Manipulator grabbing planning system and method based on visual identification

Abstract

The invention discloses a manipulator grabbing planning system and method based on visual identification, and relates to the technical field of robots, the manipulator grabbing planning system comprises a visual perception module, a coordinate conversion module, a motion planning module, a control execution module, a tail end perception module and a system integration and communication module; the visual perception module realizes multi-view target detection, semantic segmentation and three-dimensional pose estimation through a binocular depth camera; the coordinate conversion module is used for converting a target pose under a camera coordinate system into a world coordinate under a mechanical arm base coordinate system; the motion planning module is responsible for generating a mechanical arm grabbing path, optimizing a strategy and supporting generalization migration of a multi-form mechanical arm; the control execution module drives the six-axis cooperative mechanical arm to complete the grabbing action, and vision-force mixed feedback closed-loop control is achieved. The tail end sensing module monitors the grabbing state in real time through a touch sensor and dynamically adjusts a grabbing strategy; and the system integration and communication module is used for realizing real-time data interaction and cooperative work among the modules of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and specifically to a manipulator grasping planning system and method based on visual recognition. Background Technique

[0002] In the fields of modern industrial production, logistics warehousing, and services where manipulators are widely used, there are many problems in the existing technologies. Traditional manipulator grasping planning systems have limited capabilities in visual perception, mostly relying on simple visual sensors or monocular vision, making it difficult to achieve multi-view target detection, with low target recognition accuracy, unable to accurately obtain various information of the target, inaccurate image segmentation and target measurement, and unable to provide accurate basis for grasping planning. For example, it is difficult to quickly locate similar components on an industrial assembly line. The coordinate transformation method is simple and has poor accuracy, lacking dynamic calibration compensation, and it is difficult to accurately correspond the visual and manipulator motion information, affecting the grasping success rate. The motion planning is imperfect, the path planning cannot meet the kinematic constraints of the manipulator, with poor path smoothness and weak dynamic obstacle avoidance ability. The grasping strategy lacks a learning mechanism, making it difficult to adapt to different tasks and environments, and unable to achieve multi-manipulator strategy transfer. The control execution module has insufficient calculation accuracy, lacks vibration suppression means, with large manipulator motion jitter, and unable to achieve visual-force hybrid feedback closed-loop control, prone to unstable grasping or damaging objects. The end perception ability is weak, unable to monitor the surface texture change and sliding risk of the object in real time, with low force feedback accuracy and non-intelligent strategy adjustment. In addition, the system integration and communication are backward, with low data interaction efficiency among modules, lacking unified architecture support, difficult to perform dynamic re-planning in the face of anomalies, insufficient hardware synchronization accuracy, and high visual-control closed-loop delay, seriously reducing the overall performance of the system.

[0003] To solve the above problems, the present invention proposes a manipulator grasping planning system and method based on visual recognition. Summary of the Invention

[0004] The purpose of the present invention is to provide a manipulator grasping planning system and method based on visual recognition to solve the problems raised in the existing technologies.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] A manipulator grasping planning system based on visual recognition, comprising a visual perception module, a coordinate transformation module, a motion planning module, a control execution module, an end perception module, and a system integration and communication module; the visual perception module realizes multi-view target detection, semantic segmentation, and three-dimensional pose estimation through a binocular depth camera; the coordinate transformation module converts the target pose in the camera coordinate system into the world coordinate in the manipulator base coordinate system; the motion planning module is responsible for generating the manipulator grasping path and optimizing the strategy, and supports the generalization and migration of multi-form manipulators; the control execution module drives a six-axis collaborative manipulator to complete the grasping action and realizes visual-force hybrid feedback closed-loop control; the end perception module real-time monitors the grasping state through a tactile sensor and dynamically adjusts the grasping strategy; the system integration and communication module is used to realize real-time data interaction and collaborative work among the system modules.

[0007] The visual perception module includes a target detection unit, an image segmentation unit, and a target measurement unit;

[0008] The target detection unit is used to process and analyze the images in the scene to identify dynamic targets, obtain their categories and two-dimensional position information in the images, and calculate the pose information of the targets in three-dimensional space; the implementation method is as follows: First, preprocess the input images, use Gaussian filtering to denoise the images, remove the noise interference in the images, and improve the image quality; then use the histogram equalization method to enhance the image contrast and improve the recognizability of the images in low-light environments; then, based on the YOLOv5 network architecture, use the FPN structure inside the network to extract multi-level semantic information of the targets from the network feature maps at different levels, and these semantic information include the texture and edge features of the targets at the low level to the semantic category features at the high level; at the same time, combine the PointNet++ algorithm to process the point cloud data of the targets, realize the extraction of three-dimensional point cloud features, and predict the three-dimensional pose of the targets through relevant calculation methods; finally, post-process the detection results, use the non-maximum suppression algorithm to eliminate overlapping detection frames, avoid repeated detection of the same target; then use the PnP algorithm combined with RANSAC for robust estimation and output the pose of the targets in the camera coordinate system.

[0009] The image segmentation unit is used to segment the static environmental information in the scene image, separate the specific area to be segmented from the image, and provide a semantic environment description at the pixel level for the grasping planning of the manipulator, enabling the system to clearly distinguish different environmental areas. The implementation method is as follows: The DeepLabv3+ network model is adopted. This network combines the ASPP module and samples and fuses image features at different scales through dilated convolution, thereby enhancing the multi-scale feature fusion ability to capture the feature information of targets of different sizes. After being processed by the network model, pixel-level class labels are output, and each pixel is classified into the corresponding class. Then, post-processing filtering is performed on the segmentation result. Small holes in the segmentation area are filled through the closing operation in morphological operations, and adjacent target areas are connected. Then, the opening operation is used to remove small noise points in the image and retain continuous and complete areas.

[0010] The target measurement unit is used to obtain dynamic environmental parameters such as the distance between the target and the manipulator, the moving speed of the target, and the orientation angle of the target relative to the manipulator, and provide distance and motion information for the grasping planning of the manipulator. The implementation method is as follows: Based on the monocular vision geometry model, depth information is regressed through multi-frame image matching, and the target distance Z is calculated according to the formula as follows:

[0011]

[0012] where f is the camera focal length, B is the baseline length, and d is the horizontal displacement of corresponding pixel points in the left and right images. At the same time, the Lucas-Kanade algorithm is combined with IMU data. The IMU data is the data obtained from the angular rate and acceleration of the object measured in this unit. The motion trajectory of the target in consecutive image frames is analyzed, and the motion information of the target is obtained by calculating the displacement of the target pixel points in adjacent frames. Combining the frame difference method, the difference between adjacent frame images is calculated to obtain the motion area of the target, thereby obtaining the moving speed and orientation angle of the target.

[0013] The coordinate transformation module includes a hand-eye calibration unit.

[0014] The hand-eye calibration unit is used to establish the conversion relationship between the camera coordinate system and the end-effector coordinate system of the robotic arm, ensuring the correspondence between visual information and robotic arm motion information, enabling the robotic hand to grasp the target in space according to the visual recognition result. The implementation method is as follows: First, static calibration is performed. Using the Zhang-Zhengyou checkerboard calibration method, the checkerboard images collected by the camera are processed through the calibrateCamera function of OpenCV to calculate the internal parameter matrix of the camera and determine the internal parameters of the camera. Then, dynamic calibration compensation is carried out. Based on the lane vanishing point detection algorithm, the pitch angle change of the camera is monitored in real time, and the Kalman filtering algorithm is used to smooth the detected camera pitch angle data to reduce noise interference and obtain the calibration parameters. Finally, through the tf2 tool of the Robot Operating System, the conversion from the camera coordinate system to the base coordinate system of the robotic arm is realized according to the formula as follows:

[0015]

[0016] where T base represents the homogeneous transformation matrix of the base coordinate system of the robotic arm, which is used to describe the position and orientation information of the base coordinate system of the robotic arm in the global space; T gripper represents the homogeneous transformation matrix of the end-effector coordinate system of the robotic arm, reflecting the position and orientation of the end-effector coordinate system of the robotic arm; T camera is the homogeneous transformation matrix of the camera coordinate system, which describes the position and orientation of the camera coordinate system; is the inverse matrix of the homogeneous transformation matrix of the camera coordinate system; by multiplying with T gripper the conversion calculation between the camera coordinate system and the end-effector coordinate system is completed.

[0017] The motion planning module includes a path planning unit and a strategy optimization unit;

[0018] The path planning unit is used to plan the motion path for the robotic arm to grasp the target. On the premise of meeting the kinematic constraints of the robotic arm, it avoids obstacles in the working space to ensure that the robotic arm can reach the target grasping position. The implementation method is as follows: In the global planning stage, the Rapidly-exploring Random Tree Star (RRT*) algorithm is adopted to randomly sample and construct a search tree in the free space of the robotic arm, and an initial path from the starting position to the target grasping position is gradually generated. Then, combined with the A* algorithm, using the path length and target distance as heuristic functions, the initial path is optimized to improve the smoothness of the path and reduce the path turning during the motion of the robotic arm. In terms of dynamic obstacle avoidance, based on the point cloud data obtained by the lidar, the environmental map of the working space of the robotic arm is constructed in real time to identify the position and shape of the obstacles. Using the DLite algorithm, when a new obstacle is detected, the path is locally replanned to enable the robotic arm to avoid the obstacle and adjust the motion path;

[0019] The policy optimization unit is used to optimize the grasping policy of the manipulator by means of reinforcement learning, enabling the manipulator to make optimal decisions in different grasping tasks and environments, and realizing the generalization and transfer of grasping policies between multiple manipulators, thereby improving the versatility and adaptability of the system. The implementation method is as follows: A hierarchical reward mechanism is adopted to decompose the grasping task into two parts: basic reward and task reward. Based on the Deep Deterministic Policy Gradient and Experience Replay algorithms, a policy network is constructed and trained. Through continuous training and learning, the policy network can generate an optimal grasping policy according to the current environmental state and target information. At the same time, key semantic points such as the position of the manipulator fingertip and the geometric center of the object are extracted, and the interaction relationship between the object and the manipulator during the grasping process is modeled by means of interactive bisecting surface modeling, enhancing the generalization ability of the policy network in different grasping scenarios and realizing the effective transfer of grasping policies between multiple manipulators.

[0020] The control execution module includes a trajectory execution unit and a vision-force hybrid feedback unit;

[0021] The trajectory execution unit is used to convert the grasping path generated by the motion planning module into motion commands for each joint of the robotic arm, and drive the six-axis robotic arm to complete the grasping action according to the planned path. The implementation method is as follows: Use the compute_ik interface of MoveIt, combined with the KDL kinematics library, to perform inverse kinematics solution according to the kinematic model of the robotic arm, and convert the target position and pose in the Cartesian space into the angle values of each joint of the robotic arm. Then, based on a proportional-integral-derivative controller, a basic trajectory for the robotic arm joint motion is generated. By adjusting the proportional, integral, and derivative parameters, the robotic arm can track the target trajectory. At the same time, the normalized least mean square algorithm of the adaptive filter is used to process the vibration signal during the motion of the robotic arm, suppressing the jitter generated by the motion of the robotic arm, and the filtering frequency of the filter needs to be controlled within a preset threshold range;

[0022] The vision-force hybrid feedback unit is used to monitor the grasping state in real time during the grasping process of the manipulator and dynamically adjust the clamping force according to the vision information and force feedback information. The implementation method is as follows: In terms of visual servo, the camera pose error is calculated based on the image Jacobian matrix. By analyzing the difference between the target position and the desired position in the current image, the visual error e v is obtained; The PD controller is used to process the visual error, and the control quantity T is calculated according to the formula v , and the formula is as follows:

[0023]

[0024] where is the visual gain matrix, is the change rate of visual error. By adjusting the control quantity, the position of the end effector is adjusted to control the manipulator to approach the target object. In terms of force closed-loop control, the contact force information during the grasping process is collected by a six-axis force sensor installed on the manipulator. The PID controller is used to compare and process the collected contact force with the preset force threshold. When the contact force is less than the threshold, the gripper pressure is increased by 1 N each time. When the contact force is greater than the threshold, the gripper pressure is decreased by 1 N each time to prevent the object from being damaged and deformed due to excessive clamping force, and the clamping force is controlled.

[0025] The end perception module includes a visual feedback unit and a force feedback unit;

[0026] The visual feedback unit is used to continuously monitor the texture change of the object surface in real time, detect whether there is a risk of sliding during the grasping process of the object, and provide a visual basis for adjusting the grasping action of the manipulator. The implementation method is as follows: The miniature camera installed on the manipulator continuously captures the image of the object surface to obtain the object texture information. The collected image data is input into a pre-trained CNN model, which analyzes and identifies the texture features in the image to determine whether there are signs of sliding of the object. And a sliding threshold is set. When the model detects that the displacement of the object surface texture between adjacent frames exceeds this threshold, it is determined that there is a risk of sliding. At the same time, the Lucas-Kanade method is used to calculate the movement speed of the object in the image. When the calculated speed exceeds the preset threshold, the motion planning module is triggered to re-plan the grasping path to ensure the stability of the grasping process;

[0027] The force feedback unit is used to accurately measure the magnitude of the contact force when the manipulator grasps the object and adjust the clamping strategy according to the change of the force. The implementation method is as follows: First, calibrate the six-axis force sensor. By loading a standard weight on the sensor, the output data of the sensor is obtained, and the sensor is calibrated according to the weight of the standard weight to make the measurement error of the sensor less than the preset threshold range. Then, a Butterworth filter is used to filter the signal collected by the force sensor to suppress high-frequency noise interference. During the grasping process, the change of the force is monitored in real time. When it is detected that the force mutation ≥ 10 N, it is determined that a collision has occurred. When it is detected that the force drops suddenly ≥ 5 N, it is determined that a slippage has occurred. When collisions and slippages occur, the system triggers the motion planning module to regenerate the path and readjust the grasping action of the manipulator.

[0028] The system integration and communication module includes the following:

[0029] This module is used to achieve real-time data interaction and collaborative work among various system modules to ensure the operation of the system. The implementation method is as follows: The ROS2 architecture is adopted, and the topic mechanism is used to realize data publishing and subscribing among different modules, enabling each module to obtain the data published by other modules according to its own needs; the action mechanism is used to realize task interaction among modules, supporting the dynamic replanning function. When the system detects abnormal situations, each module can work together to make corresponding adjustments; at the same time, a hardware timer is used to synchronize the system hardware to ensure that the vision-control closed-loop delay is lower than the preset threshold range, guaranteeing the system's real-time response and control of visual information.

[0030] A manipulator grasping planning method based on visual recognition includes the following steps:

[0031] S1. Through the target detection unit, image segmentation unit, and target measurement unit of the visual perception module, the scene image is processed and analyzed according to their respective implementation methods to obtain the category, position, pose, distance, speed, orientation angle information of the target, and the semantic segmentation information of the static environment;

[0032] S2. Using the hand-eye calibration unit of the coordinate transformation module, the conversion relationship between the camera coordinate system and the end coordinate system of the robotic arm is established through static calibration, dynamic calibration compensation, and coordinate transformation methods;

[0033] S3. With the help of the path planning unit and strategy optimization unit of the motion planning module, the global planning, dynamic obstacle avoidance, and reinforcement learning methods are respectively adopted to generate the grasping path of the robotic arm and optimize the grasping strategy;

[0034] S4. Using the trajectory execution unit and vision-force hybrid feedback unit of the control execution module, through inverse kinematics solution, control algorithms, and feedback control methods, the planned path is converted into the actions of the robotic arm, and the grasping state is monitored in real time to dynamically adjust the clamping force;

[0035] S5. Through the visual feedback unit and force feedback unit of the end perception module, according to the image processing and force sensor data processing methods, the surface state of the object is detected and the contact force is quantified to trigger corresponding action adjustments;

[0036] S6. Through the system integration and communication module, using the ROS2 architecture and hardware synchronization methods, real-time data interaction and collaborative work among modules are achieved.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] 1. Precise Visual Perception: At the level of visual perception, through a binocular depth camera combined with advanced algorithms, multi-view object detection, precise semantic segmentation, and three-dimensional pose estimation are achieved. It can accurately identify objects and obtain detailed information in complex scenarios, while accurately segmenting the static environment, providing a high-resolution semantic environment description for grasping planning. It can also precisely measure the dynamic parameters of the object and the manipulator, significantly improving the accuracy and comprehensiveness of visual perception.

[0039] 2. Accurate Coordinate Transformation: The coordinate transformation module uses advanced calibration methods and tools to accurately establish the transformation relationship between the camera coordinate system and the end-effector coordinate system of the robotic arm, ensuring the accurate matching of visual information and robotic arm motion information. This enables the manipulator to accurately grasp objects in space based on the visual recognition results, effectively improving the grasping accuracy and success rate.

[0040] 3. Intelligent Motion Planning: The motion planning module adopts innovative algorithms. It can not only quickly plan a smooth grasping path and avoid obstacles in real time under the premise of meeting the kinematic constraints of the robotic arm, but also optimize the grasping strategy through reinforcement learning to achieve the generalization and migration of multi-form manipulators, significantly enhancing the adaptability and versatility of the system in different tasks and environments, and reducing the development and debugging costs.

[0041] 4. Reliable End-Effector Sensing: The end-effector sensing module uses a micro camera and a professional model to continuously monitor the surface texture changes of the object and the risk of slipping in real time. Combined with high-precision force sensors and intelligent gripping strategy adjustment, it significantly enhances the monitoring and control capabilities of the system during the grasping process, improving the safety and stability of grasping. Brief Description of the Drawings

[0042] Figure 1 It is a schematic diagram of the system architecture of a robotic arm grasping planning system based on visual recognition according to the present invention;

[0043] Figure 2 It is a schematic diagram of the method flow of a robotic arm grasping planning method based on visual recognition according to the present invention. Detailed Embodiments

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0045] Embodiment: As Figure 1 - Figure 2 shown, the present invention provides a technical solution,

[0046] A manipulator grasping planning system based on visual recognition, comprising a visual perception module, a coordinate transformation module, a motion planning module, a control execution module, an end perception module, and a system integration and communication module; the visual perception module realizes multi-view target detection, semantic segmentation, and three-dimensional pose estimation through a binocular depth camera; the coordinate transformation module converts the target pose in the camera coordinate system into the world coordinates in the manipulator base coordinate system; the motion planning module is responsible for generating the manipulator grasping path and optimizing the strategy, and supports the generalization and migration of multi-form manipulators; the control execution module drives a six-axis collaborative manipulator to complete the grasping action and realizes visual-force hybrid feedback closed-loop control; the end perception module real-time monitors the grasping state through a tactile sensor and dynamically adjusts the grasping strategy; the system integration and communication module is used to realize real-time data interaction and collaborative work among the system modules.

[0047] The visual perception module includes a target detection unit, an image segmentation unit, and a target measurement unit;

[0048] The target detection unit is used to process and analyze the images in the scene to identify dynamic targets, obtain their categories, two-dimensional position information in the images, and at the same time calculate the pose information of the targets in three-dimensional space; the implementation method is as follows: First, preprocess the input images, use Gaussian filtering to denoise the images, remove the noise interference in the images, and improve the image quality; then use the histogram equalization method to enhance the image contrast and improve the recognizability of the images in low-light environments; then, based on the YOLOv5 network architecture, use the FPN structure inside its network to extract multi-level semantic information of the targets from the network feature maps at different levels. These semantic information include the texture and edge features of the targets at the low level to the semantic category features at the high level; at the same time, combine the PointNet++ algorithm to process the point cloud data of the targets, realize the extraction of three-dimensional point cloud features, and predict the three-dimensional poses of the targets through relevant calculation methods; finally, post-process the detection results, use the non-maximum suppression algorithm to eliminate overlapping detection frames, and avoid repeated detection of the same target; then use the PnP algorithm combined with RANSAC for robust estimation and output the pose of the target in the camera coordinate system; this way of fusing image features and point cloud data is different from the traditional target detection methods that only rely on single visual information, and can more accurately determine the position and pose of the targets in three-dimensional space, providing key basic data for subsequent grasping planning.

[0049] The image segmentation unit is used to segment the static environmental information in the scene image, separate the specific area to be segmented from the image, and provide a pixel-level semantic environment description for the grasping planning of the manipulator, enabling the system to clearly distinguish different environmental areas. The implementation method is as follows: The DeepLabv3+ network model is adopted. This network combines the ASPP module and samples and fuses image features at different scales through dilated convolution, thereby enhancing the multi-scale feature fusion ability to capture the feature information of targets of different sizes. After being processed by the network model, pixel-level class labels are output, and each pixel is classified into the corresponding category. Then, post-processing filtering is performed on the segmentation result. Small holes in the segmentation area are filled through the closing operation in morphological operations, and adjacent target areas are connected. Then, the opening operation is used to remove small noise points in the image and retain continuous and complete areas.

[0050] The target measurement unit is used to obtain dynamic environmental parameters such as the distance between the target and the manipulator, the moving speed of the target, and the orientation angle of the target relative to the manipulator, and provide distance and motion information for the grasping planning of the manipulator. The implementation method is as follows: Based on the monocular vision geometry model, depth information is regressed through multi-frame image matching, and the target distance Z is calculated according to the formula as follows:

[0051]

[0052] where f is the camera focal length, B is the baseline length, and d is the horizontal displacement of corresponding pixel points in the left and right images. At the same time, the Lucas-Kanade algorithm is combined with IMU data, where the IMU data is the data of the angular rate and acceleration of the object measured in this unit, to analyze the motion trajectory of the target in consecutive image frames, and the motion information of the target is obtained by calculating the displacement of the target pixel points in adjacent frames. Combining the inter-frame difference method, the difference between adjacent frame images is calculated to obtain the motion area of the target, thereby obtaining the moving speed and orientation angle of the target. This method of multi-modal data fusion can more accurately perceive the dynamic state of the target and effectively solve the problems of insufficient accuracy and large environmental influence in measuring the dynamic parameters of the target, providing more reliable dynamic environmental information for the grasping planning of the manipulator.

[0053] The coordinate transformation module includes a hand-eye calibration unit.

[0054] The hand-eye calibration unit is used to establish the conversion relationship between the camera coordinate system and the end-effector coordinate system of the robotic arm, ensuring the correspondence between visual information and robotic arm motion information, enabling the robotic hand to grasp the target in space according to the visual recognition result. The implementation method is as follows: First, static calibration is performed. Using the Zhang Zhengyou checkerboard calibration method, the checkerboard images collected by the camera are processed through the calibrateCamera function of OpenCV to calculate the internal parameter matrix of the camera and determine the internal parameters of the camera. Then, dynamic calibration compensation is carried out. Based on the lane vanishing point detection algorithm, the pitch angle change of the camera is monitored in real time, and the Kalman filtering algorithm is used to smooth the detected camera pitch angle data. As an optimal estimation algorithm, Kalman filtering can effectively reduce noise interference and obtain more accurate calibration parameters. Finally, through the tf2 tool of the Robot Operating System, the conversion from the camera coordinate system to the base coordinate system of the robotic arm is achieved according to the formula. The formula is as follows:

[0055]

[0056] where T bas represents the homogeneous transformation matrix of the base coordinate system of the robotic arm, which is used to describe the position and orientation information of the base coordinate system of the robotic arm in the global space; T gripp represents the homogeneous transformation matrix of the end-effector coordinate system of the robotic arm, reflecting the position and orientation of the end-effector coordinate system of the robotic arm; T camera is the homogeneous transformation matrix of the camera coordinate system, which describes the position and orientation of the camera coordinate system; is the inverse matrix of the homogeneous transformation matrix of the camera coordinate system; by multiplying with T gripp the conversion calculation between the camera coordinate system and the end-effector coordinate system of the robotic arm is completed. This dynamic calibration compensation mechanism fully considers the possible changes in the pose of the camera during actual use, significantly improving the accuracy and stability of the conversion relationship between the camera coordinate system and the end-effector coordinate system of the robotic arm, and ensuring the precise correspondence between visual information and robotic arm motion information.

[0057] The motion planning module includes a path planning unit and a strategy optimization unit;

[0058] The path planning unit is used to plan the motion path for the robotic arm to grasp the target. On the premise of satisfying the kinematic constraints of the robotic arm, it avoids obstacles in the working space to ensure that the robotic arm can reach the target grasping position. The implementation method is as follows: In the global planning stage, the Rapidly-Exploring Random Tree Star (RRT*) algorithm is adopted to randomly sample in the free space of the robotic arm and construct a search tree, gradually generating an initial path from the starting position to the target grasping position. Then, in combination with the A* algorithm, using the path length and the distance to the target as heuristic functions, the initial path is optimized to improve the smoothness of the path and reduce the path turning during the movement of the robotic arm. In terms of dynamic obstacle avoidance, based on the point cloud data obtained by the lidar, the environmental map of the working space of the robotic arm is constructed in real time, marking the positions and shapes of the obstacles. The DLite algorithm is used to locally replan the path when a new obstacle is detected, enabling the robotic arm to avoid the obstacle and adjust the motion path.

[0059] Among them, the Rapidly-Exploring Random Tree Star (RRT*) algorithm randomly samples points in the free space of the robotic arm and gradually constructs a search tree. After each new point is sampled, the nearest node in the tree to this point is found. If the path between the new point and the nearest node has no collision, the new point is added to the tree. As the number of sampled points increases, the search tree gradually covers the entire free space, thus finding an initial path from the starting position to the target grasping position. The A* algorithm calculates the cost function for each node. The cost function usually consists of two parts: the actual cost (path length) from the starting point to the current node and the estimated cost (such as Euclidean distance) from the current node to the target node. By continuously selecting the node with the minimum cost function for expansion, the optimal path is finally found. In the present invention, the A* algorithm optimizes the initial path generated by the Rapidly-Exploring Random Tree Star (RRT*) algorithm to make the path smoother.

[0060] The dynamic obstacle avoidance solution based on lidar point cloud data is that the lidar emits laser beams and receives the reflected light, calculates the distance information of surrounding objects based on the propagation time of the light, and forms point cloud data. Through the processing and analysis of this point cloud data, the real-time environmental map of the working space of the robotic arm is constructed to clarify the positions and shapes of the obstacles. The DLite algorithm is an incremental path search algorithm based on the Dijkstra algorithm. It utilizes the results of previous searches. When a new obstacle is detected, it quickly finds a new path to avoid the obstacle by locally updating the search tree, realizing the dynamic obstacle avoidance of the robotic arm. This way of combining global planning and dynamic obstacle avoidance can better adapt to complex and changing working environments, ensuring that the robotic arm can efficiently and safely avoid obstacles during the process of grasping the target and stably reach the target grasping position.

[0061] The strategy optimization unit is used to optimize the grasping strategy of the manipulator by means of reinforcement learning, enabling the manipulator to make optimal decisions in different grasping tasks and environments, and realizing the generalization and transfer of the grasping strategies between multiple manipulators, thereby improving the versatility and adaptability of the system. The implementation method is as follows: a hierarchical reward mechanism is adopted, and the grasping task is decomposed into two parts: basic reward and task reward; a policy network is constructed and trained based on the Deep Deterministic Policy Gradient and Experience Replay algorithms. Through continuous training and learning, the policy network can generate an optimal grasping strategy according to the current environmental state and target information; at the same time, key semantic points such as the position of the manipulator fingertip and the geometric center of the object are extracted, and the interaction relationship between the object and the manipulator during the grasping process is modeled by means of interactive bisecting surface modeling, enhancing the generalization ability of the policy network in different grasping scenarios and realizing the effective transfer of the grasping strategies between multiple manipulators. This grasping strategy optimization method enables the manipulator to make optimal decisions in different grasping tasks and environments, significantly improving the versatility and adaptability of the system.

[0062] The control execution module includes a trajectory execution unit and a vision-force hybrid feedback unit;

[0063] The trajectory execution unit is used to convert the grasping path generated by the motion planning module into motion commands for each joint of the robotic arm, driving the six-axis robotic arm to complete the grasping action according to the planned path. The implementation method is as follows: use the compute_ik interface of MoveIt, combined with the KDL kinematics library, to perform inverse kinematics solution according to the kinematic model of the robotic arm, and convert the target position and pose in the Cartesian space into the angle values of each joint of the robotic arm; then, generate the basic trajectory of the robotic arm joint movement based on a proportional-integral-derivative controller. By adjusting the proportional, integral, and derivative parameters, the robotic arm can track the target trajectory; at the same time, use the normalized least mean square algorithm of the adaptive filter to process the vibration signal during the movement of the robotic arm, suppressing the jitter generated by the movement of the robotic arm, and the filtering frequency of the filter needs to be controlled within the preset threshold range;

[0064] The vision-force hybrid feedback unit is used to monitor the grasping state in real time during the grasping process of the manipulator and dynamically adjust the clamping force according to the visual information and force feedback information. The implementation method is as follows: in terms of visual servo, calculate the camera pose error based on the image Jacobian matrix. By analyzing the difference between the target position and the desired position in the current image, obtain the visual error e v ; use a PD controller to process the visual error and calculate the control quantity T according to the formula v , and the formula is as follows:

[0065]

[0066] where is the visual gain matrix, is the change rate of visual error. By adjusting the control quantity, the position of the end effector is adjusted to control the manipulator to approach the target object. In terms of force closed-loop control, the contact force information during the grasping process is collected by a six-axis force sensor installed on the manipulator. The PID controller is used to compare and process the collected contact force with the preset force threshold. When the contact force is less than the threshold, the gripper pressure is increased by 1N each time. When the contact force is greater than the threshold, the gripper pressure is decreased by 1N each time to prevent the object from being damaged and deformed due to excessive clamping force, and the control of the clamping force is realized. This control method that combines visual and force feedback enables the manipulator to adjust its actions in real time according to visual and force information during the grasping process, ensuring stable grasping and avoiding object damage. Compared with the single feedback control method, the accuracy and reliability of the grasping operation are improved.

[0067] The end perception module includes a visual feedback unit and a force feedback unit;

[0068] The visual feedback unit is used to continuously monitor the texture change of the object surface, detect whether there is a sliding risk during the grasping process of the object, and provide a visual basis for adjusting the grasping action of the manipulator. The implementation method is as follows: The micro camera installed on the manipulator continuously captures the image of the object surface to obtain the object texture information. The collected image data is input into a pre-trained CNN model, which analyzes and identifies the texture features in the image to judge whether there are sliding signs on the object. And a sliding threshold is set. When the model detects that the displacement of the object surface texture between adjacent frames exceeds this threshold, it is determined that there is a sliding risk. At the same time, the Lucas-Kanade method is used to calculate the movement speed of the object in the image. When the calculated speed exceeds the preset threshold, the motion planning module is triggered to re-plan the grasping path to ensure the stability of the grasping process. This mechanism for continuously monitoring the texture change and sliding risk of the object surface provides a timely visual basis for adjusting the grasping action of the manipulator, improving the safety and stability of the grasping process;

[0069] The force feedback unit is used to accurately measure the magnitude of the contact force when the manipulator grasps an object, and adjust the clamping strategy according to the change of the force. The implementation method is as follows: First, calibrate the six-axis force sensor. By loading standard weights onto the sensor, obtain the output data of the sensor, and calibrate the sensor according to the weight of the standard weights to make the measurement error of the sensor less than the preset threshold range. Then, use a Butterworth filter to filter the signals collected by the force sensor to suppress high-frequency noise interference. During the grasping process, continuously monitor the change of the force. When it is detected that the force mutation ≥ 10N, it is determined that a collision has occurred. When it is detected that the force drops suddenly ≥ 5N, it is determined that a slippage has occurred. In case of collision and slippage, the system triggers the motion planning module to regenerate the path and readjust the grasping action of the manipulator. This force feedback control mechanism can timely respond to abnormal force changes during the grasping process, ensure the safety of the manipulator during the grasping process, and effectively avoid object damage and manipulator failures.

[0070] The system integration and communication module includes the following:

[0071] This module is used to realize real-time data interaction and collaborative work among the system modules to ensure the operation of the system. The implementation method is as follows: Adopt the ROS2 architecture, and use the topic mechanism to realize data publishing and subscribing among different modules, so that each module can obtain the data published by other modules according to its own needs. Realize task interaction among modules through the action mechanism, support the dynamic replanning function, and when the system detects an abnormal situation, each module can work together to make corresponding adjustments. At the same time, use a hardware timer to synchronize the system hardware to ensure that the vision-control closed-loop delay is lower than the preset threshold range, and ensure the real-time response and control of the system to visual information. This loosely coupled data interaction method improves the scalability and flexibility of the system.

[0072] A manipulator grasping planning method based on visual recognition includes the following steps:

[0073] S1. Through the target detection unit, image segmentation unit, and target measurement unit of the visual perception module, process and analyze the scene image according to their respective implementation methods to obtain the category, position, pose, distance, speed, orientation angle information of the target, as well as the semantic segmentation information of the static environment.

[0074] S2. Use the hand-eye calibration unit of the coordinate transformation module to establish the conversion relationship between the camera coordinate system and the end-effector coordinate system through static calibration, dynamic calibration compensation, and coordinate transformation methods.

[0075] S3. With the help of the path planning unit and strategy optimization unit of the motion planning module, respectively adopt global planning, dynamic obstacle avoidance, and reinforcement learning methods to generate the manipulator grasping path and optimize the grasping strategy.

[0076] S4. Use the trajectory execution unit and the vision-force hybrid feedback unit of the control execution module. Through inverse kinematics solution, control algorithms, and feedback control methods, convert the planned path into the actions of the robotic arm and monitor the grasping state in real time, dynamically adjusting the clamping force.

[0077] S5. Through the vision feedback unit and the force feedback unit of the end perception module, based on image processing and force sensor data processing methods, detect the surface state of the object and quantify the contact force, triggering corresponding action adjustments.

[0078] S6. Through the system integration and communication module, using the ROS2 architecture and hardware synchronization methods, achieve real-time data interaction and collaborative work among modules.

[0079] Embodiment:

[0080] Suppose in an industrial production scenario, a robotic arm is required to grasp small parts on an assembly line.

[0081] I. Vision Perception Module

[0082] 1. Object Detection Unit: A binocular depth camera captures an image containing small parts. In the image preprocessing stage, Gaussian filtering (with a standard deviation set to 1.5) is used to remove noise, and then histogram equalization is applied to enhance the contrast. Based on the YOLOv5 network and the FPN structure, combined with the PointNet++ algorithm, the part is successfully identified as a certain type of capacitor, and its two-dimensional position in the image is at (x = 100, y = 150) pixels. Through relevant calculations, its three-dimensional pose is predicted. Suppose the rotation angle around the x-axis is 10°, the rotation angle around the y-axis is -5°, and the rotation angle around the z-axis is 3°. The non-maximum suppression algorithm is used to remove overlapping detection frames, and then the PnP algorithm combined with RANSAC is used for robust estimation to obtain the pose of the part in the camera coordinate system.

[0083] 2. Image Segmentation Unit: The DeepLabv3+ network combined with the ASPP module is used to perform semantic segmentation on the image, successfully separating regions such as the assembly line, background, and small parts, and outputting pixel-level class labels. After closing operations to fill small holes and opening operations to remove small noise points, a clear semantic environment description is provided for grasping planning.

[0084] 3. Target Measurement Unit: Calculate the distance between the part and the manipulator based on the monocular vision geometric model combined with IMU data. Assume the camera focal length f = 50mm, the baseline length B = 100mm, and the horizontal displacement d = 20 pixels of the corresponding pixel points of the left and right images obtained through multi-frame image matching. According to the formula, the distance between the part and the manipulator Z = 250mm is calculated; using the Lucas-Kanade algorithm combined with the inter-frame difference method, analyze the motion trajectory of the part in consecutive image frames, and calculate the part motion speed to be 5mm / s and the orientation angle to be 30°.

[0085] II. Coordinate Transformation Module

[0086] 1. Hand-Eye Calibration Unit: Use the Zhang-Zhengyou checkerboard calibration method to calculate the camera internal parameter matrix through the calibrateCamera function of OpenCV. In the dynamic calibration compensation stage, monitor the change of the camera pitch angle based on the lane vanishing point detection algorithm. Assume that the pitch angle change value is detected at a certain moment, and after Kalman filtering and smoothing processing, the relevant calibration parameters are obtained. Finally, through the tf2 tool of the robot operating system, according to the formula, complete the conversion from the camera coordinate system to the manipulator base coordinate system. Assume T gripp 、T camera is the known homogeneous transformation matrix, and the world coordinates of the part in the manipulator base coordinate system are calculated.

[0087] III. Motion Planning Module

[0088] 1. Path Planning Unit: In global planning, use the Rapidly-Exploring Random Tree Star (RRT*) algorithm to generate an initial path from the starting position to the target grasping position in the manipulator free space. Assume that there are many turns in the initial path, and combine the A * algorithm, using the path length and the target distance as heuristic functions, the smoothness of the optimized path is significantly improved. In dynamic obstacle avoidance, the lidar obtains point cloud data to construct a real-time environment map. Assume that a new obstacle is detected, and use the DLite algorithm to perform local replanning of the path, so that the manipulator can avoid the obstacle and reach the target grasping position.

[0089] 2. Policy Optimization Unit: Adopt a hierarchical reward mechanism, with the basic reward set to 1 point for successfully approaching the part, and the task reward set to 5 points for stably grasping the part and placing it at the specified position. Build and train a policy network based on the Deep Deterministic Policy Gradient (DDPG) and Experience Replay algorithms. Extract key semantic points such as the manipulator fingertip position and the part geometric center, and enhance the generalization ability through interactive binary surface modeling. After multiple trainings, the policy network can generate the optimal grasping policy according to the current environmental state and target information.

[0090] IV. Control Execution Module

[0091] 1. Trajectory Execution Unit: Using the compute_ik interface of MoveIt in combination with the KDL kinematics library, inverse kinematics is solved according to the robotic arm kinematic model, converting the target position and orientation in Cartesian space into the angular values of each joint of the robotic arm. Based on a proportional-integral-derivative (PID) controller, a basic trajectory for the robotic arm joint movement is generated, assuming a proportional parameter K p = 0.5, an integral parameter K i = 0.1, and a derivative parameter K d = 0.2. The normalized least mean square algorithm of the adaptive filter is used to process the robotic arm movement vibration signal. Assuming the filtering frequency is set to 250 Hz, the robotic arm movement jitter is suppressed, and the robotic arm is driven to move along the planned path.

[0092] 2. Vision-Force Hybrid Feedback Unit: In terms of visual servo, based on the image Jacobian matrix, the camera pose error is calculated. Assuming the visual gain matrix is as follows:

[0093]

[0094] According to the difference between the actual position and the desired position of the part in the current image, the visual error e v is obtained, and the visual error change rate is calculated. The position of the end effector is adjusted according to the formula. In terms of force closed-loop control, a six-axis force sensor collects the contact force. Assuming the preset force threshold is 8 N, when the contact force is less than 8 N, the gripper pressure is increased by 1 N each time; when it is greater than 8 N, the gripper pressure is decreased by 1 N each time, achieving precise control of the gripping force.

[0095] V. End Perception Module

[0096] 1. Vision Feedback Unit: The micro camera installed on the robotic arm continuously captures the surface image of the part and inputs the image data into the pre-trained CNN model. Assuming the sliding threshold is set to 0.1 pixel / frame, when the model detects that the displacement of the part surface texture between adjacent frames exceeds this threshold, it is determined that there is a sliding risk. The Lucas-Kanade method is used to calculate the part movement speed. Assuming the preset speed threshold is 10 mm / s, when the calculated speed exceeds this threshold, the motion planning module is triggered to re-plan the grasping path.

[0097] 2. Force Feedback Unit: The six-axis force sensor is calibrated using standard weights to make the measurement error less than 0.1 N. A Butterworth filter is used to suppress high-frequency noise. During the grasping process, assuming that the detected force mutation is greater than or equal to 10 N, it is determined that a collision has occurred; if the force drops suddenly by more than or equal to 5 N, it is determined that a slippage has occurred. Once a collision or slippage occurs, the motion planning module is triggered to regenerate the path to adjust the grasping action.

[0098] VI. System Integration and Communication Module

[0099] Using the ROS2 architecture, each module publishes and subscribes to data through the topic mechanism. For example, the visual perception module publishes the processed target information to the " / target_info" topic, and the motion planning module subscribes to this topic to obtain information for path planning. Task interaction is achieved through the action mechanism. For instance, when the control execution module detects a grasping anomaly, it sends an action request to the motion planning module for dynamic replanning. The system hardware is synchronized using a hardware timer. Assuming the preset visual-control closed-loop delay threshold is 50 ms, it ensures the system's real-time response and control of visual information, enabling real-time data interaction and collaborative work among modules.

[0100] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any perspective, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Thus, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed by the present invention. Any reference signs in the claims should not be construed as limiting the claimed rights.

Claims

1. A manipulator grasping planning system based on visual recognition, characterized in that: It includes a visual perception module, a coordinate transformation module, a motion planning module, a control execution module, an end perception module, and a system integration and communication module; the visual perception module realizes multi-view target detection, semantic segmentation, and three-dimensional pose estimation through a binocular depth camera; the coordinate transformation module converts the target pose in the camera coordinate system into the world coordinate in the robotic arm base coordinate system; the motion planning module is responsible for generating the robotic arm grasping path and optimizing the strategy, and supports the generalization and migration of multi-form manipulators; the control execution module drives the six-axis collaborative robotic arm to complete the grasping action and realizes the visual-force hybrid feedback closed-loop control; the end perception module monitors the grasping state in real time through a tactile sensor and dynamically adjusts the grasping strategy; the system integration and communication module is used to realize the real-time data interaction and collaborative work among the system modules.

2. The manipulator grasping planning system based on visual recognition according to claim 1, characterized in that: The visual perception module includes a target detection unit, an image segmentation unit, and a target measurement unit; The target detection unit is used to process and analyze the images in the scene to identify dynamic targets, obtain their categories, two-dimensional position information in the image, and calculate the pose information of the targets in the three-dimensional space; the implementation method is as follows: First, preprocess the input images, use Gaussian filtering to denoise the images, remove the noise interference in the images, and improve the image quality; then use the histogram equalization method to enhance the image contrast and improve the recognizability of the images in low-light environments; then, based on the YOLOv5 network architecture, use the FPN structure inside the network to extract the multi-level semantic information of the targets from the network feature maps at different levels, and these semantic information include the target's low-level texture and edge features to high-level semantic category features; at the same time, combine the PointNet++ algorithm to process the point cloud data of the targets, realize the extraction of three-dimensional point cloud features, and predict the three-dimensional poses of the targets through relevant calculation methods; finally, post-process the detection results, use the non-maximum suppression algorithm to eliminate the overlapping detection frames, avoid the repeated detection of the same target; then use the PnP algorithm combined with RANSAC for robust estimation, and output the pose of the target in the camera coordinate system. The image segmentation unit is used to segment the static environment information in the scene images, separate the areas to be segmented from the images, and provide a pixel-level semantic environment description for the grasping planning of the manipulator, so that the system can clearly distinguish different environmental areas; the implementation method is as follows: Adopt the DeepLabv3+ network model, which combines the ASPP module, and sample and fuse the image features at different scales through dilated convolution, so as to enhance the multi-scale feature fusion ability to capture the feature information of targets of different sizes. After being processed by the network model, output the pixel-level class labels and divide each pixel into the corresponding class; finally, post-process and filter the segmentation results, fill the small holes in the segmentation area through the closing operation in morphological operations, and connect the adjacent target areas. Then use the opening operation to remove the small noise points in the image and retain the continuous and complete areas. The target measurement unit is used to obtain dynamic environment parameters such as the distance between the target and the manipulator, the moving speed of the target, and the orientation angle of the target relative to the manipulator, providing distance and motion information for the grasping planning of the manipulator. The implementation method is as follows: Based on the monocular vision geometric model, the depth information is regressed through multi-frame image matching, and the target distance Z is calculated according to the formula as follows: where f is the camera focal length, B is the baseline length, and d is the horizontal displacement of corresponding pixel points in the left and right images; at the same time, the Lucas-Kanade algorithm is combined with IMU data, where the IMU data is the data of the angular rate and acceleration of the object measured in this unit, to analyze the motion trajectory of the target in consecutive image frames, and the motion information of the target is obtained by calculating the displacement of the target pixel points in adjacent frames; Combined with the inter-frame difference method, the difference between adjacent frame images is calculated to obtain the motion area of the target, thereby obtaining the moving speed and orientation angle of the target.

3. A robotic arm grasping planning system based on visual recognition according to claim 1, characterized in that: The coordinate transformation module includes a hand-eye calibration unit; The hand-eye calibration unit is used to establish the transformation relationship between the camera coordinate system and the end-effector coordinate system of the manipulator, ensuring the correspondence between visual information and manipulator motion information, enabling the manipulator to grasp the target in space according to the visual recognition result. The implementation method is as follows: First, static calibration is performed. Using the Zhang-Zhengyou checkerboard calibration method, the checkerboard images collected by the camera are processed through the calibrateCamera function of OpenCV to calculate the internal parameter matrix of the camera and determine the internal parameters of the camera; then, dynamic calibration compensation is carried out. Based on the vanishing point detection algorithm of the lane line, the pitch angle change of the camera is monitored in real time, and the Kalman filtering algorithm is used to smooth the detected camera pitch angle data to reduce noise interference and obtain the calibration parameters; finally, through the tf2 tool of the robot operating system, the conversion from the camera coordinate system to the base coordinate system of the manipulator is realized according to the formula as follows: where it is T base represents the homogeneous transformation matrix of the base coordinate system of the robotic arm, which is used to describe the position and attitude information of the base coordinate system of the robotic arm in the global space; T gripper represents the homogeneous transformation matrix of the end coordinate system of the robotic arm, reflecting the position and orientation of the end coordinate system of the robotic arm; T camera is the homogeneous transformation matrix of the camera coordinate system, describing the position and orientation of the camera coordinate system; is the inverse matrix of the homogeneous transformation matrix of the camera coordinate system; by multiplying with T gripper the conversion calculation between the camera coordinate system and the end-effector coordinate system of the robotic arm is completed.

4. The manipulator grasping planning system based on visual recognition according to claim 1, wherein: The motion planning module includes a path planning unit and a strategy optimization unit; The path planning unit is used to plan the motion path for the manipulator to grasp the target. On the premise of satisfying the kinematic constraints of the manipulator, it avoids obstacles in the working space to ensure that the manipulator can reach the target grasping position. The implementation method is as follows: In the global planning stage, the rapidly-exploring random tree star (RRT*) algorithm is adopted to randomly sample and construct a search tree in the free space of the manipulator, and an initial path from the starting position to the target grasping position is gradually generated; then, combined with the A* algorithm, using the path length and target distance as heuristic functions, the initial path is optimized to improve the smoothness of the path and reduce the path turning during the motion of the manipulator; in terms of dynamic obstacle avoidance, based on the point cloud data obtained by the lidar, the environmental map of the manipulator working space is constructed in real time to identify the positions and shapes of obstacles; the DLite algorithm is used to perform local replanning of the path when a new obstacle is detected, enabling the manipulator to avoid obstacles and adjust the motion path; The strategy optimization unit is used to optimize the grasping strategy of the manipulator by means of reinforcement learning, enabling the manipulator to make optimal decisions under different grasping tasks and environments, and realizing the generalization and transfer of grasping strategies among multiple manipulators, so as to improve the versatility and adaptability of the system. The implementation method is as follows: adopt a hierarchical reward mechanism, and decompose the grasping task into two parts: basic reward and task reward; construct and train a policy network based on the deep deterministic policy gradient and experience replay algorithm. Through continuous training and learning, the policy network can generate an optimal grasping strategy according to the current environmental state and target information; at the same time, extract key semantic points such as the position of the manipulator fingertip and the geometric center of the object, and model the interaction relationship between the object and the manipulator during the grasping process through the interactive bisecting surface modeling method, so as to enhance the generalization ability of the policy network in different grasping scenarios and realize the effective transfer of grasping strategies among multiple manipulators.

5. A robotic manipulator grasping planning system based on visual recognition according to claim 1, characterized in that: The control execution module includes a trajectory execution unit and a vision-force hybrid feedback unit; The trajectory execution unit is used to convert the grasping path generated by the motion planning module into the motion instructions of each joint of the robotic arm, and drive the six-axis robotic arm to complete the grasping action according to the planned path. The implementation method is as follows: use the compute_ik interface of MoveIt, combined with the KDL kinematics library, to perform inverse kinematics solution according to the kinematic model of the robotic arm, and convert the target position and pose in the Cartesian space into the angle values of each joint of the robotic arm; then, generate the basic trajectory of the robotic arm joint movement based on the proportional-integral-derivative controller, and by adjusting the proportional, integral and derivative parameters, enable the robotic arm to track the target trajectory; at the same time, use the normalized least mean square algorithm of the adaptive filter to process the vibration signal during the movement of the robotic arm, suppress the jitter generated by the movement of the robotic arm, and the filtering frequency of the filter needs to be controlled within the preset threshold range; The visual-force hybrid feedback unit is used to monitor the grasping state in real time during the manipulator grasping process and dynamically adjust the clamping force according to visual information and force feedback information. The implementation method is as follows: in terms of visual servo, the camera pose error is calculated based on the image Jacobian matrix, and the visual error e is obtained by analyzing the difference between the target position and the desired position in the current image. v The PD controller is used to process the visual error, and the control quantity T is calculated according to the formula. v The formula is as follows: wherein is the visual gain matrix, is the change rate of visual error. The position of the end effector is adjusted by adjusting the control quantity to control the manipulator to approach the target object. In terms of force closed-loop control, the contact force information during the grasping process is collected by a six-axis force sensor installed on the manipulator. The PID controller is used to compare and process the collected contact force with the preset force threshold. When the contact force is less than the threshold, the gripper pressure is increased by 1N each time; when the contact force is greater than the threshold, the gripper pressure is decreased by 1N each time to prevent damage and deformation of the object caused by excessive clamping force, and the control of the clamping force is realized.

6. The robotic arm grasping planning system based on visual recognition according to claim 1, wherein: The end perception module includes a vision feedback unit and a force feedback unit; The vision feedback unit is used to continuously monitor the texture change of the object surface in real time, detect whether there is a risk of sliding during the grasping process of the object, and provide a visual basis for adjusting the grasping action of the manipulator. The implementation method is as follows: continuously capture the image of the object surface through a micro camera installed on the manipulator to obtain the object texture information; input the collected image data into a pre-trained CNN model, and this model analyzes and identifies the texture features in the image to judge whether there are signs of sliding of the object; And set a sliding threshold. When the model detects that the displacement of the object surface texture between adjacent frames exceeds this threshold, it is determined that there is a risk of sliding; at the same time, use the Lucas-Kanade method to calculate the movement speed of the object in the image. When the calculated speed exceeds the preset threshold, trigger the motion planning module to re-plan the grasping path to ensure the stability of the grasping process; The force feedback unit is used to accurately measure the magnitude of the contact force when the manipulator grasps an object, and adjust the clamping strategy according to the change of the force. The implementation method is as follows: First, calibrate the six-axis force sensor. By loading a standard weight onto the sensor, obtain the output data of the sensor, and calibrate the sensor according to the weight of the standard weight to make the measurement error of the sensor less than the preset threshold range. Then, use a Butterworth filter to filter the signal collected by the force sensor to suppress high-frequency noise interference. During the grasping process, continuously monitor the change of the force. When it is detected that the force mutation ≥ 10 N, it is determined that a collision has occurred. When it is detected that the force drops suddenly ≥ 5 N, it is determined that slippage has occurred. In case of collision and slippage, the system triggers the motion planning module to regenerate the path and readjust the grasping action of the manipulator.

7. A manipulator grasping planning system based on visual recognition according to claim 1, characterized in that: The system integration and communication module includes the following: This module is used to realize the real-time data interaction and collaborative work among the system modules to ensure the operation of the system. The implementation method is as follows: Adopt the ROS2 architecture, use the topic mechanism to realize data publishing and subscribing among different modules, so that each module can obtain the data published by other modules according to its own needs. Realize task interaction among modules through the action mechanism, support the dynamic replanning function, and when the system detects an abnormal situation, each module can work together to make corresponding adjustments. At the same time, use a hardware timer to synchronize the system hardware to ensure that the vision-control closed-loop delay is lower than the preset threshold range to ensure the real-time response and control of the system to visual information.

8. A manipulator grasping planning method based on visual recognition, which is applied to a manipulator grasping planning system based on visual recognition according to any one of claims 1-7, characterized in that, Including the following steps: S1. Through the target detection unit, image segmentation unit and target measurement unit of the visual perception module, process and analyze the scene image according to their respective implementation methods to obtain the category, position, pose, distance, speed, orientation angle information of the target and the semantic segmentation information of the static environment. S2. Use the hand-eye calibration unit of the coordinate transformation module to establish the conversion relationship between the camera coordinate system and the end coordinate system of the manipulator through static calibration, dynamic calibration compensation and coordinate transformation methods. S3. With the help of the path planning unit and strategy optimization unit of the motion planning module, use global planning, dynamic obstacle avoidance and reinforcement learning methods respectively to generate the grasping path of the manipulator and optimize the grasping strategy. S4. Use the trajectory execution unit and vision-force hybrid feedback unit of the control execution module to convert the planned path into the actions of the manipulator through inverse kinematics solution, control algorithms and feedback control methods, and continuously monitor the grasping state to dynamically adjust the clamping force. S5. Through the vision feedback unit and force feedback unit of the end perception module, detect the surface state of the object and quantify the contact force according to the image processing and force sensor data processing methods, and trigger corresponding action adjustments. S6. Through the system integration and communication module, use the ROS2 architecture and hardware synchronization method to realize real-time data interaction and collaborative work among modules.

Images