Multi-target adaptive angle grasping method for visual servo mechanical arm
By combining a global camera and a hand-eye camera system, and using Mask R-CNN and MPC control algorithms, a visual servo robotic arm was able to perform multi-target adaptive angle grasping in complex environments. This solved the problem of insufficient adaptability and flexibility of traditional robotic arms in multi-target environments, and enabled precise grasping.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV OF TECH
- Filing Date
- 2023-09-25
- Publication Date
- 2026-07-07
AI Technical Summary
Traditional vision servo robotic arms struggle to adaptively grasp a large number and variety of objects in complex environments, exhibiting insufficient adaptability and flexibility.
By combining a global camera system and a hand-eye camera system, and using Mask R-CNN and MPC control algorithms, the global camera acquires object position information, while the hand-eye camera adjusts the end-effector angle to achieve multi-target adaptive grasping.
Achieving precise grasping of multiple targets in complex environments enhances the adaptability and flexibility of the robotic arm, avoiding problems such as grasping task failure and targets exceeding the field of vision.
Smart Images
Figure CN117162094B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual control technology for robotic arms, specifically to a method for multi-target adaptive angle grasping of a visual servo robotic arm based on MaskR-CNN (MaskRegion-based Convolutional Neural Network) and MPC (Model Predictive Control) control algorithms. Background Technology
[0002] With the advancement of technology, robotic arms have become one of the main types of robots used in industry. They are characterized by high reliability and long working hours, greatly improving production efficiency, saving human resources, and are widely used across various industries. However, traditional robotic arms mainly operate through a teaching method, requiring technicians to pre-program the operation, define the trajectory, execute fixed actions, and complete specific tasks based on the application scenario. This approach is primarily used in standardized assembly lines and struggles to cope with real-time changes in the external environment. With the development of machine vision technology, visual servo robotic arms are gaining increasing popularity. These arms are mainly controlled by machine vision. Commonly used visual servo robotic arms include fixed camera systems and hand-eye systems. Fixed camera systems offer a good field of view but lower control precision, while hand-eye systems, although highly precise, carry the risk of losing field of view. Image processing and control algorithms remain challenging aspects of visual servo robotic arm control. Therefore, visual servo robotic arms are often used in situations with simple targets and backgrounds, which still presents significant limitations in today's increasingly diverse production environments. Summary of the Invention
[0003] To address the limitations and low adaptability of existing visual servoing robotic arms, this invention combines a global camera system with a hand-eye camera system to provide a visual servoing control method for robotic arms that integrates Mask R-CNN and MPC control algorithms. This aims to improve the flexibility and adaptability of the visual servoing robotic arm, enabling it to accurately grasp numerous and diverse objects from adaptive angles in complex background environments. The method primarily relies on the global camera to obtain the position information of the target object, and uses the hand-eye camera to move the robotic arm to the initial position for the grasping task, ensuring the target object is within the field of view of the hand-eye camera. Then, based on the hand-eye camera, the MPC algorithm is used to control the robotic arm to adaptively adjust the end effector angle and perform the grasping task. Specifically, the method includes the following steps:
[0004] Step 1) Take pictures with the global camera and perform preprocessing.
[0005] The global camera uses a camera that can acquire depth information to take pictures of all objects in the area to be captured, and uses a threshold mean filtering algorithm to filter the image. When the absolute value of the difference between the pixel value of the center pixel and the average pixel value in the neighborhood is greater than the threshold, the filtering process is performed; otherwise, the original pixel value is retained. This smooths out noise while avoiding the loss of edge information.
[0006] Step 2) The global camera acquires object information and determines the current target to be captured.
[0007] The Mask R-CNN neural network is trained using a training set to identify all objects in the global camera image and label their categories. After object recognition, the pixel coordinates of the centroid of each object are calculated based on its shape. The objects are then sorted from left to right and top to bottom based on their centroids, and the object with the index 1 is set as the current grasping target.
[0008] Step 3) Obtain the feature points of the target and their world coordinates.
[0009] After identifying the target, edge detection and corner detection methods are used to obtain the upper surface information of the target, and four feature points are marked on its upper surface to obtain the pixel coordinates of the feature points. The world coordinates of each feature point are then calculated based on the intrinsic parameter matrix of the global camera and the rotation transformation matrix from the global camera to the world coordinate system.
[0010] Step 4) The robotic arm moves to the initial gripping position.
[0011] Before the grasping task begins, the target must be within the field of view of the hand-eye camera mounted on the end effector of the robotic arm. Therefore, the robotic arm should first be moved to an initial grasping position where the target object can be seen. Initial pixel coordinates in the hand-eye camera are set for each feature point according to the type of target object. Based on the world coordinates of the feature points obtained in step 3), the theoretical pixel coordinates in the hand-eye camera are calculated using the intrinsic parameter matrix of the hand-eye camera and the rotation transformation matrix from the world coordinates to the hand-eye camera. Then, an offline open-loop MPC algorithm is used to control the movement of the robotic arm, while continuously calculating the new theoretical pixel coordinates of the target object's feature points in the hand-eye camera under the new pose of the robotic arm, until the theoretical pixel coordinates reach near the set initial pixel coordinates. Although this process uses an open-loop algorithm and does not provide real-time feedback of the actual feature point pixel coordinates from the hand-eye camera, the theoretical pixel coordinates in the hand-eye camera are still calculated using the world coordinates of the feature points, ensuring that the target object remains within the field of view of the hand-eye camera after the robotic arm moves to the initial grasping position.
[0012] Step 5) Hand-eye camera acquires object information
[0013] The hand-eye camera uses the same recognition method as the global camera to identify objects and their types within the current field of view. It uses edge detection and corner detection to identify the upper surface of objects and marks four feature points on each object to obtain the pixel coordinates of feature points of all objects within the field of view.
[0014] Step 6) The hand-eye camera identifies the current target to be grasped.
[0015] A target recognition and tracking strategy is invented, specifically by calculating the sum of the differences between the pixel coordinates of four feature points of each object within the field of view of the hand-eye camera and the four initial pixel coordinates in step 4). The object with the smallest sum of pixel differences is the predefined target for grasping in the global camera.
[0016] Step 7) Use a hand-eye camera and an online closed-loop MPC algorithm to capture the target.
[0017] After the hand-eye camera identifies the target, the expected coordinates of the feature points in the hand-eye camera are set according to the target type. An online closed-loop MPC algorithm is used to control the robotic arm to grasp the target object. The input of the control algorithm is the sum of the differences between the current pixel coordinates and the expected pixel coordinates of the four feature points in the hand-eye camera. The output is the angular velocity of the six joints of the robotic arm. The cost function is defined as the sum of the differences between the current pixel coordinates and the expected pixel coordinates. As the sum of the pixel differences decreases, the robotic arm will first adjust the end angle to be parallel to the feature point plane, and then approach the object while maintaining the parallel state with the feature point plane, thus achieving adaptive angle grasping. At the same time, taking advantage of the ability to set constraints in MPC, speed constraints, joint angle constraints, and field of view constraints are set for the robotic arm to prevent the target object from going out of the field of view of the hand-eye camera during the grasping process.
[0018] During the grasping task, the target recognition and following strategy of this invention is used in each cycle to confirm the grasping target, avoiding the problem of grasping task failure due to target recognition errors. Specifically, the sum of the differences between the pixel coordinates of the four feature points of each object in the current hand-eye camera and the four theoretical pixel coordinates of the robotic arm in this posture is calculated. The object with the smallest sum of pixel differences is the grasping target in the current task.
[0019] The beneficial effects of this invention are as follows:
[0020] 1) This invention employs an object recognition method based on Mask R-CNN, which, after training, can identify various objects to be grasped in complex background environments. It features strong anti-interference capabilities, a wide range of recognition types, and high practicality.
[0021] 2) This invention applies a global camera system and a hand-eye camera system to the robotic arm at the same time, which can autonomously determine the order of disordered targets for grasping tasks without prior teaching, and has strong flexibility and applicability.
[0022] 3) The target recognition and tracking algorithm designed in this invention enables the robotic arm to automatically find the target to be grasped before grasping the task, move the robotic arm to a suitable initial position, and accurately identify and follow the target during the grasping task, effectively avoiding the problems of target following error and target exceeding the field of vision causing grasping task failure.
[0023] 4) This invention adopts an MPC-based control method, which can adaptively adjust the angle of the end of the robotic arm according to four feature points, making it parallel to the feature point surface, thereby improving the gripping success rate. It can also effectively handle the constraints of the robotic arm's speed, joint angle, and camera field of view, and reliably complete the visual servo grasping task. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the hardware layout of the present invention;
[0025] Figure 1 In the middle: 1-Global camera; 2-Hand-eye camera; 3-Object to be grasped; 4-Industrial computer; 5-Controller; 6-6-axis robotic arm;
[0026] Figure 2 This is a schematic diagram of the overall process of the present invention;
[0027] Figure 3 This is a schematic diagram of the object recognition and sorting process of the present invention;
[0028] Figure 4 This is a schematic diagram of the target recognition and tracking algorithm of the present invention; Figure 4 Figure a shows a schematic diagram of the target identification process before the robotic arm grasps the target. Figure 4 Figure b shows a schematic diagram of the target identification process during the robotic arm's grasping operation;
[0029] Figure 5 This is a schematic diagram of the MPC-based control method used in this invention; Figure 5 Figure a shows a schematic diagram of the open-loop MPC control method. Figure 5 Figure b shows a schematic diagram of the closed-loop MPC control method. Detailed Implementation
[0030] To make the technical problem, solution, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific examples. The platform used in this invention specifically comprises a lightweight six-axis robotic arm and its controller, a monocular camera used as a hand-eye camera, a depth camera used as a global camera, and an industrial control computer, as shown in the schematic diagram below. Figure 1As shown. 1 is the global camera, mounted at the top of the assembly, providing a top-down view of the target object. 2 is the hand-eye camera, mounted at the end of the robotic arm. 3 is the object to be grasped. 4 is the industrial control computer. 5 is the robotic arm's controller. 6 is the lightweight 6-axis robotic arm.
[0031] This invention provides a method for multi-target adaptive angle grasping of a visual servoing robotic arm based on Mask R-CNN and MPC control algorithm. The overall process of this method is as follows: Figure 2 As shown.
[0032] Step 1) Global camera capture and preprocessing
[0033] A global camera is used to photograph all objects within the capture area, and an improved mean filtering algorithm with thresholding is applied to filter the images. The filtering algorithm uses a 3×3 convolution kernel. Filtering is performed when the absolute value of the difference between the center pixel value and the mean value is greater than the set threshold; otherwise, the original pixel value is retained. Setting a reasonable threshold effectively smooths noise while avoiding the loss of edge information. The filtered pixel value is...
[0034]
[0035] Where (x,y) are pixel coordinates, g(x,y) are the filtered pixel values, I(x,y) are the unfiltered pixel values, n is the size of the region corresponding to the convolution kernel (here, n is 9), and Neighbor represents the entire image region.
[0036] Step 2) The global camera acquires object information and determines the current target to be captured.
[0037] This invention uses a neural network recognition method based on Mask R-CNN to identify objects. After training on a training set, the method can identify all objects in the global camera image and indicate the category of each object.
[0038] The object recognition and sorting process is as follows: Figure 3 As shown, after an object is identified, the centroid pixel coordinates of each object are calculated based on the object's shape. The objects are sorted from left to right and from top to bottom based on their coordinates, and the object with the number 1 is set as the current grasping target.
[0039] Step 3) Obtain the feature points of the target and their world coordinates.
[0040] Based on the above images and according to the type of target object, edge detection and corner detection methods are used to identify the upper surface of the object and four feature points on the surface. The pixel coordinates and depth information of each feature point are obtained. The world coordinates of each feature point are then calculated using the global camera intrinsic parameter matrix and the rotation transformation matrix from the global camera to the world coordinate system. According to the pinhole camera model, the world coordinates of the i-th feature point are... Global camera coordinates Image pixel coordinates The following conversion relationship exists between them:
[0041]
[0042]
[0043] in, Let be the pixel coordinates of the i-th feature point. Let i be the world coordinates of the i-th feature point. The global camera intrinsic parameter matrix is obtained through calibration. Let T be the rotation and translation transformation matrix from the world coordinate system to the global camera coordinate system, where T represents the matrix transpose.
[0044] Step 4) The robotic arm moves to the initial gripping position.
[0045] Before grasping, the target should be within the field of view of the hand-eye camera. However, after the global camera selects the target object, the target object is likely not within the field of view of the robotic arm's hand-eye camera. Therefore, this invention first uses an offline open-loop MPC algorithm to control the movement of the robotic arm, so that the target object appears in the field of view of the hand-eye camera. The algorithm flow is as follows: Figure 5 As shown in Figure a.
[0046] First, based on the object type, set the initial pixel coordinates for the grasping task in the hand-eye camera using four feature points. The coordinates of the i-th feature point are... This is used as the expected pixel coordinate for this step. Based on the feature point world coordinates obtained in step 3), the current theoretical pixel coordinates of the i-th feature point in the hand-eye camera can be calculated using the hand-eye camera intrinsic parameter matrix and the rotation transformation matrix from the world coordinate system to the hand-eye camera. Since the feature points are not within the field of view of the hand-eye camera, the calculated theoretical pixel coordinates may be negative or exceed the camera's pixel limit. The transformation formula is shown below.
[0047]
[0048]
[0049] in, Let i be the current theoretical pixel coordinates of the i-th feature point. Let be the hand-eye camera coordinates of the i-th feature point, which can also be expressed as . T represents the matrix transpose. The intrinsic parameter matrix of the hand-eye camera is obtained through camera calibration. Let be the rotation and translation transformation matrix from the world coordinate system to the hand-eye camera coordinate system.
[0050] After obtaining the current theoretical pixel coordinates, calculate the theoretical pixel coordinates of the four feature points. With expected pixel coordinates The sum of errors e = (e u ,e v ) T .
[0051]
[0052] Then, the MPC algorithm is used to control the movement of the robotic arm, with the error e as the input, and the angular velocities of the six joints of the robotic arm (U1, U2, U... 3, U4, U5, U6) T As output, it is sent to the robotic arm controller to move the robotic arm. During the calculation process, the time taken for each cycle is dynamically recorded, and then the angle change of each joint within this cycle is calculated to obtain the new joint angle. Further, the rotation transformation matrix from the new world coordinate system to the hand-eye camera coordinate system is calculated, and the current theoretical pixel coordinates of the target object's feature points in the new pose of the robotic arm are updated. The new error e is then calculated until the error is reduced to within the target range, at which point the target object is considered to have entered the field of view of the hand-eye camera.
[0053] In this open-loop MPC algorithm, since the calculated theoretical pixel coordinates may be negative or exceed the camera's pixel limit, no field-of-view constraints are set; only joint angular velocity constraints are set.
[0054] The "open-loop" approach used in this process does not rely on the real-time feedback of the actual feature point pixel coordinates from the hand-eye camera. Instead, it uses the world coordinates of the feature points and the rotation transformation matrix to calculate their theoretical pixel coordinates in the hand-eye camera in each cycle as feedback for the algorithm. This approach can simply and effectively ensure that the robotic arm moves to the vicinity of the initial grasping position, keeping the target object within the field of view of the hand-eye camera and avoiding situations where the grasping task cannot be carried out due to loss of field of view.
[0055] Step 5) The hand-eye camera acquires object information and identifies the current grasping target.
[0056] The hand-eye camera also uses an object recognition method based on Mask R-CNN to identify objects and their types within the current field of view. It uses edge detection and corner detection methods to identify the upper surface of the object and four feature points on the surface, thereby obtaining the pixel coordinates of the feature points of all objects within the field of view.
[0057] This paper also proposes a target recognition and following strategy to identify the current target to be grasped before the grasping task begins. The process is as follows: Figure 4 As shown in Figure a, specifically, the pixel coordinates of four feature points of each object within the field of view of the hand-eye camera are first marked, and the pixel coordinates of the i-th feature point are (u i ,v i ), and calculate its coordinates with the four initial pixel coordinates in step 4). The sum of the differences. The difference of the j-th object is...
[0058]
[0059] After calculating the difference, compare all the differences Dis. j The object with the smallest sum of pixel differences is considered the current grasping target in the global camera. Then, based on the pixel coordinates of the four feature points of the current target object, the robotic arm is controlled to perform the grasping task.
[0060] Step 6) Use a hand-eye camera and an online MPC algorithm to capture the target.
[0061] For each type of object, the expected coordinates of the feature points when the object is grasped are set in advance. After the hand-eye camera determines the grasping target, the closed-loop MPC algorithm is used to control the robotic arm to perform the task of grasping the target object.
[0062] The closed-loop MPC algorithm is similar to the open-loop MPC algorithm, and its process is as follows: Figure 5As shown in Figure b, an image Jacobian matrix is established from pixel coordinates to the angular velocities of the robotic arm joints, and substituted into the state equation of the MPC. The system input is defined as the sum of the differences between the current pixel coordinates and the desired pixel coordinates of the four feature points, and the output is defined as the angular velocities of the six joints of the robotic arm, which are sent to the robotic arm controller to move the robotic arm. The cost function is defined as the sum of the differences between the pixel coordinates of the four feature points. The difference of the "closed loop" is that in each cycle, the pixel coordinates of the current target object feature points are obtained by the hand-eye camera, and then a new error is calculated with the desired pixel coordinates as the input for the next cycle, until the error is reduced to the desired range, indicating that the robotic arm has reached the grasping position, and then the end effector gripper is controlled to perform the grasping operation. In this process, the pixel error of four feature points can be used to enable the robotic arm to automatically adjust the end angle to be parallel to the feature point surface, realizing adaptive angle gripping. At the same time, in order to ensure that the target object is out of the field of view of the hand-eye camera during the task, the advantages of setting constraints by MPC are used to set speed constraints, joint angle constraints and field of view constraints for the robotic arm to prevent the target object from leaving the field of view. Finally, the robotic arm can achieve precise grasping, and its error accuracy in the hand-eye camera can reach 0.001 pixels.
[0063] During the crawling task, the target identification and following strategy in step 5) is used in each cycle to confirm the crawling target, avoiding crawling task failure due to incorrect target identification. The process is as follows: Figure 4 As shown in Figure b, the process involves first obtaining the pixel coordinates of four feature points for each object within the current field of view of the hand-eye camera. Then, based on the rotation transformation matrix from the current robotic arm to the hand-eye camera, the four theoretical pixel coordinates of the robotic arm in its current posture are calculated. Finally, the sum of the differences between the four feature points and the four theoretical pixel coordinates of each object is calculated. The object with the smallest sum of pixel differences is the current grasping target.
[0064] The above embodiments represent only one implementation of the present invention and should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make several modifications without departing from the concept of the present invention, and these modifications all fall within the protection scope of the present invention.
Claims
1. A method for multi-target adaptive angle grasping with a visual servo robotic arm, characterized in that, Includes the following steps: Step 1) Take photos with the global camera and perform preprocessing: Take pictures of all objects in the area to be captured, and use a threshold mean filtering algorithm to filter the images; Step 2) The global camera acquires object information and determines the current target to be captured: The Mask R-CNN neural network is trained using the training set to identify all objects in the global camera image and label the object categories. After identifying the objects, the pixel coordinates of the centroid of each object are calculated based on its shape. The objects are sorted from left to right and from top to bottom based on the centroid, and the object with the number 1 is set as the current grasping target. Step 3) Obtain the feature points of the target and their world coordinates: After identifying the target, edge detection and corner detection methods are used to obtain the upper surface information of the target, and four feature points are marked on its upper surface to obtain the pixel coordinates of the feature points. The world coordinates of each feature point are then calculated based on the intrinsic parameter matrix of the global camera and the rotation transformation matrix from the global camera to the world coordinate system. Step 4) The robotic arm moves to the initial gripping position: According to the type of target object, the initial pixel coordinates in the hand-eye camera are set for each feature point. Based on the world coordinates of the feature point obtained in step 3), the current theoretical pixel coordinates in the hand-eye camera are calculated using the intrinsic parameter matrix of the hand-eye camera and the rotation transformation matrix from the world coordinates to the hand-eye camera. Then, the offline open-loop MPC algorithm is used to control the movement of the robotic arm. At the same time, the new theoretical pixel coordinates of the target object feature point in the hand-eye camera under the new pose of the robotic arm are continuously calculated until the theoretical pixel coordinates reach the vicinity of the set initial pixel coordinates. Step 5) The hand-eye camera acquires object information: The Mask R-CNN neural network is trained using the training set to identify objects and their types within the current field of view. Edge detection and corner detection are used to identify the upper surface of the objects, and four feature points are marked on each object to obtain the pixel coordinates of the feature points of all objects within the field of view. Step 6) The hand-eye camera identifies the current target to be grasped: Calculate the sum of the differences between the pixel coordinates of the four feature points of each object in the field of view of the hand-eye camera and the four initial pixel coordinates in step 4). The object with the smallest sum of pixel differences is the target to be grasped in the global camera. Step 7) Use a hand-eye camera and an online closed-loop MPC algorithm to capture the target: After the hand-eye camera determines the target to be grasped, the expected coordinates of the feature points in the hand-eye camera when the object is grasped are set according to the target type, and the robotic arm is controlled to grasp the target object using an online closed-loop MPC algorithm.
2. The multi-target adaptive angle grasping method of a visual servo robotic arm according to claim 1, characterized in that, The specific process of step 1) is as follows: Filtering is performed when the absolute value of the difference between the center pixel value and the average pixel value in its neighborhood is greater than a threshold; otherwise, the original pixel value is retained. The filtered pixel value is: ⑴ in, For pixel coordinates, These are the filtered pixel values. These are the pixel values before filtering. The size of the region corresponding to the convolution kernel. This represents the entire image area.
3. The multi-target adaptive angle grasping method of a visual servo robotic arm according to claim 1, characterized in that, The specific process of step 3) is as follows: Based on the pinhole camera model, the first World coordinates of a feature point Global camera coordinates Image pixel coordinates The following conversion relationship exists between them: ⑵ ⑶ in, For the first The pixel coordinates of each feature point For the first The world coordinates of a feature point The global camera intrinsic parameter matrix is obtained through calibration. Let be the rotation and translation transformation matrix from the world coordinate system to the global camera coordinate system. This represents the transpose of a matrix.
4. The multi-target adaptive angle grasping method of a visual servo robotic arm according to claim 1, characterized in that, The specific process of step 4) is as follows: First, based on the object type, set the initial pixel coordinates for the grasping task in the hand-eye camera using four feature points. The coordinates of the feature points are: This is used as the expected pixel coordinate for this step; based on the feature point world coordinates obtained in step 3), the first pixel coordinate can be calculated using the hand-eye camera intrinsic parameter matrix and the rotation transformation matrix from the world coordinate system to the hand-eye camera. The current theoretical pixel coordinates of each feature point in the hand-eye camera Since the feature points are not within the field of view of the hand-eye camera, the calculated theoretical pixel coordinates may be negative or exceed the camera's pixel limit; the transformation formula is as follows: ⑷ ⑸ in, For the first The current theoretical pixel coordinates of each feature point For the first The hand-eye camera coordinates of each feature point are represented as follows: , Indicates matrix transpose. The intrinsic parameter matrix of the hand-eye camera is obtained through camera calibration. Let be the rotation and translation transformation matrix from the world coordinate system to the hand-eye camera coordinate system; After obtaining the current theoretical pixel coordinates, calculate the theoretical pixel coordinates of the four feature points. With expected pixel coordinates The sum of errors ; ⑹ Then, the MPC algorithm is used to control the movement of the robotic arm, reducing the error. As input, the angular velocities of the robotic arm's six joints As output, it is sent to the robotic arm controller to move the robotic arm; during the calculation process, the time taken for each cycle is dynamically recorded, and then the angle change of each joint within this cycle is calculated to obtain the new joint angle. Further, the rotation transformation matrix from the new world coordinate system to the hand-eye camera coordinate system is calculated, and the current theoretical pixel coordinates of the target object's feature points in the new pose of the robotic arm are updated. Thus, the new error is calculated. The error continues until it is reduced to within the target range, at which point the target object is considered to have entered the field of view of the hand-eye camera.
5. A multi-target adaptive angle grasping method for a visual servo robotic arm according to claim 4, characterized in that, The specific process of step 6) is as follows: First, mark the pixel coordinates of the four feature points of each object within the field of view of the hand-eye camera. The pixel coordinates of the i-th feature point are: And calculate its coordinates with the four initial pixel coordinates in step 4). The sum of the differences; the first The difference between the objects is: ⑺ After calculating the difference, compare all the differences. The object with the smallest sum of pixel differences is considered the current target to be captured in the global camera. Subsequently, the robotic arm is controlled to perform the grasping task based on the pixel coordinates of the four feature points of the current target object.
6. A multi-target adaptive angle grasping method for a visual servo robotic arm according to claim 1, characterized in that, The specific process of step 7) is as follows: The input is the sum of the differences between the current pixel coordinates and the desired pixel coordinates of four feature points in the hand-eye camera. The output is the angular velocity of the six joints of the robotic arm. The cost function is defined as the sum of the differences between the current pixel coordinates and the desired pixel coordinates. As the sum of the pixel differences decreases, the robotic arm will first adjust the end angle to be parallel to the feature point plane, and then approach the object while maintaining the state of being parallel to the feature point plane, thus achieving adaptive angle gripping. At the same time, taking advantage of the ability to set constraints in MPC, speed constraints, joint angle constraints, and field of view constraints are set for the robotic arm. During the grasping task, the sum of the differences between the pixel coordinates of the four feature points of each object in the current hand-eye camera and the four theoretical pixel coordinates of the robotic arm in this posture is calculated. The object with the smallest sum of pixel differences is the grasping target in the current task.