Autonomous underwater vehicle bottom-sitting visual robotic arm grabbing method

Abstract

The application belongs to the field of autonomous operation of underwater robots, and particularly relates to a visual mechanical arm grabbing method after autonomous underwater robots sit on the bottom, which comprises the following steps: after the AUV sits on the bottom, a binocular camera is used to shoot the seabed image, and target recognition and positioning for seabed grabbing are performed; a transformation matrix of the camera coordinate system and the mechanical arm coordinate system is used to determine whether the grabbing target is in the seabed movement space of the flexible hand, if not, the AUV body pose is adjusted, and the target recognition and positioning are re-executed; if the target is in the seabed movement space of the flexible hand, the AUV performs mechanical arm movement planning and execution based on the look-up table interpolation method according to the spatial position information of the target, so that the flexible hand moves to the target position, and the flexible hand is controlled to close to complete grabbing. The visual mechanical arm grabbing movement planning method provided by the application can quickly complete the autonomous grabbing action of the mechanical arm and the flexible hand under low calculation complexity, and has small grabbing position error and high grabbing success rate.

Description

Technical Field

[0001] This invention belongs to the field of autonomous underwater robot operation, specifically a method for a vision-based robotic arm to grasp objects after an autonomous underwater robot has landed on the seabed. Background Technology

[0002] With the increasing demand for marine resource development, underwater engineering maintenance, and scientific exploration, operational autonomous underwater vehicles (AUVs) are receiving increasing attention. Among these, the autonomous grasping technology using underwater vision robotic arms is a crucial component of operational AUVs, enabling them to play a vital role in tasks such as seabed equipment maintenance, sample collection, and debris recovery. However, autonomous grasping with underwater vision robotic arms for AUVs involves several technical challenges. Unlike land-based industrial robotic arms, AUV robotic arms have very limited computational resources and training samples for vision-based grasping tasks. Furthermore, due to the AUV's carrying capacity and the impact of the robotic arm on its navigation, existing AUV robotic arm grasping technologies face three major bottlenecks:

[0003] 1. Insufficient reliability of visual perception: Underwater light attenuation, occlusion by suspended objects and the diversity of target shapes cause the accuracy of traditional recognition algorithms (such as YOLOv5, Mask R-CNN) to drop sharply (<70%) in real-world scenarios, and they rely on massive amounts of labeled data for training.

[0004] 2. Poor real-time performance of motion planning: The online inverse kinematics (IK) solution of the robotic arm requires iterative calculation, which takes more than 500ms on the AUV embedded platform, making it difficult to meet the real-time response requirements in dynamic water flow environments;

[0005] 3. Lack of spatial constraint avoidance: Existing methods often ignore the problem of limited space for the robotic arm to move after the AUV lands, and lack a rapid target reachability determination mechanism, which can easily lead to collisions or grasping failures.

[0006] Although some studies have adopted offline trajectory pre-generation schemes, these cannot adapt to changes in seabed topography and have high storage costs. Therefore, there is an urgent need for a lightweight and highly robust vision-motion collaborative grasping framework. Summary of the Invention

[0007] The purpose of this invention is to provide an AUV-based visual grasping method that is highly real-time, robust, avoids collisions with the robotic arm, and ensures the safety of the flexible hand. The main target of this invention is indicator organisms with low mobility on the seabed, providing theoretical support and technical solutions for autonomous seabed biological sampling by AUVs.

[0008] The technical solution adopted by this invention to achieve the above objectives is: a method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, comprising the following steps:

[0009] Step 1: After the AUV lands on the seabed, it takes seabed images using a binocular camera and uses the YOLOv10 network model combined with the MobileSAM instance segmentation network to identify and locate the target to be grabbed from the seabed.

[0010] Step 2: Using the transformation matrix of the camera coordinate system and the robotic arm coordinate system, determine whether the target being grasped is within the seabed motion space of the flexible hand. If it is not within the motion space, adjust the pose of the AUV body and then return to Step 1.

[0011] Step 3: If the target is within the seabed movement space of the flexible hand, control the robotic arm to move from the following posture to the ready-to-grab posture, and control the flexible hand to open.

[0012] Step 4: The AUV plans and executes the robotic arm motion based on the target's spatial position information and the lookup table interpolation method, so that the flexible hand moves to the target position and finally controls the flexible hand to close to complete the grasping.

[0013] Step 1, the identification and positioning of the target to be captured from the seabed, includes the following steps:

[0014] Step 1-1: Train a YOLOv10 recognition network for grasping targets, using a supervised training method that fuses background and foreground data to accelerate convergence;

[0015] Step 1-2: Input the binocular vision acquisition images into the trained target recognition network to obtain the seabed target recognition inference results, i.e., obtain the target detection box;

[0016] Steps 1-3: Use the target detection bounding box information based on the YOLOv10 recognition network as prompt information and input it into the trained MobileSAM instance segmentation network model to obtain all pixels of the target in the image;

[0017] Steps 1-4: Using the center of the smallest bounding rectangle of the target pixel as the grasping point, use the binocular ranging principle to perform stereo matching ranging of the grasping point to obtain the spatial coordinates of the grasping point.

[0018] Step 2, determining whether the target to be grasped is within the seabed movement space of the flexible hand, includes the following steps:

[0019] Step 2-1: Construct the forward kinematics equations of the robotic arm to obtain the expression of the coordinates of the flexible hand's palm in the robotic arm's base coordinate system;

[0020] Step 2-2: Attach ArUco codes to the connection point between the robotic arm's end effector and the flexible hand. Solve the transformation matrix between the robotic arm's base coordinate system and the camera coordinate system using hand-eye calibration based on the ArUco codes, i.e.:

[0021]

[0022] Where base represents the robot arm's base coordinate system, end represents the robot arm's end-effector coordinate system, camera represents the camera coordinate system, and object represents the calibration object. and It is fixed; the transformation matrix between the base coordinate system and the camera coordinate system is obtained by solving AX = XB.

[0023] Steps 2-3: Calculate the seabed motion space of the flexible hand in the camera coordinate system when the robotic arm is in the ready-to-grasp posture. That is, the range of the seabed that the flexible hand can touch in the binocular camera coordinate system when the robotic arm moves to the ready-to-grasp posture and the flexible hand closes.

[0024] Steps 2-4: Use the discrete ray intersection method to determine the position of the grab point based on the parity of the intersection points between the ray and the boundary of the motion space.

[0025] Steps 2-3 are specifically as follows:

[0026] Step 2-3-1: Based on the motion model of the three-axis robotic arm, the coordinates of the hand's palm in the end-effector coordinate system when the flexible hand is closed, and the joint angles in the grasping posture, obtain the function of the coordinates of the flexible hand's palm in the camera coordinate system:

[0027]

[0028] [xyz] = f(θ1,θ2,θ3)

[0029] Where T1, T2, and T3 are the transformation matrices of the corresponding joint coordinate system, and α, β, and γ are the three joint angles under the ready-to-grab posture;

[0030] Step 2-3-2: Taking the distance d between the seabed and the binocular camera when the AUV is seated as a reference, and assuming the binocular camera is mounted vertically downwards, the x and y coordinates corresponding to the camera's visible area in the camera coordinate system are:

[0031]

[0032] Where δ and σ are the horizontal and vertical angles of the camera, respectively;

[0033] Step 2-3-3: Traverse the points in the planar region from the outside in, check if there is an inverse kinematic solution, form a contour point cloud, and record the joint angle combination corresponding to the contour points;

[0034] Steps 2-3-4: Take z = d - nΔd, ..., d, ..., d + nΔd, and repeat steps b and c to obtain the contour points of multiple planes.

[0035] Steps 2-4 are specifically as follows:

[0036] Step 2-4-1: Select the planar motion region contour point cloud that is closest to the z-value of the grab point, and arrange the contour points in clockwise or counterclockwise order to form a closed polygon;

[0037] Step 2-4-2: Launch a horizontal ray from the target point to the right, traverse all contour edges formed by adjacent point clouds, and calculate the number of intersections between the ray and the edges;

[0038] Step 2-4-3: If the number of intersection points is odd, the grasping point is within the seabed movement space of the flexible hand; if the number is even, the grasping point is outside the seabed movement space of the flexible hand.

[0039] In step 3, the control of the robotic arm to move from the following posture to the ready-to-grab posture is specifically as follows:

[0040] Step 3-1: When in the following posture, the forearm and upper arm of the robotic arm extend backward parallel to the abdomen of the AUV; when in the ready-to-grab posture, the forearm and upper arm extend forward, with the flexible hand positioned above the target.

[0041] Step 3-2: Control the movement of the robotic arm joints using a fixed sequence or a specific rotation angle;

[0042] Step 3-3: During the movement of the robotic arm, the positioning information of the target is updated by the binocular camera. After the robotic arm reaches the ready sampling posture, the flexible hand is opened.

[0043] In step 4, the process of planning and executing the robotic arm motion based on the lookup table interpolation method specifically involves:

[0044] Step 4-1: Based on the positive kinematic equations and the combination of motion space contour points and corresponding joint angles obtained in Step 2, construct a coordinate point-joint angle mapping table within the seabed motion range of the flexible hand;

[0045] Step 4-2: Based on the coordinates of the capture point in the camera coordinate system obtained in Step 1, calculate the rotation angle of the joint angle using the lookup table interpolation method;

[0046] Step 4-3: Move the robotic arm sequentially in the order of joint 1, joint 3, and joint 2 to move the flexible hand to the target grasping point;

[0047] Step 4-4: After the robotic arm finishes its movement, control the flexible hand to close, completing the grasping process.

[0048] In step 4-1, a coordinate point-joint angle mapping table is constructed, specifically as follows:

[0049] Step 4-1-1: Based on the function of the coordinates of the flexible hand's palm in the camera coordinate system, calculate the coordinate values ​​and joint angles corresponding to each unit labeled h, i, j, that is:

[0050] [x(h,i,j)y(h,i,j)z(h)]=f(θ1(h,i),θ2(h,i,j),θ3(h,i,j))

[0051] Where z(h) = d + (hn)Δd, adaptive sampling is performed on θ1 and θ3 in each z = z(h) plane, that is:

[0052]

[0053] Where ε is the sampling interval attenuation coefficient of θ1, and Δ1 is the sampling step size of θ1; Let θ3 be the projection of the vector from the center of the manipulator's hand to the center of joint 3 onto the xy plane of the camera coordinate system, and let Δτ be the variable sampling step size of θ3.

[0054] Step 4-1-2: For each cell (h,i,j) in the mapping table, obtain θ1 and θ3 using the above sampling formula, and then solve for f. z (θ1,θ2,θ3)=z(h) to obtain θ2. If there are multiple solutions, the minimum value of θ2 is filled into the mapping table.

[0055] Step 4-1-3: When the difference in x-coordinates between adjacent points is less than or equal to Δx, record (x, y, θ1, θ2, θ3) and execute step 4-1-4; otherwise, change the attenuation coefficient to adjust the sampling step size of θ3 and execute step 4-1-2 again.

[0056] Step 4-1-4: Fit the straight line y = a to the point set corresponding to each group θ1(h,i). h,i x+b h,i And store the coefficients.

[0057] In step 4-2, the rotation angles of the three joints are calculated using a lookup table interpolation method, specifically as follows:

[0058] For the camera coordinate system, the coordinates are (x... c ,y c ,z c For the capture point, perform the following steps:

[0059] Step 4-2-1: Find the distance z c Closest to and greater than z c z(h);

[0060] Step 4-2-2: In the sub-table of z(h), use binary search to find the index i that satisfies the following formula:

[0061] a h,i x c +b h,i ≥|y c |and a h,i+1 xc +b h,i+1 <|y c |

[0062] Step 4-2-3: In the sub-tables of adjacent θ1(i) and θ1(i+1), use the binary search method to find the sub-tables that satisfy x. i,j <x c <x i,j+1 The index j, and the condition x i+1,k <x c <x i+1,k+1 index k;

[0063] Step 4-2-4: Based on the four nearest points found, use local linear fitting to interpolate and calculate θ1, θ2, and θ3.

[0064] The present invention has the following beneficial effects and advantages:

[0065] 1. This invention can be used by an AUV equipped with a 3-axis robotic arm and a flexible hand to autonomously grasp seabed targets using a binocular camera while the vehicle is seated on the seabed;

[0066] 2. The seabed organism identification and localization method based on the YOLOv10 network model proposed in this invention has a high identification success rate and accurate localization information;

[0067] 3. High grasping success rate of the present invention: The vision-based robotic arm grasping motion planning method proposed in this invention can quickly complete the autonomous grasping action of the robotic arm and flexible hand with low computational complexity, and the grasping position error is small and the grasping success rate is high.

[0068] 4. This invention exhibits strong robustness in recognition: This invention integrates background-foreground supervised training with instance segmentation to improve the accuracy of underwater target recognition;

[0069] 5. This invention has the effect of high efficiency in motion planning: Based on the pre-calculated mapping table and interpolation mechanism, this invention reduces the motion planning time of the robotic arm by more than 50%;

[0070] 6. The invention has high fault tolerance in grasping: The invention improves the grasping success rate to over 95% through adaptive adjustment of the AUV body posture and dynamic determination of the flexible hand's motion space; Attached Figure Description

[0071] Figure 1 This is a flowchart of the AUV landing and vision robotic arm grasping biological targets according to the present invention;

[0072] Figure 2 This is a diagram showing the installation layout of the binocular camera and robotic arm on the AUV in this embodiment;

[0073] Figure 3This is the result of biometric target recognition using a binocular camera in this embodiment;

[0074] Figure 4 This is a schematic diagram of the three-axis robotic arm structure and joint coordinate system in this embodiment;

[0075] Figure 5 This is a distribution diagram of sampling points within the seabed plane movement range of the flexible hand in this embodiment;

[0076] Figure 6a This is a schematic diagram of the robotic arm's following posture in this embodiment;

[0077] Figure 6b This is a schematic diagram showing the robotic arm preparing for the grasping posture in this embodiment;

[0078] Figure 7 This is a schematic diagram illustrating the interpolation process using four nearest neighbor points in this embodiment. Detailed Implementation

[0079] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0080] The main content of this invention includes a proposed method for seabed target identification and localization based on the YOLOv10 network framework combined with the MobileSAM instance segmentation network. This method obtains the grasping point location of the target to be grasped. Then, a proposed robotic arm motion planning algorithm based on a lookup table method can quickly and accurately move the flexible hand to the grasping point location to complete the grasping. In this embodiment, the proposed method enables an AUV to autonomously grasp and sample seabed target organisms after settling on the seabed, using a binocular camera and a three-axis robotic arm. It has advantages such as high real-time performance, high reliability, and small grasping error.

[0081] like Figure 1 The diagram shows the overall flow of the method of this invention. After the AUV lands, it uses a binocular camera and a YOLOv10 network model framework combined with a MobileSAM instance segmentation network to identify and locate the target. Based on the location information, it determines whether the target is within the movement space of the flexible hand, and adjusts the AUV's landing posture accordingly. The AUV controls the robotic arm to move from a following posture to a ready-to-grasp posture, and opens the flexible hand. Then, based on the target's spatial position information, it plans the robotic arm's motion and controls the arm's movement to move the flexible hand to the target position. Finally, it controls the flexible hand to close to complete the grasp. Further details are provided below.

[0082] For target identification and localization, the following steps are included:

[0083] Step 1) Train the recognition network for the specified target to be grasped, and introduce a supervised training method that fuses the background and foreground to achieve fast network convergence. The specific training process is as follows:

[0084] 1.1) Collect real data of underwater targets and model data simulating the shape of the targets. Simulated data can solve the problems of difficulty and high cost in acquiring real data of underwater targets;

[0085] 1.2) Label the data according to the morphological characteristics of the target. For example, three types of targets: sea cucumber, sea urchin and fish. They have great differences in morphology and can be labeled as three types. The data is divided into training set and validation set in a 4:1 ratio.

[0086] 1.3) Based on the number of target categories, modify the training parameters of the YOLOv10S model and enable the data augmentation function;

[0087] 1.4) Perform N epochs of training (N = number of samples / batch), and check the detection accuracy of the validation data at any time during the training process. If it keeps fluctuating, adjust the learning rate parameter appropriately and retrain.

[0088] Step 2) Input the binocular vision acquisition images into the trained target recognition network to obtain the seabed target recognition inference results.

[0089] The mounting method of a binocular camera is as follows: Figure 2 As shown, it is mounted on the abdomen of the AUV, in front of the robotic arm, facing directly downwards;

[0090] On the validation set using open-source data, the average accuracy of inference for recognizing sea urchins and starfish reached 90.50% and 85.30%, respectively; on the validation set using real underwater data, the accuracy reached 97.70%. The detection results are as follows... Figure 3 As shown.

[0091] Step 3) Use the detection box information output by the recognition network as prompt information and input it into the trained MobileSAM instance segmentation network model to obtain the pixel-level mask of the target object in the image;

[0092] Step 4) Using the center of the smallest bounding rectangle of the target pixel as the grab point, use the binocular ranging principle to perform stereo matching ranging of the grab point to obtain the spatial coordinates of the grab point.

[0093] The specific steps for hand-eye calibration and flexible hand seabed motion space calculation are as follows:

[0094] Step 1) Construct the forward kinematics equations of the three-axis robotic arm, specifically:

[0095] In this example, the structure of the robotic arm is as follows: Figure 4 As shown, the formula for calculating the coordinates of the flexible hand's palm in the robot arm's base coordinate system is obtained using the improved DH method:

[0096]

[0097] The parameter values ​​of the coordinate systems of each joint of the robotic arm in the example are shown in Table 1:

[0098] Table 1

[0099] i α_i(°) a_i-1(mm) d_i(mm) θ_i(°) 1 0 0 120.5 θ_1 2 90 108 0 θ_2 3 0 256.1445 0 θ_3 4 0 300 50

[0100] Step 2) Affix ArUco codes (fixed in the joint 3-coordinate system) to the connection point between the robotic arm and the flexible hand, and perform hand-eye calibration. The coordinate transformation relationship is as follows:

[0101]

[0102] Where base represents the robot arm's base coordinate system, end represents the joint's 3D coordinate system, camera represents the camera coordinate system, and object represents the calibration object. and It is fixed; the transformation matrix between the base coordinate system and the camera coordinate system is obtained by solving AX = XB.

[0103] Step 3) Calculate the motion space of the flexible hand on the seabed, that is, the range of the seabed that the flexible hand can touch in the binocular camera coordinate system when the robotic arm moves to the ready-to-grasp posture and the flexible hand closes.

[0104] Considering that the camera height is fixed after the AUV lands on the seabed, and the seabed topography changes little over a small area, the movement space of the flexible hand on the seabed is discretized into multiple parallel planes. Given the height d of the camera above the seabed after the AUV lands, the calculation method is as follows:

[0105] a. Based on the motion model of the three-axis robotic arm, the coordinates of the hand's palm in the joint 3-coordinate system when the flexible hand is closed, and the joint angles in the ready-to-grasp posture, the function of the coordinates of the hand's palm in the camera coordinate system when the flexible hand is closed is obtained:

[0106]

[0107] [xyz] = f(θ1,θ2,θ3)

[0108] Where T1, T2, and T3 are the transformation matrices of the corresponding joint coordinate systems, and α, β, and γ are the three joint angles in the ready-to-grab posture. 3 x h , 3 y h , 3 z h The coordinates of the palm of the flexible hand in the joint 3-coordinate system;

[0109] b. On the plane z = d, the camera's visible area is:

[0110]

[0111] Where δ and σ are the horizontal and vertical angles of the camera, respectively, which are 90° and 60° in this example;

[0112] c. Traverse the points within the planar region from the outside in, check for the existence of an inverse kinematic solution, form a contour point cloud, and record the joint angle combinations corresponding to the contour points;

[0113] d. Repeat steps b and c to calculate the contour points and corresponding joint angle combinations in other z-value planes to obtain a spatial contour point cloud within a certain z-value range.

[0114] In this example, the planar motion space contour points with d = 500 mm are as follows: Figure 5 As shown.

[0115] Step 4) Use the discrete ray intersection method to determine whether the spatial coordinates of the grasping point are within the seabed movement space of the flexible hand. The specific method is as follows:

[0116] a. Select the planar motion region contour point cloud that is closest to the z-value of the grab point, and arrange the contour points in clockwise or counterclockwise order to form a closed polygon;

[0117] b. Launch a horizontal ray to the right from the target point, traverse all contour edges (composed of adjacent point clouds), and calculate the number of intersections between the ray and the edges;

[0118] c. If the number of intersection points is odd, the grab point is within the range; if it is even, the grab point is outside the range.

[0119] The specific method for adjusting the AUV pose based on the determination result is as follows:

[0120] Step 1) Divide the area within the field of view of the binocular camera but outside the range of movement of the flexible hand on the seabed into five regions, such as... Figure 5 As shown;

[0121] Step 2) If the target is within these five areas, briefly suspend the AUV in the air (in this example, this is achieved by reducing the rotational speed of the AUV's rear wheel rim thrusters), then allow it to land. During this process, the following actions are performed:

[0122] a. If the target is in area A, control the main thrusters to move the AUV forward a distance L;

[0123] b. If the target is in area B, control the side thrusters to rotate the AUV clockwise by an angle α.

[0124] c. If the target is in area C, control the side thrusters to rotate the AUV counterclockwise by an angle α.

[0125] d. If the target is in area D, control the main thruster to move the AUV forward a distance L, and then immediately control the side thrusters to rotate the AUV clockwise by an angle α.

[0126] e. If the target is in area E, control the main thruster to move the AUV forward a distance L, and then immediately control the side thrusters to rotate the AUV counterclockwise by an angle α.

[0127] In this example, L is 20cm and α is 5°.

[0128] The specific method for controlling the robotic arm to move from the following posture to the grasping preparation posture is as follows:

[0129] (1) In the following posture, the forearm and upper arm of the robotic arm extend backward and are parallel to the abdomen of the AUV (e.g., Figure 6a (As shown); In the ready-to-grab posture, the forearm and upper arm extend forward, and the flexible hand is positioned above the target (as shown). Figure 6b As shown), the angle of joint 3 is the angle at which the flexible hand can reach the farthest distance. In this example, the calculated angle is -10°.

[0130] (3) To avoid collisions between the flexible hand and the seabed during the robotic arm's transition from the following posture to the grasping posture, a fixed sequence of joint movement is used to control the movement of the three joints. In this example, the joint angular velocities are the same, and the joint movement angles and sequence are as follows:

[0131] a. Joints 2 and 3 rotate simultaneously by 70°;

[0132] b. Joint 3 rotates 74°;

[0133] c. Joints 2 and 3 rotate simultaneously by 36°;

[0134] d. Joint 2 rotates 60°, while joint 1 rotates 60° simultaneously;

[0135] e. Joint 3 rotates 170° in the opposite direction;

[0136] f. Rotate joint 1 in the opposite direction by 60°.

[0137] Under this sequence of movements, the minimum distance between the robotic arm and the flexible hand and the seabed is greater than 100mm during the movement, ensuring the safety of the arm and hand;

[0138] (4) During the movement of the robotic arm, the positioning information of the target is updated by the binocular camera. After the robotic arm reaches the ready sampling posture, it opens its flexible hand.

[0139] The specific method for planning and executing the robotic arm's visual grasping motion is as follows:

[0140] (1) Using the positive kinematic equations obtained in step 2 and the combination of motion space contour points and corresponding joint angles, construct a mapping table of coordinate points and joint angle combinations within the seabed motion range of the flexible hand.

[0141] (2) Using the coordinates of the capture point in the camera coordinate system obtained in step 1, calculate the rotation angles of the three joint angles using the lookup table interpolation method;

[0142] (3) Move the robotic arm in the order of joint 1 → joint 3 → joint 2 to move the flexible hand to the target grasping point;

[0143] (4) After the robotic arm finishes its movement, control the flexible hand to close and complete the grasping.

[0144] The specific method for constructing the mapping table of coordinate points and joint angle combinations within the seabed movement range of the flexible hand is as follows:

[0145] A mapping table is constructed based on the forward kinematics equations, and the coordinate values ​​and joint angles corresponding to each unit (labeled as h, i, j) are calculated.

[0146] [x(h,i,j)y(h,i,j)z(h)]=f(θ1(h,i),θ2(h,i,j),θ3(h,i,j))

[0147] Where z(h) = d + (hn)Δd, adaptive sampling is performed on θ1 and θ3 in each z = z(h) plane to ensure uniform distribution of position points. The sampling formula is as follows:

[0148]

[0149] Where ε is the sampling interval attenuation coefficient of θ1, and Δ1 is the sampling step size of θ1; Δτ is the projection of the vector from the center of the manipulator's hand to the center of joint 3 onto the xy plane of the camera coordinate system; Δτ is the variable sampling step size of θ3; in this example, Δ1 = 4°, ε = 0.95, and the initial value of Δτ is 20°.

[0150] The specific numerical calculation method for each cell in the mapping table is as follows:

[0151] a. Using the sampling formula, we obtain z = z(h), θ1 = θ1(h,i), and θ3 = θ3(h,i,j). Then, we solve for f... z (θ1, θ2, θ3)=z(h) gets θ2(h,i,j);

[0152] b. Substitute θ2=θ2(h,i,j) to obtain x(h,i,j) and y(h,i,j);

[0153] c. If x h,i,j-1 -xh,i,j If ≤Δx, record x(h,i,j), y(h,i,j), and θ1(h,i), θ2(h,i,j), and θ3(h,i,j); otherwise, let Δτ = λΔτ and repeat steps a and b.

[0154] d. For each group θ1(h,i), the theoretical distribution of all points (x(h,i,j), y(h,i,j)) is a straight line. Record this straight line y = a. h,i x+b h,i The parameters.

[0155] In the above method, λ is the attenuation coefficient, used to adaptively adjust the sampling step size of θ3, which is set to 0.8 in this example; Δx is the expected maximum interval of the x-coordinate of the sampling point, which is set to 20mm in this example;

[0156] Since the range of motion of the flexible hand on the seabed is symmetrical along the x-axis, it is only necessary to complete the calculation of the mapping table for half of the range (i.e., θ1≥0);

[0157] There may be multiple solutions when calculating θ2 in step a. Select the smaller angle value and fill it into the table.

[0158] In this example, the distribution of coordinate points in the sub-table where z(h) = 500mm is obtained as follows: Figure 5 As shown, the coordinate points in the mapping table are evenly distributed. In the table generated using the above method, y increases with respect to θ1, and x increases with respect to θ3, meaning that x and y in the table data are sorted.

[0159] like Figure 7 As shown, the above method of using lookup table interpolation to calculate the rotation angles of the three joint angles is used for coordinates (x, y, y) in the camera coordinate system. c ,y c ,z c The specific method for capturing points is as follows:

[0160] a. Find the distance z c Closest to and greater than z c z(h);

[0161] b. In the sub-table of z(h), use binary search to find the index i that satisfies the following formula:

[0162] a h,i x c +b h,i ≥|y c |and a h,i+1 x c +b h,i+1 <|y c |

[0163] c. In the sub-tables θ1(i) and θ1(i+1), use a binary search to find the expression x. i,j <x c <x i,j+1 The index j and satisfying x i+1,k <x c <x i+1,k+1 index k;

[0164] d. Based on the four nearest points found, interpolate θ1, θ2, and θ3 using local linear fitting:

[0165] Let θ1 = μ1x + ν1y + ω1, and solve for the coefficients μ1, ν1, ω1 using the least squares method:

[0166]

[0167] Substitute the grab points to calculate the interpolation of θ1:

[0168]

[0169] Let θ² = μ²x + ν²y + ω², and solve for the coefficients μ², ν², ω² using the least squares method:

[0170]

[0171] Substitute the grab points to calculate the interpolation of θ2:

[0172] θ2=μ2x c +ν2|y c |+ω2

[0173] The calculation method for θ3 is the same as that for θ2, so it will not be repeated here.

[0174] In summary, this invention provides a visual robotic arm grasping method for an autonomous underwater robot after it has landed on the seabed. It achieves high-precision underwater target recognition and localization by integrating YOLOv10 target detection and MobileSAM instance segmentation. Hand-eye calibration and motion space discretization modeling ensure the reliability of the reachability determination of the grasping point. Furthermore, it employs lookup table interpolation motion planning instead of traditional inverse kinematics online solution, significantly reducing computational complexity. This method has the following core advantages:

[0175] 1. Strong robustness in recognition: Integrating background-foreground supervised training and instance segmentation improves the accuracy of underwater target recognition;

[0176] 2. Highly efficient motion planning: Based on pre-calculated mapping tables and interpolation mechanisms, the motion planning time for robotic arms is reduced by more than 50%;

[0177] 3. High grasping tolerance: Through adaptive adjustment of AUV body posture and dynamic judgment of flexible hand movement space, the grasping success rate is increased to over 95%;

[0178] 4. Good resource adaptability: The lightweight network model and lookup table method are suitable for AUV embedded platforms and meet the requirements of real-time operation.

[0179] This invention relates to a visual robotic arm method for grasping underwater targets after an autonomous underwater vehicle (AUV) has landed on the seabed. The robotic arm platform studied is a three-axis robotic arm equipped with a flexible gripper. Based on the robotic arm motion model, this invention studies a method for biological target recognition and localization under binocular vision and a low computational complexity robotic arm motion planning method. This method has high real-time performance and strong robustness, and is of great significance for realizing AUV seabed target grasping and improving the autonomous operation capability of AUVs.

[0180] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0181] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for a vision-based robotic arm to grasp objects after an autonomous underwater robot has landed on the seabed, characterized in that... Includes the following steps: Step 1: After the AUV lands on the seabed, it takes seabed images using a binocular camera and uses the YOLOv10 network model combined with the MobileSAM instance segmentation network to identify and locate the target to be grabbed from the seabed. Step 1, the identification and positioning of the target to be captured from the seabed, includes the following steps: Step 1-1: Train a YOLOv10 recognition network for grasping targets, using a supervised training method that fuses background and foreground data to accelerate convergence; Step 1-2: Input the binocular vision acquisition images into the trained target recognition network to obtain the seabed target recognition inference results, i.e., obtain the target detection box; Steps 1-3: Use the target detection bounding box information based on the YOLOv10 recognition network as prompt information and input it into the trained MobileSAM instance segmentation network model to obtain all pixels of the target in the image; Steps 1-4: Using the center of the smallest bounding rectangle of the target pixel as the grasping point, use the binocular ranging principle to perform stereo matching ranging of the grasping point to obtain the spatial coordinates of the grasping point; Step 2: Using the transformation matrix of the camera coordinate system and the robotic arm coordinate system, determine whether the target being grasped is within the seabed motion space of the flexible hand. If it is not within the motion space, adjust the pose of the AUV body and then return to Step 1. Step 2, determining whether the target to be grasped is within the seabed movement space of the flexible hand, includes the following steps: Step 2-1: Construct the forward kinematics equations of the robotic arm to obtain the expression of the coordinates of the flexible hand's palm in the robotic arm's base coordinate system; Step 2-2: Attach ArUco codes to the connection point between the robotic arm's end effector and the flexible hand. Solve the transformation matrix between the robotic arm's base coordinate system and the camera coordinate system using hand-eye calibration based on the ArUco codes, i.e.: ； Where, base represents the robot arm's base coordinate system, end represents the robot arm's end-effector coordinate system, camera represents the camera coordinate system, and object represents the calibration object. and It is fixed; the transformation matrix between the base coordinate system and the camera coordinate system is obtained by solving AX=XB. ; Steps 2-3: Calculate the seabed motion space of the flexible hand in the camera coordinate system when the robotic arm is in the ready-to-grasp posture. That is, the range of the seabed that the flexible hand can touch in the binocular camera coordinate system when the robotic arm moves to the ready-to-grasp posture and the flexible hand closes. Steps 2-4: Use the discrete ray intersection method to determine the position of the grab point based on the parity of the intersection points between the ray and the boundary of the motion space; Step 3: If the target is within the seabed movement space of the flexible hand, control the robotic arm to move from the following posture to the ready-to-grab posture, and control the flexible hand to open. Step 4: The AUV plans and executes the robotic arm motion based on the target's spatial position information and the lookup table interpolation method, so that the flexible hand moves to the target position and finally controls the flexible hand to close to complete the grasping.

2. The method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, as described in claim 1, is characterized in that... Steps 2-3 are specifically as follows: Step 2-3-1: Based on the motion model of the three-axis robotic arm, the coordinates of the hand's palm in the end-effector coordinate system when the flexible hand is closed, and the joint angles in the grasping posture, obtain the function of the coordinates of the flexible hand's palm in the camera coordinate system: ； ； Where T1, T2, and T3 are the transformation matrices of the corresponding joint coordinate system, and α, β, and γ are the three joint angles under the ready-to-grab posture; Step 2-3-2: Taking the distance d between the seabed and the binocular camera when the AUV is seated as a reference, and assuming the binocular camera is mounted vertically downwards, the x and y coordinates corresponding to the visible area of ​​the binocular camera in the camera coordinate system are: ； Where δ and σ are the horizontal and vertical field of view of the binocular camera, respectively; Step 2-3-3: Traverse the points in the planar region from the outside in, check if there is an inverse kinematic solution, form a contour point cloud, and record the joint angle combination corresponding to the contour points; Steps 2-3-4: Take z = d - nΔd, ..., d, ..., d + nΔd, and repeat steps b and c to obtain the contour points of multiple planes.

3. The method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, as described in claim 1, is characterized in that... Steps 2-4 are specifically as follows: Step 2-4-1: Select the planar motion region contour point cloud that is closest to the z-value of the grab point, and arrange the contour points in clockwise or counterclockwise order to form a closed polygon; Step 2-4-2: Launch a horizontal ray from the target point to the right, traverse all contour edges formed by adjacent point clouds, and calculate the number of intersections between the ray and the edges; Step 2-4-3: If the number of intersection points is odd, the grasping point is within the seabed movement space of the flexible hand; if the number is even, the grasping point is outside the seabed movement space of the flexible hand.

4. The method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, as described in claim 1, is characterized in that... In step 3, the control of the robotic arm to move from the following posture to the ready-to-grab posture is specifically as follows: Step 3-1: When in the following posture, the forearm and upper arm of the robotic arm extend backward parallel to the abdomen of the AUV; when in the ready-to-grab posture, the forearm and upper arm extend forward, with the flexible hand positioned above the target. Step 3-2: Control the movement of the robotic arm joints using a fixed sequence or a specific rotation angle; Step 3-3: During the movement of the robotic arm, the positioning information of the target is updated by the binocular camera. After the robotic arm reaches the ready-to-grab posture, it opens its flexible hand.

5. The method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, as described in claim 1, is characterized in that... In step 4, the process of planning and executing the robotic arm motion based on the lookup table interpolation method specifically involves: Step 4-1: Based on the positive kinematic equations and the combination of motion space contour points and corresponding joint angles obtained in Step 2, construct a coordinate point-joint angle mapping table within the seabed motion range of the flexible hand; Step 4-2: Based on the coordinates of the capture point in the camera coordinate system obtained in Step 1, calculate the rotation angle of the joint angle using the lookup table interpolation method; Step 4-3: Move the robotic arm sequentially in the order of joint 1, joint 3, and joint 2 to move the flexible hand to the target grasping point; Step 4-4: After the robotic arm finishes its movement, control the flexible hand to close, completing the grasping process.

6. The method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, as described in claim 5, is characterized in that... In step 4-1, a coordinate point-joint angle mapping table is constructed, specifically as follows: Step 4-1-1: Based on the function of the coordinates of the flexible hand's palm in the camera coordinate system, calculate the coordinate values ​​and joint angles corresponding to each unit labeled h, i, j, that is: ； Where z(h) = d + (hn)Δd, in each z = z(h) plane pair and Adaptive sampling is performed, and the sampling formula is as follows: ； ； in, for The sampling interval attenuation coefficient, for The sampling step size; Let Δτ be the projection of the vector from the center of the manipulator's hand to the center of joint 3 onto the xy plane of the camera coordinate system. Variable sampling step size; Step 4-1-2: For each cell in the mapping table Using the above sampling formula, we obtain and By solving get If multiple solutions exist, Enter the minimum angle value into the mapping table; Step 4-1-3: When the difference in x-coordinates between adjacent points is ≤ Δx, record... Perform step 4-1-4; otherwise, attenuate. Given the sampling step size Δτ, repeat step 4-1-2; Step 4-1-4: Each group Fitting a straight line to the corresponding point set And store the coefficients.

7. A method for a vision-based robotic arm to grasp an autonomous underwater robot after it has landed on the seabed, as described in claim 5, characterized in that... In step 4-2, the rotation angles of the three joints are calculated using a lookup table interpolation method, specifically as follows: For the camera coordinate system, the coordinates are (x... c , y c , z c For the capture point, perform the following steps: Step 4-2-1: Find the distance z c Closest to and greater than z c of ; Step 4-2-2: In In the sub-table, use binary search to find index i that satisfies the following formula: and ; Step 4-2-3: In the sub-tables of adjacent θ1(i) and θ1(i+1), use the binary search method to find the sub-tables that satisfy the condition. The index j, and satisfying index k; Step 4-2-4: Based on the four nearest points found, use local linear fitting to interpolate and calculate θ1, θ2, and θ3.

Images