An underwater mechanical arm target grabbing method and system based on flow field compensation and multi-modal feedback

Abstract

The application provides an underwater mechanical arm target grabbing method and system based on flow field compensation and multi-modal feedback, and relates to the technical fields of underwater robot operation, underwater mechanical arm grabbing, machine vision and multi-modal perception control. The system comprises a binocular image acquisition module, a flow velocity and flow direction acquisition module, a flow field compensation image enhancement module, a target detection and target region segmentation module, a target identity initialization and continuous tracking module, a binocular depth estimation and feature matching module, a pixel-level target point continuous tracking module, a PnP-P3P time sequence pose solution module, a mechanical arm end and gripper control module, and a clamping state confirmation module. The technical scheme of the application can solve the problems of obvious image degradation, unstable target detection, discontinuous target pose estimation, easy loss of target during the approach of the gripper, and difficult confirmation of the clamping state in a complex underwater environment.

Description

Technical Field

[0001] This invention relates to the technical field of underwater robot operations, and more particularly to a target grasping method and system for an underwater robotic arm based on flow field compensation and multimodal feedback. Background Technology

[0002] Underwater robots and robotic arms can be used in various scenarios, including seabed sample collection, underwater equipment installation and maintenance, subsea pipeline repair, shipwreck or foreign object salvage, marine ranching, and underwater engineering operations. Compared to land-based operating environments, underwater environments present complex challenges such as light attenuation, suspended particle scattering, color shift, water flow disturbance, camera movement, short-term target occlusion, and robot posture changes. These factors can easily lead to problems such as degraded underwater image quality, unstable target detection, discontinuous target localization, and reduced success rate of robotic arm grasping.

[0003] Existing underwater operation systems typically employ sonar, lasers, depth cameras, inertial navigation, external positioning systems, or pre-established 3D target models to assist in target identification and pose estimation. While these methods can achieve target detection and localization in some scenarios, they generally suffer from problems such as large equipment size, high system cost, high energy consumption, complex deployment, or strong dependence on prior target models. When underwater missions are temporary, target shapes are unknown, environmental disturbances are strong, or the operating space is limited, it is difficult to pre-establish complete 3D target models and stably acquire high-quality point clouds, making it difficult to meet the requirements of rapid grasping and continuous operation of underwater robotic arms.

[0004] On the other hand, when performing grasping tasks, underwater robotic arms not only need to identify the target in the initial stage, but also need to continuously acquire the target's pose as the robotic arm's end effector and gripper gradually approach the target. As the gripper gets closer to the target, the target may experience partial occlusion, sudden changes in viewpoint, detection box drift, a reduction in effective feature points, or even temporary loss of the target. If only a single frame image is relied upon for target detection and pose calculation, pose jumps or target identity changes can easily occur, thus affecting the robotic arm's end effector posture adjustment, gripper alignment, and stable grasping.

[0005] Furthermore, water flow velocity and direction affect underwater image quality, target edge sharpness, local image distortion, and the target's relative position during the robotic arm's approach. Some existing methods only use water flow information as environmental monitoring data or motion control references, failing to adequately utilize it as compensation input in image enhancement and target localization processes. This results in subsequent target detection, feature matching, and pose estimation remaining susceptible to water flow disturbances.

[0006] Therefore, it is necessary to provide a new underwater robotic arm target grasping method and system that incorporates water flow velocity and direction information into the underwater image enhancement process, and forms a complete closed loop by integrating target detection, target segmentation, continuous target tracking, binocular depth estimation, pixel-level continuous tracking, PnP-P3P temporal pose selection, and gripper multimodal feedback control, thereby improving the continuity of target localization and the stability of robotic arm grasping in complex underwater environments. Summary of the Invention

[0007] To address the technical problems in existing technologies, such as underwater image degradation, unstable target detection and tracking, easy target pose changes during grasping and approach, and difficulty in confirming stable gripping by the gripper, this invention provides a target grasping method and system for an underwater robotic arm based on flow field compensation and multimodal feedback. This invention primarily utilizes water flow velocity and direction information as image enhancement compensation input to repair degradation in binocular underwater images; it uses target detection and target region segmentation results to define the subsequent processing area; it constructs a local 2D-3D correspondence point set for the target using binocular depth estimation and feature matching; it maintains the local geometric relationship of target points in consecutive image frames using pixel-level continuous target point tracking; when the number of effective correspondence points decreases, it uses the PnP-P3P temporal pose calculation method to select the target pose with the highest continuity; finally, it combines robotic arm end-effector control, gripper control, and pressure, tactile, or slip feedback to achieve stable approaching, attitude adjustment, and gripping confirmation of the target by the underwater robotic arm.

[0008] The technical means employed in this invention are as follows: A target grasping method for an underwater robotic arm based on flow field compensation and multimodal feedback includes the following steps: Step 1: Obtain the left and right view images of the underwater operation area, as well as information on water flow velocity and direction; Step 2: Using the water flow velocity and direction information as image enhancement compensation input, underwater degradation repair is performed on the left and right view images to obtain the enhanced underwater image; Step 3: Perform target detection and target region segmentation on the enhanced underwater image to obtain the detection box, target category, confidence score, and target mask of the target to be captured; Step 4: Based on the selected target or task preset target, initialize the target tracking object, and update the target tracking object according to the target detection results and historical motion state in consecutive image frames; Step 5: Define the effective processing area in the binocular image according to the target mask, and perform feature matching and depth estimation within the effective processing area to obtain the two-dimensional-three-dimensional corresponding point set of multiple pixels in the target area; Step 6: Perform cross-frame continuous tracking on the target pixels in the two-dimensional-three-dimensional corresponding point set to obtain the position changes of the target pixels in consecutive image frames and maintain the local geometric relationship of the target. Step 7: Solve the six-degree-of-freedom pose of the target to be grasped relative to the underwater robot body, binocular camera or end effector of the robotic arm based on the two-dimensional-three-dimensional corresponding point set; when the number of effective corresponding points is reduced to three, select the pose solution with the highest continuity from multiple candidate pose solutions based on the six-degree-of-freedom pose of the previous frame. Step 8: Generate a robotic arm end-effector posture adjustment command and a gripper approach command based on the six-degree-of-freedom pose, so that the robotic arm end-effector approaches the target to be grasped along the preset grasping direction; Step 9: Collect gripper pressure, tactile or gliding feedback information, and determine whether the target to be grasped is stably gripped based on the gripper pressure, tactile or gliding feedback information; when the determination result is that it is not stably gripped, regenerate the robotic arm end-effector posture adjustment command or gripper approach command based on the current target six-degree-of-freedom pose and feedback information, until the stable gripping condition is met or the task termination condition is triggered.

[0009] Furthermore, in step 2, the water flow velocity and direction information is either a flow field compensation matrix corresponding to the image size or an image enhancement module that is input together with the underwater image after being introduced into the input channel.

[0010] Furthermore, in step 2, the underwater degradation repair includes: compensating for color shifts, scattering, blurring, and local distortions caused by water flow disturbances in the underwater image.

[0011] Furthermore, in step 5, based on the binocular camera intrinsic parameters, binocular baseline, and target area depth map, the pixels within the target mask area are converted into three-dimensional coordinate points to obtain the target local two-dimensional-three-dimensional corresponding point set.

[0012] Furthermore, in step 6, multiple positioning points within the target area are continuously tracked at the pixel level. When the detection box drifts, the target is temporarily occluded, or the gripper approaches, causing a change in the field of view, the local geometric relationship of the target is maintained based on historical tracking points.

[0013] Furthermore, in step 7, when the number of valid corresponding points is not less than four, the target six-degree-of-freedom pose is obtained by the PnP-type pose solving method; when the number of valid corresponding points is less than four, multiple candidate pose solutions are generated by the P3P method.

[0014] Furthermore, the pose solution with the highest continuity is determined based on the positional difference, attitude difference, or a weighted sum of the two between the candidate pose and the six-DOF pose of the previous frame.

[0015] The present invention also includes an underwater robotic arm target grasping system based on flow field compensation and multimodal feedback, comprising: The module includes a binocular image acquisition module, a flow velocity and direction acquisition module, a flow field compensation image enhancement module, a target detection and target region segmentation module, a target identity initialization and continuous tracking module, a binocular depth estimation and feature matching module, a pixel-level target point continuous tracking module, a PnP-P3P temporal pose calculation module, a robotic arm end effector and gripper control module, and a gripping status confirmation module. The binocular image acquisition module and the flow velocity and direction acquisition module are connected to the flow field compensation image enhancement module; The flow field compensation image enhancement module is connected to the target detection and target region segmentation module; The target detection and target region segmentation module is connected to the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module, respectively. The pixel-level target point continuous tracking module generates continuous tracking points based on the results of the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module. The PnP-P3P temporal pose calculation module outputs the target's six-degree-of-freedom pose based on continuous tracking points and corresponding two-dimensional-three-dimensional point sets. The robotic arm end effector and gripper control module generates robotic arm end effector attitude adjustment commands and gripper approach commands based on the target six-degree-of-freedom pose. The clamping status confirmation module determines the clamping status based on pressure, tactile, or slip feedback, and feeds back the clamping status information to the end effector and gripper control module of the robotic arm when the clamping is not stable.

[0016] Compared with the prior art, the present invention has the following advantages: (1) High underwater sensing stability: This invention uses water flow velocity and direction as image enhancement compensation inputs, rather than just environmental monitoring data, which can reduce the impact of water flow disturbance, scattering, blurring and local distortion on target detection, target segmentation and feature matching from the source of sensing.

[0017] (2) Low dependence on prior target model: This invention uses target mask, binocular depth estimation and feature matching to construct a local two-dimensional-three-dimensional corresponding point set of the target. It does not require the complete three-dimensional model of the target to be obtained in advance. It is suitable for underwater tasks such as sample collection, inspection, salvage and temporary capture where the target shape is unknown or the target model is incomplete.

[0018] (3) Strong target localization continuity: This invention maintains the local geometric relationship of the target point in continuous image frames by continuously tracking the target point at the pixel level, which can reduce the risk of target loss caused by detection box drift, short-term target occlusion, interference from similar targets and the proximity of the gripper.

[0019] (4) High stability of pose calculation: When the effective corresponding points are reduced to three, the present invention generates multiple candidate pose solutions through the P3P method and performs continuity screening based on the six-degree-of-freedom pose of the previous frame, so that the target pose estimation is transformed from single-frame solution to temporal continuous solution, reducing the risk of pose jump during the grasping process.

[0020] (5) Complete grasping closed loop: The present invention forms a closed loop by combining the target's six degrees of freedom pose, the robotic arm's end-effector posture adjustment, the gripper's proximity control, and pressure, tactile or gliding feedback judgment, which can improve the stability and reliability of underwater robotic arm target grasping under conditions of water flow disturbance and target partial obstruction.

[0021] In summary, the technical solution of this invention can solve problems such as significant image degradation, unstable target detection, discontinuous target pose estimation, easy loss of target during the approach of the gripper, and difficulty in confirming the gripping state in complex underwater environments. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall structure of the underwater robotic arm target grasping system of the present invention.

[0024] Figure 2 This is a schematic diagram of the underwater robotic arm target grasping method of the present invention.

[0025] Figure 3 This is a schematic diagram of the input-output relationship of the flow field compensation image enhancement module of the present invention.

[0026] Figure 4 This is a schematic diagram illustrating the binocular depth estimation and the construction of the 2D-3D corresponding point set under the target mask constraint of the present invention.

[0027] Figure 5 This is a schematic diagram of the continuous tracking of target pixels and the PnP-P3P temporal pose calculation of the present invention.

[0028] Figure 6 This is a schematic diagram of the robotic arm end effector, gripper control, and multimodal feedback closed loop of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] like Figures 1-6 As shown, this invention provides a target grasping method for an underwater robotic arm based on flow field compensation and multimodal feedback, comprising the following steps: Step 1: Obtain the left and right view images of the underwater operation area, as well as information on water flow velocity and direction; Step 2: Using the water flow velocity and direction information as image enhancement compensation input, underwater degradation repair is performed on the left and right view images to obtain enhanced underwater images. In Step 2, the water flow velocity and direction information is a flow field compensation matrix corresponding to the image size or an image enhancement module that is input together with the underwater image after being introduced into the input channel. The underwater degradation repair includes compensating for color shift, scattering, blurring, and local distortions caused by water flow disturbance in the underwater image.

[0032] Step 3: Perform target detection and target region segmentation on the enhanced underwater image to obtain the detection box, target category, confidence score, and target mask of the target to be captured; Step 4: Based on the selected target or task preset target, initialize the target tracking object, and update the target tracking object according to the target detection results and historical motion state in consecutive image frames; Step 5: Define the effective processing area in the binocular image according to the target mask, and perform feature matching and depth estimation within the effective processing area to obtain a set of two-dimensional and three-dimensional corresponding points for multiple pixels in the target area; In step 5, the pixels in the target mask area are converted into three-dimensional coordinate points according to the binocular camera intrinsic parameters, binocular baseline and target area depth map to obtain the local two-dimensional and three-dimensional corresponding point set of the target.

[0033] Step 6: Perform cross-frame continuous tracking of the target pixels in the two-dimensional-three-dimensional corresponding point set to obtain the position changes of the target pixels in consecutive image frames and maintain the local geometric relationship of the target; In step 6, perform pixel-level continuous tracking of multiple positioning points in the target area. When the detection box drifts, the target is temporarily occluded, or the gripper approaches, causing changes in the field of view, the local geometric relationship of the target is maintained based on the historical tracking points.

[0034] Step 7: Solve the six-degree-of-freedom pose of the target to be grasped relative to the underwater robot body, binocular camera, or robotic arm end effector based on the 2D-3D corresponding point set. When the number of effective corresponding points decreases to three, select the pose solution with the highest continuity from multiple candidate pose solutions based on the six-degree-of-freedom pose of the previous frame. In Step 7, when the number of effective corresponding points is not less than four, the target's six-degree-of-freedom pose is obtained through a PnP-type pose solving method; when the number of effective corresponding points is less than four, multiple candidate pose solutions are generated through a P3P method. The pose solution with the highest continuity is determined based on the positional difference, attitude difference, or weighted sum of the two between the candidate pose and the six-degree-of-freedom pose of the previous frame.

[0035] Step 8: Generate a robotic arm end-effector posture adjustment command and a gripper approach command based on the six-degree-of-freedom pose, so that the robotic arm end-effector approaches the target to be grasped along the preset grasping direction; Step 9: Collect gripper pressure, tactile or gliding feedback information, and determine whether the target to be grasped is stably gripped based on the gripper pressure, tactile or gliding feedback information; when the determination result is that it is not stably gripped, regenerate the robotic arm end-effector posture adjustment command or gripper approach command based on the current target six-degree-of-freedom pose and feedback information, until the stable gripping condition is met or the task termination condition is triggered.

[0036] The present invention also includes an underwater robotic arm target grasping system based on flow field compensation and multimodal feedback, comprising: The module includes a binocular image acquisition module, a flow velocity and direction acquisition module, a flow field compensation image enhancement module, a target detection and target region segmentation module, a target identity initialization and continuous tracking module, a binocular depth estimation and feature matching module, a pixel-level target point continuous tracking module, a PnP-P3P temporal pose calculation module, a robotic arm end effector and gripper control module, and a gripping status confirmation module. The binocular image acquisition module and the flow velocity and direction acquisition module are connected to the flow field compensation image enhancement module; The flow field compensation image enhancement module is connected to the target detection and target region segmentation module; The target detection and target region segmentation module is connected to the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module, respectively. The pixel-level target point continuous tracking module generates continuous tracking points based on the results of the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module. The PnP-P3P temporal pose calculation module outputs the target's six-degree-of-freedom pose based on continuous tracking points and corresponding two-dimensional-three-dimensional point sets. The robotic arm end effector and gripper control module generates robotic arm end effector attitude adjustment commands and gripper approach commands based on the target six-degree-of-freedom pose. The clamping status confirmation module determines the clamping status based on pressure, tactile, or slip feedback, and feeds back the clamping status information to the end effector and gripper control module of the robotic arm when the clamping is not stable.

[0037] Example 1 This embodiment provides an underwater robotic arm target grasping system based on flow field compensation and multimodal feedback. The system can be installed on an underwater robot, underwater work platform, or other mobile carrier capable of carrying an underwater robotic arm. It is used to identify targets to be grasped in complex underwater environments, estimate the target's relative pose, and control the robotic arm and grippers to complete stable grasping.

[0038] The system includes an underwater robot body, an underwater robotic arm, grippers, a binocular image acquisition module, a flow velocity and direction acquisition module, a flow field compensation image enhancement module, a target detection and target region segmentation module, a target identity initialization and continuous tracking module, a binocular depth estimation and feature matching module, a pixel-level target point continuous tracking module, a PnP-P3P temporal pose calculation module, a robotic arm end effector and gripper control module, and a gripping status confirmation module.

[0039] The underwater robot body houses an underwater robotic arm, grippers, a binocular image acquisition module, a flow velocity and direction acquisition module, and a control and computing unit. The underwater robotic arm is mounted on the underwater robot body or other underwater work platform, with its end effector connected to grippers for performing target approach, attitude adjustment, gripper closure, and target grasping actions. The grippers can be two-finger grippers, multi-finger grippers, flexible grippers, or other end effectors suitable for underwater target grasping.

[0040] The binocular image acquisition module includes a left-view camera and a right-view camera, used to acquire left-view and right-view images of the underwater working area. The left-view and right-view cameras can be mounted on the front end of the underwater robot body or near the end effector of the underwater robotic arm. The images acquired by the binocular image acquisition module are used for subsequent image enhancement, target detection, target segmentation, depth estimation, feature matching, and target pose calculation.

[0041] The flow velocity and direction acquisition module is used to collect the water flow velocity and direction in the underwater working area. This module can be positioned at the front of the underwater robot, near the robotic arm base, or near the binocular image acquisition module, ensuring that the acquired water flow information reflects the local flow field state of the target grasping area. The water flow velocity and direction not only serve as environmental recording data but also as compensation input for the flow field compensation image enhancement module.

[0042] The flow field compensation image enhancement module is connected to both the binocular image acquisition module and the flow velocity and direction acquisition module to receive the left and right view images, as well as water flow velocity and direction information. The flow field compensation image enhancement module encodes the water flow velocity and direction into a flow field compensation matrix, direction vector, or additional input channel, and performs image enhancement processing together with the original underwater image to obtain an enhanced underwater image. Image enhancement processing may include compensation for color shift, suspended particle scattering, blurring, low contrast, and local distortions caused by water flow disturbance.

[0043] The target detection and target region segmentation module is connected to the flow field compensation image enhancement module. It is used to perform target recognition and target region segmentation on the enhanced underwater image, and outputs the detection box, target category, confidence score, and target mask of the target to be captured. The target mask is used to represent the effective region of the target to be captured in the image and to limit the processing range of subsequent binocular depth estimation, feature matching, and pixel tracking.

[0044] The target identity initialization and continuous tracking module is connected to the target detection and target region segmentation module. It initializes the target tracking object based on the user-selected target or a pre-set target, and updates the target tracking object in consecutive image frames by combining target detection results, target mask, historical motion state, and target identity information. Through this module, the system can maintain the consistency of the identity of the currently target to be captured when similar targets, multiple target interference, short-term target occlusion, or detection box drift occur, avoiding accidental switching to other similar targets.

[0045] The binocular depth estimation and feature matching module is connected to the target detection and target region segmentation module, and is used to perform feature matching and depth estimation on the left and right view images under target mask constraints. This module converts multiple effective pixels within the target region into three-dimensional coordinate points based on the binocular camera intrinsics, the binocular baseline, and the target region depth information, thereby constructing a local two-dimensional-three-dimensional corresponding point set for the target.

[0046] The pixel-level target point continuous tracking module is connected to the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module, respectively, and is used to continuously track target pixels across frames in the local two-dimensional-three-dimensional corresponding point set of the target. This module can continuously obtain the positional changes of the target point in consecutive image frames and maintain the local geometric relationship of the target as the end effector of the robotic arm or gripper gradually approaches the target to be grasped.

[0047] The PnP-P3P temporal pose calculation module is connected to the pixel-level target point continuous tracking module. It is used to solve the six-DOF pose of the target to be captured based on the continuously tracked points and the local two-dimensional-three-dimensional corresponding point set of the target. When the number of effective corresponding points is not less than four, the PnP-type pose calculation method is used to obtain the target's six-DOF pose. When the number of effective corresponding points is reduced to three, the P3P method is used to generate multiple candidate pose solutions for the current frame. Based on the positional difference, attitude difference, or weighted sum of the two between the candidate pose solution of the current frame and the six-DOF pose of the previous frame, the candidate pose solution with the highest continuity is selected as the target pose output for the current frame.

[0048] The robotic arm end effector and gripper control module is connected to the PnP-P3P timing pose calculation module. It is used to generate robotic arm end effector attitude adjustment commands, gripper approach commands, and gripper opening and closing commands based on the target's six-degree-of-freedom pose, so that the underwater robotic arm end effector approaches the target to be grasped along the preset grasping direction.

[0049] The gripping status confirmation module is connected to the robotic arm end effector and gripper control module. The gripping status confirmation module may include one or more of a pressure sensor, a tactile sensor, or a slip sensor, used to collect feedback information after the gripper contacts the target to be grasped, and to determine whether the target is stably gripped. When the gripping status does not meet the stable gripping conditions, the gripping status confirmation module transmits the feedback information to the robotic arm end effector and gripper control module, which then regenerates the end effector attitude adjustment command or the gripper approach command.

[0050] Example 2 A target grasping method for an underwater robotic arm based on flow field compensation and multimodal feedback includes the following steps: Collect left and right view images of the underwater operation area, as well as information on water flow velocity and direction; The water flow velocity and direction information are used as image enhancement compensation inputs to perform underwater degradation repair on the left and right view images, resulting in enhanced underwater images. The enhanced underwater image is subjected to target detection and target region segmentation to obtain the detection box, target category, confidence score and target mask of the target to be captured; The target tracking object is initialized according to the user-selected target or task preset target, and the target tracking object is updated in consecutive image frames; Based on the target mask, feature matching and depth estimation are performed on the binocular image to obtain a set of two-dimensional and three-dimensional corresponding points for multiple pixels in the target area; The target pixels in the two-dimensional-three-dimensional corresponding point set are continuously tracked across frames to obtain the position changes of the target pixels in consecutive image frames and maintain the local geometric relationship of the target. The six-degree-of-freedom pose of the target to be grasped relative to the underwater robot body, binocular camera, or end effector is solved based on the two-dimensional-three-dimensional corresponding point set. When the number of effective corresponding points is reduced to three, the pose solution with the highest continuity is selected from multiple candidate pose solutions based on the six-degree-of-freedom pose of the previous frame. Based on the six degrees of freedom pose, generate robotic arm end-effector posture adjustment commands and gripper approach commands to make the robotic arm end-effector approach the target to be grasped along a preset grasping direction; Collect gripper pressure, tactile or gliding feedback information, and determine whether the target to be grasped is stably gripped based on the feedback information; when the stable gripping conditions are not met, readjust the end effector posture of the robotic arm or the gripper approaching action based on the current target six-degree-of-freedom pose and gripper feedback information.

[0051] This invention also provides an underwater robotic arm target grasping system based on flow field compensation and multimodal feedback, including a binocular image acquisition module, a flow velocity and direction acquisition module, a flow field compensation image enhancement module, a target detection and target region segmentation module, a target identity initialization and continuous tracking module, a binocular depth estimation and feature matching module, a pixel-level target point continuous tracking module, a PnP-P3P temporal pose calculation module, a robotic arm end effector and gripper control module, and a gripping status confirmation module.

[0052] The binocular image acquisition module is used to acquire left and right view images of the underwater operation area; the flow velocity and direction acquisition module is used to acquire the water flow velocity and direction of the current operation area; the flow field compensation image enhancement module is used to use the water flow velocity and direction as compensation input to perform degradation repair on the underwater image; the target detection and target region segmentation module is used to identify the target to be captured and output the detection box, target category, confidence score, and target mask; the target identity initialization and continuous tracking module is used to maintain the identity consistency of the selected target in consecutive frames; the binocular depth estimation and feature matching module is used to target... The depth information and corresponding 3D coordinates of pixels within the target area are obtained under mask constraints; the pixel-level target point continuous tracking module is used to track the positioning points within the target area across frames; the PnP-P3P temporal pose calculation module is used to solve the target's six-degree-of-freedom pose based on the 2D-3D corresponding point set, and to perform candidate pose continuity filtering when the number of effective points decreases; the robotic arm end effector and gripper control module is used to generate end effector attitude adjustment commands, proximity commands, and gripper opening and closing commands based on the target's six-degree-of-freedom pose; the gripping state confirmation module is used to determine whether the target is stably gripped based on pressure, tactile, or gliding feedback.

[0053] The working process of this invention includes an image enhancement perception process, a target continuous localization process, and a gripper feedback grasping process.

[0054] I. Image Enhancement Perception Process: 1. After the system starts, the binocular image acquisition module acquires the left and right view images of the underwater operation area, while the flow velocity and direction acquisition module simultaneously acquires the current water flow velocity and direction; 2. The flow field compensation image enhancement module encodes the water flow velocity and direction into a flow field compensation matrix, direction vector, or additional input channel, and inputs them together with the original underwater image into the image enhancement module to compensate for color shift, scattering, blurring, and local distortion in the underwater image; 3. The target detection and target region segmentation module identifies and segments the enhanced underwater image, and outputs the detection box, target category, confidence score and target mask of the target to be captured; 4. The target identity initialization and continuous tracking module creates a target tracking object based on the user-selected target or the task preset target, and updates the target identity in continuous image frames by combining the detection results, target mask and historical motion status.

[0055] II. Target Continuous Positioning Process: 5. The binocular depth estimation and feature matching module performs feature matching and depth estimation within the effective area defined by the target mask, filtering out background areas, water noise, and interference from similar targets; 6. Based on the intrinsic parameters of the binocular camera, the binocular baseline, and the depth map of the target area, convert multiple effective pixels in the target area into three-dimensional coordinate points to form a local two-dimensional-three-dimensional corresponding point set of the target. 7. The pixel-level target point continuous tracking module performs cross-frame continuous tracking of target pixels in the corresponding two-dimensional and three-dimensional point sets, and maintains the local geometric relationship of the target when the target is temporarily occluded, the detection box drifts, or the gripper approaches and causes changes in the field of view. 8. The PnP-P3P temporal pose calculation module selects the pose calculation method based on the number of effective correspondence points: when the number of effective correspondence points is not less than four, the PnP method is used to solve the target six-DOF pose; when the number of effective correspondence points is reduced to three, the P3P method is used to generate multiple candidate pose solutions, and the pose solution with the highest continuity is selected as the current frame output based on the position difference, attitude difference or weighted sum of the two between the candidate pose and the pose of the previous frame.

[0056] III. Gripper Feedback Grabbing Process: 9. The robotic arm end effector and gripper control module generates robotic arm end effector attitude adjustment commands and gripper approach commands based on the target's six-degree-of-freedom pose, so that the robotic arm end effector approaches the target to be grasped along the preset grasping direction; 10. When the gripper approaches and contacts the target to be gripped, the gripping status confirmation module collects gripper pressure, tactile or tactile feedback information. 11. If the feedback information meets the stable gripping conditions, the system determines that the target to be grasped has been stably gripped and completes the grasping action; 12. If the feedback information does not meet the stable gripping conditions, the system will regenerate the robotic arm end-effector attitude adjustment command or gripper approach command based on the current target six-degree-of-freedom pose and gripper feedback information until the stable gripping conditions are met or the task termination conditions are triggered.

[0057] Compared with the prior art, the present invention has the following advantages: (1) High underwater sensing stability: This invention uses water flow velocity and direction as image enhancement compensation inputs, rather than just environmental monitoring data, which can reduce the impact of water flow disturbance, scattering, blurring and local distortion on target detection, target segmentation and feature matching from the source of sensing.

[0058] (2) Low dependence on prior target model: This invention uses target mask, binocular depth estimation and feature matching to construct a local two-dimensional-three-dimensional corresponding point set of the target. It does not require the complete three-dimensional model of the target to be obtained in advance. It is suitable for underwater tasks such as sample collection, inspection, salvage and temporary capture where the target shape is unknown or the target model is incomplete.

[0059] (3) Strong target localization continuity: This invention maintains the local geometric relationship of the target point in continuous image frames by continuously tracking the target point at the pixel level, which can reduce the risk of target loss caused by detection box drift, short-term target occlusion, interference from similar targets and the proximity of the gripper.

[0060] (4) High stability of pose calculation: When the effective corresponding points are reduced to three, the present invention generates multiple candidate pose solutions through the P3P method and performs continuity screening based on the six-degree-of-freedom pose of the previous frame, so that the target pose estimation is transformed from single-frame solution to temporal continuous solution, reducing the risk of pose jump during the grasping process.

[0061] (5) Complete grasping closed loop: The present invention forms a closed loop by combining the target's six degrees of freedom pose, the robotic arm's end-effector posture adjustment, the gripper's proximity control, and pressure, tactile or gliding feedback judgment, which can improve the stability and reliability of underwater robotic arm target grasping under conditions of water flow disturbance and target partial obstruction.

[0062] In summary, the technical solution of this invention can solve problems such as significant image degradation, unstable target detection, discontinuous target pose estimation, easy loss of target during the approach of the gripper, and difficulty in confirming the gripping state in complex underwater environments.

[0063] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target grasping method for an underwater robotic arm based on flow field compensation and multimodal feedback, characterized in that, Includes the following steps: Step 1: Obtain the left and right view images of the underwater operation area, as well as information on water flow velocity and direction; Step 2: Using the water flow velocity and direction information as image enhancement compensation input, underwater degradation repair is performed on the left and right view images to obtain the enhanced underwater image; Step 3: Perform target detection and target region segmentation on the enhanced underwater image to obtain the detection box, target category, confidence score, and target mask of the target to be captured; Step 4: Based on the selected target or task preset target, initialize the target tracking object, and update the target tracking object according to the target detection results and historical motion state in consecutive image frames; Step 5: Define the effective processing area in the binocular image according to the target mask, and perform feature matching and depth estimation within the effective processing area to obtain the two-dimensional-three-dimensional corresponding point set of multiple pixels in the target area; Step 6: Perform cross-frame continuous tracking on the target pixels in the two-dimensional-three-dimensional corresponding point set to obtain the position changes of the target pixels in consecutive image frames and maintain the local geometric relationship of the target. Step 7: Solve the six-degree-of-freedom pose of the target to be grasped relative to the underwater robot body, binocular camera or end effector of the robotic arm based on the two-dimensional-three-dimensional corresponding point set; when the number of effective corresponding points is reduced to three, select the pose solution with the highest continuity from multiple candidate pose solutions based on the six-degree-of-freedom pose of the previous frame. Step 8: Generate a robotic arm end-effector posture adjustment command and a gripper approach command based on the six-degree-of-freedom pose, so that the robotic arm end-effector approaches the target to be grasped along the preset grasping direction; Step 9: Collect gripper pressure, tactile or gliding feedback information, and determine whether the target to be grasped is stably gripped based on the gripper pressure, tactile or gliding feedback information; when the determination result is that it is not stably gripped, regenerate the robotic arm end-effector posture adjustment command or gripper approach command based on the current target six-degree-of-freedom pose and feedback information, until the stable gripping condition is met or the task termination condition is triggered.

2. The underwater robotic arm target grasping method based on flow field compensation and multimodal feedback according to claim 1, characterized in that, In step 2, the water flow velocity and direction information is either a flow field compensation matrix corresponding to the image size or an image enhancement module that is input together with the underwater image after being introduced into the input channel.

3. The underwater robotic arm target grasping method based on flow field compensation and multimodal feedback according to claim 1, characterized in that, In step 2, the underwater degradation repair includes compensating for color shifts, scattering, blurring, and local distortions caused by water flow disturbances in the underwater image.

4. The underwater robotic arm target grasping method based on flow field compensation and multimodal feedback according to claim 1, characterized in that, In step 5, the pixels within the target mask area are converted into three-dimensional coordinate points based on the binocular camera intrinsic parameters, binocular baseline, and target area depth map to obtain the target local two-dimensional-three-dimensional corresponding point set.

5. The underwater robotic arm target grasping method based on flow field compensation and multimodal feedback according to claim 1, characterized in that, In step 6, multiple positioning points within the target area are continuously tracked at the pixel level. When the detection box drifts, the target is temporarily occluded, or the gripper approaches, causing changes in the field of view, the local geometric relationship of the target is maintained based on historical tracking points.

6. The underwater robotic arm target grasping method based on flow field compensation and multimodal feedback according to claim 1, characterized in that, In step 7, when the number of valid corresponding points is not less than four, the target six-degree-of-freedom pose is obtained by the PnP-type pose solving method; when the number of valid corresponding points is less than four, multiple candidate pose solutions are generated by the P3P method.

7. The underwater robotic arm target grasping method based on flow field compensation and multimodal feedback according to claim 1, characterized in that, The pose solution with the highest continuity is determined based on the positional difference, attitude difference, or weighted sum of the two between the candidate pose and the six-DOF pose of the previous frame.

8. An underwater robotic arm target grasping system based on flow field compensation and multimodal feedback, using the method described in any one of claims 1-7, characterized in that, include: The module includes a binocular image acquisition module, a flow velocity and direction acquisition module, a flow field compensation image enhancement module, a target detection and target region segmentation module, a target identity initialization and continuous tracking module, a binocular depth estimation and feature matching module, a pixel-level target point continuous tracking module, a PnP-P3P temporal pose calculation module, a robotic arm end effector and gripper control module, and a gripping status confirmation module. The binocular image acquisition module and the flow velocity and direction acquisition module are connected to the flow field compensation image enhancement module; The flow field compensation image enhancement module is connected to the target detection and target region segmentation module; The target detection and target region segmentation module is connected to the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module, respectively. The pixel-level target point continuous tracking module generates continuous tracking points based on the results of the target identity initialization and continuous tracking module and the binocular depth estimation and feature matching module. The PnP-P3P temporal pose calculation module outputs the target's six-degree-of-freedom pose based on continuous tracking points and corresponding two-dimensional-three-dimensional point sets. The robotic arm end effector and gripper control module generates robotic arm end effector attitude adjustment commands and gripper approach commands based on the target's six-degree-of-freedom pose. The clamping status confirmation module determines the clamping status based on pressure, tactile, or slip feedback, and feeds back the clamping status information to the end effector and gripper control module of the robotic arm when the clamping is not stable.

Images