A vision-based underwater hovering robot assisted motion control method

Abstract

This invention belongs to the field of motion control technology, specifically relating to a vision-based assisted motion control method for underwater floating robots. It includes the following steps: Step 1: Image coordinate system and heading calibration; Step 2: Object recognition and operation trajectory generation; Step 3: Heading control; Step 4: Fixed-point hovering. The beneficial effects of this invention are: (1) Using only a monocular uncalibrated camera, the heading and position control of the floating robot is achieved by recognizing recurring targets in the scene; (2) This invention employs a deep learning method, using a small number of on-site images to train a detection model based on data generation methods, and simultaneously generating a heading planning map based on template images and detection results; (3) Based on the trajectory planning map and odometry information, real-time target detection is combined to achieve closed-loop control of real-time heading and position.

Description

Technical Field

[0001] This invention belongs to the field of motion control technology, specifically relating to a vision-based assisted motion control method for underwater floating robots. Background Technology

[0002] Visual inspection of fuel assemblies (including those inside the reactor core and spent fuel pool), video inspection of the mating surfaces at the bottom of upper reactor components, and video inspection of the bottom of the core basket are important operations during nuclear power plant overhauls. Because the objects being inspected are highly radioactive, these operations are all conducted underwater.

[0003] Compared to general inspection objects, fuel assemblies, the bottom mating surfaces of upper in-core components, and the bottom of the core basket all have visually distinctive, regularly arranged features. For these operations, the commonly used method is to employ mobile carriers, such as underwater floating robots, dedicated mobile platforms, or cranes, to place underwater cameras vertically along the rows of features being inspected.

[0004] Among them, underwater floating robots, with their compact structure, ease of use, and non-interference with the main line, are playing an increasingly important role in underwater operations in the primary circuit pool of nuclear power plants. Currently, underwater floating robots are mostly controlled remotely, meaning operators observe the robot through its onboard camera, moving along characteristic objects and inspecting them row by row. Underwater floating robots generally have depth-keeping and heading-keeping capabilities. Depth-keeping control usually achieves high accuracy and effectiveness; however, heading-keeping control is limited by the inherent drift of the heading angle from the attitude sensor, requiring constant adjustment of the target heading angle value to ensure the robot's actual heading angle remains constant. Furthermore, during operation, underwater floating robots are affected by the umbilical cable and external water currents, making it difficult to maintain a stable horizontal position. Operators need to manually control the robot to overcome external influences and achieve horizontal trajectory tracking and positioning control. Summary of the Invention

[0005] The purpose of this invention is to provide a vision-based auxiliary motion control method for underwater floating robots. By utilizing the regularly arranged visual features on the inspected object, the method enables course control, guidance control, and positioning control of the underwater floating robot, which can effectively improve the operational efficiency of the underwater floating robot.

[0006] The technical solution of the present invention is as follows: A vision-based assisted motion control method for underwater floating robots, comprising the following steps:

[0007] Step 1: Image coordinate system and heading calibration;

[0008] Step 2: Object recognition and task trajectory generation;

[0009] Step 3: Heading control;

[0010] Step 4: Hover at a fixed point.

[0011] Step 1, image coordinate system and heading calibration, is to convert the position and direction offset detected in the image into the floating robot's body coordinate system, which is used to control the floating robot's heading and position.

[0012] In step 1, when the inspection target is above or below the floating robot, if it is above, during the installation process, first ensure that the camera's optical axis is perpendicular to the floating robot's horizontal heading plane or that the imaging plane is parallel to the heading plane, so that the center of the camera image coincides with the center of the floating robot. That is, the marker point corresponding to the center of the floating robot facing upward is located at the center of the camera image. In the image coordinate system, the x and y directions of the image coordinate system are aligned with the x and y directions of the floating robot's heading.

[0013] The calibration process in step 1 involves aligning the image coordinate system with the two-dimensional plane of the floating robot's horizontal plane through rotation and translation. The translation amount represents the distance between the center of the floating robot and the center of the image. First, the translation amount is calibrated by placing a marker directly above the center of the floating robot and obtaining the center pixel coordinates P1 of the marker in the image. The displacement (Δu, Δv) between point P1 and the image center is calculated, which is the relative coordinate system offset. When calibrating the rotation angle, the robot moves 100mm forward along the positive x and y directions respectively, and the changes in the x and y coordinates of the marker pixel (Δu1, Δv1 and (Δu2, Δv2) are detected. The rotation angle θ is then calculated using Equation 1.

[0014]

[0015] Where Δx1=100, Δy1=0, Δx2=0, Δy1=100, and s is the scale factor;

[0016] Once the solution is complete, the rotation and translation parameters in the pixel space can be obtained. A coordinate transformation matrix or an image transformation matrix can then be constructed to transform the coordinates of the acquired image to the image coordinate space aligned with the floating robot.

[0017] Step 2 involves using a deep learning framework for detection and localization. This includes data augmentation processing using real-world captured data, labeling the real-world dataset, and then randomly cropping the entire image. During the random cropping process, randomly generated rotating rectangles are created, with random variables including the rectangle's center point, length, width, and rotation angle to ensure irregular cropping edges. The rectangles must also have an area greater than 200. After cropping, the corresponding image and labeled data are obtained. Then, the cropped data is augmented by changes to rotation, scaling, brightness, contrast, color saturation, blue-green channel enhancement, and noise reduction, increasing the dataset's variability and volume.

[0018] After the dataset is generated in step 2, it is divided into training and testing sets. Then, model training begins. The detection model uses YOLOv8 and is trained using a rotated rectangle model. After training, the model is deployed to the image processing system to output real-time detection results.

[0019] Step 3 involves using a visual method to compare the target alignment direction with the set heading in real time to obtain heading offset data for heading adjustment.

[0020] Step 4 involves selecting a target after initial alignment of the heading with the planned route. The target is controlled within the center of the image. By real-time detection of the target and the selected target within the field of view, the angle between the heading formed by the line connecting the targets and the x-direction of the image is determined. That is, among the many targets in the current field of view, the target closest to the one detected in the previous frame is selected as the target. The positional deviation, angle deviation and positional deviation between the target and the image center are determined. After obtaining these, a closed-loop control method similar to that used in heading control is adopted, adding a set of closed-loop control variables for position, and performing closed-loop control for angle and direction.

[0021] The beneficial effects of the present invention are as follows: (1) Only a single uncalibrated camera is used to control the heading and position of the floating robot by identifying repeated targets in the scene; (2) The present invention adopts a deep learning method to train the detection model based on a small number of on-site images and the data generation method, and generates a heading planning map based on the template image and the detection results; (3) Based on the trajectory planning map and odometer information, the target is detected in real time to achieve closed-loop control of real-time heading and position. Attached Figure Description

[0022] Figure 1 A flowchart of a vision-based assisted motion control method for an underwater floating robot provided by the present invention;

[0023] Figure 2 A schematic diagram showing the installation position directly above the camera;

[0024] Figure 3 A schematic diagram showing the initial installation and alignment of the camera and the floating robot body;

[0025] Figure 4 This is a diagram showing the arrangement of the core stack grid.

[0026] Figure 5 Flowchart for target detection and operation trajectory generation;

[0027] Figure 6 This is a diagram illustrating the heading deviation.

[0028] Figure 7 Here is a flowchart of the heading control algorithm;

[0029] Figure 8 This is a schematic diagram of the positional deviation. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The vision-based motion control method for underwater floating robots provided in this invention is primarily aimed at scenarios where floating robots need to inspect rows of characteristic objects within reactor cores and spent fuel pools. These scenarios typically involve large areas of repetitive structural textures, making global localization difficult. Due to water resistance and current disturbances, the onboard IMU accumulates errors, making it difficult to ensure accurate forward heading. High-precision tracking and inspection operations require manual control, but due to repetitive texture interference, manual remote control is time-consuming and labor-intensive.

[0032] This method addresses the autonomous localization and navigation problem of underwater floating robots in repetitive texture environments. It proposes relative localization based on image visual features and relative heading control based on odometry information, providing floating robots with heading control, hovering control, and track tracking functions.

[0033] like Figure 1 As shown, the hardware components of the vision-based underwater floating robot assisted motion control method of the present invention include a camera installed at the center of the bottom or top surface of the floating robot, an IMU inertial navigation system installed in the floating robot, and a processor for visual data and IMU data internally installed in the floating robot. Due to the low accuracy of the IMU and the accumulation of errors, the approximate course of the floating robot can only be determined based on the IMU data. The camera provides the floating robot's accurate course by collecting environmental information above or below the floating robot's operating environment.

[0034] This method uses a calibration-free single-camera approach to acquire image information. First, image coordinate system and heading calibration are required: the image coordinate system is calibrated with the floating robot's body coordinate system to facilitate the conversion of image-detected offsets into control information for the floating robot. Next, object recognition and trajectory generation are performed: the working object units (fuel tanks, etc.) in the image are identified, and the working trajectory is determined based on the arrangement pattern of the working objects in the image. Then, heading control is implemented: during operation, the floating robot uses a pre-calibrated image and trajectory calibration matrix to determine if there is a heading deviation. If a deviation exists, the heading deviation in the image coordinate system is calculated, and then trajectory control based on image servoing is used to continuously correct the heading deviation until it is consistent with the working trajectory. Finally, hovering is performed: when a long-term targeted operation is required on a specific working unit, the floating robot needs to hover directly above or below the object. This requires locating the working object, calculating the offsets in various directions, and controlling the floating robot to reach the designated object. During the robot's operation, the offsets in each direction need to be calculated in real time to adjust the floating robot's heading and position to ensure hovering.

[0035] Therefore, the key steps involved in this invention mainly include image coordinate system and heading calibration, object recognition and operation trajectory generation, heading control and fixed-point hovering. The overall method flowchart is as follows: Figure 2 As shown.

[0036] A vision-based assisted motion control method for underwater floating robots is proposed. This method addresses the requirements for trajectory control and hovering inspection of regularly geometrically arranged targets, such as reactor cores and lower tube support grids. The overall process is as follows: Figure 1 As shown.

[0037] First, the image coordinate system and the floating robot coordinate system are calibrated. The calibration process only needs to ensure that the calibrated image coordinate system is aligned with the heading coordinate system on the horizontal plane of the floating robot, so as to ensure that the control input of the floating robot can be realized according to the deviation of the image coordinate system.

[0038] Next, it is necessary to identify recurring targets according to the scenario. For example, when inspecting the reactor core, it is necessary to identify the shelves of the reactor core, locate the position of each shelf in the image, extract the set of repeated targets, and formulate the traversal inspection track based on the set of arrangement.

[0039] After target detection and tracking generation are completed, the floating robot's track may deviate due to accumulated errors in its IMU sensor and the influence of water resistance and flow. At this time, image information is needed to detect track deviation based on the detected track information and continuously correct and adjust the floating robot's running direction to ensure the correct heading.

[0040] When it is necessary to perform fixed-point observation and measurement of a target, it is necessary to ensure that the floating robot always stays directly above or below the target to ensure continuous observation over a long period of time. However, due to water flow and insufficient accuracy of its own IMU, there will be positional deviation, making it difficult to achieve accurate hovering control. At this time, it is necessary to use image information captured by the camera to obtain the offset data of the floating robot, so as to continuously adjust the pose of the floating robot and ensure its hovering state to support continuous hovering operations.

[0041] A vision-based assisted motion control method for underwater floating robots includes the following steps:

[0042] Step 1: Image coordinate system and heading calibration

[0043] Image coordinate system and heading calibration are primarily used to transform the detected position and orientation offsets in the image into the floating robot's body coordinate system, which is used to control the floating robot's heading and position. The floating robot's design strictly considers the balance of its various components to ensure it remains level while navigating in water. Furthermore, its onboard sensors, such as the DLV, IMU, and sonar, further ensure accurate horizontal navigation attitude. During underwater inspection operations, typically targeting reactor cores, lower tube mounting grids, etc., the inspection targets are usually located above or below the floating robot. Therefore, the camera is mounted at the center of the floating robot, facing downwards or upwards. Figure 2 The image shows the mounting position of the camera directly above; a similar position is shown directly below.

[0044] Taking the top as an example, during installation, first ensure that the camera's optical axis is perpendicular to the horizontal flight plane of the floating robot, or that the imaging plane is parallel to the flight plane. It is also necessary to ensure that the center of the camera image coincides with the center of the floating robot as much as possible; that is, the marker point corresponding to the center of the floating robot should be as close as possible to the center of the camera image (e.g., ...). Figure 3 (As shown). In the image coordinate system, the x and y directions of the image coordinate system should be aligned as closely as possible with the x and y directions of the floating robot's heading to ensure the most accurate initial calibration. During installation, the alignment requirements for the center or the x and y directions do not need to be too high; only approximate alignment is required. The better the initial alignment, the more reliable the calibration accuracy will be.

[0045] Because the monocular camera cannot obtain depth information during heading and position control, it cannot convert the detected pixel offset into an absolute distance for the control of the floating robot. Therefore, an image-based closed-loop control strategy is adopted on the control side. In this case, it is only necessary to align the x and y directions of the pixel coordinate system with the x and y directions of the floating robot coordinate system. The higher the alignment accuracy, the faster the closed-loop efficiency. Therefore, the calibration process mainly involves rotation and translation alignment of the image coordinate system and the floating robot's horizontal plane in a two-dimensional plane. To eliminate scale interference (the ratio of pixel value to actual distance), the translation amount is the distance between the floating robot's center and the image center. Since underwater images suffer from geometric distortion due to refraction by multiple media, the entire calibration process is performed underwater to minimize errors caused by geometric distortion.

[0046] First, calibrate the translation amount. Place a marker (a small red ball or target plate) directly above the center of the floating robot, ensuring that the distance between the object and the camera is close to the normal shooting distance. Obtain the center pixel coordinates P1 of the marker in the image at this time. Calculate the displacement (Δu, Δv) between point P1 and the center point of the image, which is the relative coordinate system offset. When calibrating the rotation angle, move forward 100mm along the positive x and y directions respectively, and detect the changes (Δu1, Δv1) and (Δu2, Δv2) of the marker pixel coordinates in the x and y directions. Solve the equations (Formula 1) to calculate the rotation angle θ.

[0047]

[0048] Where Δx1 = 100, Δy1 = 0, Δx2 = 0, Δy1 = 100, and s is the scale factor.

[0049] After solving, the rotation and translation parameters in the pixel space can be obtained. A coordinate transformation matrix or image transformation matrix can be constructed to transform the coordinates of the acquired image to the image coordinate space aligned with the floating robot. In order to minimize interference at different scales, the coordinates are calibrated multiple times at different depths to solve the angle. The coordinate transformation matrix is ​​shown in formula (2).

[0050]

[0051] Step 2: Object Recognition and Work Trajectory Generation

[0052] Because the background of the objects being worked on is simple and the number of them is large, they are arranged in a regular pattern across the entire image (e.g., Figure 4As shown, this illustrates the arrangement of the grid structures within the entire core. Due to the influence of shooting angle and lighting, different grid structures exhibit differences in brightness, shadows, and contrast during imaging. Traditional template matching methods are prone to causing missed detections, false detections, and positioning errors. This invention employs a deep learning framework for detection and positioning. However, deep learning-based detection frameworks require a sufficient amount of training data. Directly labeling and training the model for situations with simple backgrounds, uniform data patterns, and limited real-world data collection results in generally poor model detection performance, failing to achieve satisfactory detection and positioning accuracy.

[0053] To address this, this invention proposes a data augmentation process for a small amount of real-world captured data. Since the background of the target object to be predicted is singular, with only variations in target angle, brightness, and contrast, the background of the training dataset does not require excessive variation, and there are numerous detection targets within the entire image. Therefore, the data augmentation process first accurately labels the real-world captured dataset. After labeling, random cropping is performed on the entire image. During random cropping, a randomly generated rotating rectangle is generated, with random variables including the rectangle's center point, length, width, and rotation angle, to ensure irregular cropping edges. Simultaneously, the rectangle's area is greater than 200 to ensure the cropped data is authentic and effective.

[0054] After cropping, the corresponding images and labeled data are obtained. Then, the cropped data is further enhanced by adjusting rotation, scaling, brightness, contrast, color saturation, blue-green channel enhancement, and noise, increasing the dataset's variability and volume.

[0055] After the dataset is generated, it is divided into training and testing sets before model training begins. The detection model uses YOLOv8 and is trained using a rotated bounding box model. Once training is complete, the model can be deployed to the image processing system to output real-time detection results.

[0056] After detecting multiple arranged targets across the entire image area, the center point of the detection box is obtained. Then, combined with known target arrangement data such as a grid layout diagram, the detected target positions are mapped to the layout diagram. This mapping involves two methods: First, manual specification, where the initial corresponding position is manually marked within the area of ​​the entire layout diagram; second, direct matching with the layout diagram using two-dimensional point cloud matching to obtain the correspondence between the image area and the layout diagram. To improve the accuracy and efficiency of layout diagram mapping, we adopt a combined strategy of the two methods: first, manually specifying the approximate corresponding area, and then using two-dimensional point cloud matching to obtain an accurate match with the overall layout diagram.

[0057] First, based on the approximate pose of the floating robot, its general location is determined manually. Combining this with target detection results, a general area corresponding to the global layout map is manually designated. After manual designation, the 2D point cloud formed by the detected target center points is registered with the local point cloud of the designated layout map area. Based on the shape characteristics of the point clouds, a 2D ICP method is used for iterative search, outputting high-precision matching results. After matching, the correspondence between targets within the current field of view can be obtained based on the layout map. The initial point and inspection direction can be manually designated based on the layout map to generate the work trajectory. After manually selecting the starting point and inspection route, the angle between the target connection direction and the inspection direction in the current camera's field of view is calculated. The heading is then adjusted to the initial inspection heading. The overall flowchart is as follows: Figure 5 As shown.

[0058] Step 3: Heading Control

[0059] Heading control is primarily designed to overcome the deviations in heading caused by water resistance, buoyancy, and current thrust during the movement of a floating robot. However, the IMU sensors carried by the floating robot have limited accuracy and accumulate errors, making precise heading correction impossible. Therefore, a vision-based approach is used to compare the real-time target alignment with the set heading, obtaining heading deviation data, which is then fed back to the underlying actuator for heading adjustment.

[0060] The heading control section mainly detects the heading angle. Since the operation route is set according to the arrangement direction of repeated targets when performing inspection tasks, it is only necessary to detect the angle between the arrangement direction of multiple targets in the current field of view and the heading of the floating robot represented by the pre-calibrated image direction to output the heading deviation.

[0061] After the initial image coordinate system and the floating robot coordinate system are calibrated, the camera image will be transformed according to the calibrated rotation and translation deviations to ensure that the center of the image is directly below the hovering position, and that the x and y directions of the image coincide with the x and y directions of the floating robot's horizontal navigation, so as to facilitate observation and subsequent control by the controller.

[0062] After alignment is completed and the initial operational trajectory is generated, the floating robot will adjust its course based on the alignment result between the current view and the overall target layout. It will adjust its position and orientation according to the calculated rotation and translation matrices to ensure alignment with the initially set position and operational route. In other words, it will ensure that the positive direction of the image is aligned with the flight path represented by the line connecting the target.

[0063] During its movement, the floating robot continuously detects targets within its field of view. Multiple detected targets are connected by lines along directions similar to those detected in the previous frame, and these lines represent the preset operational route. The positive x-axis or y-axis of the image (hereinafter referred to as the x-axis) represents the floating robot's forward heading. The heading deviation of the floating robot can be calculated based on the angle between the direction of the connecting line and the x-axis direction.

[0064] After obtaining the heading offset angle, a closed-loop control strategy is employed. Specifically, each time an angle offset exceeding a certain threshold is detected, if the offset is positive, the y-direction thrust is decreased by a fixed amount; if the offset is negative, the y-direction thrust is increased by a fixed amount. This continues until the angle offset falls below the threshold. The control flowchart is as follows: Figure 7 As shown:

[0065] Step 4: Fixed-point hovering

[0066] When a floating robot needs to hover above a target, it must maintain its position and attitude. Similar to heading control, the underwater environment is affected by buoyancy and current thrust, causing deviations in its course. However, the robot's IMU (Insulated Measurement Unit) is often too limited in accuracy to detect these shifts. Therefore, a vision-based approach is necessary for hovering control.

[0067] Typically, hovering detection for floating robots involves detecting the robot's own position and orientation. Orientation detection is similar to heading detection, achieved by real-time detection of the angle between the flight path direction formed by repeating targets in the current camera's field of view and the image's x-direction. Camera position detection involves detecting the offset between the center position of the target directly below and the image center.

[0068] Similar to heading control, after initial alignment of the heading with the planned flight path, a target is selected and kept within the image center. Then, by real-time detection of the target and the selected target within the field of view, the angle between the heading formed by the line connecting the targets and the image's x-direction is determined. Since targets appear repeatedly within the current field of view and are difficult to distinguish, a nearest neighbor approach is used after initial position adjustment. That is, among the many targets in the current field of view, the target closest to the one detected in the previous frame is selected as the target, and its positional deviation from the image center is determined.

[0069] After obtaining the angle and position deviations, a closed-loop control method similar to that used in heading control is adopted, adding a set of closed-loop control variables for position, and performing closed-loop control of angle and direction to ensure fixed-point hovering operation.

[0070] This invention proposes a vision-based assisted motion control method for underwater floating robots, replacing manual remote control. Despite the relatively low accuracy of the robot's IMU (Installation Unit), it overcomes the effects of water resistance, buoyancy, and water flow to achieve high-precision automatic inspection and hovering operations. This invention utilizes a single, ordinary camera, requiring no camera calibration; only pixel-level calibration between the camera image and the floating robot's coordinate system at the operating height. Employing a deep learning framework, it only requires collecting a small number of images of repeating targets, adding minimal annotations, and using data augmentation to generate training data. After training the target detection model, it can detect and locate repeating targets. The data acquisition and annotation workload is relatively small, achieving good detection and localization results, and the method has strong versatility. The operation route is generated by semi-automatically matching the detected repeating targets with a pre-known permutation map, while the operation route is manually customized, improving the method's versatility and robustness. Track control and hovering are achieved by automatically detecting targets, obtaining position and heading deviations, and employing a simple and efficient closed-loop control strategy, ensuring control accuracy and stability. The method has strong applicability and versatility.

[0071] This invention provides a vision-based assisted motion control method for underwater floating robots. It uses only a common single camera and does not require precise camera calibration. It only requires simple calibration of the camera image and the floating robot coordinate system at the operating depth to achieve accurate control of the floating robot's heading and hovering in repetitive target operation scenarios.

[0072] First, ensure the camera is properly positioned on the floating robot, ideally centered on it, so that the image is captured directly in front of the robot while it is moving horizontally. This will guarantee the accuracy of the image and the robot's calibration.

[0073] Next, the image coordinate system and the floating robot coordinate system are calibrated. A marker point is placed relative to the center position of the floating robot at the operating depth. The camera captures the marker point, and the pixel offset between the marker point's pixel coordinates and the image center coordinates is the translation offset. Moving 100mm along the positive x and y directions respectively, the changes in the marker's pixel coordinates in the x and y directions (Δu1, Δv1) and (Δu2, Δv2) are detected, and the rotation angle is calculated by solving simultaneous equations. After obtaining the two-dimensional translation and rotation variables, rotation and translation corrections are performed on each frame of the captured image to align the floating robot's center with the image center, and the floating robot's x and y directions with the image's x and y directions.

[0074] For operations involving repetitive targets, a deep learning framework is employed. This involves labeling a small number of acquired images from the field, randomly cropping the entire labeled data, and then performing data augmentation on the cropped data. The augmented data is then used to train a detection model. The resulting model is used for online real-time target detection. During actual operations, repetitive objects are first detected. The location points of these repetitive objects are matched against a known object arrangement map, and a workflow is generated based on manually labeled operational information. Combining the completed image labeling results, the floating robot is controlled to reach its initial position and heading for inspection operations.

[0075] During patrol operations, when the course deviates due to factors such as water resistance, buoyancy, and water flow, the system uses real-time detected repeating target lines to obtain the current course based on the course information from the previous moment. The angular error between the current course and the x-axis of the image is the course deviation. A closed-loop control strategy is adopted to control the course in a step-by-step manner, gradually reducing the course deviation and ensuring the accuracy of the course.

[0076] When performing hovering maneuvers on a target, it is crucial to maintain the target directly beneath the floating robot at all times, while ensuring the robot's own attitude remains constant. Based on real-time detection of multiple targets within the field of view, similar to heading control, the current direction of the floating robot is determined from the heading information of the previous moment, and the angle between this heading and the x-axis of the image is used to calculate the directional deviation. The current position of the target is obtained from the previous position of the target, and combined with the coordinates of the image center, the positional deviation of the target is calculated. A similar closed-loop control strategy is then employed to control the position and attitude of the floating robot, ensuring accurate and stable hovering operations.

Claims

1. A vision-based assisted motion control method for underwater floating robots, characterized in that, Includes the following steps: Step 1: Image coordinate system and heading calibration; Step 1, image coordinate system and heading calibration, is to transform the position and direction offset detected in the image into the floating robot's body coordinate system, which is used to control the floating robot's heading and position. Step 2: Object recognition and task trajectory generation; Based on the general pose of the floating robot, the approximate area where the floating robot is located is determined manually. Combined with the target detection results, the approximate area corresponding to the global layout map is manually specified. After the manual specification is completed, the two-dimensional point cloud formed by the center point of the detected target is registered with the local point cloud of the specified layout map area. According to the shape features of the point cloud, the two-dimensional ICP method is used for iterative search to output high-precision matching results. After the matching is completed, the correspondence between targets within the current field of view is obtained according to the layout map. The initial point and inspection direction are manually specified according to the layout map to generate the operation trajectory. After manually selecting the starting point and inspection route, the angle between the target connection direction and the inspection direction in the current camera field of view is calculated, and the heading is adjusted to the initial inspection heading. Step 3: Heading control; Step 3 involves using a visual method to compare the target arrangement direction with the set heading in real time to obtain heading offset data for heading adjustment. The heading control section detects the heading angle, identifies the angle between the arrangement direction of multiple targets in the current field of view and the heading of the floating robot represented by the pre-calibrated image direction, and outputs the heading deviation. After the initial operation trajectory is generated, the floating robot adjusts its course according to the alignment result between the current view and the overall target layout. Based on the calculated rotation and translation matrix, it adjusts the position and direction of the floating robot to ensure alignment with the initially set position and operation route, and to ensure that the positive direction of the image is aligned with the route represented by the line connecting the targets. During the movement of the floating robot, targets within its field of vision are detected in real time. Multiple detected targets are connected by lines based on the direction of similarity detected in the previous frame. These lines represent the preset operating route. Step 4: Hover at a fixed point; Step 4 involves selecting a target after initial alignment of the heading with the planned route. The target is controlled within the center of the image. After real-time detection of the target and the selected target within the field of view, the angle between the heading formed by the line connecting the targets and the x-direction of the image is determined. That is, among the many targets in the current field of view, the target closest to the one detected in the previous frame is selected as the target. The positional deviation, angle deviation and positional deviation between the target and the image center are determined. After obtaining these, a closed-loop control method is adopted, adding a set of closed-loop control values ​​for position, and performing closed-loop control for angle and direction.

2. The vision-based assisted motion control method for underwater floating robots as described in claim 1, characterized in that: The calibration process in step 1 involves aligning the image coordinate system with the two-dimensional plane of the floating robot's horizontal plane through rotation and translation. The translation amount corresponds to the distance between the center of the floating robot and the center of the image. First, the translation amount is calibrated. A marker is placed directly above the center of the floating robot, and the center pixel coordinates of the marker in the image are obtained at this time. Calculate at this time Displacement between point and the center point of the image This refers to the relative coordinate system offset. When calibrating the rotation angle, the marker is moved 100mm along the positive x and y directions respectively, and the changes in the x and y coordinates of the pixel are detected at this time. and Solve the rotation angle using Equation 1. , ， in, , The scale factor; Once the solution is complete, the rotation and translation parameters in the pixel space can be obtained. A coordinate transformation matrix or an image transformation matrix can then be constructed to transform the coordinates of the acquired image to the image coordinate space aligned with the floating robot.

3. The vision-based assisted motion control method for underwater floating robots as described in claim 1, characterized in that: In step 1, when the inspection target is above or below the floating robot, if it is above, during the installation process, first ensure that the camera's optical axis is perpendicular to the floating robot's horizontal heading plane or that the imaging plane is parallel to the heading plane, so that the camera's image center coincides with the floating robot's center. That is, the marker point corresponding to the floating robot's center facing upward is located at the camera's image center. In the image coordinate system, the x and y directions of the image coordinate system are aligned with the x and y directions of the floating robot's heading.

4. The vision-based assisted motion control method for underwater floating robots as described in claim 1, characterized in that: Step 2 involves using a deep learning framework for detection and localization. This includes data augmentation processing using real-world captured data, labeling the real-world dataset, and then randomly cropping the entire image. During the cropping process, randomly generated rotating rectangles are created, with random variables including the rectangle's center point, length, width, and rotation angle to ensure irregular cropping edges. The rectangles must also have an area greater than 200. After cropping, the corresponding image and labeled data are obtained. Then, rotation, scaling, brightness, contrast, color saturation, blue-green channel enhancement, and noise reduction are applied to the cropped data to enhance the dataset, increasing its variety and volume.

5. The vision-based assisted motion control method for an underwater floating robot as described in claim 1, characterized in that: After the dataset is generated in step 2, it is divided into training and testing sets. Then, model training begins. The detection model uses YOLOv8 and is trained using a rotated rectangle model. After training, the model is deployed to the image processing system to output real-time detection results.

Images