Vision-based underwater floating robot auxiliary motion control method

Abstract

The invention belongs to the technical field of motion control, and particularly relates to an underwater floating robot auxiliary motion control method based on vision. Comprising the following steps: step 1, calibrating an image coordinate system and a course; step 2, object identification and operation track generation; step 3, course control; and step 4, fixed-point hovering. The method has the beneficial effects that (1) only a monocular uncalibrated camera is adopted, and the course and the position of the floating robot are controlled by identifying repeated targets in a scene; (2) a deep learning method is adopted, a detection model is trained based on a data generation mode through a small number of field images, and meanwhile, a course planning map is generated based on a template image according to a detection result; and (3) realizing closed-loop control of real-time course and position by combining real-time target detection based on the track planning map and odometer information.

Description

Technical Field

[0001] The invention belongs to the technical field of motion control, and in particular relates to a vision-based auxiliary motion control method for an underwater floating robot. Background Art

[0002] Visual inspection of fuel assemblies (including: in the core and spent fuel pool), video inspection of the bottom mating surface of upper internal components, and video inspection of the bottom of the core basket are important operations during the overhaul of nuclear power plants. Since the objects of these operations have high radioactivity, these operations are carried out underwater.

[0003] Compared with general inspection objects, fuel assemblies, the bottom mating surface of upper internal components, the bottom of the core basket, etc. all have visual features arranged in a certain regular pattern. For these operations, the commonly used method is to use mobile carriers, such as underwater floating robots, special mobile platforms, cranes, etc., to place underwater cameras in the vertical direction of the inspection object and inspect the characteristic objects row by row.

[0004] Among them, underwater floating robots are increasingly important in underwater operations in the primary water pool of nuclear power plants due to their compact structure, easy use, and no main line occupation. At present, the control of underwater floating robots is mostly carried out by remote control, that is, the operator observes through the camera carried by the robot, moves along the characteristic objects, and inspects row by row. Underwater floating robots generally have the functions of depth and heading. Usually, depth control can achieve high accuracy and effect; but heading control, subject to the inherent characteristics of the heading angle of the attitude sensor drifting, needs to adjust the heading angle target value at any time to ensure that the actual heading angle of the underwater floating robot remains unchanged. In addition, during the operation, the underwater floating robot cannot be well stabilized in the horizontal position due to the influence of the umbilical cable and the external water flow. The operator needs to manually operate to overcome the external influence to achieve horizontal trajectory tracking and horizontal positioning control. Summary of the invention

[0005] The purpose of the present invention is to provide a vision-based assisted motion control method for an underwater floating robot, which utilizes the regularly arranged visual features on the inspected object to perform heading control, guidance control and positioning control on the underwater floating robot, thereby effectively improving the operating efficiency of the underwater floating robot.

[0006] The technical solution of the present invention is as follows: A vision-based underwater floating robot assisted motion control method comprises the following steps:

[0007] Step 1: Image coordinate system and heading calibration;

[0008] Step 2: Object recognition and operation trajectory generation;

[0009] Step 3: Heading control;

[0010] Step 4: Hover at a fixed point.

[0011] The step 1 is image coordinate system and heading calibration, which is to convert the position and direction offset detected in the image into the floating robot body coordinate system, so as to control the heading and position of the floating robot.

[0012] In step 1, when the inspection target is above or below the floating robot, when it is above, during the installation process, first ensure that the camera optical axis is perpendicular to the horizontal heading plane of the floating robot or the imaging plane is parallel to the heading plane, so that the camera image center coincides with the floating robot center, that is, the marking point corresponding to the center of the floating robot upward is in the center of the camera image, and in the image coordinate system, the x and y directions of the image coordinate system are aligned with the x and y directions of the heading of the floating robot body.

[0013] The calibration process in step 1 is to align the image coordinate system with the rotation and translation in the two-dimensional plane of the horizontal plane of the floating robot. The translation amount is the center of the floating robot and the center of the image. First, the translation amount is calibrated. A marker is placed directly above the center of the floating robot, and the central pixel coordinate P1 of the marker in the image at this time is obtained. The displacement (Δu, Δv) between the point P1 and the center point of the image at this time is calculated as the relative coordinate system offset. When calibrating the rotation angle, advance 100 mm along the positive direction of x and y respectively, and detect the changes (Δu1, Δv1) and (Δu2, Δv2) in the x and y directions of the pixel coordinates of the marker at this time. The rotation angle θ is solved by formula 1.

[0014]

[0015] Among them, Δx1=100, Δy1=0, Δx2=0, Δy1=100, s is the scale factor scale;

[0016] After the solution is completed, the rotation and translation parameters of the pixel space can be obtained, and the coordinate transformation matrix or image transformation matrix can be constructed to transform the acquired image coordinates into the image coordinate space aligned with the floating robot.

[0017] The step 2 is to use a deep learning framework for detection and positioning, including data enhancement processing using real shooting data, labeling the real collected data set, and after labeling, random cropping is performed in the entire image. During the random cropping process, a cropped rotated rectangular frame is randomly generated, and the random variables include the center point of the rectangle, the length and width of the rectangle, and the rotation angle of the rectangle to ensure irregular shearing edges. At the same time, the area of ​​the rectangle is greater than 200. After cropping is completed, the corresponding image and labeling data are obtained, and then rotation, scaling, brightness, contrast, color saturation, blue-green channel enhancement, noise, etc. are performed in the cropped data to enhance the data set and increase the changes and data volume of the data set.

[0018] After the data set is generated in step 2, the data set is divided into a training set and a test set, and then model training is started. The detection model uses YOLOV8 and a rotating rectangular box model for training. After the training, the model is deployed to the image processing system to output real-time detection results.

[0019] The step 3 adopts a visual method to detect the target arrangement direction in real time and compare it with the set heading to obtain the heading offset data for heading adjustment.

[0020] The step 4 is to select the work target after the initial alignment of the heading and the route direction of the work plan is completed. The work target is controlled within the center of the image. After real-time detection of the work target and the selected detection target in the field of view, the angle between the heading formed by the line connecting the work target and the x direction of the image is determined in real time. That is, among the many targets in the current field of view, the target closest to the detection in the previous frame is taken as the work target, and the position deviation between it and the center of the image is determined. After the angle deviation and position deviation are obtained, a closed-loop control method similar to that in the heading control is adopted, and a group of closed-loop control quantities of the position are added to perform closed-loop control of the angle and direction.

[0021] The beneficial effects of the present invention are: (1) only a monocular uncalibrated camera is used to control the heading and position of the floating robot by identifying repeated targets in the scene; (2) the present invention adopts a deep learning method to train the detection model based on a data generation method through a small amount of field images, and at the same time, based on the template image, according to the detection results, a heading planning map is generated; (3) based on the track planning map and odometer information, the target is detected in real time to achieve closed-loop control of real-time heading and position. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flow chart of a vision-based underwater floating robot assisted motion control method provided by the present invention;

[0023] Figure 2 This is a schematic diagram of the installation position directly above the camera;

[0024] Figure 3 Schematic diagram of the initial installation and alignment of the camera and the floating robot body;

[0025] Figure 4 This is a diagram of the core grid arrangement style;

[0026] Figure 5 Generate flow charts for target detection and operation trajectories;

[0027] Figure 6 This is a schematic diagram of heading deviation;

[0028] Figure 7 This is the flow chart of the heading control algorithm;

[0029] Figure 8 This is a schematic diagram of position deviation. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The vision-based underwater floating robot assisted motion control method provided by the present invention is mainly aimed at the floating robot in the core and spent fuel pool, which needs to check along the characteristic objects in rows. Such scenes usually have large-scale repetitive structural textures, making global positioning difficult. Due to the influence of water resistance and water flow disturbance, the IMU carried by itself has cumulative errors, making it difficult to ensure the accurate forward heading. High-precision tracking inspection operations require manual control, and due to repetitive texture interference, manual remote control operations are time-consuming and labor-intensive.

[0032] This method aims at the autonomous positioning and navigation problem of underwater floating robots in a repetitive texture environment. A relative positioning method based on image visual features and a relative heading control method based on odometer information are proposed to provide the floating robot with heading control, hovering control and track tracking functions.

[0033] like Figure 1 As shown, the hardware composition of the system used in the vision-based underwater floating robot auxiliary motion control method of the present invention includes a camera installed in the middle of the bottom or the middle of the upper surface of the floating robot, an IMU inertial navigation system installed in the floating robot, and a processor for visual data and IMU data installed inside the floating robot. Due to the low accuracy of the IMU and the accumulation of errors, the approximate heading of the floating robot can only be determined based on the IMU data. The camera provides the floating robot with an accurate heading by collecting environmental information above or below the operating environment of the floating robot.

[0034] In this method, an uncalibrated single camera method is used to obtain image information. First, the image coordinate system and heading calibration are required: the image coordinate system is calibrated with the coordinate system of the floating robot body to facilitate the conversion of the offset detected by the image into the control information of the floating robot; object recognition and operation trajectory generation: the operation object unit (fuel tank, etc.) in the image is identified, and the operation trajectory is determined according to the arrangement rule of the operation object in the image; heading control: during the operation of the floating robot, the floating robot is judged whether there is a heading offset of the floating robot through the pre-calibrated image and track calibration matrix. If there is an offset, the heading offset in the image coordinate system is calculated, and then the track control based on the image servo method is adopted to continuously correct the heading offset until it is consistent with the operation trajectory; fixed-point hovering: when a long-term targeted operation is required for a certain operation unit, the floating robot is required to hover directly above or below the object. At this time, it is necessary to locate the operation object, calculate the offset in each direction, and control the floating robot to reach above or below the specified object. During the robot's operation, it is necessary to calculate the offset in each direction in real time, adjust the heading and position of the floating robot, and ensure the hovering state.

[0035] Therefore, the key steps mainly involved in the present invention include image coordinate system and heading calibration, object recognition and operation trajectory generation, heading control and fixed-point hovering. The overall method flow chart is as follows: Figure 2 shown.

[0036] A vision-based underwater floating robot assisted motion control method is proposed to meet the inspection track control and fixed-point hovering inspection requirements of underwater floating robots in the inspection operation of targets with regular geometric arrangements, such as core and lower pipe seat grid plate inspection. The overall process of the method is as follows Figure 1 shown.

[0037] First, the image coordinate system and the floating robot coordinate system are calibrated. The calibration process only needs to ensure that the corrected image coordinate system is aligned with the heading coordinate system on the horizontal plane of the floating robot to ensure that the control input of the floating robot can be realized according to the deviation of the image coordinate system.

[0038] Next, it is necessary to identify repeated targets according to the scene. For example, when inspecting the core, it is necessary to identify the shelves of the core, locate the position of each shelf in the image, extract the set arrangement of repeated targets, and formulate a traversal inspection track based on the set arrangement.

[0039] After completing target detection and track generation, the floating robot’s own IMU sensor has accumulated errors during the tracking control process, and the water body has resistance, flow, etc., which will cause the track to deviate. At this time, image information is needed to detect the track deviation information based on the detected track information, and continuously correct and adjust the floating robot’s running direction to ensure the correct heading.

[0040] When it is necessary to conduct fixed-point observation and measurement of one of the targets, it is necessary to ensure that the floating robot always stays directly above or below the corresponding target to ensure long-term continuous observation. However, due to the water flow and insufficient IMU accuracy, there will be position offsets, making it difficult to achieve accurate hovering control. At this time, it is necessary to use the image information captured by the camera to obtain the offset data of the floating robot, so as to continuously adjust the position and posture of the floating robot and ensure its hovering state to support continuous hovering operations.

[0041] A vision-based underwater floating robot assisted motion control method comprises the following steps:

[0042] Step 1: Image coordinate system and heading calibration

[0043] The image coordinate system and heading calibration are mainly used to convert the position and direction offsets detected in the image into the coordinate system of the floating robot body, which is used to control the heading and position of the floating robot. The design of the floating robot will strictly consider the weight balance of each part of itself to ensure that the floating robot remains level when sailing in the water, and the sensors it carries, such as DLV, IMU and sonar, will further ensure accurate horizontal navigation posture. When conducting underwater inspection operations, the inspection targets are usually above or below the floating robot, usually for scenes such as the core and the lower pipe seat grid plate. Therefore, the camera will be installed in the center of the floating robot, facing downward or upward. ( Figure 2 The top camera installation position is shown in the figure, and the bottom camera installation position is similar):

[0044] Taking the top as an example, during the installation process, first ensure that the camera optical axis is perpendicular to the horizontal heading plane of the floating robot or the imaging plane is parallel to the heading plane. In advance, it is necessary to ensure that the center of the camera image coincides with the center of the floating robot as much as possible, that is, the mark point corresponding to the center of the floating robot upward is as close to the center of the camera image as possible (such as Figure 3 In the image coordinate system, the x and y directions of the image coordinate system should be aligned with the x and y directions of the heading of the floating robot as much as possible to ensure that the subsequent initial calibration is as accurate as possible. During the installation process, the center or x and y direction alignment requirements do not need to be too high, only rough alignment is required. The better the initial alignment effect, the more reliable the calibration accuracy will be.

[0045] Since the monocular camera cannot obtain depth information during the heading and position control process, it is impossible to convert the detected pixel offset into an absolute distance for floating robot control. Therefore, an image-based closed-loop control strategy is adopted on the control side. At this time, it is only necessary to align the x and y directions of the pixel coordinate system with the x and y directions of the floating robot coordinate system. The higher the alignment accuracy, the faster the closed-loop efficiency. Therefore, the calibration process is mainly the rotation and translation alignment of the image coordinate system and the two-dimensional plane of the floating robot horizontal plane. In order to eliminate the interference of scale (the ratio of pixel value to actual distance), the translation amount is the center of the floating robot and the center of the image. Since underwater images will have geometric distortion due to the refraction of multi-layer media, in order to minimize the errors caused by geometric distortion, the entire calibration process is carried out underwater.

[0046] First, calibrate the translation amount, place a marker (red ball or target plate) just above the center of the floating robot, ensure that the distance between the object and the camera is close to the normal working shooting distance, obtain the center pixel coordinate P1 of the marker in the image at this time, and calculate the displacement (Δu, Δv) between point P1 and the center point of the image at this time, which is the relative coordinate system offset. When calibrating the rotation angle, move 100mm along the positive direction of x and y respectively, detect the changes in the x and y directions of the pixel coordinates of the marker at this time (Δu1, Δv1) and (Δu2, Δv2), and solve the rotation angle θ by the simultaneous equations (Formula 1).

[0047]

[0048] Wherein Δx1=100, Δy1=0, Δx2=0, Δy1=100, and s is the scale factor scale.

[0049] After the solution is completed, the rotation and translation parameters of the pixel space can be obtained, and the coordinate transformation matrix or image transformation matrix can be constructed to transform the acquired image coordinates to the image coordinate space aligned with the floating robot. In order to eliminate the interference problem at different scales as much as possible, multiple calibrations are performed at different depths as much as possible to solve the angle. The coordinate transformation matrix is ​​shown in formula (2).

[0050]

[0051] Step 2: Object recognition and trajectory generation

[0052] Since the objects are located in a single background and are numerous, they are arranged regularly in the entire image format (e.g. Figure 4As shown, the arrangement pattern of the grids in the entire core is shown). Due to the influence of shooting angle and illumination, there are differences in brightness, shadow, contrast, etc. when different grids are imaged. The traditional template matching method is very likely to cause missed detection, false detection and positioning errors. The present invention uses a deep learning framework for detection and positioning. The detection framework based on deep learning requires enough training data. For such cases with a single background, a single data style, and a small amount of real collected data, direct annotation is performed to obtain a training set for training. The obtained model detection effect is general, and it is difficult to achieve good detection and positioning accuracy.

[0053] To this end, the present invention proposes a set of data enhancement processing procedures for a small amount of real shooting data. Since the background of the object to be predicted is single, and only the changes in the target angle, brightness, contrast, etc. are required, the background of the training data set does not need too many changes, and there are more detection targets in the entire image format. Therefore, the data enhancement process will first accurately mark the real collected data set, and after the marking is completed, random cropping will be performed on the entire image. During the random cropping process, a cropped rotated rectangle is randomly generated, and the random variables include the center point of the rectangle, the length and width of the rectangle, and the rotation angle of the rectangle to ensure irregular shearing edges. At the same time, the area of ​​the rectangle is greater than 200 to ensure that the cropped data is true and valid.

[0054] After cropping, the corresponding image and annotation data are obtained. Then, the cropped data is rotated, scaled, and the brightness, contrast, color saturation, blue-green channel enhancement, noise, etc. are changed to further enhance the data set and increase the changes and data volume of the data set.

[0055] After the data set is generated and divided into training and test sets, model training can begin. The detection model uses YOLOV8 and the rotating rectangular box model is used for training. After the training is completed, the model can be deployed to the image processing system to output real-time detection results.

[0056] After detecting multiple arranged targets in the entire image format, the center point position of the detection frame is obtained, and then the detected target position is matched with the layout diagram in combination with known target arrangement data such as the grid layout diagram. This part of the correspondence includes two methods. First, directly rely on manual designation to manually mark the area where the initial corresponding position is in the entire layout diagram; second, directly match the layout diagram, use two-dimensional points for matching, and obtain the correspondence between the image format and the layout diagram. In order to improve the accuracy and efficiency of the layout diagram correspondence, we adopt a strategy that combines the two methods. First, manually specify the approximate corresponding area, and then use two-dimensional point cloud matching to obtain an accurate match with the overall layout diagram.

[0057] First, the general posture of the floating robot is used to manually determine the approximate area where the floating robot is located. Combined with the target detection results, the approximate area corresponding to the global layout diagram is manually specified. After the manual specification is completed, the two-dimensional point cloud formed by the detected target center point is aligned with the local point cloud of the specified layout diagram area. According to the shape characteristics of the point cloud composition, the two-dimensional ICP method is used for iterative search to output high-precision matching results. After the matching is completed, the correspondence between the targets within the current field of view can be obtained according to the layout diagram. According to the layout diagram, the initial point and inspection direction can be manually specified to generate the operation trajectory. After manually selecting the starting point and the inspection route, the angle between the target connection direction and the inspection direction of the current camera shooting field of view is calculated to adjust the heading to the initial inspection heading. The overall flow chart is as follows Figure 5 shown.

[0058] Step 3: Heading Control

[0059] The main purpose of heading control is to overcome the deviation of the heading of the floating robot due to the influence of water resistance, buoyancy, water thrust, etc. during its movement. However, the IMU sensor carried by the floating robot has limited accuracy and cumulative errors, and cannot achieve accurate heading correction control. Therefore, a visual method is used to obtain the heading deviation data by comparing the target arrangement direction with the set heading in real time, and then feed it back to the underlying driver for heading adjustment.

[0060] The heading control part is mainly about the detection of heading angle. When performing inspection tasks, the operation route is set according to the arrangement direction of repeated targets. It is only necessary to detect the angle between the arrangement direction of multiple targets formed in the current field of view and the heading of the floating robot represented by the pre-calibrated image direction to output the heading deviation.

[0061] After the initial image coordinate system and the floating robot coordinate system are calibrated, the camera image will be transformed according to the calibrated rotation and translation deviations to ensure that the center of the image is directly below the hovering point, and the x and y directions of the image coincide with the x and y directions of the floating robot's horizontal navigation, so as to facilitate observation and subsequent control by the control personnel.

[0062] After the alignment is completed and the initial operation trajectory is generated, the floating robot will adjust its heading according to the alignment result of the current view and the overall target layout, and adjust the position and direction of the floating robot according to the calculated rotation and translation matrix to ensure alignment with the initially set position and operation route. In other words, it ensures that the positive direction of the image is aligned with the route represented by the target line.

[0063] During the movement of the floating robot, the targets in the field of view are detected in real time. The detected multiple targets will form a line in the direction close to the detection direction of the previous frame. The line represents the preset operation route. The positive x-axis or y-axis of the image represents the forward heading of the floating robot (hereinafter represented by the x-axis). The heading offset of the floating robot can be calculated based on the angle between the direction of the line and the x-axis direction.

[0064] After obtaining the heading deviation angle, a closed-loop control strategy is used for control. That is, each time the angle deviation is detected to be greater than a certain threshold, if the positive angle deviation is greater than the threshold, the y direction is pushed down by a fixed amount; if the negative angle deviation is greater than the threshold, the y direction is pushed up by a fixed amount. This continues until the angle deviation is less than the threshold. The control flow chart is as follows: Figure 7 As shown:

[0065] Step 4: Fixed-point hovering

[0066] When a floating robot needs to hover above a target, it needs to control its own position and posture to remain unchanged. Similar to heading control, the robot's heading is offset by the buoyancy of the water and the thrust of the water flow in an underwater environment. However, the robot's own IMU is usually difficult to perceive the drift of its own posture due to its limited accuracy. A visual approach is required for hovering control.

[0067] Usually, the hovering detection of floating robots includes the detection of the robot's own position and posture. The detection of posture is similar to the detection of heading, which is achieved by real-time detection of the angle between the route direction composed of repeated targets in the current camera field of view and the x-direction of the image. The position detection of the camera is to detect the offset between the center position of the target directly below and the center position of the image.

[0068] The application scenario is similar to heading control. After the heading is initially aligned with the route direction of the operation plan, the operation target is selected and controlled within the center of the image. At this time, after real-time detection of the operation target and the selected detection target in the field of view, the angle between the heading formed by the connection line of the operation target and the x-direction of the image is determined in real time. Since the operation target appears repeatedly in the current field of view and is difficult to distinguish, the nearest neighbor method is used for determination after the initial position adjustment is completed. That is, among the many targets in the current field of view, the target closest to the detection in the previous frame is used as the operation target, and its position deviation from the center of the image is determined.

[0069] After the angle deviation and position deviation are obtained, a closed-loop control method similar to that in heading control is adopted to increase a set of closed-loop control quantities for positions, and perform closed-loop control of angle and direction to ensure fixed-point hovering operations.

[0070] The invention proposes a vision-based underwater floating robot assisted motion control method, which replaces manual remote control. When the precision of its own IMU is low, it overcomes the influence of water resistance, buoyancy and water flow, and realizes high-precision automatic inspection and hovering operation. The invention can be realized by using an ordinary single camera. The camera itself does not need to be calibrated. It only needs to calibrate the camera image and the floating robot coordinate system at the pixel level at the working height. Using the framework of deep learning, it only needs to collect a small number of repeated target images, generate training data by a small amount of annotation, and train the target detection model to realize the detection and positioning of repeated targets. The workload of data collection and annotation is small, and good detection and positioning effects can be achieved. The method has strong versatility. The generation of the operation route adopts the repeated targets obtained by detection, and semi-automatically matches them with the previously known arrangement diagram. The operation route is manually customized, which improves the versatility and robustness of the method. Track control and fixed-point hovering adopt automatic detection of targets to obtain position and heading deviations, and adopt a simple and efficient closed-loop control strategy to complete, and the control accuracy and stability are guaranteed. The method has strong applicability and versatility.

[0071] The present invention discloses a vision-based assisted motion control method for an underwater floating robot. Only an ordinary single camera is used, and no precise camera calibration is required. It only needs to ensure a simple calibration between the camera image and the floating robot coordinate system at the operating depth, so as to achieve accurate control of the floating robot's heading and hovering in repeated target operation scenarios.

[0072] First, fully ensure the installation position of the camera on the floating robot body, try to make the camera in the middle of the floating robot, and ensure that the image is taken directly relative to the horizontal navigation of the floating robot. This can fully ensure the effect of image and floating robot calibration.

[0073] Then, the image coordinate system and the floating robot coordinate system are calibrated. A marker point is placed at the operating depth relative to the center position of the floating robot. The camera shoots the marker point, and the pixel offset between the pixel coordinates of the marker point and the image center coordinates is the translation offset. Move forward 100 mm in the positive direction of x and y respectively, and detect the changes (Δu1, Δv1) and (Δu2, Δv2) in the x and y directions of the pixel coordinates of the marker at this time, and solve the rotation angle by solving the simultaneous equations. After obtaining the two-dimensional translation and rotation variables, each frame of the captured image is rotated and translated so that the center of the floating robot is aligned with the center of the image, and the x and y directions of the floating robot are aligned with the x and y directions of the image.

[0074] When working in a repetitive target scene, a deep learning framework is used to annotate a small number of images acquired on site, randomly crop the entire annotated data, and then perform data enhancement on the cropped data. The detection model is trained using the expanded data. The obtained model is used for online real-time target detection. In the actual operation process, the repetitive objects are first detected, the positioning points of the repetitive objects are matched with the known object arrangement diagram, and the operation route is generated according to the manually annotated operation information. Combined with the completed image calibration results, the floating robot is controlled to reach the initial position and heading for inspection operations.

[0075] During the inspection process, when the heading is offset due to factors such as water resistance, buoyancy and water flow, the repeated target lines detected in real time are used to obtain the current heading based on the heading information at the previous moment. The angular error between the current heading and the x-axis of the image is the heading deviation. A closed-loop control strategy is used to control the heading in a step-by-step manner to gradually reduce the heading deviation and ensure the accuracy of the heading.

[0076] When performing a fixed-point hovering operation on a target, it is necessary to keep the operating object directly under the floating robot at all times, and the floating robot's own posture remains unchanged. At this time, based on the multiple targets in the field of view detected in real time, similar to the heading control method, the current floating robot direction is obtained based on the heading information at the previous moment, and the direction deviation is obtained by the angle with the x-axis of the image. The corresponding target position at the current moment is obtained based on the position of the operating object at the previous moment, and the position deviation of the operating object is obtained by combining the coordinates of the center position of the image. A similar closed-loop control strategy is used to control the position and posture of the floating robot to ensure accurate and stable hovering operations.

Claims

1. A vision-based underwater floating robot assisted motion control method, characterized in that: The steps include: Step 1: Image coordinate system and heading calibration; Step 2: Object recognition and operation trajectory generation; Step 3: Heading control; Step 4: Hover at a fixed point.

2. A vision-based underwater floating robot assisted motion control method as claimed in claim 1, characterized in that: The step 1 is image coordinate system and heading calibration, which is to convert the position and direction offset detected in the image into the floating robot body coordinate system, so as to control the heading and position of the floating robot.

3. A vision-based underwater floating robot assisted motion control method as claimed in claim 2, characterized in that: In step 1, when the inspection target is above or below the floating robot, when it is above, during the installation process, first ensure that the camera optical axis is perpendicular to the horizontal heading plane of the floating robot or the imaging plane is parallel to the heading plane, so that the camera image center coincides with the floating robot center, that is, the marking point corresponding to the center of the floating robot upward is in the center of the camera image, and in the image coordinate system, the x and y directions of the image coordinate system are aligned with the x and y directions of the heading of the floating robot body.

4. A vision-based underwater floating robot assisted motion control method as claimed in claim 2, characterized in that: The calibration process in step 1 is to align the image coordinate system with the rotation and translation in the two-dimensional plane of the horizontal plane of the floating robot. The translation amount is the center of the floating robot and the center of the image. First, the translation amount is calibrated. A marker is placed directly above the center of the floating robot, and the central pixel coordinate P1 of the marker in the image at this time is obtained. The displacement (Δu, Δv) between the point P1 and the center point of the image at this time is calculated as the relative coordinate system offset. When calibrating the rotation angle, advance 100 mm along the positive direction of x and y respectively, and detect the changes (Δu1, Δv1) and (Δu2, Δv2) in the x and y directions of the pixel coordinates of the marker at this time. The rotation angle θ is solved by formula 1. Among them, Δx1=100, Δy1=0, Δx2=0, Δy1=100, s is the scale factor scale; After the solution is completed, the rotation and translation parameters of the pixel space can be obtained, and the coordinate transformation matrix or image transformation matrix can be constructed to transform the acquired image coordinates into the image coordinate space aligned with the floating robot.

5. The vision-based underwater floating robot assisted motion control method according to claim 1, characterized in that: The step 2 is to use a deep learning framework for detection and positioning, including data enhancement processing using real shooting data, labeling the real collected data set, and after labeling, random cropping is performed in the entire image. During the random cropping process, a cropped rotated rectangular frame is randomly generated, and the random variables include the center point of the rectangle, the length and width of the rectangle, and the rotation angle of the rectangle to ensure irregular shearing edges. At the same time, the area of ​​the rectangle is greater than 200. After cropping is completed, the corresponding image and labeling data are obtained, and then rotation, scaling, brightness, contrast, color saturation, blue-green channel enhancement, noise, etc. are performed in the cropped data to enhance the data set and increase the changes and data volume of the data set.

6. A vision-based underwater floating robot assisted motion control method as claimed in claim 5, characterized in that: After the data set is generated in step 2, the data set is divided into a training set and a test set, and then model training is started. The detection model uses YOLOV8 and a rotating rectangular box model for training. After the training, the model is deployed to the image processing system to output real-time detection results.

7. The vision-based underwater floating robot assisted motion control method according to claim 1, characterized in that: The step 3 adopts a visual method to detect the target arrangement direction in real time and compare it with the set heading to obtain the heading offset data for heading adjustment.

8. The vision-based underwater floating robot assisted motion control method according to claim 1, characterized in that: The step 4 is to select the work target after the initial alignment of the heading and the route direction of the work plan is completed. The work target is controlled within the center of the image. After real-time detection of the work target and the selected detection target in the field of view, the angle between the heading formed by the line connecting the work target and the x direction of the image is determined in real time. That is, among the many targets in the current field of view, the target closest to the detection in the previous frame is taken as the work target, and the position deviation between it and the center of the image is determined. After the angle deviation and position deviation are obtained, a closed-loop control method similar to that in the heading control is adopted, and a group of closed-loop control quantities of the position are added to perform closed-loop control of the angle and direction.

Images