Underwater robot autonomous inspection system based on binocular vision

Abstract

The invention discloses an underwater robot autonomous inspection system based on binocular vision, and the system remarkably improves the autonomous inspection capability of an underwater robot in a complex environment through the technologies of multi-sensor fusion, deep learning, path optimization and precise control, achieves the high-precision positioning, depth information extraction and smooth path planning, and improves the inspection efficiency. And an innovative technical solution is provided for health monitoring and maintenance of underwater structures.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater robots and their inspection techniques, and in particular relates to an underwater robot autonomous inspection system based on binocular vision. Background Art

[0002] With the development and utilization of global marine resources, the importance of underwater structures in marine engineering has become increasingly prominent. The construction and maintenance of infrastructure such as bridge piers, offshore wind power platforms, offshore oil platforms and submarine cables are key tasks in marine engineering. Traditional underwater inspection methods mainly rely on manual divers or remotely operated underwater robots (ROVs).

[0003] There are many significant disadvantages of using manual divers to conduct underwater inspections. First, the operating costs are high. Not only do divers need to be equipped with professional and expensive diving equipment, such as high-performance diving suits, accurate depth gauges, and reliable communication equipment, but they also need to arrange a professional support team, including surface monitoring personnel and equipment maintenance personnel, which undoubtedly greatly increases the cost of manpower and material resources.

[0004] At the same time, diving operations are extremely dangerous. The underwater environment is complex and changeable, and undercurrents and whirlpools may occur at any time. If you are not careful, you will be swept away by the current and endanger your life. In addition, divers also face health risks such as decompression sickness. Every dive is like a game of danger.

[0005] In addition, the operation is greatly restricted by factors such as water depth, current, and light. As the water depth increases, the water pressure increases dramatically, which puts a huge test on the divers' physical endurance; the turbulent current will interfere with the divers' movements, making it difficult for them to work stably at the target location; and in deep sea areas or extreme environments, the light is weak or even almost dark, which seriously affects the divers' vision, making it difficult to accurately observe and judge the underwater situation. Therefore, it is difficult for artificial divers to carry out long-term and efficient inspection work in deep sea or extreme environments.

[0006] Although the use of ROV can solve the limitations of manual inspections to a certain extent, its operation still relies on manual control and is subject to limitations such as operating depth and communication bandwidth. Especially under complex water conditions, the image recognition and positioning accuracy of the ROV system often cannot meet the needs of high-precision inspections.

[0007] In addition, the particularity of the underwater environment, such as low light, strong current, high turbidity, etc., poses severe challenges to existing visual processing systems. Due to the scattering and absorption of light, underwater images usually show low contrast, blur, distortion and other characteristics, making it difficult for monocular vision systems to accurately obtain effective depth information and target recognition results. Although multi-view vision or other sensors can partially make up for this deficiency, their accuracy and real-time performance are still affected by complex environmental conditions.

[0008] At the same time, the challenges faced by underwater robots in positioning and navigation cannot be ignored. The dynamic changes in the underwater environment lead to great challenges for traditional positioning algorithms in terms of error accumulation and drift. Conventional positioning sensors (such as GPS) cannot be used in underwater environments due to physical limitations, which creates a technical bottleneck for underwater robots to achieve high-precision autonomous inspections.

[0009] Especially in complex underwater environments, robots need to have sufficient autonomy to complete tasks, which usually depends on the precise planning and control of the robot's motion trajectory. Therefore, how to achieve high-precision positioning, depth information extraction, path planning and autonomous inspection of underwater robots in complex environments through multi-sensor fusion, image processing and intelligent control technology has become a technical hotspot and research difficulty in the field of underwater robots. Summary of the invention

[0010] The present invention provides an autonomous inspection system for an underwater robot based on binocular vision, which can realize high-precision positioning, depth information extraction, path planning and autonomous inspection of the underwater robot in a complex environment.

[0011] The present invention provides an underwater robot autonomous inspection system based on binocular vision, comprising: A positioning module includes a front-end tracking unit and a back-end optimization unit, wherein the front-end tracking unit is used to obtain a predicted initial posture through an IMU, track feature points of the left camera image through an LK optical flow method, and then obtain the visual posture of the current frame through a PnP algorithm, obtain a key frame by correcting the predicted initial posture through the visual posture of the current frame, and construct a local map of the sliding window through map points obtained by binocular triangulation of the key frame and the right camera image, and map points obtained by binocular triangulation of the key frame and its previous and subsequent frames, and the back-end optimization unit is used to optimize the key frame in the sliding window and its corresponding map points and camera posture to obtain the actual posture and speed; A perception module is used to perform semantic segmentation on the camera left image through the YOLOv8 method to realize target object recognition, match the camera left image and the camera right image using a stereo matching algorithm to obtain a depth map, obtain the depth information of the target based on the depth map and the recognized target, obtain local point cloud data through the camera intrinsic parameter based on the depth information of the target, calculate the point cloud normal vector and local surface curvature based on the local point cloud data, obtain the geometric features of the target based on the local surface curvature, and construct the local 3D features of the target based on the geometric features of the target; The planning module is used to estimate the initial search trajectory through the depth meter and the ultra-short baseline system, select the trajectory control points based on the actual posture and local 3D features of the target object, and allocate the time consumption of each trajectory segment. The initial search trajectory is optimized based on the trajectory control points and the time consumed by each trajectory segment through the Minimum Snap algorithm to obtain the target posture and target speed; The control module is used to construct attitude, position and velocity errors through actual posture and velocity and corresponding target posture and velocity, control the attitude of the autonomous cruise system by using a sliding mode control method based on the attitude error, and control the position and velocity of the autonomous cruise system by using a double-loop control structure based on the position and velocity errors, thereby realizing autonomous cruising of the autonomous cruise system on underwater structures.

[0012] Preferably, the perception module includes target recognition, target extraction and feature calculation; The target recognition is used to perform real-time semantic segmentation and target detection on the left camera image using the YOLOv8 model, so as to identify the category and position of the target object; The target extraction is used to calculate the depth information of each pixel through a stereo matching algorithm based on the camera left image and the camera right image, thereby obtaining a depth map, and obtaining the depth of the target object through the depth map and the category and position of the identified target object; The feature calculation is used to obtain local point cloud data through camera intrinsic parameters based on the depth information of the target object, calculate the point cloud normal vector through least squares fitting based on the local point cloud data, obtain the local surface curvature through the point cloud normal vector calculation, obtain the geometric features of the target object based on the local surface curvature, and construct the local 3D features of the target object based on the geometric features of the target object.

[0013] Preferably, the target extraction is used to calculate the depth information of each pixel point based on the camera left image and the camera right image through a stereo matching algorithm, including: After matching the camera left image with the camera right image, the depth value of each point in the image is obtained using the triangulation principle to construct a three-dimensional point cloud of the underwater environment.

[0014] Preferably, the initial trajectory search based on the depth meter and the ultra-short baseline system estimation includes: Preliminary positioning information is obtained through the depth meter and ultra-short baseline system, and a global path search is performed for the target object based on the positioning information to form an initial search trajectory.

[0015] Preferably, the trajectory control points are selected based on the actual posture and local 3D features of the target object, including: In the initial search trajectory process, the geometric information of the target object is extracted through surface fitting and feature extraction technology based on the actual posture and local 3D features of the target object. The geometric information of the target object is analyzed to obtain the key structural area, and the trajectory control points passed by the inspection are screened out based on the key structural area.

[0016] Preferably, the initial search trajectory is optimized based on the trajectory control point and the time between the trajectory control points by using the Minimum Snap algorithm to obtain the target posture and target speed, including: Time is allocated to the path between trajectory control points. Based on the trajectory control points and the time allocated to each trajectory control point, the initial search trajectory is optimized by minimizing the high-order derivatives of the trajectory through the Minimum Snap algorithm to obtain the target posture and target velocity.

[0017] Preferably, the control module includes a synovial controller, a position controller and a thrust distribution unit; Wherein, the sliding mode controller is used to obtain the attitude control law through the sliding mode control method based on the attitude error between the current actual attitude and the current target attitude; The position controller is used to obtain a position control law by minimizing the position error and speed error at time t+k using a dual-loop control structure; The thrust distribution unit is used to distribute the thrust based on the attitude control law and the position control law, so as to realize path tracking and attitude control.

[0018] Preferably, the position controller includes a PID controller and an MPC controller; Wherein, the PID controller is used to obtain an initial position control law based on the error between the current actual position and the target position; The MPC controller is used to minimize the position error and speed error at future moments to optimize the initial position control law to obtain the position control law.

[0019] Preferably, it also includes a reconstruction module, which is used to obtain local point cloud data, key frames and corresponding actual postures, align the key frames and the corresponding actual postures with the local point cloud data, and then use the TSDF algorithm to optimize the aligned point cloud data to obtain a three-dimensional model point cloud, and render the target object model based on the three-dimensional model point cloud using 3DGS rendering technology.

[0020] Compared with the prior art, the present invention has the following beneficial effects: Through multi-sensor fusion, deep learning, path optimization and precise control technology, the present invention significantly improves the autonomous inspection capability of underwater robots in complex environments, realizes high-precision positioning, deep information extraction, and smooth path planning, and provides innovative technical solutions for the health monitoring and maintenance of underwater structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an underwater robot autonomous inspection with underwater pile foundations as target structures provided by a specific embodiment of the present invention; Figure 2 A block diagram of an underwater robot autonomous inspection system provided by a specific embodiment of the present invention; Figure 3 A schematic diagram of the structure of an underwater binocular vision module provided by a specific embodiment of the present invention; Figure 4 A schematic diagram of tight coupling optimization of positioning modules provided by a specific embodiment of the present invention; Figure 5 A schematic diagram of local trajectory optimization provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] The present invention proposes an underwater robot autonomous inspection system based on binocular vision, aiming to solve the problems of insufficient positioning accuracy, difficulty in extracting depth information, and unstable path planning and control in existing underwater inspection technologies. The system achieves high-precision positioning and state estimation by fusing the inertial measurement unit (IMU) with binocular vision information, uses the YOLOv8 model for real-time target recognition and depth information extraction of underwater structures, and estimates the morphological characteristics of structures in combination with local point cloud analysis. In order to optimize the motion trajectory of the underwater robot, the system uses the MinimumSnap algorithm to generate a smooth trajectory based on the 3D feature information of the underwater structure, and achieves accurate path tracking and posture stabilization through a sliding mode controller and PID+MPC control strategy. In addition, the system optimizes point cloud data using the TSDF algorithm by fusing positioning and perception data, and combines 3DGS rendering technology for high-precision three-dimensional reconstruction.

[0024] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an exemplary embodiment of the present invention, that is, an underwater robot autonomous inspection diagram using an underwater pile foundation as a target structure. The process uses Figure 3 The underwater binocular vision module shown.

[0025] The specific embodiment of the present invention provides an underwater robot autonomous inspection system based on binocular vision, such as Figure 2As shown, it includes a positioning module, a perception module, a planning module, a control module and a reconstruction module.

[0026] Among them, the positioning module provided by the specific embodiment of the present invention realizes high-precision positioning of the underwater robot by fusing IMU and binocular vision information through tight coupling (tight coupling: jointly optimizing the observation values ​​and state estimates of multiple sensors; loose coupling method processes the data of each sensor separately, and then combines and optimizes the results). The positioning module includes a front-end tracking unit and a back-end optimization unit. The front-end tracking unit uses the IMU to predict the initial pose through the pre-integration method to obtain the predicted initial pose, that is, the preliminary motion estimate. The purpose of pre-integration is to integrate the acceleration and angular velocity information measured by the IMU between two key frames into the motion change within a period of time, and then calculate the position, speed and rotation information of the system. This process can greatly reduce the amount of calculation, effectively reduce the errors caused by IMU data noise and deviation, and provide a more accurate initial estimate for the subsequent fusion of vision and IMU information.

[0027] In a specific embodiment, the specific steps of obtaining the predicted initial posture include: bringing the acceleration and angular velocity measured by the IMU between time i and time j into the pre-integration model to obtain the position, velocity and rotation information of the underwater robot, which is expressed as follows: , define the pre-integrated quantities of position, velocity and angle as follows: , where q represents the quaternion rotation, represents the rotation angle, a represents the acceleration, Represents the gravitational acceleration in the world coordinate system, t is the current time, p is the position, v is the velocity, and q is the rotation.

[0028] At the same time, a specific embodiment of the present invention uses the LK optical flow method to track feature points of the camera's left image, uses the PnP algorithm to solve the current frame pose, and uses Kalman filtering to fuse the visual pose solved by the PnP algorithm with the predicted initial pose predicted by the IMU to achieve pose optimization while screening out key frames.

[0029] In a specific embodiment, the method for selecting key frames includes: 1. disparity: when the disparity between the current frame and the previous key frame is large, it is selected as the key frame; 2. number of feature points: when the number of tracked feature points in the current frame is small, it is selected as the key frame; 3. time interval: when the time interval from the previous key frame is long, it is selected as the key frame.

[0030] The specific embodiment of the present invention constructs a local map of the sliding window by using the map points obtained by binocular triangulation of the key frame and the camera right image, and the map points obtained by binocular triangulation of the key frame and its front and rear frames, and the selected key frame and its corresponding right image are binocularly triangulated, and at the same time, the key frame and its front and rear frame images are also triangulated. Through the triangulation process, the corresponding three-dimensional map points can be obtained from the two-dimensional feature points in the key frame, and these map points provide a preliminary representation of the three-dimensional structure of the local environment. In order to improve the positioning accuracy and the quality of the map, we use the sliding window technology to optimize the state of these map points and key frames. Specifically, the size of the sliding window is set to 10 frames, that is, the system only focuses on the most recent 10 frames of images and related map information. With the addition of new frames, the sliding window will move the earliest frame out of the window, and the latest frame and map point will be added to the optimization process. In the sliding window, all key frames and their corresponding map points and camera poses participate in the optimization together. By minimizing objective functions such as visual reprojection error and IMU residual, the data in the sliding window is continuously optimized, thereby improving the accuracy of pose estimation and map points. The position, velocity, acceleration, angular velocity, and pre-integrated information of the IMU in the sliding window are jointly modeled with the position, posture, 2D feature points, and 3D map points of the camera to form an overall optimization problem. The LM optimization method is used to perform BA (Bundle Adjustment) tight coupling optimization on the above system. The tight coupling strategy is as follows: Figure 4 As shown. The constructed least squares optimization objective function is as follows: ,in is the IMU residual, The IMU residual refers to the error between the pose estimated by the IMU pre-integration and the pose solved by the vision; the reprojection error is the error between the projection of the map point in the three-dimensional space onto the camera image plane and the corresponding feature point in the actual observed image.

[0031] After the back-end optimization provided in the specific embodiment of the present invention, the position, posture and speed of the underwater robot are obtained, and the position and posture are returned to the IMU pre-integration process as initial values ​​to predict the posture of the next stage.

[0032] The perception module provided in the specific embodiment of the present invention is used to realize depth information extraction and target recognition of underwater structures through the YOLOv8 method and the stereo matching algorithm, and semantic segmentation of the camera left image is performed through the YOLOv8 method to realize target recognition. This recognition method can effectively cope with challenges such as low contrast, blur and insufficient lighting in complex underwater environments, thereby ensuring efficient recognition and classification of structures.

[0033] In one embodiment, a convolutional neural network (CNN) sampling YOLOv8 performs feature extraction on an input underwater image and outputs a category label, a bounding box, and a position of the structure, thereby achieving accurate target positioning.

[0034] The specific embodiment of the present invention uses a stereo matching algorithm to match the camera left image and the camera right image to obtain the depth information of each pixel, thereby obtaining a depth map. In a specific embodiment, by matching the left and right images, the depth value of each point in the image is obtained using the triangulation principle, thereby constructing a three-dimensional point cloud of the underwater environment. The specific depth information extraction can be achieved by the following formula: , where Z is the depth of the pixel, f is the focal length of the camera, B is the baseline distance between the two cameras, and d is the parallax of the corresponding point in the image. By obtaining accurate depth information of underwater structures, the accuracy of underwater visual perception is further enhanced.

[0035] After acquiring the local point cloud data of the structure, the specific embodiment of the present invention further calculates the point cloud normal vector and the surface curvature to estimate the morphological characteristics of the underwater structure.

[0036] Specifically, the normal vector calculation is based on the neighborhood points of the set local point cloud, and the normal vector of the point is calculated by least squares fitting: , is the point cloud normal vector, are the coordinates of the local neighborhood points, For each point’s weight, is the center of the point cloud. Then, the curvature of the local surface is calculated through the normal vector information to further evaluate the geometric features and surface state of the underwater structure. Through this series of depth information extraction, point cloud computing and geometric feature estimation, the present invention can accurately obtain the three-dimensional morphological features of the underwater structure, providing strong support for the subsequent underwater robot trajectory planning and target structure reconstruction.

[0037] The planning module provided in a specific embodiment of the present invention is used to search and optimize the inspection path based on the local 3D features of the underwater structure: the planning module is used to estimate the initial search trajectory through a depth meter and an ultra-short baseline system (USBL), and based on the actual posture and the actual posture of the local 3D features of the target object, the initial search trajectory is optimized based on the trajectory control points and the time between the trajectory control points through the Minimum Snap algorithm to obtain the target posture and target speed.

[0038] In a specific embodiment, the present invention estimates an initial trajectory search based on a depth meter and an ultra-short baseline system, including: The embodiment of the present invention obtains preliminary positioning information through a depth meter and an ultra-short baseline system, performs a global path search for a target object based on the positioning information, and forms an initial search trajectory.

[0039] In a specific embodiment, the embodiment of the present invention selects trajectory control points based on the actual posture and local 3D features of the target object, including: In the initial trajectory search process, the embodiment of the present invention extracts the geometric information of the target object through surface fitting and feature extraction technology based on the local 3D features of the target object. The geometric information includes local point cloud information to extract the surface shape, curvature, edge, etc. of the target object. The underwater robot can obtain the key structural area by analyzing the geometric information of the target object, and screen out the trajectory control points passed by the inspection based on the key structural area.

[0040] The underwater robot provided by the specific embodiment of the present invention uses a depth meter and an ultra-short baseline system (USBL) to obtain preliminary positioning information, and based on this, performs a global path search for the underwater structure to be inspected to form a preliminary inspection trajectory. The global path planning process takes into account the complexity of the underwater environment and the relative position of the underwater structure to ensure that the robot can effectively cover the key areas of the target structure. Subsequently, the 3D features are extracted in combination with the local point cloud information of the target structure to screen out the trajectory control points. Specifically, the local point cloud obtains geometric information such as the shape, curvature, and edge of the surface of the underwater structure through surface fitting and feature extraction technology. According to these features, the area with high curvature, complex shape or edge features is screened as the trajectory control point for inspection. In the screening process, in addition to considering the geometric features, it is also necessary to consider the safe distance between the robot and the target object to ensure that the robot will not collide with the target object during the inspection process. Therefore, the selection of control points needs to cover important areas and ensure safety, so as to optimize the inspection path and improve the inspection efficiency and accuracy. After screening out the control points, it is necessary to allocate time for each sub-trajectory formed by these control points. The specific time allocation strategy is as follows: 1. Allocate time based on regional importance: allocate more time for detailed inspection of areas with high curvature, complex surfaces or potential fault areas (such as cracks, joints, etc.); allocate less time for relatively flat and simple areas; 2. Allocate time based on trajectory segment length: allocate more time to longer sub-segments of the trajectory, and less time to shorter sub-segments to maintain the balance of the overall path; 3. Allocate time based on safety distance and speed limit: when the robot approaches a complex area of ​​the target object, in order to ensure safety, reduce the speed and appropriately increase the time to ensure the smooth operation of the trajectory; 4. Allocate time based on maximum acceleration and deceleration limits: according to the maximum acceleration and deceleration limits of the robot, reasonably arrange the time of the acceleration and deceleration process. Sharp acceleration and deceleration will affect the stability and accuracy of the robot, so more time is needed for a smooth transition.

[0041] The embodiment of the present invention takes into account the dynamic constraints of the underwater robot during movement and uses the Minimum Snap algorithm to optimize the local path. The optimization of the local trajectory is shown in the following figure: Figure 5 As shown in the figure. The Minimum Snap algorithm generates a smooth trajectory that complies with dynamic constraints by minimizing the high-order derivatives of the trajectory, thereby avoiding drastic changes and instability in the robot's motion. The algorithm can reduce energy consumption and mechanical load during motion while maintaining the smoothness of motion, effectively improving the robot's maneuverability and stability in complex underwater environments. The optimized trajectory not only meets the robot's dynamic constraints, but also enables the underwater robot to perform inspection tasks more smoothly in narrow underwater spaces through smoothing. The final smooth trajectory provides a feasible and efficient inspection path for the underwater robot, ensuring that the robot can complete comprehensive inspection of underwater structures with a stable posture and precise control.

[0042] The control module provided by the specific embodiment of the present invention realizes the precise path tracking control of the underwater robot based on the planned local target trajectory and positioning feedback. The planning module calculates the expected position, posture and speed information at the current moment according to the local target trajectory. These expected values ​​constitute the goals of the control module for path tracking, guiding the robot to perform inspection tasks along the predetermined trajectory in actual operation. At the same time, with the help of the positioning module mentioned in the first step, the robot can obtain the current actual position, posture and speed information in real time, and compare it with the expected value to achieve precise trajectory tracking. In the control system, the error between the actual position, posture and speed information and the expected value is transmitted to the control module as an input signal.

[0043] The control module provided by the specific embodiment of the present invention consists of two parts, including a posture controller and a position controller. The posture controller adopts a sliding mode control method to achieve the posture stability of the robot. The sliding mode controller can effectively deal with model uncertainty and external disturbances in the system, ensuring that the underwater robot can accurately maintain its posture and remain stable in a complex water flow environment.

[0044] The goal of the sliding mode control provided in the specific embodiment of the present invention is to make the system state tend to the sliding mode surface and remain on the surface. The control law can be expressed as: , ,in, is the output of the sliding mode controller, the force required for the underwater robot to move in each axis; is the sliding surface function, which is a function of the system state error; is the attitude error, is the rate of change of the error, and is the design parameter of the sliding surface function; k is the control gain; is a sign function, which is used to push the system state into the sliding surface. Through this control law, the sliding mode controller can enable the underwater robot to quickly and accurately adjust the attitude error to make it approach zero.

[0045] The position controller provided in the specific embodiment of the present invention adopts a dual-loop control structure, in which the inner loop is a PID controller based on position error and the outer loop is a model predictive control (MPC) controller based on speed. The PID controller adjusts the control input by calculating the error between the current position and the desired position. The initial position control law can be expressed as: ,in, is the output of the PID controller, i.e., the initial position control rate, which is specifically an initial control signal of the MPC controller as the inner loop of the position controller; is the current position error; , , The proportional, integral and differential gains respectively, the PID controller reduces the position error and smoothly guides the robot to the target position by adjusting the control input.

[0046] The MPC controller provided in the specific embodiment of the present invention optimizes the future state of the robot according to the prediction model and outputs the optimal control input. The goal of MPC is to obtain the best control strategy by minimizing the future prediction error and constraints. The optimization problem can be expressed as: ,in, For at the moment , that is, the predicted value of the robot's position and velocity state at the future moment, which is obtained by iteratively bringing the current position and velocity state of the underwater robot and the control input into the robot's kinematic and dynamic model; The target state for the future moment; solving in continuous optimization and iteration This sequence finally obtains the output of the MPC controller, which is the force required for the movement of each axis for the underwater robot. The output of the PID controller is As the initial sequence before optimization iteration; is the regularization factor of the control input; N is the length of the prediction time domain. By optimizing this objective function, MPC can generate the optimal control input based on the current state and future goals of the robot at the current moment to ensure that the robot's trajectory tracking error is minimized.

[0047] The specific embodiment of the present invention can realize accurate path tracking and attitude control of the underwater robot through the cooperation of the above two controllers. The sliding mode controller ensures the rapid stabilization of the robot's attitude, while the combination of PID and MPC controllers ensures the accuracy and stability of position tracking. The combination of sliding mode control and PID+MPC dual-loop control strategy effectively reduces the error accumulation and dynamic instability of the robot in a complex underwater environment, thereby improving the stability and efficiency of the inspection task.

[0048] The reconstruction module provided in the specific embodiment of the present invention integrates the relevant data of the positioning module and the perception module, and performs high-quality three-dimensional reconstruction of the underwater target structure through the 3DGS rendering technology.

[0049] The perception module provided in the specific embodiment of the present invention obtains the three-dimensional depth data of the underwater structure in real time and generates a local point cloud through the depth information extraction technology. At the same time, the positioning module records the key frame and the corresponding camera pose information, including the rotation matrix R and translation vector t of the camera, which helps to accurately align the point cloud and image data under different perspectives.

[0050] In order to achieve accurate three-dimensional reconstruction, a specific embodiment of the present invention first establishes a correspondence between point clouds by matching point clouds and aligning key frame poses. In this process, standard point cloud registration technology is used to ensure accurate alignment of point clouds from each perspective by minimizing reprojection errors. Next, the Truncated Signed Distance Function (TSDF) algorithm is used to optimize the point cloud. The TSDF method maps each point cloud surface into a distance field and performs a weighted update based on the distance from the point to the surface to obtain a dense depth field representation. Specifically, for each three-dimensional point P, the TSDF value d(P) is calculated by calculating the distance from the point to the nearest surface. For each update of the distance function, it can be weighted by the following formula: , It is the newly added point cloud data. is the value in the current distance field, is a weighting factor. By repeatedly updating the distance values ​​of all points, the TSDF algorithm can gradually eliminate noise and optimize the point cloud to generate a high-quality underwater 3D model point cloud.

[0051] Finally, the optimized point cloud is reconstructed in three dimensions using 3DGS (3D Graphics System) rendering technology to form an accurate visualization model of the underwater structure. 3DGS technology can map the optimized three-dimensional point cloud data to the view through pixel-by-pixel rendering calculations, and perform realistic rendering in combination with underwater lighting models, material models, etc. This process can calculate the light intensity of each rendered pixel using the following formula: ,in, I is the light intensity of the pixel, is the lighting model, P is the position of the midpoint in three-dimensional space, L is the light source direction, V is the viewing direction, T is the distance that light travels. Through precise lighting calculation and efficient rendering technology, 3DGS can generate high-quality 3D models of underwater structures, provide clear visual effects, and help in-depth analysis and evaluation of structures during underwater inspection tasks.

[0052] In summary, the present invention provides an underwater robot autonomous inspection system and method based on binocular vision, aiming to solve the problems of insufficient accuracy, low efficiency, difficulty in manual operation and difficulty in ensuring safety in traditional underwater inspection methods.

[0053] Firstly, by fusing IMU and binocular vision information, a tight coupling method is adopted to achieve high-precision positioning of the underwater robot, and the positioning accuracy is improved through an optimization algorithm to provide reliable posture information for subsequent tasks.

[0054] Secondly, based on YOLOv8 semantic segmentation and stereo matching technology, the system can perform target recognition and depth information extraction of underwater structures in real time, and accurately estimate the morphological characteristics of the structures.

[0055] In the third step, the system accurately plans the inspection path by combining local 3D features and optimization algorithms (such as the Minimum Snap algorithm) to generate a smooth trajectory that conforms to dynamic constraints, ensuring that the robot can complete the inspection task efficiently and stably.

[0056] In the fourth step, sliding mode control and PID+MPC dual-loop control strategy are used to realize the path tracking and attitude control of the underwater robot, ensuring the stability and accuracy of the underwater robot in complex environments.

[0057] Finally, the system integrates the data from the positioning and perception modules, optimizes the point cloud through the TSDF algorithm, and uses 3DGS rendering technology to achieve high-quality three-dimensional reconstruction of underwater target structures, providing accurate three-dimensional data support for subsequent defect detection and health assessment.

[0058] The above contents are only the technical ideas of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solutions in accordance with the technical ideas proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. An autonomous inspection system for underwater robots based on binocular vision, characterized in that: include: A positioning module includes a front-end tracking unit and a back-end optimization unit, wherein the front-end tracking unit is used to obtain a predicted initial posture through an IMU, track feature points of the left camera image through an LK optical flow method, and then obtain the visual posture of the current frame through a PnP algorithm, obtain a key frame by correcting the predicted initial posture through the visual posture of the current frame, and construct a local map of the sliding window through map points obtained by binocular triangulation of the key frame and the right camera image, and map points obtained by binocular triangulation of the key frame and its previous and subsequent frames, and the back-end optimization unit is used to optimize the key frame in the sliding window and its corresponding map points and camera posture to obtain the actual posture and speed; A perception module is used to perform semantic segmentation on the camera left image through the YOLOv8 method to realize target object recognition, match the camera left image and the camera right image using a stereo matching algorithm to obtain a depth map, obtain the depth information of the target based on the depth map and the recognized target, obtain local point cloud data through the camera intrinsic parameter based on the depth information of the target, calculate the point cloud normal vector and local surface curvature based on the local point cloud data, obtain the geometric features of the target based on the local surface curvature, and construct the local 3D features of the target based on the geometric features of the target; The planning module is used to estimate the initial search trajectory through the depth meter and the ultra-short baseline system, select the trajectory control points based on the actual posture and local 3D features of the target object, and allocate the time consumption of each trajectory segment. The initial search trajectory is optimized based on the trajectory control points and the time consumed by each trajectory segment through the Minimum Snap algorithm to obtain the target posture and target speed; The control module is used to construct attitude, position and velocity errors through actual posture and velocity and corresponding target posture and velocity, control the attitude of the autonomous cruise system by using a sliding mode control method based on the attitude error, and control the position and velocity of the autonomous cruise system by using a double-loop control structure based on the position and velocity errors, thereby realizing autonomous cruising of the autonomous cruise system on underwater structures.

2. The underwater robot autonomous inspection system based on binocular vision according to claim 1 is characterized in that: The perception module includes target recognition, target extraction and feature calculation; The target recognition is used to perform real-time semantic segmentation and target detection on the left camera image using the YOLOv8 model, so as to identify the category and position of the target object; The target extraction is used to calculate the depth information of each pixel through a stereo matching algorithm based on the camera left image and the camera right image, thereby obtaining a depth map, and obtaining the depth of the target object through the depth map and the category and position of the identified target object; The feature calculation is used to obtain local point cloud data through camera intrinsic parameters based on the depth information of the target object, calculate the point cloud normal vector through least squares fitting based on the local point cloud data, obtain the local surface curvature through the point cloud normal vector calculation, obtain the geometric features of the target object based on the local surface curvature, and construct the local 3D features of the target object based on the geometric features of the target object.

3. The underwater robot autonomous inspection system based on binocular vision according to claim 2 is characterized in that: The target extraction is used to calculate the depth information of each pixel point based on the camera left image and the camera right image through a stereo matching algorithm, including: After matching the camera left image with the camera right image, the depth value of each point in the image is obtained using the triangulation principle to construct a three-dimensional point cloud of the underwater environment.

4. The underwater robot autonomous inspection system based on binocular vision according to claim 1 is characterized in that: Initial trajectory search based on depth meter and ultra-short baseline system estimation, including: Preliminary positioning information is obtained through the depth meter and ultra-short baseline system, and a global path search is performed for the target object based on the positioning information to form an initial search trajectory.

5. The underwater robot autonomous inspection system based on binocular vision according to claim 1 is characterized in that: The trajectory control points are selected based on the actual posture and local 3D features of the target object, including: In the initial search trajectory process, the geometric information of the target object is extracted through surface fitting and feature extraction technology based on the actual posture and local 3D features of the target object. The geometric information of the target object is analyzed to obtain the key structural area, and the trajectory control points passed by the inspection are screened out based on the key structural area.

6. The underwater robot autonomous inspection system based on binocular vision according to claim 1 is characterized in that: The Minimum Snap algorithm is used to optimize the initial search trajectory based on the trajectory control points and the time between the trajectory control points to obtain the target pose and target speed, including: Time is allocated to the path between trajectory control points. Based on the trajectory control points and the time allocated to each trajectory control point, the initial search trajectory is optimized by minimizing the high-order derivatives of the trajectory through the Minimum Snap algorithm to obtain the target posture and target velocity.

7. The underwater robot autonomous inspection system based on binocular vision according to claim 1 is characterized in that: The control module includes a synovial controller, a position controller and a thrust distribution unit; Wherein, the sliding mode controller is used to obtain the attitude control law through the sliding mode control method based on the attitude error between the current actual attitude and the current target attitude; The position controller is used to obtain a position control law by minimizing the position error and speed error at time t+k using a dual-loop control structure; The thrust distribution unit is used to distribute the thrust based on the attitude control law and the position control law, so as to realize path tracking and attitude control.

8. The underwater robot autonomous inspection system based on binocular vision according to claim 7 is characterized in that: The position controller includes a PID controller and an MPC controller; Wherein, the PID controller is used to obtain an initial position control law based on the error between the current actual position and the target position; The MPC controller is used to minimize the position error and speed error at future moments to optimize the initial position control law to obtain the position control law.

9. The underwater robot autonomous inspection system based on binocular vision according to claim 1 is characterized in that: It also includes a reconstruction module, which is used to obtain local point cloud data, key frames and corresponding actual postures, align the key frames and the corresponding actual postures with the local point cloud data, and then use the TSDF algorithm to optimize the aligned point cloud data to obtain a three-dimensional model point cloud, and render the target object model based on the three-dimensional model point cloud using 3DGS rendering technology.

Images