A multi-modal autonomous working system and navigation method for ship cleaning

Abstract

The application discloses a kind of multi-modal autonomous operation systems and navigation method for ship cleaning. The method first synchronously acquires observation data and processes to obtain recovered image sequence, relative motion state and gravity direction;Subsequently, dynamic false features are removed to obtain static matching features and static map points;Then, point re-projection constraint, geometric edge line re-projection constraint, Hessian standard plane constraint and inertial pre-integration constraint are established, and the system absolute pose is obtained by sliding window nonlinear optimization;Finally, based on the recovered image sequence, fine-grained grading labels are output;Based on the system absolute pose and the distance observation data, the global 2.5D probability grid map is cumulatively updated, and the global navigation path and local speed instruction are generated in combination with the preset navigation target pose. The application provides system absolute pose, fine-grained grading labels, global navigation path and local speed instruction for ship cleaning operation.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of ship automation, underwater robots and machine vision navigation. Specifically, it relates to a multimodal autonomous operation system and anti-interference navigation method for complex marine environments and homogeneous paint surfaces on ship hulls. Background Technology

[0002] Ships operate in the high-salt-spray, high-humidity marine environment for extended periods, making their hull plating highly susceptible to complex electrochemical corrosion. The adhesion of marine organisms such as barnacles and algae often damages the physical structure of the anti-corrosion coating, inducing deep pitting corrosion and significantly increasing the surface roughness of the hull. This type of fouling not only weakens the structural strength of the steel plates but also increases the ship's drag, leading to approximately 5%–10% additional power loss and significantly increasing shipping costs and carbon emissions.

[0003] Traditional ship maintenance primarily relies on manual scaffolding for rust removal and cleaning. This method is time-consuming and carries serious safety hazards such as high concentrations of dust, loud noise, and falls from heights. To improve operational efficiency, the industry has gradually introduced automated maintenance equipment. However, when faced with the real and complex marine conditions, existing automated equipment still has significant technical limitations in terms of mobility, perception robustness, and process closed-loop feedback.

[0004] In underwater near-wall operations, mobile vehicles need to navigate under conditions such as curved hull surfaces, attachments, and water current disturbances. Existing systems often struggle to simultaneously handle stable positioning, dynamic interference filtering, and near-wall obstacle avoidance planning under weak texture conditions, resulting in a lack of continuous data link between positioning results and actual navigation requirements.

[0005] At the visual navigation and perception level, underwater cleaning scenarios exhibit typical "dynamic, weak texture" composite characteristics. Cavitation bubbles, turbid silt stirred up by high-pressure jets, and schools of fish create severe high-dynamic interference, easily disrupting the static world assumption of traditional visual simultaneous localization and mapping systems and introducing numerous mismatch points. Simultaneously, the homogeneous anti-rust paint covering the large areas of the hull lacks corner and edge geometric information, forming a visual "feature desert." This directly leads to inevitable and severe cumulative drift in visual odometry during long-distance cruise operations.

[0006] At the target assessment and process control level, most existing defect detection algorithms remain at the coarse-grained "rust / non-rust" binary classification stage, resulting in perception results that are difficult to support differentiated operational decisions. Different evolution stages, such as surface rust, paint peeling, and deep pitting corrosion, have different appearance characteristics; the lack of fine-grained classification models will limit the level of precision and intelligence in ship cleaning operations.

[0007] In summary, how to achieve high-precision pose estimation under strong dynamic and weak texture interference, and how to effectively integrate the pose estimation results with map building, navigation planning, and fine-grained corrosion perception results, has become an urgent problem to be solved in the intelligent development of ship maintenance equipment. Summary of the Invention

[0008] Purpose of the invention: To address the problems of image degradation, dynamic pseudo-feature interference, insufficient spatial constraints in weak texture areas, unstable local obstacle avoidance planning, and insufficient fine-grained corrosion perception capabilities in existing ship cleaning operations in complex underwater environments, this invention proposes a multimodal autonomous operation system and navigation method for ship cleaning, in order to improve underwater navigation and positioning, local environment adaptation, and corrosion perception capabilities.

[0009] The above objectives are achieved through the following technical solutions:

[0010] This invention first provides a multimodal autonomous operation navigation method for ship cleaning, which includes the following steps:

[0011] S1. Spatiotemporal data preprocessing and anisotropy restoration: Synchronously acquire binocular image observation data stream, distance observation data and body dynamics time series data of the working environment, and perform time alignment; preprocess the binocular image observation data stream to obtain the restored image sequence, and perform continuous pre-integration on the body dynamics time series data to obtain the relative motion state and gravity direction of adjacent time moments;

[0012] S2. Decoupling of Static and Dynamic Features and Non-rigid Dynamic Filtering: Candidate 2D matching point pairs from adjacent keyframes are extracted from the restored image sequence obtained in S1, and the symmetrical epipolar geometric distance of each candidate 2D matching point pair is calculated. For each candidate 2D matching point pair, 3D points are predicted. Based on the relative motion state obtained in S1, the 3D points in the camera coordinate system of the previous keyframe are transformed to the camera coordinate system of the next keyframe, and the 3D geometric consistency residual is calculated. Based on the symmetrical epipolar geometric distance and the 3D geometric consistency residual, mask removal is performed on the corresponding candidate 2D matching point pairs to obtain retained static matching features. The retained static matching features are triangulated to obtain static map points.

[0013] S3. The point reprojection constraints formed by preserving static matching features and static map points, the geometric edge line reprojection constraints formed by geometric edge line features, and the Hessian canonical plane constraints formed by Hessian canonical plane features are entered into the back-end optimization to obtain the absolute pose of the system.

[0014] S4. Fine-grained rust classification and navigation planning: Based on the restored image sequence obtained in S1, extract the region of interest of rust defects, and classify them by fusing color, texture and semantic features to output fine-grained classification labels; output the fine-grained classification labels to the external operation strategy module as the basis for selecting differentiated cleaning strategies in the future.

[0015] The system acquires the preset navigation target pose, and updates the global 2.5D probabilistic grid map based on the system absolute pose obtained in S3 and the distance observation data collected in S1. A global navigation path is generated using the A* algorithm, starting from the grid corresponding to the system absolute pose and ending at the grid corresponding to the preset navigation target pose. A local obstacle sub-map centered on the current system absolute pose is extracted from the global 2.5D probabilistic grid map, and the current motion state is determined based on the system absolute pose at adjacent times. A local velocity command is generated using the dynamic window method. The global navigation path and the local velocity command are sent to the mobile carrier execution unit to execute the corresponding navigation operation.

[0016] The present invention also provides a multimodal autonomous operation system for ship cleaning to implement the above-mentioned navigation method, comprising:

[0017] A multimodal environment data acquisition array is configured to acquire binocular image observation data streams, distance observation data, and body dynamics time-series data of the working environment in real time and synchronously, and send the binocular image observation data streams, the distance observation data, and the body dynamics time-series data to the edge computing processing unit;

[0018] An edge computing processing unit is electrically connected to the multimodal environment data acquisition array and internally deploys an anti-interference state estimation engine, a corrosion fine-grained perception module, and a navigation planning module. The navigation planning module is configured to receive a preset navigation target pose. The edge computing processing unit is configured to execute the navigation method according to any one of claims 1 to 5 based on the binocular image observation data stream, the distance observation data, the body dynamics time-series data, and the preset navigation target pose, and output the system absolute pose, fine-grained classification label, global navigation path, and local velocity command. The corrosion fine-grained perception module outputs the fine-grained classification label to an external operation strategy module, and the fine-grained classification label is not used as input for localization, mapping, or navigation planning.

[0019] The mobile carrier execution unit is communicatively connected to the edge computing processing unit and is configured to receive the global navigation path and the local speed command, and to perform navigation operations based on the local speed command.

[0020] The advantages of this invention compared to the prior art are:

[0021] By restoring images from binocular image observation data streams and continuously pre-integrating the body dynamics time-series data, restored image sequences, relative motion states, and gravity directions are obtained, providing continuous input for subsequent feature processing and pose optimization. By comparing the symmetrical epipolar geometric distance and the 3D geometric consistency residual with their respective thresholds, dynamic pseudo-features are eliminated, and static matching features and static map points are retained, reducing the impact of dynamic interference on backend optimization. By jointly incorporating inertial pre-integration constraints, point reprojection constraints, geometric edge line reprojection constraints, and Hessian canonical plane constraints into sliding window optimization, the system's absolute pose is obtained, enhancing geometric constraints in weakly textured regions. By using fine-grained grading labels as the grading basis for subsequent selection of differentiated cleaning strategies by the external operation strategy module, and ensuring their independence from the data links of positioning, mapping, and navigation planning, the misuse of perception labels for navigation calculations is avoided. By generating global navigation paths and local velocity commands based on the system's absolute pose, distance observation data, and preset navigation target pose, the coordination between navigation and perception during ship cleaning operations is improved. Attached Figure Description

[0022] Figure 1 For logical roadmap;

[0023] Figure 2 A multi-feature compensation model and optimization framework diagram for weak texture of homogeneous paint surface (point-line-surface).

[0024] Figure 3 Structure diagram of the multi-dimensional feature fusion corrosion fine-grained classification algorithm;

[0025] Figure 4 A trajectory comparison chart;

[0026] Figure 5 A comparison chart of positioning errors in the X direction;

[0027] Figure 6 This is a schematic diagram of the underwater mobile carrier structure used to implement the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 As shown, this embodiment provides a multimodal autonomous operation system and navigation method for ship cleaning. Figure 1The data flow is demonstrated after processing binocular image observation data, distance observation data, and body dynamics time-series data through S1, S2, S3, and S4: S1 obtains the restored image sequence, relative motion state, and gravity direction; S2 obtains static matching features and static map points from candidate 2D matching point pairs; S3 obtains the system's absolute pose; the rust fine-grained classification branch of S4 outputs fine-grained classification labels for use by the external operation strategy module, while the map building and navigation planning branch outputs the global navigation path and local velocity commands. The two branches are independent of each other. The core implementation steps are detailed below:

[0030] S1. Spatiotemporal Data Preprocessing and Anisotropic Restoration. The system synchronously acquires binocular image observation data streams, distance observation data, and body dynamics time-series data. The binocular image observation data streams include the left and right eye images synchronously acquired by the binocular vision sensors at each acquisition moment. The body dynamics time-series data includes the angular velocity and linear acceleration output by the high-frequency inertial measurement unit. The binocular image observation data streams, distance observation data, and body dynamics time-series data are aligned using hardware timestamps. To address image degradation caused by underwater light absorption, scattering, and suspended matter, contrast restoration, edge restoration, and anisotropic diffusion processing are performed on the acquired left and right eye images, respectively, to obtain restored left and right eye images. The obtained restored left and right eye images are then arranged into a restored image sequence according to the acquisition timestamps. Continuous pre-integration is performed on the time-series data of the body dynamics to obtain the relative motion state and gravity direction at adjacent time points, which are used by S2 to calculate the three-dimensional geometric consistency residual and S3 to construct the inertial pre-integration constraint; the distance observation data is retained for S4 to cumulatively update the global 2.5D probabilistic grid map.

[0031] S2. Decoupling of static and dynamic characteristics and filtering of non-rigid dynamics. This addresses dynamic disturbances such as fish schools, bubbles, and suspended particles. Figure 1 As shown, candidate two-dimensional matching point pairs of adjacent keyframes are extracted from the restored image sequence obtained in S1, and RANSAC is used to estimate the fundamental matrix of the candidate two-dimensional matching point pairs to obtain the fundamental matrix. Based on the fundamental matrix Calculate the symmetric epipolar geometric distance for each candidate 2D matching point pair. The geometric distance of the symmetrical epipolar line As shown in formula (1): In the formula, and These are the same candidate 2D matching point pairs in adjacent keyframes. Time and Two-dimensional homogeneous coordinates at time; This is the fundamental matrix obtained based on robust estimation of all candidate two-dimensional matching point pairs; The symmetric epipolar geometric distance between the candidate two-dimensional matching point pair is given; the superscript T indicates matrix transpose.

[0032] For each candidate two-dimensional matching point pair, respectively using Time and The left and right reconstructed images at the corresponding time points are subjected to binocular triangulation to obtain... 3D points in the camera coordinate system and 3D points in the camera coordinate system Based on the relative motion state obtained from S1, from Transformation of camera coordinate system to The camera coordinate system at any given time is used to obtain the predicted 3D points. and the predicted three-dimensional points and The Euclidean distance between them serves as the three-dimensional geometrically consistent residual. The three-dimensional geometric consistency residual r3ᴰ is shown in formula (2): In the formula, and For the same candidate two-dimensional matching point pair Time camera coordinate system and Three-dimensional points obtained by binocular triangulation in the time-lapse camera coordinate system; and These are the rotation matrix and translation vector in the relative motion state obtained in S1, respectively; Indicates will from Transformation of camera coordinate system to The predicted 3D points obtained after the camera coordinate system is set at a given time.

[0033] To determine whether each candidate 2D matching point pair satisfies the static rigid body assumption, the symmetric epipolar geometric distance is... Distance threshold from preset polar lines Compare and use the three-dimensional geometric consistency residuals Compared with the preset three-dimensional residual threshold Comparison. When or When the corresponding candidate two-dimensional matching point pairs are identified as dynamic pseudo-features, masking is performed to remove them; and when both conditions are met... and Candidate two-dimensional matching point pairs are determined as retained static matching features. The retained static matching features are triangulated to obtain static map points, and the retained static matching features and the static map points are passed to S3 as point observation input.

[0034] S3, weak texture heterogeneous feature tight coupling positioning.

[0035] like Figure 2 As shown in the figure, this figure illustrates how the point reprojection constraints formed by preserving static matching features and static map points, the geometric edge line reprojection constraints formed by geometric edge line features, and the Hessian canonical plane constraints formed by Hessian canonical plane features are incorporated into the back-end optimization framework under the condition of weak texture on a homogeneous paint surface. S3 establishes point reprojection constraints using the preserving static matching features and static map points obtained in S2; extracts the geometric edge line features corresponding to the hull geometric edges and welds from the restored image sequence obtained in S1, and uses six-dimensional Plück coordinates to parameterize the geometric edge line features, as shown in formula (3).

[0036] In the formula, The six-dimensional Plück coordinates are for the geometric edge line features; The moment vector of the geometric edge line features relative to the origin. Let be the direction vector of the geometric edge line feature, and and Satisfies orthogonal constraints; This represents a six-dimensional real vector space.

[0037] During the back-end sliding window nonlinear optimization process, when iteratively updating the parameters of the geometric edge line features, the Plück coordinates are projected onto a four-degree-of-freedom orthogonal representation space, and an increment is applied to the corresponding tangent space. A set of locally coplanar points is determined from the static map points obtained in S2, and a Hessian canonical plane feature is fitted. This Hessian canonical plane feature is used to form the Hessian canonical plane residual. The back-end sliding window uses the pose, velocity, gyroscope bias, and accelerometer bias of each keyframe as the state variable to be estimated, X. Based on the relative motion state and gravity direction obtained from S1, it constructs the inertial pre-integration residual. The point reprojection residuals are formed based on the static matching features and static map points obtained from S2. Based on the geometric edge line features, a geometric edge line reprojection residual is constructed. Based on the aforementioned Hessian canonical plane features, Hessian canonical plane residuals are constructed. When the sliding window moves out of the earliest keyframe, the state variables and observation factors associated with that keyframe are marginalized to form prior constraints that retain the state variables, and marginalized prior residuals are constructed. The above five types of residuals are used to construct a full-dimensional joint cost function, which is shown in formula (4).

[0038] In the formula, Let X be the full-dimensional joint cost function with the state variable to be estimated as the independent variable; To marginalize prior residuals, The inertial pre-integral residual is formed by the relative motion state and the direction of gravity obtained from S1. The point reprojection residuals obtained from S2, consisting of preserved static matching features and static map points, The reprojection residuals of geometric edge lines, which are composed of geometric edge line features. The Hessian canonical plane residual is formed by the plane features of the Hessian canonical form. , , , and These are the preset weights for the five types of residuals mentioned above.

[0039] Finally, the absolute pose of the system is obtained by iterative optimization using the Levenberg-Marquardt nonlinear least squares algorithm.

[0040] S4. Rust fine-grained classification and navigation planning. For example... Figure 3 As shown, S4 includes two independent branches: a fine-grained corrosion classification branch and a map building and navigation planning branch. The fine-grained corrosion classification specifically uses the restored image sequence obtained in S1 as input, performs region-of-interest localization of corrosion defects, fusion of color, texture, and semantic features, principal component analysis, and multilayer perceptron classification, outputting fine-grained classification labels. These labels are then provided to the external operation strategy module as the basis for selecting differentiated cleaning strategies. When the map building and navigation planning branch first obtains the system absolute pose output by S3, it uses the unified coordinate system where the system absolute pose is located as a reference and builds a global 2.5D probabilistic raster map according to a preset raster resolution and initial occupancy probability. Subsequently, based on the continuously output system absolute pose of S3, it transforms the distance observation data collected and retained by S1 to the unified coordinate system, cumulatively updates the global 2.5D probabilistic raster map, and outputs a global navigation path and local velocity command in conjunction with the preset navigation target pose. The specific process of S4 is as follows:

[0041] S41 Lightweight Rust Anchoring: For the hull region in the restored image sequence obtained in S1, a YOLOv8 network embedding spatial attention and channel attention modules is used to locate the region of interest (ROI) for rust defects. The spatial attention module is used to highlight the edges and positional relationships of suspected rust areas, and the channel attention module is used to enhance feature channels related to the appearance of rust. The YOLOv8 network embedding the spatial attention and channel attention modules outputs the ROI for the rust defects.

[0042] S42 Color, Texture and Semantic Feature Fusion: Feature cascading is carried out within the region of interest of the rust defects located in S41. In the HSV color space, formulas (5), (6) and (7) are used to calculate the mean, standard deviation and skewness of each color channel, respectively; mean (mean) (Reflecting the overall color tone) Standard deviation (Reflecting the breadth of color distribution) Skewness (skewness) (Reflecting the asymmetry of color distribution) sᵢ = (1 / N) ∑ⱼ₌1ᴺ ((pᵢⱼ − μᵢ) / σᵢ)³ (7) In the formula, Let be the j-th pixel value of the i-th HSV color channel within the region of interest for the corrosion defect, and N be the total number of pixels in that region. , and These are the mean, standard deviation, and skewness of the i-th color channel, respectively. These statistics form the apparent feature vector.

[0043] The region of interest (ROI) of the corrosion defect is converted into a grayscale image. Under a preset pixel spacing and preset grayscale levels, grayscale co-occurrence matrices are constructed in four directions: 0°, 45°, 90°, and 135°. The angular second moment, contrast, correlation, and inverse difference moment in each of the four directions are calculated, and the mean values ​​of these parameters are calculated to obtain texture feature vectors. A joint feature vector is formed by globally average pooling the apparent feature vector (composed of the mean, standard deviation, and skewness of each color channel), the texture feature vector, and the feature map output from the YOLOv8 network backbone.

[0044] S43 PCA Dimensionality Reduction and Multilayer Perceptron Refined Classification: In this embodiment, the fine-grained classification of rust is pre-defined into three categories: light surface rust, moderate spalling, and severe pitting corrosion. The joint feature vector obtained in S42 is dimensionality reduced using the principal component analysis module to obtain principal component subspace features. These principal component subspace features are then input into a three-hidden-layer multilayer perceptron, and the probability distribution of each category is obtained through a normalized exponential function. Three fine-grained classification labels—light surface rust, moderate spalling, and severe pitting corrosion—are output. These fine-grained classification labels are then output to the external operation strategy module as the basis for selecting differentiated cleaning strategies in the future.

[0045] S44 Map Construction and Path Planning: Based on the system absolute pose obtained in S3, the distance observation data retained in S1 is transformed to a unified coordinate system and the global 2.5D probabilistic grid map is updated cumulatively. Taking the grid corresponding to the current system absolute pose as the starting point and the grid corresponding to the preset navigation target pose as the ending point, the global navigation path is searched on the global 2.5D probabilistic grid map using the A* algorithm. A local obstacle sub-map centered on the current system absolute pose is extracted from the global 2.5D probabilistic grid map for use in evaluating local velocity candidates using the dynamic window method.

[0046] Then, based on the system's absolute pose at adjacent moments, the current linear velocity and angular velocity are determined. A dynamic window is constructed by combining preset maximum linear velocity, maximum angular velocity, linear acceleration constraints, angular acceleration constraints, and safety distance constraints. Local velocity candidates are generated within the velocity space (u, ω) defined by the dynamic window. The comprehensive evaluation value of each local velocity candidate is calculated using formula (8). : in Candidate values ​​for linear velocity. Candidate values ​​for angular velocity; , and These are, respectively, a normalized global navigation path heading consistency evaluation term, a safe distance evaluation term based on the local obstacle subgraph, and a speed evaluation term; , and The preset weights for the corresponding evaluation items.

[0047] Among the local velocity candidates that satisfy the safety distance constraint and motion constraint, the candidate that makes G(u,ω) reach its maximum value is selected as the local velocity command.

[0048] The global navigation path and the local speed command are sent to the mobile vehicle execution unit, thereby completing the navigation operation under the constraints of the global 2.5D probabilistic grid map and the local obstacle submap.

[0049] The present invention provides a multimodal autonomous operation system for ship cleaning, comprising a multimodal environmental data acquisition array configured to acquire binocular image observation data stream, distance observation data, and body dynamics time-series data of the operation environment in real time and synchronously, and to send the binocular image observation data stream, the distance observation data, and the body dynamics time-series data to an edge computing processing unit;

[0050] An edge computing processing unit is electrically connected to the multimodal environment data acquisition array. It internally deploys an anti-interference state estimation engine, a corrosion fine-grained perception module, and a navigation planning module. The navigation planning module is configured to receive a preset navigation target pose. The edge computing processing unit is configured to execute the navigation method based on the binocular image observation data stream, the distance observation data, the body dynamics time-series data, and the preset navigation target pose, outputting the system absolute pose, fine-grained classification labels, global navigation path, and local velocity commands. The corrosion fine-grained perception module outputs the fine-grained classification labels to an external operation strategy module; these labels are not used as input for localization, mapping, or navigation planning.

[0051] The mobile carrier execution unit is communicatively connected to the edge computing processing unit and is configured to receive the global navigation path and the local speed command, and to perform navigation operations based on the local speed command.

[0052] The multimodal environmental data acquisition array integrates a binocular vision sensor, a high-frequency inertial measurement unit, and a distance sensor. The binocular vision sensor and the high-frequency inertial measurement unit are time-stamp aligned via a hardware synchronization trigger circuit. The binocular vision sensor synchronously acquires left and right eye images at each acquisition moment to form a binocular image observation data stream. The high-frequency inertial measurement unit acquires body dynamics time-series data, and the distance sensor acquires distance observation data. The binocular image observation data stream is used to obtain a restored image sequence, retain static matching features, and static map points. The body dynamics time-series data is used to obtain the relative motion state and gravity direction. The distance observation data and the system's absolute pose are used to cumulatively update the global 2.5D probabilistic grid map.

[0053] like Figure 6 As shown, the mobile carrier execution unit can be implemented using the underwater mobile carrier illustrated. This mobile carrier includes a sealed hull, a frame for mounting sensors and a computing unit, and thruster assemblies disposed around the frame. Binocular image observation data streams, body dynamics timing data, and distance observation data acquired by the binocular vision sensor, inertial measurement unit, and distance sensor are transmitted to the edge computing processing unit. This processing unit sends the global navigation path and local velocity commands obtained in step S4 to the mobile carrier execution unit, which then executes the corresponding heading and velocity commands. Figure 6 This is used to illustrate the scenario of the invention being mounted and executed on an underwater mobile carrier.

[0054] Experimental verification:

[0055] In a preferred embodiment, to verify the navigation and positioning performance of the patented method in a weakly textured underwater environment, the ORB-SLAM3 baseline was selected as a comparison method, and the patented method was used as the complete method for experimental comparison. During the experiment, both methods were analyzed based on the same underwater image sequence and reference trajectory to ensure consistent comparison conditions.

[0056] like Figure 4 As shown in the figure, the trajectory comparison results between the ORB-SLAM3 baseline, the method of this patent, and the reference trajectory are presented. Figure 4 As can be seen, within the local trajectory segment shown, the trajectory obtained by the method of this patent maintains a good overall fit with the reference trajectory; in contrast, the ORB-SLAM3 baseline shows more significant deviations from the reference trajectory at locations with obvious local turns and trajectory changes. This result demonstrates that the method of this patent can improve trajectory following performance even when underwater image quality is limited and local texture is insufficient.

[0057] like Figure 5 As shown in the figure, the positioning error variation in the X direction between the ORB-SLAM3 baseline and the method of this patent is illustrated. Figure 5 It can be seen that the X-direction error of the ORB-SLAM3 baseline fluctuates significantly in some time periods, with obvious error peaks; the error curve of the proposed method exhibits relatively smaller overall fluctuations, remaining at a low level for most time periods. This result demonstrates that the proposed method can achieve certain improvements in X-direction positioning accuracy and error stability.

[0058] In summary, combining Figure 4 The trajectory comparison results and Figure 5 The X-direction positioning error results show that, under the same experimental conditions, the method of this patent has better trajectory following ability and more stable positioning error performance than the ORB-SLAM3 baseline, which can provide a more stable pose estimation basis for underwater autonomous navigation and positioning in ship cleaning operations.

Claims

1. A multimodal autonomous operation navigation method for ship cleaning, characterized in that, The method includes the following steps: S1. Spatiotemporal data preprocessing and anisotropy restoration: Synchronously acquire binocular image observation data streams, distance observation data, and body dynamics time series data of the working environment, and perform time alignment; The binocular image observation data stream is preprocessed to obtain a restored image sequence, and the body dynamics time series data is continuously pre-integrated to obtain the relative motion state and gravity direction at adjacent time points; S2. Decoupling of static and dynamic features and non-rigid dynamic filtering: Extract candidate 2D matching point pairs from adjacent keyframes in the restored image sequence obtained in S1, and calculate the symmetrical epipolar geometric distance of each candidate 2D matching point pair; predict 3D points for each candidate 2D matching point pair; based on the relative motion state obtained in S1, transform the 3D points in the camera coordinate system of the previous keyframe to the camera coordinate system of the next keyframe, and calculate the 3D geometric consistency residual; perform mask removal on the corresponding candidate 2D matching point pairs based on the symmetrical epipolar geometric distance and the 3D geometric consistency residual to obtain retained static matching features; The preserved static matching features are triangulated to obtain static map points; S3. The point reprojection constraints formed by preserving static matching features and static map points, the geometric edge line reprojection constraints formed by geometric edge line features, and the Hessian canonical plane constraints formed by Hessian canonical plane features are entered into the back-end optimization to obtain the absolute pose of the system. S4. Fine-grained rust classification and navigation planning: Based on the restored image sequence obtained in S1, extract the region of interest of rust defects, and classify them by fusing color, texture and semantic features to output fine-grained classification labels; output the fine-grained classification labels to the external operation strategy module as the basis for selecting differentiated cleaning strategies in the future. Obtain the preset navigation target pose, and based on the system absolute pose obtained in S3 and the distance observation data collected in S1, update the global 2.5D probabilistic grid map cumulatively. Using the grid corresponding to the system absolute pose as the starting point and the grid corresponding to the preset navigation target pose as the ending point, use the A* algorithm to generate a global navigation path. The system extracts a local obstacle submap centered on the current system absolute pose from the global 2.5D probabilistic grid map, determines the current motion state based on the system absolute pose at adjacent time points, and generates local velocity commands using a dynamic window method. The global navigation path and the local velocity commands are then sent to the mobile carrier execution unit to execute the corresponding navigation operation.

2. The multimodal autonomous operation navigation method for ship cleaning according to claim 1, characterized in that, The binocular image observation data stream in step S1 includes the left and right eye images synchronously acquired by the binocular vision sensor at each acquisition time; the preprocessing of the binocular image observation data stream involves contrast restoration, edge restoration, and anisotropic diffusion processing to smooth underwater scattering noise and preserve directional high-frequency edge textures, resulting in the left and right eye restored images, which are then arranged into a restored image sequence according to the acquisition timestamp.

3. The multimodal autonomous operation navigation method for ship cleaning according to claim 1, characterized in that, Step S2 specifically includes the following sub-steps: S21. Extract candidate two-dimensional matching point pairs from adjacent keyframes in the restored image sequence obtained in S1, and obtain the fundamental matrix by estimating the fundamental matrix of the candidate two-dimensional matching point pairs using RANSAC. Based on the fundamental matrix Calculate the symmetric epipolar geometric distance for each candidate 2D matching point pair. The geometric distance of the symmetrical epipolar line As shown in formula (1): In the formula, and These are the same candidate 2D matching point pairs in adjacent keyframes. Time and Two-dimensional homogeneous coordinates at time; This is the fundamental matrix obtained based on robust estimation of all candidate two-dimensional matching point pairs; The epipolar geometric distance between the candidate two-dimensional matching point pairs is given by t; the superscript T indicates matrix transpose. S22. For each candidate two-dimensional matching point pair, respectively using Time and The left and right reconstructed images at the corresponding time points are subjected to binocular triangulation to obtain... 3D points in the camera coordinate system and 3D points in the camera coordinate system Based on the relative motion state obtained from S1, from Transformation of camera coordinate system to The camera coordinate system at any given time is used to obtain the predicted 3D points. and the predicted three-dimensional points and The Euclidean distance between them serves as the three-dimensional geometrically consistent residual. The three-dimensional geometric consistency residual r3ᴰ is shown in formula (2): In the formula, and For the same candidate two-dimensional matching point pair in Time camera coordinate system and Three-dimensional points obtained by binocular triangulation in the time-lapse camera coordinate system; and These are the rotation matrix and translation vector in the relative motion state obtained in S1, respectively; Indicates will from Transformation of camera coordinate system to Predicted 3D points obtained after the camera coordinate system is set at a specific time; S23, Geometric distance of symmetrical polar lines Distance threshold from preset polar lines Compare and use the three-dimensional geometric consistency residuals Compared with the preset three-dimensional residual threshold Compare: when or When the corresponding candidate two-dimensional matching point pairs are identified as dynamic pseudo-features, mask removal is performed. Will simultaneously satisfy and Candidate two-dimensional matching point pairs are determined as retaining static matching features; The preserved static matching features are triangulated to obtain static map points.

4. A multimodal autonomous operation navigation method for ship cleaning according to claim 3, characterized in that, The steps specifically include the following processes: S31. Establish point reprojection constraints using the retained static matching features and static map points obtained in S2; extract geometric edge line features from the restored image sequence obtained in S1, and use six-dimensional Plück coordinates to parameterize the geometric edge line features, as shown in formula (3): In the formula, The six-dimensional Plück coordinates represent the geometric edge line features; The moment vector of the geometric edge line features relative to the origin. Let be the direction vector of the geometric edge line feature, and and Satisfies orthogonal constraints; Represents a six-dimensional real vector space; S32. During the nonlinear optimization process of the back-end sliding window, when iteratively updating the parameters of the geometric edge line features, the Plück coordinates are projected onto a four-degree-of-freedom orthogonal representation space, and an increment is applied to the corresponding tangent space; a set of locally coplanar points is determined from the static map points obtained in S2, and a Hessian canonical plane feature is fitted, which is used to form the Hessian canonical plane residual. ; The back-end sliding window uses the pose, velocity, gyroscope bias, and accelerometer bias of each keyframe as the state variable to be estimated, and constructs the inertial pre-integration residual based on the relative motion state and gravity direction obtained from S1. The point reprojection residuals are formed based on the static matching features and static map points obtained from S2. Based on the geometric edge line features, a geometric edge line reprojection residual is constructed. Based on the aforementioned Hessian canonical plane features, Hessian canonical plane residuals are constructed. When the sliding window moves out of the earliest keyframe, the state variables and observation factors associated with that keyframe are marginalized to form prior constraints that retain the state variables, and marginalized prior residuals are constructed. The above five types of residuals are used to construct a full-dimensional joint cost function, which is shown in formula (4): In the formula, Let X be the full-dimensional joint cost function with the state variable to be estimated as the independent variable; To marginalize prior residuals, The inertial pre-integral residual is formed by the relative motion state and the direction of gravity obtained from S1. The point reprojection residuals obtained from S2, consisting of preserved static matching features and static map points, The reprojection residuals of geometric edge lines, which are composed of geometric edge line features. The Hessian canonical plane residual is formed by the plane features of the Hessian canonical form. , , , and These are the preset weights for the five types of residuals mentioned above; S33. The absolute pose of the system is obtained by performing iterative optimization using the Levenberg-Marquardt nonlinear least squares algorithm.

5. A multimodal autonomous operation navigation method for ship cleaning according to claim 1, characterized in that, The feature extraction and fusion mechanism for classifying corrosion defects in step S4 includes the following sub-steps: S41. Based on the restored image sequence obtained in S1, the YOLOv8 network with embedded spatial attention module and channel attention module is used to locate the region of interest of the corrosion defect. S42. Calculate the mean, standard deviation, and skewness of each color channel in the color space model for the region of interest of the corrosion defect, and construct the appearance feature vector; at the same time, construct the texture feature vector based on the gray-level co-occurrence matrix; perform global average pooling on the feature map output by the YOLOv8 network backbone to obtain a high-dimensional semantic feature vector, and concatenate the appearance feature vector, the texture feature vector, and the high-dimensional semantic feature vector to obtain a joint feature vector; S43. The fine-grained rust classification categories are pre-defined as three categories: light surface rust, moderate spalling, and severe pitting corrosion. The joint feature vector obtained in S42 is reduced in dimensionality through the principal component analysis module to obtain the principal component subspace features. The principal component molecular space features are input into a three-hidden-layer multilayer perceptron, and the probability distribution of each category is obtained through a normalized exponential function. Three fine-grained classification labels of light surface rust, moderate peeling, and severe pitting corrosion are output. The fine-grained classification labels are output to an external operation strategy module as the basis for selecting differentiated cleaning strategies in the future. S44. Map Construction and Path Planning: Based on the system absolute pose obtained in S3, the distance observation data retained in S1 is transformed to a unified coordinate system and the global 2.5D probabilistic grid map is updated cumulatively. Taking the grid corresponding to the current system absolute pose as the starting point and the grid corresponding to the preset navigation target pose as the ending point, the global navigation path is searched on the global 2.5D probabilistic grid map using the A* algorithm. A local obstacle sub-map centered on the current system absolute pose is extracted from the global 2.5D probabilistic grid map for use in evaluating local velocity candidates using the dynamic window method. Then, based on the system's absolute pose at adjacent moments, the current linear velocity and angular velocity are determined. A dynamic window is constructed by combining preset maximum linear velocity, maximum angular velocity, linear acceleration constraints, angular acceleration constraints, and safety distance constraints, within the velocity space defined by this dynamic window. Local velocity candidates are generated internally, and the comprehensive evaluation value of each local velocity candidate is calculated using formula (8). : in Candidate values ​​for linear velocity. Candidate values ​​for angular velocity; , and These are, respectively, a normalized global navigation path heading consistency evaluation term, a safe distance evaluation term based on the local obstacle subgraph, and a speed evaluation term; , and The preset weights for the corresponding evaluation items; Among the local velocity candidates that satisfy both the safety distance constraint and the motion constraint, select the one that makes... Candidates for obtaining the maximum value are used as local velocity commands; S44. The global navigation path and the local speed command are sent to the mobile carrier execution unit, thereby completing the navigation operation under the constraints of the global 2.5D probabilistic grid map and the local obstacle sub-map.

6. A multimodal autonomous operation system for ship cleaning, used to implement the navigation method according to any one of claims 1 to 5, characterized in that, include: A multimodal environment data acquisition array is configured to acquire binocular image observation data streams, distance observation data, and body dynamics time-series data of the working environment in real time and synchronously, and send the binocular image observation data streams, the distance observation data, and the body dynamics time-series data to the edge computing processing unit; The edge computing processing unit is electrically connected to the multimodal environment data acquisition array and internally deploys an anti-interference state estimation engine, a corrosion fine-grained perception module, and a navigation planning module. The navigation planning module is configured to receive a preset navigation target pose. The edge computing processing unit is configured to execute the navigation method according to any one of claims 1 to 5 based on the binocular image observation data stream, the distance observation data, the body dynamics time series data, and the preset navigation target pose, and output the system absolute pose, fine-grained classification labels, global navigation path, and local velocity command; wherein, the corrosion fine-grained perception module outputs the fine-grained classification labels to the external operation strategy module, and the fine-grained classification labels are not used as input for positioning, mapping, or navigation planning; The mobile carrier execution unit is communicatively connected to the edge computing processing unit and is configured to receive the global navigation path and the local speed command, and to perform navigation operations based on the local speed command.

7. A multimodal autonomous operation system for ship cleaning according to claim 6, characterized in that, The multimodal environmental data acquisition array integrates a binocular vision sensor, a high-frequency inertial measurement unit, and a distance sensor. The binocular vision sensor and the high-frequency inertial measurement unit are time-stamp aligned via a hardware synchronization trigger circuit. The binocular vision sensor synchronously acquires left and right eye images at each acquisition moment to form a binocular image observation data stream. The high-frequency inertial measurement unit acquires body dynamics time-series data, and the distance sensor acquires distance observation data. The binocular image observation data stream is used to obtain a restored image sequence, retain static matching features, and static map points. The body dynamics time-series data is used to obtain the relative motion state and gravity direction. The distance observation data and the system's absolute pose are used to cumulatively update the global 2.5D probabilistic grid map.

Images