Ship scene image processing method based on visual stereo matching technology

Abstract

The invention discloses a ship scene image processing method based on a visual stereo matching technology, and the method comprises the steps: combining the geometric robustness of an improved A-SGM with the semantic priori features of a DepthAnything model, obtaining the geometric priori parallax distribution from a binocular image pair through the A-SGM, and taking the geometric priori parallax distribution as a potential initial geometric code; then, scale and offset correction is carried out by using a depth estimation result and an A-SGM parallax, and a fusion volume (SEG Volume) is constructed in a feature space. On the basis, an iterative updating framework based on GRU is introduced, and the initial parallax is gradually optimized, so that a parallax estimation result with higher precision and higher robustness is obtained. A structure coding geometry (SEG Volume) is constructed by fusing efficient energy aggregation of A-SGM and semantic feature extraction capability of a deep network, and a GRU module is utilized to perform multi-step dynamic geometry updating, so that high-robustness and high-real-time visual stereo matching is realized in complex environments where water mist, light reflection, weak texture and the like exist on the surface of a ship.

Description

Technical Field

[0001] This invention relates to the field of visual technology, and more specifically, to a method for processing ship scene images based on visual stereo matching technology. Background Technology

[0002] Visual stereo matching, a common problem in the field of computer vision, aims to recover dense depth information of a scene from binocular images. In recent years, visual stereo matching technology has not only been widely used in autonomous driving and robot navigation, but also has significant application value in ship surface cleaning operations.

[0003] In rust removal and cleaning operations on ship hulls, the goal of visual stereo matching is to perceive the depth information of obstacles on the hull surface in real time, enabling precise obstacle avoidance and path planning. In these operations, cleaning equipment typically needs to move along the hull surface, and the vision system must identify obstacle positions in real time to ensure accurate obstacle avoidance. The environment for ship surface cleaning operations typically has the following significant characteristics: 1. Excessive water mist and reflective interference: During rust removal or high-pressure cleaning, tiny water mist particles will be diffused in the air, and the metal surface will be wet and smooth, resulting in decreased image contrast, increased reflection, and blurred texture features; 2. Complex lighting conditions: The lighting on the hull surface is extremely uneven, with alternating changes in sunlight reflection and shadows, which makes the matching cost function susceptible to brightness drift. 3. High real-time requirements: In dynamic operations, each frame of image needs to be reconstructed in milliseconds, and traditional high-precision algorithms often cannot meet the requirements of real-time operation.

[0004] Against this backdrop, how to achieve highly robust and real-time visual stereo matching in complex environments (such as water mist, reflection, and weak texture) has become an urgent problem to be solved in the field of ship surface cleaning operations.

[0005] (1) Matching method based on local window Early stereo matching methods primarily relied on local pixel blocks (such as SAD, SSD, NCC, etc.) for similarity measurement, determining the correspondence by finding the position with the minimum cost within the disparity search range. While these methods are computationally efficient, they are prone to mismatches in areas with little texture, severe occlusion, or significant lighting variations, leading to inaccurate depth estimation.

[0006] (2) Matching method based on global optimization To overcome the limitations of local methods, global stereo matching methods introduce energy function modeling and optimization strategies, optimizing matching results globally through smoothing constraints. A typical example is the semi-global matching (SGM) algorithm. SGM achieves a trade-off between accuracy and computational efficiency by accumulating and optimizing energy across multiple paths, and has been widely used in applications such as autonomous driving, drones, and remote sensing. However, SGM still suffers from significant errors when handling large parallax, weak textures, and occluded regions, and it is heavily reliant on manually designed cost functions and energy models, making it difficult to adapt to complex scenes.

[0007] (3) Matching methods based on deep learning In recent years, deep learning techniques such as Convolutional Neural Networks (CNNs) and Transformers have been introduced into visual stereo matching tasks. These methods typically include steps such as feature extraction, cost volume construction, cost aggregation, and disparity regression. Among them, cost volume aggregation methods based on 3D convolutions (such as GC-Net and PSMNet) can learn contextual information and obtain high-precision matching results in an end-to-end framework. However, these methods generally suffer from the following problems: ① High computational and storage overhead: 3D convolution operations have high computational complexity in voxel space, making it difficult to meet the real-time requirements of ship cleaning operations; ② Strong dependence on training data: The model has limited generalization ability and its performance drops significantly when applied across scenarios. Due to the significant differences between the ship surface environment and common autonomous driving or indoor scenarios (such as water mist, reflection, curved surface structure, etc.), the model's generalization ability drops significantly when applied across scenarios, and it needs to be readjusted or fine-tuned. (4) Deep estimation method based on pre-trained model and large-scale data In recent years, with the development of large-scale visual models, general depth estimation frameworks such as DepthAnything have emerged. These models utilize large-scale pre-training and multimodal feature modeling to generate robust depth maps under unsupervised or weakly supervised conditions. Compared to traditional methods, they possess stronger cross-domain generalization ability and global contextual understanding. However, DepthAnything is more biased towards monocular depth estimation, its depth results lack geometric consistency constraints, and there is still room for improvement in accuracy for fine structures and edge regions.

[0008] (5) Laser-based depth measurement method In ship cleaning and rust removal operations, another common environmental sensing technology is depth measurement based on LiDAR or structured light. These methods calculate target distance by actively emitting lasers and measuring the time-of-flight or phase difference of the reflected signals, achieving high depth accuracy in simple, clean environments. However, ship cleaning operations often involve high-pressure water jets and atomized sprays, causing numerous water droplets to scatter or reflect the laser signal, resulting in attenuation or shift in the echo signal, leading to significant noise or even failure in depth measurement.

[0009] In summary, while existing SGM methods offer advantages in computational efficiency, they suffer from insufficient accuracy in complex and ever-changing shipboard operating environments. Deep learning methods, while improving accuracy, struggle with real-time deployment. Traditional lidar methods, though mature, face challenges in ranging failures and insufficient real-time performance in highly reflective and water-mist-laden environments such as ship surface cleaning.

[0010] The most similar solution to this invention is shown in the appendix. Figure 1 As shown, the source references are: J. Cheng et al., "MonSter: Marry Monodepth to Stereo UnleashesPower," 2025 IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR), Nashville, TN, USA, 2025, pp. 6273-6282, doi: 10.1109 / CVPR52734.2025.00588. keywords: {Geometry;Computer vision;Codes;Foundationmodels;Depth measurement;Pipelines;Noise;Robustness;Pattern recognition;Iterative methods}.

[0011] The specific implementation steps of this method are as follows: Feature Extraction: Multi-scale features are extracted from the left and right views using a deep convolutional neural network to obtain feature maps containing rich contextual information. Cost Volume Construction: Features from the left and right views are matched using group-wise correlation and feature interpolation methods to construct a high-dimensional cost volume, preserving similarity information in the disparity search space. Volume Aggregation and Update: An iterative aggregation module based on 3D convolutions and recurrent units (such as GRUs) is used to hierarchically update and pass information to the cost volume, thereby progressively refining the disparity estimation results. Disparity Prediction and Optimization: The updated cost volume is decoded and regressed using a regression and post-processing module, outputting high-precision disparity estimation results pixel-by-pixel.

[0012] In summary, existing methods suffer from the following drawbacks in complex environments such as water mist, glare, and weak texture during ship surface cleaning operations: 1. While existing SGM methods have advantages in computational efficiency, their accuracy is insufficient in complex and variable ship operating environments; 2. Deep learning methods based on 3D convolution have excessive computational overhead, making them unsuitable for real-time deployment; 3. Although monocular depth estimation models have strong generalization capabilities, their geometric consistency is insufficient, resulting in poor model generalization in the complex environment of ship surface cleaning operations; 4. Traditional lidar methods suffer from ranging failures and insufficient real-time performance in ship surface cleaning operations characterized by high glare and water mist. Summary of the Invention

[0013] One object of the present invention is to solve at least the above-mentioned problems and to provide at least the advantages that will be described later.

[0014] To address the above issues, this invention proposes a stereo matching method that combines the geometric robustness of an improved A-SGM with the semantic prior features of the DepthAnything model. This method constructs a Structure Encoded Geometry (SEG Volume) by fusing the efficient energy aggregation of A-SGM with the semantic feature extraction capabilities of deep networks, and utilizes a GRU module for multi-step dynamic geometric updates. This enables highly robust and real-time visual stereo matching even in complex environments such as water mist, reflections, and weak textures on ship surfaces.

[0015] To achieve these objectives and other advantages according to the present invention, a method for processing ship scene images based on visual stereo matching technology is provided, comprising the following steps: S1. Real-time left and right image data of the ship surface cleaning operation area are acquired by a binocular camera, and the image data is transmitted to the computing unit. The binocular images are preprocessed to remove water mist interference, suppress reflection overexposure, normalize illumination and correct color consistency to obtain a standardized image suitable for depth calculation. The enhanced semi-global matching algorithm A-SGM is used to calculate the initial disparity map and output the corresponding geometric confidence. The depth of monocular images is estimated using the deep learning model DepthAnything to obtain the global prior depth and uncertainty. The initial disparity of A-SGM is consistently fused with the prior depth of the depth model, and scale calibration and confidence weighting are performed to obtain a robust initial fused disparity. S2. Centered on fused parallax, construct a structure-encoded geometry (SEG) volume at the feature scale, including grouped correlation features, SGM multipath energy encoding, deep prior likelihood, and position encoding; unify the representation and spatial aggregation of multi-source geometric priors and image features; S3. The SEG Volume is aggregated using a 3D-Hourglass aggregation structure, and combined with two-dimensional contextual features, the matching reliability of reflective and water mist-obscured areas is enhanced. S4. The disparity is iteratively updated in multiple stages through the environment-adaptive convolutional gated recurrent unit EAG-ConvGRU, gradually outputting the disparity residual and obtaining the final high-precision disparity map. S5. Perform selective convolution smoothing, left-right consistency check and hole repair on the disparity map, and output a depth map and confidence map for obstacle avoidance and path planning of the ship cleaning robot.

[0016] Preferably, in step S1, the preprocessing of the binocular images includes: S1-1. A water mist suppression algorithm is used to estimate the transmittance of the image and correct fine particle fogging. S1-2. Use a reflective area detection module (such as brightness threshold + local contrast analysis) to suppress overexposure of metallic reflections; S1-3. Perform color consistency and brightness normalization on the image; S1-4. Use histogram equalization or adaptive contrast enhancement (CLAHE) to improve the visibility of hull texture.

[0017] Preferably, in step S1, the enhanced semi-global matching algorithm (A-SGM) is used to calculate the initial disparity map and output the corresponding geometric confidence score, including: S1-6, Perform cost aggregation in 16 path directions; S1-7. Introduce an attention guidance coefficient to enhance stability in areas with weak textures; S1-8. Generate an initial confidence map based on path consistency and cost sharpness.

[0018] Preferably, in step S1, the initial disparity of the A-SGM is consistently fused with the prior depth of the depth model, and scale calibration and confidence weighting are performed to obtain a robust initial fused disparity; including: S1-9. Convert the depth results of DepthAnything to parallax format; S1-10. Perform scale and offset calibration through high-confidence regions; S1-11. Use fusion weights to perform weighted fusion of SGM and depth prior to form a robust initial disparity map.

[0019] Preferably, step S2 includes: S2-1, Construction of the Groupwise Correlation Volume; The splicing of S2-2 and A-SGM multipath energy encoders; S2-3, DepthAnything Prior Depth Likelihood Encoding; S2-4. Fuse the parallax offset window and position encoding to form the SEG Volume.

[0020] Preferably, step S3 includes: S3-1, Downsampling three layers to extract cross-scale volume context; S3-2, Feature fusion is performed at the bottleneck layer; S3-3, upsample three layers and fuse them with aligned jump connection features to form the aggregated structured geometry V_AGG.

[0021] Preferably, step S4 includes: S4-1. Query the geometry slice based on the current disparity estimate; S4-2. Perform parallax distortion alignment on the features in the right image; S4-3. Input geometric features, left-right consistency features, and historical states into ConvGRU; S4-4, Output the disparity residual Δd and update the disparity; S4-5. Adjust the update step size based on the reflection / water mist confidence map.

[0022] Preferably, step S5 includes: S9-1. Generate dynamic convolution kernels based on context-guided filtering and smooth depth edges; S9-2, Perform left-right consistency checks and remove unreliable pixels; S9-3, Perform hole repair and sub-pixel fitting to improve depth accuracy; S9-4 Output the final depth map and confidence map for path planning of the ship cleaning robot.

[0023] The present invention has at least the following beneficial effects: 1. This invention combines the geometric optimization advantages of A-SGM with the global representation capabilities of DepthAnything to construct a structured coded geometry and achieve efficient stereo matching through an iterative update mechanism. This enables highly robust and real-time visual stereo matching in complex environments such as water mist, reflection, and weak texture on ship surfaces.

[0024] 2. Improved technology through deep integration of A-SGM and DepthAnything: By using robust scale registration and confidence weighting, the structurally consistent A-SGM and the globally generalized monocular prior relative depth are transformed into a "same scale, same reference frame" fusion, resulting in a more robust parallax. Improvements: Errors in areas with weak textures / lighting variations are significantly reduced, initial values ​​are closer to reality, and the search radius and iteration difficulty are reduced.

[0025] 3. Improved technology for SEG Volume of structure-encoded geometry: Based on the traditional relevance volume representation, A-SGM energy, monocular prior likelihood and geometric position encoding are introduced and unified into the volume representation; Improvements: The cost volume peaks are sharper and less ambiguous, enhancing the ability to distinguish between occlusion and duplicate textures, and reducing the burden of 3D aggregation.

[0026] Environment-Adaptive Convolutional Gated Recurrent Unit (EAG-ConvGRU) Improvement Techniques: In each iteration, the weights are adaptively adjusted and updated based on the environmental confidence level (such as the detection results of water mist and reflective areas), thereby effectively suppressing the propagation of errors in abnormal areas.

[0027] Improvements: While maintaining accuracy close to that of traditional 3D convolution aggregation, the computational cost and memory usage of the model are significantly reduced; suitable for real-time deployment environments.

[0028] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the method that most closely resembles existing technologies. Figure 2 This is a flowchart of the method of the present invention; Figure 3 Flowchart of the improved GRU algorithm of this invention; Figure 4 The flowchart of the improved A-SGM algorithm of this invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0031] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not imply the presence or addition of one or more other elements or combinations thereof.

[0032] This invention addresses the complex environment encountered during ship surface cleaning operations, such as: ① the presence of water mist, reflections, rust layers, and residual coatings on the surface; ② the weakening of textures and uneven lighting due to curved and angular structures; ③ high real-time requirements (depth maps need to be output in seconds to support obstacle avoidance by robots). Traditional visual stereo matching algorithms are prone to depth jumps, mismatches, and low-confidence regions in such scenarios. To address this, this invention proposes a structural coding geometry iterative optimization framework that integrates improved A-SGM geometry optimization with the global prior of the pre-trained deep network DepthAnything, achieving a dynamic balance between geometric consistency and depth robustness in complex environments. The specific implementation of this invention is described in detail below with reference to the accompanying drawings.

[0033] like Figure 2-4 As shown, this invention provides a method for processing ship scene images based on visual stereo matching technology, including the following steps: Step S1. In this invention, an improved SGM (Attention-SGM, or A-SGM for short) module with an attention mechanism is first used to perform preliminary matching of the binocular images. This module introduces adaptive attention weights and a reflective region suppression mechanism on the basis of the traditional semi-global matching (SGM) framework, which can maintain high confidence in geometric consistency even in complex environments such as water mist, reflection, and low texture on the ship surface.

[0034] (1) Dynamic path weight adjustment Traditional SGM (Short-Range Memory Gain) uses a fixed weighting method when accumulating energy in multiple directions (e.g., 8 or 16 paths), making it susceptible to interference from local abnormal pixels. This invention introduces path attention weights. The contribution of each path is dynamically adjusted based on local features (such as gradient direction, edge strength, and texture complexity). in, For pixels Total matching energy under disparity d (the smaller the value, the more reliable the match). For along direction Energy accumulation value, These are the adaptive path weights output by the lightweight attention network. When water mist, reflective areas, or shadows are detected, the model automatically reduces the weights in that direction to mitigate the impact of false matches, thereby improving overall geometric consistency.

[0035] (2) Adaptive cost function for illumination and reflection For highly reflective areas on the hull surface, traditional brightness-based matching costs (such as Census or NCC) are prone to failure. This invention introduces a spectral consistency attention term to modify the cost function: in, The cost of the final matching after fusion. This refers to the original matching cost (such as traditional costs like Census, SAD, NCC, etc.). It is estimated from local brightness non-uniformity and reflection intensity. This enhances the similarity features for channel attention. This mechanism ensures that the system matches based on structural features rather than brightness differences in damp, reflective, or coating-residual areas, improving robustness.

[0036] (3) Path consistency confidence mapping While obtaining the initial disparity map, this invention calculates the path consistency confidence score. : in, This represents the disparity value calculated along the path direction r. A smaller variance indicates higher consistency among estimates from multiple directions, which can be used as a confidence weight map for subsequent fusion. This confidence level shows a significant decrease in areas of strong light reflection or water mist, thus helping the downstream network automatically identify unreliable areas.

[0037] With the above improvements, the A-SGM module has the following advantages: ① It has stronger robustness to complex areas such as low-texture and damp surfaces; ② It has an adaptive suppression effect on reflection and water mist interference; ③ It can output disparity maps and confidence maps in real time, providing high-quality initial estimates for subsequent geometric fusion; ④ The computational load is only slightly higher than that of the standard SGM, which can still meet the real-time operation requirements of ship surface robots.

[0038] Subsequently, the present invention uses the geometrically consistent initial disparity output by the above-mentioned A-SGM. With confidence plot The global depth prior output by the DepthAnything network Perform geometric consistency fusion to obtain robust initial disparity estimates. .

[0039] (4) Geometric consistency fusion process First, based on the camera's focal length Compared with baseline The depth predicted by DepthAnything Convert to parallax form .

[0040] If there is scale drift in monocular depth, robust linear fitting is performed using the high-confidence region of SGM: in, Estimated by RANSAC.

[0041] Consider occlusion mask during fusion and confidence weight The fusion formula is: in, Finally, the initial parallax after fusion is obtained. Simultaneously output confidence plot .

[0042] Step S2. After completing geometric consistency fusion and obtaining initial disparity With confidence level Subsequently, this invention constructs a Structured Encoding Geometry Volume (SEG Volume) to unify and spatially aggregate multi-source geometric priors and image features. This structure not only retains the local correlation information in traditional stereo matching, but also explicitly embeds elements such as SGM energy distribution, DepthAnything global depth semantics, and scene confidence weights, thereby significantly improving the matching stability under complex lighting and environmental interference on ship surfaces.

[0043] (1) Multi-source feature alignment and normalization Because the disparity information from A-SGM and DepthAnything has different distribution scales, the two are first compared on... Unify the resolution to match the feature scale. Extract convolutional features from the left and right input images separately. and the initial parallax Scale normalization is performed to ensure that the displacement sampling and depth distribution are consistent during cost volume construction.

[0044] In actual ship cleaning scenarios, cameras have certain overhead angles and reflective interference. Therefore, this invention introduces local brightness correction and reflection masking in the normalization stage to perform smooth interpolation processing on overexposed or water mist-covered pixels, thereby reducing false feature interference.

[0045] (2) Construction of Groupwise Correlation Volume Divide the feature channels into Groups, each group calculates local correlation: This step is used to capture similarity features under disparity shift and generate the basic correlation volume. For the curved reflective areas of the hull, traditional correlation calculations are easily affected by texture loss. Therefore, this invention introduces channel attention weights on the correlation volume. The response of each channel is adaptively weighted to highlight geometrically stable areas (such as rivet edges and rust lines) and suppress specular reflection areas.

[0046] (3) SEG Volume splicing and output With d 0 Centered on a single image, a SEG cost matrix [B,G,D,H / 4,W / 4] is constructed that integrates group correlation, A-SGM energy, and deep prior confidence. Each voxel in the cost matrix not only contains similarity information of left and right image features but also embeds the following three types of structural encodings: A-SGM energy information: To supplement geometric consistency information, the energy accumulated by the A-SGM on multiple paths is... Normalization is performed to form an energy body. . DepthAnything confidence information: The depth prediction results of DepthAnything After being converted to parallax form, and Calculate the Gaussian likelihood at the same scale to obtain .

[0047] Geometric position encoding: records the relative spatial position of pixels in the image plane and the parallax dimension, used by the network to capture surface structure changes, for ( Perform sine / polynomial encoding By concatenating the above codes, we obtain: The Structure Encoded Geometry (SEG Volume) is essentially a multimodal fusion voxel tensor that integrates local correlations, global priors, and positional information. For ship surfaces, it can capture geometrical abrupt changes in details such as welds, rivets, and paint boundaries, as well as stable semantic features of water mist and reflective areas.

[0048] Step S3. After obtaining the Structure Encoded Geometry (SEG Volume), this step uses a 3D Hourglass aggregation structure to perform multi-scale fusion of volume information and combines it with two-dimensional contextual features to extract a more robust geometric representation.

[0049] First, the SEG Volume constructed in step S2 is input into a 3D convolutional hourglass aggregation network. This network adopts a symmetrical "downsampling-bottleneck-upsampling" structure, progressively extracting geometric context features from local to global in 3D space. Specifically, the hourglass module captures spatial correlation information at different scales through three layers of progressive downsampling, and in the bottleneck layer, it uses multi-branch residual blocks and attention mechanisms to fuse SGM energy information and depth prior features, thereby forming a geometric representation with global consistency constraints. Subsequently, skip connections are established between the upsampling path and the corresponding downsampling feature layer to achieve the restoration of fine structure and the fusion of high-level semantics, outputting the aggregated volume representation Vagg.

[0050] During the aggregation process, this invention proposes a context enhancement mechanism to address complex optical phenomena on ship surfaces, such as strong reflections, water mist scattering, and local occlusion. This mechanism utilizes two-dimensional semantic and edge contextual features (Ctx) from the left image, dynamically weighting the 3D aggregation features through lightweight attention modules (e.g., channel attention (SE) or spatial attention (CBAM)). For reflective areas, the network suppresses unreliable brightness variation features through cross-layer contextual dependencies; for water mist and blurred areas, the network maintains geometric continuity using global context, effectively avoiding false matches and holes.

[0051] Step S4. After volume aggregation is completed, this invention introduces an Environment Adaptive Convolutional Gated Recurrent Unit (EAG-ConvGRU) module to perform multi-stage iterative updates (T times) on the disparity estimation. Based on the traditional ConvGRU structure, this module introduces an environmental attention mechanism and a confidence-gated update to adapt to common complex interference factors in ship operation scenarios, such as water mist, strong reflections, paint gloss, and local fluid occlusion.

[0052] By dynamically suppressing unreliable regions and enhancing structural continuity, this stage achieves multi-step incremental optimization of depth geometry. The specific process is as follows: 1. Volume query and alignment For current disparity estimation Trilinear or bilinear interpolation is performed on the disparity dimension of the geometry to extract voxel features consistent with the current depth. At the same time, according to The feature map on the right is warped using a disparity field, so that the left and right features are aligned under the current disparity estimation, thereby improving matching consistency.

[0053] 2. Feature fusion and state update: voxel features The difference between the left and right features after remapping (mutual information or cosine similarity) and the hidden state from the previous step. Input the improved EAG-ConvGRU module.

[0054] Unlike traditional ConvGRUs, the structure of this invention introduces an environmental confidence channel into the gating mechanism. Its sources include: Water mist occlusion detection (based on brightness scattering model and local entropy). Reflection detection (using color saturation and high-frequency energy distribution); Matching confidence map (estimated jointly by cost sharpness and path consistency).

[0055] EAG-ConvGRU Update Gate With Reset Door From traditional input and Joint modulation to achieve dynamic regional weighting: in, In order to achieve the current parallax Below, voxel features extracted from Vagg using trilinear interpolation are analyzed. It uses the Sigmoid activation function, with an output range of 0 to 1. The hidden state after the (t-1)th iteration update The environmental confidence feature map is obtained by fusing reflection detection, water mist estimation, and matching uncertainty. During the update process, the model adaptively enhances the update intensity of high-confidence regions (e.g., weld edges, geometric polygons) and suppresses noise propagation from water mist obstruction and reflective areas. The output hidden state... Residual disparity is predicted after a lightweight convolutional decoder: 3. Iterative optimization and residual update: ConvGRU outputs a new hidden state. Then, the residual parallax is predicted by a lightweight decoder: in, The disparity estimate at the t-th iteration is updated through progressive refinement, which can achieve global consistency and edge smoothness in the multi-stage inference process.

[0056] 4. Stable updates and trimming: To prevent error accumulation in abnormal areas (such as areas with strong reflections or water mist), a confidence mask is used to control the update step size.

[0057] When the sharpness or confidence of a local cost is below a threshold, ConvGRII adaptively reduces the update weight of that region, thereby maintaining the stability of the overall structure.

[0058] The EAG-ConvGRU is used to iteratively update the output of multi-stage disparity residuals Δd, which are gradually refined to obtain the final disparity d*. The GRU's loop mechanism enables the model to: ① remember the geometric structure of the previous stage; ② adaptively adjust the weights of anomalous regions (such as areas with strong reflections or areas obscured by water mist).

[0059] By continuously improving geometric consistency during multi-step inference, the final high-precision parallax d* output can maintain depth continuity and edge sharpness on complex hull surfaces.

[0060] Step S5. To further improve the smoothness and local detail fidelity of the final depth map, this invention introduces context-guided filtering and dynamic convolution kernel generation mechanism in the output stage.

[0061] Adaptive convolutional kernel weights are predicted for each pixel location by a 2D context network (Ctx). This is to match changes in local geometric features. The dynamic kernel is generated based on local gradients, reflection features, and confidence levels, and can be represented as: in For context-sensitive convolutional prediction functions, For feature maps, To integrate confidence information.

[0062] For parallax results One or more guided filters based on dynamic kernels are performed to smooth the local depth surface. After filtering, quadratic curve fitting is performed locally on the probabilistic volume or cost surface to obtain sub-pixel disparity estimates. Left-right consistency (LR-check) verification is performed on the output disparity to eliminate unreliable spurious matching regions. Holes caused by occlusion or water mist are repaired using guided interpolation or bilateral filtering to restore continuous depth. The confidence map, calculated by combining volume entropy, gradient consistency, and update stability, is used for safety planning and risk assessment in subsequent robotic tasks.

[0063] In the output stage, the final depth map and confidence map output by this invention can be directly used for robot 3D localization, surface modeling and cleaning path planning.

[0064] Figure 2 This is a flowchart of the method of the present invention. The present invention utilizes an initial disparity map generated by the A-SGM method and monocular depth estimation results provided by DepthAnything. First, the geometric prior disparity distribution is obtained on the binocular image pair using A-SGM as a potential initial geometric code. Then, scale and offset corrections are performed using the depth estimation results and A-SGM disparity, and a fusion volume (Structure Encoding Geometry, SEG Volume) is constructed in the feature space. Based on this, an iterative update framework based on GRU is introduced to progressively optimize the initial disparity, thereby obtaining more accurate and robust disparity estimation results. Further improvements in this process include utilizing multi-scale feature enhancement, consistency constraints of geometric priors from different sources, and a disparity-feature fusion mechanism, outlining potential optimization paths and error correction procedures.

[0065] The present invention has the following technical innovations: 1. A-SGM and DepthAnything geometric consistency fusion mechanism: Using the high confidence region Ω of A-SGM as the anchor point, robust scale-shift registration of DepthAnything disparity is performed using RANSAC / LAD; The design path consistency-network uncertainty joint confidence α(p) is dynamically weighted and fused to generate the initial disparity d. 0 .

[0066] Unlike existing technologies, traditional methods either use only SGM (which is prone to failure in weak textures) or only monocular depth (geometric drift); this invention is the first to fuse the two under the same geometric reference frame and the same confidence framework.

[0067] 2. SEG Volume of Structure Coding Geometry: Simultaneously embed ① group correlation Vcorr, ② A-SGM path energy Vsgm, ③ monocular prior likelihood Vda, and ④ geometric / positional encoding Vgeo within a single 4D volume (B,G,D,H / 4,W / 4). Four heterogeneous pieces of information are cascaded after being normalized by the channel to form a compact expression with "one code, multiple meanings".

[0068] Unlike existing technologies, which only use correlation or deep learning features and do not encode traditional energy and data-driven priors in a unified manner, this invention is the first to combine manual energy and deep learning likelihood into a single entity.

[0069] 3. Volume Query - ConvGRU Iterative Update: Using the current disparity dt as an index, 1×1×1 features Qvol are extracted from the aggregate V_AGG on demand using trilinear sampling to avoid repeated 3D convolution; Let Qvol, context Ctx, gradient ∇IL, dt, and c be the values ​​of Qvol, context Ctx, gradient ∇IL, dt, and c. 0 The quintuple is simultaneously fed into ConvGRU to predict the residual Δdt; The entire process involves 2D convolution for iteration, while 3D convolution is performed only once in the preceding Hourglass stage.

[0070] Unlike existing technologies, traditional 3D-CNN requires T 3D convolutions on the entire cost volume; this invention reduces the 3D operation to 1 + T 2D samplings, reducing the computational complexity by an order of magnitude.

[0071] As described above, this invention proposes a visual stereo matching method for obstacles in ship surface cleaning operations. This method combines the geometric consistency constraints of traditional stereo matching with the strong representation ability of the DepthAnything depth estimation network, and achieves iterative geometric feature enhancement and disparity optimization through a cyclic update unit (GRU).

[0072] (1) Initial data collection and integration: (1-1) First, based on the binocular image input (Left-Right Pair), an attention-guided semi-global matching algorithm (A-SGM) is used to generate initial dense disparity estimation results. A-SGM introduces an attention weight modulation mechanism on the basis of the traditional SGM multi-path energy aggregation framework (usually including 8 or 16 paths in the horizontal, vertical and diagonal directions), and adaptively adjusts the path cost according to the local texture intensity, reflective features and water mist confidence of the image.

[0073] This mechanism enables ASGM to automatically suppress the effects of unreliable paths and enhance geometric consistency in complex ship surface environments (such as paint reflection, water mist obstruction, and weld areas), thereby obtaining an initial parallax map that is structurally continuous, has smooth edges, and is robust to environmental changes.

[0074] (1-2) The DepthAnything network is used to perform monocular depth estimation on the left eye image to obtain semantically rich depth prediction results. Since monocular depth lacks absolute scale information, this invention uses a linear fitting method (compute_scale_shift function) to align the scale and offset of monocular depth and A-SGM disparity, thereby ensuring the consistency of the two in terms of physical scale.

[0075] (2) Construction of the Structure Encoding Geometry (SEG Volume): (2-1) The scale-corrected SGM disparity results and the feature maps output by DepthAnything are input into the feature fusion module. This module aggregates multi-scale features through three-dimensional convolutional units (Conv3D) to construct a structure-encoded geometry (SEG Volume) that includes structural consistency and semantic prior.

[0076] (2-2) The construction of the SEG Volume follows the following mathematical form: in, This represents the geometric cost volume obtained through A-SGM. Represents depth feature volume, These are the feature maps for the left and right images, respectively. This is a 3D convolution fusion operator. This structure achieves explicit collaboration between depth estimation and stereo matching geometry.

[0077] (3) Dynamic updates and disparity optimization based on GRU: (3-1) After the SEG Volume is constructed, it is input into the multi-level gated cyclic unit (GRU) update module to achieve time-step-based geometric consistency optimization. Each time... The update formula is as follows: in Encode the voxel features of the structure geometry in the current iteration. This represents the predicted disparity increment.

[0078] (3-2) During training, the GRU module simultaneously receives the depth feature gradient from DepthAnything and the structural consistency signal from A-SGM, achieving joint backpropagation update of semantic priors and geometric constraints. The final output disparity result has higher boundary preservation and region consistency.

[0079] (3-3) In the iterative optimization process, the invention also introduced a REMP module (ResidualEdge-awareMulti-scalePrediction) based on residual correction, which is used to repair local occlusion and texture missing areas, and further improve the continuity and robustness of disparity estimation.

[0080] (4) Analysis of the fusion optimization path: (4-1) This invention further defines three fusion paths: Geometry-Guided Path: Guides the geometric alignment of deep features using the A-SGM cost volume as the primary constraint; Depth-GuidedPath: Primarily uses the features output by DepthAnything to guide the semantic smoothing of disparity distribution; HybridCoupledPath: This method fuses features from both approaches and dynamically adjusts the fusion weights within the GRU using a gating mechanism.

[0081] (4-2) Experiments have shown that different paths exhibit differentiated characteristics in specific scenarios: In low-texture, low-light, or high-noise scenes, depth-dominated paths provide robust estimation stability. In scenarios with high geometric complexity or occlusion, geometry-dominated paths ensure stereo consistency. Figure 3 The process of GRU optimizing disparity in multiple iterations is described: 1. Initial inputs are SEG Volume and initial disparity estimate. ; 2. Perform the following operations in each iteration: ① Extract current geometric features ; ② Calculate the parallax residual ; ③ Update the hidden state through the GRU gating mechanism ; ④ Update parallax estimation ; 3. After the iteration is complete, output the final high-precision parallax result. .

[0082] This structure enables the network to simulate multipath aggregation and global consistency adjustment in traditional energy optimization processes, thereby achieving geometrically adaptive optimization in an end-to-end framework.

[0083] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A ship scene image processing method based on a visual stereo matching technique, characterized in that, The method comprises the following steps: S1, acquiring real-time left and right image data of a ship surface cleaning operation area by a binocular camera, and transmitting the image data to a computing unit, pre-processing the binocular image, calculating an initial disparity map by using an enhanced semi-global matching algorithm A-SGM, and outputting corresponding geometric confidence; using a deep learning model DepthAnything to perform depth estimation on the monocular image to obtain global prior depth and uncertainty; performing consistency fusion on the initial disparity of A-SGM and the prior depth of the depth model, performing scale calibration and confidence weighting to obtain a robust initial fused disparity; S2, constructing a structure encoding geometry SEG Volume on a feature scale with the fused disparity as the center, and uniformly representing and spatially aggregating multi-source geometric priors and image features; S3, using a 3D-Hourglass aggregation structure to perform volume aggregation on the SEG Volume, and combining two-dimensional context features to enhance the matching reliability of the reflective light and water mist occlusion area; S4, performing multi-stage iterative updating of the disparity by an environment adaptive convolution gate recurrent unit EAG-ConvGRU, gradually outputting disparity residuals and obtaining a final high-precision disparity map; S5, performing selective convolution smoothing, left-right consistency checking and hole repair on the disparity map, and outputting a depth map and a confidence map for ship cleaning robot obstacle avoidance and path planning.

2. The ship scene image processing method based on the visual stereo matching technology according to claim 1, characterized in that, In S1, the binocular image is pre-processed, including: S1-1, using a water mist suppression algorithm to estimate the transmittance of the image and correct the fine particle atomization; S1-2, using a reflective light area detection module to suppress metal reflective overexposure; S1-3, performing color consistency and brightness normalization on the image; S1-4, using histogram equalization or adaptive contrast enhancement method to improve the visibility of ship texture.

3. The ship scene image processing method based on the visual stereo matching technology according to claim 1, characterized in that, In S1, the enhanced semi-global matching algorithm A-SGM is used to calculate the initial disparity map and output the corresponding geometric confidence, including: S1-6, cost aggregation is performed in 16 path directions; S1-7, an attention guide coefficient is introduced to strengthen the stability of the weak texture area; S1-8, an initial confidence map is generated based on path consistency and cost sharpness.

4. The ship scene image processing method based on the vision stereo matching technology according to claim 3, characterized in that, In S1, the initial disparity of A-SGM and the prior depth of the depth model are fused for consistency, scale calibration and confidence weighting to obtain a robust initial fused disparity; including: S1-9, converting the depth result of DepthAnything into disparity format; S1-10, performing scale and offset calibration through high-confidence areas; S1-11, using fusion weights to weight and fuse SGM and depth priors to form a robust initial disparity map.

5. The ship scene image processing method based on the visual stereo matching technology according to claim 4, characterized in that, The step S2 comprises: S2-1, construction of Groupwise Correlation Volume; S2-2, splicing of A-SGM multi-path energy encoding volume; S2-3, DepthAnything prior depth likelihood encoding; S2-4, forming SEG Volume by fusing disparity offset window and position encoding.

6. The ship scene image processing method based on the vision stereo matching technology according to claim 5, characterized in that, The step S3 comprises: S3-1, downsampling three layers to extract cross-scale volume context; S3-2, feature fusion in bottleneck layer; S3-3, upsampling three layers and fusing with aligned skip connection features to form an aggregated structured geometry V_AGG.

7. The ship scene image processing method based on the vision stereo matching technology according to claim 6, characterized in that, The step S4 comprises: S4-1, querying geometry slices according to current disparity estimation; S4-2, aligning right image features through disparity warping; S4-3, inputting geometry features, left-right consistency features and historical state into ConvGRU; S4-4, outputting disparity residual Δd and updating disparity; S4-5, adjusting update step according to reflection / water mist confidence map.

8. The ship scene image processing method based on the vision stereo matching technology according to claim 7, characterized in that, The step S5 comprises: S9-1, generating dynamic convolution kernel based on context guided filtering and smoothing depth edge; S9-2, performing left-right consistency check and eliminating unreliable pixels; S9-3, performing hole filling and sub-pixel fitting to improve depth accuracy; S9-4, outputting final depth map and confidence map for ship cleaning robot path planning.

Images