A method for underwater depth estimation based on cross-modal alignment and fusion of visual and sonar images

Abstract

This invention discloses an underwater depth estimation method based on the fusion of visual and sonar dual-modal images, belonging to the fields of underwater robotics, deep learning, and 3D perception fusion technology. Step 1: Convert the original 2D sonar image into an STR image with the same resolution and spatial alignment as the RGB image; Step 2: Construct a dual-branch encoder consisting of RGB and STR branches; Step 3: Construct a multi-scale feature reconstruction decoder to recursively decode the multi-scale fused features output in Step 2; Step 4: Construct a global context modeling module based on a lightweight visual Transformer to model long-range dependencies of the decoded high-dimensional features, and output the final underwater absolute depth map using a depth regression head. This invention addresses the technical challenges of scale ambiguity in existing underwater depth estimation methods using monocular vision, and the noise introduced by modal misalignment due to differences in imaging mechanisms during sonar-visual fusion, which limits fusion efficiency.

Description

Technical Field

[0001] This invention belongs to the fields of underwater robots, deep learning and 3D perception fusion technology, and specifically relates to an underwater depth estimation method based on the fusion of visual and sonar dual-modal images. Background Technology

[0002] With the increasing development of marine resources and the growing number of underwater operations, the application of underwater robots (such as AUVs and ROVs) in areas such as marine 3D reconstruction, seabed target attitude estimation, underwater visual mapping, and navigation has become crucial. In these applications, acquiring high-precision, real-time underwater scene depth maps with absolute scale information is the core foundation for achieving autonomous obstacle avoidance and environmental perception by robots. Currently, the technical solutions for acquiring underwater depth information mainly fall into the following categories, but all have significant limitations: 1. Direct Measurement Sensor Solutions: Commonly used depth acquisition devices include time-of-flight sensors, underwater lidar, structured light, and binocular vision systems. However, the underwater environment is extremely complex, with light experiencing severe attenuation and scattering in water. Lithium and structured light sensors are not only expensive, but their effective measurement distance is limited by water transparency; binocular vision systems, on the other hand, heavily rely on matching environmental features, and in low-light, turbid, or monochromatic water environments, matching accuracy drops significantly, leading to depth estimation failure.

[0003] 2. Monocular Vision-Based Depth Estimation Schemes: Monocular cameras have attracted attention due to their small size, low power consumption, and flexible deployment. Traditional monocular depth estimation methods mainly rely on underwater imaging physical models (such as dark channel priors) to calculate transmittance and background light, thereby indirectly obtaining depth. However, these methods are highly dependent on wavelength-dependent optical parameters and have poor robustness in turbid waters. In recent years, although deep learning-based monocular depth estimation methods have improved in accuracy, the monocular imaging process is essentially a dimensionality reduction process of projecting from three dimensions to two dimensions, resulting in inherent "scale ambiguity." That is, based solely on a single RGB image, the algorithm cannot accurately determine the absolute physical size and true distance of objects, leading to a lack of reliable absolute scale references in the generated depth map.

[0004] 3. Existing Sonar and Vision Fusion Schemes: Multibeam imaging sonar utilizes sound wave reflection for imaging, offering advantages such as long propagation distance, immunity to water turbidity and lighting conditions, and the ability to provide accurate absolute ranging information. Theoretically, combining sonar with RGB images can compensate for the scale deficiency of monocular vision. However, the imaging principles of sonar and RGB images are fundamentally different (sonar is based on range-azimuth imaging, while RGB is based on lens refraction projection), resulting in a lack of natural spatial alignment between the two modalities. Existing sonar and vision fusion techniques often simply superimpose images at the image or feature layer, ignoring the positional misalignment of the two modalities in pixel space. Since pixels at the same location in an image do not represent the same semantic object in sonar and vision, this forced fusion due to "misalignment" introduces significant noise. In summary, there is currently a lack of a method that can effectively align the features of both acoustic and visual modes and achieve high-precision depth perception, addressing the technical issues of scale ambiguity in monocular vision in existing underwater depth estimation methods, as well as the noise introduced and the limitations on fusion efficiency caused by modal misalignment due to differences in imaging mechanisms during sonar and vision fusion. Summary of the Invention

[0005] This invention provides an underwater depth estimation method based on cross-modal alignment fusion of visual and sonar images. It addresses the technical challenges of scale ambiguity in monocular vision in existing underwater depth estimation methods, as well as the noise introduced by modal misalignment due to differences in imaging mechanisms during the fusion of sonar and vision, which limits the fusion efficiency.

[0006] This invention is achieved through the following technical solution: An underwater depth estimation method based on cross-modal alignment and fusion of visual and sonar images, the estimation method comprising the following steps: Step 1: Extract the original 2D sonar image Convert to an STR image with the same resolution and spatial alignment as the RGB image. ; Step 2: Construct a two-branch encoder consisting of an RGB branch and a STR branch; Step 3: Construct a multi-scale feature reconstruction decoder to recursively decode the multi-scale fused features output in Step 2; Step 4: Construct a global context modeling module based on a lightweight visual Transformer to model long-range dependencies of the decoded high-dimensional features and output the final underwater absolute depth map using a depth regression head.

[0007] Furthermore, step 1 specifically includes the following steps: Step 1.1: Construct an optical camera imaging model; Step 1.2: Construct a sonar imaging model; Step 1.3: Perform cross-modal mapping function The derivation aligns steps 1.1 and 1.2; Step 1.4: The multi-plane reconstruction strategy was used to achieve spatial geometric alignment between the converted STR image and the RGB image, providing a reliable physical distance scale for subsequent dual-branch feature fusion.

[0008] Furthermore, step 1.1 specifically involves defining the coordinates of a three-dimensional point in the world coordinate system as follows: The point projected onto the camera pixel coordinate system. The mapping relationship is represented as follows: (1) in: For three-dimensional points Depth value in camera coordinate system; The camera extrinsic parameter matrix is ​​derived from the rotation matrix. Translation vector composition; This is a dimension transformation matrix; This is the camera intrinsic parameter matrix, including focal length. and principal point coordinates ; Step 1.2 specifically involves, Two-dimensional sonar imaging is described as a weak perspective projection model; sonar image pixels 3D points in the world coordinate system The mapping relationship is represented as follows: (2) in: The transformation matrix representing the transformation from the sonar coordinate system to the world coordinate system; This is the sonar dimension transformation matrix; Here is the sonar intrinsic parameter matrix, where and Based on sonar resolution A defined scaling factor This is the offset. The tilt coefficient; Step 1.3 specifically involves, The result is obtained from formula (2) through pseudo-inverse operation. Relative representation of sonar pixels: (3) in: This represents the Moore-Penrose pseudo-inverse operation; Substituting equation (3) into equation (1), we obtain the mapping function from sonar pixels to camera pixels. : (4) Under this mapping, the corresponding camera coordinate system depth value The calculation is as follows: (5) Step 1.4 specifically involves, Step 1.4.1: Set the sonar vertical elevation angle range Evenly divided into A subplane; Step 1.4.2: For each subplane, apply the corresponding independent transformation matrix. and mapping function sonar images Projected onto the camera's viewpoint, generating a layered aligned image. ; Step 1.4.3: Pixel positions in the image The value at that location is assigned the calculated absolute depth. ; Step 1.4.4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation Layer-aligned images are concatenated and stitched along the channel dimension to generate a final STR image containing coarse absolute distance prior information. : (6) The converted STR image achieves spatial geometric alignment with the RGB image.

[0009] Furthermore, step 2 specifically includes the following steps: Step 2.1: Initial feature downsampling and spatial perception; Step 2.2: Employ a symmetric parallel feature extraction path; Step 2.3: Use an asynchronous multi-scale information fusion strategy to fuse the features extracted in Step 2.2.

[0010] Furthermore, step 2.1 specifically involves receiving the input optical RGB image. and the aligned STR image output from step one. The output goes to two branches of the perception module; Both branches of the perception module have an initial perception module at the beginning to perform initial convolutional encoding on the features. This module consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and a max pooling layer connected sequentially. After processing by the initial perception module, an initial visual feature map is generated. Compared with the initial scale feature map ; Step 2.2 specifically involves each branch of the encoder containing four cascaded feature extraction stages, with each stage employing a symmetrical downsampling strategy to ensure the spatial correspondence between modalities. The first four stages of ResNet34, pre-trained on ImageNet, are used as RGB feature encoding layers. Used to extract multi-scale semantic and texture features ; Four sequentially connected convolutional blocks are used as the STR feature encoding layer. Each convolutional block consists of two consecutive... Unit composition, used to extract multi-scale auxiliary depth features ; The input at each scale combines the fusion result from the previous scale, and its recursive formula is as follows: (7) Step 2.3 specifically involves, Gated lightweight fusion and cross-attention deep fusion are used to generate fused features across four scales in the feature channel dimension. ; The gated lightweight fusion specifically involves, in the shallow feature stage, using the features extracted from the STR branch as auxiliary information and concatenating them with the RGB features through channels; and adaptively modulating the participation level of the STR information by introducing learnable gating coefficients; the calculation logic is as follows: (8) The cross-attention deep fusion specifically involves performing deep modal interactions using a cross-attention mechanism at the highest-level semantic stage; dynamically capturing the complex nonlinear relationships between the two heterogeneous modalities by calculating the attention weight distribution between RGB and STR features; its calculation logic is as follows: (9) Finally, step 2 outputs a fused feature set containing multi-scale information. This provides high-dimensional feature representations for subsequent resolution reconstruction and depth regression. in, To represent different scale levels, the initial fused features are set as follows: .

[0011] Furthermore, step 3 specifically includes the following steps: Step 3.1: Decoder input and initialization; Step 3.2: Recursive feature upsampling and splicing fusion; Step 3.3: Final feature refinement and high-dimensional output.

[0012] Furthermore, step 3.1 specifically involves the multi-scale feature reconstruction decoder using the multi-scale fused features output by the dual-branch encoder. As input; The initial decoding process incorporates the highest-level fusion features from the encoding phase. As the initial decoding feature, let:

[0013] in, It is a low-resolution feature map with high-dimensional semantic information; Step 3.2 specifically involves the decoder comprising three consecutive decoding layers, reconstructing the feature map space size recursively from deep to shallow; for each decoding layer, the following sub-steps are performed: Using the transpose convolution operator Decoding features of the next higher level Perform an upsampling operation to double its spatial resolution; A skip connection mechanism is introduced to fuse the upsampled feature map with the corresponding scale of the feature map from the encoding stage. Perform splicing along the channel dimension; Apply feature concatenation and fusion and convolution thinning operators to the concatenated feature maps. Further encoding features yields the decoded features for the current level. The recursive reconstruction process is represented as follows: (10) Step 3.3 specifically involves, after completing the three-layer recursive decoding, processing the final decoded features. Perform two additional convolution thinning operations in sequence. To further refine the pixel-level feature representation, a high-resolution decoded feature map is finally obtained. : (11) in, This is the final output feature.

[0014] Furthermore, step 4 specifically includes the following steps: Step 4.1: Parallel processing of global and local features; Step 4.2: Feature fusion and enhanced representation; Step 4.3: Absolute depth map regression.

[0015] Furthermore, step 4.1 specifically involves the mViT module reconstructing the feature map output by the decoder using multi-scale features. As input, global context and local details are captured synchronously through parallel branches: First, the input feature map is embedded using a convolutional embedding method. The image is divided into several non-overlapping blocks, with the block size set to p=16; then it is mapped to a set of patchtokens and fed into TransformerEncoder for global self-attention modeling, thereby realizing statistical modeling of the long-range dependencies of the entire image. Through a The convolutional layers perform local spatial refinement operations on the original feature map to preserve pixel-level spatial structure information and obtain local features; Specifically, step 4.2 involves the mViT module fusing the global features output by the Transformer with the local features generated by the convolutional branches through a pixel-wise dot product operation. This generates an attention map that integrates global context and local details, ultimately resulting in a globally enhanced feature representation that preserves the original spatial resolution while incorporating long-range dependency information. : (12) Specifically, step 4.3 involves finally using a deep regression head composed of multiple convolutional layers to process the high-dimensional features. Mapping to a single-channel depth map outputs a pixel-level initial depth feature map; then, the following transformations are performed sequentially to obtain the final prediction result: It is mapped to the range [0,1] by the Sigmoid activation function; Map it to a preset effective depth range through a linear transformation. Inside; The process can be represented as follows: (13) in, This represents the convolution operation in the depth regression head. This is the final predicted underwater absolute depth map.

[0016] An underwater depth estimation system based on cross-modal alignment and fusion of visual and sonar images is disclosed. The estimation system employs the aforementioned underwater depth estimation method based on cross-modal alignment and fusion of visual and sonar images. The estimation system includes... Sonar mode alignment module: aligns the original 2D sonar image... Convert to an STR image with the same resolution and spatial alignment as the RGB image. ; Dual-branch encoder building block: The dual-branch encoder consists of an RGB branch and a STR branch; Multi-scale feature reconstruction decoder module: recursively decodes the output multi-scale fused features; A global context modeling module based on a lightweight visual Transformer: performs long-range dependency modeling on the decoded high-dimensional features and uses a depth regression head to output the final underwater absolute depth map.

[0017] The beneficial effects of this invention are: This invention constructs a supervised cross-modal alignment and fusion framework, utilizes the sonar modal alignment mechanism to eliminate spatial mapping deviations between two sensors, thereby fully leveraging the absolute ranging advantage of sonar to compensate for the scale deficiency of monocular vision, achieving highly reliable and high-precision depth acquisition of complex underwater scenes, and significantly improving the three-dimensional environmental perception capability of underwater robots.

[0018] This invention utilizes a dual-branch encoder structure to achieve multi-scale depth fusion of visual texture features and sonar depth prior features; then, a multi-scale decoder is used to reconstruct and restore spatial resolution; a lightweight visual Transformer module is used to capture global contextual information, and combined with multiple convolutional layers to achieve depth map regression, thereby achieving high-precision depth perception in complex and turbid waters. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the sonar mode alignment module of the present invention. Figure 3 This is a diagram of the mViT module architecture of the present invention.

[0021] Figure 4 This is a diagram showing the underwater depth estimation effect of the RSFNet proposed in this invention. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] The following is in conjunction with the appendix to this application specification. Figure 1-4 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0027] Implementation Method 1 This embodiment provides an underwater depth estimation method based on cross-modal alignment and fusion of visual and sonar images. A sonar modal alignment module maps misaligned sonar information to the imaging space of an optical camera, constructing a STR (Sonar To RGB) image with spatial consistency and absolute scale prior. A dual-branch encoder structure is used to achieve multi-scale depth fusion of visual texture features and sonar depth prior features. Subsequently, a multi-scale decoder is used to reconstruct and restore spatial resolution. A lightweight visual Transformer module is used to capture global contextual information, combined with multiple convolutional layers to perform depth map regression, thereby achieving high-precision depth perception in complex and turbid waters.

[0028] This invention addresses the technical challenges of scale ambiguity in monocular vision-based underwater depth estimation and the noise and fusion efficiency limitations introduced by modal misalignment due to differences in imaging mechanisms during sonar-visual fusion. It proposes a cross-modal depth estimation method based on a sonar modal alignment mechanism. The core of this method lies in the design of a sonar modal alignment module. By establishing a geometric transformation relationship between the optical camera pinhole imaging model and the sonar weak perspective projection model, and combining this with a multi-subplane projection reconstruction strategy, the absolute ranging information of the sonar is accurately projected onto the camera imaging space. This constructs an aligned image with spatial consistency and absolute scale prior, effectively eliminating spatial mapping deviations between heterogeneous modalities from a physical perspective. It fully leverages the ranging advantages of sonar in complex waters to compensate for the scale deficiency of visual sensors, breaking through the technical bottlenecks of existing fusion algorithms in modal alignment and scale compensation, and achieving high-precision and reliable depth perception in complex underwater scenes.

[0029] The estimation method includes the following steps: Step 1: Extract the original 2D sonar image Convert to a spatially aligned STR (SonarToRGB) image with the same resolution as the RGB image. Its core lies in establishing a cross-modal mapping function to solve the spatial mismatch problem caused by the different imaging mechanisms of sensors (sonar uses range-azimuth imaging, while cameras use perspective projection imaging). The principle diagram of sonar mode alignment is shown below. Figure 2 As shown, Step 2: Construct a dual-branch encoder consisting of an RGB branch and a STR branch; simultaneously mine the semantic texture information of the optical image and the absolute scale information of the sonar-aligned image, and achieve deep integration of heterogeneous modal features through a multi-scale interaction mechanism; Step 3: Construct a multi-scale feature reconstruction decoder to recursively decode the multi-scale fused features output in Step 2; gradually restore the spatial resolution of the feature map and effectively fuse shallow structural information with deep semantic representation; Step 4: Construct a global context modeling module based on the lightweight visual Transformer (mViT), with the model architecture as follows: Figure 3 As shown, long-range dependency modeling is performed on the decoded high-dimensional features, and the final underwater absolute depth map is output using a depth regression head.

[0030] This invention first simultaneously acquires optical and sonar images of the underwater scene under test. Then, a sonar modal alignment module reads pre-calibrated intrinsic and extrinsic parameters of the camera and sonar sensors, projects the sonar pixel coordinates onto the camera imaging plane using a cross-modal mapping function, and constructs an aligned depth prior image (STR) with spatial consistency and absolute scale prior information using a multi-plane reconstruction strategy. Next, the optical image and the aligned depth prior image are simultaneously input into a dual-branch encoder. Parallel feature extraction branches are used to mine visual semantic texture and sonar scale features respectively, and asynchronous feature interaction is performed at multiple scale levels through gating fusion and cross-attention mechanisms to generate a multi-scale fused feature set. Then, a multi-scale feature reconstruction decoder is used to gradually restore the image spatial resolution through transposed convolution upsampling and skip connection mechanisms to achieve the organic fusion of deep semantic information and shallow structural information, outputting a high-resolution decoded feature map. Finally, a lightweight global context modeling module captures long-range dependencies and enhances feature representation, and pixel-level depth mapping is performed in conjunction with a depth regression head to output a high-precision underwater depth map with true physical scale.

[0031] Furthermore, step 1 specifically includes the following steps: Step 1.1: Construct an optical camera imaging model; Step 1.2: Construct a sonar imaging model; Step 1.3: Perform cross-modal mapping function The derivation aligns steps 1.1 and 1.2; Step 1.4: The multi-plane reconstruction strategy was used to achieve spatial geometric alignment between the converted STR image and the RGB image, providing a reliable physical distance scale for subsequent dual-branch feature fusion.

[0032] Furthermore, step 1.1 specifically involves defining the coordinates of a three-dimensional point in the world coordinate system as follows: The point projected onto the camera pixel coordinate system. The mapping relationship is represented as follows: (1) in: For three-dimensional points Depth value in camera coordinate system; The camera extrinsic parameter matrix represents the transformation from the camera coordinate system to the world coordinate system, and is derived from the rotation matrix. Translation vector composition; This is a dimension transformation matrix used to map homogeneous coordinates to three-dimensional coordinates; This is the camera intrinsic parameter matrix, including focal length. and principal point coordinates ; Step 1.2 specifically involves, Two-dimensional sonar imaging is described as a weak perspective projection model; sonar image pixels 3D points in the world coordinate system The mapping relationship is represented as follows: (2) in: The transformation matrix representing the transformation from the sonar coordinate system to the world coordinate system; This is the sonar dimension transformation matrix; since two-dimensional sonar lacks elevation angle information, the dimension along the vertical axis is ignored in the initial derivation. Here is the sonar intrinsic parameter matrix, where and Based on sonar resolution A defined scaling factor This is the offset. The tilt coefficient; Specifically, step 1.3 involves eliminating intermediate variables in order to align the sonar information to the visual space. ; The result is obtained from formula (2) through pseudo-inverse operation. Relative representation of sonar pixels: (3) in: This represents the Moore-Penrose pseudo-inverse operation; Substituting equation (3) into equation (1), we obtain the mapping function from sonar pixels to camera pixels. : (4) Under this mapping, the corresponding camera coordinate system depth value The calculation is as follows: (5) Specifically, step 1.4 involves the fact that a single-pixel sonar point actually corresponds to a vertical sector in three-dimensional space (affected by the elevation angle). (Impact), when mapped to the camera plane, is not a unique point; this invention employs a multi-plane reconstruction strategy: Step 1.4.1: Set the sonar vertical elevation angle range Evenly divided into A subplane; Step 1.4.2: For each subplane, apply the corresponding independent transformation matrix. and mapping function sonar images Projected onto the camera's viewpoint, generating a layered aligned image. ; Step 1.4.3: Pixel positions in the image The value at that location is assigned the calculated absolute depth. ; Step 1.4.4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation Layer-aligned images are concatenated and stitched along the channel dimension to generate a final STR image containing coarse absolute distance prior information. : (6) The converted STR image achieves spatial geometric alignment with the RGB image, providing a reliable physical distance scale for subsequent dual-branch feature fusion.

[0033] Furthermore, step 2 specifically includes the following steps: Step 2.1: Initial feature downsampling and spatial perception; Step 2.2: Employ a symmetric parallel feature extraction path; Step 2.3: Use an asynchronous multi-scale information fusion strategy to fuse the features extracted in Step 2.2.

[0034] Furthermore, step 2.1 specifically involves receiving the input optical RGB image. and the aligned STR image output from step one. The output goes to two branches of the perception module; Both branches of the perception module have an initial perception module at the beginning to perform initial convolutional encoding on the features. This module consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and a max pooling layer connected sequentially. After processing by the initial perception module, an initial visual feature map is generated. Compared with the initial scale feature map The resolution of the initial feature map is uniformly downsampled to one-quarter of the original input size. Step 2.2 specifically involves each branch of the encoder containing four cascaded feature extraction stages, with each stage employing a symmetrical downsampling strategy to ensure the spatial correspondence between modalities. Visual feature branch: The first four stages of ResNet34 pre-trained on ImageNet are used as RGB feature encoding layers. Used to extract multi-scale semantic and texture features ; Sonar Scale Branch: Four sequentially connected convolutional blocks are used as the STR feature encoding layer. Each convolutional block consists of two consecutive... Unit composition, used to extract multi-scale auxiliary depth features ; Feature recursion logic: The input at each scale combines the fusion result of the previous scale, and its recursive formula is as follows: (7) Specifically, step 2.3 involves generating fused features using a dynamic fusion mechanism of "shallow lightweight fusion and deep strong interaction" across the four scales of the feature channel dimension. ; Gated lightweight fusion and cross-attention deep fusion are used to generate fused features across four scales in the feature channel dimension. ; The gated lightweight fusion (scale) Specifically, in the shallow feature stage, the features extracted from the STR branch are used as auxiliary information and concatenated with the RGB features through channels; by introducing learnable gating coefficients, the participation level of STR information is adaptively modulated, aiming to preserve more spatial details and suppress geometric errors in the modality alignment process; its calculation logic is as follows: (8) The cross-attention deep fusion (scale) Specifically, in the highest-level semantic stage, a cross-attention mechanism is used to perform deep modal interactions; by calculating the attention weight distribution between RGB and STR features, the complex nonlinear relationship between the two heterogeneous modalities is dynamically captured, thereby using the absolute distance information of sonar to achieve scale completion of visual features; its calculation logic is as follows: (9) Finally, step 2 outputs a fused feature set containing multi-scale information. This provides high-dimensional feature representations for subsequent resolution reconstruction and depth regression. in, To represent different scale levels, the initial fused features are set as follows: .

[0035] Furthermore, step 3 specifically includes the following steps: Step 3.1: Decoder input and initialization; Step 3.2: Recursive feature upsampling and splicing fusion; Step 3.3: Final feature refinement and high-dimensional output.

[0036] Furthermore, step 3.1 specifically involves the multi-scale feature reconstruction decoder using the multi-scale fused features output by the dual-branch encoder. As input; The initial decoding process incorporates the highest-level fusion features from the encoding phase. As the initial decoding feature, let:

[0037] in, It is a low-resolution feature map with high-dimensional semantic information; Step 3.2 specifically involves the decoder comprising three consecutive decoding layers, reconstructing the feature map space size recursively from deep to shallow; for each decoding layer, the following sub-steps are performed: Transposed convolution upsampling: using the transposed convolution operator Decoding features of the next higher level Perform an upsampling operation to double its spatial resolution; Skip-connect concatenation: A skip-connect mechanism is introduced to combine the upsampled feature map with the fused features at the corresponding scale from the encoding stage. The process involves splicing along the channel dimension; this operation aims to organically combine the shallow spatial details preserved during the encoding stage with the high-level semantic information during the decoding stage.

[0038] Convolutional Feature Refinement: Applying feature concatenation and fusion and convolutional refinement operators to the concatenated feature maps. Further encoding features yields the decoded features for the current level. The recursive reconstruction process is represented as follows: (10) Step 3.3 specifically involves, after completing the three-layer recursive decoding, processing the final decoded features. Perform two additional convolution thinning operations in sequence. To further refine the pixel-level feature representation, a high-resolution decoded feature map is finally obtained. : (11) in, The final output features maintain global consistency while ensuring the integrity of spatial details, providing high-quality feature input for subsequent global context modeling.

[0039] Furthermore, step 4 specifically includes the following steps: This step involves constructing a global context modeling module based on a lightweight visual Transformer (mViT), with the model architecture as follows... Figure 3 As shown, long-range dependency modeling is performed on the decoded high-dimensional features, and the final underwater absolute depth map is output using a depth regression head.

[0040] Step 4.1: Parallel processing of global and local features; Step 4.2: Feature fusion and enhanced representation; Step 4.3: Absolute depth map regression.

[0041] Furthermore, step 4.1 specifically involves the mViT module reconstructing the feature map output by the decoder using multi-scale features. As input, global context and local details are captured synchronously through parallel branches: Global branch: First, the input feature map is embedded using convolutional embedding. The image is divided into several non-overlapping patches, with the patch size set to p=16. Then, the patches are mapped to a set of patchtokens and fed into TransformerEncoder for global self-attention modeling, thereby realizing statistical modeling of the long-range dependencies of the entire image. Local branch: Meanwhile, through a The convolutional layers perform local spatial refinement operations on the original feature map to preserve pixel-level spatial structure information and obtain local features; Specifically, step 4.2 involves the mViT module fusing the global features output by the Transformer with the local features generated by the convolutional branches through a pixel-wise dot product operation. This generates an attention map that integrates global context and local details, ultimately resulting in a globally enhanced feature representation that preserves the original spatial resolution while incorporating long-range dependency information. : (12) Specifically, step 4.3 involves finally using a deep regression head composed of multiple convolutional layers to process the high-dimensional features. Mapping to a single-channel depth map outputs a pixel-level initial depth feature map; then, the following transformations are performed sequentially to obtain the final prediction result: It is mapped to the range [0,1] by the Sigmoid activation function; Map it to a preset effective depth range through a linear transformation. Inside; The process can be represented as follows: (13) in, This represents the convolution operation in the depth regression head. This is the final predicted underwater absolute depth map. Specific implementation examples: 1) An RGB-Sonar paired real underwater depth dataset was collected in an indoor simulated inflatable pool using a zed mini binocular camera and an M750d multibeam sonar. The dataset was named RSRUD, and the depth ground plane was generated using the high-precision stereo matching algorithm built into the zed API.

[0043] 2) Within the Gazebo simulation environment, the DAVE underwater simulator was extended and developed, and images with the same timestamp were acquired by combining a virtual Kinect camera and an M750d sonar. The RGB-Sonar paired simulation dataset was constructed and named RSSUD. The depth GT can be directly obtained by the Kinect camera.

[0044] 3) The method in this invention is trained using the real dataset RSSRD and the simulated dataset RSSUD, respectively. The parameter configuration in the sonar mode alignment module adopts the parameters obtained synchronously when the dataset is created. The training is performed for 100 epochs, and the model weights for the two datasets are obtained respectively.

[0045] 4) The two datasets were tested separately and input into RSFNet for depth map inference. The inference performance was significantly improved compared to other single-modal (pure RGB input) underwater depth estimation methods, achieving absolute leading levels in metrics such as SILog, AbsRel, and RMSE.

[0046] 5) The overall model was statistically analyzed for the number of parameters and computational cost. The number of parameters was 32.43 M, the computational cost was 137.65 GFLOPs, and the single-frame inference speed was 31.7 FPS. It has reached the leading level in the field in terms of lightweight design and can be easily ported to a lightweight platform in future work.

[0047] Implementation Method 2 This embodiment provides an underwater depth estimation system based on cross-modal alignment and fusion of visual and sonar images. The estimation system uses the underwater depth estimation method based on cross-modal alignment and fusion of visual and sonar images as described in Embodiment 1. The estimation system includes... Sonar mode alignment module: aligns the original 2D sonar image... Convert to a spatially aligned STR (SonarToRGB) image with the same resolution as the RGB image. Its core lies in establishing a cross-modal mapping function to solve the spatial mismatch problem caused by the different imaging mechanisms of sensors (sonar uses range-azimuth imaging, while cameras use perspective projection imaging). The principle diagram of sonar mode alignment is shown below. Figure 2 As shown, Dual-branch encoder building module: The dual-branch encoder consists of an RGB branch and an STR branch; it simultaneously mines the semantic texture information of optical images and the absolute scale information of sonar-aligned images, and achieves deep integration of heterogeneous modal features through a multi-scale interaction mechanism; Multi-scale feature reconstruction decoder module: recursively decodes the output multi-scale fused features; gradually restores the spatial resolution of the feature map and effectively fuses shallow structural information with deep semantic representation; A global context modeling module based on a lightweight visual Transformer: The model architecture is as follows Figure 3 As shown, long-range dependency modeling is performed on the decoded high-dimensional features, and the final underwater absolute depth map is output using a depth regression head.

[0048] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An underwater depth estimation method based on cross-modal alignment and fusion of visual and sonar images, characterized in that, The estimation method includes the following steps: Step 1: Extract the original 2D sonar image Convert to an STR image with the same resolution and spatial alignment as the RGB image. ; Step 2: Construct a two-branch encoder consisting of an RGB branch and a STR branch; Step 3: Construct a multi-scale feature reconstruction decoder to recursively decode the multi-scale fused features output in Step 2; Step 4: Construct a global context modeling module based on a lightweight visual Transformer to model long-range dependencies of the decoded high-dimensional features and output the final underwater absolute depth map using a depth regression head.

2. The underwater depth estimation method according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1.1: Construct an optical camera imaging model; Step 1.2: Construct a sonar imaging model; Step 1.3: Perform cross-modal mapping function The derivation aligns steps 1.1 and 1.2; Step 1.4: The multi-plane reconstruction strategy was used to achieve spatial geometric alignment between the converted STR image and the RGB image, providing a reliable physical distance scale for subsequent dual-branch feature fusion.

3. The underwater depth estimation method according to claim 2, characterized in that, Step 1.1 specifically involves defining the coordinates of a three-dimensional point in the world coordinate system as follows: The point projected onto the camera pixel coordinate system The mapping relationship is represented as follows: (1) in: For three-dimensional points Depth value in camera coordinate system; The camera extrinsic parameter matrix is ​​derived from the rotation matrix. Translation vector composition; This is a dimension transformation matrix; This is the camera intrinsic parameter matrix, including focal length. and principal point coordinates ; Step 1.2 specifically involves, Two-dimensional sonar imaging is described as a weak perspective projection model; sonar image pixels 3D points in the world coordinate system The mapping relationship is represented as follows: (2) in: The transformation matrix representing the transformation from the sonar coordinate system to the world coordinate system; This is the sonar dimension transformation matrix; Here is the sonar intrinsic parameter matrix, where and Based on sonar resolution A defined scaling factor This is the offset. The tilt coefficient; Step 1.3 specifically involves, The result is obtained from formula (2) through pseudo-inverse operation. Relative sonar pixel representation: (3) in: This represents the Moore-Penrose pseudo-inverse operation; Substituting equation (3) into equation (1), we obtain the mapping function from sonar pixels to camera pixels. : (4) Under this mapping, the corresponding camera coordinate system depth value The calculation is as follows: (5) Step 1.4 specifically involves, Step 1.4.1: Set the sonar vertical elevation angle range Evenly divided into A subplane; Step 1.4.2: For each subplane, apply the corresponding independent transformation matrix. and mapping function sonar images Projected onto the camera's viewpoint, generating a layered aligned image. ; Step 1.4.3: Pixel positions in the image The value at that location is assigned the calculated absolute depth. ; Step 1.4.4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation Layer-aligned images are concatenated and stitched along the channel dimension to generate a final STR image containing coarse absolute distance prior information. : (6) The converted STR image achieves spatial geometric alignment with the RGB image.

4. The underwater depth estimation method according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Initial feature downsampling and spatial perception; Step 2.2: Employ a symmetric parallel feature extraction path; Step 2.3: Use an asynchronous multi-scale information fusion strategy to fuse the features extracted in Step 2.

2.

5. The underwater depth estimation method according to claim 4, characterized in that, Step 2.1 specifically involves receiving the input optical RGB image. and the aligned STR image output from step one. The output goes to two branches of the perception module; Both branches of the perception module have an initial perception module at the beginning to perform initial convolutional encoding on the features. This module consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and a max pooling layer connected sequentially. After processing by the initial perception module, an initial visual feature map is generated. Compared with the initial scale feature map ; Step 2.2 specifically involves each branch of the encoder containing four cascaded feature extraction stages, with each stage employing a symmetrical downsampling strategy to ensure the spatial correspondence between modalities. The first four stages of ResNet34, pre-trained on ImageNet, are used as RGB feature encoding layers. Used to extract multi-scale semantic and texture features ; Four sequentially connected convolutional blocks are used as the STR feature encoding layer. Each convolutional block consists of two consecutive... Unit composition, used to extract multi-scale auxiliary depth features ; The input at each scale combines the fusion result from the previous scale, and its recursive formula is as follows: (7) Step 2.3 specifically involves, Gated lightweight fusion and cross-attention deep fusion are used to generate fused features across four scales in the feature channel dimension. ; The gated lightweight fusion specifically involves, in the shallow feature stage, using the features extracted from the STR branch as auxiliary information and concatenating them with the RGB features through channels; and adaptively modulating the participation level of the STR information by introducing learnable gating coefficients; the calculation logic is as follows: (8) The cross-attention deep fusion specifically involves performing deep modal interactions using a cross-attention mechanism at the highest-level semantic stage; dynamically capturing the complex nonlinear relationships between the two heterogeneous modalities by calculating the attention weight distribution between RGB and STR features; its calculation logic is as follows: (9) Finally, step 2 outputs a fused feature set containing multi-scale information. This provides high-dimensional feature representations for subsequent resolution reconstruction and depth regression. in, To represent different scale levels, the initial fused features are set as follows: .

6. The underwater depth estimation method according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Decoder input and initialization; Step 3.2: Recursive feature upsampling and splicing fusion; Step 3.3: Final feature refinement and high-dimensional output.

7. The underwater depth estimation method according to claim 6, characterized in that, Specifically, step 3.1 involves the multi-scale feature reconstruction decoder using the multi-scale fused features output by the dual-branch encoder. As input; The initial decoding process incorporates the highest-level fusion features from the encoding phase. As the initial decoding feature, let: in, It is a low-resolution feature map with high-dimensional semantic information; Step 3.2 specifically involves the decoder comprising three consecutive decoding layers, reconstructing the feature map space size recursively from deep to shallow; for each decoding layer, the following sub-steps are performed: Using the transpose convolution operator Decoding features of the next higher level Perform an upsampling operation to double its spatial resolution; A skip connection mechanism is introduced to fuse the upsampled feature map with the corresponding scale of the feature map from the encoding stage. Perform splicing along the channel dimension; Apply feature concatenation and fusion and convolution thinning operators to the concatenated feature maps. Further encoding features yields the decoded features for the current level. The recursive reconstruction process is represented as follows: (10) Step 3.3 specifically involves, after completing the three-layer recursive decoding, processing the final decoded features. Perform two additional convolution thinning operations in sequence. To further refine the pixel-level feature representation, a high-resolution decoded feature map is finally obtained. : (11) in, This is the final output feature.

8. The underwater depth estimation method according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1: Parallel processing of global and local features; Step 4.2: Feature fusion and enhanced representation; Step 4.3: Absolute depth map regression.

9. The underwater depth estimation method according to claim 8, characterized in that, Specifically, step 4.1 involves the mViT module reconstructing the feature map output by the decoder using multi-scale features. As input, global context and local details are captured synchronously through parallel branches: First, the input feature map is embedded using a convolutional embedding method. The image is divided into several non-overlapping blocks, with the block size set to p=16; then it is mapped to a set of patchtokens and fed into TransformerEncoder for global self-attention modeling, thereby realizing statistical modeling of the long-range dependencies of the entire image. Through a The convolutional layers perform local spatial refinement operations on the original feature map to preserve pixel-level spatial structure information and obtain local features; Specifically, step 4.2 involves the mViT module fusing the global features output by the Transformer with the local features generated by the convolutional branches through a pixel-wise dot product operation. This generates an attention map that integrates global context and local details, ultimately resulting in a globally enhanced feature representation that preserves the original spatial resolution while incorporating long-range dependency information. : (12) Specifically, step 4.3 involves finally using a deep regression head composed of multiple convolutional layers to process the high-dimensional features. Mapped to a single-channel depth map, outputting a pixel-level initial depth feature map; The following transformations are then performed sequentially to obtain the final prediction result: It is mapped to the range [0,1] by using the Sigmoid activation function; Map it to a preset effective depth range through a linear transformation. Inside; The process can be represented as follows: (13) in, This represents the convolution operation in the depth regression head. This is the final predicted underwater absolute depth map.

10. An underwater depth estimation system based on cross-modal alignment and fusion of visual and sonar images, characterized in that, The estimation system uses an underwater depth estimation method based on cross-modal alignment and fusion of visual and sonar images as described in any one of claims 1-9, the estimation system comprising, Sonar mode alignment module: aligns the original 2D sonar image... Convert to an STR image with the same resolution and spatial alignment as the RGB image. ; Dual-branch encoder building block: The dual-branch encoder consists of an RGB branch and a STR branch; Multi-scale feature reconstruction decoder module: recursively decodes the output multi-scale fused features; A global context modeling module based on a lightweight visual Transformer: performs long-range dependency modeling on the decoded high-dimensional features and uses a depth regression head to output the final underwater absolute depth map.

Images