A dual-encoder underwater image segmentation method based on SAM2 and DINOv2

Abstract

This invention relates to the field of computer vision and image processing technology. It discloses an underwater image segmentation method based on a dual encoder using SAM2 and DINOv2, comprising: acquiring underwater RGB images from publicly available marine organism segmentation benchmark datasets RMAS and MAS3K to obtain an original image set; preprocessing the original image set to obtain a standard image set; and adapting and adjusting the standard image set to form a first dataset. This invention employs a frozen pre-training parameter and lightweight adaptation strategy, reducing reliance on large amounts of labeled data and enabling the model to quickly adapt to different underwater scenarios while maintaining stability in complex environments such as lighting changes and noise interference. Furthermore, through lightweight designs such as feature adaptation and hierarchical fusion, it retains the powerful transfer capabilities of the dual encoder while strictly controlling the model's parameter scale and computational complexity, making it more flexible, efficient, and widely applicable in practical applications.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and more specifically, to an underwater image segmentation method based on dual encoders SAM2 and DINOv2. Background Technology

[0002] Underwater image segmentation refers to the processing and analysis of images in an underwater environment to identify and distinguish different objects or regions within the image. Due to complex underwater optical conditions, severe light attenuation, water disturbance, and background noise, underwater images often suffer from blurriness, distortion, and low contrast, making image segmentation even more challenging.

[0003] Traditional segmentation methods rely on manually designed features, making it difficult to adapt to feature variations in complex underwater environments, resulting in limited segmentation accuracy and generalization ability. Existing deep learning segmentation methods based on single encoders either focus on capturing local details but lack global semantic guidance, or focus on global semantic understanding but lose fine-grained structural information. This leads to problems such as incomplete segmentation and misidentification of targets in underwater scenarios with blurred target boundaries and severe background interference. Furthermore, underwater image data annotation is costly and highly diverse, making it difficult for a single model to quickly adapt to the domain differences of various underwater environments. Existing methods often suffer from high computational overhead and demanding hardware resource requirements, limiting their application in embedded scenarios such as underwater robots and real-time monitoring equipment, and failing to meet the comprehensive requirements of practical applications for segmentation accuracy, generalization ability, and deployment efficiency. Summary of the Invention

[0004] To overcome the above-mentioned deficiencies of the prior art, embodiments of the present invention provide an underwater image segmentation method based on dual encoders SAM2 and DINOv2.

[0005] To achieve the above objectives, the present invention provides the following technical solution: An underwater image segmentation method based on dual encoders SAM2 and DINOv2, the method comprising: S101: Collect underwater RGB images from the publicly available marine organism segmentation benchmark datasets RMAS and MAS3K to obtain the original image set. Preprocess the original image set to obtain the standard image set. Adapt and adjust the standard image set to form the first dataset. S102: Optimize the SAM2 encoder and DINOv2 encoder respectively, and extract the first feature map and the second feature map from the first dataset using the optimized SAM2 encoder and DINOv2 encoder respectively; S103: Align the first feature map with the second feature map to form an aligned feature map, fuse the aligned feature map according to the resolution features, and output the fused feature map; S104: Perform multiple upsampling operations on the fused feature map and synchronously connect across layers to generate a segmentation probability map. Build a segmentation model and train it. Use the trained segmentation model in combination with the segmentation probability map to perform operations and obtain the segmentation mask. Furthermore, the adaptation adjustment based on the standard image set includes: The standard image set was adjusted into tensors of various fixed sizes using bilinear interpolation, with each size being a multiple of sixteen (n being an integer).

[0006] Furthermore, the optimization of the SAM2 encoder and the DINOv2 encoder respectively includes: Optimize the SAM2 encoder: Based on the hierarchical encoder of SAM2, it consists of four stages of hierarchical blocks. Each hierarchical block integrates a multi-head self-attention mechanism and a feedforward network. Freeze all pre-trained parameters of the encoder and insert independent lightweight adapters at the input of each hierarchical block to introduce learnable parameters, thus obtaining the optimized SAM2 encoder.

[0007] Furthermore, the optimization of the SAM2 encoder and the DINOv2 encoder respectively also includes: The self-supervised DINOv2-Base model was selected as an auxiliary model, and all pre-trained parameters of the encoder were frozen.

[0008] Further, aligning the first feature map with the second feature map includes: The spatial dimensions of the second feature map are adjusted to match the resolution of the first feature map using bilinear interpolation, and the number of channels in the second feature map is mapped to the corresponding channel dimension of the first feature map to obtain the aligned feature map. The first feature map has four layers, the second feature map has one layer, and the alignment feature map has four layers.

[0009] Furthermore, the step of fusing and aligning feature maps based on resolution features includes: The shallow features in the second feature map are subjected to global average pooling, and modulated by introducing a first scaling factor and a first offset factor. The modulation formula is as follows: ; In the formula: , Indicates the scaling factor and offset factor; Simultaneously, a spatial gating branch is introduced, using the shallow features of the dual encoders and their absolute difference as input, which are then concatenated with convolution and sigmoid activation to generate a pixel-level gating map. Then, the output features are obtained through fusion, and the formula is: ; in: For learnable fusion coefficients, This represents depthwise separable convolution. This represents pixel-level multiplication operations.

[0010] Furthermore, the step of fusing and aligning feature maps based on resolution features also includes: Global average pooling and global max pooling are performed on the deep features in the second feature map, and the concatenation results in a feature vector. A second scaling factor and a second offset factor are then introduced for modulation.

[0011] Furthermore, the specific operation for obtaining the segmentation mask is as follows: Set a segmentation threshold. If the probability value is greater than or equal to the segmentation threshold, the region is determined to be the target region. If the probability value is less than the segmentation threshold, the region is determined to be the background region. Convert the segmentation probability map into a binary segmentation mask. The binary segmentation mask is subjected to morphological erosion and dilation operations, and then smoothed by Gaussian filtering to obtain the segmentation mask.

[0012] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the underwater image segmentation method based on SAM2 and DINOv2 as described above.

[0013] A computer-readable storage medium storing a computer program that, when executed, implements the underwater image segmentation method based on SAM2 and DINOv2 as described above.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This application discloses an underwater image segmentation method based on SAM2 and DINOv2 dual encoders, comprising: acquiring underwater RGB images from publicly available marine organism segmentation benchmark datasets RMAS and MAS3K to obtain an original image set; preprocessing the original image set to obtain a standard image set; and adapting and adjusting the standard image set to form a first dataset. This invention effectively fuses a first feature map and a second feature map through the collaborative work of the dual encoders, thereby overcoming common problems in underwater images such as boundary blurring and target camouflage, and improving segmentation accuracy. Secondly, by employing a frozen pre-training parameter and lightweight adaptation strategy, the dependence on a large amount of labeled data is reduced, enabling the model to quickly adapt to different underwater scenarios while maintaining stability in complex environments such as illumination changes and noise interference. Furthermore, through lightweight designs such as feature adaptation and hierarchical fusion, while retaining the powerful transfer capabilities of the dual encoders, the parameter scale and computational complexity of the model are strictly controlled, making it more flexible, efficient, and widely applicable in practical applications. Attached Figure Description

[0015] Figure 1 A flowchart of an underwater image segmentation method based on dual encoders SAM2 and DINOv2 provided by the present invention; Figure 2 A schematic diagram of the structure of an electronic device provided by the present invention; Figure 3 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention; Figure 4 This is a diagram of the complementary framework of the dual encoders in this invention; Figure 5 This is a structural diagram of the characteristic linear modulation mechanism in this invention; Figure 6 This is a structural diagram of the multi-branch hybrid attention module in this invention; Figure 7 A visual comparison of different methods for predicting segmentation masks in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: Please see Figure 1 As shown, this embodiment discloses an underwater image segmentation method based on dual encoders SAM2 and DINOv2, the method comprising: S101: Collect underwater RGB images from the publicly available marine organism segmentation benchmark datasets RMAS and MAS3K, compatible with JPG and PNG formats, to obtain the original image set. Preprocess the original image set to obtain the standard image set. Adapt and adjust the standard image set to form the first dataset.

[0018] Among them, RMAS is a real marine animal statistical data dataset containing 3014 images (2514 training images and 500 test images), and MAS3K is a marine animal statistical 3000 dataset containing 3103 images (1769 training images and 1141 test images), covering complex scenes such as mimicry camouflage, color matching, and transparent camouflage. The original image resolution ranges from [640x480, 4096x3072].

[0019] Preprocessing the original image set specifically involves performing pixel value standardization on the original image set to eliminate the influence of differences caused by different imaging devices; For the "ocean snow" noise formed by suspended particles in the image, a 3x3 adaptive median filter is used for initial suppression, filtering out isolated noise points while preserving the edge details of the target.

[0020] As one specific implementation method, the adaptation and adjustment based on the standard image set includes: To meet the input size requirements of dual encoders, the standard image set is adjusted into tensors of various fixed sizes using bilinear interpolation, with each size being a multiple of sixteen (n being an integer). This adapts to the Transformer encoder block processing mechanism, balancing computational overhead and feature extraction accuracy.

[0021] It should be noted that the tensor size of the Hiera encoder adapted to SAM2 is... The tensor size for the DINOv2 encoder is .

[0022] In addition, to improve the generalization ability of the model, this embodiment only uses two lightweight augmentation strategies (50% probability each) of random vertical flipping and random horizontal flipping during the training phase to expand the distribution range of training data, alleviate overfitting, and not destroy the inherent shape of the underwater target; no data augmentation is performed during the testing phase, only standardization and size adaptation operations are performed to ensure that the test results objectively reflect the actual performance of the model.

[0023] S102: Optimize the SAM2 encoder and DINOv2 encoder respectively, and extract the first feature map and the second feature map from the first dataset using the optimized SAM2 encoder and DINOv2 encoder respectively.

[0024] The optimization approach for the encoder is to adopt a core strategy of freezing pre-trained parameters and lightweight adaptation. This strategy not only fully preserves the original advantages of SAM2 and DINOv2 in their respective fields, but also allows for rapid adaptation to the special scenarios of underwater images, avoiding overfitting caused by retraining a large number of parameters. At the same time, it strictly controls the overall computational cost of the model to ensure that it can meet the training and inference requirements of a single 24GB GPU, thus balancing training efficiency and model performance.

[0025] For example, the optimization of the SAM2 encoder and the DINOv2 encoder respectively includes: Optimize the SAM2 encoder: Based on the SAM2 hierarchical encoder, it consists of four stages of hierarchical blocks. Each hierarchical block integrates a multi-head self-attention mechanism and a feedforward network, which has a strong ability to capture fine-grained structures and can accurately extract the edge, texture and other detailed features of underwater targets. In order to avoid retraining from destroying its original detail extraction ability, all pre-trained parameters of the encoder are frozen, and an independent lightweight adapter is inserted at the input of each hierarchical block to introduce learnable parameters, resulting in the optimized SAM2 encoder.

[0026] It should be added that the adapter adopts a compact structure of linear layer downsampling-Gaussian error linear unit activation-linear layer upsampling-Gaussian error linear unit activation, introducing only about 3 million learnable parameters, accounting for 5% of the total model parameters, to achieve efficient domain difference pairs without adding too much additional computational burden.

[0027] After processing by the optimized SAM2 encoder, the first dataset outputs four layers of first feature maps with different resolutions and levels of abstraction: S221 (resolution 88x88, number of channels 144, focusing on shallow edge details), S222 (resolution 44x44, number of channels 288, focusing on mid-level local contour features), S223 (resolution 22x22, number of channels 576, capturing deep overall structural features), and S224 (resolution 11x11, number of channels 1152, extracting high-level abstract structural features). These provide a rich and accurate foundation of local details for subsequent fusion with the second feature map.

[0028] The optimization of the SAM2 encoder and the DINOv2 encoder respectively also includes: The self-supervised DINOv2-Base model was selected as an auxiliary model, and all pre-trained parameters of the encoder were frozen. This model, through self-supervised learning on massive amounts of general images, possesses excellent global semantic understanding and generalization transfer capabilities, and can quickly adapt to underwater scenes without any fine-tuning. To fully utilize its pre-trained semantic advantages, all approximately 86 million pre-trained parameters of the encoder were frozen to avoid a decrease in semantic capture capability due to parameter adjustments. After inputting the adapted tensor, the encoder outputs a high-dimensional second feature map (37x37 resolution, 1024 channels). This feature map contains high-level semantic information such as marine life, underwater rocks, and water background, and has good scale consistency and semantic discriminative ability. It can effectively compensate for the shortcomings of the optimized SAM2 encoder in global semantic capture and provide key semantic guidance for accurate fusion with the first feature map.

[0029] like Figure 4 As shown, the dual encoder complementary framework consists of a SAM2 encoder and a DINOv2 encoder. Upsampling is performed by the size alignment module in the adapter, and the adapted and aligned features will continue to participate in subsequent cross-modal fusion. The U-shaped fusion decoder is a top-down structure consisting of multiple decoder modules connected in series. Each decoder module contains a difference-aware fusion module, a dual convolutional layer, and a multi-branch hybrid attention module.

[0030] S103: Align the first feature map with the second feature map to form an aligned feature map, fuse the aligned feature map according to the resolution features, and output the fused feature map.

[0031] The step of aligning the first feature map with the second feature map includes: The spatial dimensions of the second feature map are adjusted to match the resolution of the first feature map using bilinear interpolation, and the number of channels in the second feature map is mapped to the corresponding channel dimension of the first feature map to obtain the aligned feature map. As a specific implementation method, aligning the first feature map and the second feature map specifically involves: First, the spatial dimensions of the second feature map output by DINOv2 are downsampled to 88x88, 4444, 22x22, and 11x11 respectively using bilinear interpolation to match the resolution of the first feature map of each layer of SAM2. Then, using four independent depthwise separable wavelet convolutional modules, the number of channels of the second feature map is mapped to the channel dimension of the corresponding layer of SAM2. By leveraging the characteristics of wavelet transform, the receptive field is expanded with a small number of parameters, improving the feature representation capability. Finally, four aligned feature maps are output: S321 (144x88x88), S322 (288x44x44), S323 (576x22x22), and S324 (1152x11x11). The feature map fusion and alignment based on resolution features includes: For high-resolution features, corresponding to the first two layers of SAM2 features S221 and S222 and the aligned DINOv2 features S321 and S322, in order to preserve the edge and detail structure captured by SAM2 and inject semantic cues from DINOv2, the shallow features in the second feature map are globally averaged and pooled. Modulation is then achieved through a multilayer perceptron activated by two layers of 1x1 convolutional units with Gaussian error linear units, introducing a first scaling factor and a first offset factor. The modulation formula is as follows: ; In the formula: , Indicates the scaling factor and offset factor; The structure of the characteristic linear modulation mechanism is as follows Figure 5 As shown, this mechanism extracts channel statistics of shallow features through global average pooling, generates scaling coefficient γ and offset coefficient β through a multilayer perceptron activated by two layers of 1×1 convolution and Gaussian error linear units, and then modulates the original features at the channel level to achieve lightweight semantic injection.

[0032] High-resolution features: The core carries local detail information and spatial geometric information, including the edge contour, texture details, corner points, and precise pixel-level position of small targets. It is the key to achieving accurate alignment of segmentation boundaries and correct segmentation of small targets. It corresponds to a small receptive field, focuses on local areas of the input image, and can accurately capture subtle texture and edge changes. It has a strong ability to depict local details, but cannot perceive the global scene structure.

[0033] Low-resolution features: The core carries high-level semantic information and global contextual information, including the target's category attributes, the contextual relationship of the global scene, and the overall structure of large targets. It can solve the core classification problem of "what category is this area", effectively reduce category confusion, correspond to a large receptive field, and can cover a large area or even the global area of ​​the input image. It has strong overall perception of large targets and scene-level semantic understanding capabilities, and can establish cross-regional contextual associations.

[0034] The distinction between the two is based on: 1. the spatial downsampling ratio of the feature map relative to the input image (the width and height of the feature map), and 2. the type of information carried by the feature and the range of the receptive field.

[0035] Simultaneously, a spatial gating branch is introduced, using the shallow features of the dual encoders and their absolute difference as input, which are then concatenated with convolution and sigmoid activation to generate a pixel-level gating map. Furthermore, semantic injection is selectively amplified only in regions with local consistency and high confidence, and then the output features are obtained through fusion, achieving a precise combination of local details and local semantics. The formula is as follows: ; in The fusion coefficients are learnable and automatically optimized through backpropagation during model training. Constrained by the loss function and fusion logic, the initial values ​​are set in the (0,1) interval. After training convergence, the value of α typically stabilizes within the (0,1) interval as well. This represents depthwise separable convolution. This represents pixel-level multiplication operations.

[0036] The feature map fusion and alignment based on resolution features also includes: For low-resolution features, corresponding to the last two layers of SAM2 features S223 and S224 and the corresponding DINOv2 features S323 and S324, the focus is on global semantic consistency. Global average pooling and global max pooling operations are performed on the deep features in the second feature map, and the concatenated features are used to obtain the feature vector. The same method as for high-resolution fusion is used to introduce a second scaling factor and a second offset factor for modulation, achieving cross-branch semantic pairs in a lightweight manner. Since the deep features are highly abstract and spatial details are no longer dominant, pixel gating is not introduced. Fusion is completed only through channel-level global modulation, which not only reduces computational overhead and parameter scale, but also significantly alleviates cross-branch semantic drift and improves overall semantic coherence and distribution stability.

[0037] By employing a hierarchical fusion approach, prioritizing details in shallow layers and semantics in deeper layers, the fusion effect can be adapted to the feature characteristics. The fused feature map consists of four layers: S331 (144x88x88), S332 (288x44x44), S333 (576x22x22), and S334 (1152x11x11). The fused feature map simultaneously integrates the fine-grained local structural details of SAM2 and the global semantic information of DINOv2, with high spatial and semantic pairing accuracy, providing a high-quality feature foundation for target boundary reconstruction and detail recovery in the subsequent decoding stage.

[0038] S104: Perform multiple upsampling operations on the fused feature map and synchronously connect across layers to generate a segmentation probability map. Build and train a segmentation model. Use the trained segmentation model in conjunction with the segmentation probability map to obtain the segmentation mask.

[0039] To reconstruct the boundary details and semantic consistency of underwater targets, the decoder adopts a top-down U-shaped structure and works in conjunction with feature upsampling and cross-layer connections.

[0040] The process of performing multiple upsampling operations on the fused feature map and simultaneously connecting across layers involves three upsampling operations using transposed convolutions with a stride of 2, successively increasing the resolution of the deepest S334 feature map to 22x22, 44x44, and 88x88. After each upsampling, a cross-layer connection is made with the corresponding layer's fused feature map output by the encoder, supplementing the underlying detail information through concatenation. The entire decoding path consists of three decoding blocks, each containing a difference-aware fusion module, a dual convolutional layer, and a multi-branch hybrid attention module. Finally, a 1x1 convolutional layer maps the number of feature map channels to the number of segmentation categories, generating a segmentation probability map that provides the foundation for subsequent outputs. Ablation experiments validate that removing the difference-aware fusion module reduces the average intersection-over-union (IoU) ratio by 1.2 percentage points, and removing the multi-branch hybrid attention module reduces the average IoU ratio by 1.4 percentage points.

[0041] The decoding block has enhanced feature discrimination capabilities and is adapted to underwater scenes with low contrast and blurred boundaries. It includes the following modules: The core objective of difference-aware fusion is to guide the model to focus on discriminative regions by leveraging feature differences and suppressing redundant background interference. First, the difference maps of the two input features—upsampled features and cross-layer connection features—are calculated to depict their spatial distribution trends, highlighting key areas such as target boundaries and camouflaged regions. Then, the difference information is jointly input with the two original features into a lightweight convolutional structure to generate corresponding spatial weight maps, achieving adaptive adjustment of the feature response. Finally, the output features are obtained through convolutional fusion, using the following fusion formula: It achieves difference-guided feature integration with low computational overhead, enhancing the expression of effective information. For upsampling features, It is a cross-layer connectivity feature; The dual convolutional layer local feature optimization, as a feature local optimization unit, receives the feature map output by the difference-aware fusion module. Through a series structure of 3x3 convolution, batch normalization, Gaussian error linear unit activation, and 3x3 convolution, batch normalization, and Gaussian error linear unit activation, it expands the local receptive field, captures the local dependencies between features, stabilizes the feature distribution, and enhances the nonlinear expressive power of features, laying a high-quality foundation for subsequent attention module processing.

[0042] Please see Figure 6This multi-branch hybrid attention processing addresses the issues of feature information loss and insufficient saliency modeling caused by single-channel compression in high-channel outputs of dual convolutional layers. First, batch normalization is performed on the input features to stabilize their distribution. Then, the channel dimension is divided into three functional branches in a 2:1:1 ratio. The identity branch directly retains some features without any operations to ensure structural continuity. The channel attention branch employs an efficient channel attention mechanism, extracting channel statistics through global average pooling and establishing local dependencies between adjacent channels using one-dimensional convolutions, achieving efficient channel recalibration and key semantic enhancement. The spatial attention branch introduces an improved spatial attention mechanism based on convolutional block attention modules, combining channel pooling to generate a saliency spatial weight map and expanding the receptive field through two-way dilated depthwise convolutions to capture contextual information at different scales. The outputs of the three branches are concatenated and input into a depthwise separable convolutional multilayer perceptron to achieve channel dimensionality reduction and feature fusion. To ensure the consistency of residual connections, a channel matching projection mechanism is introduced, with the following formula: ; ; ; In the formula: This indicates the final output. This represents the main branch transformation function of the module. For the projection mapping of input features, Indicates the identity branching characteristic, Indicates channel attention. Indicates spatial attention; Simultaneously, a random depth discarding strategy is incorporated into the residual path to enhance the network's generalization ability. The final output undergoes a nonlinear transformation using a Gaussian error linear unit activation function, improving the separability and stability of the feature distribution.

[0043] After successive optimization and upsampling operations across three decoding blocks, the final output feature map size is 88x88, consistent with the resolution of the shallow feature map of the SAM2 encoder. The number of channels corresponds to the number of segmentation categories; for binary classification, it is 1D, and for multi-class classification, it is the number of corresponding categories. After processing with either the Sigmoid or Softmax activation function, the final segmentation probability map is generated. Each pixel value in the probability map represents the confidence level of belonging to the corresponding category at that location, providing a direct basis for subsequent segmentation results.

[0044] The segmentation model is built and trained, specifically as follows: To ensure the model accurately learns the feature patterns of underwater target segmentation, a weighted intersection-over-union (IoU) loss and a binary cross-entropy loss are used as the core training objectives. The combined loss formula is as follows: ,in, For weighted average loss, For binary cross-entropy loss, For all segmentation outputs, a weighted intersection-union loss assigns higher weights to target pixels, while a binary cross-entropy loss optimizes pixel-level classification accuracy. The combination of these two methods effectively alleviates the imbalance in the proportion of underwater target pixels. Simultaneously, to enhance the model's learning performance on features at different levels, all segmentation outputs from the decoder are processed... Applying a deep supervision strategy, the total loss formula is: Where G is the true label of the image, the multi-layer loss constraint guides the model to converge quickly and improves the segmentation accuracy.

[0045] The entire model training process was implemented using the Torch deep learning framework. The training hardware consisted of an Intel Core i9-1290 OK processor, 64GB of DDR5 memory, and a single NVIDIA RTX 3090 GPU with 24GB of VRAM, ensuring efficient and stable operation during training. The optimizer employed adaptive weight decay, with an initial learning rate of 0.001, a weight decay coefficient of 0.0001, and betas parameters set to (0.9, 0.999). A cosine decay strategy was used to dynamically adjust the learning rate. The training epochs were set to 30 rounds, with the first 3 rounds serving as a warm-up phase, gradually increasing the learning rate to the initial value. Subsequent rounds decayed the learning rate according to a cosine function to avoid oscillations in the later stages of training. During training, the batch size was set to 8, and a gradient clipping strategy was employed with a maximum norm of 1.0 to prevent gradient explosion and ensure stable model training.

[0046] The specific steps for obtaining the segmentation mask are as follows: If the task requires binary classification, a segmentation threshold is set. In this embodiment, the value is 0.5. If the probability value is greater than or equal to the segmentation threshold, the region is determined to be the target region. If the probability value is less than the segmentation threshold, the region is determined to be the background region. The segmentation probability map is then converted into a binary segmentation mask. The binary segmentation mask is subjected to morphological erosion and dilation operations of 3x3 structuring elements to eliminate small noise areas and fill tiny holes inside the target. Then, a 3x3 Gaussian filter is used for smoothing to improve the visual consistency and integrity of the segmentation results. The standard deviation is set to 0.5 to obtain the segmentation mask.

[0047] If the task requires multi-class classification, then the category with the highest probability for each pixel is selected as the segmentation result.

[0048] Furthermore, to further demonstrate the advantages of this method, this invention first conducted extensive experiments on two publicly available MAS datasets, comparing it with three mainstream methods, including typical marine organism segmentation methods (such as marine animal statistical networks, edge consistency detection networks, and deep fusion networks), camouflage target detection methods (such as scaling networks, next-generation scaling networks, and camouflage target detection network version 2), and related methods adapted to downstream tasks of the SAM model, such as underwater segmentation adaptation models, dual segmentation all-encompassing models, dual-branch segmentation models, dual-guided segmentation networks, segmentation base models, segmentation base model 2, and segmentation model 2 joint network.

[0049] First, we compared experiments using real marine animal statistics datasets with experiments using the Marine Animal Statistics 3000 dataset. The results are shown in Tables 1 and 2. Table 1: First Data Comparison Table

[0050] Table 2: Second Data Comparison Table

[0051] Experimental results demonstrate that this method performs exceptionally well on publicly available marine biological segmentation benchmark datasets. Specifically, on real marine animal statistical data datasets, the average intersection-union ratio (IU) reaches 0.765, the structural similarity measure reaches 0.889, and the mean absolute error is as low as 0.019. On the Marine Animal Statistical 3000 dataset, the average IU reaches 0.840, and the mean absolute error is as low as 0.015. All evaluation indicators are superior to existing mainstream segmentation methods, and this method can meet the precise data requirements of practical applications such as marine resource exploration and ecological monitoring.

[0052] Furthermore, experiments were conducted on a dataset of 10,000 underwater salient targets for comparison. The results are shown in Table 3. Table 3: Third Data Comparison Table

[0053] Experimental results validated that this method was compared with 17 state-of-the-art salient target detection methods, achieving best performance across all four core metrics. Compared to the current second-ranked transparency-guided underwater salient detection method, this method improves performance by 1.47, 0.55, and 1.59 percentage points in structural similarity metric, maximum E-metric, and maximum F-metric, respectively, and further reduces the mean absolute error by 0.0022. These results demonstrate that even in more complex and data-rich scenarios, this method maintains stronger robustness and generalization ability in target segmentation.

[0054] To visually demonstrate the advantages of the proposed method, in Figure 7This paper presents a visual comparison of several existing advanced methods. Eleven representative underwater images were selected from two datasets, with the following selection strategy: Five images were chosen from the real marine animal statistics dataset, focusing on overall structure (a), high-quality scenes (b), scenes with imaging difficulties due to underwater light attenuation (c), and complex scenes containing multiple animals (e); from the Marine Animal Statistics 3000 dataset, typical images corresponding to six marine animal camouflage methods were selected: mimicry camouflage (f), color matching (g), deformation camouflage (h), internal organ camouflage (i), motion camouflage (j), and transparency camouflage (k). Experimental results show that the method of this invention can achieve underwater target detection with higher accuracy, more complete target structure, and richer details, generating results with better visual effects, higher segmentation quality, clear boundaries, and complete structure.

[0055] Example 2: Please see Figure 2 As shown, this embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements any of the above-described methods for underwater image segmentation based on SAM2 and DINOv2 dual encoders. Since the electronic device described in this embodiment is the one used to implement the underwater image segmentation method based on SAM2 and DINOv2 using dual encoders in this application embodiment, those skilled in the art can understand the specific implementation and various variations of the electronic device in this embodiment based on the underwater image segmentation method based on SAM2 and DINOv2 described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art to implement the underwater image segmentation method based on SAM2 and DINOv2 using dual encoders in this application embodiment falls within the scope of protection of this application.

[0056] Example 3: Please see Figure 3 As shown, this embodiment discloses a computer-readable storage medium, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, it implements any of the above-described methods for underwater image segmentation based on dual encoders SAM2 and DINOv2.

[0057] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters, weights, and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0058] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0059] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0060] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0061] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0062] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0063] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0065] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-encoder underwater image segmentation method based on SAM2 and DINOv2, characterized in that, The method includes: S101: Collect underwater RGB images from the publicly available marine organism segmentation benchmark datasets RMAS and MAS3K to obtain the original image set. Preprocess the original image set to obtain the standard image set. Adapt and adjust the standard image set to form the first dataset. S102: Optimize the SAM2 encoder and DINOv2 encoder respectively, and extract the first feature map and the second feature map from the first dataset using the optimized SAM2 encoder and DINOv2 encoder respectively; S103: Align the first feature map with the second feature map to form an aligned feature map, fuse the aligned feature map according to the resolution features, and output the fused feature map; S104: Perform multiple upsampling operations on the fused feature map and synchronously connect across layers to generate a segmentation probability map. Build and train a segmentation model. Use the trained segmentation model in conjunction with the segmentation probability map to obtain the segmentation mask.

2. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders as described in claim 1, characterized in that, The adaptation and adjustment based on the standard image set includes: The standard image set was adjusted into tensors of various fixed sizes using bilinear interpolation, with each size being a multiple of sixteen (n being an integer).

3. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders as described in claim 1, characterized in that... The optimization of the SAM2 encoder and the DINOv2 encoder respectively includes: Optimize the SAM2 encoder: Based on the hierarchical encoder of SAM2, it consists of four stages of hierarchical blocks. Each hierarchical block integrates a multi-head self-attention mechanism and a feedforward network. Freeze all pre-trained parameters of the encoder and insert independent lightweight adapters at the input of each hierarchical block to introduce learnable parameters, thus obtaining the optimized SAM2 encoder.

4. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders according to claim 3, characterized in that, The optimization of the SAM2 encoder and the DINOv2 encoder respectively also includes: The self-supervised DINOv2-Base model was selected as an auxiliary model, and all pre-trained parameters of the encoder were frozen.

5. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders according to claim 4, characterized in that, The step of aligning the first feature map with the second feature map includes: The spatial dimensions of the second feature map are adjusted to match the resolution of the first feature map using bilinear interpolation, and the number of channels in the second feature map is mapped to the corresponding channel dimension of the first feature map to obtain the aligned feature map. The first feature map has four layers, the second feature map has one layer, and the alignment feature map has four layers.

6. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders according to claim 1, characterized in that, The feature map fusion and alignment based on resolution features includes: The shallow features in the second feature map are subjected to global average pooling, and modulated by introducing a first scaling factor and a first offset factor. The modulation formula is as follows: ； In the formula: , This represents the scaling factor and the offset factor. Indicates shallow features; Simultaneously, a spatial gating branch is introduced, using the shallow features of the dual encoders and their absolute difference as input, which are then concatenated with convolution and sigmoid activation to generate a pixel-level gating map. Then, the output features are obtained through fusion, and the formula is: ； in: For learnable fusion coefficients, This represents depthwise separable convolution. This represents pixel-level multiplication operations.

7. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders according to claim 6, characterized in that, The feature map fusion and alignment based on resolution features also includes: Global average pooling and global max pooling are performed on the deep features in the second feature map, and the concatenation results in a feature vector. A second scaling factor and a second offset factor are then introduced for modulation.

8. The underwater image segmentation method based on SAM2 and DINOv2 using dual encoders according to claim 1, characterized in that, The specific steps for obtaining the segmentation mask are as follows: Set a segmentation threshold. If the probability value is greater than or equal to the segmentation threshold, the region is determined to be the target region. If the probability value is less than the segmentation threshold, the region is determined to be the background region. Convert the segmentation probability map into a binary segmentation mask. The binary segmentation mask is subjected to morphological erosion and dilation operations, and then smoothed by Gaussian filtering to obtain the segmentation mask.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the underwater image segmentation method based on SAM2 and DINOv2 according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the underwater image segmentation method based on SAM2 and DINOv2 according to any one of claims 1 to 8.

Images