Deep-sea image instance segmentation system based on lightweight hybrid visual structure

Abstract

The application discloses a deep-sea image instance segmentation system based on a light-weight mixed visual structure and belongs to the field of instance segmentation. The application aims to solve the problem that a model based on natural image training has poor adaptability to the degradation characteristics of deep-sea images, low feature extraction quality and influences the segmentation result. The application comprises a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module and an integrated weak supervision guiding module. The feature extraction module receives inputted original data frames, including image data frames and video data frames. Features are extracted based on the image data frames and the video data frames. The feature extraction strategy of the feature map is adjusted in real time by receiving the pseudo mask from the integrated weak supervision guiding module. The image feature fusion module receives the multi-scale feature map transmitted by the feature extraction module and obtains a fused feature map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of instance segmentation, and in particular to a deep-sea image instance segmentation system based on a lightweight hybrid visual structure. BACKGROUND

[0002] With the continuous growth of the demand for ocean resource development and environmental monitoring, intelligent analysis of deep-sea images has become one of the key capabilities in underwater robot systems. Deep-sea image segmentation, as an important task of image understanding, aims to accurately divide meaningful regions from underwater images or videos, such as pipelines, reefs, sediments, artificial structures, various seabed topographies, or marine organisms. This task not only has broad prospects in applications such as marine ecological investigation, seabed terrain modeling, and pipeline detection, but also is the basis for realizing intelligent perception of deep-sea scenes.

[0003] Current mainstream image segmentation techniques mainly focus on natural images, and representative methods include:

[0004] Fully Convolutional Network (FCN) methods: such as U-Net, DeepLab series, SegNet, etc., which rely on multi-layer convolution and up-sampling structures to complete pixel-level prediction. Transformer architecture segmentation models: such as SETR, MaskFormer, etc., which use self-attention mechanisms to model global context and achieve leading performance on certain public datasets. SAM (Segment Anything Model) general-purpose models: with strong zero-shot segmentation capabilities, attracting widespread attention in the natural image field. Weakly supervised or few-shot segmentation methods: such as BoxInst, BBAM, PseudoSeg, etc., which attempt to train models with a small number of bounding boxes or image-level labels to reduce dependence on expensive annotations.

[0005] Although image segmentation techniques have made great progress in recent years, especially with the widespread application of deep learning methods, semantic segmentation and instance segmentation have achieved high accuracy in the natural image field. However, directly applying these methods to deep-sea image segmentation tasks still faces the following problems:

[0006] (1) Most existing methods are based on natural image training and lack adaptability to the unique degradation characteristics of deep-sea images (such as low brightness, color shift, blur, and reflection), resulting in poor feature quality extracted by the model in actual underwater images, ultimately affecting the accuracy and boundary clarity of the segmentation results;

[0007] (2) Mainstream models such as Mask R-CNN and SAM have large structures and redundant calculations, which are unstable in deep-sea images, often resulting in blurred boundaries, broken biological individuals, or over-segmentation, and the model volume is large, with slow inference speed;

[0008] (3) Deep-sea images often contain dense structures (such as sediment particles, seabed rock fissures, coral clusters, etc.). Existing models are prone to problems such as biological individuals sticking together, occlusion without separation, and edge breakage in scenarios with overlapping biological individuals and blurred boundaries, resulting in a significant decrease in segmentation accuracy and a lack of modeling mechanism for complex spatial relationships.

[0009] (4) Most image segmentation models rely on a large number of pixel-level masks for supervised training, but the annotation cost of deep-sea images is extremely high, the manual efficiency is low, and data resources are scarce, which restricts the generalization and transfer capabilities of models in deep-sea missions.

[0010] (5) In deep-sea video missions, existing models often process each frame independently, and cannot use the temporal information of previous and next frames to suppress redundancy or enhance continuity, resulting in inconsistent processing of moving organisms (such as defects in pipeline inspections, sediment disturbances, etc.) and unstable segmentation problems. Summary of the Invention

[0011] To address the shortcomings of existing technologies, this invention proposes a deep-sea image instance segmentation system based on a lightweight hybrid visual structure, comprising a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module, and an integrated weakly supervised guidance module.

[0012] The feature extraction module receives the raw data frame, extracts multi-scale feature maps, low-level high-resolution feature maps, and time-step feature maps, and adjusts the extraction strategy in real time according to the pseudo-mask of the integrated weak supervision guidance module.

[0013] The image feature fusion module receives multi-scale feature maps, adjusts the weights through a multi-scale fusion path combined with an underwater adaptive attention mechanism, enhances key features and boundary information to generate a fused feature map, and sends it to the instance segmentation module. At the same time, it uses a pseudo-mask to guide attention to focus on salient regions.

[0014] The dynamic feature fusion module fuses the feature map with the time step features. Figure Two The second fusion generates a dynamic feature fusion map, which is then transmitted to the instance segmentation module.

[0015] The instance segmentation module uses a dense instance-aware decoder to receive fused feature maps, low-level high-resolution feature maps, dynamic fused feature maps and pseudo-masks. For deep-sea organism aggregation scenarios, it generates a corresponding segmentation mask for each individual and combines it with the low-level high-resolution feature map to generate a segmentation result image.

[0016] The integrated weakly supervised guidance module receives the original data frame and bounding box annotations, generates a pseudo-mask through the SAM model and sends it to the instance segmentation module, while providing region guidance for the feature extraction module and attention weight learning guidance for the image feature fusion module.

[0017] As a further improvement of the present invention, obtaining the feature map in the feature extraction module includes: using a lightweight hybrid spatiotemporal encoder as the underlying feature extractor and embedding a deep learning Transformer model with a compact attention mechanism to obtain the feature map.

[0018] As a further improvement of the present invention, the lightweight hybrid spatiotemporal encoder used in the image feature extraction module adopts a hybrid CNN + Transformer architecture.

[0019] As a further improvement of the present invention, the multi-scale feature fusion path in the image fusion module refers to the bottom-up and top-down fusion path, specifically the flow of multi-scale feature maps from the shallower to the deeper layers of the lightweight hybrid spatiotemporal encoder.

[0020] As a further improvement of the present invention, the instance segmentation module uses an instance-aware decoder to segment deep-sea organisms into instances.

[0021] As a further improvement of the present invention, for dense clusters, an instance center prediction and learning clusterer, instance embedding and metric learning or a repulsion mechanism are adopted. At the same time, a boundary refinement module is used, and with the help of fused feature maps, low-level high-resolution feature maps, and dynamic fused feature maps, the candidate biological individual regions are initially predicted based on pseudo-masks to generate segmentation result images and complete the instance segmentation of deep-sea organisms.

[0022] The beneficial effects of this invention are:

[0023] To address the issue of poor underwater image quality, an underwater adaptive module is added. This module can adaptively adjust image features, enhance recognition capabilities under adverse conditions such as low light and turbid water, and improve the model's ability to identify and segment individual underwater organisms.

[0024] To more accurately delineate the outlines of living organisms, especially for each complex individual, the dense instance segmentation decoder employs a boundary refinement subnetwork or module. This module uses high-resolution feature maps from earlier layers of the encoder (passed through skip connections) to fine-tune the initially predicted mask boundaries, thereby improving the fitting accuracy for complex shape boundaries.

[0025] The bounding boxes are directly input into the system. First, they are labeled on the original data frame. Then, the original data frame image and its corresponding bounding box labels are used together as input for the training process. This can reduce the workload of manual labeling because drawing simple boxes is much faster and cheaper than creating precise pixel-level masks.

[0026] It can accept several temporal step feature maps from multiple consecutive frames and fuse information from different time points through a temporal fusion unit to obtain a dynamic fused feature map, thereby enhancing the understanding of dynamic biological individuals. Instance segmentation using the dynamic fused feature map and the underlying high-resolution feature map can utilize temporal information from consecutive frames for redundancy suppression or continuity enhancement.

[0027] Aimed at overcoming the shortcomings of existing technologies in terms of real-time performance, robustness, processing capabilities for specific biological individuals, and data efficiency, this study focuses particularly on efficient deployment on AUV / ROV platforms. It is expected to achieve real-time inference on typical AUV / ROV embedded platforms, improve segmentation accuracy in scenes containing dense schools of fish and complex boundaries, enhance robustness in low-light and high-turbidity environments, and achieve near-fully supervised performance using bounding box-supervised training, thereby significantly reducing annotation costs.

[0028] This can be implemented using deep learning frameworks such as PyTorch or TensorFlow. The training process can employ optimizers like AdamW, and appropriate learning rate scheduling strategies and data augmentation methods can be selected based on the specific task and dataset. Attached Figure Description

[0029] Figure One This is a system framework diagram for this embodiment;

[0030] Figure Two This is a diagram showing the instance segmentation result in this embodiment. Detailed Implementation

[0031] In this embodiment, the original data frame and bounding box annotation data are input into the deep-sea image instance segmentation system based on a lightweight hybrid visual structure;

[0032] The original data frame includes image data frames and video data frames; the bounding box annotation data is annotated on the original data frame, and then the original data frame and its corresponding bounding box annotation data are used together as input to the deep-sea image instance segmentation system based on lightweight hybrid visual structure.

[0033] In this implementation, the SAM (Segment Anything Model) model has been pre-trained on a large number of original data frames.

[0034] like Figure One As shown, the deep-sea image instance segmentation system based on a lightweight hybrid vision structure adopts the underwater vision network Hydro-Sight-Net. This network uses an improved U-Net encoder-decoder architecture, including a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module, and an integrated weakly supervised guidance module.

[0035] The feature extraction module employs a lightweight hybrid spatiotemporal encoder to receive input raw data frames, including image data frames and video data frames. Based on these image and video data frames, it extracts feature maps: a lightweight convolutional neural network (CNN) is used as the underlying feature extractor, i.e., the lightweight hybrid spatiotemporal encoder, embedding a deep learning Transformer model with a compact attention mechanism to balance efficiency and feature extraction capability, synergistically optimizing local feature extraction, global context understanding, and computational efficiency. A channel attention mechanism or a lightweight online image enhancement subnetwork is introduced into the lightweight hybrid spatiotemporal encoder to adapt to underwater characteristics, thereby obtaining feature maps. These feature maps include multi-scale feature maps, a low-level high-resolution feature map, and several time-step feature maps. The multi-scale feature maps are transmitted to the image feature fusion module; the low-level high-resolution feature map is transmitted to the instance segmentation module; and the several time-step feature maps are transmitted to the dynamic feature fusion module, used to adjust the feature map extraction strategy in real-time based on the pseudomask received from the integrated weakly supervised guidance module.

[0036] In the deep-sea environment, organisms often occlude or overlap. To address this, embedding a Transformer introduces an attention mechanism, which can better understand these complex spatial relationships, improve the distinguishability of individual organisms, and enhance the feature map extraction capability under low light by aggregating global information and reducing the impact of local noise and blur.

[0037] Specifically, the image feature extraction module uses a lightweight hybrid spatiotemporal encoder with a hybrid CNN + Transformer architecture:

[0038] A lightweight convolutional neural network (CNN) is used as the backbone, and downsampling (such as convolution with a stride of 2) is performed after each convolutional layer to extract features at different spatial scales. Typically, three feature layers are formed: Stage 1 (1 / 4 size), Stage 2 (1 / 8), and Stage 3 (1 / 16).

[0039] Adding Transformer to supplement global awareness: Embed a compact Transformer module after feature layer Stage 2 or feature layer Stage 3:

[0040] The convolutional output is flattened into tokens and input into the transformer block. A self-attention mechanism is introduced to enhance each location's understanding of other regions of the image, thus improving the shortcomings of CNNs in global modeling.

[0041] The Transformer module is designed to address the specific challenges of underwater imagery, such as low contrast, color distortion, and blur. This module learns to assign higher weights to features at specific spatial locations that contain sharper object boundaries.

[0042] The method for extracting multi-scale feature maps is as follows:

[0043] First, shallow features of underwater image data frames are extracted using convolutional layers in a CNN:

[0044] A typical convolutional layer outputs a shallow feature map, FeatureMap1, which is 1 / 4 the size of the original image.

[0045] Continue convolutional extraction to output a low-resolution feature map, FeatureMap2, with a size of 1 / 16. Gradually reduce the spatial resolution to improve semantics.

[0046] A Transformer module is embedded on a feature map of size 1 / 16 to model long-range dependencies and enhance global semantics, thereby outputting a multi-scale feature map FeatureMap3.

[0047] The method for extracting the underlying high-resolution feature map is as follows:

[0048] To address the issues of blurred boundaries and the fusion of biological entities, Hydro-Sight-Net uses direct branching or skip connections in the shallow layers of its lightweight hybrid spatiotemporal encoder to preserve high-resolution features.

[0049] Typically, a feature path is separated at feature layer 1 or feature layer 2. After 1×1 convolution to compress the channel, the high-level features are used for semantics, and the low-level features are used for fine boundaries, thus outputting a high-resolution low-level feature map.

[0050] When extracting time-step feature maps from video data, the lightweight hybrid spatiotemporal encoder processes the video frame by frame, treating each video frame as a static image to extract its features.

[0051] Specifically, a lightweight convolutional neural network is used to extract hierarchical features at different scales. A Transformer module is embedded in the feature layer Stage 3 of the convolutional network to capture the global context information of the video for that specific frame. This process is repeated for each frame in the video sequence, thereby generating several time step feature maps for each independent time step.

[0052] The extraction strategy for real-time adjustment of feature maps is performed during the training phase;

[0053] During training, the pseudomask guides the lightweight encoder to update its internal parameters through backpropagation, prompting the encoder to learn the region defined by the pseudomask, thereby generating a stronger activation function and producing higher quality features, while suppressing the response of the background region.

[0054] Through the above methods, the lightweight hybrid spatiotemporal encoder is effectively trained, making it better at recognizing and emphasizing features relevant to biological individuals.

[0055] The image feature fusion module receives multi-scale feature maps transmitted by the feature extraction module; constructs a multi-scale feature fusion path based on the multi-scale feature maps; dynamically adjusts feature weights by combining an underwater adaptive attention mechanism to enhance the expression of key underwater image features and boundary information, thereby obtaining a fused feature map; the fused feature map is transmitted to the instance segmentation module; it is used to guide the pseudo-mask guidance attention mechanism received from the integrated weakly supervised guidance module to focus on salient regions during the multi-scale feature map fusion process, thereby improving fusion accuracy;

[0056] The multi-scale feature fusion path mentioned here refers to the bottom-up and top-down fusion paths, specifically the flow of multi-scale feature maps from shallower to deeper layers of the lightweight hybrid spatiotemporal encoder.

[0057] The top-down path involves propagating semantically richer features from deeper layers back to shallower layers. This is achieved through upsampling (increasing the spatial resolution of feature maps) and combining them with multi-scale feature maps from the bottom-up path at the corresponding scale. This allows the network to utilize contextual information from deeper layers to refine detailed features.

[0058] At the feature fusion points of the bottom-up and top-down paths, a Transformer module is introduced to enable the network to learn how to best combine features from different scales, thereby assigning higher weights to the most relevant features for the task of segmenting underwater objects.

[0059] Feature fusion involves taking top-down and bottom-up features of the same scale, weighting and summing them point-by-point, or concatenating them, and then processing the information through convolution to generate richer fused features. The fusion process also uses an underwater adaptive attention module to adjust the weights of different channels or spatial locations, making the features more adaptable to the underwater environment. This means that deep features contain overall semantic information about the object, but have lower resolution and less detail; shallow features have high resolution but shallow semantics. The top-down path upsamples this useful semantic information from deeper layers back to the shallower layers, combining it with shallower details to make the shallower features more accurate and meaningful. The goal is to combine high-resolution details with deep semantics to improve segmentation quality.

[0060] Specifically, the top-down path starts by upsampling the features from the deepest layer of the multi-scale feature map. At each fusion node, the features from the top-down path are combined with the features from the bottom-up path at the corresponding scale. Before fusing the features, an underwater adaptive attention module is applied to weight the features to adapt to the underwater environment, thereby generating a fused feature map.

[0061] The Transformer module performs global correlation modeling on the input feature map. Specifically, it transforms the features extracted by the CNN into a series of feature sequences (tokens), and uses a self-attention mechanism to calculate the correlation between each position in the sequence and all other positions, capturing long-distance dependencies. The final output feature map contains both local information and enhanced global context awareness.

[0062] The underwater adaptive attention mechanism dynamically calculates attention weights for each multi-scale feature map's feature fusion points as input feature maps. These weights are not fixed and are learned for each input image during training. This module, once trained, can identify underwater image-specific attributes such as color cast, low contrast, and blur, and adjust the weights accordingly to highlight the feature channels or spatial locations most informative for the segmentation task. This process enables the model to adaptively amplify key features and suppress noise, thereby enhancing its recognition and segmentation capabilities in complex underwater environments. Furthermore, the weights themselves are automatically learned through backpropagation, driven by the overall segmentation task loss.

[0063] The image feature fusion module incorporates an attention mechanism that learns to focus on important regions when fusing feature maps of different scales. During training, a pseudo-mask provided by the weakly supervised ensemble guidance module marks the location of the organisms of interest. The loss function incorporating this pseudo-mask information guides the attention module to assign higher weights to features within these salient regions. Through this mechanism, the network learns to prioritize information corresponding to the organisms during the fusion process, thereby generating more accurate fused feature representations and improving segmentation performance.

[0064] The dynamic feature fusion module receives a fused feature map transmitted by the image feature fusion module; receives several time-step feature maps transmitted by the image feature extraction module; and performs fusion again based on the fused feature map and the several time-step feature maps to obtain a dynamic feature fusion map, which is then transmitted to the instance segmentation module.

[0065] Specifically, the dynamic feature fusion module calculates the motion of pixels between frames and distorts and aligns the features of the moving organism before fusion. A Convolutional Long Short-Term Memory (ConvLSTM) network is used to process the feature map sequence. ConvLSTM assists in analyzing the current frame by maintaining an internal memory of past frames and simultaneously applying convolutional operations in both spatial and temporal dimensions. This allows it to directly learn spatiotemporal features that represent motion and temporal changes, and dynamically assign importance weights to different frames in the sequence, enabling the model to focus on the most crucial moments in time.

[0066] Spatiotemporal features refer to the ability of a dynamic feature fusion module in a video scene to accept several temporal feature maps from multiple consecutive frames and fuse information from different time points through a temporal fusion unit, thereby enhancing the understanding of dynamic biological individuals. Ultimately, the output is a dynamic feature fusion map that integrates spatial multi-scale information and temporal information.

[0067] For videos of moving fish schools, feature maps and fused feature maps at different time steps are received, and various fusion methods are employed, including optical flow guidance, recurrent / convolutional LSTM, 3D convolution, and temporal attention. These methods can be dynamically enabled or disabled based on task requirements or input data type. The final result is a dynamically fused feature map.

[0068] like Figure Two As shown, the instance segmentation module employs a dense instance-aware decoder, receiving a fused feature map transmitted by the image feature fusion module; receiving a low-level high-resolution feature map transmitted by the feature extraction module; receiving a dynamic fused feature map transmitted by the dynamic feature fusion module; and receiving a pseudo-mask transmitted by the integrated weak supervision guidance module. This is used to generate a corresponding segmentation mask for each individual organism in cases of clustering of deep-sea organisms, based on the fused feature map, the dynamic fused feature map, and the pseudo-mask. The segmentation result image is generated by combining the segmentation mask with the low-level high-resolution feature map, thereby completing the instance segmentation of deep-sea organisms.

[0069] In response to the frequent aggregation of deep-sea organisms, a dense instance perception decoder is used to distinguish and segment organisms that are crowded or overlapping.

[0070] Specifically, for dense clusters, instance center prediction and learning clusterers, instance embedding and metric learning or the introduction of repulsion mechanisms are used. At the same time, a boundary refinement module is used, and with the help of fused feature maps, low-level high-resolution feature maps, and dynamic fused feature maps, the candidate biological individual regions are initially predicted based on pseudo-masks to generate segmentation result images and complete the instance segmentation of deep-sea organisms.

[0071] When segmenting biological individuals, different processing methods are adopted according to the state of the biological individual.

[0072] For static images of deep-sea organisms, a segmentation process is performed using fused feature maps and low-level high-resolution feature maps. The fused feature maps are fed in parallel into three highly complementary decoding paths to achieve maximum robustness and accuracy.

[0073] The first approach utilizes a multi-task Mask Head as the primary component for the system's classification function. This module outputs two key pieces of information in parallel for each candidate biological region: first, it predicts the specific category of the organism and outputs a confidence score quantifying the reliability of the prediction; second, it directly generates a segmentation mask for the candidate organism associated with the predicted category. This approach can complete classification and segmentation in a single step for handling regular, well-defined targets.

[0074] The candidate biological individual regions mentioned here are obtained through pseudo-masked regions and clustering algorithms.

[0075] Path two is based on instance embedding heads, whose core task is to achieve accurate individual separation. This module generates a high-dimensional unique identity vector for each pixel, and then a clustering algorithm performs unsupervised grouping based on the proximity relationships of the unique identity vectors in the feature space. The advantage of this path is that it does not require prior knowledge of the number of targets and can effectively separate individuals that severely overlap in physical space.

[0076] Path three introduces instance center prediction and a learning-based clustering mechanism. This path also operates on the fused feature map, but it predicts a center offset vector pointing to the geometric center of its respective instance for each foreground pixel. All pixels are mapped into an implicit instance center space based on their predicted offset vectors, forming an initial instance affiliation distribution. Subsequently, a learning-based clusterer groups these affiliation centers or embedding vectors. Compared to traditional clustering methods, this learning-based clusterer is better able to adapt to the complex conditions unique to underwater images, such as blurring, occlusion, and dense distribution. It automatically defines cluster boundaries through learning, ultimately dividing the pixels of the fused feature map into several subsets. Each subset corresponds to a segmentation mask, and each mask contains all pixels associated with a certain inferred center point.

[0077] Next, the decision-making phase begins. This phase first adopts the segmentation mask output by the masking path, which possesses both a high confidence score and a clear category label. Then, in ambiguous or conflicting regions where the masking path prediction failed or had low confidence, the module simultaneously references the outputs of path two and path three. Through internal decision-making logic, the most reliable segmentation result is selected to fill in and correct the ambiguous regions.

[0078] In addition, in order to assign species identification to these masks that are supplemented from path two or path three and do not carry category information, the system will use the spatial overlap (IoU) to make them inherit the predicted category with the highest score at the corresponding position of the mask.

[0079] Ultimately, this initial robust segmentation mask, formed through the complementary advantages of three paths and the intelligent fusion of multi-source information, and with each biological individual bearing a category identifier, is uniformly fed into the boundary refinement sub-network. This network utilizes the underlying high-resolution feature map to perform pixel-level fine-tuning and correction on the edges of the segmentation mask, thereby outputting a segmentation result image in which each biological individual is not only accurately identified and clearly separated, but also correctly classified and labeled, and has a highly smooth contour.

[0080] In addition to the strategy of predicting candidate biological regions, Hydro-Sight-Net further introduces instance embedding and metric learning mechanisms to enhance individual separation capabilities in scenarios with dense distribution of biological individuals and severe occlusion. During the decoding phase, Hydro-Sight-Net learns to generate a fixed-dimensional embedding vector for each foreground pixel on the feature map, ensuring that the embedding vectors of all foreground pixels from the same instance are close to each other in the feature space, while the embeddings of pixels from different instances have sufficient spacing. To achieve these characteristics, a discriminative loss function is introduced during the training phase for optimization. This loss function consists of three terms: a variance term within a class, which constrains the embedding vectors of pixels within the same instance to converge towards their class center, enhancing internal consistency; a distance term between classes, which increases the distance between the class centers of different instances, improving discriminability; and a regularization term, which prevents the embedding values ​​from growing unbounded and stabilizes the training process. This loss guides the network to form a clear and separable cluster structure in the embedding space. During the inference phase, after the network outputs the embedding vector for each pixel, the density-based spatial clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is used to unsupervisedly group the pixels based on the Euclidean distance between the embedding vectors. DBSCAN does not require specifying the number of candidate biological individual regions and can automatically identify each biological individual in the foreground based on the distribution density of points in the embedding space, effectively eliminating outliers.

[0081] To more accurately delineate the outline of living organisms, especially for those with complex shapes, the decoder employs a boundary refinement subnetwork or module. This module uses high-resolution feature maps from earlier layers of the encoder (passed through skip connections) to fine-tune the initially predicted mask boundaries, thereby improving the fitting accuracy for complex shape boundaries.

[0082] For segmenting active biological individuals, a dynamic fusion feature map and a low-level high-resolution feature map are used for instance segmentation. The segmentation process is the same as that for segmenting static biological individuals, except that the fusion feature map used is changed from a static fusion feature map to a dynamic fusion feature map.

[0083] The integrated weakly supervised guidance module receives the input raw data frame and its corresponding bounding box annotation data; it combines the raw data frame and its corresponding bounding box annotation data to generate a pixel mask through the SAM model, which is then used as a pseudo-mask and transmitted to the instance segmentation module; it provides region guidance to the feature extraction module; and it provides fusion attention weights for learning biological individuals to the image feature fusion module.

[0084] The bounding boxes are directly input into the system. They are first labeled on the original data frame, and then the original data frame image and its corresponding bounding box labels are used together as input for the training process. This aims to reduce the workload of manual labeling, because drawing simple boxes is much faster and cheaper than creating precise pixel-level masks.

[0085] Combining the original data frame and its corresponding bounding box annotation data, SAM generates multiple candidate masks in its mask decoding head, selects the optimal mask through an internal confidence scoring mechanism, and outputs a binary pixel mask corresponding to the bounding box region.

[0086] During the training phase of Hydro-Sight-Net, the binary pixel mask is used as a pseudo-mask to replace manual annotation and guide the training of the network.

[0087] This module can generate pseudomasks using a pre-trained SAM model. When the input only contains bounding box annotations of deep-sea organisms, the SAM model can automatically generate corresponding binary pixel masks as pseudomasks based on these bounding boxes. Because SAM is pre-trained on a large amount of data, the pseudomasks it generates are usually of a certain quality and can provide useful supervision signals for the training of Hydro-Sight-Net.

[0088] Furthermore, this integrated weakly supervised guidance module designs a composite loss function to guide the network's training. This loss function typically includes the following components:

[0089] Apply the pseudomask generated by SAM or a pseudomask generated in combination with other methods. Calculate the standard segmentation loss on the generated pseudomask.

[0090] Loss terms are defined based on the alignment between the bounding box and the prediction mask or other constraints, such as bounding box alignment loss or internal consistency loss.

[0091] Based on the specific design of the integrated weakly supervised guidance module, it includes instance center prediction loss, boundary point prediction loss, or instance embedding metric learning loss.

[0092] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A deep-sea image instance segmentation system based on a lightweight hybrid visual structure, characterized in that, It includes a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module, and an integrated weakly supervised guidance module; The feature extraction module receives raw data frames, which include image data frames and video data frames. Multi-scale feature maps and low-level high-resolution feature maps are extracted from the input image data frames; time step feature maps of each frame are extracted from the video sequence composed of the input video data frames, and the extraction strategy is adjusted in real time according to the pseudo-mask of the integrated weak supervision guidance module. The image feature fusion module receives multi-scale feature maps, adjusts the weights through a multi-scale fusion path combined with an underwater adaptive attention mechanism, enhances key features and boundary information to generate a fused feature map, and sends it to the instance segmentation module. At the same time, it uses a pseudo-mask to guide attention to focus on salient regions. The dynamic feature fusion module fuses the fused feature map with the time step feature map a second time to generate a dynamic feature fusion map, which is then transmitted to the instance segmentation module. The instance segmentation module uses a dense instance-aware decoder to receive fused feature maps, low-level high-resolution feature maps, dynamic fused feature maps and pseudo-masks. For deep-sea organism aggregation scenarios, it generates a corresponding segmentation mask for each individual and combines it with the low-level high-resolution feature map to generate a segmentation result image. The integrated weakly supervised guidance module receives the original data frame and bounding box annotations, generates a pseudo-mask through the SAM model and sends it to the instance segmentation module, while providing region guidance for the feature extraction module and attention weight learning guidance for the image feature fusion module.

2. The deep-sea image instance segmentation system based on a lightweight hybrid visual structure according to claim 1, characterized in that, The feature extraction module obtains the feature map by: using a lightweight hybrid spatiotemporal encoder as the underlying feature extractor and embedding a deep learning Transformer model with a compact attention mechanism, thereby obtaining the feature map.

3. The deep-sea image instance segmentation system based on a lightweight hybrid visual structure according to claim 2, characterized in that, The image feature extraction module uses a lightweight hybrid spatiotemporal encoder with a hybrid CNN + Transformer architecture.

4. The deep-sea image instance segmentation system based on a lightweight hybrid visual structure according to claim 3, characterized in that, The multi-scale feature fusion path in the image feature fusion module refers to the bottom-up and top-down fusion paths, specifically the flow of multi-scale feature maps from shallower to deeper layers of the lightweight hybrid spatiotemporal encoder.

5. The deep-sea image instance segmentation system based on a lightweight hybrid visual structure according to claim 1, characterized in that, The instance segmentation module uses an instance-aware decoder to segment deep-sea organisms into instances.

6. The deep-sea image instance segmentation system based on a lightweight hybrid visual structure according to claim 5, characterized in that, For dense clusters, we employ instance center prediction and learning clusterers, instance embedding and metric learning, or introduce a repulsion mechanism. We also use a boundary refinement module, fused feature maps, low-level high-resolution feature maps, and dynamically fused feature maps, and preliminarily predict candidate biological individual regions based on pseudomasks to generate segmentation result images, thus completing the instance segmentation of deep-sea organisms.

Images