Deep sea image instance segmentation system based on lightweight mixed visual structure

Abstract

The invention discloses a deep sea image instance segmentation system based on a lightweight hybrid visual structure, belongs to the field of instance segmentation, and aims to solve the problems that a model based on natural image training is poor in adaptability to degradation characteristics of deep sea images, low in feature extraction quality and affects a segmentation result. The system comprises a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module and an integrated weak supervision guidance module. The feature extraction module receives input original data frames including image data frames and video data frames; extracting a feature map based on the image data frame and the video data frame; the extraction strategy is used for adjusting a feature map of the pseudo mask received from the integrated weak supervision guiding module in real time; and the image feature fusion module receives the multi-scale feature map transmitted by the feature extraction module and obtains a fused feature map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of instance segmentation, and particularly relates to a deep-sea image instance segmentation system based on a light-weight hybrid visual structure. BACKGROUND

[0002] With the continuous growth of the demand for ocean resource development and environmental monitoring, intelligent analysis of deep-sea images has become one of the key capabilities in underwater robot systems. Deep-sea image segmentation, as an important task of image understanding, aims to accurately divide meaningful regions from underwater images or videos, such as pipelines, reefs, sediments, artificial structures, various seabed topographies, or marine organisms. This task not only has broad prospects in applications such as marine ecological investigation, seabed terrain modeling, and pipeline detection, but also is the basis for realizing intelligent perception of deep-sea scenes.

[0003] Current mainstream image segmentation techniques mainly focus on natural images, and representative methods include: Fully Convolutional Network (FCN) methods: such as U-Net, DeepLab series, SegNet, etc., which rely on multi-layer convolution and up-sampling structures to complete pixel-level prediction. Transformer architecture segmentation models: such as SETR, MaskFormer, etc., which use self-attention mechanisms to model global context and achieve leading performance on certain public datasets. SAM (Segment Anything Model) general-purpose models: with strong zero-shot segmentation capabilities, attracting widespread attention in the natural image field. Weakly supervised or few-shot segmentation methods: such as BoxInst, BBAM, PseudoSeg, etc., which attempt to train models with a small number of bounding boxes or image-level labels to reduce dependence on expensive annotations.

[0004] Although image segmentation techniques have made great progress in recent years, especially with the widespread application of deep learning methods, semantic segmentation and instance segmentation have achieved high accuracy in the natural image field. However, directly applying these methods to deep-sea image segmentation tasks still faces the following problems: (1) Most existing methods are based on natural image training and lack adaptability to deep-sea image-specific degradation characteristics (such as low brightness, color shift, blur, and reflection), resulting in poor feature quality extracted by the model in actual underwater images, ultimately affecting the accuracy and boundary clarity of the segmentation results; (2) Mainstream models such as Mask R-CNN and SAM have large structures and redundant calculations, which are unstable in deep-sea images, often resulting in blurred boundaries, broken biological individuals, or over-segmentation, and the model volume is large, with slow inference speed; (3) Deep-sea images often contain dense structures (such as sediment particles, seafloor cracks, coral clusters, etc.). Existing models are prone to problems such as individual adhesion, occlusion, and edge fracture in scenes with overlapping organisms and blurred boundaries. The segmentation accuracy is significantly reduced, and there is a lack of a modeling mechanism for complex spatial relationships. (4) Most image segmentation models rely on a large number of pixel-level masks for supervised training. However, the labeling cost of deep-sea images is extremely high, the manual work efficiency is low, and the data resources are scarce, which restricts the generalization and migration capabilities of the models in deep-sea missions. (5) In deep-sea video tasks, existing models often process each frame independently and are unable to use the temporal information of previous and next frames to suppress redundancy or enhance continuity, resulting in incoherent processing of moving biological individuals (such as defects in pipeline inspections, sediment disturbances, etc.) and unstable segmentation problems. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes a deep-sea image instance segmentation system based on a lightweight hybrid visual structure, which includes a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module and an integrated weak supervision guidance module; The feature extraction module receives the original data frame, extracts the multi-scale feature map, the bottom-level high-resolution feature map, and the time-step feature map, and adjusts the extraction strategy in real time according to the pseudo mask of the integrated weak supervision guidance module; The image feature fusion module receives multi-scale feature maps, adjusts the weights through a multi-scale fusion path combined with an underwater adaptive attention mechanism, enhances key features and boundary information to generate a fused feature map that is passed to the instance segmentation module, and uses a pseudo mask to guide attention to focus on salient areas; Dynamic feature fusion module, which combines the fusion feature map with the time step feature Figure Two Second fusion, generating a dynamic feature fusion map and transmitting it to the instance segmentation module; The instance segmentation module uses a dense instance-aware decoder to receive the fused feature map, the underlying high-resolution feature map, the dynamic fused feature map, and the pseudo mask. For deep-sea biological aggregation scenes, it generates a corresponding segmentation mask for each individual and combines it with the underlying high-resolution feature map to generate a segmentation result image. The integrated weakly supervised guidance module receives the original data frame and bounding box annotations, generates a pseudo mask through the SAM model and passes it to the instance segmentation module. It also provides regional guidance for the feature extraction module and attention weight learning guidance for the image feature fusion module.

[0006] As a further improvement of the present invention, obtaining the feature map in the feature extraction module includes: using a lightweight hybrid spatiotemporal encoder as the underlying feature extractor, and embedding a deep learning Transformer model with a compact attention mechanism to obtain the feature map.

[0007] As a further improvement of the application, the lightweight hybrid space-time encoder used by the image feature extraction module adopts a hybrid CNN + Transformer architecture.

[0008] As a further improvement of the application, the multi-scale feature fusion path in the image fusion module refers to the bottom-up and top-down fusion path, specifically referring to the flow of multi-scale feature maps from the shallow layer to the deep layer of the lightweight hybrid space-time encoder.

[0009] As a further improvement of the application, the instance segmentation module uses an instance-aware decoder for instance segmentation of deep-sea organisms.

[0010] As a further improvement of the application, for dense clusters, an instance center prediction and learning-based clusterer, instance embedding and metric learning, or an introduced repulsive force mechanism are used, and a boundary refinement module is used, with the help of fusion feature maps, low-level high-resolution feature maps, dynamic fusion feature maps, and according to the preliminary prediction of candidate biological individual regions based on the pseudo mask, a segmentation result image is generated, and the instance segmentation of deep-sea organisms is completed.

[0011] The application has the following advantages: In view of the poor quality of underwater images, an underwater adaptive module is added, which can adaptively adjust the image features, enhance the recognition ability under poor conditions such as insufficient light and turbid water, and improve the recognition and segmentation ability of the model for underwater organisms.

[0012] In order to more accurately outline the outline of the organism, especially for each biological individual with complex morphology, the dense instance segmentation decoder uses a boundary refinement subnetwork or module. This module uses high-resolution feature maps from early layers of the encoder (delivered through a skip connection) to fine-tune the preliminary predicted mask boundary, thereby improving the fitting accuracy of complex shape boundaries.

[0013] The bounding box is directly input to the system, and is labeled on the original data frame first, and then the original data frame image and its corresponding bounding box label are used as input for the training process, which can reduce the workload of manual labeling, because it is faster and less costly to draw a simple box than to make an accurate pixel-level mask.

[0014] A plurality of time step feature maps of continuous multiple frames can be accepted, and the information at different time points can be fused through a time fusion unit to obtain a dynamic fusion feature map, thereby enhancing the understanding of dynamic biological individuals. Using dynamic fusion feature maps and low-level high-resolution feature maps for instance segmentation can utilize temporal information of previous and subsequent frames for redundancy suppression or continuity enhancement.

[0015] Aiming to overcome the deficiencies of the prior art in real-time performance, robustness, specific biological individual processing capability and data efficiency, with a particular focus on efficient deployment on AUV / ROV platforms. Real-time inference is expected to be achieved on typical AUV / ROV embedded platforms, improving segmentation accuracy in scenarios involving dense fish schools and complex boundaries, enhancing robustness in low-light and high-turbidity environments, and achieving near-full-supervised performance using bounding box supervision training, thereby significantly reducing annotation costs.

[0016] It can be implemented using deep learning frameworks such as PyTorch or TensorFlow. The training process can use optimizers such as AdamW, and appropriate learning rate scheduling strategies and data augmentation methods can be selected according to specific tasks and datasets. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure One is a system framework diagram in the embodiment; Figure Two is an instance segmentation result diagram in the embodiment. DETAILED DESCRIPTION

[0018] In this embodiment, the original data frame and the bounding box annotation data are input into the deep-sea image instance segmentation system based on the lightweight hybrid visual structure; wherein the original data frame includes image data frames and video data frames; the bounding box annotation data is annotated on the original data frame, and then the original data frame and its corresponding bounding box annotation data are taken together as the input of the deep-sea image instance segmentation system based on the lightweight hybrid visual structure; In this embodiment, the SAM (Segment Anything Model) model has been pre-trained on a large number of original data frames.

[0019] As shown in Figure One The deep-sea image instance segmentation system based on the lightweight hybrid visual structure adopts the underwater visual network Hydro-Sight-Net, which uses an improved U-Net type encoder-decoder architecture, including a feature extraction module, an image feature fusion module, a dynamic feature fusion module, an instance segmentation module, and an integrated weak supervision guide module.

[0020] The feature extraction module adopts a lightweight hybrid space-time encoder to receive input raw data frames, including image data frames and video data frames; based on the image data frames and video data frames, a feature map is extracted: a lightweight convolutional neural network (CNN) is used as a bottom feature extractor, i.e., a lightweight hybrid space-time encoder, a deep learning Transformer model embedded with a compact attention mechanism is used to balance efficiency and feature extraction capability, and local feature extraction, global context understanding and computational efficiency are optimized in cooperation, a channel attention mechanism is introduced in the adopted lightweight hybrid space-time encoder or a lightweight online image enhancement sub-network is integrated to adapt to underwater characteristics, so as to obtain a feature map; the feature map includes a multi-scale feature map, a bottom high-resolution feature map and a plurality of time step feature maps; the multi-scale feature map is transmitted to an image feature fusion module; the bottom high-resolution feature map is transmitted to an instance segmentation module; and the plurality of time step feature maps are transmitted to a dynamic feature fusion module; the feature map extraction strategy is adjusted in real time based on the pseudo mask received from the integrated weak supervision guiding module. In the deep-sea environment, organisms often have occlusion and overlap. In view of this situation, the Transformer is embedded to introduce the attention mechanism, which can better understand the complex spatial relationships, improve the individual differentiation of organisms, and reduce the influence of local noise and blur by aggregating global information, and enhance the feature extraction capability in weak light. Specifically, the lightweight hybrid space-time encoder used by the image feature extraction module adopts a hybrid CNN + Transformer architecture: A lightweight convolutional neural network (CNN) is used as the backbone, and downsampling (such as convolution with a stride of 2) is performed after each convolutional layer to extract features of different spatial scales. Generally, three feature layers are formed: Stage 1 (1 / 4 size), Stage 2 (1 / 8), and Stage 3 (1 / 16).

[0021] Add a Transformer to supplement global perception: embed a compact Transformer module after the feature layer Stage 2 or the feature layer Stage 3: Flatten the convolution output into tokens and input them into the transformer block. Introduce a self-attention mechanism to enhance the understanding of each position to other regions of the image and improve the shortcoming of CNN in global modeling.

[0022] The design of the Transformer module takes into account the specific challenges of underwater images, such as low contrast, color distortion and blur. This module can learn to give higher weights to features of specific spatial positions that contain clearer object boundaries.

[0023] The method for extracting a multi-scale feature map is as follows: First, the shallow features of the underwater image data frame are extracted using the convolutional layer in the CNN: The typical convolutional layer outputs a shallow feature map FeatureMap1 with a size of 1 / 4 of the original image size; Continuing the convolutional extraction, a low-resolution feature map FeatureMap2 is outputted with a size of 1 / 16, gradually reducing the spatial resolution and improving the semantics.

[0024] The Transformer module is embedded on the feature map with a size of 1 / 16 to model long-range dependencies and enhance global semantics, thereby outputting a multi-scale feature map FeatureMap3.

[0025] The extraction method of the bottom high-resolution feature map is as follows: To solve the problems of boundary blur and biological individual fusion, Hydro-Sight-Net directly branches or jump connects skip connection to reserve high-resolution features at the shallow layer of the lightweight hybrid space-time encoder: Usually, a branch of features is separated at the feature layer Stage 1 or the feature layer Stage 2, and after 1x1 convolution compression, the high-level features are used for semantics and the low-level features are used for fine boundaries, thereby outputting a bottom high-resolution feature map; When extracting time step feature maps for video data, the lightweight hybrid space-time encoder processes the video in frames, and each video frame is processed as a static image to extract its features.

[0026] Specifically, a lightweight convolutional neural network is used to extract hierarchical features of different scales. A Transformer module is embedded at the feature layer Stage 3 of the convolutional network to capture the global context information of the specific frame. This process is repeated for each frame in the video sequence, thereby generating several time step feature maps for each independent time step. The extraction strategy of real-time adjustment of feature maps is performed during the training phase. During the training process, the pseudo mask guides the lightweight encoder to update its internal parameters through backpropagation, prompting the encoder to learn the regions defined by the pseudo mask, thereby generating stronger activation functions and higher-quality features while suppressing the response of the background region.

[0027] In this way, the lightweight hybrid space-time encoder is effectively trained to be better at recognizing and emphasizing features related to biological individuals.

[0028] The image feature fusion module receives the multi-scale feature maps transmitted by the feature extraction module; a multi-scale feature fusion path is constructed based on the multi-scale feature maps, the feature weight is dynamically adjusted combined with the underwater adaptive attention mechanism, the expression and boundary information of the key features of the underwater image are enhanced, and a fused feature map is obtained; the fused feature map is transmitted to the instance segmentation module; and the pseudo mask received from the integrated weak supervision guiding module is used to guide the attention mechanism to focus on the significant area in the multi-scale feature map fusion process, so that the fusion precision is improved. The multi-scale feature fusion path mentioned here refers to the bottom-up and top-down fusion paths, specifically, the flow of multi-scale feature maps from the shallow layer to the deep layer of the lightweight hybrid space-time encoder.

[0029] The top-down path involves propagating richer semantic features from the deeper layer back to the shallower layer. This is achieved by upsampling operations (increasing the spatial resolution of the feature map) and combining them with multi-scale feature maps from the bottom-up path at the corresponding scale. This allows the network to utilize context information from the deeper layer to refine detailed features.

[0030] At the feature fusion point of the bottom-up and top-down paths, a Transformer module is introduced, which aims to let the network learn how to best combine features from different scales, giving higher weights to the most relevant features for the task of segmenting underwater objects.

[0031] Feature fusion is a process of point-by-point weighted addition or splicing of features of the same scale from the top-down and bottom-up paths, followed by convolution to fuse information and generate more rich fused features. The underwater adaptive attention module is also used to adjust the weights of different channels or spatial positions during the fusion process, making the features more suitable for underwater environments. This means that deep features contain overall semantic information about objects, but the resolution is low and the details are unclear; shallow features have high resolution but shallow semantics. The top-down path is to pass these useful semantic information from the deep layer back to the shallow layer through upsampling, and combine it with the details of the shallow layer to help the shallow layer features be more accurate and meaningful. The purpose of this is to combine high-resolution details and deep semantics to improve segmentation quality.

[0032] Specifically, the top-down path starts with upsampling the features from the deepest layer of the multi-scale feature map. At each fusion node, the features from the top-down path are combined with the features of the corresponding scale from the bottom-up path, and before the fused features, the underwater adaptive attention module is applied to weight the features to adapt to the underwater environment, thereby generating a fused feature map.

[0033] The Transformer module models the global correlation of the input feature map. Specifically, it converts the CNN-extracted features into a series of feature tokens, calculates the correlation between each position and all other positions in the sequence through self-attention mechanism, and captures long-distance dependencies. The output feature map contains both local information and enhanced global context awareness.

[0034] The underwater adaptive attention mechanism dynamically calculates attention weights for the feature fusion points of each multi-scale feature map. These weights are not fixed and are learned for each input image during training. After training, the module can identify the unique properties of underwater images, such as color bias, low contrast, and blur, and adjust the weights accordingly to highlight the feature channels or spatial positions that are most informative for the segmentation task. This process enables the model to adaptively amplify key features and suppress noise, thereby enhancing its recognition and segmentation capabilities in complex underwater environments. In addition, the weights are automatically learned through the backpropagation process driven by the overall segmentation task loss; The attention mechanism in the image feature fusion module can learn to focus on important regions when fusing feature maps of different scales. During training, the pseudo mask provided by the weakly supervised integrated weakly supervised guidance module indicates the location of the individual organism of interest, and the loss function containing this pseudo mask information guides the attention module to assign higher weights to the features in these prominent regions. Through this mechanism, the network learns to prioritize processing information corresponding to the individual organism during fusion, thereby generating more accurate fused feature representations and improving segmentation performance.

[0035] The dynamic feature fusion module receives the fused feature map transmitted by the image feature fusion module; receives several time step feature maps transmitted by the image feature extraction module; based on the fused feature map and several time step feature maps, fuses again to obtain a dynamic feature fusion map, which is transmitted to the instance segmentation module; Specifically, the dynamic feature fusion module warps and aligns the features of moving organisms before fusion by calculating the motion of inter-frame pixels. Convolutional Long Short-Term Memory Network (ConvLSTM) is used to process the feature map sequence. ConvLSTM maintains an internal memory about past frames to assist in analyzing the current frame, and applies convolution operations in both spatial and temporal dimensions, which can directly learn spatio-temporal features that represent motion and temporal changes, and dynamically assign importance weights to different frames in the sequence, enabling the model to focus on the most critical moments in time.

[0036] The space-time feature refers to that in a video scene, the dynamic feature fusion module can accept a plurality of time step feature maps of continuous multiple frames, and fuse information at different time points through a time fusion unit, thereby enhancing the understanding of dynamic biological individuals. Finally, a dynamic feature fusion map that fuses spatial multi-scale information and time information is output.

[0037] For fish school videos in a moving state, feature maps at different time steps and fused feature maps are received, and a plurality of fusion methods are adopted, including optical flow guidance, cyclic / convolution LSTM, 3D convolution or time attention, etc. These methods can be dynamically enabled or disabled according to task requirements or input data types. Finally, a dynamic feature fusion map is obtained.

[0038] As shown in Figure Two The instance segmentation module adopts a dense instance-aware decoder to receive the fused feature map transmitted by the image feature fusion module, receive the low-level high-resolution feature map transmitted by the feature extraction module, receive the dynamic fused feature map transmitted by the dynamic feature fusion module, and receive the pseudo mask transmitted by the integrated weakly supervised guidance module. For the case where deep sea organisms appear to be gathered, the corresponding segmentation mask is generated for each biological individual according to the fused feature map, the dynamic fused feature map and the pseudo mask, and the segmentation result image is generated by combining the low-level high-resolution feature map through the segmentation mask, thereby completing the instance segmentation of deep sea organisms. For the case where deep sea organisms frequently gather, a dense instance-aware decoder is used to distinguish and segment biological individuals that are crowded or overlapped together.

[0039] Specifically, for dense clusters, an instance center prediction and learning type clusterer, instance embedding and metric learning or introduction of repulsive force mechanism are adopted, and a boundary refinement module is used, with the aid of the fused feature map, the low-level high-resolution feature map, the dynamic fused feature map and according to the pseudo mask, the candidate biological individual region is preliminarily predicted, the segmentation result image is generated, and the instance segmentation of deep sea organisms is completed.

[0040] For biological individual segmentation, different processing methods are adopted according to the state of the biological individual.

[0041] For deep sea organism images in a stationary state, the fused feature map and the low-level high-resolution feature map are used to perform a segmentation process. The fused feature map is sent into three functionally highly complementary decoding paths in parallel to achieve extreme robustness and precision.

[0042] Path one is through a multi-task Mask Head, which is the main bearer of the system's classification function. This module outputs two key pieces of information for the candidate biological individual region in parallel: first, it predicts the specific category of the organism and outputs a confidence score quantifying the reliability of the prediction; second, it directly generates a candidate individual segmentation mask associated with the predicted category. This path can complete classification and segmentation at one time for regular and clear targets.

[0043] The candidate biological individual region mentioned here is obtained by a pseudo mask region and using a clustering algorithm.

[0044] Path two is based on an Instance Embedding Head, whose core task is to achieve accurate individual separation. This module generates a high-dimensional unique identity vector for each pixel, which is then grouped unsupervisedly by a clustering algorithm based on the proximity of unique identity vectors in the feature space. The advantage of this path is that it does not need to know the target number in advance and can effectively separate individuals that are severely overlapped in physical space.

[0045] Path three introduces an instance center prediction and learning-based clustering mechanism. This path also acts on the fused feature map, but it predicts a center offset vector for each foreground pixel that points to its own instance geometric center. All pixels are mapped to an implicit instance center space according to their predicted offset vectors, forming a preliminary instance attribution distribution. Then, the learning-based clusterer groups these attribution centers or embedding vectors. Compared to traditional clustering methods, this learning-based clusterer can better adapt to the complex conditions of underwater images, such as blur, occlusion, and dense distribution, automatically define the boundaries of clusters through learning, and finally divide the pixels of the fused feature map into several subsets, each corresponding to a segmentation mask, and each mask contains all the pixels associated with a certain inferred center point.

[0046] Then, the decision-making stage is entered. This stage first adopts the segmentation mask output by the Mask Head path that has both a high confidence score and a clear category label. Then, in the ambiguous or conflicting regions where the Mask Head prediction fails or the confidence is low, the module will simultaneously refer to the output results of Path Two and Path Three. Through internal decision logic, the most reliable segmentation result is selected to fill in and correct the ambiguous regions.

[0047] In addition, in order to give these masks from Path Two or Path Three, which do not have category information, a species identification, the system will make them inherit the highest scoring predicted category of the Mask Head at the corresponding position according to the spatial overlap IoU.

[0048] Finally, this preliminary robust segmentation mask, which is formed by the complementary advantages of the three paths and the intelligent fusion of multi-source information, and each biological individual is labeled with a category, is uniformly sent to the boundary refinement subnetwork. This network uses the bottom layer high-resolution feature map to perform pixel-level fine polishing and correction on the edge of the segmentation mask, thereby outputting a segmentation result image in which each biological individual is not only accurately identified and clearly separated, but also correctly classified and identified with a highly smooth contour.

[0049] In addition to the strategy based on candidate biological individual region prediction, Hydro-Sight-Net further introduces an instance embedding and metric learning mechanism to enhance the individual separation capability in the dense distribution of biological individuals and the severely occluded scene. Hydro-Sight-Net learns to generate a fixed-dimensional embedding vector for each foreground pixel on the feature map in the decoding stage, so that all foreground pixels from the same instance have embedding vectors close to each other in the feature space, and the embedding of pixels from different instances has sufficient distance. In order to achieve the above characteristics, a discriminative loss function is introduced in the training stage for optimization. The loss function is composed of three parts: the intra-class aggregation loss Variance Term, which constrains the embedding vectors of pixels within the same instance to move closer to their class centers, enhancing internal consistency; the inter-class separation loss Distance Term, which pulls the distance between class centers of different instances, improving the discrimination; and the regularization term Regularization Term, which prevents the embedding value from unbounded growth, stabilizing the training process. This loss guides the network to form a clear and separable clustering structure in the embedding space. In the inference stage, after the network outputs the embedding vector of each pixel point, the density clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is used to unsupervisedly group pixels according to the Euclidean distance between embedding vectors. DBSCAN does not need to specify the number of candidate biological individual regions, and can automatically identify each biological individual in the foreground according to the distribution density of points in the embedding space, and effectively exclude outliers.

[0050] In order to more accurately outline the contour of the biological body, especially for biological individuals with complex shapes, the decoder uses a boundary refinement subnetwork or module. This module uses high-resolution feature maps from early layers of the encoder (delivered through skip connections) to fine-tune the mask boundary of the preliminary prediction, thereby improving the fitting accuracy of complex shape boundaries.

[0051] For the segmentation of active biological individuals, dynamic fused feature maps and low-level high-resolution feature maps are used for instance segmentation. The segmentation process is consistent with the segmentation method of static biological individuals. However, the fused feature maps used are changed from static fused feature maps to dynamic fused feature maps.

[0052] The integrated weakly supervised guidance module receives input raw data frames and their corresponding bounding box annotation data; it uses the SAM model to generate pixel masks as pseudo masks for the instance segmentation module; it provides region guidance to the feature extraction module; and it provides fusion attention weight learning biological individuals to the image feature fusion module. The bounding box is directly input into the system, and the original data frame is labeled first, and then the original data frame image and its corresponding bounding box annotation are used as input for the training process, aiming to reduce the workload of manual annotation, as it is faster and less costly to draw a simple box than to create an accurate pixel-level mask.

[0053] In combination with the original data frame and its corresponding bounding box annotation data, SAM generates multiple candidate masks in its mask decoding head and selects the optimal mask through an internal confidence scoring mechanism, outputting a binary pixel mask corresponding to the bounding box region.

[0054] During the training phase of Hydro-Sight-Net, the binary pixel mask is used as a pseudo mask to replace manual annotation and guide network training.

[0055] This module can use a pre-trained SAM model to generate pseudo masks. When only the bounding box annotation of deep-sea organisms is input, the SAM model can automatically generate the corresponding binary pixel mask as a pseudo mask based on these bounding boxes. Since SAM is pre-trained on a large amount of data, the pseudo masks generated by it usually have a certain quality and can provide useful supervision signals for the training of Hydro-Sight-Net.

[0056] In addition, the integrated weakly supervised guidance module will design a composite loss function to guide network training. This loss function usually contains the following parts: Acting on the pseudo masks generated by SAM or other methods. Calculate the standard segmentation loss on the generated pseudo masks.

[0057] Loss terms defined based on the alignment degree between the bounding box and the predicted mask or other constraints, such as bounding box alignment loss or internal consistency loss.

[0058] According to the specific design of the integrated weakly supervised guidance module, it contains instance center prediction loss, boundary point prediction loss, or instance embedding metric learning loss.

[0059] The above-described embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A deep-sea image instance segmentation system based on a lightweight hybrid visual structure, characterized by: It includes feature extraction module, image feature fusion module, dynamic feature fusion module, instance segmentation module and integrated weak supervision guidance module; The feature extraction module receives the original data frame, extracts the multi-scale feature map, the bottom-level high-resolution feature map, and the time-step feature map, and adjusts the extraction strategy in real time according to the pseudo mask of the integrated weak supervision guidance module; The image feature fusion module receives multi-scale feature maps, adjusts the weights through a multi-scale fusion path combined with an underwater adaptive attention mechanism, enhances key features and boundary information to generate a fused feature map that is passed to the instance segmentation module, and uses a pseudo mask to guide attention to focus on salient areas; The dynamic feature fusion module fuses the fusion feature map with the time step feature map twice to generate a dynamic feature fusion map and transmit it to the instance segmentation module; The instance segmentation module uses a dense instance-aware decoder to receive the fused feature map, the underlying high-resolution feature map, the dynamic fused feature map, and the pseudo mask. For deep-sea biological aggregation scenes, it generates a corresponding segmentation mask for each individual and combines it with the underlying high-resolution feature map to generate a segmentation result image. The integrated weakly supervised guidance module receives the original data frame and bounding box annotations, generates a pseudo mask through the SAM model and passes it to the instance segmentation module. It also provides regional guidance for the feature extraction module and attention weight learning guidance for the image feature fusion module.

2. The deep-sea image instance segmentation system based on lightweight hybrid visual structure according to claim 1 is characterized in that: Obtaining the feature map in the feature extraction module includes: using a lightweight hybrid spatiotemporal encoder as an underlying feature extractor, and embedding a deep learning Transformer model with a compact attention mechanism to obtain the feature map.

3. The deep-sea image instance segmentation system based on lightweight hybrid visual structure according to claim 2, characterized in that: The lightweight hybrid spatiotemporal encoder used in the image feature extraction module adopts a hybrid CNN + Transformer architecture.

4. The deep-sea image instance segmentation system based on lightweight hybrid visual structure according to claim 3 is characterized in that: The multi-scale feature fusion path in the image fusion module refers to a bottom-up and top-down fusion path, specifically refers to the flow of multi-scale feature maps from shallower layers to deeper layers of the lightweight hybrid spatiotemporal encoder.

5. The deep-sea image instance segmentation system based on lightweight hybrid visual structure according to claim 1, characterized in that: The instance segmentation module uses an instance-aware decoder to perform instance segmentation of deep-sea organisms.

6. The deep-sea image instance segmentation system based on lightweight hybrid visual structure according to claim 5, characterized in that: For dense clusters, instance center prediction and learning clusterer, instance embedding and metric learning or repulsive force mechanism are adopted, and boundary refinement module is used at the same time. With the help of fusion feature map, underlying high-resolution feature map, dynamic fusion feature map, and pseudo mask, the individual area of ​​candidate organisms is preliminarily predicted to generate segmentation result image and complete the instance segmentation of deep-sea organisms.

Images