A marine animal image segmentation method based on Mamba key token perception fine-grained feature refinement

Abstract

The application discloses a marine animal image segmentation method based on Mamba key Token perception fine-grained feature refinement, comprising: acquiring multi-scale coding features of an image to be segmented; introducing a KFFR-Mamba refinement framework in each decoding stage; positioning a key Token position based on current input features and performing position calibration to obtain a calibrated key Token position; constructing a feature sequence based on the calibrated key Token position, adjusting a scanning order for rearrangement; performing feature refinement on the rearranged feature sequence to obtain enhanced features of the current decoding stage; and generating a segmentation result based on the enhanced features output by the decoding stage with the highest spatial resolution. The application can model the boundary, slender structure and local shape change area of marine animals, reduce the interference of redundant areas in a complex background, improve the positioning accuracy of key areas and the cross-scale context expression capability, and improve the accuracy, structural integrity and fine-grained boundary quality of marine animal segmentation under the condition of reasonable calculation overhead.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a marine animal image segmentation method based on Mamba key token perception fine-grained feature refinement. Background Technology

[0002] Marine animal segmentation aims to accurately separate marine life from complex marine scenes, providing crucial visual information for applications such as marine ecological analysis, fish resource assessment, aquaculture management, and perception for autonomous underwater robots. However, marine environments are typically characterized by low visibility, complex backgrounds, and varied appearances, making it difficult to accurately distinguish marine animals from their surroundings. Furthermore, marine animals often exhibit irregular shapes and complex boundaries, further increasing the difficulty of fine-grained segmentation. These factors make marine animal segmentation more challenging than segmentation in natural image scenes.

[0003] In early research, Convolutional Neural Networks (CNNs) made significant progress in marine-related segmentation tasks by effectively learning hierarchical feature representations. However, the limited receptive field of CNNs can increase the difficulty of distinguishing marine animals in complex backgrounds. In recent years, Transformer methods have been introduced to improve performance by capturing global contextual information, a characteristic particularly suitable for marine animal segmentation tasks. In particular, models based on SegmentAnythingModel (SAM) benefit from their pre-training on over a billion natural images, thus possessing strong representational capabilities. However, both Transformer and SAM suffer from high computational and parameter requirements. SAM, in particular, is not specifically designed for underwater scenes, making it difficult to accurately recover target boundaries and fine-grained local structures.

[0004] State-space models like Mamba have gained attention due to their ability to model the global context with linear complexity. However, these models tend to perform dense feature modeling across the entire feature map and treat all spatial locations equally, lacking targeted modeling of regions that truly influence segmentation results. In complex marine scenes, not all tokens contribute equally to visual understanding. Therefore, locating key tokens with high discriminative value from complex underwater feature maps and performing targeted, fine-grained enhancements is a pressing issue that current technologies need to address. Furthermore, existing feature sampling or fusion methods based on fixed grid positions cannot accurately align with real target boundaries and fine structures, resulting in insufficient feature representation of edge regions. Especially in situations with low contrast, motion blur, local occlusion, and target camouflage, coarse spatial location selection can easily cause key regions to deviate from their true structural positions, further affecting subsequent feature aggregation and segmentation prediction. Therefore, how to further calibrate the spatial positions of key tokens after selection to more accurately align them with the boundaries and local structures of marine animals is also a crucial technical problem that needs to be solved. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a marine animal image segmentation method based on Mamba key token-aware fine-grained feature refinement, specifically proposing a key token-aware fine-grained (KFFR-Mamba) refinement framework. This framework consists of two main sub-modules: a Region-Aware Key-Token Localization (RKL) module and a Global-Aware Fine-Grained Refinement (GFR) module. After extracting multi-scale features, the RKL module first identifies and calibrates informative key tokens based on cross-scale differences and local redundancy; subsequently, the GFR module refines the features of these tokens by introducing relevant coarse-scale context through Mamba with a learnable scan order. Experimental results on multiple datasets demonstrate that the method of this invention achieves competitive performance in marine animal segmentation tasks with relatively low computational overhead.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] A marine animal image segmentation method based on Mamba key token-aware fine-grained feature refinement includes:

[0008] The image to be segmented is acquired, and multi-scale encoded features are extracted using an encoder.

[0009] A multi-stage decoder is constructed, and the KFFR-Mamba refinement framework is introduced in each decoding stage; the encoded features at the corresponding scale are fused with the enhanced features output from the previous coarser-scale decoding stage as the current input features;

[0010] The key token position is located based on the current input features, and the key token position is continuously calibrated to obtain the calibrated key token position.

[0011] Based on the location of the key token, a feature sequence is constructed, and the scanning order of the feature sequence is adjusted to obtain the rearranged feature sequence. The rearranged feature sequence is then refined to obtain the enhanced features for the current decoding stage.

[0012] The segmentation result of the image to be segmented is generated based on the enhanced features output from the decoding stage with the highest spatial resolution.

[0013] As one implementation method, the key token location based on the current input features includes: calculating the key token prior response based on the cross-scale difference response between the current input features and the enhanced features in the coarser-scale decoding stage, as well as the local similarity response of the current input features; dividing the current input features into non-overlapping local units, and selecting key token locations in the local units based on the key token prior response.

[0014] As one implementation, calculating the key token prior response based on the cross-scale difference response between the current input features and the enhanced features from the coarser-scale decoding stage, as well as the local similarity response of the current input features, includes: aligning the enhanced features from the coarser-scale decoding stage to the current scale to obtain the aligned enhanced features from the coarser-scale decoding stage at any spatial location on the current input features. Calculate cross-scale differential response :

[0015] ,

[0016] in, This represents the current input feature in the i-th decoding stage. This indicates enhanced features in the coarser-scale decoding stage after alignment. Represents the L2 norm distance;

[0017] Calculate the mean cosine similarity between any spatial location on the current input feature and its four neighboring locations, as the local similarity response. Based on cross-scale difference responses and local similarity responses, the prior response of the key token is calculated:

[0018]

[0019] in, The exponential coefficient representing the cross-scale differential response. The exponential coefficient represents the local similarity response.

[0020] As one implementation method, dividing the current input features into non-overlapping local units and filtering key token positions in the local units based on the key token prior response includes: dividing the current input features into several non-overlapping local units, and selecting the N positions with the highest key token prior response in each local unit as key token positions.

[0021] As one implementation method, the step of continuously calibrating the key token position to obtain the calibrated key token position includes: obtaining a point position offset vector through a position calibration network based on the current input features, the aligned coarser-scale stage enhancement features, and the key token prior response; and adding the point position offset vector to the key token position to obtain the calibrated key token position.

[0022] As one implementation method, the step of constructing a feature sequence based on the location of the calibrated key tokens and adjusting the scanning order of the feature sequences to obtain a rearranged feature sequence includes: sampling the current stage key token corresponding to the location of the calibrated key token from the current input features; sampling the cross-scale context token corresponding to the location of the calibrated key token from the aligned coarser-scale stage enhancement features; concatenating and fusing them to obtain a cross-scale token group; organizing the cross-scale token group into a feature sequence according to its spatial location; obtaining the prior response of the calibrated key token location; generating a scanning order offset through a scanning order prediction network; and rearranging the feature sequence based on the scanning order offset to obtain the rearranged feature sequence.

[0023] As one implementation, the step of refining the rearranged feature sequence to obtain enhanced features for the current decoding stage includes: refining the rearranged feature sequence using a sequence modeling network, wherein the sequence modeling network includes multiple scanning paths, including a basic scanning path and a reverse scanning path; the scanning order offset is applied to the basic scanning path, and the reverse scanning path is obtained by flipping the basic scanning path with the applied scanning order offset; mapping the outputs of the multiple scanning paths back to the original order of the feature sequence and fusing them to obtain a refined feature sequence; and locally merging the refined feature sequence with the current input features to obtain enhanced features for the current decoding stage.

[0024] This invention also provides a marine animal image segmentation system based on Mamba key token-aware fine-grained feature refinement, used to implement the marine animal image segmentation method described above, comprising:

[0025] The feature acquisition module is used to acquire multi-scale encoded features of the image to be segmented.

[0026] The feature fusion module is used to construct a multi-stage decoder and introduces the KFFR-Mamba refinement framework in each decoding stage; it fuses the encoded features at the corresponding scale with the enhanced features output from the previous coarser-scale decoding stage as the current input features.

[0027] The node localization and calibration module is used to locate the key token position based on the current input features and perform continuous position calibration on the key token position to obtain the calibrated key token position.

[0028] The feature refinement module is used to construct a feature sequence based on the location of the calibrated key tokens, adjust the scanning order of the feature sequence to obtain a rearranged feature sequence, and refine the features of the rearranged feature sequence to obtain the enhanced features of the current decoding stage.

[0029] The results output module is used to generate segmentation results for the image to be segmented based on the enhanced features output from the decoding stage with the highest spatial resolution.

[0030] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the marine animal image segmentation method as described above.

[0031] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the marine animal image segmentation method as described above.

[0032] The beneficial effects of this invention are:

[0033] The method of this invention introduces the KFFR-Mamba refinement module in all decoding stages, so that the decoding features at different scales can be enhanced around the structurally sensitive regions, thereby improving the feature recovery capability from coarse to fine.

[0034] From a mechanistic perspective, this invention first utilizes cross-scale differences and local similarity to measure the importance of spatial location: cross-scale differences highlight locations where representations differ between different decoding scales; these locations typically correspond to target boundaries, local abrupt changes, or regions with significant scale variations. Local similarity reflects the structural consistency between the current token and its four neighboring tokens; locations with lower similarity are more likely to belong to boundaries, details, or foreground-background transition regions. Based on these two types of clues, this invention can prioritize key tokens that contribute more to the segmentation results, reducing the interference of complex backgrounds and low-information regions on feature refinement.

[0035] After locating the key token, this invention further performs continuous spatial calibration on the coarse key token position by offsetting the point position, adjusting it to a position that is closer to the actual structure of marine animals. This improves the spatial alignment between the key area and the target boundary, fin structure, tail details and other slender structures, and alleviates the problems of boundary misalignment, local missing parts and incomplete outlines.

[0036] During the refinement stage, this invention organizes local detail information at the current scale and semantic context information at the coarse scale into a key token sequence at the key token location. The structural relationships between key tokens are modeled using a learnable scan order and SS2D-based Mamba sequence modeling. Compared to a fixed scan order, a learnable scan order can better adapt to the irregular contours and complex morphologies of marine animals. Compared to dense refinement of the entire image, refinement around key tokens allows more computation to be allocated to boundaries, slender structures, and regions with local shape variations, improving fine-grained segmentation quality while controlling computational overhead.

[0037] In terms of specific performance metrics, this invention achieves superior results on three marine animal segmentation benchmarks: RMAS, Mask3K, and DeepFish. The results demonstrate that this invention can reduce pixel-level prediction errors and improve foreground prediction stability, structure preservation, and region consistency across different marine animal segmentation datasets. For example, on the RMAS dataset, the MAE of this invention is 0.0206, lower than MAS-SAM's 0.0229, Dual-SAM's 0.0220, multi-SAM's 0.0221, and Samba's 0.0223; on the Mask3K dataset, the MAE of this invention is 0.0233, lower than MAS-SAM's 0.0258, Dual-SAM's 0.0252, multi-SAM's 0.0263, and Samba's 0.0276; on the DeepFish dataset, the MAE of this invention is 0.0053, lower than MasNet's 0.0086, TC-USOD's 0.0070, MAS-SAM's 0.0092, multi-SAM's 0.0062, and Samba's 0.0109.

[0038] Regarding computational complexity, the number of floating-point operations and the number of parameters in the method of this invention are as follows: Compared to MAS-SAM's 141.03G, Dual-SAM's 325.68G, multi-SAM's 252.11G, and Samba's 94.37GFLOPs, this invention achieves superior overall segmentation results with lower or relatively lower computational costs. Therefore, the performance improvement of this invention does not solely rely on increased model size, but rather stems from the synergistic effect of key token-aware localization, point position offset calibration, cross-scale key token aggregation, and learnable scan order Mamba refinement.

[0039] In summary, this invention can focus on modeling marine animal boundaries, slender structures, and regions with local shape variations, reducing interference from redundant regions in complex backgrounds, improving the localization accuracy of key regions and cross-scale contextual expression capabilities, and enhancing the accuracy, structural integrity, and fine-grained boundary quality of marine animal segmentation while maintaining reasonable computational overhead. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0041] Figure 2 This diagram illustrates a qualitative comparison with state-of-the-art methods in marine animal segmentation. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] A marine animal image segmentation method based on Mamba key token-aware fine-grained feature refinement, such as... Figure 1 As shown, it includes the following steps:

[0044] Step 1: Obtain the image to be segmented and extract multi-scale encoded features using an encoder.

[0045] Specifically, the encoder can be a Transformer encoder, a Mamba-based encoder, or a VMamba-based encoder, etc. This embodiment constructs a VMamba-based encoder to segment the image. After inputting the VMamba-based encoder, four scales of encoded features are obtained, denoted as... ,in Indicates the feature scale index. Indicates the first Encoding features at each scale, and A larger value indicates a lower spatial resolution and a higher semantic level. The coding features at each scale can be represented as ,in Indicates the first The height of the feature map at each scale. Indicates the first The width of the feature map at each scale. Indicates the first The number of channels in a scale feature map.

[0046] Step 2: Construct a multi-stage decoder and introduce the KFFR-Mamba refinement framework in each decoding stage. The encoded features at the corresponding scale are fused with the enhanced features output from the previous coarser-scale decoding stage and used as the input features of the KFFR-Mamba refinement framework in the current decoding stage.

[0047] A four-stage decoder is constructed, and a KFFR-Mamba refinement framework is introduced in each decoding stage. This framework includes a Region-Aware Key Token Localization (RKL) module and a Global-Aware Fine-Grained Refinement (GFR) module. The decoder performs feature recovery in a coarse-to-fine order, processing the features sequentially. , No. , No. and the The decoding stage. In the first... Each decoding stage, the current input feature is denoted as... , Encoding features at the current scale Enhanced features compared to the output of the previous coarser decoding stage After upsampling and fusion, the following results were obtained:

[0048]

[0049] in, Indicates an upsampling operation. This indicates the input fusion operation during the decoding phase. Therefore Input the first feature The KFFR-Mamba module with +1 decoding stage utilizes the enhanced features obtained from the coarser-scale decoding stage as cross-scale context to perform key token localization and refinement, outputting the first... Enhanced features for each decoding stage.

[0050] For the The coarsest-scale decoding stage lacks features enhanced by a coarser-scale decoding stage. From coding features After decoding and transforming the input, KFFR-Mamba is obtained based solely on the current input features. Execution of key token positioning and refinement:

[0051] .

[0052] Step 3: Input the current input features into the RKL module of the KFFR-Mamba thinning framework, generate a key token prior map using cross-scale differences and local similarities, and obtain the calibrated key token positions through local cell selection and point position offset.

[0053] In the In each decoding stage, the current input features are input into the RKL module.

[0054] For decoding stages other than the coarsest scale, first enhance the features of the coarser-scale decoding stages. Align to current input features The spatial resolution yields enhanced features in the coarser-scale decoding stage after alignment. , Indicates from the first The first scale to the first Spatial alignment is performed at each scale. Then, the current input features are... any spatial location Calculate cross-scale differential response :

[0055] ,

[0056] in, This represents the L2 norm distance. The larger the value, the higher the current position. The more pronounced the difference in representation between the current-scale features and the coarser-scale features, the more likely the location corresponds to a target boundary, a local shape change, or a foreground-background transition region; for the th In the coarsest-scale decoding stage, since there are no coarser-scale decoding stages to enhance features, the calculation of cross-scale difference responses is omitted.

[0057] For the current input features any spatial location on Take the set of four neighboring positions of its top, bottom, left, and right adjacent positions. ,set up for Calculate the position of any neighborhood location in the given information. The mean cosine similarity with the four neighboring locations serves as a local similarity response. :

[0058]

[0059] in, This represents the cosine similarity calculation after channel-dimensional normalization. The lower the value, the higher the position. The weaker the local consistency between a location and its neighboring locations, the more likely that location is situated in a boundary, elongated structure, or region of abrupt local texture change.

[0060] Based on cross-scale differential responses and local similarity responses, the prior responses of key tokens are calculated, and a prior graph of key tokens is constructed. The formula for calculating the key token prior response is as follows:

[0061]

[0062] in, The exponential coefficient representing the cross-scale differential response. The exponential coefficient representing the local similarity response. and Optimize the parameters for the model. The larger the value, the more likely that position is a key token position that requires further refinement. For the first... In the coarsest-scale decoding stage, since cross-scale difference responses are omitted, its key token prior graph is constructed only from local similarity responses:

[0063]

[0064] The current input feature map is divided into several non-overlapping local units, denoted as... ,in This represents a local cell index, where the size of the local cell is... , or In this embodiment, we take Within each local unit, the N positions with the highest prior responses to the key token are selected as the key token positions. N can be 1 to 3, and is 1 in this embodiment; key token position The selection process is represented as follows:

[0065]

[0066] All key token locations are grouped into a key token location set: , Indicates the first The total number of key token locations in each decoding stage. By selecting key token locations, dense locations across the entire image are compressed into a set of structurally sensitive candidate locations that contribute more to the segmentation results. This reduces the interference of complex backgrounds and low-information regions on feature refinement, while also reducing redundant computations in subsequent Mamba sequence modeling.

[0067] Because the key token locations are still constrained by a discrete grid, they may not be accurately aligned with real boundaries or slender structures. Therefore, a point position offset predictor is used for continuous position calibration. Point position offset predictor Based on the current input features, the aligned coarser-scale stage-enhanced features, and the key token prior map, the spatial offset is predicted to obtain the point position offset vector. , constitute the first Point position offset diagram for each decoding stage:

[0068]

[0069] in, Represents the two-dimensional coordinates on the current input feature map. This represents the hyperbolic tangent function, used to limit the offset magnitude.

[0070] For the For each key token location, extract the corresponding offset vector from the point position offset map: Add the point position offset vector to the key token position to obtain the calibrated key token position. :

[0071]

[0072] All key token locations are grouped into a key token location set: . This will serve as the spatial anchor point for the current stage key token and cross-scale context token sampling in the GFR module. Point position offset can adjust the key token to a position closer to the actual structure of marine animals, thereby improving the spatial alignment between key areas and target boundaries, fin structures, tail details and other slender structures, and mitigating problems such as boundary misalignment, local missing parts and incomplete outlines.

[0073] Step 4: Based on the calibrated key token positions, sample the current stage key token and the coarser-scale stage context token, and then refine the key token sequence with structure awareness through scan order offset and SS2D-Mamba to obtain enhanced features.

[0074] Execute the GFR module, for the first Each decoding stage begins with the current input features. The current stage key token is identified by mid-sampling at the key token position; then, at the same key token position, features are enhanced from the aligned coarser-scale stage. Mid-sample cross-scale context tokens, and then concatenate and fuse them to obtain a cross-scale token group that marks the position of key tokens. :

[0075]

[0076] in, This indicates the operation of sampling features according to a given continuous spatial location. This indicates a channel dimension or token dimension concatenation operation; for the first... In the coarsest-scale decoding stage, since there are no coarser-scale stages to enhance features,... Based solely on the current input features exist The current stage key tokens obtained from sampling constitute the key tokens. Cross-scale token groups enable each calibrated key token to simultaneously carry local details at the current scale and available coarser-scale context information.

[0077] Organize all cross-scale token groups corresponding to the key token positions into a key token sequence according to their spatial location: ;in, This refers to the operation of arranging multiple cross-scale token groups into a sequence according to their spatial location. Because... It only includes tokens at the key token locations, rather than all tokens on the entire feature map, thus reducing redundant sequence modeling and computation while preserving structurally sensitive information.

[0078] In the set of key token locations Upsampling key token prior graph This yields the prior response corresponding to the key token sequence; a scan order offset predictor is then used. Predict the scan order offset of the key token sequence: ;in, This indicates the set of key token locations. Upsampling key token prior graph The obtained prior response. Scan order offset. Used to adjust the key token sequence The relative scanning order of each token in the Mamba database allows it to no longer process tokens according to a fixed grid order, but to adaptively determine the scanning path based on the content of structurally sensitive areas.

[0079] Using scan order offset Key Token Sequence The key tokens are rearranged in descending order, and the rearranged sequence is input into the SS2D-based Mamba submodule for feature refinement, resulting in a refined key token sequence: ;in, Indicates the offset by the scan order Guided sequence rearrangement operation. This represents the Mamba state-space sequence modeling operation based on SS2D. In actual implementation, the SS2D-based Mamba submodule uses four scan paths for modeling, including two basic scan paths and two reverse scan paths. The two basic scan paths correspond to different two-dimensional scan sequences, including row-first scan and transpose scan, with a scan sequence offset. It operates on two basic scan paths, and the two reverse scan paths are offset by the corresponding application scan order. The basic scan path is obtained by flipping it; the outputs of the four scan paths are then mapped back to the key token sequence. The key tokens are sequentially sequenced and merged to obtain a structure-aware key token representation, i.e., a refined key token sequence.

[0080] The refined key token sequence With current input features Perform local merging to obtain the enhanced features for the current scale decoding stage: ;in, This indicates the refined key token sequence. According to the set of key token locations Merge into current input features The local fusion operation in the model enhances only the structure-sensitive key token regions, rather than densely refining the entire feature map, resulting in enhanced features. It continues to participate in the input fusion of the next finer-scale decoding stage, and serves as the output and feature source for segmentation prediction in the current scale decoding stage.

[0081] Step 5: At each decoding stage, a segmentation prediction map is output through the segmentation prediction head, and the segmentation prediction map with the highest spatial resolution is used as the final marine animal segmentation result.

[0082] Specifically, for the first Each decoding stage will enhance the features. Input the segmentation prediction head to obtain the stage prediction results: ;in, Indicates the first The marine animal segmentation prediction map output by each decoding stage. Indicates the first Each decoding stage corresponds to a segmentation prediction head. The segmentation prediction map uses the highest spatial resolution. As the final result of marine animal segmentation.

[0083] The training process of the overall network consisting of the encoder, decoder, and segmentation prediction head described above is as follows:

[0084] Animal images from marine scenes are acquired as training and testing data, and each image is labeled with a binary segmentation mask. For any input image, let be... ,in Represents the space of real numbers. This represents the input image of a marine animal. Indicates the image height. Indicates image width, number This represents the red, green, and blue color channels. (Compared to the input image) The corresponding real binary segmentation mask is denoted as These serve as foreground labels for marine animals used during supervised training. The input images are uniformly adjusted to... The spatial resolution is input into the overall network, and the prediction results of each decoding stage are obtained using the above method.

[0085] The entire network is trained using deep supervised learning. For the first... Prediction results of each decoding stage A supervision loss is constructed, consisting of binary cross-entropy loss, structural similarity loss, and intersection-union ratio loss. :

[0086]

[0087] in, Represents the binary cross-entropy loss. Represents structural similarity loss. This indicates the loss compared to the intersection of the two.

[0088] The total loss for the four phases is:

[0089]

[0090] in, Indicates the first The weights of the supervision loss in each decoding stage are set to a default value. .

[0091] During training, the encoder is initialized using VMamba-S pre-trained weights and trained using the AdamW optimizer. epochs, initial learning rate set to And every training The learning rate will be set per epoch. The attenuation factor is used for attenuation, and the batch size is set to [value missing]. Through the above training process, the model learns from the input image To the final segmentation prediction The end-to-end mapping relationship.

[0092] This embodiment also provides a marine animal image segmentation system based on Mamba key token-aware fine-grained feature refinement, including:

[0093] The feature acquisition module is used to acquire multi-scale encoded features of the image to be segmented.

[0094] The feature fusion module is used to construct a multi-stage decoder and introduces the KFFR-Mamba refinement framework in each decoding stage; it fuses the encoded features at the corresponding scale with the enhanced features output from the previous coarser-scale decoding stage as the current input features.

[0095] The node localization and calibration module is used to locate the key token position based on the current input features and perform continuous position calibration on the key token position to obtain the calibrated key token position.

[0096] The feature refinement module is used to construct a feature sequence based on the location of the calibrated key tokens, adjust the scanning order of the feature sequence to obtain a rearranged feature sequence, and refine the features of the rearranged feature sequence to obtain the enhanced features of the current decoding stage.

[0097] The results output module is used to generate segmentation results for the image to be segmented based on the enhanced features output from the decoding stage with the highest spatial resolution.

[0098] This embodiment also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the marine animal image segmentation method as described above.

[0099] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the marine animal image segmentation method as described above.

[0100] Effect verification

[0101] To verify the effectiveness of the method of this invention, comparative experiments were conducted on three marine animal segmentation benchmarks (RMAS, Mask3K, and DeepFish) using the same dataset. Existing methods included MasNet, iGAN, MAS-SAM, Dual-SAM, multi-SAM, TC-USOD, SENet, and Samba. The parameters MAE, maxF, and avgF were set. , and FLOPs and Params were used as metrics for evaluating segmentation quality, respectively, to assess computational cost and model size. All experiments were performed on a single NVIDIA GeForce RTX 4090 GPU. The encoder was initialized with pre-trained weights using VMamba-S, and the model was trained with AdamW for 300 epochs at an initial learning rate of 5×10⁻⁶. -4 The batch size is set to 4, and the input resolution is fixed at 512×512. The batch size is also set to 4.

[0102] The quantitative comparison results are shown in Table 1:

[0103] Table 1. Quantitative comparison results with existing methods on RMAS, Mask3K, and DeepFish.

[0104]

[0105] As shown in Table 1, compared with existing methods, the method of the present invention achieves lower MAE in most cases, and in... , , , and Competitive results were achieved. This demonstrates that the method of the present invention can reduce pixel-level prediction errors and produce more stable foreground predictions. In particular... and The improvement indicates that the method of the present invention can maintain reliable segmentation quality under different threshold settings, while and The improvements indicate better structure preservation and region-level consistency. These improvements are primarily due to the proposed key-token-aware fine-grained refinement strategy. Unlike methods that uniformly process the entire feature map, this invention focuses more on high-resolution regions with information, such as target boundaries, elongated structures, and local shape variations. Therefore, it can enhance important local details while reducing redundant modeling of low-information regions.

[0106] Regarding model complexity, the method of this invention also demonstrates a reasonable trade-off between accuracy and computational cost. Compared with SAM-based methods such as MAS-SAM, Dual-SAM, and multi-SAM, the method of this invention achieves better segmentation performance with a moderate number of parameters and FLOPs. Although some lightweight methods have lower computational costs, they are generally less accurate on most metrics. This indicates that the performance improvement of the method of this invention does not simply come from an increase in model size, but from a more efficient refinement design, that is, allocating computation to structurally sensitive regions.

[0107] Figure 2 The qualitative results further support the quantitative results in Table 1. In underwater scenes with low contrast, complex backgrounds, blurred boundaries, and irregular target shapes, some existing methods are prone to problems such as incomplete foreground regions, boundary leakage, or loss of local details. In contrast, the method of this invention can generate more complete target regions and clearer contours, and can also better preserve irregular boundaries, elongated structures, and local details near the foreground-background transition area. These visualization results are consistent with... , and The consistent improvement indicates that the method of the present invention is effective for fine-grained detail preservation and structure-aware segmentation.

[0108] To verify the effectiveness of the two core modules, two offset designs, and the insertion stage of the refinement module in the method of this invention, ablation experiments were conducted. The experimental results are shown in Tables 2 to 4.

[0109] Table 2. Ablation experiment results on the effectiveness of the core module

[0110]

[0111] First, the individual and joint contributions of the RKL and GFR modules were evaluated, as shown in Table 2. Introducing either the RKL module (or the GFR module (Baseline+RKL)) consistently improved baseline performance, while the full model achieved the best overall performance. This result indicates that the RKL and GFR modules play different but complementary roles: the RKL module enhances the localization capability of information-rich regions, while the GFR module further strengthens the structural refinement capability of the selected regions.

[0112] The method of this invention incorporates two learnable offset mechanisms: keypoint position offset in the RKL module and scan order offset in the GFR module. To verify its necessity, Table 3 compares four settings: removing both offsets, removing only the keypoint position offset, removing only the scan order offset, and using the full model. In the table, PO and SO represent the keypoint position offset and scan order offset, respectively. It can be observed that both offset designs positively contribute to the final performance. Removing the keypoint position offset weakens the alignment between coarse candidate locations and structurally sensitive regions, while removing the scan order offset reduces the flexibility of local structural sequence modeling. The full model achieves the best results, indicating that the two offset designs have a combined benefit.

[0113] Table 3. Ablation experiments with two offset designs

[0114]

[0115] The method of this invention includes sparse key token selection. As shown in Table 4, without sparse key token selection and instead refining all spatial tokens, the model achieves only a very limited improvement in accuracy, but FLOPs increase to 152.46 G. In contrast, the method of this invention selects more discriminative tokens from structurally sensitive regions through key token priors, focusing on refining only the selected locations, reducing the computational cost to 86.01 G while maintaining near-or better segmentation performance. This result demonstrates that key token selection can effectively reduce redundant computation and improve the balance between accuracy and efficiency.

[0116] Table 4. Ablation experiments on key token selection in the RKL module

[0117]

[0118] In the table, "w / o selection" means that instead of sparse key token selection, dense refinement is used.

[0119] Finally, the impact of different cross-scale token sources in the GFR module on model performance was evaluated. Since the GFR module performs fine-grained enhancement on calibrated key tokens by aggregating cross-scale context tokens, Table 5 compares four settings: using only the current scale token (Current), introducing the most recent coarser-scale decoding feature (Current+N1), introducing the two most recent coarser-scale decoding features (Current+N2), and using all available coarser-scale decoding features (Current+All).

[0120] It can be observed that the model performance is relatively weak when using only the current-scale token, but as coarser-scale decoding features are gradually introduced, all indicators show a continuous upward trend. This result indicates that cross-scale context tokens can provide effective high-level semantic information and structural supplementary clues for key token refinement, helping to enhance the model's ability to express target boundaries, local shape changes, and structurally sensitive regions. The complete model achieves the best results, demonstrating that GFR can fully utilize the complementary information between multi-scale decoding features to achieve more effective structure-aware fine-grained feature refinement.

[0121] Table 5. Ablation experiments on cross-scale token origination in the GFR module

[0122]

[0123] Overall, the above ablation results verify that the gain of the method of the present invention does not come from a single isolated design, but from the synergistic effect of multiple carefully designed components, including key region localization, sparse region selection, learnable offset modeling, and staged refinement in the later decoder.

[0124] The above description represents a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A marine animal image segmentation method based on Mamba key token-aware fine-grained feature refinement, characterized in that, include: The image to be segmented is acquired, and multi-scale encoded features are extracted using an encoder. A multi-stage decoder is constructed, and the KFFR-Mamba refinement framework is introduced in each decoding stage; the encoded features at the corresponding scale are fused with the enhanced features output from the previous coarser-scale decoding stage as the current input features; The key token position is located based on the current input features, and the key token position is continuously calibrated to obtain the calibrated key token position. Based on the location of the key token, a feature sequence is constructed, and the scanning order of the feature sequence is adjusted to obtain the rearranged feature sequence. The rearranged feature sequence is then refined to obtain the enhanced features for the current decoding stage. The segmentation result of the image to be segmented is generated based on the enhanced features output from the decoding stage with the highest spatial resolution.

2. The method according to claim 1, characterized in that, The method of locating key token positions based on current input features includes: calculating the prior response of key tokens based on the cross-scale difference response between the current input features and the enhanced features in the coarser-scale decoding stage, as well as the local similarity response of the current input features; dividing the current input features into non-overlapping local units, and selecting key token positions in the local units based on the prior response of key tokens.

3. The method according to claim 2, characterized in that, The calculation of the key token prior response based on the cross-scale difference response between the current input features and the enhanced features from the coarser-scale decoding stage, as well as the local similarity response of the current input features, includes: aligning the enhanced features from the coarser-scale decoding stage to the current scale, obtaining the aligned enhanced features from the coarser-scale decoding stage at any spatial location on the current input features. Calculate cross-scale differential response : ， in, This represents the current input feature in the i-th decoding stage. This indicates enhanced features in the coarser-scale decoding stage after alignment. Represents the L2 norm distance; Calculate the mean cosine similarity between any spatial location on the current input feature and its four neighboring locations, as the local similarity response. Based on cross-scale difference responses and local similarity responses, the prior response of the key token is calculated: in, The exponential coefficient representing the cross-scale differential response. The exponential coefficient represents the local similarity response.

4. The method according to claim 2, characterized in that, The step of dividing the current input features into non-overlapping local units and filtering key token positions in the local units based on the prior response of the key token includes: dividing the current input features into several non-overlapping local units, and selecting the N positions with the highest prior response of the key token in each local unit as key token positions.

5. The method according to claim 3, characterized in that, The step of continuously calibrating the key token position to obtain the calibrated key token position includes: obtaining the point position offset vector through the position calibration network based on the current input features, the aligned coarser-scale stage enhancement features, and the key token prior response; and adding the point position offset vector to the key token position to obtain the calibrated key token position.

6. The method according to claim 3, characterized in that, The process of constructing a feature sequence based on the location of the calibrated key tokens and adjusting the scanning order of the feature sequences to obtain a rearranged feature sequence includes: sampling the current stage key token corresponding to the location of the calibrated key token from the current input features; sampling the cross-scale context token corresponding to the location of the calibrated key token from the aligned coarser-scale stage enhancement features; concatenating and fusing them to obtain a cross-scale token group; organizing the cross-scale token group into a feature sequence according to its spatial location; obtaining the prior response of the calibrated key token location; generating a scanning order offset through a scanning order prediction network; and rearranging the feature sequence based on the scanning order offset to obtain the rearranged feature sequence.

7. The method according to claim 6, characterized in that, The step of refining the rearranged feature sequence to obtain enhanced features for the current decoding stage includes: refining the rearranged feature sequence using a sequence modeling network, wherein the sequence modeling network includes multiple scanning paths, including a basic scanning path and a reverse scanning path; the scanning order offset is applied to the basic scanning path, and the reverse scanning path is obtained by flipping the basic scanning path with the applied scanning order offset; mapping the outputs of the multiple scanning paths back to the original order of the feature sequence and fusing them to obtain a refined feature sequence; and locally merging the refined feature sequence with the current input features to obtain enhanced features for the current decoding stage.

8. A marine animal image segmentation system based on Mamba key token-aware fine-grained feature refinement, used to implement the method as described in any one of claims 1-7, characterized in that, include: The feature acquisition module is used to acquire multi-scale encoded features of the image to be segmented. The feature fusion module is used to fuse the encoded features at the corresponding scale with the enhanced features output from the previous coarser-scale decoding stage as the current input features; The node localization and calibration module is used to locate the key token position based on the current input features and perform continuous position calibration on the key token position to obtain the calibrated key token position. The feature refinement module is used to construct a feature sequence based on the location of the calibrated key tokens, adjust the scanning order of the feature sequence to obtain a rearranged feature sequence, and refine the features of the rearranged feature sequence to obtain the enhanced features of the current decoding stage. The results output module is used to generate segmentation results for the image to be segmented based on the enhanced features output from the decoding stage with the highest spatial resolution.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images