Marine organism image segmentation method and device

Abstract

This invention relates to a method and apparatus for marine biological image segmentation, belonging to the field of marine image target segmentation. To address the issue of improving the segmentation effect of underwater scenes, the key point is that the first stitched feature is obtained by passing through the efficient attention layer and feedforward network layer of the second model of the adaptive local-global feature fusion model to obtain the second feature of the first stitched feature; the second stitched feature and the second feature of the first stitched feature are stitched together to obtain the third stitched feature; a spatial attention map is obtained based on the third stitched feature; the second stitched feature is multiplied by the spatial attention map to obtain the first feature of the second stitched feature; the second feature of the first stitched feature is multiplied by (1-) to obtain the third feature of the second stitched feature; the first feature of the second stitched feature is added to the third feature of the second stitched feature to obtain the third stitched feature; and a biological image segmentation prediction map is obtained based on the third stitched feature. Experimental results show that the model performs excellently in underwater scene segmentation tasks.

Description

Technical Field

[0001] This invention belongs to the field of image target segmentation, and relates to a unified multimodal feature extraction and adaptive local-global feature fusion model and method for RGB-X marine organism image segmentation. Background Technology

[0002] Marine organism segmentation (MAS) is a critical task aimed at segmenting marine life in complex marine scenes. It plays a vital role in marine engineering, marine ecological resources, and management. Underwater environments present challenges such as insufficient light, low visibility, and cluttered background information. For example, tangled seaweed, suspended sediments, and coral reefs of varying shapes make distinguishing foreground from background in terms of color and texture extremely challenging. Furthermore, the camouflage of many marine organisms makes their appearance and texture highly similar to the surrounding background, posing a significant challenge to accurate segmentation of underwater images.

[0003] Early methods for segmenting marine targets employed convolutional neural networks (CNNs), utilizing the local receptive fields of convolutional kernels to extract features, enabling CNNs to capture spatial structural information in images, such as the shape and texture of objects. However, due to fixed parameters such as kernel size and stride, long-range dependencies in the image may be lost. Transformers perform better in processing image data; through a self-attention mechanism, they can directly compute the relationship between any two positions in a sequence, regardless of distance, and can effectively capture long-range dependencies. However, the computational complexity of the self-attention mechanism is high, consuming significant computational resources when processing large-scale data or complex tasks, severely impacting computational efficiency. The recently proposed Segmentation All Model (SAM) has made significant progress in the segmentation field. It is not only structurally robust but also trained using tens of millions of data points, improving its effectiveness. However, SAM training focuses more on some common scenarios, which differ significantly from underwater scenarios. To better adapt SAM to marine target segmentation, methods such as MAS-SAM and Dual-SAM have efficiently fine-tuned the SAM model, improving its segmentation performance in underwater scenes.

[0004] To improve underwater scene segmentation, some methods utilize auxiliary information to supplement RGB information. Depth information provides geometric details and enhances boundary recognition capabilities. Hong et al. introduced depth information as auxiliary information to enhance RGB information, improving the information completeness of the dataset. Optical flow information can capture pixel motion information between consecutive frames, and moving regions can be identified by comparing pixel displacements between adjacent frames. Zhang et al. cleverly used optical flow information as an auxiliary tool, significantly improving the performance of the segmentation task. Gamma information can adjust the nonlinear response of image brightness, improving the visual quality of the image and thus assisting segmentation. Zhang et al. used gamma information to improve segmentation performance, but there is a problem of excessively large parameters. These existing methods often focus on a specific combination of modes, which may only be applicable to limited marine scenes. At the same time, they often employ diverse and complex methods for multi-scale and multi-modal feature fusion, making the structural design complex. Summary of the Invention

[0005] To address the problem of improving the segmentation effect of underwater scenes, in a first aspect, a marine organism image segmentation method according to some embodiments of this application includes...

[0006] The first modality image is input from the attention layer and fine-tuned using LoRA to obtain the first feature of the first modality image;

[0007] The second modality image is input from the attention layer and fine-tuned using LoRA to obtain the first feature of the second modality image;

[0008] The first feature of the first modality image is input into the feedforward network layer, and through the adapter, the second feature of the first modality image is obtained.

[0009] The first feature of the second modality image is input into the feedforward network layer, and through the adapter, the second feature of the second modality image is obtained.

[0010] The second features of the first modality image and the second features of the second modality image are input into the first model of the adaptive local-global feature fusion model and concatenated to obtain the first concatenated feature;

[0011] The first concatenated feature is input into the multi-kernel convolutional block of the first model of the adaptive local-global feature fusion model. The convolutional kernels process the first concatenated feature in parallel and output the first feature of the first concatenated feature of the convolutional kernel. The first feature of the first concatenated feature is concatenated to obtain the second concatenated feature.

[0012] The first splicing feature is used to obtain the second feature of the first splicing feature through the efficient attention layer and feedforward network layer of the first model of the adaptive local-global feature fusion model;

[0013] The second concatenated feature and the second feature of the first concatenated feature are concatenated to obtain the third concatenated feature. Based on the third concatenated feature, the spatial attention map is obtained. ;

[0014] Combine the second splicing feature with the spatial attention map Multiplying them yields the first feature of the second concatenation feature. The second feature of the first concatenation feature is then multiplied by (1-...). Multiplying them together yields the third feature of the first concatenation feature;

[0015] The second splicing feature is added to the first feature and the third feature to obtain the fourth splicing feature. The biological image segmentation prediction map is obtained based on the fourth splicing feature.

[0016] The first modal image includes RGB images of marine organisms, and the second modal image includes depth maps or optical flow maps of marine organisms.

[0017] A marine organism image segmentation method according to some embodiments of this application, wherein a predicted organism image segmentation map is obtained based on a fourth stitching feature, including...

[0018] The fourth concatenated feature and the first output of the second model of the previous adaptive local-global feature fusion model are concatenated with the second model of the adaptive local-global feature fusion model to obtain the fifth concatenated feature.

[0019] The fifth concatenated feature is input into the multi-kernel convolutional block of the second model of the adaptive local-global feature fusion model. The convolutional kernels process the fifth concatenated feature in parallel and output the first feature of the fifth concatenated feature of the convolutional kernel. The first feature of the fifth concatenated feature is concatenated to obtain the sixth concatenated feature.

[0020] The fifth concatenation feature is obtained by using the efficient attention layer and feedforward network layer of the second model of the adaptive local-global feature fusion model to obtain the second feature of the fifth concatenation feature;

[0021] The sixth concatenated feature and the second feature of the fifth concatenated feature are concatenated to obtain the seventh concatenated feature. Based on the seventh concatenated feature, the spatial attention map is obtained. ;

[0022] The sixth stitching feature is combined with the spatial attention map. Multiplying them yields the first feature of the sixth concatenation feature. The second feature of the fifth concatenation feature is then multiplied by (1- Multiplying them together yields the third feature of the fifth splicing feature;

[0023] The first feature of the sixth splicing feature is added to the third feature of the fifth splicing feature to obtain the eighth splicing feature. The biological image segmentation prediction map is obtained based on the eighth splicing feature.

[0024] The marine organism image segmentation method according to some embodiments of this application also includes

[0025] The eighth concatenated feature of the second output of the second model of the adaptive local-global feature fusion model at each level is upsampled and convolved in parallel to obtain the intermediate prediction maps.

[0026] All intermediate prediction maps are input into the same convolutional layer for convolution operation, and the final biological image segmentation prediction map is output.

[0027] According to some embodiments of the present application, a marine organism image segmentation method is used, the method being based on a network model of multimodal feature extraction and adaptive local-global feature fusion, wherein the network model includes:

[0028] The SAM encoder includes a Transformer layer, which includes a self-attention layer, a feedforward network layer, a separable LoRA, and a separable adapter.

[0029] Neck module: Modal image input to neck module, output of second feature of modal image;

[0030] An adaptive local-global feature fusion model, including multi-kernel convolutional blocks, efficient attention layers, and feedforward network layers;

[0031] The Transformer layer includes a parameter-freezing self-attention layer and a parameter-freezing feedforward network layer.

[0032] Among them, a multi-kernel convolution block includes several convolution kernels of different sizes;

[0033] The adaptive local-global feature fusion model includes a first model and a second model.

[0034] According to some embodiments of the present application, a marine organism image segmentation method is provided, wherein the SAM encoder includes at least two Transformer blocks, each block including at least one Transformer layer;

[0035] The adaptive local-global feature fusion network model includes at least three types.

[0036] According to some embodiments of the marine organism image segmentation method of this application, the SAM encoder includes four Transformer blocks, each block including three Transformer layers;

[0037] There are five network models that adapt to local-global feature fusion.

[0038] According to some embodiments of the marine organism image segmentation method of this application, the Transformer layer of the SAM encoder includes a first input, a first output, and a second output;

[0039] The first input to the Transformer layer is the first output of the previous Transformer layer. The first output and the second output are the second features of the modality image. The input to the first-level Transformer layer is the modality image, and the last-level Transformer layer only includes the second output.

[0040] According to some embodiments of the marine organism image segmentation method of this application, the first model of the adaptive local-global feature fusion model includes a first input, a second input, and a first output; the second model includes a first input, a second input, a first output, and a second output.

[0041] The first input of the first model is the second feature of the first modality image, the second input is the second feature of the second modality image, and the first output is the fourth concatenated feature;

[0042] The first input of the second model is the first output of the first model of the adaptive local-global feature fusion model at this level, and the second input is the first output of the second model of the previous adaptive local-global feature fusion model. The first output and the second output are the eighth concatenated features.

[0043] The first-level adaptive local-global feature fusion model includes only the first model, and the first output is used as the second input of the second model of the second-level adaptive local-global feature fusion model. The first input of the first model of the first-level adaptive local-global feature fusion model is the second feature of the first modality image output by the neck module, and the second input is the second feature of the second modality image output by the neck module.

[0044] Among them, for other levels of adaptive local-global feature fusion models, the first input of the first model of the adaptive local-global feature fusion model at this level is the second feature of the first modal image of the second output of the last Transformer layer of the Transformer block at the corresponding level, and the second input is the second feature of the second modal image of the second output of the last Transformer layer of the Transformer block at the corresponding level; wherein, the first level Transformer block corresponds to the last level of the adaptive local-global feature fusion model.

[0045] In this model, the second model of the final stage adaptive local-global feature fusion model only includes the second output.

[0046] According to some embodiments of the marine organism image segmentation method of this application, wherein the first model of the adaptive local-global feature fusion module is:

[0047] First splicing feature :

[0048] = ( )

[0049] In the formula, This represents the second feature of the first modality image. This represents the second feature of the second modality image. Indicates splicing;

[0050] The first feature of the first splicing feature :

[0051]

[0052] In the formula, GULE() represents the activation function. Represents depthwise convolution. This represents pointwise convolution. Indicates the convolution operation;

[0053] Second splicing feature :

[0054]

[0055] In the formula, Indicates the output of the convolution kernel. Indicates the size of the convolution kernel;

[0056] The second feature of the first splicing feature :

[0057] =FFN( )

[0058]

[0059] In the formula, Indicates a query along the EA attention mechanism The softmax function is applied to each row. Indicates the key along the EA attention mechanism The softmax function is applied to each column, where V represents the value, where... and From input features Obtained from.

[0060] Third splicing feature :

[0061]

[0062] Spatial attention map :

[0063]

[0064] In the formula, σ represents the sigmoid function. It is a 1×1 kernel convolutional layer;

[0065] Fourth splicing feature :

[0066] .

[0067] According to some embodiments of the marine organism image segmentation method of this application, the second model of the adaptive local-global feature fusion module is as follows:

[0068] Fifth splicing feature

[0069] '= ( )

[0070] In the formula, This represents the first output of the second model of the preceding adaptive local-global feature fusion model;

[0071] The first feature of the fifth splicing feature :

[0072]

[0073] Sixth splicing feature :

[0074]

[0075] The second feature of the fifth splicing feature ';

[0076] '=FFN( )

[0077]

[0078] Seventh splicing feature :

[0079]

[0080] Spatial attention map :

[0081]

[0082] Eighth splicing feature :

[0083] .

[0084] In a second aspect, an electronic device is also provided, comprising: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the electronic device to perform the method described in the first aspect above.

[0085] Thirdly, a computer-readable storage medium is also provided, which stores a computer program that, when run on a computer, causes the computer to perform the method described in the first aspect above.

[0086] Beneficial Effects: This invention proposes a marine biological image segmentation device and method, including a multimodal feature extraction and adaptive local-global feature fusion structure. This framework can be applied to various modal combinations, such as RGB-D (RGB + depth map) or RGB-O (RGB + optical flow map). The framework of this invention comprises two main parts: (1) a unified multimodal feature extraction module; and (2) an adaptive local-global feature fusion module (ALGFF). Firstly, this invention uses SAM as the backbone network and employs parallel LoRA and adapters to extract features simultaneously from RGB and auxiliary information (depth map or optical flow map). Secondly, this invention handles multimodal and multi-scale feature fusion in a concise and unified manner. Specifically, this invention first performs a coarse fusion of the two features, and then extracts local and global information through two branches, utilizing multi-kernel convolutional blocks and Transformer layers respectively. These two types of information are combined with a dynamic spatial attention map to enhance the fused features, enabling them to adapt to the fusion of different features, thereby enhancing the segmentation capability of marine targets. Experimental results show that the proposed method achieves state-of-the-art performance on three datasets, demonstrating its efficiency and effectiveness.

[0087] The experiment also showed that, Figure 3In the overall structure segmentation task (lines 1-2), the proposed method can accurately capture the global morphological features of the target object, effectively avoiding the structural breakage or contour offset problems common in other methods. For the segmentation of camouflaged marine organisms (lines 3-5), even if the target organism has a high visual similarity with the surrounding marine environment, this method can still accurately distinguish the target from the background, successfully solving the problem of target omission or missegmentation in camouflaged scenarios. In the fine-grained boundary segmentation task (lines 6-7), the proposed method has more delicate details of the target edge, and can achieve clear and continuous boundary segmentation, which is significantly better than the blurring effect of other methods at the boundary.

[0088] Compared with other methods, the segmentation results proposed in this invention are clearer overall and have higher target integrity. Even when faced with complex images with rich details, this method can still stably output high-quality segmentation results, effectively overcoming the limitations of complex scenes on segmentation performance.

[0089] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0090] Figure 1 The overall structure of the proposed framework.

[0091] Figure 2 The Adaptive Local-Global Feature Fusion Module (ALGFF) structure, (a) with the multimodal-multiscale fusion process (b) ALGFF contains two branches, including the fifth, middle and first adaptive local-global feature fusion modules.

[0092] Figure 3 Visual comparison of different methods for predicting segmentation. Detailed Implementation

[0093] Marine organism segmentation aims to segment marine life in complex marine scenes and is of great significance in underwater intelligent research. Due to the complexity of underwater scenes, relying solely on a single RGB image or learning only from specific combinations of multimodal information may not achieve ideal results. Therefore, this invention proposes a unified multimodal feature extraction and adaptive feature fusion structure for RGB-X marine target segmentation. This structure is applicable to various marine organism segmentation scenarios, such as RGB-D (RGB + depth map) and RGB-O (RGB + optical flow map). Specifically, we first fine-tune the SAM encoder through parallel LoRa and adapters to extract RGB information and auxiliary information separately. Subsequently, we propose an adaptive local-global feature fusion (ALGFF) module to gradually fuse multimodal and multi-scale features in a simple and dynamic manner. Experimental results on RGB-D and RGB-O datasets show that the model performs excellently in underwater scene segmentation tasks.

[0094] Marine Target Segmentation: Marine target segmentation faces numerous challenges due to low visibility and the presence of particulate matter in underwater environments. Historically, many methods employed hand-designed features. Initially, Convolutional Neural Networks (CNNs) were the preferred model for marine target segmentation due to their superior performance in extracting information at different levels. Existing techniques have improved model performance in complex underwater environments through transfer learning and image restoration. This paper utilizes CNNs for color balancing and dehazing of degraded underwater images. Adaptive optical flow selection is used to extract motion information from video sequences and is combined with CNNs for fish segmentation. Despite achieving good segmentation results, CNNs have limitations in capturing global contextual information and long-range dependencies.

[0095] RGB+X Object Detection: Many methods use only RGB information for extraction during segmentation. To further utilize this information and achieve the desired segmentation effect, this invention combines RGB with auxiliary information to improve the accuracy of underwater environment segmentation. Depth and / or optical flow information will be used as auxiliary information to enhance the accuracy of underwater environment segmentation.

[0096] This disclosure proposes a marine organism image segmentation method based on a segmentation device that includes a parallel feature extraction module and an adaptive local-global feature fusion module.

[0097] A parallel feature extraction module extracts multi-scale features from RGB and auxiliary information, implemented using a SAM encoder with parallel LoRA and an adapter. An adaptive local-global feature fusion module dynamically fuses multi-scale and multi-modal features in a unified structure to generate a prediction map. These key components will be described in detail below. The overall structure is as follows: Figure 1 As shown.

[0098] Figure 1 The overall structure of the proposed framework is shown. It consists of two main parts: parallel feature extraction and adaptive local-global feature fusion. Parallel feature extraction is a SAM encoder with parallel LoRA and an adapter, used to extract RGB multi-scale features and auxiliary information. Adaptive local-global feature fusion realizes multi-scale feature fusion and multi-modal feature fusion. The PP module is used for progressive prediction.

[0099] Parallel feature extraction: Input RGB image and auxiliary information images (Depth information or optical flow information), this invention aims to obtain relevant multi-scale and multi-modal features. and As mentioned earlier, the Segmentation All Model (SAM) exhibits excellent performance in conventional segmentation tasks. However, the marine environment is complex, and marine objects possess unique characteristics in terms of shape, texture, and lighting conditions, making SAM potentially unsuitable for direct application in marine target segmentation. In image segmentation, combining RGB information with auxiliary information (such as depth information and optical flow information) often significantly improves segmentation results and accuracy. For marine target segmentation, RGB information provides basic target features, such as color and appearance, while auxiliary information (here referred to as X, representing depth information or optical flow information) supplements key information about the target's spatial location and motion state. RGB and X information form a complementary relationship at the feature level. Based on the above, this invention aims to extract effective features from two different modalities (RGB data and X data) as efficiently as possible without altering the overall structure of SAM.

[0100] To achieve this goal, this invention performs efficient fine-tuning of the SAM (Separate Aperture Model) to better adapt it to ocean target segmentation tasks, thereby improving segmentation accuracy and effectiveness. For example... Figure 1 As shown, this invention freezes the parameters of the SAM encoder and trains only the parallel LoRA and adapter, enabling efficient fine-tuning of the SAM encoder with a small number of parameters.

[0101] The SAM encoder contains four blocks with Transformer layers, each block containing three Transformer layers, which can be used to extract features at four scales for one modality. Features from layers 3, 6, 9, and 12 are extracted, which are the features of the last layer of each block. Each Transformer layer contains a self-attention (SA) layer and a feedforward network (FFN) layer.

[0102] First, this invention applies LoRA to the SA layer, which updates parameters through low-rank matrix simulation to reduce computational load and memory consumption. Its expression is as follows:

[0103] (1)

[0104] in, This represents the original pre-trained weight matrix. It is a low-rank, dimension-reduced matrix. It is a low-rank, high-dimensional matrix. This represents the input feature vector. This indicates the output feature vector.

[0105] In this invention, it is used to update the queries and keys of the self-attention layer in the SAM encoder to extract fine-grained global relationships in ocean images, thereby producing a more accurate representation. The k parameter used in the Lora implementation is the parameter of the original SAM model and has been frozen, so it does not participate in training updates. Only the q and v parameters have trainable incremental parts added through the Lora module.

[0106] Specifically, for the first Input features of each encoder block and This invention applies the shared parameter W of the SA layer. V Get the values ​​of RGB and X information , This invention utilizes shared freeze parameters in the SA layer. Get query and key .

[0107] Based on the separable learning LoRA, fine values ​​are generated for the inputs using formula (1). , Query and key .

[0108] The SA layer outputs for both modes can be represented as follows:

[0109] (2)

[0110] in, This represents a self-attention mechanism applied to multimodal inputs (RGB and auxiliary modal information). Through LoRA fine-tuning, SA enables the model to capture long-range spatial dependencies in the image, thereby improving segmentation accuracy.

[0111] Then, this invention applies an adapter to the FFN layer. The adapter inserts a small number of trainable parameters into the original structure, enabling it to efficiently adapt to new tasks. It is described below:

[0112]

[0113] in, and These are the weights of the two linear projections. To save computational costs, Typically much smaller than the input features The dimension. Here, we take the first... Taking the FFN layer of an encoder block as an example, consider the input features from two modal SA layers. and This invention uses two different adapters and a frozen FFN layer to obtain the final feature of the i-th scale after certain optimization. The process is described as follows:

[0114]

[0115] in, , These represent adapters for RGB and auxiliary data, respectively. Through specific LoRA and adapters, this invention is able to obtain features from four encoder blocks. and .

[0116] Highest-scale features and Generated by a neck module in the SAM encoder.

[0117] Both LoRA and the adapter use very few parameters. This allows for effective parameter fine-tuning, making the pre-trained SAM encoder more suitable for marine environments. The improved encoder is used to extract multi-scale RGB and auxiliary information, which is progressively fused in the decoder section.

[0118] Adaptive Local-Global Feature Fusion: Due to the complexity of underwater scenes, adaptively fusing multimodal and multi-scale information is crucial for improving robustness and accuracy. Previous research has shown that CNNs efficiently extract local information, while Transformers excel at capturing global information; the two can complement each other. In this invention, a simple yet effective module called the Adaptive Local-Global Feature Fusion Module (ALGFF) is proposed.

[0119] like Figure 2 As shown in (a), with any two features and As input, ALGFF first obtains a rough combination of these two features by concatenation. Then, it employs a flexible structure, simultaneously extracting and adaptively combining... It incorporates local and global information to enhance the fusion process. This structure is highly adaptable to various scenarios. It is suitable not only for the fusion of multi-scale features but also for the fusion of multiple modal combinations such as RGB-D and RGB-O. It also simplifies the decoder structure, avoiding the complex structures used in existing methods.

[0120] The ALGFF proposed in this invention comprises two branches: a local branch and a global branch. The adaptive local-global feature fusion module consists of two identical ALGFFs. Please refer to [link / reference needed]. Figure 1 Features are obtained from the four encoder blocks of this invention. and and the neck module obtains features and It has a corresponding input relationship with the adaptive local-global feature fusion module. Figure 1 The sequence of encoder blocks is called a stage, and the order of stages is determined by the input order. yes The preceding part. The input adaptive local-global feature fusion module is defined as ALGF .

[0121] Please refer to Figure 1 and Figure 2 (b) The two ALGFFs of the adaptive local-global feature fusion module, such as the input of the fifth adaptive local-global feature fusion module, are , , Figure 2 (b) shows that the fifth adaptive local-global feature fusion module has only one ALGFF. The other adaptive local-global feature fusion modules have two ALGFFs.

[0122] Specifically, for the first ALGFF of the adaptive local-global feature fusion module:

[0123] Local branches primarily utilize CNN layers to extract information from adjacent regions. To extract local information at different levels, this invention employs an improved convolutional variant called a multi-kernel CNN (MK-CNN) block. It contains kernels of 3×3, 5×5, 7×7, and 9×9 sizes, which are processed in parallel to capture details at different levels. Simultaneously, to maintain the efficiency of the proposed module, this invention uses depthwise separable convolutions (depth and point convolutions) to significantly reduce the number of parameters. Therefore, it ensures an expanded receptive field, and its computational complexity is similar to standard convolutions, without introducing additional computational overhead or a larger number of parameters. Specifically, for the coarsely fused features of the input... The output of the local branch is:

[0124] = ( )

[0125]

[0126]

[0127] in, It is the output of each CNN kernel. It is the size of the convolution kernel. It's a convolution operation. and It is depthwise separable convolution, including depthwise convolution. Pointwise convolution GULE() represents the activation function. This represents a convolution operation. Through this design, the model can effectively extract multi-level local information with low computational cost.

[0128] The global branch is used to extract long-distance relationships between any two locations in the input features. The Transformer's self-attention layer is well-suited for this purpose, but it significantly increases computational cost when multiplying queries and keys. To avoid this problem, this invention employs an efficient attention layer (EA) and an FFN layer. For the efficient attention layer (EA), please refer to Shen, Z.; Zhang, M.; Zhao, H.; Yi, S.; Li, H. Efficient attention: Attention with linear complexities. In Proceedings of the Proceedings of the IEEE / CVF winter conference on applications of computer vision, 2021, pp. 3531–3539.

[0129] The efficient attention layer optimizes the traditional dot product attention method by multiplying the key and value. Specifically, it first computes... To obtain a size of The matrix (where c represents the number of manually configurable channels) is then multiplied by the query. This reduces the computational complexity from... Reduce to Where n is the number of pixels in the feature, while maintaining equivalence with traditional attention mechanisms. The FFN layer, based on the features provided by the attention mechanism, uses non-linear transformations and dimensionality operations to deeply explore and enhance the feature representation, enabling the model to learn more complex patterns and abstract concepts.

[0130]

[0131] in, and These represent queries along the EA attention mechanism. Each line and the key to the EA attention mechanism The softmax function is applied to each column. Among them, and From input features Obtained from.

[0132] Next, we go through the FFN layer:

[0133] =FFN( )

[0134] The outputs of the two branches can be described as follows:

[0135]

[0136]

[0137] in, This represents the overall calculation of a local branch. This represents the overall computation of the global branch. Then, local and global information are fused using a dynamically learned spatial attention map.

[0138]

[0139]

[0140]

[0141] Where σ is the sigmoid function, It is a 1×1 kernel convolutional layer. It's a spatial attention map, so local and global information at each pixel can be fused with different weights. It varies with location and input, allowing the network to assign more weight to local details or global context as needed. This design allows the module to adapt to various fusion scenarios.

[0142] Specifically, for the second ALGFF of the adaptive local-global feature fusion module:

[0143] The output of the local branch is:

[0144] = ( )

[0145]

[0146]

[0147] in, It is the output of each CNN kernel. It is the size of the convolution kernel. It's a convolution operation. and It is depthwise separable convolution, including depthwise convolution. Pointwise convolution GULE() represents the activation function. This indicates a convolution operation.

[0148] The output of the global branch is:

[0149]

[0150] in, and These represent queries along the EA attention mechanism. Each line and the key to the EA attention mechanism The softmax function is applied to each column. Among them, and From input features Obtained from.

[0151] Next, we go through the FFN layer:

[0152] =FFN( )

[0153] The outputs of the two branches can be described as follows:

[0154]

[0155]

[0156] in, This represents the overall calculation of a local branch. This represents the overall computation of the global branch. Then, local and global information are fused using a dynamically learned spatial attention map.

[0157]

[0158]

[0159]

[0160] Where σ is the sigmoid function, It is a 1×1 kernel convolutional layer. It's a spatial attention map, so local and global information at each pixel can be fused with different weights. It varies with location and input, allowing the network to assign more weight to local details or global context as needed. This design allows the module to adapt to various fusion scenarios.

[0161] As described above, the first ALGFF is used to progressively achieve the fusion of multimodal and multi-scale features from coarse to fine, such as... Figure 2 As shown in (b), when processing the i-th scale, this invention first fuses multimodal features of the same scale. That is, using... and X and Y are used as inputs to the ALGFF module to obtain multimodal fusion features. As shown below:

[0162]

[0163] The second ALGFF combines multi-scale feature fusion with the structure of the ALGFF module. Since the feature fusion process is performed from coarse to fine, multi-scale feature fusion involves fusion features from larger scales. This means that to obtain the final fusion features at that scale... As shown below:

[0164]

[0165] Note that when i=5, since there are no larger-scale features, only multimodal fusion is performed, not multiscale fusion. Through these processing steps, the model of this invention can segment images more accurately.

[0166] Progressive Prediction: Finally, this invention implements progressive prediction. Each extracted feature, after passing through the ALGFF module, generates an intermediate prediction map. This invention adjusts the size of the intermediate prediction maps through upsampling and convolution. Finally, this invention uses a 1×1 convolution to combine all intermediate prediction maps to generate the final prediction map. The process is as follows:

[0167]

[0168]

[0169] in, Represents a 1×1 convolution. It's upsampling. It is the first A prediction mask. This is the final generated prediction image. The proposed ALGFF module optimizes information transfer between different layers through progressive feature fusion and retains useful information to the maximum extent. Adaptive spatial fusion operation solves the information conflict problem in the feature fusion process. This invention uses a pyramid structure and the proposed ALGFF module to effectively utilize RGB and auxiliary information, making underwater segmentation more efficient and precise.

[0170] During the training phase, this invention employs three commonly used loss functions: Binary Cross-Entropy (BCE) loss, SSIM loss, and IoU loss. The BCE loss function compares the predicted probability of each pixel with the ground truth label to optimize foreground / background segmentation. The SSIM loss function enhances the similarity between the predicted image details and the ground truth image in terms of brightness, contrast, and structure. The IoU loss function directly improves localization accuracy by calculating the overlap rate between the predicted and ground truth bounding boxes. Each intermediate prediction image in this invention corresponds to a loss function operation. Finally, this invention sums the loss functions and calculates the average to obtain the final loss function.

[0171]

[0172]

[0173] in, This represents the loss for the i-th level prediction. This represents the final predicted loss.

[0174] experiment:

[0175] 1. Dataset and Evaluation Metrics

[0176] To demonstrate that the proposed structure is applicable to both RGB+D and RGB+O marine object detection, this invention utilizes three commonly used datasets and four metrics. The RMAs dataset contains 3014 marine images featuring marine animals. The Mask3k dataset, focused on marine animal segmentation, contains 3103 images, 193 of which are background images. The scene backgrounds in both datasets are complex and noisy, and depth maps help distinguish the foreground. Additionally, the DeepFish dataset contains videos of 72 species of underwater fish, where optical flow aids in detecting moving objects.

[0177] In this invention, the same structure is used to train both RGB-D and RGB-O marine target detection models. To train the RGB-D marine target detection model on the RMAs dataset, 1769 images were used for training and 1141 images for testing. To train the RGB-D marine target detection model on the Mask3k dataset, 2514 images were used for training and 500 images for testing. Furthermore, to train the RGB-F marine target detection model on the DeepFish dataset, 3107 images were used for training and 609 images for testing.

[0178] This invention uses the following six metrics to evaluate the model's performance: structural similarity metric (Sα), average augmented alignment metric (Em), mean absolute error (MAE), maximum F-score (Fm), number of parameters, and number of floating-point operations (FLOPs). These metrics provide a more comprehensive evaluation and capture various aspects of segmentation quality.

[0179] 2. Implementation Details

[0180] The model of this invention was tested using the PyTorch toolkit and an RTX 4090 GPU. The model initializes the SAM encoder from a pre-trained SAM-B dataset; the initialization of other modules is randomized. During training, the SAM encoder is frozen, and only the remaining modules are fine-tuned. The AdamW optimizer is used to update parameters. The initial learning rate is set to 0.001, and the weight decay is set to 0.1. The learning rate is reduced by a factor of 10 every 22 epochs. The total number of training epochs is set to 50. The batch size is set to 6. The input image size is 512×512.

[0181] 3. Comparison with state-of-the-art methods

[0182] In this section, the method of this invention is compared with other methods on three common datasets. The comparisons are based on previous methods, namely TFL-Net, MAS-SAM, TC-USOD, and Dual-SAM. Both qualitative and quantitative results demonstrate that the method of this invention has significant advantages.

[0183] Quantitative Comparison. Table 1 describes the quantitative comparison results with other methods. Compared with other RGBD methods, the method of this invention outperforms these methods in all four metrics across three datasets. On the large-scale dataset RMAS, the metrics of this invention are generally higher than other methods, with improvements across all metrics. In dual-input scenarios, the model of this invention has fewer trainable parameters and lower floating-point operations (FLOPs) compared to other dual-input models. Compared with single-input models, the model of this invention achieves a significant performance improvement without a significant increase in the number of parameters.

[0184] As shown in Table 1, TFL-Net has the fewest parameters. However, since TFL-Net is not designed for underwater scenes, its overall performance is not ideal. Although the USOD10K dataset targets underwater scenes, the Transformer-based methods on this dataset do not use SAM (SegmentAnythingModel), resulting in relatively poor segmentation performance. While SAM-MAS is designed for underwater scenes and uses SAM, it is a single-input model, leading to relatively poor overall performance. The method of this invention was also compared with Dual-SAM, which uses RGB-D (RGB + depth map) and RGB-G (RGB + gamma) as inputs. Because Dual-SAM uses two SAM encoders in the encoder part and has a relatively complex decoder structure, its floating-point operations (FLOPs) and number of parameters are relatively high. In contrast, the model structure of this invention is simpler and more efficient, showing superior performance compared to other methods. Furthermore, on three other datasets, the method of this invention also shows significant improvements in various metrics. In summary, the method of this invention improves the segmentation capability of the model and achieves significant improvements compared with other methods.

[0185] Table 1. Performance comparison on RMAS, MASK3k, and DeepFish

[0186]

[0187] Qualitative comparison. To intuitively verify the segmentation effectiveness of the proposed method, this invention... Figure 3The paper presents qualitative comparison examples of the proposed method with other mainstream methods, and the results fully highlight the significant advantages of the proposed method. Specifically, in the overall structure segmentation task (lines 1-2), the proposed method can accurately capture the global morphological features of the target object, effectively avoiding the structural breakage or contour offset problems common in other methods; for the camouflaged marine organism segmentation scene (lines 3-5), even if the target organism has a high visual similarity with the surrounding marine environment, the proposed method can still accurately distinguish the target from the background, successfully solving the problem of target under-detection or mis-segmentation in camouflaged scenes; in the fine-grained boundary segmentation task (lines 6-7), the proposed method has more delicate characterization of the target edge, and can achieve clear and continuous boundary segmentation, which is significantly better than the blurring effect of other methods at the boundary.

[0188] Compared with other methods, the proposed method produces clearer segmentation results and higher target integrity. Even when faced with complex images with rich details, this method can still consistently output high-quality segmentation results, effectively overcoming the limitations of complex scenes on segmentation performance. Overall, the proposed method achieves a significant improvement in segmentation performance. For example, the invention introduces effective auxiliary information highly adapted to the task in the method design. This auxiliary information can provide the segmentation model with more accurate target localization clues and feature discrimination criteria, thereby greatly improving the model's ability to handle complex scenes, camouflaged targets, and fine-grained boundaries.

[0189] 4 Ablation Studies

[0190] In this section, the invention experimentally verifies the effectiveness of different modules in the model. The invention reports results on the RMA dataset; other datasets also show similar trends.

[0191] The effect of dual-branch architecture. In this work, the present invention fine-tunes the encoder portion of the SAM by employing parallel LoRA and adapters, and utilizes auxiliary information. Table 2 shows the effect of dual-branch architecture. It can be observed that the model with auxiliary information achieves better performance on all metrics compared to the single-branch model. This demonstrates the effectiveness of the RGB+X architecture of the present invention in feature extraction.

[0192] Table 2. Performance of the dual-branch system

[0193]

[0194] The Effects of Multimodal and Multiscale Information. In this work, the invention explores the impact of multimodal and multiscale information on the model. As shown in Table 3, all metrics are significantly low when using only the baseline model. Incorporating multimodal information improves segmentation performance because RGB and auxiliary information complement each other. Further addition of multiscale information enhances model performance by utilizing information from both modalities. This method integrates the features of multimodal and multiscale information, making it highly suitable for segmentation tasks involving marine organisms.

[0195] Table 3. Performance effects of multimodal and multiscale information.

[0196]

[0197] The effect of ALGFF. In this work, the present invention uses the ALGFF (Adaptive Local-Global Feature Fusion) module for multi-scale feature fusion and multi-modal fusion. As shown in Table 4, the present invention explores the influence of different factors within the ALGFF module. It can be observed that the performance is worst when the ALGFF module is not used, with all index values ​​generally being low. When the local branch is incorporated, this branch processes and captures features of different scales in the image in parallel through convolutional kernels of different sizes, while taking into account both detailed and global information. The results in the table show that the segmentation performance is significantly improved after adding the local branch. In addition, after introducing adaptive fusion: in the multi-modal fusion process, it dynamically adjusts the weights and fusion strategies of different modal features to solve problems such as noise interference and modality loss; in the multi-scale fusion process, adaptive fusion dynamically adjusts the weights and interaction methods of features of different scales to solve problems such as semantic gaps between multi-scale features. According to the data in the table, all indexes show significant improvement after incorporating adaptive fusion. Obviously, the model performance is greatly enhanced after using the ALGFF module.

[0198] Table 4. Performance comparison using different modules.

[0199]

[0200] This invention proposes a novel feature learning framework for marine target segmentation. The framework uses SAM as the backbone network and fine-tunes it with LoRA and an adapter to extract RGB, depth, and optical flow features. In the decoding part, combining the advantages of CNN and Transformer, an Adaptive Local-Global Feature Fusion (ALGFF) module is proposed. Through multimodal fusion and multi-scale feature processing, more efficient autonomous fusion can be achieved. The ALGFF module is used to process multi-scale and multimodal features. Experiments on three databases demonstrate the effectiveness of the proposed method.

[0201] Based on the above embodiments, this application also provides a computer program that, when run on a computer, causes the computer to execute the methods provided in the above embodiments.

[0202] Based on the above embodiments, this application also provides a computer storage medium storing a computer program, which, when executed by a computer, causes the computer to perform the method provided in the above embodiments.

[0203] The storage medium can be any available medium that a computer can access. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0204] Based on the above embodiments, this application also provides a chip for reading a computer program stored in a memory and implementing the method provided in the above embodiments.

[0205] Based on the above embodiments, this application provides a computer program product that implements the methods provided in the above embodiments when the computer program product is run on an electronic device.

[0206] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0207] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0210] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for segmenting marine organism images, characterized in that, The method is based on a network model for multimodal feature extraction and adaptive local-global feature fusion, wherein the network model includes: The SAM encoder includes a Transformer layer, which includes a self-attention layer, a feedforward network layer, a separable LoRA, and a separable adapter. Neck module: Modal image input to neck module, output of second feature of modal image; An adaptive local-global feature fusion model, including multi-kernel convolutional blocks, efficient attention layers, and feedforward network layers; The Transformer layer includes a parameter-freezing self-attention layer and a parameter-freezing feedforward network layer. Among them, a multi-kernel convolution block includes several convolution kernels of different sizes; The adaptive local-global feature fusion model includes a first model and a second model. The marine organism image segmentation method includes The first modality image is input from the attention layer and fine-tuned using LoRA to obtain the first feature of the first modality image; The second modality image is input from the attention layer and fine-tuned using LoRA to obtain the first feature of the second modality image; The first feature of the first modality image is input into the feedforward network layer, and through the adapter, the second feature of the first modality image is obtained. The first feature of the second modality image is input into the feedforward network layer, and through the adapter, the second feature of the second modality image is obtained. The second features of the first modality image and the second features of the second modality image are input into the first model of the adaptive local-global feature fusion model and concatenated to obtain the first concatenated feature; The first concatenated feature is input into the multi-kernel convolutional block of the first model of the adaptive local-global feature fusion model. The convolutional kernels process the first concatenated feature in parallel and output the first feature of the first concatenated feature of the convolutional kernel. The first feature of the first concatenated feature is concatenated to obtain the second concatenated feature. The first splicing feature is used to obtain the second feature of the first splicing feature through the efficient attention layer and feedforward network layer of the first model of the adaptive local-global feature fusion model; The second concatenated feature and the second feature of the first concatenated feature are concatenated to obtain the third concatenated feature. Based on the third concatenated feature, the spatial attention map is obtained. ; Combine the second splicing feature with the spatial attention map Multiplying them yields the first feature of the second concatenation feature. The second feature of the first concatenation feature is then multiplied by (1-...). Multiplying them together yields the third feature of the first concatenation feature; The second splicing feature is added to the first feature and the third feature to obtain the fourth splicing feature. The biological image segmentation prediction map is obtained based on the fourth splicing feature. The first modal image includes RGB images of marine organisms, and the second modal image includes depth maps or optical flow maps of marine organisms.

2. The marine organism image segmentation method according to claim 1, characterized in that, in, The biological image segmentation prediction map is obtained based on the fourth stitching feature, including The fourth concatenated feature and the first output of the second model of the previous adaptive local-global feature fusion model are concatenated with the second model of the adaptive local-global feature fusion model to obtain the fifth concatenated feature. The fifth concatenated feature is input into the multi-kernel convolutional block of the second model of the adaptive local-global feature fusion model. The convolutional kernels process the fifth concatenated feature in parallel and output the first feature of the fifth concatenated feature of the convolutional kernel. The first feature of the fifth concatenated feature is concatenated to obtain the sixth concatenated feature. The fifth concatenation feature is obtained by using the efficient attention layer and feedforward network layer of the second model of the adaptive local-global feature fusion model to obtain the second feature of the fifth concatenation feature; The sixth concatenated feature and the second feature of the fifth concatenated feature are concatenated to obtain the seventh concatenated feature. Based on the seventh concatenated feature, the spatial attention map is obtained. ; The sixth splicing feature is combined with the spatial attention map. Multiplying them yields the first feature of the sixth concatenation feature. The second feature of the fifth concatenation feature is then multiplied by (1- Multiplying them together yields the third feature of the fifth splicing feature; The first feature of the sixth splicing feature is added to the third feature of the fifth splicing feature to obtain the eighth splicing feature. The biological image segmentation prediction map is obtained based on the eighth splicing feature.

3. The marine organism image segmentation method according to claim 2, characterized in that, Also includes The eighth concatenated feature of the second output of the second model of the adaptive local-global feature fusion model at each level is upsampled and convolved in parallel to obtain the intermediate prediction maps. All intermediate prediction maps are input into the same convolutional layer for convolution operation, and the final biological image segmentation prediction map is output.

4. The marine organism image segmentation method according to claim 1, characterized in that, in, The SAM encoder includes at least two Transformer blocks, each of which includes at least one Transformer layer; The adaptive local-global feature fusion network model includes at least three types.

5. The marine organism image segmentation method according to claim 4, characterized in that, in, The SAM encoder consists of four Transformer blocks, each of which contains three Transformer layers; There are five network models that adapt to local-global feature fusion.

6. The marine organism image segmentation method according to claim 5, characterized in that, in, The Transformer layer of the SAM encoder includes a first input, a first output, and a second output; The first input to the Transformer layer is the first output of the previous Transformer layer. The first output and the second output are the second features of the modality image. The input to the first Transformer layer is the modality image, and the last Transformer layer only includes the second output. The adaptive local-global feature fusion model includes a first input, a second input, and a first output in its first model; and a second model includes a first input, a second input, a first output, and a second output. The first input of the first model is the second feature of the first modality image, the second input is the second feature of the second modality image, and the first output is the fourth concatenated feature; The first input of the second model is the first output of the first model of the adaptive local-global feature fusion model at this level, and the second input is the first output of the second model of the previous adaptive local-global feature fusion model. The first output and the second output are the eighth concatenated features. The first-level adaptive local-global feature fusion model includes only the first model, and the first output is used as the second input of the second model of the second-level adaptive local-global feature fusion model. The first input of the first model of the first-level adaptive local-global feature fusion model is the second feature of the first modality image output by the neck module, and the second input is the second feature of the second modality image output by the neck module. Among them, for other levels of adaptive local-global feature fusion models, the first input of the first model of the adaptive local-global feature fusion model at this level is the second feature of the first modal image of the second output of the last Transformer layer of the Transformer block at the corresponding level, and the second input is the second feature of the second modal image of the second output of the last Transformer layer of the Transformer block at the corresponding level; wherein, the first level Transformer block corresponds to the last level of the adaptive local-global feature fusion model. In this model, the second model of the final stage adaptive local-global feature fusion model only includes the second output.

7. The marine organism image segmentation method according to claim 6, characterized in that, in, The first model of the adaptive local-global feature fusion module: First splicing feature : = ( ) In the formula, This represents the second feature of the first modality image. This represents the second feature of the second modality image. Indicates splicing; The first feature of the first splicing feature : In the formula, GULE() represents the activation function. Represents depthwise convolution. This represents pointwise convolution. Indicates the convolution operation; Second splicing feature : In the formula, Indicates the output of the convolution kernel. Indicates the size of the convolution kernel; The second feature of the first splicing feature : In the formula, Indicates a query along the EA attention mechanism The softmax function is applied to each row. Keys along the EA attention mechanism The softmax function is applied to each column, where V represents the value, where... and From input features Obtained; Third splicing feature : Spatial attention map : In the formula, σ represents the sigmoid function. It is a 1×1 kernel convolutional layer; Fourth splicing feature : 。 8. The marine organism image segmentation method according to claim 7, characterized in that, in, The second model of the adaptive local-global feature fusion module: Fifth splicing feature ( ) In the formula, This represents the first output of the second model of the preceding adaptive local-global feature fusion model; The first feature of the fifth splicing feature : Sixth splicing feature : The second feature of the fifth splicing feature ; Seventh splicing feature : Spatial attention map : Eighth splicing feature : 。 9. A marine organism image segmentation device, characterized in that, include: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the apparatus to perform the method of any one of claims 1-8.

Images