SAM2 multi-task perception binary segmentation method based on hybrid expert adapter

Abstract

The invention discloses an SAM2 multi-task perception binary segmentation method based on a hybrid expert adapter. The method is specifically implemented according to the following steps: step 1, constructing a data set and an encoder; step 2, constructing a hybrid expert adapter module; and step 3, constructing a task awareness gating module. According to the method, a pre-trained SAM2 is taken as a main network, on the basis of freezing the main body weight, a standard adapter and a MoE-Adapter are respectively deployed on odd and even layers of an encoder, and a lightweight expert sub-network and a dynamic gating strategy are combined, so that unified processing of multiple tasks such as salient target detection, camouflage target identification, marine animal segmentation and the like is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-task binary segmentation methods, and in particular relates to a SAM2 multi-task perception binary segmentation method based on a hybrid expert adapter. Background Art

[0002] In the field of computer vision, image segmentation technology is the core foundation for object recognition and scene understanding. Binary segmentation tasks are widely used in many key scenarios, including medical image analysis, industrial defect detection, and autonomous driving. The Segment Anything Model (SAM) and its upgraded version, SAM2, have achieved significant breakthroughs in image segmentation thanks to their powerful zero-shot segmentation capabilities, enabling accurate segmentation of objects in various image types. However, SAM2 still has limitations in terms of task specificity and model generalization for diverse binary segmentation tasks.

[0003] Traditional multi-task segmentation methods often employ fixed network structures and parameter sharing mechanisms, making them difficult to adapt to the complex differences and dynamic changes between tasks. This leads to performance degradation and insufficient segmentation accuracy when handling diverse tasks. Furthermore, single models are prone to overfitting or underfitting when handling multiple tasks, failing to fully realize their potential. While the Mixture of Experts (MoE) mechanism can improve the model's ability to handle complex tasks to a certain extent, effectively combining it with SAM2 to achieve task-aware multi-task segmentation still presents challenges in adaptability and optimization. Summary of the Invention

[0004] The purpose of the present invention is to provide a SAM2 multi-task-aware binary segmentation method based on a hybrid expert adapter, which solves the problem that the prior art SAM2 model is prone to overfitting or underfitting when processing diverse binary segmentation tasks.

[0005] The technical solution adopted by the present invention is a binary segmentation method based on the SAM2 multi-task perception of a hybrid expert adapter, which is specifically implemented according to the following steps: Step 1: Build the dataset and encoder; Step 2, constructing a hybrid expert adapter module; Step 3: Build a task awareness gating module.

[0006] The present invention is also characterized in that: Step 1 is implemented as follows: Step 1.1, collect sample images covering various tasks; Step 1.2: Label the sample images collected in step 1.1 in labelme to generate a true image, where salient objects are white pixels and background is black pixels; Step 1.3: Divide all sample images and true value images into training sets and test sets according to the ratio; Step 1.4: Use the pre-trained SAM2 as the backbone network to extract image features and build the encoder; In step 1.5, the dataset image resolution is adjusted to 352×352 and input into the SAM2 backbone in the encoder to obtain convolutional multi-scale features.

[0007] Step 2 is implemented as follows: In step 2.1, we design multiple lightweight task-specific sub-networks as expert networks. For different binary segmentation tasks, we construct multiple simple feedforward neural networks as expert networks in the MoE-Adapter. Each expert network consists of several fully connected layers and is responsible for processing input features to capture unique representations for a specific task. Step 2.2: Create a library containing detailed text descriptions of each task. Use the CLIP text encoder to encode the task text to obtain the corresponding task feature vector. When the user enters the task text, the CLIP text encoder is also used to encode it into a feature vector. By calculating the cosine similarity between this vector and the feature vectors of each task in the task library, the best matching task is found. The calculation formula is as follows: (1) in, is the normalized feature of the user input text, is the normalized feature of the predefined task text, is the similarity score between the two, according to the formula (2) Select the corresponding task, then refine the matched task text features through a projection layer, convert the task ID into a one-hot encoded vector, and fuse the two using the alpha fusion mechanism. The formula is as follows: (3) Finally, the fused vector is normalized to obtain the gating weights used to control the expert network in the MoE-Adapter: wexperts=Norm(f gate)(4); Step 2.3: Implement the expert network output fusion and residual connection, and perform weighted summation on the outputs of each expert network according to the gating weights generated by the task-aware gating module, as shown in the formula: (5) in, is the fused output, is the gating weight of the k-th expert network, E k is the output of the k-th expert network, n is the number of expert networks. The fused features are then passed through a lightweight final fusion layer, which consists of a convolutional layer and an activation function to further enhance the compatibility of the features with the current task. Finally, the residual connection method is used to add the features processed by the fusion layer to the original features input to the MoE-Adapter. The formula is as follows: (6) This enables the model to retain important information in the original input features while learning task-specific features, improving the model's stability and learning effect, and effectively completing the forward transmission process of the MoE-Adapter.

[0008] Step 2.1 is implemented as follows: Step 2.1.1: Determine the number of expert networks in the MoE-Adapter based on the characteristics and complexity of different binary segmentation tasks. Each expert network adopts a feedforward neural network structure, consisting of an input layer, several hidden layers, and an output layer. Step 2.1.2, use the random initialization method to assign weights and biases to each expert network. For weights, use normal distribution for initialization, with the mean set to 0 and the standard deviation set to 0.01. The formula is ,in Represents the weights from layer i to layer j. For bias, all are initialized to 0, that is, b j =0, b j represents the bias of the jth layer.

[0009] Step 2.1.1 is implemented as follows: Step 2.1.1.1: Analyze multiple binary segmentation tasks, including salient object detection, camouflaged object detection, marine animal segmentation, mirror image detection, polyp segmentation, and shadow detection, to study the characteristic differences between the target object and the background in different tasks. In step 2.1.1.2, analyze the differences in the characteristics of each task in step 2.1.1.1 to determine the number of expert networks in the MoE-Adapter. If the characteristics of different tasks are significantly different, the number of expert networks can be appropriately increased to better capture the unique characteristics of each task.

[0010] Step 2.1.2 is implemented as follows: Step 2.1.2.1, select the initialization method. When initializing the expert network parameters, consider the impact of various initialization methods on model training and select the Kaiming initialization method to initialize the weights of the expert network; Step 2.1.2.2, perform parameter initialization operation. For each expert network, initialize the weights according to the selected Kaiming initialization method. Assume that the weight matrix of a layer of the expert network is W, and the number of input neurons in this layer is n int , the number of output neurons is n out , according to Kaiming initialization formula, weight W ij The initialization value has a mean of 0 and a standard deviation of The normal distribution of , for the bias, initialize them all to 0.

[0011] Step 2.2 is implemented as follows: Step 2.2.1: Build and encode the task text description library. Create a detailed task text description library for different binary segmentation tasks. Step 2.2.2, calculate the similarity and generate the gating weight. When the user enters the task text, the CLIP text encoder is also used to encode it into a feature vector f task,i , by calculating the vector and each task feature vector in the task feature library f task,j The cosine similarity is used to determine the most matching task. The cosine similarity calculation formula is as follows: (7) in, S (i,j) Represents the similarity score between user input text i and task j. From the similarity matrix S, the task with the highest score is selected as the matching result, that is, (8) After finding a matching task, the text feature vector of the task f task,j Feature refinement is performed through a projection layer, which consists of a fully connected layer and an activation function. f proj , and convert the task ID into a one-hot encoding vector f onehot , use alpha blending, use alpha blending mechanism to f onehot and fproj Fusion, the formula is as follows: (9) Finally, the fused vector f gate After normalization, the gating weights used to control the expert network in the MoE-Adapter are obtained. The formula is as follows: (10) In this way, the gating weights suitable for the current task are dynamically generated based on the task text entered by the user, enabling the MoE-Adapter to adaptively allocate resources and focus on feature learning of related tasks.

[0012] Step 2.2.1 is implemented as follows: Step 2.2.1.1: Collect and organize text descriptions for different tasks. For binary segmentation tasks such as salient object detection, camouflaged object detection, marine animal segmentation, mirror image detection, polyp segmentation, and shadow detection, collect relevant text description information. For each task, filter and organize the collected text descriptions, remove duplicate, ambiguous, or inaccurate content, and ensure that the text descriptions accurately and comprehensively cover the key features and objectives of the task. In step 2.2.1.2, use the CLIP text encoder to encode the task text. Input the organized text description of each task into the CLIP text encoder. During the encoding process, the text description set of each task is input into the CLIP text encoder in batches of a certain size. The CLIP text encoder first performs lexical analysis and tokenization on the input text, converting the text into a series of tokens. It then uses a multi-layer Transformer architecture to extract features and semantically fuse these tokens, ultimately outputting a fixed-dimensional vector representation.

[0013] Step 3 is implemented as follows: Step 3.1: Use the gating weights output by the task-aware gating module to regulate the hybrid expert adapter. W experts Input into the hybrid expert adapter, these gating weights will dynamically adjust the contribution of each expert network in the feature processing process according to different task requirements; Step 3.2: Decoding prediction based on multi-scale features First, the multi-scale features extracted by the encoder are processed to convert the features containing rich spatial information extracted in the early stage into f 1 is input to the spatial attention module, through which the spatial attention features are generated Sf, and its calculation formula is as follows: (11) Among them, sigmoid activation function, δ is ReLU activation function, and is a fully connected layer, then Sf Perform matrix multiplication with the feature flow of other layers to highlight the feature information of important areas in space. Then, the multiplied result is combined with the feature rich channel information extracted in the later stage. f 4. Splice in the channel dimension to obtain splicing features f concat , then f concat Double convolution operation is performed to further enhance spatial information. The convolution operation uses convolution kernels of different sizes. The formula is as follows: (12) After that, the convolutional features are input to the channel upsampling module and combined with the channel attention processed f The 4 features are spliced ​​again to enrich the channel information. Finally, the final fusion features are processed by a convolutional layer to obtain the prediction result. The convolution kernel size of the convolutional layer is set according to the task requirements to complete the binary segmentation prediction of the input image and output the prediction mask.

[0014] Step 3.1 is implemented as follows: Step 3.1.1: Receive the output of the task-aware gating module and obtain the gating weight generated by the task-aware gating module. W experts ,When obtaining the gating weights, ensure the accuracy and integrity of data transmission, and prevent the weight information from being lost or erroneous during the transmission process; Step 3.1.2, based on the gating weight regulation expert network, the received gating weight W experts Applied to each expert network in the MoE-Adapter, the output of each expert network is weighted according to the size of the gating weight.

[0015] The beneficial effects of the present invention are: This paper presents a SAM2 multi-task-aware binary segmentation method based on a hybrid-expert adapter. It introduces a hybrid-expert adapter (MoE-Adapter) and a task-aware gating module (TAGM), leveraging a pre-trained SAM2 backbone network to achieve efficient and unified multi-task binary segmentation. This method overcomes the limitations of traditional standalone modeling by integrating diverse tasks such as salient object detection, camouflaged object recognition, and marine animal segmentation into a single framework. Leveraging a text-driven dynamic gating mechanism and a lightweight expert network, the model automatically matches and focuses on specific features based on user-entered task descriptions. This allows for flexible adaptation to new tasks without retraining, significantly reducing the cost and complexity of multi-task deployment. By freezing the SAM2 main body weights, task adaptation is achieved solely through fine-tuning of the adapter and gating module. This approach retains the backbone network's powerful feature extraction capabilities while balancing computational efficiency with a lightweight design. It demonstrates segmentation accuracy and robustness that surpass existing methods on various task benchmark datasets. Its versatility and flexibility provide a unified solution for diverse fields such as medical image analysis, autonomous driving scene understanding, and remote sensing monitoring. It effectively responds to target characteristics and complex background challenges in different scenarios, and promotes multi-task visual processing technology towards efficiency, generalization, and practicality. It has significant theoretical innovation value and engineering application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the overall network architecture diagram of the binary segmentation method based on the SAM2 multi-task perception of the hybrid expert adapter of the present invention; Figure 2 1 is a network architecture diagram of the hybrid expert adapter module in step 2 of the SAM2 multi-task perception binary segmentation method based on the hybrid expert adapter of the present invention; Figure 3 This is the architecture diagram of the task awareness gating module in step 3 of the binary segmentation method of SAM2 multi-task perception based on the hybrid expert adapter of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 The present invention is based on the SAM2 multi-task perception binary segmentation method of the hybrid expert adapter, which is specifically implemented according to the following steps: Step 1: Build the dataset and encoder, as Figure 1 As shown; Step 1 is implemented as follows: Step 1.1, collect sample images covering various tasks; Step 1.2: Label the sample images collected in step 1.1 in labelme to generate a true value image, where the salient objects are white pixels and the background is black pixels; Step 1.3: Divide all sample images and true value images into training sets and test sets according to the ratio; Step 1.4: Use the pre-trained SAM2 as the backbone network to extract image features and build the encoder; In step 1.5, the dataset image resolution is adjusted to 352×352 and fed into the SAM2 backbone in the encoder to obtain convolutional multi-scale features. Step 2, constructing a hybrid expert adapter module; Step 3: Build a task awareness gating module.

[0019] Example 2 The present invention is based on the SAM2 multi-task perception binary segmentation method of the hybrid expert adapter, wherein step 2 is specifically implemented according to the following steps: Step 2.1, design multiple lightweight task-specific sub-networks as expert networks, such as Figure 2 As shown. For different binary segmentation tasks, multiple simple feedforward neural networks are constructed as expert networks in MoE-Adapter. Each expert network consists of several fully connected layers and is responsible for processing input features to capture unique representations of specific tasks. For example, in the salient target detection task, the expert network focuses on learning features related to target saliency, such as color contrast and edge information; for the camouflaged target detection task, the expert network focuses on extracting subtle difference features in similar areas between the target and the background. These expert networks are independent of each other and can process input features in parallel, providing the model with diverse task feature extraction capabilities; In step 2.2, we construct a task-aware gating module (TAGM) to generate gating weights. First, we create a library containing detailed text descriptions of each task. For example, in the camouflaged target detection task, the text descriptions include "objects with a similar color to the background" and "hidden objects with textures similar to the background." In the marine animal segmentation task, the descriptions include "marine life in complex underwater environments" and "marine animals with unique appearances." We use the CLIP text encoder to encode the task text and obtain the corresponding task feature vector. When the user enters the task text, the CLIP text encoder also encodes it into a feature vector. The cosine similarity between this vector and the feature vectors of each task in the task library is calculated to find the most matching task. The calculation formula is as follows: (1) in, is the normalized feature of the user input text, is the normalized feature of the predefined task text, is the similarity score between the two, according to the formula (2) Select the corresponding task, then refine the matched task text features through a projection layer, convert the task ID into a one-hot encoded vector, and fuse the two using the alpha fusion mechanism (α=0.3). The formula is as follows: (3) Finally, the fused vector is normalized to obtain the gating weights used to control the expert network in the MoE-Adapter: wexperts=Norm(f gate)(4); Step 2.3: Implement the expert network output fusion and residual connection, and perform weighted summation on the outputs of each expert network according to the gating weights generated by the task-aware gating module, as shown in the formula: (5) in, is the fused output, is the gating weight of the k-th expert network, E k is the output of the k-th expert network, n is the number of expert networks. The fused features are then passed through a lightweight final fusion layer, which consists of a convolutional layer and an activation function to further enhance the compatibility of the features with the current task. Finally, the residual connection method is used to add the features processed by the fusion layer to the original features input to the MoE-Adapter. The formula is as follows: (6) This enables the model to retain important information in the original input features while learning task-specific features, improving the model's stability and learning effect, and effectively completing the forward transmission process of the MoE-Adapter.

[0020] Example 3 The present invention is based on the SAM2 multi-task perception binary segmentation method of the hybrid expert adapter, wherein step 3 is specifically implemented according to the following steps: Step 3.1, build the task awareness gating module, such as Figure 3 As shown in the figure, the gate weight output by the task-aware gating module is used to regulate the hybrid expert adapter, and the gate weight generated by the task-aware gating module (TAGM) is converted to W expertsThese gating weights are fed into the Mixed-of-Experts Adapter (MoE-Adapter), where they dynamically adjust the contribution of each expert network to feature processing based on the specific task requirements. For example, when performing salient object detection, the gating weights give the expert network that excels at extracting salient features a dominant position in information processing; whereas, in the camouflaged object detection task, the expert network that can capture subtle differences in features is given greater attention. In this way, the MoE-Adapter can adaptively focus on the key features of different tasks, improving the model's adaptability and processing capabilities for various tasks. Step 3.1 is implemented as follows: Step 3.1.1: Receive the output of the Task-Aware Gating Module and obtain the gating weight generated by the Task-Aware Gating Module (TAGM) W experts The gating weight is calculated based on the matching results between the user-input task text and the predefined task text library. It exists in the form of a vector. Each element of the vector corresponds to the weight coefficient of an expert network in the Hybrid Expert Adapter (MoE-Adapter), reflecting the importance of each expert network in the current task processing. When obtaining the gating weight, the accuracy and integrity of the data transmission are ensured to prevent the loss or error of weight information during the transmission process. Step 3.1.2, based on the gating weight regulation expert network, the received gating weight W experts Applied to each expert network in MoE-Adapter, the output of each expert network is weighted according to the size of the gating weight. The output of each expert network is weighted according to the size of the gating weight. For example, if there are n expert networks, their outputs are , the corresponding gating weight , the weighted comprehensive output is as follows: (7) In this process, the gating weight plays a dynamic adjustment role. For expert networks that are more closely related to the current task, the corresponding gating weight is larger, and the corresponding output accounts for a higher proportion in the comprehensive results, thereby highlighting the task-specific features extracted by these expert networks; conversely, the output of expert networks that are less closely related to the task has less impact on the comprehensive results. In this way, the MoE-Adapter can flexibly adjust the degree of dependence on each expert network according to different task requirements, achieving efficient processing of input features and accurate capture of task-related features; Step 3.2: Decoding prediction based on multi-scale features First, the multi-scale features extracted by the encoder are processed to convert the features containing rich spatial information extracted in the early stage intof 1 is input to the spatial attention module, through which the spatial attention features are generated Sf , and its calculation formula is as follows: (8) Among them, sigmoid activation function, δ is ReLU activation function, and is a fully connected layer, then Sf Perform matrix multiplication with the feature flow of other layers to highlight the feature information of important areas in space. Then, the multiplied result is combined with the feature rich channel information extracted in the later stage. f 4. Splice in the channel dimension to obtain splicing features f concat , then f concat Double convolution operation is performed to further enhance spatial information. The convolution operation uses convolution kernels of different sizes. The formula is as follows: (9) After that, the convolutional features are input to the channel upsampling (CUP) module and combined with the channel attention processed f The 4 features are spliced ​​again to enrich the channel information. Finally, the final fusion features are processed by a convolutional layer to obtain the prediction result. The convolution kernel size of the convolutional layer is set according to the task requirements to complete the binary segmentation prediction of the input image and output the prediction mask.

[0021] Example 4 The present invention is based on the SAM2 multi-task perception binary segmentation method of the hybrid expert adapter, wherein step 2.1 is specifically implemented as follows: Step 2.1.1, determine the number and structure of expert networks. Determine the number of expert networks in the MoE-Adapter based on the characteristics and complexity of different binary segmentation tasks. For example, for the three tasks with relatively large task differences, namely salient target detection, camouflaged target detection, and marine animal segmentation, the number of expert networks is initially set to three. Each expert network adopts a feedforward neural network structure, consisting of an input layer, several hidden layers, and an output layer. The number of neurons in the input layer is determined by the dimension of the input features. Assuming that the dimension of the feature output from the SAM2 encoder is d, the number of neurons in the input layer is d. Set up 2-3 hidden layers, and the number of neurons in each layer decreases according to a certain ratio, such as the number of neurons in the first hidden layer is d×0.8, and the number of neurons in the second hidden layer is d×0.6, to gradually extract more abstract task-related features. The number of neurons in the output layer is adjusted according to the requirements of subsequent tasks. If it is used for feature fusion later, the number of neurons in the output layer should be consistent with the dimension of the input features to ensure that the output features can smoothly participate in the fusion process; Step 2.1.2, initialize the expert network parameters. Use the random initialization method to assign weights and biases to each expert network. For weights, use normal distribution for initialization, with the mean set to 0 and the standard deviation set to 0.01. The formula is ,in Represents the weights from layer i to layer j. For bias, all are initialized to 0, that is, b j =0, b j Represents the bias of the jth layer. This initialization method allows the expert network to have different initial states at the beginning of training, avoiding the homogenization problem caused by all networks starting training from the same starting point, and is conducive to the network learning different task feature representations.

[0022] Example 5 The present invention is based on the SAM2 multi-task perception binary segmentation method of the hybrid expert adapter, wherein step 2.1.1 is specifically implemented according to the following steps: Step 2.1.1.1, analyze the differences in task characteristics. Conduct in-depth analysis of various binary segmentation tasks such as salient target detection, camouflaged target detection, marine animal segmentation, mirror detection, polyp segmentation, and shadow detection. Study the differences in the characteristics of target objects and backgrounds in different tasks. For example, in salient target detection, salient objects usually have high contrast, unique colors or textures, and other features that are easy to attract attention; while the objects of camouflaged target detection try to blend with the background, and their colors and textures are highly similar to the background, with subtle differences. In the marine animal segmentation task, marine animals have diverse morphologies and are affected by factors such as underwater environmental lighting and water quality. The degree of distinction from the background varies greatly in different scenarios. By analyzing the differences in these task characteristics in detail, we can provide a basis for the subsequent determination of the number and structure of expert networks, so that each expert network can learn the key features of a specific task in a targeted manner; In step 2.1.1.2, analyze the differences in task features in step 2.1.1.1 to determine the number of expert networks in the MoE-Adapter. If there are significant feature differences between tasks, the number of expert networks can be appropriately increased to better capture the unique characteristics of each task. Assuming that the tasks are categorized into three categories: salient targets, camouflage and similarity fusion, and objects in special environments (such as marine animals and objects affected by light and shadow), the initial number of expert networks is set to three. For each expert network, the number of neurons in the input layer is determined by the dimensionality of the input features. If the feature map output from the SAM2 encoder has dimensions C × H × W (C is the number of channels, H is the height, and W is the width), and it is flattened into a one-dimensional vector with a length of d = C × H × W, then the number of neurons in the input layer is set to d. Two to three hidden layers are set to gradually extract abstract features. To ensure that different expert networks can better learn the characteristics of different tasks, the number of neurons in the hidden layers of each expert network is set differently. For example, the number of neurons in the first hidden layer of the expert network responsible for learning features related to salient object detection is set to d×0.7, as salient object features are relatively obvious and do not require overly complex intermediate layers. For the expert network handling camouflaged object detection, the number of neurons in the first hidden layer is set to d×0.8, increasing the complexity of feature learning due to the need to exploit subtle feature differences. For expert networks targeting tasks such as marine animal segmentation, the number of neurons in the first hidden layer is set to d×0.75. The number of neurons in the second hidden layer is further reduced from the first layer, to d×0.5, d×0.6, and d×0.55, respectively. The number of neurons in the output layer is adjusted based on the requirements of subsequent tasks. If used for fusion with other features, the number of neurons in the output layer remains consistent with the dimensionality of the input features to facilitate feature fusion, enabling the expert network to better serve different tasks.

[0023] Step 2.1.2 is implemented as follows: Step 2.1.2.1, select the initialization method. When initializing the expert network parameters, consider the impact of various initialization methods on model training, and compare common initialization methods, such as random initialization, Xavier initialization, Kaiming initialization, etc. Although random initialization is simple, it may cause unstable network training; Xavier initialization is suitable for activation functions such as sigmoid and tanh, and can better maintain the flow of data in the network; Kaiming initialization is more suitable for ReLU and its variant activation functions, which can effectively avoid the problem of gradient disappearance or explosion. Since the expert network in this model uses ReLU as the activation function, after comprehensive evaluation, the Kaiming initialization method was finally selected to initialize the weights of the expert network to ensure that the network can converge quickly and learn stably in the early stages of training; Step 2.1.2.2, perform parameter initialization operation. For each expert network, initialize the weights according to the selected Kaiming initialization method. Assume that the weight matrix of a layer of the expert network is W, and the number of input neurons in this layer is n int , the number of output neurons is n out , according to Kaiming initialization formula, weight W ij The initialization value has a mean of 0 and a standard deviation of The normal distribution of Initialize all biases to 0. During network training, biases are continuously adjusted as gradient descent progresses. Initializing them to 0 ensures consistency during the initial training phase. In practice, use functions provided by deep learning frameworks such as PyTorch to initialize parameters. For example, in PyTorch, for the fully connected layer fc, use nn.init.kaiming_norma to initialize weights and nn.init.constant_(fc.bias, 0) to initialize biases. This ensures that the parameters of each expert network are correctly initialized, laying a solid foundation for subsequent training.

[0024] Example 6 The present invention is based on the SAM2 multi-task perception binary segmentation method of the hybrid expert adapter, wherein step 2.2 is specifically implemented according to the following steps: Step 2.2.1, build and encode the task text description library, and create a detailed task text description library for different binary segmentation tasks. Taking the camouflaged target detection task as an example, the description content includes "objects with textures similar to the background", "hidden objects with colors similar to the surrounding environment", "objects with difficult-to-distinguish outlines and blending into the background", etc.; for the marine animal segmentation task, the description can be set as "various marine organisms existing in complex marine environments", "marine animals affected by the light and shadow of seawater", etc. Use the CLIP text encoder to encode the text description of each task. The CLIP text encoder converts the input text into a high-dimensional vector representation, so that the text of different tasks is measurable in the semantic space. During the encoding process, the text description of each task is input into the CLIP text encoder in turn to obtain the corresponding task feature vector. For example, the text description of task j is encoded by CLIP to obtain the feature vector ftask,j. The feature vectors of all tasks constitute the task feature library, which provides basic data for subsequent task matching and gating weight generation; Step 2.2.1 is implemented as follows: Step 2.2.1.1: Collect and organize text descriptions for different tasks. For binary segmentation tasks such as salient object detection, camouflaged object detection, marine animal segmentation, mirror detection, polyp segmentation, and shadow detection, relevant text descriptions were extensively collected from professional literature, domain knowledge, and previous research results. For example, for salient object detection, reference was made to literature in visual perception and computer vision to compile descriptions such as "a visual focal area with high contrast" and "an area where a prominent object attracts human attention." For camouflaged object detection, research on animal camouflage mechanisms in biology and related computer vision papers was consulted to obtain descriptions such as "a hidden object with a color and texture highly similar to the background" and "an object that evades detection by mimicking environmental features." For marine animal segmentation, descriptions such as "marine life under different lighting conditions" and "groups or individuals of marine animals with diverse shapes and colors" were compiled based on marine biology data and underwater image processing research. For each task, the collected text descriptions were screened and organized to remove duplicate, ambiguous, or inaccurate content to ensure that the text descriptions accurately and comprehensively cover the key features and objectives of the task. In step 2.2.1.2, use the CLIP text encoder to encode the task text. Input the organized text description of each task into the CLIP text encoder. The CLIP (Contrastive Language - Image Pretraining) text encoder is pre-trained based on large-scale image-text pairs, and can map text into a high-dimensional semantic vector space, so that semantically similar texts are closer in the space, while texts with large semantic differences are farther apart. During the encoding process, for each task's text description set, it is input into the CLIP text encoder in sequence according to a certain batch size (for example, the batch size is set to 32). The CLIP text encoder first performs lexical analysis and tokenization on the input text, converting the text into a series of tokens, and then performs feature extraction and semantic fusion on these tokens through a multi-layer Transformer architecture, and finally outputs a vector representation of a fixed dimension (assuming 512 dimensions). For example, for the k-th text description of task j text j,k , after being processed by the CLIP text encoder, the feature vector is obtained f task，j,k For each task, perform an average pooling operation on the feature vectors corresponding to all its text descriptions to obtain a unified feature vector for the task f task,j ,These feature vectors of all tasks together constitute the task feature library, which provides the key data foundation for subsequent ,task matching and gating weight calculation; Step 2.2.2, calculate the similarity and generate the gating weight. When the user enters the task text, the CLIP text encoder is also used to encode it into a feature vector f task,i , by calculating the vector and each task feature vector in the task feature library f task,j The cosine similarity is used to determine the most matching task. The cosine similarity calculation formula is as follows: (10) in, S (i,j) Represents the similarity score between user input text i and task j. From the similarity matrix S, the task with the highest score is selected as the matching result, that is, (11) After finding a matching task, the text feature vector of the task f task,j Feature refinement is performed through a projection layer, which consists of a fully connected layer and an activation function (such as ReLU), and we get fproj , and convert the task ID into a one-hot encoding vector f onehot , use alpha fusion, use alpha fusion mechanism (α=0.3) f onehot and f proj Fusion, the formula is as follows: (12) Finally, the fused vector f gate After normalization, the gating weights used to control the expert network in the MoE-Adapter are obtained. The formula is as follows: (13) In this way, the gating weights suitable for the current task are dynamically generated based on the task text entered by the user, enabling the MoE-Adapter to adaptively allocate resources and focus on feature learning of related tasks.

[0025] Example 6 To verify the effectiveness of the proposed method, training and testing were performed on five benchmark datasets representing different tasks. The last row shows the evaluation index results of the proposed method. The experimental results show that the proposed method outperforms existing deep learning-based methods in comprehensive evaluation indicators and exhibits high detection performance in various datasets: Table 1 Evaluation metrics of different methods on the camouflaged object detection benchmark dataset

[0026] Table 2 Evaluation metrics of different methods on salient object detection benchmark datasets

[0027] Table 3 Evaluation metrics of different methods on the mirror detection benchmark dataset

[0028] Table 4 Evaluation indicators of different methods on the marine animal segmentation benchmark dataset

[0029] Table 5 Evaluation indicators of different methods on the polyp segmentation benchmark dataset

[0030] This multi-task-aware binary segmentation method based on the Hybrid Expert Adapter (SAM2) uses a hybrid expert adapter (MoE-Adapter) to construct a SAM2 extension framework and a task-aware gating mechanism, effectively addressing the inefficiencies of independent modeling and repetitive training in traditional methods. This method utilizes a pretrained SAM2 network as the backbone, freezing its main weights. By deploying standard adapters and MoE-Adapters in the even and odd layers of the encoder, this method, combined with lightweight expert subnetworks and a dynamic gating strategy, achieves unified processing of multiple tasks, including salient object detection, camouflaged object recognition, and marine animal segmentation. The Task-Aware Gating Module (TAGM) uses the CLIP text encoder to semantically match user input text with a predefined task library, generating an expert network with adaptive gating weights, enabling the model to dynamically focus on key features based on task requirements. Furthermore, a multi-scale feature enhancement and decoding strategy effectively integrates spatial details and channel semantic information, improving segmentation accuracy. Experimental results show that the present invention achieves state-of-the-art performance on 18 benchmark datasets, combining efficient parameter utilization, flexible task scalability and robust generalization capabilities. It provides a unified and efficient segmentation solution for multiple fields such as medical image analysis and autonomous driving scene understanding, significantly reducing the cost and complexity of multi-task deployment.

Claims

1. A binary segmentation method based on SAM2 multi-task perception of hybrid expert adapter, characterized by: Please follow the steps below to implement it: Step 1: Build the dataset and encoder; Step 2, constructing a hybrid expert adapter module; Step 3: Build a task awareness gating module.

2. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1, collect sample images covering various tasks; Step 1.2: Label the sample images collected in step 1.1 in labelme to generate a true image, where salient objects are white pixels and background is black pixels; Step 1.3: Divide all sample images and true value images into training sets and test sets according to the ratio; Step 1.4: Use the pre-trained SAM2 as the backbone network to extract image features and build the encoder; In step 1.5, the dataset image resolution is adjusted to 352×352 and input into the SAM2 backbone in the encoder to obtain convolutional multi-scale features.

3. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 1, characterized in that: The step 2 is specifically implemented according to the following steps: In step 2.1, we design multiple lightweight task-specific sub-networks as expert networks. For different binary segmentation tasks, we construct multiple simple feedforward neural networks as expert networks in the MoE-Adapter. Each expert network consists of several fully connected layers and is responsible for processing input features to capture unique representations for a specific task. Step 2.2: Create a library containing detailed text descriptions of each task. Use the CLIP text encoder to encode the task text to obtain the corresponding task feature vector. When the user enters the task text, the CLIP text encoder is also used to encode it into a feature vector. By calculating the cosine similarity between this vector and the feature vectors of each task in the task library, the best matching task is found. The calculation formula is as follows: （1） in, is the normalized feature of the user input text, is the normalized feature of the predefined task text, is the similarity score between the two, according to the formula （2） Select the corresponding task, then refine the matched task text features through a projection layer, convert the task ID into a one-hot encoded vector, and fuse the two using the alpha fusion mechanism. The formula is as follows: （3） Finally, the fused vector is normalized to obtain the gating weights used to control the expert network in the MoE-Adapter: wexperts=Norm(f gate)(4); Step 2.3: Implement the expert network output fusion and residual connection, and perform weighted summation on the outputs of each expert network according to the gating weights generated by the task-aware gating module, as shown in the formula: （5） in, is the fused output, is the gating weight of the k-th expert network, E k is the output of the k-th expert network, n is the number of expert networks. The fused features are then passed through a lightweight final fusion layer, which consists of a convolutional layer and an activation function to further enhance the compatibility of the features with the current task. Finally, the residual connection method is used to add the features processed by the fusion layer to the original features input to the MoE-Adapter. The formula is as follows: （6） This enables the model to retain important information in the original input features while learning task-specific features, improving the model's stability and learning effect, and effectively completing the forward transmission process of the MoE-Adapter.

4. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 3, characterized in that: The step 2.1 is specifically implemented according to the following steps: Step 2.1.1: Determine the number of expert networks in the MoE-Adapter based on the characteristics and complexity of different binary segmentation tasks. Each expert network adopts a feedforward neural network structure, consisting of an input layer, several hidden layers, and an output layer. Step 2.1.2, use the random initialization method to assign weights and biases to each expert network. For weights, use normal distribution for initialization, with the mean set to 0 and the standard deviation set to 0.

01. The formula is ,in Represents the weights from layer i to layer j. For bias, all are initialized to 0, that is, b j =0, b j represents the bias of the jth layer.

5. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 4, characterized in that: The step 2.1.1 is specifically implemented according to the following steps: Step 2.1.1.1: Analyze multiple binary segmentation tasks, including salient object detection, camouflaged object detection, marine animal segmentation, mirror image detection, polyp segmentation, and shadow detection, to study the characteristic differences between the target object and the background in different tasks. In step 2.1.1.2, analyze the differences in the characteristics of each task in step 2.1.1.1 to determine the number of expert networks in the MoE-Adapter. If the characteristics of different tasks are significantly different, the number of expert networks can be appropriately increased to better capture the unique characteristics of each task.

6. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 4, characterized in that: The step 2.1.2 is specifically implemented according to the following steps: Step 2.1.2.1, select the initialization method. When initializing the expert network parameters, consider the impact of various initialization methods on model training and select the Kaiming initialization method to initialize the weights of the expert network; Step 2.1.2.2, perform parameter initialization operation. For each expert network, initialize the weights according to the selected Kaiming initialization method. Assume that the weight matrix of a layer of the expert network is W, and the number of input neurons in this layer is n int , the number of output neurons is n out , according to Kaiming initialization formula, weight W ij The initialization value has a mean of 0 and a standard deviation of The normal distribution of , for the bias, initialize them all to 0.

7. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 3, characterized in that: The step 2.2 is specifically implemented as follows: Step 2.2.1: Build and encode the task text description library. Create a detailed task text description library for different binary segmentation tasks. Step 2.2.2, calculate the similarity and generate the gate weight. When the user enters the task text, the CLIP text encoder is also used to encode it into a feature vector. f task,i , by calculating the vector and each task feature vector in the task feature library f task,j The cosine similarity is used to determine the most matching task. The cosine similarity calculation formula is as follows: （7） in, S (i,j) Represents the similarity score between user input text i and task j. From the similarity matrix S, the task with the highest score is selected as the matching result, that is, （8） After finding a matching task, the text feature vector of the task f task,j Feature refinement is performed through a projection layer, which consists of a fully connected layer and an activation function. f proj , and convert the task ID into a one-hot encoding vector f onehot , use alpha blending, use alpha blending mechanism to f onehot and f proj Fusion, the formula is as follows: （9） Finally, the fused vector f gate After normalization, the gating weights used to control the expert network in the MoE-Adapter are obtained. The formula is as follows: （10） In this way, the gating weights suitable for the current task are dynamically generated based on the task text entered by the user, enabling the MoE-Adapter to adaptively allocate resources and focus on feature learning of related tasks.

8. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 7, characterized in that: The step 2.2.1 is specifically implemented according to the following steps: Step 2.2.1.1: Collect and organize text descriptions for different tasks. For binary segmentation tasks such as salient object detection, camouflaged object detection, marine animal segmentation, mirror image detection, polyp segmentation, and shadow detection, collect relevant text description information. For each task, filter and organize the collected text descriptions, remove duplicate, ambiguous, or inaccurate content, and ensure that the text descriptions accurately and comprehensively cover the key features and objectives of the task. In step 2.2.1.2, use the CLIP text encoder to encode the task text. Input the organized text description of each task into the CLIP text encoder. During the encoding process, the text description set of each task is input into the CLIP text encoder in batches of a certain size. The CLIP text encoder first performs lexical analysis and tokenization on the input text, converting the text into a series of tokens. It then uses a multi-layer Transformer architecture to extract features and semantically fuse these tokens, ultimately outputting a fixed-dimensional vector representation.

9. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 1, characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1: Use the gating weights output by the task-aware gating module to regulate the hybrid expert adapter. W experts Input into the hybrid expert adapter, these gating weights will dynamically adjust the contribution of each expert network in the feature processing process according to different task requirements; Step 3.2: Decoding prediction based on multi-scale features First, the multi-scale features extracted by the encoder are processed to convert the features containing rich spatial information extracted in the early stage into f 1 is input to the spatial attention module, through which the spatial attention features are generated Sf , and its calculation formula is as follows: （11） Among them, sigmoid activation function, δ is ReLU activation function, and is a fully connected layer, then Sf Perform matrix multiplication with the feature flow of other layers to highlight the feature information of important areas in space. Then, the multiplied result is combined with the feature rich channel information extracted in the later stage. f 4. Splice in the channel dimension to obtain splicing features f concat , then f concat Double convolution operation is performed to further enhance spatial information. The convolution operation uses convolution kernels of different sizes. The formula is as follows: （12） After that, the convolutional features are input to the channel upsampling module and combined with the channel attention processed f The 4 features are spliced ​​again to enrich the channel information. Finally, the final fusion features are processed by a convolutional layer to obtain the prediction result. The convolution kernel size of the convolutional layer is set according to the task requirements to complete the binary segmentation prediction of the input image and output the prediction mask.

10. The SAM2 multi-task perception binary segmentation method based on hybrid expert adapter according to claim 9, characterized in that: The step 3.1 is specifically implemented according to the following steps: Step 3.1.1: Receive the output of the task-aware gating module and obtain the gating weight generated by the task-aware gating module. W experts ,When obtaining the gating weights, ensure the accuracy and integrity of data transmission, and prevent the weight information from being lost or erroneous during the transmission process; Step 3.1.2, based on the gating weight regulation expert network, the received gating weight W experts Applied to each expert network in the MoE-Adapter, the output of each expert network is weighted according to the size of the gating weight.

Images