An efficient parameter fine-tuning method for large visual models

Abstract

The present invention discloses a method for efficiently fine-tuning parameters of a large visual model, comprising: designing a convolutional-side adapter to extract image features extracted by a SAM image encoder layer by layer by stacking convolutional-side adapter modules; designing a multi-scale refinement module to construct a hierarchical feature representation; designing a feature fusion decoder to fuse the hierarchical feature representations output by the convolutional-side adapter module and the multi-scale refinement module with the convolutional-side adapter branch features, fusing features of different scales during the decoding process to obtain a refined segmentation result; and performing gradient backpropagation through a loss function to update the parameters of the convolutional-side adapter, the multi-scale refinement module, and the feature fusion decoder. The present invention significantly outperforms similar SAM fine-tuning models on public datasets in the fields of camouflaged target detection, salient object detection, and shadow detection. At the same time, it maintains a low parameter count for all modules involved in fine-tuning, reducing computational and storage costs.

Description

Technical Field

[0001] The invention belongs to the technical field of computer image processing, and in particular relates to an efficient parameter fine-tuning method based on a SAM visual large model. Background Art

[0002] Although large visual models trained on large-scale general datasets have good image feature extraction capabilities, they still have significant limitations when facing complex and diverse visual scenes and downstream tasks. Therefore, how to make large models better adapt to different downstream tasks is an important issue in the application of large models.

[0003] The Segment Anything Model (SAM) is an interactive image segmentation model. Given user prompts such as points, bounding boxes, or masks, SAM returns a refined segmentation mask. However, in the task of camouflaged object detection, SAM struggles to achieve accurate segmentation in complex camouflaged scenes. To address this issue, Chen et al. [Chen, T., Zhu, L., Deng, C., Cao, R., Wang, Y., Zhang, S., ... & Mao, P. (2023). Sam-adapter: Adapting segment anything in underperformed scenes. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 3367-3375).] introduced lightweight adapters layer by layer into the SAM image encoder, freezing the backbone network during training. By fine-tuning the adapters, the SAM image encoder is adapted to downstream tasks, enabling SAM to more effectively represent camouflaged objects. Tests on three public datasets for camouflaged object detection show that this approach improves SAM's performance in camouflaged scenes. In addition, SAM-Adapter also shows good results in shadow detection tasks. In the salient object detection task, in order to enable SAM to detect prominent objects in the image, Cui et al. [Cui, R., He, S., & Qiu, S. (2023). Adaptive LowRank Adaptation of Segment Anything to Salient Object Detection.arXiv preprint arXiv:2308.05426.] used AdaLoRA to fine-tune SAM to obtain SSOM for salient object detection. A comprehensive qualitative and quantitative evaluation on five RGB-SOD benchmark datasets demonstrated the superiority of the scheme.

[0004] The above work has explored fine-tuning SAM using PEFT technology. However, existing fine-tuning methods fail to fully utilize SAM's powerful feature extraction capabilities, and performance on various downstream task datasets still requires significant improvement. Furthermore, existing technologies all directly apply PEFT to SAM. However, as a general image segmentation model, if SAM can be specifically designed based on the characteristics of the segmentation task, its performance on downstream tasks can be further improved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes an efficient parameter fine-tuning method based on the SAM large-scale visual model. By designing a lightweight convolutional side adapter, the fine-tuning method fully leverages the advantages of SAM pre-training on large-scale datasets. Furthermore, based on the characteristics of the segmentation task, a multi-scale refinement module and feature fusion decoder are designed to extract detailed features from high-resolution images, resulting in more refined segmentation results.

[0006] The present invention is achieved through the following technical solutions:

[0007] A method for efficiently fine-tuning parameters of large visual models, including:

[0008] Design a convolutional side adapter to extract the image features extracted by the SAM image encoder layer by layer by stacking convolutional side adapter modules;

[0009] Design a multi-scale refinement module to construct hierarchical feature representation. The SAM image encoder sets the corresponding multi-scale refinement module layer by layer.

[0010] Design a feature fusion decoder to fuse the hierarchical feature representations and convolutional adapter branch features output by the convolutional side adapter module and the multi-scale refinement module. During the decoding process, features of different scales are fused to obtain refined segmentation results.

[0011] Gradient backpropagation is performed through the loss function to update the parameters of the three modules: convolutional side adapter, multi-scale refinement module and feature fusion decoder.

[0012] Furthermore, the convolution side adapter includes two 1*1 convolution modules. The first 1*1 convolution module expands the compressed input features to the SAM image encoder feature dimension, and then fuses them with the output features of the SAM image encoder through element-by-element addition. The fused features are returned to the SAM image encoder; the second 1*1 convolution module compresses the fused features to the convolution side adapter input feature dimension as the input of the next layer of convolution side adapter.

[0013] Furthermore, in the multi-scale refinement module, the dimension of the SAM encoder feature is first compressed by a 1*1 convolution module, and then a higher-resolution hierarchical feature representation is obtained by a deconvolution module, and then a lightweight gating unit is used to act on high-resolution features of different scales.

[0014] Furthermore, the high-resolution hierarchical feature representation output by the gating unit is fused with the output of the previous multi-scale refinement module by pixel-by-pixel addition, and the obtained fused feature representation serves as the input of the next multi-scale refinement module.

[0015] Furthermore, the feature fusion decoder injects the hierarchical feature representation into the convolution side adapter branch feature during the decoding process. Specifically, for the convolution side adapter branch feature, it is first convolved by 1*1, and then fused with high-resolution features of different scales respectively; for the hierarchical feature representation, the pooling operation is used to downsample the hierarchical feature representation to obtain the key features. For the high-resolution feature representation of different scales, a two-stage injection method is used to inject the key features and the hierarchical feature representation into the convolution side adapter branch feature respectively.

[0016] Furthermore, the hierarchical feature pooling process is as follows: first, the hierarchical features are processed by 1*1 convolution, and then global average pooling and global maximum pooling are performed on the processed features, and the results are added pixel by pixel.

[0017] Furthermore, the two-stage injection first injects key features into the convolution-side adapter branch features, including splicing and 3*3 convolution operations along the feature dimension, then performs upsampling operations to align the feature scales, and then performs splicing and 3*3 convolution operations along the feature dimension with the high-resolution hierarchical feature representation.

[0018] Furthermore, for the disguised target detection task and the salient object detection task, binary cross entropy (BCE) loss and IOU loss are used:

[0019]

[0020] For the shadow detection task, balanced binary cross entropy loss is used:

[0021]

[0022] In the above formula, A represents the real mask, B represents the predicted mask; y i is the binary label of the i-th pixel 0 or 1, 0 represents background pixels and 1 represents foreground pixels; p(y i) is the probability that the predicted output pixel i belongs to the foreground, and N is the number of pixels involved in calculating the loss. BBCE loss introduces two parameters w1 and w2 based on BCE loss, where w1 is the ratio of foreground pixels in the pixels involved in calculating the loss, and w2 is the ratio of the number of background pixels involved in calculating the loss to the number of foreground pixels.

[0023] The present invention also provides an image segmentation model training method. The image segmentation model is constructed using the above-mentioned parameter efficient fine-tuning method of the large visual model. The training steps are as follows:

[0024] Step 1: Model initialization. For the ViT backbone network, the image encoder weights trained on a large-scale dataset using SAM are initialized and the parameters are frozen. For the convolutional side adapter, multi-scale refinement module, and feature fusion decoder modules, random initialization is used.

[0025] Step 2: Extract visual features of the input image through the SAM image encoder, and extract the features of the SAM image encoder layer by layer through stacked convolutional side adapter modules;

[0026] Step 3: Construct hierarchical feature representation through multi-scale refinement module;

[0027] Step 4: Fuse the hierarchical feature representation and convolutional side adapter branch features through the feature fusion decoder;

[0028] Step 5: Calculate the loss, backpropagate the gradient, and update the parameters of the convolutional side adapter, multi-scale refinement module, and feature fusion decoder.

[0029] Step 6: Repeat steps 2 to 5 to train the model using the training set data until the model converges;

[0030] Step 7: Verify the model segmentation performance through the validation set.

[0031] The present invention also provides an image segmentation method, which inputs the image to be segmented into the image segmentation model trained by the above-mentioned image segmentation model training method, and obtains the final segmentation output through reasoning.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This paper introduces the side network idea into the fine-tuning of SAM for the first time and designs a lightweight convolutional side adapter to fully extract the features of the SAM image encoder.

[0034] This paper specifically considers the characteristics of the segmentation task during the SAM fine-tuning process and proposes a multi-scale refinement module and a feature fusion decoder. Refined target location information is obtained through high-resolution hierarchical features, which are fully integrated during the decoding process to obtain a refined segmentation result.

[0035] This paper significantly outperforms similar SAM fine-tuning work on public datasets in three areas: camouflaged object detection (COD), salient object detection (SOD), and shadow detection. It also maintains a low parameter count for all modules involved in fine-tuning, reducing computational and storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the overall architecture of the image segmentation model in an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the convolution-side adapter structure of the image segmentation model in an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of the multi-scale refinement module structure of the image segmentation model in an embodiment of the present invention;

[0040] Figure 5 Schematic diagram of the feature fusion decoder structure of the image segmentation model in an embodiment of the present invention;

[0041] Figure 6 This is a comparison chart of the effects of the image segmentation model in an embodiment of the present invention and the existing methods in the field of camouflaged target detection. DETAILED DESCRIPTION

[0042] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present invention and the features within the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0043] The following is a detailed description of the process of image segmentation using the image segmentation model constructed based on the SAM parameter efficient fine-tuning method provided by the present invention. Figure 1 As shown, the following steps are included:

[0044] Step 1: Model initialization.

[0045] The image segmentation model structure constructed based on the SAM parameter efficient fine-tuning method of the present invention is as follows: Figure 2 As shown in the figure, the ViT backbone network uses image encoder weights initialized using SAM trained on large-scale datasets. Random initialization is used for the Convolutional Side Adapter (CSA), Multi-scale Refine Module (MRM), and Feature Fusion Decoder (FFD) modules. During model training, the parameters of the ViT backbone network are frozen, and only the parameters of the CSA, MRM, and FFD modules are updated.

[0046] Step 2: CSA branch feature extraction.

[0047] Given an image Extract visual features of images through SAM image encoder On the other hand, by stacking CSA modules, the features of the SAM image encoder are extracted layer by layer to obtain image features that are suitable for downstream tasks.

[0048] CSA can well extract the features of the SAM image encoder and bridge the gap between it and downstream task data.

[0049] The structure of CSA is as follows Figure 3 As shown, in CSA, there are two 1*1 convolution modules. The first 1*1 convolution module compresses the feature F csa Expand to SAM image encoder feature F vit The output feature embedding of the SAM image encoder is then fused by an addition operation. The second 1*1 convolution module compresses the fused feature into the input feature F csa dimension, as the input of the next layer of CSA.

[0050] For the input image I, the SAM image encoder first converts the image into a feature vector through the patch embed operation. Input features of CSA branch By Linear projection acquisition, the specific operation can be expressed as:

[0051]

[0052] Where MLP represents the fully connected layer.

[0053] Given the output features of the i-1th CSA and the output features of the j-1th layer SAM image encoder Then the operation of the i-th CSA can be expressed as:

[0054]

[0055] where conv 1×1 Represents a 1*1 convolution module, which includes 1*1 convolution, batch normalization and activation operations; is the input of the j-th layer SAM image encoder, is the input of the i-th CSA.

[0056] Step 3: Image refinement and feature extraction.

[0057] Considering that in segmentation tasks, obtaining more refined segmentation results requires the image segmentation model to have a strong ability to describe detailed features such as object edges. However, the SAM image encoder downsamples the image by a factor of 16 during the patch embedding phase, which can make it difficult to extract target location information. Therefore, to fully extract target location information, the present invention designs a multi-scale refinement module (MRM) to obtain higher-resolution features with more details.

[0058] Each layer of the SAM image encoder is attached with a corresponding MRM module, which upsamples the feature embeddings of each layer of the image encoder through MRM to construct a hierarchical feature representation. The structure of MRM is as follows Figure 4 As shown, for a given j-1th layer SAM image encoder output feature and the hierarchical features of the i-1th layer MRM output First, compress it through a 1*1 convolution module The feature dimension of Then Obtain higher resolution hierarchical feature representation through deconvolution module The above process can be formally expressed as:

[0059]

[0060] Where deconv represents the deconvolution module, which Perform upsampling by 2 times and 4 times respectively to obtain high-resolution features of two different scales.

[0061] To collect richer image detail features, it is necessary to fuse the obtained high-resolution features with the hierarchical features output by the previous MRM layer. To control the degree of feature fusion and avoid introducing invalid features, the present invention uses a lightweight gating unit to act on high-resolution features of different scales. The operation of the gating unit can be formally expressed as:

[0062]

[0063] Among them, Tanh represents the hyperbolic tangent function, Linear represents the linear layer, and ReLU is the activation function. Represents element-wise multiplication.

[0064] Finally, the two features are added element by element to achieve feature fusion:

[0065]

[0066] Step 4: Feature fusion and decoding.

[0067] Through lightweight FFD, hierarchical features are represented Inject the feature F obtained by CSA csa , to enhance the feature representation and obtain the refined segmentation mask. The structure of FFD is as follows Figure 5 As shown, for the input feature F csa , first obtain F through 1*1 convolution in,csa .

[0068] For hierarchical feature representation The pooling operation is used to downsample the hierarchical feature representation to obtain key features. The hierarchical feature pooling process can be expressed as:

[0069]

[0070] Where GAP and GMP represent global average pooling and global maximum pooling, respectively.

[0071] For high-resolution features of different scales, a two-stage injection method is used to inject them into F in,csa First, the key feature F obtained by the pooling operation key,mrm Injection F in,csa , further integrate the complete high-resolution hierarchical features F mrm Injection F in,csa . With a 2x upsampling resolution feature Taking the injection process as an example, the fusion operation process can be formally expressed as follows:

[0072]

[0073] Among them, [,] represents the splicing operation along the channel dimension, and UpSample represents 2x upsampling.

[0074] After that, the high-resolution features are upsampled by 4 times Injection F stage2,csa Process and 2x upsampling of high-resolution features Injection Fin,csa Through two-stage injection, the hierarchical feature representation is fully integrated with the CSA branch features to obtain an enhanced feature representation, enabling the decoding process to obtain more refined segmentation results.

[0075] Step 5: Calculate the loss, backpropagate the gradient, and update the parameters of the three modules CSA, MRM, and FFD to adapt them to the data patterns of different downstream tasks.

[0076] For the disguised target detection task and the salient object detection task, binary cross entropy (BCE) loss and IOU loss are used:

[0077]

[0078] where y i is the binary label of the i-th pixel 0 or 1, 0 represents background pixels and 1 represents foreground pixels. i ) is the probability that the i-th pixel belongs to the foreground, and N is the number of pixels involved in calculating the loss.

[0079]

[0080] Where A represents the real mask and B represents the predicted mask.

[0081] For the shadow detection task, balanced binary cross entropy (BBCE) loss is used:

[0082]

[0083] BBCE loss introduces two parameters, w1 and w2, based on BCE loss, where w1 is set to the ratio of foreground pixels in the pixels involved in calculating the loss, and w2 is set to the ratio of the number of background pixels to the number of foreground pixels involved in calculating the loss.

[0084] Step 6: Repeat steps 2 to 5 using the training set data to train the model until the model converges. During model training, freeze the parameters of the ViT backbone network and only update the parameters of the CSA, MRM, and FFD modules. Test the model performance using the validation set.

[0085] Step 7: In the inference phase, the test image is used to complete the inference through the forward process from step 2 to step 4, and then post-processing is performed. Specifically, the segmentation mask obtained by FFD is up-sampled by bilinear interpolation to obtain a mask of the target output size, and finally a binarization operation is performed to obtain the final segmentation output.

[0086] Figure 6The figure shows the comparison of the image segmentation model TS-SAM proposed in this paper with SAM, SAMAdapter and the most advanced domain-specific models on some images in the COD10K dataset. It can be seen that the proposed method is significantly better than other methods in predicting image details, indicating the superiority of the proposed method.

[0087] Compared with similar methods, the image segmentation model of the present invention shows higher accuracy in the fields of camouflaged object detection (COD), shadow segmentation and salient object detection (SOD), and also achieves competitive performance compared with a variety of advanced field-specific methods.

[0088] The segmentation effect of the image segmentation model is tested by detecting disguised targets.

[0089] Table 1 Quantitative comparison of TS-SAM with SAM, SAM-Adapter, and SOTA domain-specific models on four datasets of COD tasks (CHAMELEON, CAMO, COD10K, and NC4K)

[0090]

[0091] Table 1 shows the results of TS-SAM compared with the existing state-of-the-art models on four commonly used COD datasets. The present invention achieves competitive performance on all four datasets with only 4.4% of the parameters fine-tuned and no specific modules designed specifically for the task. In particular, on the two largest datasets, COD10K and NC4K, TS-SAM achieves SOTA performance in most indicators. For the COD10K dataset, the image segmentation model provided by the present invention is 100% better than the second best method, SARNet, in S α , The performance of the three indicators was improved by 3.3% and 4.4% respectively, and the MAE was reduced by 19%. α , The SOTA performance was achieved on the three indicators of MAE. α Compared with SARNet, the performance of the proposed method has been improved by 1.8%, which shows that the image segmentation model provided by the present invention has strong generalization ability.

[0092] It should be noted that the SAM-adapter also uses adapter technology to fine-tune the SAM, achieving a certain performance improvement over the original SAM on the COD task. However, the TS-SAM of the present invention achieves significant improvements on all metrics reported by the SAM-adapter on the three datasets by serially connecting convolutional-side adapters, introducing SAM feature maps into convolutional-side adapter training, and using multi-scale refinement modules. This demonstrates that the TS-SAM solution has significant advantages over the SAM-adapter for extracting SAM image encoder features and adapting them to downstream tasks.

[0093] The segmentation effect of the image segmentation model is tested through shadow detection.

[0094] Table 2 Quantitative comparison of TS-SAM with SAM, SAM-Adapter and SOTA domain-specific models on the ISTD dataset of the shadow detection task

[0095]

[0096] Results on the ISTD dataset are shown in Table 2. As shown in Table 2, by introducing a lightweight adapter, SAM-Adapter significantly improves the performance of SAM on the shadow detection task. TS-SAM achieves significant performance improvements over SAM-Adapter. Without any special design for the shadow detection task, TS-SAM achieves performance close to that of the best domain-specific models.

[0097] The segmentation effect of the image segmentation model is tested through salient object detection.

[0098] Table 3 Quantitative comparison of TS-SAM with SAM, SSOM and SOTA domain-specific models on five datasets (DUTS, ECSSD, OMRON, HKU-IS and PASCAL-S) for SOD tasks

[0099]

[0100] Table 3 shows the comparison of the results of TS-SAM with SAM, SSOM and SOTA domain-specific models on five commonly used SOD datasets. Compared with the similar SAM-based method SSOM, TS-SAM shows obvious advantages, and the MAE index on all five datasets is much lower than SSOM. This shows that the fine-tuning method proposed in this paper is better than the AdaLora-based fine-tuning scheme proposed by SSOM. Compared with the advanced domain-specific methods, TS-SAM still achieves competitive performance with only a small number of parameters fine-tuned. α ,E φAnd MAE indicators, TS-SAM achieved SOTA performance on the ECSSD, OMRON and PASCAL-S datasets, and also ranked high on the DUTS and HKU-IS datasets. In terms of indicators, the performance of TS-SAM is relatively weak, which will be the direction of future improvements.

[0101] Table 4 Effectiveness of module components on different datasets

[0102]

[0103] The proposed modules, CSA, MRM, and FFD, were studied on four disguised object detection datasets: CHAMELEON, CAMO, COD10K, and NC4K. The modules used in TS-SAM were implemented on a SAM baseline model, and the results are shown in Table 4. The results show that the SAM baseline model performs poorly when only the decoder is fine-tuned. First, the effectiveness of the modules working independently was verified, including the introduction of only the CSA module and the simultaneous introduction of the MRM and FFD modules. Both settings yielded significant performance improvements compared to the baseline model while adding only a small number of trainable parameters (5.49M and 2.35M, respectively). The results demonstrate that the proposed modules can effectively extract visual information from the SAM image encoder and adapt it to downstream tasks. Second, a model combining all three proposed modules achieved the best performance across all metrics on the four datasets. These results demonstrate that the proposed CSA, MRM, and FFD modules complement each other, with the MRM and FFD modules, designed specifically for segmentation tasks, helping the CSA achieve better performance.

[0104] The present invention has been described in detail above through the embodiments, but the contents described are only exemplary embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. The scope of protection of the present invention is defined by the claims. Any use of the technical solution described in the present invention, or any person skilled in the art who, inspired by the technical solution of the present invention, designs a similar technical solution within the essence and scope of protection of the present invention to achieve the above-mentioned technical effects, or any equivalent changes and improvements made to the scope of application, shall still fall within the scope of protection covered by the patent of the present invention.

Claims

1. An efficient parameter fine-tuning method for a large visual model, characterized by: include: Design a convolutional side adapter to extract the image features extracted by the SAM image encoder layer by layer by stacking convolutional side adapter modules; Design a multi-scale refinement module to construct hierarchical feature representation. The SAM image encoder sets the corresponding multi-scale refinement module layer by layer. Design a feature fusion decoder to fuse the hierarchical feature representations and convolutional adapter branch features output by the convolutional side adapter module and the multi-scale refinement module. During the decoding process, features of different scales are fused to obtain refined segmentation results. The feature fusion decoder injects the hierarchical feature representation into the convolution side adapter branch feature during the decoding process. Specifically, for the convolution side adapter branch feature, it first passes 1*1 convolution and then fuses with high-resolution features of different scales respectively; for the hierarchical feature representation, it uses pooling operation to downsample the hierarchical feature representation to obtain key features. For high-resolution feature representation of different scales, it adopts a two-stage injection method to inject the key features and hierarchical feature representation into the convolution side adapter branch feature respectively. Gradient backpropagation is performed through the loss function to update the parameters of the three modules: convolutional side adapter, multi-scale refinement module and feature fusion decoder.

2. The method for efficient parameter fine-tuning of a large visual model according to claim 1, characterized in that: The convolution side adapter includes two 1*1 convolution modules. The first 1*1 convolution module expands the compressed input features to the SAM image encoder feature dimension, and then fuses them with the output features of the SAM image encoder through element-by-element addition. The fused features are returned to the SAM image encoder; the second 1*1 convolution module compresses the fused features to the convolution side adapter input feature dimension as the input of the next layer of convolution side adapter.

3. The method for efficient parameter fine-tuning of a large visual model according to claim 1, characterized in that: In the multi-scale refinement module, the dimension of the SAM encoder feature is first compressed by a 1*1 convolution module, and then a higher-resolution hierarchical feature representation is obtained by a deconvolution module. Then, a lightweight gating unit is used to act on high-resolution features of different scales.

4. The method for efficient parameter fine-tuning of a large visual model according to claim 3, characterized in that: The high-resolution hierarchical feature representation output by the gated unit is fused with the output of the previous multi-scale refinement module by pixel-by-pixel addition, and the obtained fused feature representation serves as the input of the next multi-scale refinement module.

5. The method for efficient parameter fine-tuning of a large visual model according to claim 1, characterized in that: The hierarchical feature pooling process is as follows: first, the hierarchical features are processed through 1*1 convolution, and then the processed features are subjected to global average pooling and global maximum pooling, and the results are added pixel by pixel.

6. The method for efficient parameter fine-tuning of a large visual model according to claim 1, characterized in that: The two-stage injection is specifically as follows: first, the key features are injected into the convolution side adapter branch features, including splicing and 3*3 convolution operations along the feature dimension, then upsampling operations are performed to align the feature scales, and then splicing and 3*3 convolution operations are performed with the high-resolution hierarchical feature representation along the feature dimension.

7. The method for efficient parameter fine-tuning of a large visual model according to claim 1, characterized in that: For the disguised target detection task and the salient object detection task, binary cross entropy loss and IOU loss are used: For the shadow detection task, balanced binary cross entropy loss is used: In the above formula, A represents the real mask and B represents the predicted mask; y i is the binary label of the i-th pixel 0 or 1, 0 represents background pixels and 1 represents foreground pixels; p(y i ) is the probability that the predicted output pixel i belongs to the foreground, and N is the number of pixels involved in calculating the loss. BBCE loss introduces two parameters w1 and w2 based on BCE loss, where w1 is the ratio of foreground pixels in the pixels involved in calculating the loss, and w2 is the ratio of the number of background pixels involved in calculating the loss to the number of foreground pixels.

8. A method for training an image segmentation model, characterized in that: The image segmentation model is constructed using the method described in any one of claims 1 to 7, and the training steps are as follows: Step 1: Model initialization. For the ViT backbone network, the image encoder weights trained on a large-scale dataset using SAM are initialized and the parameters are frozen. For the convolutional side adapter, multi-scale refinement module, and feature fusion decoder modules, random initialization is used. Step 2: Extract visual features of the input image through the SAM image encoder, and extract the features of the SAM image encoder layer by layer through stacked convolutional side adapter modules; Step 3: Construct hierarchical feature representation through multi-scale refinement module; Step 4: Fuse the hierarchical feature representation and convolutional side adapter branch features through the feature fusion decoder; Step 5: Calculate the loss, backpropagate the gradient, and update the parameters of the convolutional side adapter, multi-scale refinement module, and feature fusion decoder. Step 6: Repeat steps 2 to 5 to train the model using the training set data until the model converges; Step 7: Verify the model segmentation performance through the validation set.

9. An image segmentation method, characterized in that: The image to be segmented is input into the image segmentation model trained by the method according to claim 8, and the final segmentation output is obtained through inference.

Images