A medical image segmentation method based on frequency domain enhancement and multi-scale hollow convolution

Abstract

The present application belongs to the field of medical image processing and deep learning, and proposes a medical image segmentation method based on frequency domain enhancement and multi-scale hollow convolution. Based on the SAM model, the original framework interface is retained, the image encoder is modified, and two types of adapters are introduced. The frequency domain enhancement adapter enhances the key frequency components through Fourier transform and channel attention, improving the detail representation; the multi-scale adapter extracts multi-scale information and combines attention to strengthen the lesion representation. The enhanced features are input into the decoder to output the mask, and parameter-efficient fine-tuning is adopted to reduce the training overhead and improve the segmentation accuracy and generalization performance.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing, artificial intelligence and deep learning technology. Specifically, it relates to a method and system for efficient fine-tuning of medical image segmentation parameters and feature enhancement based on Segment Anything Model (SAM). In particular, it relates to introducing a frequency domain enhancement adapter and a multi-scale dilated convolution adapter only in the image encoder of SAM, and improving the accuracy and generalization ability of medical segmentation through an efficient parameter fine-tuning strategy. Background Technology

[0002] Medical image segmentation is a crucial step in medical image analysis, aiming to automatically extract organ or lesion regions from medical images. However, due to the inherent characteristics of medical images, such as weak contrast, noise interference, blurred boundaries, large target scale spans, and limited training samples due to high annotation costs, segmentation models still face challenges in terms of accuracy, generalization, and training stability.

[0003] In recent years, basic vision models have brought new paradigms to segmentation tasks. Among them, the Segment Anything Model (SAM) uses an "image encoder + cue encoder + mask decoder" framework to generate segmentation masks in different scenarios and has strong transfer potential. The image encoder of SAM usually adopts a Transformer-based visual coding structure, which can provide high-quality general visual representations.

[0004] However, directly applying SAM to medical image segmentation still has the following shortcomings: (1) Significant domain differences: SAM is mainly trained based on natural images. The imaging mechanism and texture statistics of medical images are quite different from those of natural images. When directly transferring zero samples or a small number of samples, problems such as inaccurate target boundaries and missing small structures are likely to occur; (2) High cost and easy overfitting of full fine-tuning: The SAM backbone parameters are large. Under the condition of small medical samples, full fine-tuning training is costly and easy to overfit, resulting in a decrease in cross-center generalization; (3) Insufficient details and multi-scale context: Medical segmentation often requires accurate high-frequency details such as boundaries / textures, as well as robust perception and fusion of structures of different scales. Existing improvement methods often improve performance by increasing the decoder and introducing additional pyramids / multi-scale structures, which often brings additional parameters and computational overhead, which is not conducive to balancing performance and generalization under the constraint of "freezing large models and low-cost adaptation".

[0005] Therefore, there is a need for a medical image segmentation scheme based on SAM that can achieve effective domain adaptation by introducing only a small number of incremental modules inside the image encoder, while minimizing the range of original parameter updates, and simultaneously enhancing the ability to model high-frequency details and multi-scale context. Summary of the Invention

[0006] To address the issues of inaccurate boundary values ​​and missed segments of fine structures due to domain differences in medical images, as well as the high cost of full-scale fine-tuning, which leads to large parameter counts, high training costs, and overfitting on small samples, this invention provides a medical image segmentation method based on the Segment Anything Model (SAM). This method freezes the original SAM parameters and inserts a frequency domain enhancement adapter and a multi-scale dilated convolution adapter only into the image encoder. The adapter parameters, cue encoder, and mask decoder are then trained to achieve efficient medical domain adaptation and improved segmentation performance.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A medical image segmentation method based on frequency domain enhancement and multi-scale dilated convolution includes the following steps:

[0009] Step 1: Obtain the medical image to be segmented and its annotations, and perform preprocessing; the preprocessing includes at least size and resolution unification and intensity normalization, and optional data augmentation during the training phase.

[0010] Step 2: Input the preprocessed medical image into the SAM image encoder to obtain image embedding features; introduce a frequency domain enhancement adapter into the attention output projection path of the image encoder to perform bottleneck dimensionality reduction processing on the attention projection output features, so that the attention projection output features are X∈R H×W×C The formula for bottleneck dimensionality reduction D (D≪C) is as follows:

[0011]

[0012] in, Bottleneck features after dimensionality reduction (usually shape) ), Layer Normalization normalizes features to stabilize training. The weight matrix of a dimension-reduced linear mapping, with common shapes: ,Bundle Mapped to ; : Bias vector of a dimension-reduced linear mapping; Non-linear activation functions enhance expressive power.

[0013] The bottleneck feature is subjected to a two-dimensional fast Fourier transform and amplitude-phase decomposition, as shown in the following formula:

[0014] ,

[0015] in, : The complex spectrum in the frequency domain after two-dimensional Fourier transform; Two-dimensional Fast Fourier Transform; Amplitude spectrum; : Modulus of a complex number; Phase spectrum; Complex phase angle.

[0016] Channel attention is constructed on the amplitude spectrum, and key frequency components are adaptively enhanced according to the attention weights to highlight frequency domain information that is more discriminative for the segmentation task. The amplitude spectrum gain enhancement formula is as follows:

[0017] ;

[0018] in, : Enhanced amplitude spectrum. Channel attention weights, typically within a certain range. or "1 + A": Gain enhancement, avoiding suppressing the original amplitude (compared to pure multiplication). More stable); Element-wise multiplication means multiplying elements in the same position.

[0019] Then, the phase is reconstructed to reconstruct the spectrum and returned to the spatial domain via inverse FFT, as shown in the following formula:

[0020]

[0021] in, : Enhanced frequency domain complex spectrum. Euler form of phase Convert to a compound exponent. Phase spectrum; : Two-dimensional inverse fast Fourier transform; Enhanced features after inverse transformation back to the spatial domain. : Take the real part.

[0022] Then it is fused with the residual to enhance high-frequency details such as boundaries and textures, as shown in the following formula:

[0023]

[0024] in, The increment of the FFT output; : Output features after fusion.

[0025] Dimensional Upgrading and Residual Fusion:

[0026] in, The low-dimensional enhanced features are those that have been enhanced in the frequency domain, recovered to the spatial domain by inverse fast Fourier transform, and then subjected to batch normalization and random deactivation. The weight matrix is ​​a linear mapping for increasing dimensionality. s is the bias vector for the linear mapping to higher dimensions; s is the scaling factor; This is the feature increment of the frequency domain enhancement adapter output; The original characteristics before inputting this adapter; The output features are those after residual fusion.

[0027] The calculation process for the three-way multi-scale branch is as follows:

[0028]

[0029]

[0030]

[0031] in, =2, =4, This represents the bottleneck characteristics after dimensionality reduction; This represents the output of a 1×1 convolution branch; This represents the output of a 3×3 depth-separable dilated convolution branch with an expansion rate of d1=2; This represents the output of a 3×3 depth-separable dilated convolution branch with an expansion rate of d2=4. Represents a 1×1 convolution; Indicates the expansion rate 3×3 depthwise convolution; This represents a 1×1 pointwise convolution. Step four: Input the image embedding features enhanced by the two types of adapters into the mask decoder of SAM, and output the segmentation mask under no prompt condition. No manual point cues, bounding box cues, or coarse mask cues are input during the training and inference phases, thus achieving prompt-free automatic segmentation for medical image segmentation tasks.

[0032] Step 5: During the training phase, the original image encoder parameters of SAM are frozen, while the cue encoder and mask decoder remain trainable. At the same time, the frequency domain enhancement adapter and the multi-scale dilated convolution adapter are jointly optimized. During the inference phase, the trained weights are loaded to achieve the segmentation output of medical images.

[0033] Compared with existing technologies, the present invention has at least the following beneficial effects: it achieves effective adaptation to the medical domain with very few trainable parameters and low training cost; it enhances boundary and texture details through frequency domain enhancement; it improves the ability to model the context of structures at different scales through multi-scale hole attention enhancement; residual and approximate identity initialization ensure training stability and do not change the original input and output interface of SAM, which facilitates engineering integration and deployment. Attached Figure Description

[0034] Figure 1 : Overall flowchart of the medical image segmentation method of this invention;

[0035] Figure 2 : Structural diagram of the frequency domain enhancement adapter;

[0036] Figure 3 : Structure diagram of a multi-scale dilated convolution adapter; Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0038] In the description of this invention, it is necessary to understand that the orientation or positional relationship indicated by terms such as "up", "down", "left", "right", "inner", "outer", "top", and "bottom" are based on the orientation or positional relationship shown in the accompanying drawings. The purpose is only to facilitate the description of this invention and to simplify the description. It is not intended to indicate or imply that the component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0039] This invention provides a detailed description of its implementation process in a typical medical image segmentation project. Based on the Segment Anything Model (SAM), this implementation freezes only the SAM's image encoder, inserts a frequency domain enhancement adapter and a multi-scale dilated convolution adapter into the SAM's image encoder, and trains the parameters of both adapters, the cue encoder, and the mask decoder to achieve efficient parameter fine-tuning and improved accuracy. A medical image segmentation method based on frequency domain enhancement and enhanced multi-scale dilated convolution includes the following steps:

[0040] Step S1: Data collection and annotation;

[0041] Step S101: Obtain the medical image data of the target task and its pixel-level annotation;

[0042] Step S102: Data partitioning. Divide the dataset into a training set, a validation set, and a test set in an 8:1:1 ratio;

[0043] Step S103: Spatial and Intensity Preprocessing. The input image is resized / resampled to the project-defined resolution; intensity normalization is performed; the labeled mask is resampled and aligned at the same scale to ensure spatial consistency between the image and the mask.

[0044] Step S104: Data Augmentation. Randomly apply flipping, rotation, scaling, random cropping, elastic deformation, intensity perturbation (brightness / contrast / noise), etc.; during the verification and testing phase, random augmentation is turned off, and only deterministic preprocessing is retained.

[0045] Step S2: Model building and adapter insertion;

[0046] Step S201: Load the SAM base model. Construct the Image Encoder, Prompt Encoder, and Mask Decoder for SAM, and load the pre-trained weights; keep the original input and output interfaces of SAM unchanged;

[0047] Step S202: Adapter Insertion Strategy. Two types of adapters are inserted only within the Image Encoder: (1) a frequency domain enhancement adapter is inserted in the projection path of the attention module output of the Transformer block to enhance the detail representation of the attention path; (2) a multi-scale dilated convolution adapter is inserted at the feedforward / fusion position of the Transformer block as a trainable branch of the MLP bypass to extract and fuse multi-scale context. The insertion position is preferably located before the residual addition to ensure that the adapter output can be stably integrated into the backbone features through the residual path;

[0048] Step S203: Feature shape definition. Let a certain layer feature be... B represents the batch size, H and W represent the spatial dimensions, and C represents the number of channels. Both types of adapters use a bottleneck structure to perform calculations in a low-dimensional D space (D can be set according to a trade-off between computing power and accuracy), then increase the dimension back to C and add it to the residual for output, in order to control the number of parameters and computational overhead.

[0049] Step S3: Implementation steps of the frequency domain enhancement adapter;

[0050] Step S301: Normalization and Dimensionality Reduction. Optional layer normalization is performed on the input features; the channel dimension C is mapped to the bottleneck dimension D through a linear dimensionality reduction layer, and the bottleneck features are obtained through nonlinear activation;

[0051] Step S302: Frequency Domain Enhancement. The bottleneck features are rearranged into (B, D, H, W) and a two-dimensional fast Fourier transform is performed; amplitude-phase decomposition is performed; channel attention is constructed on the amplitude spectrum and gain enhancement is applied to the amplitude spectrum; after reconstructing the spectrum using the original phase, an inverse fast Fourier transform is performed to return to the spatial domain and the real part is taken; the corresponding formulas are as follows:

[0052] Step S303: Dimensional Upgrading and Residual Fusion. Batch normalization and random deactivation are performed on the enhanced features, which are then mapped back to the channel dimension C through the dimensionality-upgrading linear layer. After multiplying by a scaling factor, the result is added to the residual and output. The dimensionality-upgrading linear layer preferably uses zero initialization or approximately identity initialization to ensure that the model behavior is consistent with the original SAM in the early stages of training, which is beneficial for stable convergence.

[0053] Step S304: Deep Adaptation. This involves linearly projecting the backbone features of the same resolution. Before entering the frequency domain enhancement, it is added to the bottleneck feature to provide a low-dimensional guiding signal shared across layers, thereby improving training stability and consistency.

[0054] Step S4: Implementation steps of the multi-scale dilated convolution adapter;

[0055] Step S401: Dimensionality reduction and 2D rearrangement. After performing layer normalization on the input features, the dimensionality is reduced to the bottleneck dimension D, and rearranged into a (B,D,H,W) 2D feature map;

[0056] Step S402: Multi-scale dilated convolution branch. Construct at least three branches: branch 1 is a 1×1 convolution; branch 2 is a 3×3 depthwise separable dilated convolution with dilation rate d1; branch 3 is a 3×3 depthwise separable dilated convolution with dilation rate d2, and d2≠d1 (preferably d1=2, d2=4), used to obtain contextual information of different receptive fields;

[0057] Step S403: Scale-Adaptive Fusion and Residuals. Softmax is applied to the learnable scale weights to obtain fusion coefficients. The outputs of each branch are weighted and summed to obtain the fusion features. These are then added to the bottleneck residuals and batch normalization is performed to stabilize training. The fusion formula is as follows:

[0058]

[0059] Multi-scale fusion features; ": Scales the output of each branch according to its weight; +: sums the elements one by one to obtain the fusion result.

[0060] Step S404: Attention Enhancement. Perform channel attention enhancement on the fused features; and optionally enable spatial attention to perform region weighting on the two-dimensional feature map to further enhance key locations such as organ edges and lesion areas;

[0061] Step S405: Dimension Upgrading and Bypass Fusion. The enhanced bottleneck features are rearranged back to (B,H,W,D), mapped back to (B,H,W,C) through a dimension-upgrading linear layer, multiplied by a scaling factor, and used as the MLP bypass output. This bypass output is then fused with the main MLP output and the residual to obtain the Transformer block output.

[0062] Step S5: To achieve efficient parameter fine-tuning, during the training phase, it is preferable to freeze the original parameters of the SAM image encoder (ImageEncoder) while keeping the prompt encoder (Prompt Encoder) and mask decoder in a trainable state, and jointly train the relevant parameters of the introduced frequency domain enhancement adapter and multi-scale dilated convolution adapter. Specifically, the `requires_grad` parameter of the original image encoder parameters is set to `False`, while the parameters in the prompt encoder, mask decoder, and adapter modules remain trainable. The prompt encoder participates in training in a no-prompt mode. Trainable parameters include at least: the dimensionality reduction linear layer parameters, the dimensionality increase linear layer parameters, the frequency domain amplitude channel attention parameters, normalization parameters, and scaling parameters in the frequency domain enhancement adapter; and the dimensionality reduction linear layer parameters, the dimensionality increase linear layer parameters, the parameters of each convolutional branch, the scale fusion weights (`scale_weights`), the channel attention parameters, and the spatial attention parameters in the multi-scale dilated convolution adapter. By employing the above strategies, while preserving the original visual representation capabilities of SAM, the overall size of trainable parameters can be effectively reduced, fine-tuning efficiency can be improved, and the model's task adaptability in small-sample medical image segmentation scenarios can be enhanced.

[0063] Step S6: Input the encoder features into the segmentation decoder to output a segmentation mask;

[0064] Step S601: Construct a data loader. Batch loading and online enhancement are preferred to ensure image and mask alignment; for class-imbalanced tasks, a sampling equalization strategy can be used.

[0065] Step S602: Forward Propagation. The input batch of medical images is processed by the SAM image encoder to obtain image embedding features. These features are then input into the SAM mask decoder under no-prompt conditions, outputting a segmentation probability map or mask logits. No point cues, box cues, or mask cues are used during either the training or inference phases.

[0066] Step S603: Loss Function. Preferably, the Dice loss and cross-entropy loss are combined:

[0067]

[0068] Where L represents the total loss function; Indicates Dice loss; This represents the binary cross-entropy loss based on logits; The loss weight coefficient is 0 ≤ ≤1. Dice loss is used to constrain the overlap between the predicted segmented region and the ground truth labeled region, while binary cross-entropy loss is used to constrain the pixel-level foreground / background classification results;

[0069] For tasks with higher boundary requirements, boundary loss or contour constraint loss can be optionally added.

[0070] Step S604: Backpropagation and optimization. The AdamW optimizer is preferably used to update trainable parameters; the learning rate can employ cosine annealing or a piecewise decay strategy; different learning rates can be set for the adapter and decoder to balance convergence speed and stability.

[0071] Step S605: Validation and Early Stopping. Calculate metrics such as Dice and IoU on the validation set every few rounds; when the metrics no longer improve, perform early stopping and save the optimal weights.

[0072] Step S06: Testing and Evaluation. Load the trained model weights, calculate Dice, IoU, precision, recall, and optional boundary F1 scores on the test set, and output statistical results by case / organ.

[0073] Implementation Example 1

[0074] Compared to traditional medical image segmentation methods that primarily process single-modality medical images, this invention proposes a medical image segmentation method based on a frequency domain enhancement adapter and an enhanced multi-scale dilated convolution adapter. By introducing a frequency domain enhancement adapter and an enhanced multi-scale dilated convolution adapter during the image encoding process, and combining a promptless automatic segmentation mechanism with a mask decoding optimization strategy, the method effectively improves the model's feature representation ability for regions with blurred boundaries, significant scale changes, and complex structural details, thereby solving the problem of insufficient accuracy in medical image segmentation.

[0075] To verify the effectiveness of the method of this invention, five publicly available medical image segmentation datasets—Kvasir-SEG, BUSI, Synapse, Chest, and ISIC—were selected for experimental verification, with the Dice coefficient used as the main evaluation metric. The comparison methods covered existing methods such as traditional convolutional neural network segmentation methods, U-Net and its improved models, Transformer-based medical image segmentation models, and SAM-based medical image segmentation adaptation models. In this context, FCN stands for Fully Convolutional Network; Unet stands for U-shaped Network; TransUnet stands for U-Net Network with Transformer structure; SwinUnet stands for U-Net Network with Swin Transformer structure; MedSAM stands for Medical SAM Adapter, a SAM adaptation method for medical image segmentation that improves the adaptability of medical image segmentation by introducing a lightweight adapter module into the SAM; and SAMed stands for Customized Segment Anything Model for Medical Image Segmentation, a SAM model customized for medical image segmentation tasks, which typically employs efficient parameter fine-tuning strategies to improve the segmentation performance of SAM in medical scenarios. The above English abbreviations or names are commonly used model names or algorithm abbreviations in this field. Retaining their English forms is to maintain consistency with publicly available literature and common technical names, avoiding inaccuracies in technical meaning due to direct translation.

[0076] Experimental results show that the method of this invention achieves good segmentation performance on multiple publicly available medical image segmentation datasets. For example, on the Kvasir-SEG dataset, the Dice coefficient of the method of this invention reaches 92.47%, which is higher than FCN's 86.50%, Unet's 78.22%, SwinUnet's 91.01%, and MedSAM's 91.53%. On the Synapse dataset, the method of this invention reaches 95.03%, which is higher than TransUnet's 94.72%, SAMed's 94.64%, and MedSAM's 94.86%. On the ISIC dataset, the method of this invention reaches 92.65%, which is slightly higher than MedSAM's 92.64% and MSA's 92.59%. In addition, on the Chest dataset, the method of this invention reaches 96.81%, which is basically on par with MedSAM's 96.82% and higher than many comparative methods. The above results demonstrate that the present invention, through a frequency domain enhancement adapter and a multi-scale dilated convolution adapter, can effectively improve the ability to represent boundary details and model multi-scale context in medical image segmentation.

Claims

1. A medical image segmentation method based on frequency domain enhancement and multi-scale dilated convolution, characterized in that, Includes the following steps: Step S1: Obtain the medical image to be segmented and perform preprocessing, the preprocessing including at least size normalization and intensity normalization; Step S2: Input the preprocessed medical image into a Transformer-based image encoder to obtain multi-layer features, wherein the image encoder contains several Transformer blocks; Step S3: In the output projection path of the attention module of the Transformer block, the frequency domain enhancement adapter is called to enhance the features to obtain attention-enhanced features; Step S4: In the Transformer block, the multi-scale dilated convolution adapter is called to perform multi-scale context enhancement on the features to obtain context-enhanced features; Step S5: Fuse the attention enhancement features, context enhancement features, and residual features of the Transformer block to output encoder features; Step S6: Input the encoder features into the mask decoder to output the segmented mask image.

2. The method according to claim 1, characterized in that, The frequency domain enhancement adapter includes: performing layer normalization on the input features and reducing the dimensionality of the input features through a first linear mapping to obtain bottleneck features; performing a two-dimensional fast Fourier transform on the bottleneck features to obtain a frequency domain complex spectrum, and decomposing the frequency domain complex spectrum into amplitude components and phase components; performing global pooling and convolution on the amplitude components to obtain channel attention weights, and adaptively modulating and enhancing the amplitude components according to the channel attention weights to improve the response intensity of key frequency components; then combining the enhanced amplitude components with the phase components to reconstruct the frequency domain complex spectrum, and restoring it to the spatial domain through an inverse fast Fourier transform; finally, after batch normalization, random deactivation, and dimensionality increase through a second linear layer, adding it to the residual branch to obtain the output features.

3. The method according to claim 1 or 2, characterized in that, The frequency domain enhancement adapter introduces a deep adaptation feature (deep_adapt) after dimensionality reduction. This deep adaptation feature is obtained by linear projection of features with the same resolution as the backbone and is added to the bottleneck feature before entering the frequency domain enhancement.

4. The method according to claim 1, characterized in that, The multi-scale dilated convolution adapter includes: rearranging the input features into a two-dimensional feature map after layer normalization and linear dimensionality reduction; extracting multi-scale features using a 1×1 convolution branch, a 3×3 depthwise separable dilated convolution branch with an inflation rate of 2, and a 3×3 depthwise separable dilated convolution branch with an inflation rate of 4, wherein the depthwise separable dilated convolution branch includes depthwise dilated convolution and 1×1 pointwise convolution; weighting and fusing the outputs of the three branches using learnable scale weights normalized by softmax, and adding the residuals with the dimensionality-reduced bottleneck features; performing batch normalization, channel attention enhancement, activation, random deactivation, and linear dimensionality increase on the features after residual addition, and further performing region weighting through spatial attention, and obtaining context-enhanced features after scaling.

Images