Medical image segmentation method fusing SAM global modeling and U-Net local optimization

Abstract

The invention provides a medical image segmentation method fusing SAM global modeling and U-Net local optimization, and belongs to the technical field of medical image segmentation of image segmentation. The technical problems that the segmentation precision is insufficient and edge details are lost due to the fact that global semantic features and local detail features cannot be considered at the same time in the segmentation process of a traditional method are solved. According to the technical scheme, the method comprises the following steps of 1, preprocessing a medical image; step 2, constructing a segmentation model fusing a visual Transform and a convolutional network; 3, training the segmentation model by using a supervised learning method with a binary cross entropy loss function; and step 4, inputting an image to be segmented into the trained model, and generating a segmentation result of the medical image. The medical image segmentation method has the beneficial effects that the accuracy and robustness of medical image segmentation are effectively improved by combining the global modeling capability of SAM and the local optimization capability of U-Net.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation, and in particular relates to a medical image segmentation method integrating SAM global modeling and U-Net local optimization. Background Art

[0002] Medical image segmentation is an important technology for clinical diagnosis and treatment. Traditional methods such as threshold segmentation and region growing are limited by complex noise and low contrast problems, resulting in poor segmentation results. In recent years, deep learning-based convolutional neural network (CNN) methods, such as U-Net, have made significant progress by effectively capturing local detail information of images through encoder-decoder structures and skip connections. However, U-Net is limited by the receptive field of convolution operations and has difficulty in modeling global context information.

[0003] The Visual Transformer (ViT) can capture global semantic features through the self-attention mechanism, making up for the shortcomings of CNN, but its ability to model detailed information is weak, the amount of computation is large when processing high-resolution medical images, and the edges of targets are easily blurred.

[0004] How to solve the above technical problems is the subject faced by the present invention. Summary of the invention

[0005] The purpose of the present invention is to provide a medical image segmentation method that integrates SAM global modeling and U-Net local optimization, which has the characteristics of full automation and does not require human intervention during the segmentation process.

[0006] In order to achieve the above-mentioned invention object, the present invention adopts a technical solution: a medical image segmentation method integrating SAM global modeling and U-Net local optimization, comprising the following steps:

[0007] Step 1: preprocessing the medical image, adjusting the image to a uniform resolution, and performing a data enhancement operation; in the step 1, preprocessing the medical image, adjusting the image to a uniform resolution, and performing a data enhancement operation includes the following steps:

[0008] Step 1.1: Image normalization:

[0009] Normalize the pixel value range of the medical image to the interval [0, 1]. The specific processing method is as follows:

[0010] Input medical image Where H is the height, W is the width, and C is the number of channels. Then normalize

[0011] Normalization formula:

[0012]

[0013] Where: I(x, y) represents the original pixel value at (x, y), I min and I max Represent the minimum and maximum pixel values ​​in the original image, I norm (x, y) is the normalized pixel value.

[0014] Finally, the normalized image is output

[0015] Step 1.2: Image resizing: resize the medical image to a fixed target resolution to ensure the consistency of the input size.

[0016] Input normalized medical image I∈R H×W×C Then the image is scaled using bilinear interpolation. The specific process is:

[0017]

[0018] Where: w(x′, i) and w(y′, j) are weight coefficients, calculated based on the linear relationship between pixel coordinates, and (x′, y′) represents the pixel coordinates in the target image.

[0019] Finally, the output image is of uniform size

[0020] Step 1.3: Data augmentation. Perform data augmentation operations on the image and its label (mask), including the following random transformations:

[0021] 1) Randomly flip the image and the corresponding label horizontally with probability P = 0.5:

[0022] I flip (x, y) = I resize (H′-x-1, y)

[0023] 2) Randomly flip the image and label vertically with probability P = 0.5:

[0024] I flip (x, y) = I resize (x, W′-y-1)

[0025] 3) Randomly rotate the image and label within an angle range (plus or minus 15 degrees) and use interpolation to maintain edge information.

[0026] 4) Randomly crop regions from the image and resize to the target resolution.

[0027] 5) Add Gaussian noise to simulate the noise interference in real medical images and increase the robustness of the model to noise:

[0028] Inoise =I resize +N(0,σ 2 )

[0029] in: Indicates that the mean is 0 and the variance is σ 2 Gaussian noise.

[0030] 6) During data augmentation, the label image needs to perform the same transformation operation to ensure that the label is aligned with the image space.

[0031] Step 1.4: Output preprocessing results. After normalization, resizing, and data augmentation, the images and their labels are converted into a unified format:

[0032]

[0033] Where: B is the batch size, C is the number of channels, H' and W' are the target resolutions.

[0034] Preprocessed image I pre and M pre It will be used as input and passed to the subsequent fusion segmentation model for training and reasoning;

[0035] Step 2: Combine the visual Transformer branch with the convolutional network branch, make full use of global semantic features and local detail features, and build an efficient medical image segmentation model. Specifically, it includes the following two core processes:

[0036] Step 2.1: The visual Transformer branch extracts global semantic features. The visual Transformer branch is mainly responsible for extracting the global semantic information of the input medical image. The specific process is as follows:

[0037] First input the preprocessed image The input image is split into a series of non-overlapping image blocks (Patch) through a convolutional layer (16×16 size patch), and each patch is mapped to a high-dimensional feature representation:

[0038] P = Conv2D(I pre , stride = 16)

[0039] Then output the features in: is the number of image patches and D is the embedding feature dimension.

[0040] Next, we add position encoding to each image block and introduce position information so that the Transformer can distinguish the spatial position of each image block:

[0041] Ppas =P+PositionalEncoding

[0042] The feature P with position encoding pos Input to the multi-layer Transformer block for processing. First, the self-attention mechanism is used to capture the long-distance dependencies between image blocks and extract global context information; then the feedforward network (MLP layer) is used to further enhance the feature expression ability. Finally, after being processed by the multi-layer Transformer block, the global semantic features are obtained and output:

[0043] F Tansformer = TransformerBlocks(P pos )

[0044] Finally, the high-dimensional feature F output by Transformer is transformed through a 1×1 convolutional layer Transformer Project to a lower channel dimension:

[0045] F global =Conv2D(F Transformer )

[0046] Output global features:

[0047]

[0048] Step 2.2: The convolutional network branch extracts local detail features. The convolutional network branch is mainly responsible for capturing the details and local features of the input image to supplement the edge and texture information missing from the Transformer branch. The specific process is as follows:

[0049] First input the preprocessed image And the input image is processed by stacking convolution blocks layer by layer (such as GhostConv2D module):

[0050] F local =ConvBlocks(I pre )

[0051] In each convolution block, local detail features are extracted through convolution operations, and then the normalization layer is used to accelerate training and improve feature stability. Finally, the activation function is used to introduce nonlinear feature expression capabilities. In addition, downsampling is used between each convolution layer to gradually reduce the spatial resolution of the feature map and increase the number of channels.

[0052] Then output the final local features:

[0053]

[0054] Step 2.3: Weighted feature fusion. Use a weighted fusion module to combine the global features of the visual Transformer branch and the local features of the convolutional network branch. The specific operations are as follows:

[0055] Input the global features of step 2.1 and the local features of step 2.2, and use the attention mechanism to dynamically assign the importance weights of global features and local features:

[0056] F fusion =α·F global +(1-α)·F local

[0057] The weighted fusion process enables the model to flexibly adjust the importance of global and local information according to the characteristics of the input data. Finally, the fused features are output:

[0058]

[0059] By fusing global and local information, the model’s semantic understanding and edge positioning capabilities of the target area are improved, providing high-quality feature representation for the subsequent decoder module to restore the segmentation results.

[0060] Step 3: Use a binary cross entropy loss function to supervise the segmentation model. The former is used to minimize the cross entropy error between the predicted segmentation result and the true label, and the latter is used to solve the category imbalance problem in the segmentation task and further improve the accuracy of the segmentation result and the generalization ability of the model. The specific process is as follows:

[0061] Step 3.1: Take the preprocessed medical image X and the corresponding true segmentation label Y as input for model training.

[0062] The dimension of the input medical image X is

[0063] The true label Y in the segmentation task is a single-channel binary image with a dimension of

[0064] Step 3.2: Input the input medical image X into the segmentation model that integrates the visual Transformer and the convolutional network. The model gradually extracts features and generates segmentation prediction results through the following main stages.

[0065] First, the input image X is processed by the visual Transformer branch to extract the global semantic features of the image. The image X first passes through the Patch Embedding layer to split the image into patches of fixed size and map them into high-dimensional embedding vectors.

[0066]

[0067] Where P is the size of the Patch and D is the number of Transformer feature channels.

[0068] Feature extraction is performed through multiple layers of Transformer Block. In each Transformer Block, the self-attention mechanism is used to calculate the long-distance dependencies between patches and extract global semantic information. The attention-weighted features are then nonlinearly transformed and enhanced using a feedforward network (MLP).

[0069] F global = Transformer(X′)

[0070] Through the channel projection operation, the feature channel number D output by Transformer is mapped to 256 dimensions, which is convenient for fusion with the convolution branch:

[0071]

[0072] Next, while image X is processed by the visual Transformer branch, image X passes through the convolutional network branch, performs convolution operations and feature downsampling layer by layer, and extracts local detail features of the image. Feature extraction is performed by stacking multiple convolutional layers, and each convolution operation extracts detail information such as edges and textures:

[0073]

[0074] Then the global feature F extracted by the visual Transformer branch is global proj Local features F extracted by the sum-product network branch local Perform weighted fusion to preserve the complementarity of global and local information:

[0075]

[0076] The fusion module here includes a weighted self-attention mechanism to dynamically weight the two features to control the importance of global and local features.

[0077] Finally, the decoder reconstructs the features and the fused features F fusion is input to the decoder, which restores the spatial resolution of the image by progressive upsampling:

[0078]

[0079] Final Output It is the segmentation result predicted by the model, and its dimension is consistent with the true label Y.

[0080] Step 3.3: Loss function calculation. The segmentation result output by the model And the true label Y is input into the binary cross entropy (BCE) loss function to calculate the segmentation error of the current batch:

[0081]

[0082] Where: Y i is the value of the i-th pixel in the true label (0 or 1), The probability value of the i-th pixel predicted by the model.

[0083] Step 3.4: Back propagation and parameter update. Based on the calculated loss value L BCE , the gradient of loss to model parameters is calculated by back-propagation algorithm:

[0084]

[0085] Use an optimizer (such as Adam) to update the model parameters W:

[0086]

[0087] Where: W t represents the current model parameters, W t+1 Represents the updated model parameters.

[0088] Step 3.5: Iterative training. Input all training data into the model in batches, repeat the process of forward propagation, loss calculation, back propagation and parameter update to complete a training cycle (Epoch). Continue to iterate multiple Epochs until the model converges or reaches the specified training round.

[0089] Step 4: Input the image to be segmented into the trained model to generate the segmentation result of the medical image.

[0090] Step 4.1: Input the medical image to be segmented into the trained segmentation model. The image has a uniform resolution and format after preprocessing. The shape of the input image is [B, 3, H, W].

[0091] Step 4.2: Feature extraction, feature extraction is divided into visual Transformer branch and convolutional network branch. The visual Transformer branch is the visual Transformer branch of the model that extracts the global semantic features of the input image through multiple layers of Transformer blocks and outputs a global feature map. At this time, the feature map has been calculated through multiple layers and contains long-distance dependency information and global context semantics. The convolutional network branch is the convolutional network branch of the model that uses multiple convolutional layers to extract local detail features of the input image, such as edges, textures, and local morphological information of small targets, and outputs a local feature map.

[0092] Step 4.3: Weighted fusion module. The global feature map output by the visual Transformer branch and the local feature map output by the convolutional network branch are input into the weighted fusion module. The weighted fusion module balances the importance of the two types of features through weighted operations, so that global features and local features are effectively complemented. The output feature map retains global semantic information and integrates local detail features, which can better locate and segment the target area.

[0093] Step 4.4: Decoder restores resolution. The fused feature map is passed to the decoder module for step-by-step upsampling and restoration.

[0094] The decoder gradually restores the spatial resolution of the feature map through multi-layer upsampling operations, making it close to the original resolution of the input image. At the same time, in each upsampling stage, the decoder and the corresponding encoder output are skipped to integrate high-level abstract semantics and low-level spatial details to improve the accuracy of the segmentation results.

[0095] Step 4.5: Generate segmentation results. In the last layer of the decoder, a 1×1 convolutional layer is used to adjust the number of channels to the number of output categories. In the last layer of the decoder, a 1×1 convolutional layer with a Sigmoid activation function is used to convert the feature map into a probability map. Then, the threshold T=0.5 is set, and the probability map is binarized, and pixels greater than the threshold are marked as the target category (1), and the rest are marked as background (0). The specific operations are as follows:

[0096]

[0097] M binary is the final binary segmentation image.

[0098] Step 4.6: Output the generated probability map as the final medical image segmentation result. The output result is a segmentation map of shape [B, 1, H, W], where the value of each pixel represents the probability that it belongs to the target category.

[0099] Compared with the prior art, the present invention has the following beneficial effects:

[0100] 1. The present invention combines the global modeling capability of the visual Transformer with the local feature extraction capability of the U-Net, which can capture global context information and retain the detailed features of the image, thereby improving the segmentation accuracy.

[0101] 2. The present invention designs a weighted fusion module to dynamically adjust the weights of global features and local features according to the importance of the features, thereby achieving efficient complementarity of multi-scale information and improving the performance of the segmentation model.

[0102] 3. The present invention combines the self-attention mechanism in the visual Transformer to enhance the model's ability to recognize and model the target area, avoid background noise interference, and improve the accuracy of the segmentation results.

[0103] 4. In the present invention, through effective image preprocessing and data enhancement methods, combined with the fusion design of global and local features, the model shows good robustness and generalization ability on medical images of different resolutions and qualities.

[0104] 5. The present invention effectively optimizes the model training process and reduces segmentation errors by introducing the binary cross entropy loss function and supervised learning method, while maintaining computational efficiency, which is suitable for the actual application requirements of medical image segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0106] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0107] Figure 2 This is the flow chart of the visual Transformer branch of the SAM encoder of the present invention.

[0108] Figure 3 This is a flow chart of the U-Net encoder convolutional network branches and decoder of the present invention.

[0109] Figure 4 This is a schematic diagram of the weighted fusion module of the present invention. DETAILED DESCRIPTION

[0110] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0111] Example 1

[0112] In this embodiment, the experiment of this embodiment is implemented on the Windows 11 platform using the Python language, and the computer platform implementing this embodiment has a 4060ti 8g GPU and 32GB of running memory. During the experiment, the model is tested and evaluated on the ISIC2017 public dataset, and compared with other methods using the same dataset.

[0113] The specific implementation steps of the above method are as follows:

[0114] Step 1: Preprocess the medical image, adjust the image to a uniform resolution, and perform data augmentation operations. Figure 1 As shown in the figure, after the medical image is input, the image is first preprocessed to unify the image data format and enhance the diversity of data samples. First, the pixel values ​​of the input medical image are mapped to a uniform range (0 to 1) to eliminate the differences caused by different data sources. Then the input image is adjusted to a uniform resolution through bilinear interpolation. Then, random transformation operations are performed on the image, including horizontal and vertical flipping, random rotation, cropping, etc., and the same transformation is performed on the image mask to ensure that the data is aligned with the label space. Finally, the preprocessed medical image and its corresponding label are output to provide standardized input for subsequent feature extraction and model training.

[0115] Step 2: Construct a segmentation model that integrates the visual Transformer and convolutional networks, where the visual Transformer branch is used to extract global semantic features; the convolutional network branch is used to capture detailed local features; and the weighted fusion module is used to perform weighted fusion of the global features extracted by the visual Transformer branch and the local features extracted by the convolutional network branch. Figure 1 As shown in the figure, after the image preprocessing module, the image is input into the SAM branch (Transformer) and the U-Net branch (convolutional network) in parallel.

[0116] In the SAM branch (visual Transformer encoder), the preprocessed medical image is first input. Then Figure 2 As shown in the figure, the image is divided into patches of fixed size through the Patch Embedding operation and embedded in a high-dimensional space. Then, after adding position encoding, the patch embedding vector passes through the multi-layer Transformer block layer by layer, and the global semantic features are extracted through the self-attention mechanism. Finally, through channel projection and bilinear interpolation operations, the number of channels of the feature map is adjusted to 256, and it is downsampled to the target size, and the global features are output.

[0117] In the U-Net branch (convolutional network encoder), the preprocessed medical image is first input, and local detail features are extracted layer by layer through multi-layer convolution operations. Figure 3 As shown in the figure, the encoder part captures image edge, texture and other detail features through small receptive field convolution operations, and gradually reduces the resolution and increases the number of channels through downsampling operations. Finally, each layer output of the encoder is transmitted to the corresponding position of the decoder through a jump connection to retain detail information and output local features.

[0118] Finally Figure 4As shown in the figure, the global features from the SAM branch and the local features from the U-Net branch are input, weights are dynamically assigned through feature weighting operations, and the advantages of global features and local features are combined to generate fused features and output them.

[0119] Step 3: Use supervised learning with a binary cross entropy loss function to train the segmentation model to minimize the cross entropy error between the predicted segmentation results and the true labels. Figure 3 As shown in the figure, the fused features output from step 2 are input to the decoder module. The decoder restores the spatial resolution of the feature map by upsampling layer by layer. At each decoding layer, a skip connection is used to combine the intermediate features of the U-Net encoder with the current decoder features to ensure the fusion of high-level semantics and low-level detail information. The last layer of the decoder maps the number of feature channels to 1 channel through 1X1 convolution to generate a probability map. Finally, the Sigmoid activation function is used to normalize the output results to generate a pixel-level segmentation probability map.

[0120] Step 4: Input the image to be segmented into the trained model to generate the segmentation result of the medical image. Set the threshold T = 0.5 and convert the probability map into a binary segmentation result. The generated segmentation result is a binary mask map, and the segmentation result is compared with the true label to calculate the Dice coefficient, IoU and other indicators to evaluate the segmentation performance.

[0121] The present embodiment is compared with the current mainstream skin disease image segmentation model on the ISIC2017 dataset, and the results are shown in Table 1.

[0122] Table 1 Performance comparison of segmentation models on the ISIC2017 dataset

[0123] Segmentation Model QUR DSC Acc UNet 76.98 86.99 95.65 TransFuse 79.21 88.40 96.17 UTNetV2 77.35 87.23 95.84 UNeXt-S 78.26 87.80 95.95 The method of this embodiment 80.48 90.13 96.18

[0124] Table 1 compares the performance of different segmentation models on the ISIC 2018 dataset, and the evaluation indicators include Dice similarity coefficient (DSC), mean intersection over union (mIoU) and accuracy (Acc). From the results, it can be seen that the method of this embodiment is superior to other comparison models in terms of mIoU, DSC and Acc. Specifically, mIoU reaches 80.48, which is about 3.5% higher than U-Net, indicating that this method has significantly improved the segmentation accuracy of the target area and the background. The DSC index is 90.13, which is 1.7% to 3.14% higher than other methods, reflecting the advantages of this method in maintaining the shape and integrity of the target area. The Accuracy reaches 96.18, slightly higher than TransFuse and UNeXt-S, showing that this method performs well in the accuracy of overall pixel segmentation.

[0125] This embodiment combines the global feature modeling of SAM with the local detail optimization of U-Net and uses a weighted fusion module to achieve efficient feature complementarity, which not only ensures the global semantic understanding of the segmentation, but also enhances the retention of local detail information, thereby achieving the best segmentation effect in multiple evaluation indicators, verifying the effectiveness and superiority of this method.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical image segmentation method integrating SAM global modeling and U-Net local optimization, characterized in that: The following steps are involved: Step 1: Preprocess the medical image, adjust the image to a uniform resolution, and perform data augmentation operations; Step 2: Construct a segmentation model that integrates the visual Transformer and convolutional networks, where the visual Transformer branch is used to extract global semantic features; the convolutional network branch is used to capture detailed local features; the global features extracted by the visual Transformer branch are weightedly fused with the local features extracted by the convolutional network branch through the weighted fusion module; Step 3: Train the segmentation model using a supervised learning method with a binary cross entropy loss function to minimize the cross entropy error between the predicted segmentation results and the true labels; Step 4: Input the image to be segmented into the trained model to generate the segmentation result of the medical image.

2. According to claim 1, a medical image segmentation method integrating SAM global modeling and U-Net local optimization is characterized in that: The step 1 specifically includes the following steps: Step 1.1: Normalize the medical image and map the pixel values ​​to a uniform range (0-1) to eliminate the impact of differences in data sources; Step 1.2: Adjust the image to a uniform target resolution through bilinear interpolation; Step 1.3: Use data augmentation technology to perform random transformation operations on the image, including random flipping, rotation, and cropping. For the label mask in the image segmentation task, the same preprocessing operation as the image needs to be performed to ensure the spatial alignment of the label and the image; Step 1.4: After this step, the output is the preprocessed medical image and its corresponding label in a unified format.

3. The medical image segmentation method integrating SAM global modeling and U-Net local optimization according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: The visual Transformer branch receives the preprocessed medical image as input and gradually extracts the global semantic features of the image through multiple layers of Transformer blocks. In each Transformer block, the self-attention mechanism is used to calculate the long-range dependencies between various regions in the image, thereby capturing the global context information of the image. Step 2.2: The convolutional network branch takes the preprocessed medical image as input and extracts local detail features by stacking multiple convolutional layers. In each convolution operation, a small receptive field is used to capture local information in the medical image. Step 2.3: Through the weighted fusion module, the global features extracted by the visual Transformer branch are dynamically weighted and fused with the local features of the convolutional network branch to achieve effective complementarity between global context and local details and accurately represent the target area; Step 2.4: The fused features are passed to the decoder module to gradually restore the spatial resolution of the image and generate the final segmentation result.

4. The medical image segmentation method integrating SAM global modeling and U-Net local optimization according to claim 1, characterized in that: The step 3 specifically includes the following steps: Step 3.1: Initialize the parameters of the segmentation model. During the model initialization phase, assign initial weights to the visual Transformer branch and the convolutional network branch to ensure that the model starts training from a consistent parameter distribution. Step 3.2: Define the training objective, that is, minimize the error between the model's predicted segmentation result and the true label, and use the binary cross-entropy loss function Binary Cross-Entropy Loss, BCE Loss as the optimization objective, and its formula is: Where: N represents the total number of all pixels in the segmented image; Y i is the value of the i-th pixel in the true label, 0 or 1, is the probability value of the i-th pixel predicted by the model; Step 3.3: Input the preprocessed training samples to the segmentation model. The training data includes medical images and the corresponding ground truth masks. The model first extracts global features through the visual Transformer branch, then extracts local detail features through the convolutional network branch, and finally generates predicted segmentation results through the weighted fusion module and the decoder module. Step 3.4: Calculate the error between the predicted result and the true label. For each training sample, use the above formula to calculate the binary cross entropy loss value to measure the accuracy of the model prediction. Step 3.5: Update the model parameters. Use the gradient descent-based optimization algorithm to update the model parameters to minimize the value of the binary cross entropy loss function. The specific parameter update formula is: in: θ t+1 represents the updated model parameters; θ t Represents the current model parameters; η represents the learning rate; Represents the loss gradient of the current parameter; Step 3.6: Repeat the above process until the training termination condition is reached.

Images