Medical image segmentation method based on fusion of CNN and lightweight transformer

Abstract

The application discloses a medical image segmentation method based on CNN and light-weighted Transformer fusion, and relates to the technical field of image segmentation.Compared with previous medical image segmentation methods, the application solves the problems of insufficient long-distance dependence modeling and insufficient multi-scale feature fusion of a traditional convolutional neural network; through enhancing local feature extraction and global feature modeling capability, precise segmentation of complex shape and fuzzy boundary tumor regions is realized; the STABlock module is adopted, deep convolution and element-by-element multiplication are combined to enhance feature expression capability, and mixed weighted pooling is used to reduce feature information loss; a dual attention mechanism module of an LT module is introduced to capture long-distance dependence and improve global semantic expression of segmentation; a light-weighted LT module is used, a local feature receiver is used to optimize local detail extraction, and a feedback module is combined to reduce redundant calculation.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a medical image segmentation method based on the fusion of CNN and lightweight Transformer. Background Technology

[0002] Medical image segmentation is a crucial task in modern medical image analysis, widely applied in disease diagnosis, surgical planning, and treatment efficacy evaluation. By segmenting tumor regions in medical images, doctors can accurately locate lesions and quantify their extent, thus providing strong evidence for clinical diagnosis and treatment. However, due to the complexity of medical images—such as the diverse morphologies of tumors, blurred boundaries, high similarity to surrounding tissues, and interference from image noise—tumor segmentation has always faced significant challenges.

[0003] Currently, deep learning-based segmentation methods have become mainstream. Compared with traditional rule-based segmentation methods, they can automatically extract features and achieve higher segmentation accuracy. However, these methods also have certain limitations, including insufficient ability to handle boundary conditions, low model efficiency, and inadequate optimization for multi-scale feature fusion.

[0004] Existing technologies propose a classic convolutional neural network architecture—Unet—specifically designed for medical image segmentation. Its key feature is achieving accurate segmentation by progressively downsampling and upsampling the input image and combining multi-scale features. Unet has a symmetrical "U"-shaped structure, comprising an encoder and a decoder. The encoder consists of a series of convolutional and pooling layers, using max pooling to progressively reduce the spatial resolution of the feature map while increasing the number of feature channels for extracting high-level semantic features. The decoder progressively restores the spatial resolution through deconvolutional layers or upsampling operations, fusing the encoder's features through skip connections to retain fine-grained spatial information and enhance boundary segmentation accuracy.

[0005] While the above methods can meet the needs of most scenarios, research and practical application of these methods and existing technologies have revealed at least the following shortcomings:

[0006] 1. Insufficient modeling of long-distance dependencies: Traditional convolutional neural networks (such as UNet) rely on fixed-size convolutional kernels to extract local features, making it difficult to capture long-distance dependencies between pixels. This is especially true in medical imaging of tumors, where it is necessary to identify complex tumor shapes or long-distance contextual information, resulting in limited performance.

[0007] 2. Insufficient multi-scale feature fusion: In the process of multi-scale feature fusion, the traditional skip connection mechanism may lead to conflicts between semantic and spatial information, especially when high-level semantic features and low-level detailed features are combined, which may result in ambiguity or deviation.

[0008] To address the aforementioned issues, this invention proposes a medical image segmentation method based on the fusion of CNN and lightweight Transformer. By improving existing UNet and TransUNet models, an innovative architecture, STRNet, combining CNN and Transformer, is designed to solve problems in medical image segmentation such as insufficient long-range dependencies, defects in multi-scale feature fusion, excessive computational complexity, and weak local detail extraction capabilities. Simultaneously, through lightweight design and robustness enhancement, the model's application value in real-world clinical scenarios is improved, meeting the demands for efficient and accurate tumor segmentation in medical images. Summary of the Invention

[0009] The purpose of this invention is to propose a medical image segmentation method based on the fusion of CNN and lightweight Transformer to solve the problems mentioned in the background art:

[0010] 1. Insufficient modeling of long-distance dependencies: Traditional convolutional neural networks (such as UNet) rely on fixed-size convolutional kernels to extract local features, making it difficult to capture long-distance dependencies between pixels. This is especially true in medical imaging of tumors, where it is necessary to identify complex tumor shapes or long-distance contextual information, resulting in limited performance.

[0011] 2. Insufficient multi-scale feature fusion: In the process of multi-scale feature fusion, the traditional skip connection mechanism may lead to conflicts between semantic and spatial information, especially when high-level semantic features and low-level detailed features are combined, which may result in ambiguity or deviation.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A medical image segmentation method based on the fusion of CNN and lightweight Transformer includes the following steps:

[0014] Acquire medical images and preprocess them, then input the preprocessed medical images into the medical image tumor segmentation model;

[0015] Multi-scale medical image features are obtained by performing multi-scale feature extraction and downsampling on the preprocessed medical images input by the encoder.

[0016] The medical image fusion features are obtained by fusing features from multi-scale medical image features through a decoder and restoring the feature resolution.

[0017] Lightweight optimization of multi-scale medical image features is performed based on a lightweight module.

[0018] The segmented tumor structure is output based on the medical imaging tumor segmentation model.

[0019] Preferably, the encoder is composed of several stacked STABlock modules. In the STABlock module, depthwise convolution, element-wise multiplication and hybrid pooling are combined to extract multi-scale medical image features.

[0020] Preferably, the hybrid pooling method weights the results of max pooling and average pooling, and dynamically adjusts the weights using proportional weights; specifically as follows:

[0021] Y=&Max(x)+(1-&)Avg(x),&∈[0,1]

[0022] Where Y is the hybrid pooling output; & is the proportional weight; Max(x) is the max pooling result; and Avg(x) is the average pooling result.

[0023] Preferably, the decoder gradually recovers the feature resolution through several layers of TFM and fuses the multi-scale features output by the encoder; the TFM fuses spatial and semantic information through feature fusion operations, and combines downsampling and upsampling mechanisms, as well as multi-scale feature integration methods to perform feature fusion of medical image features to obtain medical image fusion features.

[0024] Preferably, in the downsampling and upsampling mechanism, downsampling is performed by max pooling, and upsampling is performed by bilinear interpolation.

[0025] Preferably, the lightweight module includes an LT module;

[0026] The LT module includes a local feature receiver, a dual attention mechanism module, and a forward feedback module. The local feature receiver is used to receive the input data of the LT module. The output of the local feature receiver is connected to the dual attention mechanism module. The output of the dual attention mechanism module is added to the input of the dual attention mechanism module and used as the input of the forward feedback module. The forward feedback module adds the input of the forward feedback module and the output of the forward feedback module and used as the output of the LT module.

[0027] The local feature receiver is used to extract local detail features of the fused features by employing deep convolution combined with a residual structure.

[0028] The dual attention mechanism module is used to optimize the weight distribution of local features while modeling the global context using a dual attention mechanism. The dual attention mechanism includes feature aggregation and feature distribution. The feature aggregation is used to generate a global descriptor and capture the relationship between distant pixels. The feature distribution is used to reallocate weights and enhance the feature representation of the tumor region.

[0029] The forward feedback module is used to combine depthwise separable convolutional layers, convolutional layers, and activation functions to perform feature transformation on the output of the dual attention mechanism module.

[0030] Compared with existing technologies, this invention provides a medical image segmentation method based on the fusion of CNN and lightweight Transformer, which has the following beneficial effects:

[0031] This invention achieves accurate segmentation of tumor regions with complex shapes and blurred boundaries by enhancing local feature extraction and global feature modeling capabilities. It employs the STABlock module, combining depthwise convolution and element-wise multiplication to enhance feature representation, while reducing feature information loss through hybrid weighted pooling. The dual attention mechanism of the LT module captures long-distance dependencies, improving the global semantic representation of the segmentation. This invention maintains high segmentation performance while reducing computational resource requirements, making the model suitable for resource-constrained scenarios. It uses a lightweight LT module, optimizing local detail extraction through a local feature receiver, and reduces redundant computation by combining it with a feedback module. Attached Figure Description

[0032] Figure 1 This is the overall network architecture diagram mentioned in Embodiment 1 of the present invention;

[0033] Figure 2 This is a schematic diagram of the STABlock module structure mentioned in Embodiment 1 of the present invention;

[0034] Figure 3 This is a schematic diagram of the hybrid pooling structure mentioned in Embodiment 1 of the present invention;

[0035] Figure 4 This is a schematic diagram of the TFM module structure mentioned in Embodiment 1 of the present invention;

[0036] Figure 5 This is a schematic diagram of the local feature receiver structure mentioned in Embodiment 1 of the present invention;

[0037] Figure 6 This is a schematic diagram of the dual attention mechanism module structure mentioned in Embodiment 1 of the present invention;

[0038] Figure 7 This is a schematic diagram of the Double-attention structure mentioned in Embodiment 1 of the present invention;

[0039] Figure 8 This is a schematic diagram of the forward feedback module structure mentioned in Embodiment 1 of the present invention;

[0040] Figure 9 This is an overall framework diagram of the LT module mentioned in Embodiment 1 of the present invention. Detailed Implementation

[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0042] This invention achieves accurate segmentation of tumor regions with complex shapes and blurred boundaries by enhancing local feature extraction and global feature modeling capabilities. It employs the STABlock module, combining depthwise convolution and element-wise multiplication to enhance feature representation, while reducing feature information loss through hybrid weighted pooling. A dual attention mechanism module of the LT module is introduced to capture long-distance dependencies, improving the global semantic representation of the segmentation. This invention reduces computational resource requirements while maintaining high segmentation performance, making the model suitable for resource-constrained scenarios. A lightweight LT module is used to optimize local detail extraction through a local feature receiver, and a feedback module is combined to reduce redundant computation. Specifically, it includes the following:

[0043] Example 1:

[0044] Please see Figure 1-9 This invention relates to a medical image segmentation method based on the fusion of CNN and lightweight Transformer, comprising:

[0045] Medical images are acquired and preprocessed, then input into a medical imaging tumor segmentation model; the details are as follows:

[0046] Based on acquisition devices such as X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET), medical image data is acquired. The acquired medical image data is then converted into a format that can be processed by the medical image tumor segmentation model. The converted medical image data is then subjected to noise reduction processing, such as mean filtering, median filtering, and Gaussian filtering. Data contrast can also be enhanced and normalized to improve the processing efficiency of the subsequent medical image tumor segmentation model.

[0047] Multi-scale medical image features are obtained by performing multi-scale feature extraction and downsampling on the preprocessed medical images input by the encoder.

[0048] The structure of the medical imaging tumor segmentation model can be referenced. Figure 1 This includes modules such as STABlock, TFM, and LT, which process the preprocessed medical images to obtain the final segmentation structure. (See reference...) Figure 2The encoder is mainly composed of multiple stacked STABlock modules, used for efficient extraction of multi-scale features from preprocessed medical images. In STABlock, the input preprocessed medical image undergoes a 7x7 depthwise separable convolution, effectively reducing model parameters and computational cost; it also focuses on local features and enhances the non-linear expressive power between features through element-wise feature multiplication. After the depthwise separable convolution, the preprocessed medical image is divided into two paths: one path undergoes a 1x1 convolution, outputting a feature dimension of ratio*dim; the other path undergoes another 1x1 convolution, also outputting a feature dimension of ratio*dim, where ratio is a scaling factor. The output of one of the 1x1 convolutions is activated by the ReLU6 function. The ReLU6 function can be defined as y = min(max(0,x),6), which limits the output range of neurons and prevents the gradient vanishing problem. The preprocessed medical image after the ReLU6 activation function is multiplied by the features from the other path, and then undergoes another 1x1 convolution, outputting a feature dimension of dim. Then, it goes through a 7x7 depthwise separable convolution (Dwconv), and is then added to the original input, i.e. the preprocessed medical image, to obtain the final output.

[0049] Simultaneously, the encoder combines max pooling (preserving salient features) and average pooling (suppressing noise) by dynamically adjusting the proportional weights to reduce information loss during pooling. The definitions are as follows:

[0050] Y=&Max(x)+(1-&)Avg(x),&∈[0,1]

[0051] Where Y is the hybrid pooling output; & is the proportional weight; Max(x) is the max pooling result; Avg(x) is the average pooling result;

[0052] For details, please refer to Figure 3 In the model, a maximum pooling ratio of 0.8 and an average pooling ratio of 0.2 are used. This hybrid pooling method changes the pooling adjustment rules by proportionally allocating pooling, which to some extent solves the problems encountered when maximum pooling and average pooling are used alone.

[0053] The medical image fusion features are obtained by fusing features from multi-scale medical image features through a decoder and restoring the feature resolution.

[0054] The decoder gradually recovers the feature resolution through multiple layers of TFM (Feature Fusion Module) and fuses the multi-scale features output by the encoder. Specifically:

[0055] For details, please refer to the TFM module. Figure 4Input data enters the TFM module. The module has three main input paths, which first undergo a series of operations, including 1x1 convolution (conv1x1), downsampling, and upsampling, to ensure consistent channel count and resolution, resulting in three new feature maps: C1, C2, and C3. These three feature maps are then concatenated along the channel dimension. Next, three spatial attention weights are generated using 1x1 convolution and the softmax function, and then multiplied element-wise with C1, C2, and C3 respectively, with the results summed. By downsampling and upsampling feature maps of different resolutions and then concatenating them, spatial and semantic information is fused through a 1x1 convolutional layer, introducing a residual connection structure to avoid information loss. Max pooling is used for downsampling to reduce high-resolution features to low-resolution features; bilinear interpolation is used for upsampling to restore feature resolution.

[0056] The merged data then undergoes a series of operations, including 1x1 convolution and softmax operations. During this process, the data is further processed through feature fusion at multiple nodes (A1, A2, A3, B1, B2, B3). After this series of processing and fusion, the final output is an F1 score. In the high-resolution feature recovery process, shallow features from the encoder are fused with progressively upsampled features from the decoder.

[0057] Lightweight optimization of multi-scale medical image features is performed based on a lightweight module; details are as follows:

[0058] The lightweight module uses the LT module as its core component and mainly includes a local feature receiver, a dual attention mechanism module, and a forward feedback module.

[0059] The Local Feature Receiver (LTM) employs depthwise convolution combined with a residual structure to effectively extract local detail features from an image. For details, please refer to [link / reference needed]. Figure 5 Local Feature Receivers (LTMs) include depthwise separable convolution operations with a kernel size of 3x3, which decompose standard convolution operations into depthwise convolutions and pointwise convolutions, effectively reducing the number of model parameters and computational cost. They also include the ReLU activation function, which introduces non-linear characteristics to help the model learn complex patterns. Finally, through addition operations, such as residual connections, the input is directly added to the output after a series of operations. This helps solve the gradient vanishing problem in deep neural networks, allowing networks to be stacked deeper without significantly degrading performance.

[0060] The Dual Attention Mechanism (DAM) module employs a dual attention strategy (feature aggregation and feature distribution) to optimize the weight distribution of local features while modeling the global context. Feature aggregation primarily generates a global descriptor, capturing the relationships between distant pixels; feature distribution mainly reallocates weights, strengthening the feature representation of important regions. For example... Figure 6 , 7 As shown.

[0061] Reference Figure 6 The DAM divides the input into two parts. One part undergoes channel dimensionality reduction through 1x1 convolution, and then passes it through a dual attention mechanism for feature aggregation and distribution to enhance feature representation. The resulting output is residually concatenated with the channel-dimensionality-reduced output, and then subjected to another 1x1 convolution for channel dimensionality reduction to restore the original dimensionality. The other part directly passes through the STABlock module (used in the encoder above), whose output is residually concatenated with the output from the dual attention mechanism. Finally, the output is passed through the ReLU activation function to introduce non-linear features, yielding the final output.

[0062] Reference Figure 7 The input to the dual attention mechanism is a tensor of shape c, h, w, where c is the number of channels, and h and w are the height and width, respectively.

[0063] The input tensor is first subjected to different 1x1 convolution operations (conv1x1) to obtain three different tensors: A: shape c, h, w; B: shape c_n, h, w; V: shape c_n, h, w. The dimensionality-reduced feature A is then reshaped to obtain tempA(c_m, h*w). B and C are then reshaped and normalized using the Softmax function to obtain attention maps (c_n, h*w) and attention vectors (c_n, h*w). tempA is multiplied by the attention maps to obtain a global descriptor. The global descriptor is then multiplied by the attention vectors to obtain a new feature representation Z. Finally, the resolution and channel dimensions of this new representation Z are adjusted to obtain an output with the same resolution and channel dimensions as the input. This mechanism effectively captures long-range dependencies in the feature maps, enhances the representational power of the features, and thus improves the model's performance.

[0064] The structure of the Feedforward Module (CFB) can be referred to Figure 8The input to CFB is a feature map with 256 channels (C). The input feature map first passes through a 3x3 depthwise separable convolution (DW3x3). Depthwise separable convolution effectively reduces the number of parameters and computational cost of the model; the output feature map still has 256 channels. The feature map after the depthwise separable convolution then enters two 1x1 convolutional layers (1x1conv). The first 1x1 convolutional layer expands the number of channels from 256 to 1024. The second 1x1 convolutional layer similarly expands the number of channels from 256 to 1024. After the first 1x1 convolutional layer, the feature map enters the ReLU activation function (ReLU), which outputs 1024 channels. The feature map after the ReLU activation function is multiplied by the feature map after the second 1x1 convolutional layer. The summed feature map then passes through another 1x1 convolutional layer, reducing the number of channels from 1024 to 256, restoring it to the same number of channels as the input.

[0065] The structure of LT can be referenced. Figure 9 The LTM (Local Feature Transformer) is located at the top of the LT (Low-Level Model) and is used to extract local features. The input of the DAM (Distributed Aspect Model) is connected to the output of the LTM, and it is used to perform attention processing on the features to enhance their expressive power. The output of the DAM is split into two paths: one path is directly connected to the input of the CFB (Concurrent Feature Buffer), and the other path is connected to the input of the CFB after an addition operation (+). The CFB (Concurrent Feature Buffer) is located below the DAM and is used for feedback and fusion between features. The output of the CFB is then added again to become the output of the entire LT module. Through the collaborative work of multiple sub-modules, the LT module achieves multi-level processing and optimization of input features, thereby improving the performance of the model.

[0066] The model outputs segmented tumor structures based on medical imaging tumor segmentation. The output is typically an image or labeled map of the same size as the input image, with the tumor region clearly marked. This output can be a binary image (1 for tumor regions, 0 for non-tumor regions) or a probability map (the probability that each pixel belongs to the tumor region). Doctors or researchers can use these outputs to further analyze the tumor's location, size, shape, and other information to aid in diagnosis and treatment.

[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on the fusion of CNN and lightweight Transformer, characterized in that, Includes the following steps: Acquire medical images and preprocess them, then input the preprocessed medical images into the medical image tumor segmentation model; The encoder performs multi-scale feature extraction and downsampling on the preprocessed medical images to obtain multi-scale medical image features. The encoder is composed of several stacked STABlock modules. In the STABlock module, depthwise convolution, element-wise multiplication and hybrid pooling are combined to extract multi-scale medical image features. The hybrid pooling method weights the results of max pooling and average pooling, and dynamically adjusts the weights using proportional weights; specifically as follows: in, For hybrid pooling output; For proportional weighting; This is the result of max pooling; This is the result of average pooling; The decoder performs feature fusion on multi-scale medical image features to obtain fused features and restore feature resolution, thus obtaining medical image fused features. The decoder gradually restores feature resolution through several layers of TFM and fuses the multi-scale features output by the encoder. The TFM fuses spatial and semantic information through feature fusion operations, and combines downsampling and upsampling mechanisms, as well as multi-scale feature integration methods to perform feature fusion on medical image features, thus obtaining medical image fused features. In the downsampling and upsampling mechanisms, downsampling is performed using max pooling, and upsampling is performed using bilinear interpolation. Lightweight optimization of multi-scale medical image features is performed based on a lightweight module. The lightweight module includes an LT module; The LT module includes a local feature receiver, a dual attention mechanism module, and a forward feedback module. The local feature receiver is used to receive the input data of the LT module. The output of the local feature receiver is connected to the dual attention mechanism module. The output of the dual attention mechanism module is added to the input of the dual attention mechanism module and used as the input of the forward feedback module. The forward feedback module adds the input of the forward feedback module and the output of the forward feedback module and used as the output of the LT module. The local feature receiver is used to extract local detail features of the fused features by employing deep convolution combined with a residual structure. The dual attention mechanism module is used to optimize the weight distribution of local features while modeling the global context using a dual attention mechanism. The dual attention mechanism includes feature aggregation and feature distribution. The feature aggregation is used to generate a global descriptor and capture the relationship between distant pixels. The feature distribution is used to reallocate weights and enhance the feature representation of the tumor region. The forward feedback module is used to combine the depthwise separable convolutional layer, the convolutional layer and the activation function to perform feature transformation on the output of the dual attention mechanism module; The segmented tumor structure is output based on the medical imaging tumor segmentation model.

Images