Medical image segmentation method based on context information and multi-scale feature fusion

Abstract

The application belongs to the technical field of medical image segmentation, and specifically discloses a medical image segmentation method based on context information and multi-scale feature fusion, which comprises the following steps: acquiring medical image data and constructing a medical image segmentation model with a double-U-shaped encoder-decoder architecture; inputting the medical image data into a first U-shaped network for feature extraction to obtain multi-scale context features; element-wise multiplying the medical image data and the multi-scale context features; inputting the element-wise multiplication result into a second U-shaped network for global information modeling and local boundary feature enhancement to output final fusion features; and splicing the multi-scale context features and the final fusion features to obtain a medical image segmentation result. The application solves the problems of low segmentation accuracy, high computational complexity, poor segmentation effect on small polyps and inability to achieve efficient segmentation of existing medical image segmentation methods when segmenting polyps.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation technology, specifically relating to a medical image segmentation method based on the fusion of contextual information and multi-scale features. Background Technology

[0002] The gastrointestinal tract is a vital component of the human digestive system, and gastrointestinal polyps are common abnormalities with a potential risk of malignant transformation. Accurate polyp segmentation is crucial for cancer prevention and early screening. Timely detection of malignant polyps and appropriate treatment are essential. For high-risk polyps, medical image segmentation is necessary to accurately formulate preoperative plans and determine the most suitable surgical methods and treatment strategies. However, the extremely small size, indistinct boundaries, and low contrast of micropolyps, coupled with their susceptibility to occlusion or confusion within the complex background of the gastrointestinal tract, present significant challenges to automated segmentation.

[0003] With the widespread application of deep learning technology, CNNs have become a mainstream segmentation method. Leveraging their powerful feature extraction capabilities, CNNs can automatically learn useful information from images, especially in complex and varied polyp images. However, despite the significant achievements of CNNs in image processing, U-Net, as a typical encoder-decoder architecture, provides a powerful tool for medical image segmentation tasks. In recent years, the visual Transformer has made significant progress in medical image segmentation tasks due to its superior long-range dependency modeling capabilities. Compared to CNNs, which mainly rely on local convolution operations, the Transformer computes global feature correlations through a self-attention mechanism, making the model more accurate in capturing small targets and fine-grained features.

[0004] While the aforementioned methods significantly improve the performance of medical image segmentation, they typically require substantial computational resources, making them difficult to deploy in resource-constrained environments. Furthermore, most existing methods still suffer from insufficient feature representation of small targets, loss of boundary details, and difficulty in effectively separating polyps from the background. Therefore, current research should focus on designing a lightweight, efficient, and small-target-sensitive medical image segmentation method that can accurately identify small polyps in complex backgrounds while reducing computational overhead, thus enabling broader clinical applicability. Summary of the Invention

[0005] The purpose of this invention is to address the problems of low segmentation accuracy, high computational complexity, and poor segmentation effect on small polyps in existing medical image segmentation methods, which cannot achieve efficient segmentation. This invention proposes a medical image segmentation method based on the fusion of contextual information and multi-scale features.

[0006] The technical solution of this invention is: a medical image segmentation method based on contextual information and multi-scale feature fusion, comprising the following steps:

[0007] Medical image data of gastrointestinal polyps were acquired, and a medical image segmentation model was constructed. The medical image segmentation model adopted a dual U-shaped encoder-decoder architecture, which included a first U-shaped network and a second U-shaped network connected in sequence.

[0008] Medical image data is input into the first U-shaped network for feature extraction to obtain multi-scale contextual features;

[0009] Multiply medical imaging data and multi-scale contextual features element-wise;

[0010] The result of element-wise multiplication is input into a second U-shaped network for global information modeling and local boundary feature enhancement, and the final fused feature is output.

[0011] The multi-scale contextual features and the final fusion features are concatenated to obtain the medical image segmentation results of gastrointestinal polyps.

[0012] Preferably, the first U-shaped network includes a first encoder, a lightweight hollow space pyramid pooling module, and a first decoder connected in sequence; the first encoder and the first decoder are connected in skip connections.

[0013] The jump connection is equipped with a coordinate attention mechanism, which captures cross-channel information, direction-aware features, and position-sensitive features, and retains spatial information during feature recovery.

[0014] The first encoder is a VGG19 pre-trained network used to receive medical image data and perform multi-level feature extraction to obtain multi-level feature maps.

[0015] The lightweight hollow spatial pyramid pooling module is used to receive the multi-level feature map and perform multi-level convolution processing on the multi-level features to improve the perception ability of local regions.

[0016] The first decoder is used to receive the multi-level feature map and the multi-level feature map after multi-level convolution processing, and then recover and refine the spatial information of the medical image data feature map to obtain multi-scale context features.

[0017] Preferably, the coordinate attention mechanism retains spatial information during feature recovery by capturing cross-channel information, orientation-aware features, and position-sensitive features, specifically:

[0018] Global average pooling is performed on the multi-level feature map. The features after global average pooling are projected onto the horizontal and vertical directions respectively, and global spatial information in the horizontal and vertical directions is captured respectively to obtain horizontal projection features and vertical projection features.

[0019] The horizontal and vertical projection features are concatenated to obtain a joint feature map that integrates spatial information from both directions. The number of channels in the concatenated joint feature map is then compressed using a 1*1 lightweight convolution.

[0020] The Canny edge detection operator is applied to detect edge regions of specific intensities in the joint feature map of spatial information after channel compression, while suppressing irrelevant background regions, to obtain a feature map after edge detection.

[0021] The feature map after edge detection is normalized to eliminate gradient instability during training, and a non-linear activation function is used to activate the normalized features to enhance the expression of non-linear edge features.

[0022] The activated features are segmented, and the segmented features are divided into high-frequency feature maps and low-frequency feature maps by Fourier transform. The high-frequency feature maps contain the target boundary features and detailed information of complex shapes, while the low-frequency feature maps contain the global contour information of the target.

[0023] By convolution, the original number of channels in the high-frequency and low-frequency parts is restored, ensuring that the high-frequency and low-frequency feature maps maintain the same dimension as the multi-level feature maps of the original input of the skip connections. Then, the Sigmoid activation function is used to generate channel weights for the high-frequency and low-frequency feature maps. Through the generated channel weights, the contribution of each channel is dynamically adjusted to enhance the information expression of important channels.

[0024] The generated channel weights are multiplied element-wise with the multi-level feature map, which effectively integrates the detailed information of the target with the global semantic information.

[0025] Preferably, the lightweight hollow spatial pyramid pooling module includes a first convolutional branch, a second convolutional branch, a third convolutional branch, a fourth convolutional branch, and a lightweight attention channel; the input terminals of the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch are all input terminals of the lightweight hollow spatial pyramid pooling module, and the output terminals of the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch are all connected to the input terminal of the lightweight attention channel;

[0026] The lightweight attention channel comprises a first two-dimensional convolutional layer, a batch normalization layer, and a first ReLU activation layer connected in sequence.

[0027] Preferably, the second U-shaped network includes a second encoder, a gated self-attention feature fusion module, and a second decoder; the second encoder and the second decoder are connected in a skip connection.

[0028] The input of the gated self-attention feature fusion module is connected to the other output of the first encoder and the output of the second encoder, and the output of the gated self-attention feature fusion module is connected to the input of the second decoder.

[0029] Preferably, the second U-shaped network performs global information modeling and local boundary feature enhancement, specifically as follows:

[0030] The result of element-wise multiplication of medical image data and multi-scale contextual features is input into the second encoder to extract global spatial features.

[0031] Multi-level feature maps and global spatial features are input into a gated self-attention feature fusion module for weighted fusion to obtain weighted fused features.

[0032] The weighted fused features are input into the second decoder, and the weighted fused features are upsampled by bilinear interpolation to obtain the upsampled features;

[0033] The attention weights are obtained by calculating the cross-attention between the upsampled features and the skip connections between the second encoder and the second decoder.

[0034] The attention weights are summed with the upsampled features, and then upsampled again to obtain the final fused features.

[0035] Preferably, the gated self-attention feature fusion module includes a self-attention mechanism submodule, a gated mechanism submodule, and a channel attention mechanism submodule;

[0036] The self-attention mechanism submodule is used to receive the multi-level feature map and the global spatial features, calculate the attention weights, and perform weighted processing on the input features according to the attention weights to obtain self-attention weighted features.

[0037] The channel attention mechanism submodule is used to receive the multi-level feature map and the global spatial features, and to perform channel weighting on the input features to obtain channel attention weighted features;

[0038] The gating mechanism submodule is used to receive self-attention weighted features and channel attention weighted features, and to perform convolution and sigmoid activation on the input features to generate gating weights. The gating weights are then used to perform weighted fusion of the self-attention weighted features and channel attention weighted features to obtain weighted fused features.

[0039] Preferably, the self-attention mechanism submodule calculates attention weights, and then weights the multi-level feature maps and global spatial features based on these attention weights to obtain self-attention weighted features, specifically:

[0040] Multi-level feature maps and global spatial features are respectively input into three two-dimensional convolutional layers for processing to generate query vectors, key vectors, and numerical vectors.

[0041] Input the query vector, key vector, and numerical vector into the reshaping unit to reshape the dimensions of the vector space;

[0042] The reshaped query vector and key vector are input into the attention layer to calculate the attention weights.

[0043] The self-attention weighted features are obtained by weighting the dimension-reshaped numerical vectors according to the attention weights.

[0044] The channel attention mechanism submodule performs channel weighting on the multi-level feature map and global spatial features to obtain channel attention-weighted features, specifically:

[0045] Perform global average pooling and global max pooling on multi-level feature maps and global spatial features;

[0046] The features obtained from global average pooling and global max pooling are concatenated.

[0047] The concatenated result is subjected to convolution and sigmoid activation to obtain channel weights. The input features are then weighted according to the channel weights to obtain channel attention-weighted features.

[0048] Preferably, the self-attention mechanism submodule includes a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, a fourth two-dimensional convolutional layer, a first reshaping layer, a dot product unit, a sealing layer, a softmax layer, an attention layer, a second reshaping unit, and a third reshaping unit; the inputs of the second, third, and fourth two-dimensional convolutional layers are all inputs of the self-attention mechanism submodule; the outputs of the second and third two-dimensional convolutional layers are both connected to the input of the first reshaping unit; the output of the first reshaping unit is connected to the input of the dot product unit, the output of the dot product unit is connected to the input of the sealing layer, the output of the sealing layer is connected to the input of the softmax layer, the output of the softmax layer is connected to the input of the attention layer, the output of the attention layer is connected to the input of the second reshaping unit, and the output of the second reshaping unit is the output of the self-attention mechanism submodule; the output of the fourth two-dimensional convolutional layer is connected to the input of the third reshaping unit, and the output of the third reshaping unit is connected to another input of the attention layer;

[0049] The channel attention mechanism submodule includes a global max pooling layer and a global average pooling layer, a fourth reshaping unit, a first connection unit, a fifth two-dimensional convolutional layer, a ReLU activation layer, a sixth two-dimensional convolutional layer, a first sigmoid activation layer, and a first weighting unit connected in sequence. The input terminals of the global max pooling layer, the global average pooling layer, and the first weighting unit are all input terminals of the channel attention mechanism submodule. The output terminal of the global max pooling layer is connected to the other input terminal of the first connection unit. The output terminal of the first weighting unit is the output terminal of the channel attention mechanism submodule.

[0050] The gating mechanism submodule includes a seventh two-dimensional convolutional layer, a second sigmoid activation layer, a second weighting unit, and a second connection unit connected in sequence; the input of the second weighting unit is also connected to the output of the self-attention mechanism submodule and the output of the channel attention mechanism submodule; the input of the seventh two-dimensional convolutional layer is the output of the self-attention mechanism submodule and the output of the channel attention mechanism submodule.

[0051] Preferably, the training method of the medical image segmentation model is as follows: training is performed using a composite loss function; the composite loss function includes weighted cross-entropy loss and Dice coefficient loss. In the early stage of training, weighted cross-entropy loss is mainly used to achieve rapid convergence and alleviate the class imbalance problem. As training progresses, the weights are adjusted in a segmented round, gradually reducing the weight of weighted cross-entropy loss and increasing the weight of Dice coefficient loss to optimize segmentation accuracy and boundary details.

[0052] The weighted cross-entropy loss is expressed as follows:

[0053]

[0054] in, This represents the weighted cross-entropy loss. Indicates the number of pixels. Indicates the weight of the target class. The pixel value representing the actual label. This represents the pixel value of the predicted label. Represented by natural base Logarithmic function with base 0. Indicates the weight of the background class. Indicates the index of each pixel;

[0055] The formula for expressing the Dice coefficient loss is as follows:

[0056]

[0057] in, This represents the Dice coefficient loss. Indicates the number of smoothing terms;

[0058] The formula for the composite loss function is as follows:

[0059]

[0060] in, Represents the composite loss function. This represents the weight of the Dice coefficient loss.

[0061] The beneficial effects of this invention are:

[0062] This invention enhances the multi-scale feature extraction capability of medical images of gastrointestinal polyps by combining a dual U-shaped encoder-decoder architecture with a pre-trained network (VGG19) and a lightweight structure. Simultaneously, through a dynamic composite loss function, weighted cross-entropy loss is primarily used in the early training phase (to alleviate the background pixel dominance problem), while Dice loss weights are gradually increased in the later stages (to optimize boundary alignment). This optimizes boundary details and class imbalance, significantly improving the segmentation accuracy and robustness of medical images (especially micropolyps) while maintaining low computational cost, providing an efficient and reliable automated tool for clinical diagnosis. Attached Figure Description

[0063] Figure 1 The flowchart shown is a medical image segmentation method based on contextual information and multi-scale feature fusion provided in Embodiment 1 of the present invention.

[0064] Figure 2 The diagram shown is a flowchart of the medical image segmentation method based on contextual information and multi-scale feature fusion provided in Embodiment 1 of the present invention.

[0065] Figure 3 The diagram shown is a structural diagram of the lightweight void space pyramid pooling (L-ASPP) module provided in Embodiment 1 of the present invention.

[0066] Figure 4 The diagram shows the codec structure of the second U-shaped network provided in Embodiment 1 of the present invention.

[0067] Figure 5 The diagram shows the structure of the gated self-attention feature fusion module provided in Embodiment 1 of the present invention.

[0068] Figure 6 The diagram shows the segmentation results and uncertainty visualization results obtained by the medical image segmentation method based on contextual information and multi-scale feature fusion provided in Embodiment 2 of the present invention. Detailed Implementation

[0069] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the drawings are merely exemplary and are intended to illustrate the principles and spirit of the invention, and are not intended to limit the scope of the invention.

[0070] Example 1:

[0071] like Figure 1 As shown, a medical image segmentation method based on the fusion of contextual information and multi-scale features includes the following steps:

[0072] S1. Obtain medical image data of gastrointestinal polyps and construct a medical image segmentation model; wherein, the medical image segmentation model adopts a dual U-shaped encoder-decoder architecture, including a first U-shaped network, an element-wise multiplication module, a second U-shaped network and a connection module connected in sequence;

[0073] Specifically, two public datasets and one internal dataset were used for preprocessing to construct a medical image segmentation model. The CVC-ClinicDB dataset collected 612 polyp images from 31 colonoscopy videos, with a resolution of 384*288 pixels. The Kvasir dataset collected 1000 polyp images from endoscopic equipment, with resolutions ranging from 332*487 to 1920*1072 pixels; it includes both original images and corresponding labeled images. The practicality and effectiveness of the proposed medical image segmentation model were verified using a polyp dataset from a hospital. This dataset included 105 polyp data from 45 patients, with a pixel resolution of 1920*1082, and was annotated by a physician and validated by an experienced gastroenterologist.

[0074] S2. Input the medical image data into the first U-shaped network for feature extraction to obtain multi-scale contextual features;

[0075] S3. Input the medical image data and multi-scale contextual features into the element-wise multiplication module to perform the multiplication operation;

[0076] S4. Input the output of the element-wise multiplication module into the second U-shaped network for global information modeling and local boundary feature enhancement, and output the final fused features;

[0077] S5. Input the multi-scale contextual features and the final fusion features into the connection module for connection, and output the medical image segmentation results of gastrointestinal polyps.

[0078] In this embodiment, the first U-shaped network includes a first encoder, a lightweight void space pyramid pooling module, and a first decoder connected in sequence.

[0079] The first encoder uses a VGG19 pre-trained network as the backbone network to receive medical image data and perform multi-level feature extraction. The shallow layer captures local details, and the deep layer captures abstract morphological features to obtain a multi-level feature map.

[0080] The lightweight hollow spatial pyramid pooling (L-ASPP) module is used to receive the multi-level feature map and perform multi-level convolution processing on the multi-level features to improve the perception ability of local regions.

[0081] In the first U-shaped network, the first encoder uses a pre-trained VGG19 model and performs four downsampling operations; the corresponding first decoder performs four upsampling operations. The first encoder and first decoder are connected via skip connections; these skip connections incorporate a coordinate attention mechanism, which preserves spatial information during feature recovery by capturing cross-channel information, orientation-aware features, and position-sensitive features. The first decoder, through skip connections with the corresponding modules of the encoder and a series of convolution, activation, and normalization operations, gradually recovers and refines the spatial information of the feature maps, obtaining multi-scale contextual features.

[0082] The coordinate attention mechanism in the skip connections works as follows: First, the input features are globally averaged and projected onto the horizontal and vertical directions respectively. The horizontal projection outputs a size of (H, 1, C), and the vertical projection outputs a size of (1, W, C), capturing global spatial information in the horizontal and vertical directions respectively. Then, the horizontally and vertically projected features are concatenated to obtain a joint feature map integrating spatial information from both directions. The concatenated features are then compressed using a 1*1 lightweight convolution to reduce computational cost while maintaining feature expressiveness. After convolution, the Canny edge detection operator is applied to detect edge regions of specific intensities in the image while suppressing irrelevant background regions. The edge-detected feature map is normalized to eliminate gradient instability during training, and a non-linear activation function is used to activate the normalized features, further enhancing the expression of non-linear edge features. After feature concatenation, the features are segmented and divided into high-frequency and low-frequency components using Fourier transform. The high-frequency component contains details of the target boundary features and complex shapes, while the low-frequency component contains global contour information of the target. This improvement helps the model to more accurately locate and identify regions of interest. Convolution is used to restore the original number of channels in the features, ensuring that the feature map maintains the same dimensionality as the feature map of the original input from the skip connections. A sigmoid activation function is then used to generate channel weights on the feature map. These weights dynamically adjust the contribution of each channel, enhancing the information representation of important channels. The generated channel weights are then multiplied element-wise with the original input features, effectively fusing the target's detailed information with global semantic information.

[0083] In this embodiment, as Figure 3 As shown, the lightweight dilated spatial pyramid pooling module includes a first convolutional branch, a second convolutional branch, a third convolutional branch, a fourth convolutional branch, and a lightweight attention channel. The inputs of the first, second, third, and fourth convolutional branches are all inputs of the lightweight dilated spatial pyramid pooling module, and the outputs of the first, second, third, and fourth convolutional branches are all connected to the input of the lightweight attention channel. The lightweight attention channel includes a first two-dimensional convolutional layer, a batch normalization layer, and a first ReLU activation layer connected in sequence.

[0084] The first convolutional branch reduces the dimensionality of the multi-level feature map through 1*1 convolution and adjusts the number of channels; the second convolutional branch expands the receptive field through dilation convolution; the third and fourth convolutional branches use depthwise separable convolutions with different dilation rates to extract multi-scale features, followed by batch normalization and ReLU activation to enhance feature representation, and finally dynamically adjust the channel weights through 1*1 convolution and ReLU activation function.

[0085] In this embodiment, the second U-shaped network includes a second encoder, a gated self-attention feature fusion module, and a second decoder; the second encoder and the second decoder are connected in a skip connection.

[0086] The input of the gated self-attention feature fusion module is connected to the other output of the first encoder and the output of the second encoder, and the output of the gated self-attention feature fusion module is connected to the input of the second decoder.

[0087] like Figure 4 As shown, the second encoder uses depthwise separable convolution as the basic operator, uses 3*3 convolution kernels for spatial feature extraction, and sets up a multi-scale dilated convolution structure. The multi-scale dilated convolution structure expands the receptive field layer by layer through a progressive dilation rate design (2, 4, 6), effectively capturing global context information. After each convolutional layer, a batch normalization layer and a ReLU activation function are added. When the second decoder performs feature fusion, it first upsamples the feature maps passed from the two encoders through the skip connection using bilinear interpolation to align their spatial resolution. Then, it calculates the cross attention between the decoder output and the skip connection, and adds the attention weights to the current decoder feature map to form the fused feature map.

[0088] The second U-shaped network performs global information modeling and local boundary feature enhancement, specifically as follows:

[0089] The result of element-wise multiplication of medical image data and multi-scale contextual features is input into the second encoder to extract global spatial features.

[0090] Multi-level feature maps and global spatial features are input into the gated self-attention feature fusion module for weighted fusion to obtain weighted fused features;

[0091] The weighted fused features are input into the second decoder, and the weighted fused features are upsampled by bilinear interpolation to obtain the upsampled features;

[0092] The attention weights are calculated by performing cross-attention calculations on the upsampled features and the skip connections between the second encoder and the second decoder. (To deeply fuse multi-scale features and enhance semantic understanding, cross-attention calculations are performed on the features derived from the upsampling operation and the features transmitted by the skip connection mechanism. Attention weights are obtained by accurately measuring the correlation between the two at the spatial layout and semantic connotation levels, providing a crucial basis for the efficient integration of subsequent features.) The attention weights are then summed with the upsampled features and upsampled again to obtain the final fused features. The specific calculation formula is as follows:

[0093]

[0094]

[0095]

[0096]

[0097] in, This represents the first internal hop connection of the second U-shaped network. Indicates upsampling, This represents the first convolutional kernel with a size of 1*1. and This represents the feature map output by the second encoder of the second U-shaped network. This represents the second internal hop connection of the second U-shaped network. This represents the feature map obtained after processing by the cross-attention mechanism. This indicates attention calculation. This represents the feature map of the current decoder. Indicates addition, This represents the feature map of the final output of the second U-shaped network.

[0098] In this embodiment, as Figure 5As shown, the gated self-attention feature fusion module includes a self-attention mechanism submodule, a gated mechanism submodule, and a channel attention mechanism submodule. The core function of the self-attention mechanism module is to capture the long-range dependency between two U-shaped branch features, while the channel attention mechanism module focuses on extracting local features at multiple scales, and is particularly good at capturing the edge details of small polyps.

[0099] The self-attention mechanism submodule is used to receive the multi-level feature map and the global spatial features, calculate the attention weights, and perform weighted processing on the input features according to the attention weights to obtain self-attention weighted features.

[0100] The channel attention mechanism submodule is used to receive the multi-level feature map and the global spatial features, and to perform channel weighting on the input features to obtain channel attention weighted features;

[0101] The gating mechanism submodule is used to receive self-attention weighted features and channel attention weighted features, and to perform convolution and sigmoid activation on the input features to generate gating weights. The gating weights are then used to weight and fuse the self-attention weighted features and channel attention weighted features to obtain a weighted fused feature. The specific calculation formula is as follows:

[0102]

[0103]

[0104] in, This represents the weighted fusion feature map generated after the input feature map is processed by gating weights. This represents the final feature map obtained after the input feature map has undergone gating weight processing and a series of subsequent weighting operations. This indicates that the feature map output by the self-attention mechanism is processed through a gating mechanism. This represents the feature map output by the gating mechanism processing the channel attention mechanism. This represents the output feature map after the input feature map has been adjusted by the self-attention mechanism. Indicates splicing, This represents the output feature map after the input feature map has been adjusted using the channel attention mechanism. Indicates the number of channels.

[0105] In this embodiment, the self-attention mechanism submodule calculates attention weights, and performs weighted processing on the multi-level feature map and global spatial features based on the attention weights to obtain self-attention weighted features, specifically:

[0106] Multi-level feature maps and global spatial features are respectively input into three two-dimensional convolutional layers for processing to generate query vectors, key vectors, and value vectors.

[0107] The query vector, key vector, and numerical vector are input into the reshaping unit to reshape the dimension of the vector space, reducing the number of output channels for the query vector and key vector. To optimize computational efficiency and control model complexity, the number of output channels for the query vector and key vector is reduced to one-eighth of the number of input channels. Subsequently, the spatial dimension of these three vectors is adjusted by the reshaping operation to flatten the spatial dimension, thereby ensuring that attention computation can be carried out smoothly in the channel dimension of each pixel.

[0108] The reshaped query vector and key vector are input into the attention layer to calculate the multiplication matrix value between them, yielding the attention weights. These attention weights accurately reflect the similarity between features at different locations. To ensure the attention weights conform to the probability distribution requirements, the softmax function is used to normalize each row of the attention weight matrix, ultimately obtaining the weight of each pixel in the channel. The reshaped numerical vectors are then weighted and summed according to the attention weights to obtain the self-attention weighted features. The specific calculation formula is as follows:

[0109]

[0110] Indicates reshaping, Indicates the input tensor. Indicates the number of channels. This represents the softmax function. Indicates the scaling factor. This represents the final self-attention weighted feature. Indicates a weighted operation. Represents matrix multiplication;

[0111] The channel attention mechanism submodule performs channel weighting on the multi-level feature map and global spatial features to obtain channel attention-weighted features, specifically:

[0112] Global average pooling and global max pooling are performed on multi-level feature maps and global spatial features. The specific calculation formulas are as follows:

[0113]

[0114]

[0115] in, This represents the features after global average pooling. Indicates global average pooling. This represents the features after global max pooling. Indicates global max pooling;

[0116] The features obtained from global average pooling and global max pooling are concatenated using the following formula:

[0117]

[0118] in, This indicates the splicing result. Indicates splicing;

[0119] The concatenated result is subjected to convolution and sigmoid activation to obtain channel weights. The input features are then weighted according to these channel weights to obtain channel attention-weighted features. The specific calculation formula is as follows:

[0120]

[0121]

[0122]

[0123] in, This represents the output feature obtained after activation by the ReLU function. Represents the ReLU function. This represents the sigmoid activation function. This represents the output feature obtained after activation by the sigmoid activation function. This represents the feature map output after undergoing channel attention mechanism adjustments and optimizations.

[0124] In this embodiment, the self-attention mechanism submodule includes a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, a fourth two-dimensional convolutional layer, a first reshaping layer, a dot product unit, a sealing layer, a softmax layer, an attention layer, a second reshaping unit, and a third reshaping unit. The inputs of the second, third, and fourth two-dimensional convolutional layers are all inputs of the self-attention mechanism submodule. The outputs of the second and third two-dimensional convolutional layers are both connected to the input of the first reshaping unit. The output of the first reshaping unit is connected to the input of the dot product unit, the output of the dot product unit is connected to the input of the sealing layer, the output of the sealing layer is connected to the input of the softmax layer, the output of the softmax layer is connected to the input of the attention layer, and the output of the attention layer is connected to the input of the second reshaping unit, which is the output of the self-attention mechanism submodule. The output of the fourth two-dimensional convolutional layer is connected to the input of the third reshaping unit, and the output of the third reshaping unit is connected to another input of the attention layer.

[0125] The channel attention mechanism submodule includes a global max pooling layer and a global average pooling layer, a fourth reshaping unit, a first connection unit, a fifth two-dimensional convolutional layer, a ReLU activation layer, a sixth two-dimensional convolutional layer, a first sigmoid activation layer, and a first weighting unit connected in sequence. The input terminals of the global max pooling layer, the global average pooling layer, and the first weighting unit are all input terminals of the channel attention mechanism submodule. The output terminal of the global max pooling layer is connected to the other input terminal of the first connection unit. The output terminal of the first weighting unit is the output terminal of the channel attention mechanism submodule.

[0126] The gating mechanism submodule includes a seventh two-dimensional convolutional layer, a second sigmoid activation layer, a second weighting unit, and a second connection unit connected in sequence; the input of the second weighting unit is also connected to the output of the self-attention mechanism submodule and the output of the channel attention mechanism submodule; the input of the seventh two-dimensional convolutional layer is the output of the self-attention mechanism submodule and the output of the channel attention mechanism submodule.

[0127] In this embodiment, the second U-shaped network performs global information modeling and local boundary feature enhancement on medical image data, specifically as follows:

[0128] The first U-shaped network and medical image data are input into the element-wise multiplication module, and the first output image data is output.

[0129] The first output data is input into the second encoder, and spatial features are extracted using the depthwise separable convolutional operator of the second encoder. A multi-scale dilated convolutional structure is introduced, and the receptive field is expanded layer by layer through a progressive dilation rate design (2, 4, 6) to effectively capture the first output data.

[0130] In this embodiment, the training method of the medical image segmentation model is specifically as follows: training is performed using the Adam optimization algorithm based on a composite loss function;

[0131] The composite loss function includes Weighted Cross-Entropy Loss (WCE) and Dice coefficient loss. By dynamically adjusting the contributions of the two losses, the problem of segmentation accuracy and class imbalance can be effectively addressed. In the initial training phase, training primarily uses the weighted cross-entropy loss, setting its weight relatively large to facilitate rapid convergence and alleviate class imbalance. As training progresses, the weights are adjusted in segmented rounds, gradually decreasing the weight of the weighted cross-entropy loss and increasing the weight of the Dice coefficient loss to optimize segmentation accuracy and boundary details. The formula for the weighted cross-entropy loss is:

[0132]

[0133] in, This represents the weighted cross-entropy loss. Indicates the number of pixels ( ), Indicates the weight of the target class. Indicates the weight of the background class. and To adjust the importance of different categories, in this embodiment... Set as the ratio of background pixel count to target pixel count to enhance the model's focus on the target region. The pixel value representing the actual label (0 or 1, where 0 represents the background and 1 represents the target). This represents the pixel value of the predicted label. Represented by natural base Logarithmic function with base 0. Indicates the index of each pixel;

[0134] The formula for expressing the Dice coefficient loss is as follows:

[0135]

[0136] in, This represents the Dice coefficient loss. Indicates the number of smoothing terms;

[0137] The formula for the composite loss function is as follows:

[0138]

[0139] in, Represents the composite loss function. This represents the weight of the Dice coefficient loss.

[0140] The training process utilizing the Adam optimization algorithm is as follows: The medical image segmentation model proposed in this invention is developed using Python 3.8.0 and Tensorflow+Keras 2.7.0 software environment; training is performed on an RTX 4090D (24GB) GPU. The Adam optimizer is used to accelerate model convergence, with the initial learning rate set to 0.0001, batch size set to 4, and number of iterations set to 300. Furthermore, a corresponding callback function is used to stop training if the validation metrics of the medical image segmentation model do not improve within 50 iterations. During the initial training phase, the learning rate is gradually increased from the initial value of 0.0001 to the preset learning rate of 0.001; when the validation loss no longer decreases, the learning rate is reduced by a factor of 10, up to a maximum reduction of 20 times. This combined strategy stabilizes the training process of the medical image segmentation model in the early stages of training, while finely adjusting the learning rate in the later stages to improve the convergence speed of the medical image segmentation model.

[0141] In this embodiment, the segmentation performance of the medical image segmentation model proposed in this invention is evaluated using Dice coefficient, mIoU, Precision, Recall, Hausdorff distance (HD), and mean surface distance (MSD). The specific calculation formula is as follows:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] in, This indicates that the positive case was correctly identified as a positive case. This indicates that a positive case was incorrectly identified as a negative case. This indicates that a negative case was correctly identified as a negative case. This indicates that a negative case was incorrectly identified as a positive case.

[0149] Example 2:

[0150] Based on Example 1, the embodiments of the present invention are scientifically demonstrated through experiments to illustrate the beneficial effects of the medical image segmentation method based on contextual information and multi-scale feature fusion proposed in this invention.

[0151] The medical image segmentation model constructed in this invention was compared with other representative models on the evaluation metrics of polyp image segmentation results in Kvasir-SEG, CVC-ClinicDB, and internal datasets. The experimental results were quantitatively compared as follows: Figure 6 As shown in Tables 1, 2 and 3.

[0152] Table 1 Quantitative results of the Kvasir-SEG dataset

[0153]

[0154] Table 2 Quantitative results of the CVC-ClinicDB dataset

[0155]

[0156] Table 3 Quantitative results of the internal dataset

[0157]

[0158] As shown in Tables 1, 2, and 3, based on numerical evaluations of Dice, mIoU, Recall, Precision, HD, and MSD values ​​using different methods, the model proposed in this invention outperforms most other comparative models. Qualitative comparative experimental results of the polyp segmentation network constructed in this invention with other representative networks are as follows: Figure 6 As shown, the prediction result of the medical image segmentation model of the present invention is closest to the standard segmentation mask, and it also best matches the original edge division of the standard segmentation mask in terms of edge processing details.

[0159] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A medical image segmentation method based on contextual information and multi-scale feature fusion, characterized in that, Includes the following steps: Medical image data of gastrointestinal polyps were acquired, and a medical image segmentation model was constructed. The medical image segmentation model adopted a dual U-shaped encoder-decoder architecture, which included a first U-shaped network and a second U-shaped network connected in sequence. Medical image data is input into the first U-shaped network for feature extraction to obtain multi-scale contextual features; Multiply medical imaging data and multi-scale contextual features element-wise; The element-wise multiplication result is input into the second U-shaped network for global information modeling and local boundary feature enhancement, and the final fused feature is output. The multi-scale contextual features and the final fusion features are concatenated to obtain the medical image segmentation results of gastrointestinal polyps; The first U-shaped network includes a first encoder, a lightweight hollow space pyramid pooling module, and a first decoder connected in sequence; the first encoder and the first decoder are connected in skip connections. The jump connection is equipped with a coordinate attention mechanism, which captures cross-channel information, direction-aware features, and position-sensitive features, and retains spatial information during feature recovery. The first encoder is a VGG19 pre-trained network used to receive medical image data and perform multi-level feature extraction to obtain multi-level feature maps. The lightweight hollow spatial pyramid pooling module is used to receive the multi-level feature map and perform multi-level convolution processing on the multi-level features to improve the perception ability of local regions. The first decoder is used to receive the multi-level feature map and the multi-level feature map after multi-level convolution processing, and then recover and refine the spatial information of the medical image data feature map to obtain multi-scale context features; The second U-shaped network includes a second encoder, a gated self-attention feature fusion module, and a second decoder; there are skip connections between the second encoder and the second decoder; The input of the gated self-attention feature fusion module is connected to the other output of the first encoder and the output of the second encoder, and the output of the gated self-attention feature fusion module is connected to the input of the second decoder. The second U-shaped network performs global information modeling and local boundary feature enhancement, specifically as follows: The result of element-wise multiplication of medical image data and multi-scale contextual features is input into the second encoder to extract global spatial features. Multi-level feature maps and global spatial features are input into a gated self-attention feature fusion module for weighted fusion to obtain weighted fused features. The weighted fused features are input into the second decoder, and the weighted fused features are upsampled by bilinear interpolation to obtain the upsampled features; The attention weights are obtained by calculating the cross-attention between the upsampled features and the skip connections between the second encoder and the second decoder. The attention weights are summed with the upsampled features, and then upsampled again to obtain the final fused features.

2. The medical image segmentation method based on contextual information and multi-scale feature fusion according to claim 1, characterized in that, The coordinate attention mechanism captures cross-channel information, orientation-aware features, and position-sensitive features, while preserving spatial information during feature recovery. Specifically: Global average pooling is performed on the multi-level feature map. The features after global average pooling are projected onto the horizontal and vertical directions respectively, and global spatial information in the horizontal and vertical directions is captured respectively to obtain horizontal projection features and vertical projection features. The horizontal and vertical projection features are concatenated to obtain a joint feature map that integrates spatial information from both directions. The number of channels in the concatenated joint feature map is then compressed using a 1*1 lightweight convolution. The Canny edge detection operator is applied to detect edge regions of specific intensities in the joint feature map of spatial information after channel compression, while suppressing irrelevant background regions, to obtain a feature map after edge detection. The feature map after edge detection is normalized to eliminate gradient instability during training, and a non-linear activation function is used to activate the normalized features to enhance the expression of non-linear edge features. The activated features are segmented, and the segmented features are divided into high-frequency feature maps and low-frequency feature maps by Fourier transform. The high-frequency feature maps contain the target boundary features and detailed information of complex shapes, while the low-frequency feature maps contain the global contour information of the target. By convolution, the original number of channels in the high-frequency and low-frequency parts is restored, ensuring that the high-frequency and low-frequency feature maps maintain the same dimension as the multi-level feature maps of the original input of the skip connections. Then, the Sigmoid activation function is used to generate channel weights for the high-frequency and low-frequency feature maps. Through the generated channel weights, the contribution of each channel is dynamically adjusted to enhance the information expression of important channels. The generated channel weights are multiplied element-wise with the multi-level feature map, which effectively integrates the detailed information of the target with the global semantic information.

3. The medical image segmentation method based on contextual information and multi-scale feature fusion according to claim 1, characterized in that, The lightweight hollow spatial pyramid pooling module includes a first convolutional branch, a second convolutional branch, a third convolutional branch, a fourth convolutional branch, and a lightweight attention channel; the input terminals of the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch are all input terminals of the lightweight hollow spatial pyramid pooling module, and the output terminals of the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch are all connected to the input terminal of the lightweight attention channel; The lightweight attention channel comprises a first two-dimensional convolutional layer, a batch normalization layer, and a first ReLU activation layer connected in sequence.

4. The medical image segmentation method based on contextual information and multi-scale feature fusion according to claim 1, characterized in that, The gated self-attention feature fusion module includes a self-attention mechanism submodule, a gated mechanism submodule, and a channel attention mechanism submodule; The self-attention mechanism submodule is used to receive the multi-level feature map and the global spatial features, calculate the attention weights, and perform weighted processing on the input features according to the attention weights to obtain self-attention weighted features. The channel attention mechanism submodule is used to receive the multi-level feature map and the global spatial features, and to perform channel weighting on the input features to obtain channel attention weighted features; The gating mechanism submodule is used to receive self-attention weighted features and channel attention weighted features, and to perform convolution and sigmoid activation on the input features to generate gating weights. The gating weights are then used to perform weighted fusion of the self-attention weighted features and channel attention weighted features to obtain weighted fused features.

5. The medical image segmentation method based on contextual information and multi-scale feature fusion according to claim 4, characterized in that, The self-attention mechanism submodule calculates attention weights, and then performs weighted processing on the multi-level feature maps and global spatial features based on these attention weights to obtain self-attention weighted features, specifically: Multi-level feature maps and global spatial features are respectively input into three two-dimensional convolutional layers for processing to generate query vectors, key vectors, and numerical vectors. Input the query vector, key vector, and numerical vector into the reshaping unit to reshape the dimensions of the vector space; The reshaped query vector and key vector are input into the attention layer to calculate the attention weights. The self-attention weighted features are obtained by weighting the dimension-reshaped numerical vectors according to the attention weights. The channel attention mechanism submodule performs channel weighting on the multi-level feature map and global spatial features to obtain channel attention-weighted features, specifically: Perform global average pooling and global max pooling on multi-level feature maps and global spatial features; The features obtained from global average pooling and global max pooling are concatenated. The concatenated result is subjected to convolution and sigmoid activation to obtain channel weights. The input features are then weighted according to the channel weights to obtain channel attention-weighted features.

6. The medical image segmentation method based on contextual information and multi-scale feature fusion according to claim 4, characterized in that, The self-attention mechanism submodule includes a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, a fourth two-dimensional convolutional layer, a first reshaping layer, a dot product unit, a sealing layer, a softmax layer, an attention layer, a second reshaping unit, and a third reshaping unit. The inputs of the second, third, and fourth two-dimensional convolutional layers are all inputs of the self-attention mechanism submodule. The outputs of the second and third two-dimensional convolutional layers are both connected to the input of the first reshaping unit. The output of the first reshaping unit is connected to the input of the dot product unit, the output of the dot product unit is connected to the input of the sealing layer, the output of the sealing layer is connected to the input of the softmax layer, the output of the softmax layer is connected to the input of the attention layer, and the output of the attention layer is connected to the input of the second reshaping unit. The output of the second reshaping unit is the output of the self-attention mechanism submodule. The output of the fourth two-dimensional convolutional layer is connected to the input of the third reshaping unit, and the output of the third reshaping unit is connected to another input of the attention layer. The channel attention mechanism submodule includes a global max pooling layer and a global average pooling layer, a fourth reshaping unit, a first connection unit, a fifth two-dimensional convolutional layer, a ReLU activation layer, a sixth two-dimensional convolutional layer, a first sigmoid activation layer, and a first weighting unit connected in sequence. The input terminals of the global max pooling layer, the global average pooling layer, and the first weighting unit are all input terminals of the channel attention mechanism submodule. The output terminal of the global max pooling layer is connected to the other input terminal of the first connection unit. The output terminal of the first weighting unit is the output terminal of the channel attention mechanism submodule. The gating mechanism submodule includes a seventh two-dimensional convolutional layer, a second sigmoid activation layer, a second weighting unit, and a second connection unit connected in sequence; the input of the second weighting unit is also connected to the output of the self-attention mechanism submodule and the output of the channel attention mechanism submodule; the input of the seventh two-dimensional convolutional layer is the output of the self-attention mechanism submodule and the output of the channel attention mechanism submodule.

7. The medical image segmentation method based on contextual information and multi-scale feature fusion according to claim 1, characterized in that, The training method of the medical image segmentation model is as follows: training is performed using a composite loss function. The composite loss function includes weighted cross-entropy loss and Dice coefficient loss. In the early stage of training, weighted cross-entropy loss is mainly used to achieve rapid convergence and alleviate the class imbalance problem. As training progresses, the weights are adjusted in a segmented round, gradually reducing the weight of weighted cross-entropy loss and increasing the weight of Dice coefficient loss to optimize segmentation accuracy and boundary details. The weighted cross-entropy loss is expressed as follows: in, This represents the weighted cross-entropy loss. Indicates the number of pixels. Indicates the weight of the target class. The pixel value representing the actual label. This represents the pixel value of the predicted label. Represented by natural base Logarithmic function with base 0. Indicates the weight of the background class. Indicates the index of each pixel; The formula for expressing the Dice coefficient loss is as follows: in, This represents the Dice coefficient loss. Indicates the number of smoothing terms; The formula for the composite loss function is as follows: in, Represents the composite loss function. This represents the weight of the Dice coefficient loss.

Images