Semi-supervised medical image segmentation method based on enhanced self-attention features

Abstract

This paper presents a semi-supervised medical image segmentation method based on enhanced self-attention features. The method comprises the following steps: constructing a semi-supervised medical image segmentation network based on enhanced self-attention features, utilizing local features extracted by a convolutional network to enhance the global features extracted by a Transformer network, and improving the quality of pseudo-annotations generated by the Transformer network; calculating supervised and unsupervised losses using a cross-entropy loss function and a dice loss function, and training and optimizing the semi-supervised segmentation model using the weighted sum of the two losses. Experimental testing has demonstrated that this method can achieve good segmentation results using only partially annotated data, making it easier for doctors to assist in diagnosing conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of semi-supervised medical image segmentation, and in particular to a semi-supervised medical image segmentation method based on enhanced self-attention features. Background Art

[0002] Semi-supervised medical image segmentation methods improve upon traditional medical image segmentation methods in terms of the number of image labels required. Semi-supervised learning methods can utilize a small amount of labeled data alongside a large amount of unlabeled data to train deep learning models, effectively alleviating the need for the large amount of labeled data required to train a medical image segmentation model while maintaining satisfactory performance. This feature of semi-supervised learning can accelerate clinical data annotation and model development, saving doctors and experts time and effort and facilitating diagnosis.

[0003] Existing semi-supervised medical image segmentation frameworks primarily generate predicted pseudo-annotations using two or more segmentation networks, then calculate a consistency loss between these multiple pseudo-annotations to complete training on unlabeled data. Therefore, the quality of the generated pseudo-annotations is closely related to semi-supervised performance. With the advancement of Transformer networks in computer vision, existing work has introduced Transformer networks into semi-supervised frameworks, combining them with convolutional networks to perform semi-supervised segmentation tasks. This approach improves semi-supervised medical image segmentation performance by combining local image features extracted by convolutional networks with global image features extracted by Transformer networks. For example, Luo et al. published "Semi-supervised Medical Image Segmentation via Cross Teaching Between CNNs and Transformers." This work replaces one convolutional branch with a Transformer-based segmentation network based on a cross-pseudo-supervised model. This simple combination of convolutional and Transformer networks performs cross-consistency supervision on the output, improving semi-supervised segmentation performance. In the field of medical image segmentation, the Transformer model was introduced because of its ability to capture long-distance dependencies, but its direct application in medical scenarios still faces significant challenges: First, medical lesions often present small scales and irregular distributions, and the global self-attention mechanism of the Transformer tends to weaken local detail features, resulting in the loss of key pathological information; second, existing semi-supervised frameworks mostly use a simple serial architecture of convolution and Transformer, which lacks dynamic multi-scale feature interaction and makes it difficult to achieve efficient complementarity between local and global features of the image; finally, traditional pseudo-annotation generation strategies are sensitive to noise and can easily cause the model to misidentify normal pixel tissue when data is scarce.

[0004] Therefore, how to effectively utilize the features extracted from convolutional networks and Transformer networks to improve the accuracy and noise sensitivity of semi-supervised medical image segmentation is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In response to the shortcomings of the prior art, the present invention aims to provide a semi-supervised medical image segmentation method based on enhanced self-attention features. This method can enhance the self-attention global features extracted from the Transformer network. The self-attention feature enhancement module is introduced into the Transformer network, and multiple stacking is performed to achieve multi-scale segmentation of medical images and effectively extract global features. The convolutional network synergistically enhances the Transformer network. Finally, through consistency regularization, the image information learned by the convolutional network and the Transformer network are mutually supervised to complete the semi-supervised learning process. This method well preserves key pathological information, realizes dynamic multi-scale feature interaction, and improves the recognition accuracy of normal pixel tissue under small sample data.

[0006] To achieve the above object, the technical solution of the present invention is:

[0007] In a first aspect, the present invention provides a semi-supervised medical image segmentation method based on enhanced self-attention features, the method comprising the following contents:

[0008] Obtain a case database, where each case corresponds to a lesion medical image and a true annotated image;

[0009] Constructing a semi-supervised medical image segmentation network based on enhanced self-attention features:

[0010] The semi-supervised image segmentation network based on enhanced self-attention features includes a Transformer network and a convolutional network; the encoder of the convolutional network is used to extract multi-scale local features of the image;

[0011] The Transformer network includes an image block embedding coding module, a position encoder, several stacked Transformer encoders, and a Transformer decoder jump-connected to each Transformer encoder; the Transformer encoder includes a multi-head self-attention, layer normalization, self-attention feature enhancement module, a feedforward neural network, layer normalization and an image block merging module connected in sequence, wherein the input of the self-attention feature enhancement module is the intermediate-scale local features output by the convolutional network and the result of the multi-head self-attention after layer normalization processing and the input residual connection of the multi-head self-attention; the output of the self-attention feature enhancement module is processed by the feedforward neural network and layer normalization, and then connected to the output residual of the self-attention feature enhancement module, and then processed by the image block merging module to obtain the output of the current Transformer encoder;

[0012] Training a semi-supervised medical image segmentation network based on enhanced self-attention features:

[0013] The case samples in the case database are divided into labeled data and unlabeled data according to a set ratio, where the amount of labeled data is much smaller than the amount of unlabeled data. The labeled data is recycled and packaged with the unlabeled data and fed into a semi-supervised medical image segmentation network based on enhanced self-attention features. The labeled and unlabeled data are simultaneously fed into the Transformer network and the convolutional network.

[0014] During the training process, the cross entropy loss and the mean of the dice loss between the predicted and true labels are calculated for the labeled data. The dice loss is calculated between the predicted labels of the convolutional network and the Transformer network for the unlabeled data. The parameters are updated using deep learning gradient backpropagation, and the trained convolutional network is continuously trained and optimized.

[0015] Use the trained convolutional network to perform medical image segmentation.

[0016] Furthermore, the convolutional network adopts a U-net convolutional network, and the network structure is: [d 4, d 16, d 32, d64, d 128, d 256, u 256, u 128, u 64, u 32, u 16, u 4], where d represents a convolution-batch normalization-ReLU structure, [d 4] represents a convolution-batch normalization-ReLU structure with a [kernel size of 3×3, a step size of 1, and a number of layers of 4]; u represents a convolution-pooling upsampling structure, and [u 256] represents a convolution-pooling upsampling structure with a [kernel size of 3×3, a step size of 1, a pooling window size of 2×2, and a number of layers of 256].

[0017] Furthermore, part of the Transformer network structure is: [d 4, t 16, t 32, t 64, t 128,t 256, u 256, u 128, u 64, u 32, u 16, u 4], where t represents the Transformer encoder, [t16] represents a Transformer encoder with [image block size of 4×4, number of attention heads of 4, and number of layers of 16]; d represents the convolution-batch normalization-ReLU structure, [d 4] represents a convolution-batch normalization-ReLU structure with [kernel size of 3×3, stride of 1, and number of layers of 4]; u represents the convolution-pooling upsampling structure, and [u 256] represents a convolution-pooling upsampling structure with [kernel size of 3×3, stride of 1, pooling window size of 2×2, and number of layers of 256].

[0018] Furthermore, the self-attention feature enhancement module includes a global average pooling operation, an average pooling operation, a maximum pooling operation, a 1×1 convolution operation, a ReLU activation function, and a Sigmoid function; a local feature of an intermediate scale output by the convolutional network and the result of the multi-head self-attention of the Transformer encoder being connected with the input residual of the multi-head self-attention after layer normalization are spliced ​​to obtain the feature , for features The maximum pooling operation, average pooling operation and global average pooling operation are performed respectively. The result after the global average pooling operation is processed by a 1×1 convolution operation, a ReLU activation function, and a 1×1 convolution operation to obtain the first branch result;

[0019] The results of the maximum pooling operation and the average pooling operation are concatenated and then processed by a 1×1 convolution operation, a ReLU activation function, and a 1×1 convolution operation to obtain the second branch result;

[0020] The results of the first branch and the second branch are added element by element through the broadcast mechanism, and then the output value is mapped to the (0,1) interval through the Sigmoid function to generate the attention weight map;

[0021] The generated attention weight map is compared with the feature Perform element-by-element multiplication to obtain the output of the self-attention feature enhancement module .

[0022] Furthermore, the lesion medical image includes medical image data generated based on computed tomography (CT) or magnetic resonance imaging (MRI) technology.

[0023] In a second aspect, the present invention further protects a computer-readable storage medium having a computer program stored thereon, which can implement the steps of the method when executed by a processor.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This paper innovatively introduces a self-attention feature enhancement module inside the Transformer encoder, and uses local features in the convolutional network to enhance the self-attention image features extracted by the Transformer, solving the sensitivity of pseudo-annotations to noise and improving the segmentation accuracy under the semi-supervised framework.

[0026] (2) The self-attention feature enhancement module in the present invention captures local features (combined features) and global features (the results after global average pooling operation) through a parallel pooling strategy, and combines it with Sigmoid dynamic weighted fusion to effectively suppress background noise and enhance lesion response; combined with the CNN and Transformer dual-branch semi-supervised network, it uses jump connections to achieve multi-scale feature weight optimization fusion to solve the problem of low efficiency in the use of complementary information, and combines consistency regularization to constrain the spatial consistency of the dual-branch prediction results, significantly improving the quality of pseudo-annotation in semi-supervised scenarios, thereby improving the representation ability of complex anatomical structures while ensuring computational efficiency.

[0027] (3) The present invention deeply combines the image features extracted from two different networks in a semi-supervised framework, which is conducive to obtaining a high-performance medical image segmentation model. In actual medical clinical scenarios, in order to use deep learning training to obtain a high-performance segmentation model, the data set requires high-quality labeled images, but it is often very difficult to obtain high-quality medical image segmentation annotations, which highlights the importance of semi-supervised training. Compared with the existing technology, the present invention uses the local features extracted by the convolutional network to enhance the Transformer network medical image segmentation performance in a semi-supervised framework to improve the performance of the Transformer in medical image segmentation, so as to improve the quality of the pseudo-annotations generated by the Transformer in the semi-supervised learning framework, which helps the model to better understand the input medical images in all aspects, especially for complex and tiny lesions, so that the model can more accurately identify the key information and features in the image, so as to promote the model to more comprehensively understand the nature of complex lesions and improve the accuracy of semi-supervised segmentation.

[0028] (4) The present invention fully combines the image features extracted from different networks, and fully combines the local image features extracted from the convolutional network and the long-distance dependency features of the image extracted from the Transformer network, which helps the deep learning model to deeply understand medical images from different perspectives. The self-attention feature enhancement module can enhance the model's representation ability of medical images by fully integrating the features extracted from different networks, improve the accuracy and robustness of image analysis tasks, and achieve efficient complementarity between local features and global features. Using the convolutional network that performs well in the field of medical image segmentation to enhance the global features extracted by the Transformer network can not only improve the performance of the Transformer network in the field of medical image segmentation, but also improve the performance and effect of the model in semi-supervised medical image analysis tasks, solving the problem that the Transformer is prone to weakening local detail features in medical images. Finally, the results on the public medical image dataset show that under semi-supervised conditions, when there is only 10% of the labeled data, the average dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) are 66.21% and 10.97 mm respectively. The performance of the present invention is excellent, surpassing most semi-supervised methods, proving the efficiency of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a structural diagram of a semi-supervised medical image segmentation network based on enhanced self-attention features according to an embodiment of the present invention.

[0030] Figure 2 Schematic diagram of the structure of a self-attention feature enhancement module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to further introduce the technical solutions of the present invention and are not intended to limit the scope of protection of this application.

[0032] The present invention provides a semi-supervised medical image segmentation method based on enhanced self-attention features, comprising the following steps:

[0033] Step 1: Load case data and perform medical image data preprocessing: Obtain data from a disease-specific image segmentation dataset. The image segmentation dataset is stored in a case database saved in the h5 file format. Each case corresponds to a lesion medical image and a ground-truth annotated image. The lesion medical image is generated using technologies such as computed tomography (CT) or magnetic resonance imaging (MRI). The ground-truth annotated image is the data annotated by a professional doctor after performing a medical diagnosis on the case.

[0034] All medical image data were preprocessed, including slicing the 3D data and removing the slice data without lesions, obtaining the patient name and slice list for each case, and normalizing and standardizing the images to facilitate subsequent segmentation network training.

[0035] The h5 format is a file type based on HDF5. It is a flexible and efficient data storage and management format, especially suitable for scientific computing, engineering fields, and the storage and processing of large-scale data.

[0036] Step 2: Build a semi-supervised medical image segmentation network based on enhanced self-attention features:

[0037] The semi-supervised image segmentation network based on enhanced self-attention features adopts Figure 1 Semi-supervised frameworks presented, including Transformer networks and convolutional networks;

[0038] The convolutional network includes an encoder and a decoder. The encoder of the convolutional network is used to extract multi-scale local features of the image, and the decoder is used to map the extracted local features back to the original image size and perform pixel-level classification or segmentation.

[0039] The Transformer network includes an image block embedding coding module, a position encoder, several stacked Transformer encoders, and a Transformer decoder with jump connections to each Transformer encoder.

[0040] The Transformer encoder consists of a sequentially connected multi-head self-attention module, layer normalization, a self-attention feature enhancement module, a feedforward neural network, layer normalization, and an image patch merging module. The multi-head self-attention performs weighted processing on the input image patch sequence to capture long-range dependencies in the image. By stacking multiple Transformer encoders, image features can be gradually extracted and modeled.

[0041] The input of the self-attention feature enhancement module is the intermediate-scale local features output by the convolutional network and the result of the multi-head self-attention being connected with the input residual of the multi-head self-attention after layer normalization processing; the output of the self-attention feature enhancement module is connected with the output residual of the self-attention feature enhancement module after the feedforward neural network and layer normalization processing, and then processed by the image block merging module to obtain the output of the current Transformer encoder; the number of Transformer encoders is consistent with the number of intermediate scales of the multi-scale local features extracted by the convolutional network; the intermediate scale is the output scale of the encoding layer excluding the first layer encoding and the last layer encoding of the encoder in the convolutional network.

[0042] This paper achieves a strong correlation between convolutional networks and Transformer networks, with the intermediate scales of the encoders and decoders of both networks being four. The image features of each Transformer encoder are output to the next layer of the network, which can be the next Transformer encoder or a Transformer decoder of the corresponding scale. In a multi-layer Transformer encoder, the image features of each layer carry different levels of information.

[0043] In the Transformer network, the input medical image is processed into an image block sequence through the image block embedding coding module, and then the position code is added through the position encoder and input into the Transformer encoder to extract the global long-range dependency of the medical image.

[0044] The present invention is based on a dual-branch mutual learning architecture. The convolutional network sequentially extracts multi-scale local features through multi-level convolution kernels, and deeply embeds residual jump connections to retain anatomical structure details; the Transformer network branch captures long-range spatial dependencies, and iteratively enhances local-global feature interactions through a shift window strategy, effectively integrating the local response of CNN with the global dependency of Transformer, dynamically adjusting the feature importance through the attention mechanism, suppressing redundant background interference, and strengthening the representation significance of the lesion area, thereby achieving collaborative optimization of the convolutional network and Transformer network features to meet the characteristics of heterogeneous feature distribution of medical images. In the encoding stage, the CNN network branch focuses on the extraction of local texture and edge details, and aggregates neighborhood information layer by layer through multi-layer convolution; the Transformer network branch focuses on global context modeling and captures long-range dependencies. In the decoding stage, both branches fuse multi-scale features through jump connections, and combine consistency regularization to constrain the spatial consistency of the dual-branch prediction results.

[0045] Step 6: Train a semi-supervised medical image segmentation network based on enhanced self-attention features:

[0046] The case samples in the case database are divided into labeled data and unlabeled data according to a set ratio, where the amount of labeled data is much smaller than the amount of unlabeled data. The labeled data is recycled and packaged with the unlabeled data and fed into a semi-supervised medical image segmentation network based on enhanced self-attention features. The labeled and unlabeled data are simultaneously fed into the Transformer network and the convolutional network.

[0047] During the training process, the cross entropy loss and the mean of the dice loss between the predicted annotations and the true annotations are calculated for the labeled data. The dice loss is calculated between the predicted annotations of the convolutional network and the Transformer network for the unlabeled data. The parameters are updated using deep learning gradient backpropagation, and the trained convolutional network is continuously trained and optimized to obtain a high-performance medical image segmentation model.

[0048] Specifically, for the labeled data, the predicted label output by the convolutional network branch is , the prediction output of the Transformer network branch is labeled , calculate the cross entropy loss and the mean of the dice loss between the predicted labels of the two branches and the true labels, and get the supervision loss of the two branches respectively 、 The performance of the segmentation network on the labeled data can be evaluated by calculating the mean of the cross entropy loss and the dice loss.

[0049] For unlabeled data, the predicted annotation output by the convolutional network branch is , the prediction output of the Transformer network branch is labeled , calculate the unsupervised loss for the predicted labels of the two branches of each unlabeled sample By calculating the loss, the outputs of the convolutional network and the Transformer network can be gradually made consistent. At the same time, under the image of the labeled data, the predicted output of the unlabeled data is made closer to the true value, thereby improving the semi-supervised segmentation performance.

[0050] The training process uses the weighted summation of supervised loss and unsupervised loss to obtain the total loss value, and then uses the gradient descent algorithm in deep learning to optimize the network parameters to minimize the loss function, thereby optimizing all parameters, including the weight parameters in the convolutional network, Transformer network, and self-attention feature enhancement module. These parameters can be adjusted in the direction of smaller overall loss until the parameters converge, thus obtaining a trained convolutional network obtained through semi-supervised training.

[0051] During the training process of the present invention, the unsupervised loss function based on the Dice coefficient drives the distribution alignment of unlabeled data, prompting the CNN network and the Transformer network branch to complement each other at the feature representation level: the local characteristics of the CNN network make up for the insufficient representation of small structures by the Transformer network, while the global features of the Transformer network can enhance CNN's understanding of the spatial coherence of the lesion area. Under the action of the self-attention feature enhancement module, the interaction of the two features improves the segmentation robustness.

[0052] The self-attention feature enhancement module in this invention can significantly improve the feature discrimination ability of multi-head self-attention. The specific structure is as follows: Figure 2 As shown in the figure: the self-attention feature enhancement module includes a global average pooling operation, an average pooling operation, a maximum pooling operation, a 1×1 convolution operation, a ReLU activation function and a Sigmoid function; the self-attention feature enhancement module has two inputs, one is an intermediate-scale local feature output by the encoder of the convolutional network, and the other is the result of the multi-head self-attention of the Transformer encoder of the corresponding scale after layer normalization and the input residual connection of the multi-head self-attention; the two inputs are spliced ​​to obtain the feature , for features Perform maximum pooling operation, average pooling operation and global average pooling operation respectively.

[0053] The result after the global average pooling operation is processed by a 1×1 convolution operation, a ReLU activation function, and a 1×1 convolution operation to obtain the first branch result;

[0054] The results of the maximum pooling operation and the average pooling operation are spliced ​​together, and then processed by a 1×1 convolution operation, a ReLU activation function, and a 1×1 convolution operation to obtain the second branch result;

[0055] The results of the first branch and the second branch are added element by element through the broadcast mechanism, and then the output value is mapped to the (0,1) interval through the Sigmoid function to generate the attention weight map;

[0056] The generated attention weight map is compared with the feature Perform element-by-element multiplication to obtain the output of the self-attention feature enhancement module .

[0057] Maximum pooling can highlight the significant features in the feature map and extract the maximum response value in each local area, which is crucial for capturing areas with obvious feature differences in medical images; average pooling can obtain the average response of the feature map, while retaining the overall feature trend, smoothing the noise to a certain extent and making the feature more stable; and global average pooling can integrate feature information from a global perspective and obtain the global features of the entire feature map. The process of the three parallel pooling processes is expressed as follows:

[0058]

[0059]

[0060]

[0061] in, 、 and They represent the results of maximum pooling MaxPool, average pooling AvgPool and global average pooling GapAvgPool respectively.

[0062] Assumptions , where H and W represent the length and width of the image respectively, and C represents the number of channels of the image. After the three parallel pooling operations, we can get 、 ,Will and Combined features .

[0063] Will and combined features They are processed respectively by 1×1 convolution operations. The role of 1×1 convolution is to reduce or increase the dimension of features, adjust the number of channels, and realize the linear combination of features and information integration without changing the size of the feature map. It can further optimize the representation of features on the basis of retaining the advantages of the features after pooling, so that it is more in line with the needs of subsequent processing. Then, the features processed by the 1×1 convolution operation are processed by the ReLU activation function. The ReLU activation function can introduce nonlinear factors, enhance the expressive power of the model, and learn more complex feature patterns. The ReLU activation function sets all negative input values ​​to 0 and only retains positive input values. This can effectively suppress noise, while highlighting features with positive responses, and paying more attention to feature information that has a positive contribution to the segmentation task. This part of the process is:

[0064]

[0065]

[0066] in, is the 1×1 convolution kernel weight, is the bias term of the convolution process, is the number of channels after convolution; 、 They are 、 The result after convolution and ReLU activation function processing.

[0067] Then, the two features after ReLU activation function processing 、 The 1×1 convolution operation is then used again. Since the resulting features have different sizes, they are then broadcasted and element-by-element added. Finally, the output values ​​are mapped to the (0, 1) interval using the Sigmoid function to generate an attention weight map. This adaptively assigns weights to different features, emphasizing important features and deemphasizing unimportant ones, thereby focusing attention on the features. This process is as follows:

[0068]

[0069]

[0070]

[0071]

[0072] in, is the 1×1 convolution kernel weight, is the bias term; Represents the attention weight map obtained after enhanced adjustment; It is the result of element-by-element addition after broadcasting; Respectively for The result after performing the convolution operation again.

[0073] Finally, the attention weight map and features Perform element-by-element multiplication as follows:

[0074]

[0075] Among them, "⊙" represents element-by-element multiplication, thereby incorporating the attention weight into the original features, performing weighted adjustment on the original features according to the attention weight, and suppressing unimportant features.

[0076] Experiments show that the self-attention feature enhancement module captures multi-dimensional feature representations through a parallel pooling strategy, dynamically adjusts feature importance, suppresses redundant background interference, and enhances the representation significance of the lesion area, while maintaining computational efficiency and significantly improving the recognition accuracy of tiny lesions and complex anatomical structures.

[0077] The Transformer decoder adopts a layered upsampling strategy and combines it with a position-sensitive attention mechanism to gradually map the global features output by the Transformer encoder to the original image resolution, and finally generates a segmentation prediction map covering the entire domain through pixel-by-pixel classification.

[0078] Furthermore, the convolutional network adopts the U-net convolutional network, and its network structure is: [d 4, d 16, d 32, d64, d 128, d 256, u 256, u 128, u 64, u 32, u 16, u 4], where d represents the convolution-batch normalization-ReLU structure, indicating the encoding operation, [d 4] represents a [kernel size of 3×3, step size of 1, and number of layers of 4] convolution-batch normalization-ReLU structure; [d 16] represents a [kernel size of 3×3, step size of 1, and number of layers of 16] convolution-batch normalization-ReLU structure; [d 32] represents a [kernel size of 3×3, step size of 1, and number of layers of 32] convolution-batch normalization-ReLU structure, and so on;

[0079] u represents a convolution-pooling upsampling structure and denotes the decoding operation. [u 256] indicates a convolution-pooling upsampling structure with a kernel size of 3×3, a stride of 1, a pooling window size of 2×2, and 256 layers; [u 128] indicates a convolution-pooling upsampling structure with a kernel size of 3×3, a stride of 1, a pooling window size of 2×2, and 128 layers. A skip connection mechanism is used in the decoder: low-level features are upsampled to half the original resolution through transposed convolutions, concatenated with adjacent high-level features along the channel dimension, and then sigmoid normalization is used to suppress background noise, gradually restoring the original image resolution.

[0080] Furthermore, part of the Transformer network structure is: [d 4, t 16, t 32, t 64, t 128,t 256, u 256, u 128, u 64, u 32, u 16, u 4], where t represents the Transformer encoder, [t16] represents a Transformer encoder with [image block size of 4×4, number of attention heads of 4, number of layers of 16]; [t32] represents a Transformer encoder with [image block size of 4×4, number of attention heads of 4, number of layers of 32]; d represents the convolution-batch normalization-ReLU structure, [d 4] represents a convolution-batch normalization-ReLU structure with [kernel size of 3×3, stride of 1, number of layers of 4]; u represents the convolution-pooling upsampling structure, [u 256] represents a convolution-pooling upsampling structure with a kernel size of 3×3, a stride of 1, a pooling window size of 2×2, and 256 layers; [u 128] represents a convolution-pooling upsampling structure with a kernel size of 3×3, a stride of 1, a pooling window size of 2×2, and 128 layers, and so on.

[0081] Furthermore, the cross entropy loss function is defined as:

[0082] ,

[0083] The dice loss function is defined as: ,

[0084] Among them, N represents the total number of pixels, J represents the number of categories, represents the probability (0 or 1) that the i-th pixel in the true annotation belongs to the j-th category, Indicates the probability that the i-th pixel in the predicted probability distribution belongs to the j-th class; Represents the probability of the i-th pixel in the predicted probability distribution, The probability of the i-th pixel in the true annotation; Dice loss represents dice loss; CEloss represents cross entropy loss.

[0085] The structures of the self-attention feature enhancement module are: [gap 16, gap 32, gap 64, gap 128] and [[max16, avg 16], [max 32, avg 32], [max 64, avg 64], [max128, avg 128]]. Among them, [gap16] represents a global average pooling structure of [kernel size 1×1, stride 1, and number of layers 16], which is a global average pooling-convolution-ReLU structure; [max 16, avg 16] represents a hybrid pooling structure of [kernel size 1×1, stride 1, and number of layers 16], which is a maximum pooling-average pooling-convolution-ReLU structure. In this embodiment, four Transformer encoders are set. The number of layers of the global average pooling structure of the self-attention feature enhancement module in the four Transformer encoders are 16, 32, 64, and 128 respectively, and the number of layers of the mixed pooling structure of maximum pooling-average pooling-convolution-ReLU are 16, 32, 64, and 128 respectively.

[0086] Example 1:

[0087] This example uses the BraTS2019 dataset for brain tumor segmentation, created by researchers at the University of Oregon. It aims to provide a standard benchmark for evaluating the performance of brain tumor segmentation algorithms. The dataset contains 259 cases, each containing a 155×240×240 3D magnetic resonance image.

[0088] The following details each step and parameter settings:

[0089] Step 1: Screening of raw data.

[0090] Because this method is applied to 2D segmentation of medical images and aims to improve the model's segmentation performance, the 3D images in the dataset were segmented into 2D slices. Slices containing lesions were selected. The same processing was performed on the 3D annotated images to ensure a one-to-one correspondence between the images and the annotations. After processing, a total of 17,399 valid slices were found, representing the selected 2D medical images.

[0091] Step 2: Raw data preprocessing.

[0092] The 2D medical images selected in step 1 are preprocessed by first normalizing the data. The normalization formula is: , and are the minimum and maximum pixel values ​​of the feature in the global dataset, Represents the result of normalized output.

[0093] Normalization is performed to reduce the image pixel values ​​to the range [0, 1]. Each slice is then resized to a single-channel medical image with a size of [256, 256]. The original annotated data in this example is classified into four categories: the whole tumor (WT), tumor enhancement (ET), tumor core (TC), and image background. The original annotated pixel values ​​are 0, 1, 2, and 4, where 0 represents the image background and 1, 2, and 4 represent the three aforementioned lesions. To facilitate network training, the annotated pixel values ​​are adjusted to 0, 1, 2, and 3.

[0094] Step 3: Dataset division.

[0095] The data set processed in step 2 is randomly divided into a training set, a validation set, and a test set, and the data in the three sets do not overlap. The training set is used to train the segmentation network, the validation set is used to select the highest performance model saved during the training process, and the test set is used to test the performance of the training model. In this embodiment, the ratio of the three is set to 7:1:2.

[0096] Step 4: Load medical images and annotated image data.

[0097] The data from step 3 is saved as .h5 files. Each slice is named after the case and the slice number. Each .h5 file contains an image and its corresponding annotation data in .npy format. During training, the h5py library in Python is used to iterate over all files in the training set folder containing the slice image data. The medical images and annotation data of the cases stored therein are then extracted for use in segmentation network training.

[0098] Step 5: Build a semi-supervised medical image segmentation network based on enhanced self-attention features:

[0099] In a convolutional network, the input image size is 256×256. After the convolutional layers and downsampling operations in the encoder, the size of the feature image gradually decreases and the number of channels gradually increases. The convolution operation is a convolution with a kernel of 3, a stride of 1, and a padding of 1. The number of layers in each encoding layer is 4, 16, 32, 64, 128, and 256, respectively. The corresponding number of layers in the decoder is 256, 128, 64, 32, 16, and 4, respectively. Except for 4 and 256, the features output by the encoding layers of other layers are called intermediate-scale local features. The downsampling operation in the decoder is a 2x max pooling.

[0100] After encoding in the intermediate-scale coding layer, the number of image channels of the obtained multi-scale local features is [16, 32, 64, 128] respectively.

[0101] In the Transformer network, the input image needs to be divided into image blocks before being input into the Transformer encoder, and each image block needs to be embedded and encoded to facilitate input into the Transformer encoder. In this embodiment, the image block size is set to 4×4. Image block embedding coding is to convert each image block into a vector representation. The image block conversion operation is: [bc (h p1) (w p2) ->b (hw) (p1 p2 c)], where b is the size of each batch of data, c is the number of channels of the input image, h and w are the size of the input image, and p1 and p2 are the size of the image block. At the same time, in order to introduce the position information of the image block, the position code needs to be added to the embedding vector to help the network understand the spatial arrangement order of the image blocks. Here, absolute position code is used. It is then input into the Transformer encoder.

[0102] The Transformer decoder gradually restores the image size through upsampling and combines the extracted multi-scale features to ultimately produce the predicted output. The number of Transformer encoders matches the number of intermediate scales of the image extracted by the convolutional network, both four, coupling the Transformer encoder with the convolutional network.

[0103] Step 6: Calculate the training loss, continuously optimize the segmentation network, and complete the image segmentation task.

[0104] During the training process, the cross entropy loss function and dice loss function are used to calculate the supervised loss and unsupervised loss, and the two are comprehensively optimized through a weighted strategy to improve the training effect.

[0105] During the training process, for the labeled data, the cross entropy loss and the mean of the dice loss between the predicted label and the true label are calculated to obtain the supervised loss of the two branches respectively. 、 , then the supervision loss of these two branches is added to obtain the supervision loss of the segmentation network as a whole For unlabeled data, an unsupervised loss is calculated between the predicted labels of the convolutional network and the Transformer network. The weighted sum of the supervised loss and the unsupervised loss is used to obtain the overall loss of the segmentation network. The calculation formula is: ,

[0106] in for overall loss; is the dynamic loss weight, which is set to 0.1 in this embodiment.

[0107] The segmentation network is continuously optimized using a stochastic gradient descent optimizer using the overall loss.

[0108] This example uses case image slices as units, training a segmentation network with only 5% and 10% of the labeled data, respectively, to complete the segmentation task. The convolutional network (U-net) used in this task serves as the final test network. The two segmentation evaluation metrics used in this task are DSC and HD95. The Dice Similarity Coefficient is a statistic that measures the degree of overlap between two sets; the greater the similarity between the predicted and true annotations, the higher the Dice Similarity Coefficient. The 95% Hausdorff distance is often used to assess the consistency between predicted and true boundaries; the greater the similarity between the predicted and true annotations, the lower the Dice Similarity Coefficient.

[0109] When the convolutional network is trained using all labeled data, the two evaluation metrics, DSC and HD95, are 68.57% and 10.79 mm, respectively. In contrast, when the proposed method uses only 5% of the labeled data in the training set, DSC and HD95 are 64.01% and 15.34 mm, respectively; and when using only 10% of the labeled data, DSC and HD95 are 66.21% and 10.97 mm, respectively. This demonstrates that the proposed method, even when trained with only a portion of the labeled data, can achieve segmentation performance that reaches the level of fully supervised training, demonstrating its effectiveness.

[0110] When performing semi-supervised medical image segmentation tasks, the present invention innovatively proposes a semi-supervised method based on mutual learning between convolutional networks and Transformer networks with the assistance of a self-attention feature enhancement module. The local features extracted by the convolutional network are further utilized to enhance the Transformer network's ability to extract global image features, thereby improving the quality of pseudo-annotations generated by the Transformer network and thereby improving the performance of the semi-supervised medical image segmentation model.

[0111] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A semi-supervised medical image segmentation method based on enhanced self-attention features, characterized in that: The method includes the following: Obtain a case database, each case corresponding to a lesion medical image and a true annotated image; the annotated image is divided into four categories: the whole tumor (WT) area, the tumor enhancement (ET) area, the tumor core (TC) area, and the image background; Constructing a semi-supervised medical image segmentation network based on enhanced self-attention features: The semi-supervised medical image segmentation network based on enhanced self-attention features includes a Transformer network and a convolutional network; the encoder of the convolutional network is used to extract multi-scale local features of the image; The Transformer network includes an image block embedding coding module, a position encoder, several stacked Transformer encoders, and a Transformer decoder jump-connected to each Transformer encoder; the Transformer encoder includes a multi-head self-attention, layer normalization, self-attention feature enhancement module, a feedforward neural network, layer normalization and an image block merging module connected in sequence, wherein the input of the self-attention feature enhancement module is the intermediate scale local features output by the convolutional network and the result of the multi-head self-attention being connected with the input residual of the multi-head self-attention after layer normalization processing; the output of the self-attention feature enhancement module is processed by the feedforward neural network and layer normalization, and then connected with the output residual of the self-attention feature enhancement module, and then processed by the image block merging module to obtain the output of the current Transformer encoder; the number of Transformer encoders is consistent with the number of intermediate scales of the multi-scale local features extracted by the convolutional network; The self-attention feature enhancement module includes a global average pooling operation, an average pooling operation, a maximum pooling operation, a 1×1 convolution operation, a ReLU activation function, and a Sigmoid function; a local feature of an intermediate scale output by the convolutional network and the result of the multi-head self-attention of the Transformer encoder after layer normalization are connected with the input residual of the multi-head self-attention to obtain the feature Z l , for feature Z l The maximum pooling operation, average pooling operation and global average pooling operation are performed respectively. The result after the global average pooling operation is processed by a 1×1 convolution operation, a ReLU activation function, and a 1×1 convolution operation to obtain the first branch result; The results of the maximum pooling operation and the average pooling operation are concatenated and then processed by a 1×1 convolution operation, a ReLU activation function, and a 1×1 convolution operation to obtain the second branch result; The results of the first branch and the second branch are added element by element through the broadcast mechanism, and then the output value is mapped to the (0,1) interval through the Sigmoid function to generate the attention weight map; The generated attention weight map is compared with the feature Z l Perform element-by-element multiplication to obtain the output Z of the self-attention feature enhancement module l+1 ; The self-attention feature enhancement module strengthens the saliency of the lesion representation, significantly improving the recognition accuracy of small lesions and complex anatomical structures while maintaining computational efficiency; Training a semi-supervised medical image segmentation network based on enhanced self-attention features: The case samples in the case database are divided into labeled data and unlabeled data according to a set ratio. The labeled data is recycled and packaged with the unlabeled data and fed into a semi-supervised medical image segmentation network based on enhanced self-attention features. The labeled and unlabeled data are simultaneously fed into the Transformer network and the convolutional network. During the training process, the cross entropy loss and the mean of the dice loss between the predicted and true labels are calculated for the labeled data. The dice loss is calculated between the predicted labels of the convolutional network and the Transformer network for the unlabeled data. The parameters are updated using deep learning gradient backpropagation, and the trained convolutional network is continuously trained and optimized. Use the trained convolutional network to perform medical image segmentation.

2. The semi-supervised medical image segmentation method based on enhanced self-attention features according to claim 1, characterized in that The convolutional network adopts the U-net convolutional network.

3. The semi-supervised medical image segmentation method based on enhanced self-attention features according to claim 1, characterized in that The lesion medical image includes medical image data generated based on computed tomography (CT) or magnetic resonance imaging (MRI) technology.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 3 can be implemented.

Images