Brain tumor segmentation method and brain tumor segmentation system based on semi-supervised learning

Abstract

The invention provides a brain tumor segmentation method and system based on semi-supervised learning, and relates to the technical field of medical image segmentation, and the method comprises the steps: obtaining an MRI image data set, and obtaining a training data set and a test data set; constructing a segmentation model; inputting the training data set and the test data set into the segmentation model to obtain first prediction results corresponding to the CNN subnet and the Transform subnet, and second prediction results corresponding to the CNN subnet and the Transform subnet; on the basis of a dual uncertain sensing strategy, generating a weight of pseudo supervision loss in the optimized segmentation model according to the unlabeled data and the labeled data; updating the weight of the optimized segmentation model based on an SGD optimization algorithm, iterating for a preset number of times, and storing the weight of the model; testing by using the test data to obtain a plurality of test weights, and selecting the corresponding test weight with the best test effect as the optimal weight; and the segmentation precision is improved.

Description

Technical Field

[0001] The present application generally relates to the field of medical image segmentation technology, and more particularly to a brain tumor segmentation method and a brain tumor segmentation system based on semi-supervised learning. Background Art

[0002] Early diagnosis and precise treatment of brain tumors are the key to improving patient survival rates and quality of life. Medical imaging technologies (such as magnetic resonance imaging, MRI) play an important role in the discovery and evaluation of brain tumors. Accurate segmentation of medical imaging data can assist doctors in identifying tumor areas, assessing tumor size, morphology and location, and providing a reliable basis for subsequent treatment planning and efficacy monitoring. In recent years, automatic segmentation methods based on deep learning have made significant progress in the field of medical imaging. They use computers to automatically extract image features, significantly improving the efficiency and accuracy of segmentation. However, these methods usually rely on a large amount of high-quality annotated data, and the annotation process of medical imaging data is complex and labor-intensive, making it difficult to meet actual clinical needs.

[0003] Semi-supervised learning technology provides a new possibility for solving the above problems. By making full use of a large amount of unlabeled data and a small amount of labeled data, semi-supervised learning can significantly improve the learning ability of the model while reducing the labeling cost. Most of the existing semi-supervised methods are based on convolutional neural networks, such as Unet and its variants, which makes the network lack attention to global features. Recently, the application of Transformer variants in visual tasks has brought new ideas to the task of medical image segmentation. Unlike convolutional neural networks, Transformer can capture global features more effectively and establish long-distance dependencies between different regions through the self-attention mechanism, so as to better handle the complex morphology and fuzzy boundaries of brain tumors, but it is heavily dependent on large-scale high-quality labeled data. Therefore, we provide a brain tumor segmentation method and brain tumor segmentation system based on semi-supervised learning to solve the above problems. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a brain tumor segmentation method and a brain tumor segmentation system based on semi-supervised learning that improve the robustness and segmentation accuracy of the model.

[0005] In a first aspect, the present application provides a brain tumor segmentation method based on semi-supervised learning, comprising the following steps: Acquire an MRI image data set, preprocess the MRI image data set, and divide the preprocessed MRI image data set according to a first preset ratio to obtain a training data set and a test data set; the MRI image data set includes multiple modality image data; the training data set includes labeled data and unlabeled data divided according to a second preset ratio; According to the convolutional neural network and the Transformer model, a segmentation model is constructed; the segmentation model includes a CNN subnet and a Transformer subnet; the decoders of the CNN subnet and the Transformer subnet are both added with a pyramid prediction structure; Inputting the training data set and the test data set into the segmentation model, using the CNN subnet and the Transformer subnet to infer the same unlabeled data, respectively, to obtain first prediction results corresponding to the CNN subnet and the Transformer subnet respectively; and simultaneously using the CNN subnet and the Transformer subnet to infer the same labeled data, respectively, to obtain second prediction results corresponding to the CNN subnet and the Transformer subnet respectively; According to the labeled data and the corresponding second prediction results, the supervision losses corresponding to the CNN subnet and the Transformer subnet are calculated; pseudo labels are generated according to the two first prediction results and the pseudo labels are mutually guided to learn and update parameters to obtain an optimized segmentation model; and based on the dual uncertainty perception strategy, the weight of the pseudo supervision loss is generated in the optimized segmentation model according to the unlabeled data and the labeled data; Based on the SGD optimization algorithm, the weights of the optimized segmentation model are updated, the preset number of iterations are performed and the model weights are saved; and the test data is used for testing to obtain multiple test weights, and the corresponding test weights with the best test effect are selected as the optimal weights.

[0006] According to the technical solution provided in the embodiment of the present application, the MRI image data set is preprocessed, specifically comprising the following steps: The background area irrelevant to the brain in each example of the modality image data is cropped and the cropped image data is normalized to obtain a preprocessed MRI image data set.

[0007] According to the technical solution provided in the embodiment of the present application, after obtaining the training data set and the test data set, before inputting the training data set and the test data set into the segmentation model, the following steps are also included: Performing data enhancement operation on all data of the training data set and the test data set to obtain enhanced training data set and test data set.

[0008] According to the technical solution provided in the embodiment of the present application, the supervision loss corresponding to the CNN subnet is calculated according to the following formula: ; in, is the supervision loss corresponding to the CNN subnet, is the number of decoders corresponding to the CNN subnet, is the number of decoder layers, is the cross entropy loss, For DICE loss, is the multi-scale prediction result of the i-th layer decoder, Labels for annotated data.

[0009] According to the technical solution provided in the embodiment of the present application, based on the dual uncertainty perception strategy, the weight of the pseudo-supervision loss is generated according to the unlabeled data and the labeled data in the optimized segmentation model, and the following steps are specifically included: Comparing the Dice scores predicted by the last-layer decoders of the CNN subnet and the Transformer subnet to obtain a comparison result, and generating a first weight and a second weight according to the comparison result and the performance of the CNN subnet and the Transformer subnet; According to the corresponding prediction results of the CNN subnet and the Transformer subnet, a corresponding uncertainty estimation map is determined, and the Kullback-Leibler divergence between each prediction result and a multi-scale preset average value is calculated, a plurality of the Kullback-Leibler divergences are averaged, and then the voxels are divided into a foreground area and a background area according to the pseudo-label; the uncertainty mean is calculated for the foreground area and the background area respectively, and a third weight and a fourth weight are generated according to the uncertainty mean and an exponential function; The first weight, the second weight, the third weight and the fourth weight are used as weights of the pseudo supervision loss.

[0010] According to the technical solution provided in the embodiment of the present application, the following steps are also included: Based on the multi-scale consistency constraint, respectively calculating a first average value of the first prediction result of the CNN subnet and a second average value of the second prediction result of the Transformer subnet; A multi-scale consistency loss is calculated according to the first average value and the second average value.

[0011] According to the technical solution provided in the embodiment of the present application, the output loss value is calculated according to the following formula: ; in, is the output loss value, is the sum of the supervision losses of the CNN subnet and the Transformer subnet, is the weight coefficient, is the third weight, is the first weight, is the pseudo-supervision loss corresponding to the CNN subnet, The fourth weight, is the second weight, is the pseudo-supervision loss corresponding to the Transformer subnet, is the multi-scale consistency loss.

[0012] In a second aspect, the present application provides a brain tumor segmentation system based on semi-supervised learning, which is used to implement the above-mentioned brain tumor segmentation method based on semi-supervised learning. The brain tumor segmentation system includes: User management module, which is used for user login and registration, uses MySQL database to store registration information and encrypts passwords with bcrypt algorithm; A data loading module, the data loading module is used to obtain an MRI image data set and pre-process the MRI image data set; An inference module, wherein the inference module is used to update the weight of the optimized segmentation model based on the SGD optimization algorithm, iterate a preset number of times and save the model weight; and use the test data to perform testing to obtain multiple test weights, and select the corresponding test weight with the best test effect as the optimal weight; A visualization module is used to display 3D morphology and 2D slices, and supports reviewing historical diagnosis results; the historical diagnosis results at least include images and corresponding annotations.

[0013] It can be seen from the above technical solution that the present application has at least the following beneficial effects: The present application provides a brain tumor segmentation method based on semi-supervised learning, comprising: obtaining an MRI image data set, preprocessing the MRI image data set, and dividing the preprocessed MRI image data set according to a first preset ratio to obtain a training data set and a test data set; the MRI image data set includes multiple modality image data; the training data set includes labeled data and unlabeled data divided according to a second preset ratio; a segmentation model is constructed according to a convolutional neural network and a Transformer model; the segmentation model includes a CNN subnet and a Transformer subnet; the decoders of the CNN subnet and the Transformer subnet are both added with a pyramid prediction structure; the training data set and the test data set are input into the segmentation model, and the CNN subnet and the Transformer subnet are used to infer the same unlabeled data respectively, to obtain the CNN subnet and the Transformer subnet. The invention discloses a method for generating a segmentation model of the invention and a method for generating a segmentation model of the invention. The method comprises the following steps: generating a first prediction result corresponding to each of the unlabeled data and the labeled data by using the CNN subnet and the Transformer subnet respectively; at the same time, the CNN subnet and the Transformer subnet are used to infer the same labeled data respectively, so as to obtain the second prediction results corresponding to the CNN subnet and the Transformer subnet respectively; according to the labeled data and the corresponding second prediction results, the supervision loss corresponding to the CNN subnet and the Transformer subnet is calculated; according to the two first prediction results, pseudo labels are generated respectively and the learning and updating parameters are guided mutually to obtain the optimized segmentation model; and based on the dual uncertainty perception strategy, the weight of the pseudo supervision loss is generated according to the unlabeled data and the labeled data in the optimized segmentation model; based on the SGD optimization algorithm, the weight of the optimized segmentation model is updated, the model weight is saved after a preset number of iterations; and the test data is used for testing to obtain multiple test weights, and the test weight corresponding to the best test effect is selected as the optimal weight.

[0014] In terms of data utilization, this application adopts a semi-supervised learning strategy to divide the training data set into labeled data and unlabeled data according to a second preset ratio. While utilizing a small amount of labeled data, it fully mines the information in a large amount of unlabeled data, effectively alleviating the problem of scarce labeled data, improving model performance while reducing data labeling costs. The segmentation model integrates convolutional neural networks (CNNs) and Transformer models to construct an architecture including CNN subnets and Transformer subnets. The decoders of the CNN subnets and the Transformer subnets are both added with pyramid prediction structures; CNN is good at extracting local features and can accurately capture local details of brain tumors; Transformers perform well in processing long sequence data and global dependencies, and can grasp the image structure as a whole. The combination of the two enables the model to take into account local and global features, and segment complex brain tumor boundaries and shapes more accurately, thereby improving segmentation accuracy. In terms of model architecture, this application uses CNN subnets and Transformer subnets to infer the same unlabeled data and labeled data separately, calculates supervision losses based on prediction results, and specifies learning and updating parameters to each other by generating pseudo labels. This method promotes information interaction between the two subnetworks, allowing the model to be continuously optimized during training, improving the ability to learn brain tumor characteristics, and enhancing the generalization and stability of the model. Generate pseudo-supervisory loss weights based on the dual uncertainty perception strategy, dynamically adjust weights according to the characteristics of unlabeled data and labeled data, and reasonably balance the impact of supervision loss and pseudo-supervision loss on model training, avoid excessive interference of erroneous information in unlabeled data with the model, and ensure that the model can learn stably and effectively when using unlabeled data, further improving segmentation performance. The SGD optimization algorithm is used to update the weight iterations of the segmentation model for a preset number of times and save the model weights; and the test data is used for testing to obtain multiple test weights, and the corresponding test weights with the best test results are selected as the optimal weights to optimize in the direction of reducing the loss function, improve training efficiency, quickly find the optimal weights, and reduce computing resource consumption and training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0016] Figure 1 Flowchart of the brain tumor segmentation method based on semi-supervised learning.

[0017] Figure 2 Structural diagram of the brain tumor segmentation system based on semi-supervised learning.

[0018] Figure 3 This is the framework diagram of the segmentation model.

[0019] Figure 4Framework diagram of the dual uncertainty perception strategy based on unlabeled data.

[0020] Figure 5 Schematic diagram of the VNet model based on convolutional neural network.

[0021] Figure 6 Schematic diagram of the Swin-UNETR model based on the Transformer architecture.

[0022] Figure 7 Schematic diagram of the original image for the slice illustration.

[0023] Figure 8 A schematic diagram of the actual image is shown for the slice illustration.

[0024] Fig. 9 Schematic diagram of the model inference image for the slice illustration.

[0025] Numbers in the figure: 410, user management module; 420, data loading module; 430, reasoning module; 440, visualization module. DETAILED DESCRIPTION

[0026] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] like Figure 1 As shown, the present application provides a brain tumor segmentation method based on semi-supervised learning, comprising the following steps: S100, obtaining an MRI image data set, preprocessing the MRI image data set, and dividing the preprocessed MRI image data set according to a first preset ratio to obtain a training data set and a test data set; the MRI image data set includes multiple modality image data; the training data set includes labeled data and unlabeled data divided according to a second preset ratio.

[0029] It should be noted that MRI (magnetic resonance imaging) is a commonly used medical imaging technology. MRI images of different modalities can provide different tissue information, such as T1-weighted images, T2-weighted images, etc. The combination of these different modal data can provide richer information for brain tumor segmentation. Modal image data refers to brain tumor image data; the source of the MRI image data set is, for example, the publicly available BraTS2020 dataset. For example: MRI image data of brain tumors is obtained from the publicly available BraTS2020 dataset, and the T2 modal images are selected as experimental data, with a total of 369 modal image data.

[0030] The MRI image data set is preprocessed, specifically including the following steps: The background area irrelevant to the brain in each modality image data is cropped and the cropped image data is normalized to obtain a preprocessed MRI image data set.

[0031] Here, in each modality image data, background areas that are not related to the brain are identified and removed. Since the images obtained during MRI scanning may contain a large number of areas outside the brain, such as scalp and tissue outside the skull, these background information are not directly helpful for the brain tumor segmentation task, but may increase data noise and interfere with model learning. By removing irrelevant areas by cropping, we can focus on the brain area, reduce the proportion of background areas outside the brain area, and improve the validity of the data. For example, image segmentation technology, such as threshold segmentation, morphological operations, or more complex deep learning-based segmentation methods, is used to determine the boundaries of the brain area, and then the background part is cropped off. For example, the image is first grayed, and then a suitable threshold is set according to the difference in grayscale values ​​between the brain tissue and the background, and the image is divided into two parts: the brain and the background. Then, morphological operations (such as dilation and corrosion) are used to optimize the segmentation results, and finally cropped according to the segmented brain area boundaries.

[0032] In addition, the cropped modal image data is normalized. Normalization is to map the data to a specific range, usually [0, 1] or [-1, 1], in order to unify the scale of the data and avoid the model being overly sensitive or insensitive to certain features during training due to the large difference in pixel value ranges of different images, thereby affecting the convergence speed and performance of the model. Normalization methods include linear normalization or Z-score normalization.

[0033] The cropped and normalized MRI image data set helps to improve the stability and convergence speed of model training.

[0034] The first preset ratio is, for example, 4:1, and the second preset ratio is, for example, 1:9. The training data set is used to train the model, and the test data set is used to evaluate the performance of the model to verify its segmentation ability on unseen data. The labeled data contains manually labeled brain tumor regions, which provide accurate information for the supervised learning part, while the unlabeled data is the part that uses its potential information in semi-supervised learning. The unlabeled data only contains image information and has no predetermined tumor region label.

[0035] In addition, after obtaining the training data set and the test data set, before inputting the training data set and the test data set into the segmentation model, the following steps are also included: Data enhancement operations are performed on all data in the training data set and the test data set to obtain enhanced training data set and test data set.

[0036] Here, the purpose of data augmentation is to improve the generalization ability of the model. Data augmentation methods include random flipping, selection, and random cropping. Random flipping includes horizontal flipping and vertical flipping. Horizontal flipping is to flip the image along the vertical axis, while vertical flipping is to flip along the horizontal axis. This operation can increase the diversity of data, especially for learning some symmetrical features. It also allows the model to have a more comprehensive understanding of the left-right and up-down features. Random cropping is performed to a size of 96×96×96. The random cropping operation ensures that each cropped image contains the lesion area.

[0037] S200. Construct a segmentation model based on the convolutional neural network and the Transformer model; the segmentation model includes a CNN subnet and a Transformer subnet; the decoders of the CNN subnet and the Transformer subnet are both added with a pyramid prediction structure.

[0038] It should be noted that convolutional neural networks (CNNs) have excellent feature extraction capabilities in the field of image processing and can effectively extract local features of images. For example, through the combination of convolutional layers and pooling layers, feature information at different levels can be extracted from images. The Transformer model has advantages in processing sequence data and has achieved great success in the field of natural language processing. Applying it to image segmentation can consider the global dependencies in the image and make up for the deficiency of CNN that only considers local information. By combining the two, the local feature extraction capability of CNN and the global dependency processing capability of Transformer can be used to capture image information more comprehensively and improve segmentation performance.

[0039] Here, the CNN subnet uses VNet as the basic network, such as Figure 5As shown in the figure, the network structure adopts an encoder-decoder structure. The encoder part gradually extracts image features through 3D convolution operations and downsampling convolution layers, while reducing the spatial dimension of the feature map; the decoder part gradually restores the resolution of the feature map through upsampling deconvolution blocks, and combines jump connections to fuse the shallow features in the encoder with the deep features in the decoder. Specifically, Figure 5 In the encoding and decoding stages, 3×3×3 convolution operations are used to complete feature extraction. Residual connection operations are applied at each stage. Downsampling is completed by convolution operations with a kernel size of 2×2×2 and a stride of 2, which halves the size of the feature but doubles the channel data. Upsampling is achieved through trilinear interpolation. The nonlinear function used in the network is ReLU.

[0040] The Transformer subnet uses Swin-UNETR, such as Figure 6 As shown in the figure, the network structure diagram is a U-shaped network based on the Transformer architecture, and the input data will be divided into multiple 3D patch blocks. Each layer of the encoder is composed of a Transformer module based on a shift window mechanism, and downsampling operations are performed by patch merging. The features of different resolutions of the encoder layer are input to the decoder through jump connections. The decoder concatenates the feature map of the corresponding level with it in series, and then sends it to the residual block containing a 3×3×3 convolution operation for feature extraction. The upsampling operation of the decoder is completed through a deconvolution operation. Specifically, Figure 6 The input first enters the patch partition layer to generate a sequence of 3D patch blocks, and then enters the LinearEmbed layer for feature mapping. The encoder consists of four stages. The feature extraction of each stage relies on the transformer block. The feature extraction idea of ​​the transformer block is mainly derived from the swin-transformer, and the feature extraction operation is completed by combining the multi-head self-attention mechanism based on the shift window mechanism. The downsampling operation at the end of each stage of the encoder mainly achieves halving of the feature resolution through patch merging operations, and doubles the number of feature channels by relying on the linear layer. The feature output generated by each stage of the encoder is input into the residual block containing 3×3×3 convolution operations for feature extraction, and then spliced ​​with the features of the previous stage that have been upsampled by upsampling at the decoder end. The upsampling of the feature map is achieved by deconvolution operations. The spliced ​​features are continued to be sent to the residual block with the same structure as before for merging.

[0041] A pyramid prediction mechanism is added to the decoder of each subnet mentioned above. The upsampling multiples of the decoders at different levels are adjusted according to the resolution of the output feature map, and the feature map is finally restored to the same size as the input image. In the last layer of the decoder, only the convolution operation is retained, and the upsampling step is omitted.

[0042] S300. Input the training data set and the test data set into the segmentation model, use the CNN subnet and the Transformer subnet to infer the same unlabeled data, and obtain the first prediction results corresponding to the CNN subnet and the Transformer subnet respectively; at the same time, use the CNN subnet and the Transformer subnet to infer the same labeled data, and obtain the second prediction results corresponding to the CNN subnet and the Transformer subnet respectively.

[0043] It should be noted that the training data set and the test data set are input into the constructed segmentation model. The reasoning process for unlabeled data: The CNN subnet predicts each pixel or area in the image based on the local features it has learned, and obtains the first prediction result. This prediction result may be the probability distribution of each pixel belonging to a brain tumor or other tissue. Similarly, the Transformer subnet infers the unlabeled data based on its understanding of the global information and obtains the corresponding first prediction result. The reasoning process for labeled data: The CNN subnet predicts the category of each pixel based on the known labeling information and the learned features, and obtains the second prediction result. At the same time, the Transformer subnet infers the labeled data and also produces the corresponding second prediction result. This step allows the two subnets to learn in a supervised manner and adjust their own feature extraction and prediction capabilities according to the labeled information.

[0044] Here, the above-mentioned reasoning process refers to that the segmentation model calculates the predicted segmentation result through forward propagation based on the input modality image data, that is, determines whether each pixel in the image belongs to the brain tumor area or the background area.

[0045] S400. According to the labeled data and the corresponding second prediction results, the supervision losses corresponding to the CNN subnet and the Transformer subnet are calculated respectively; pseudo labels are generated according to the two first prediction results and the two labels are mutually guided to learn and update parameters to obtain an optimized segmentation model; and based on the dual uncertainty perception strategy, the weights of the pseudo supervision losses are generated in the optimized segmentation model according to the unlabeled data and the labeled data.

[0046] It should be noted that the supervised loss is calculated based on the labeled data and the corresponding second prediction results generated by the CNN subnet and the Transformer subnet. Here, the supervised loss function is the cross entropy loss, which guides the training of the model by comparing the difference between the predicted results and the true annotations, so that the predicted results are as close to the true annotations as possible. Here, the supervised loss is a deep supervised loss based on the multi-scale prediction results.

[0047] Pseudo-labels are information similar to labels generated by the prediction results of the model on unlabeled data. Although they are not real label information, they can provide additional supervision signals for model training. Pseudo-labels are generated based on the first prediction results of the two subnets on unlabeled data. Some thresholds or other rules can be used to convert the predicted probability distribution into a certain label. By allowing the two subnets to learn using the pseudo-labels generated by each other, they promote each other and adjust their own parameters to achieve joint optimization, thereby improving the model's learning ability for unlabeled data.

[0048] In addition, based on the dual uncertainty perception strategy, the weight of the pseudo-supervision loss is generated according to the unlabeled data and the labeled data in the optimized segmentation model. This strategy takes into account the uncertainty of the data and dynamically adjusts the weight of the pseudo-supervision loss according to the uncertainty of the data in order to better balance the impact of labeled and unlabeled data on model training and avoid the erroneous information brought by unlabeled data affecting model performance.

[0049] Furthermore, based on the dual uncertainty perception strategy, the weight of the pseudo-supervision loss is generated according to the unlabeled data and the labeled data in the optimized segmentation model, and the following steps are specifically included: Compare the Dice scores predicted by the last-layer decoders of the CNN subnet and the Transformer subnet to obtain a comparison result, and generate a first weight and a second weight according to the comparison result and the performance of the CNN subnet and the Transformer subnet; According to the corresponding prediction results of the CNN subnet and the Transformer subnet, the corresponding uncertainty estimation map is determined, and the Kullback-Leibler divergence between each prediction result and the multi-scale preset average value is calculated, and multiple Kullback-Leibler divergences are averaged and then the voxels are divided into foreground areas and background areas according to the pseudo labels; the uncertainty mean is calculated for the foreground area and the background area respectively, and the third weight and the fourth weight are generated according to the uncertainty mean and the exponential function; The first weight, the second weight, the third weight, and the fourth weight are used as weights of the pseudo-supervision loss.

[0050] Among them, the Dice score is a metric commonly used to evaluate segmentation performance, especially in the field of medical image segmentation. For the prediction results of the last level decoder of the CNN subnet and the Transformer subnet, the Dice score between it and the labeled data (for labeled data) or the pseudo label (for unlabeled data) is calculated. By calculating the Dice score, the performance of the two subnets on the current prediction can be evaluated.

[0051] Comparing the Dice scores of the CNN subnet and the Transformer subnet can intuitively show which subnet has better segmentation performance on the current data. For example, if the Dice score of the CNN subnet is higher than that of the Transformer subnet, it means that the CNN subnet has a better segmentation effect in this prediction.

[0052] Based on the comparison results and the performance of the CNN subnet and the Transformer subnet, the first weight and the second weight are generated. For example, if the Dice score of the CNN subnet is higher, a higher weight may be assigned to it because it performs better in the current situation and should have a greater impact on the final segmentation result.

[0053] According to the corresponding prediction results of the CNN subnet and the Transformer subnet, an uncertainty estimation map is determined. The uncertainty estimation map is designed to quantify the prediction uncertainty of each pixel, reflecting the model's uncertainty about whether the pixel belongs to a certain category (such as a tumor area or a background area). For each prediction result, the Kullback-Leibler (KL) divergence between it and the multi-scale preset average is calculated. KL divergence is used to measure the difference between two probability distributions, which can represent the difference between the prediction result and a preset average here.

[0054] After averaging multiple Kullback-Leibler divergences, the voxels are divided into foreground areas (such as tumor areas) and background areas (such as normal tissue areas) according to the pseudo-labels. According to the pseudo-labels, it can be clear which voxels belong to the tumor and which belong to the background area, and then these voxels are divided into different areas to prepare for the subsequent calculation of the uncertainty of different areas. The uncertainty mean is calculated for the foreground area and the background area respectively. The uncertainty (such as KL divergence) in the foreground area and the background area is averaged to obtain the average uncertainty of the two areas. The third weight and the fourth weight are generated according to the uncertainty mean and the exponential function. The first weight, the second weight, the third weight and the fourth weight are used as the weights of the pseudo-supervisory loss. When calculating the pseudo-supervisory loss, different weights will weight the losses of different parts, making the loss calculation more flexible and accurate.

[0055] Through the dual uncertainty perception strategy, the performance of different subnets and the sample-level confidence of the prediction results are comprehensively considered, and the pseudo-supervisory loss weights generated can help the model use unlabeled data more reasonably, reducing the negative impact of low-confidence pseudo-labels on model training. In addition, the weights are dynamically adjusted according to the performance of different subnets and the uncertainty of the prediction results, making the model training process more flexible and adaptive. Different subnets may perform differently on different data and at different stages. In this way, they can play to their respective strengths while avoiding their respective weaknesses, achieving better coordination and optimization throughout the training process.

[0056] S500, based on the SGD optimization algorithm, the weights of the optimized segmentation model are updated, the preset number of iterations are performed and the model weights are saved; and the test data is used for testing to obtain multiple test weights, and the corresponding test weights with the best test effect are selected as the optimal weights.

[0057] It should be noted that the stochastic gradient descent (SGD) optimization algorithm is used to update the weights of the optimized segmentation model according to the calculated weights of the supervised loss and pseudo-supervised loss. In each iteration, the parameters of the model are adjusted according to the gradient of the loss function to gradually converge the model. By loading the model weights saved in fixed iteration batches during the training process, testing on the test set, and selecting the best result as the optimal model weight, it is used for the actual brain tumor segmentation task to perform accurate brain tumor segmentation operations on new MRI images.

[0058] Specifically, during training, the SGD optimizer is used to optimize the model parameters of the segmentation model. For example, the momentum is set to 0.9, the weight decay is 0.0001, the learning rate is adjusted using the cosine annealing strategy, the maximum learning rate is 0.01, the minimum learning rate is 0.0001, the total number of training iterations is set to 12000, and the learning rate is adjusted every 4000 iterations. The training batch is set to 4, including two labeled data and two unlabeled data.

[0059] During the training phase, labeled data and unlabeled data are input into two sub-networks respectively. Labeled data includes image data and the corresponding tags , where H, W, D are the height, width, and depth of the input data; the unlabeled data only includes image data Each subnet generates four outputs with the same size as the input data based on the pyramid prediction structure. These outputs are passed through the Softmax layer to generate multi-scale prediction results. , , and , and the corresponding label Calculate the supervision loss. For example, the supervision loss corresponding to the CNN subnet is a linear combination of the cross entropy loss and the Dice loss, which is calculated according to the following formula: ; in, is the supervision loss corresponding to the CNN subnet, is the number of decoders corresponding to the CNN subnet, is the number of decoder layers, is the cross entropy loss, For DICE loss, is the multi-scale prediction result of the i-th layer decoder, Labels for annotated data.

[0060] The calculation method of the supervision loss corresponding to the Transformer subnet and the supervision loss corresponding to the CNN subnet is the same, so it will not be repeated here.

[0061] The total supervision loss of the network is calculated according to the following formula: ; in, is the total supervision loss of the network, is the supervision loss corresponding to the CNN subnet, is the supervision loss corresponding to the Transformer subnet.

[0062] In order to reduce the impact of unreliable pseudo-labels on the model optimization direction, a dual uncertainty perception strategy is introduced.

[0063] Based on the multi-scale predictions generated by each subnetwork based on unlabeled data, the uncertainty estimation map is calculated and used to estimate the sample-level uncertainty of the pseudo-label. The result is expressed as the coefficient , Represents, as one of the weights of pseudo-supervision loss. The details are as follows: By calculating the Kullback-Leibler divergence between the multi-scale prediction and the multi-scale prediction average Q, the four results are averaged and expressed as: ; ; ; Where C represents the number of channels. Representing Q channels, Corresponding to the i-th decoder layer prediction uncertainty estimate. The values ​​of different voxels in represent the uncertainty of the voxel at the current resolution, and the average value of the uncertainty at different resolutions is used as an estimate of the uncertainty of the pseudo-labels generated by this subnetwork.

[0064] To avoid foreground regions being diluted by a large number of background voxels, the pseudo labels are used to calculate sample-level uncertainty. The voxels in are divided into the foreground voxel set and background voxel set , calculate their average values ​​and add them up to get , and finally Apply an exponential function to generate a weight factor for the pseudo-supervisory loss . Weight factor Such as: ; ; Where N represents the number of elements in the set and M represents the number of elements in the set.

[0065] From the perspective of the model, the confidence of the pseudo-label is further evaluated. When the model deviates in the initial stage of training or the training direction, its performance usually fluctuates greatly. When a network experiences large performance fluctuations, the pseudo-label generated at this time is not reliable enough, thereby releasing inappropriate teaching signals. Therefore, in each iteration, by using labeled data to monitor the performance of subnetwork segmentation and weighting the pseudo-supervision loss according to the performance of both parties, the quality of pseudo-supervision can be effectively improved. The details are as follows: Convert the predictions of the last decoder layer into a binary map And calculate the Dice coefficient ,as follows: ; in, Represents the dot product, which is the sum of the multiplication of elements at corresponding positions. Represents a norm. When a network's prediction result on the labeled data has a higher Dice coefficient score than another subnet, the pseudo-labels it generates are considered more reliable and continue to maintain their influence. Conversely, for the subnet with poor performance, an adaptive weight coefficient is generated based on the difference in the Dice coefficients predicted by the two subnets. To reduce its pseudo-supervision effect on another subnet.

[0066] ; ; in, Pseudo-supervisory loss for weighting CNN branches, Pseudo-supervisory loss for weighting Transformer branches, The value of is 3.

[0067] Pseudo labels are generated by a single encoding of the output, and they may contain unreliable topological structures, which will mislead the optimization direction of the model. In addition, the extreme prediction differences between different learning paradigms will also affect the stability of model training. Therefore, explicit consistency regularization is introduced. The details are as follows: By imposing a consistency constraint on the average prediction of each subnet at different frequencies, we can ensure that its predictions converge stably. This constraint incorporates multi-scale information and is called multi-scale consistency loss. This is achieved by minimizing the L2 distance between them. like: ; in, Represents the average of multi-scale predictions of the Transformer branch, Represents the average of multi-scale predictions of CNN branches. Represents the L2 norm.

[0068] The loss function of the proposed semi-supervised method consists of three parts: supervision loss, pseudo-supervision loss of the CNN branch, and pseudo-supervision loss of the Transformer branch. like: ; ; ; in, represents the prediction of the last layer of decoder for unlabeled data, Represents the CNN branch, Represents the Transformer branch. Set to This factor is used in the model training process. represents the total number of iterations, Represents the current iteration number; is the output loss value, is the sum of the supervision losses of the CNN subnet and the Transformer subnet, is the weight coefficient, is the third weight, is the first weight, is the supervision loss corresponding to the CNN subnet, The fourth weight, is the second weight, is the supervision loss corresponding to the Transformer subnet, is the multi-scale consistency loss.

[0069] The effectiveness of the proposed semi-supervised segmentation method is verified by using the test set data. The test set data is input into the model, and then the model inference results are compared with the true labels to analyze the model segmentation performance. The experimental results show that the model can use only 10% of the labeled data, combined with a large amount of unlabeled data, so that the model segmentation Dice index is greater than 85%, and the ASD (Average Surface Distance) index reaches 2.63, which fully proves the effectiveness of this method. Figure 7 , Figure 8 and Fig. 9 Visualize the model inference results. Figure 7 The original image diagram is a slice diagram. Figure 8 The real image diagram for the slice diagram is shown in Figure 1. Fig. 9 Schematic diagram of the model inference image for the slice illustration.

[0070] like Figure 3 As shown in the figure, the subscripts c and t represent the CNN subnet and the Transformer subnet respectively, and f() represents the network. c and P t Represents the model's predictions for the labeled data. and represents the model's prediction for unlabeled data. X and Y represent labeled data and their corresponding labels, respectively. Represents unlabeled data. and Indicates based on and Generated pseudo labels. c and k c They are pseudo labels generated based on the pseudo-supervision stage and the supervision stage. Uncertain estimate of w t and k t They are pseudo labels generated based on the pseudo-supervision stage and the supervision stage. uncertain estimates.

[0071] like Figure 4As shown in the figure, in the dual uncertainty strategy, the pseudo-supervision stage generates a process for uncertainty estimation of pseudo labels, that is, a process for generating weights w. The four multi-scale predictions are respectively calculated with the average of the multi-scale predictions to calculate the KL divergence, and four uncertainty estimation maps at different frequencies are obtained. They are added together and averaged to obtain a new uncertainty estimation map, which is used as the uncertainty estimation map of the pseudo-label. Then, according to the guidance of the pseudo-label prediction, the voxels corresponding to the pseudo-label foreground and the pseudo-label background in the uncertainty map are respectively formed into a set, and the average of the two sets is calculated. The results are added to obtain the weights w. This operation of averaging and adding the foreground and background voxels separately is to prevent a large number of background voxels from drowning out the contribution of a small number of foreground voxels to the pseudo-label sample-level uncertainty estimation, and to alleviate the impact of sample imbalance in medical images on training.

[0072] This application combines the advantages of different learning paradigms through cross-pseudo-supervision between VNet and Swin-UNETR, effectively taps the potential of unlabeled data and improves the segmentation performance of the network. At the same time, by adding a pyramid structure at the decoder end of each subnet, the network's learning ability for multi-scale features is enhanced; by adding a dual uncertainty perception strategy during the training process, the network can adaptively adjust the pseudo-supervision loss weights, thereby reducing the impact of low-confidence pseudo-labels on model training and improving the robustness of the model; by adding multi-scale consistency constraints to subnet predictions, the model training process is stabilized, and the impact of topological errors on network training when generating unique-hot-encoded pseudo-labels is alleviated, and the segmentation accuracy of the model is now improved.

[0073] The present application also provides a brain tumor segmentation system based on semi-supervised learning, which is used to implement the above-mentioned brain tumor segmentation method based on semi-supervised learning, such as Figure 2 As shown, the brain tumor segmentation system includes: User management module 410, which is used for user login and registration, uses MySQL database to store registration information and encrypts passwords with bcrypt algorithm; A data loading module 420, the data loading module 420 is used to obtain an MRI image data set and pre-process the MRI image data set; The reasoning module 430 is used to update the weight of the optimized segmentation model based on the SGD optimization algorithm, iterate a preset number of times and save the model weight; and use the test data to test to obtain multiple test weights, and select the corresponding test weight with the best test effect as the optimal weight; Visualization module 440, visualization module 440 is used to display 3D morphology and 2D slices, and support reviewing historical diagnosis results; the historical diagnosis results at least include images and corresponding annotations.

[0074] It should be noted that the user management module 410 implements user login and registration functions, uses the MySQL database to store user information, and uses the bcrypt algorithm to encrypt user passwords to ensure data security and user privacy protection.

[0075] Data loading module 420: responsible for loading brain image data and performing preprocessing operations, including normalization and cropping operations.

[0076] Reasoning module 430: loads the segmentation model weights generated in the training phase, creates a folder to save the prediction results, performs the segmentation task on the input brain image data, generates the reasoning results and stores them in the specified path for subsequent analysis and visualization.

[0077] Visualization module 440: post-processes the segmentation results and displays the characteristics of the brain lesion area in an intuitive form, supports 2D and 3D view display, and provides user annotation function. The module allows users to review historical segmentation results, which is convenient for segmentation effect evaluation and lesion comparison analysis.

[0078] The user management module 410 includes: Users can add new account information through the registration function. When registering, they need to set up an account and enter a password twice to complete the registration; The user enters the account number and password according to the account information created during registration. The system then verifies the account number and password information by comparing them with those in the database. If they are correct, the user enters the brain tumor segmentation interface.

[0079] The data loading module 420 includes: Data reading means automatically jumping to the data saving folder, manually selecting the image data to be segmented, the image data format is nii format, and then reading the nii data; Data preprocessing refers to cropping the data, removing unnecessary background areas, effectively speeding up the module reasoning speed, and then normalizing the data.

[0080] The segmentation model in the inference module 430 is a VNet network with an encoder-decoder structure. The encoder part gradually extracts image features through 3D convolution operations and downsampling convolution layers, while reducing the spatial dimension of the feature map; the decoder part gradually restores the resolution of the feature map through upsampling deconvolution blocks, and combines jump connections to fuse the shallow features in the encoder with the deep features of the decoder.

[0081] Post-processing in the visualization module 440 refers to extracting the contours by using the segmentation mask and superimposing them on the original image, which will make the slices clearer and more intuitive when displayed to the user. The 3D view display is based on the VTK library, which realizes the 3D morphological display of the brain tumor by performing three-dimensional reconstruction of the slices and combining smoothing processing.

[0082] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A brain tumor segmentation method based on semi-supervised learning, characterized in that: The following steps are involved: Acquire an MRI image data set, preprocess the MRI image data set, and divide the preprocessed MRI image data set according to a first preset ratio to obtain a training data set and a test data set; the MRI image data set includes multiple modality image data; the training data set includes labeled data and unlabeled data divided according to a second preset ratio; According to the convolutional neural network and the Transformer model, a segmentation model is constructed; the segmentation model includes a CNN subnet and a Transformer subnet; the decoders of the CNN subnet and the Transformer subnet are both added with a pyramid prediction structure; Inputting the training data set and the test data set into the segmentation model, using the CNN subnet and the Transformer subnet to infer the same unlabeled data, respectively, to obtain first prediction results corresponding to the CNN subnet and the Transformer subnet respectively; and simultaneously using the CNN subnet and the Transformer subnet to infer the same labeled data, respectively, to obtain second prediction results corresponding to the CNN subnet and the Transformer subnet respectively; According to the labeled data and the corresponding second prediction results, the supervision losses corresponding to the CNN subnet and the Transformer subnet are calculated; pseudo labels are generated according to the two first prediction results and the pseudo labels are mutually guided to learn and update parameters to obtain an optimized segmentation model; and based on the dual uncertainty perception strategy, the weight of the pseudo supervision loss is generated in the optimized segmentation model according to the unlabeled data and the labeled data; Based on the SGD optimization algorithm, the weight of the optimized segmentation model is updated, the preset number of iterations is performed, and the model weight is saved; The test data is used to perform a test to obtain a plurality of test weights, and the test weight corresponding to the best test effect is selected as the optimal weight.

2. The brain tumor segmentation method based on semi-supervised learning according to claim 1, characterized in that: Preprocessing the MRI image data set specifically includes the following steps: The background area irrelevant to the brain in each example of the modality image data is cropped and the cropped image data is normalized to obtain a preprocessed MRI image data set.

3. The brain tumor segmentation method based on semi-supervised learning according to claim 1, characterized in that: After obtaining the training data set and the test data set, and before inputting the training data set and the test data set into the segmentation model, the following steps are also included: Performing data enhancement operation on all data of the training data set and the test data set to obtain enhanced training data set and test data set.

4. The brain tumor segmentation method based on semi-supervised learning according to claim 1, characterized in that: The supervision loss corresponding to the CNN subnet is calculated according to the following formula: ； in, is the supervision loss corresponding to the CNN subnet, is the number of decoders corresponding to the CNN subnet, is the number of decoder layers, is the cross entropy loss, For DICE loss, is the multi-scale prediction result of the i-th layer decoder, Labels for annotated data.

5. The brain tumor segmentation method based on semi-supervised learning according to claim 1, characterized in that: Based on the dual uncertainty perception strategy, the weight of the pseudo-supervision loss is generated according to the unlabeled data and the labeled data in the optimized segmentation model, and the following steps are specifically included: Comparing the Dice scores predicted by the last-layer decoders of the CNN subnet and the Transformer subnet to obtain a comparison result, and generating a first weight and a second weight according to the comparison result and the performance of the CNN subnet and the Transformer subnet; According to the corresponding prediction results of the CNN subnet and the Transformer subnet, a corresponding uncertainty estimation map is determined, and the Kullback-Leibler divergence between each prediction result and a multi-scale preset average value is calculated, a plurality of the Kullback-Leibler divergences are averaged, and then the voxels are divided into a foreground area and a background area according to the pseudo-label; the uncertainty mean is calculated for the foreground area and the background area respectively, and a third weight and a fourth weight are generated according to the uncertainty mean and an exponential function; The first weight, the second weight, the third weight and the fourth weight are used as weights of the pseudo supervision loss.

6. The brain tumor segmentation method based on semi-supervised learning according to claim 1, characterized in that: The following steps are also included: Based on the multi-scale consistency constraint, respectively calculating a first average value of the first prediction result of the CNN subnet and a second average value of the second prediction result of the Transformer subnet; A multi-scale consistency loss is calculated according to the first average value and the second average value.

7. The method for brain tumor segmentation based on semi-supervised learning according to claim 6, characterized in that: The output loss value is calculated according to the following formula: ； in, is the output loss value, is the sum of the supervision losses of the CNN subnet and the Transformer subnet, is the weight coefficient, is the third weight, is the first weight, is the pseudo-supervision loss corresponding to the CNN subnet, The fourth weight, is the second weight, is the pseudo-supervision loss corresponding to the Transformer subnet, is the multi-scale consistency loss.

8. A brain tumor segmentation system based on semi-supervised learning, used to implement the brain tumor segmentation method based on semi-supervised learning according to any one of claims 1 to 7, the brain tumor segmentation system comprising: A user management module (410), the user management module (410) is used for user login and registration, using a MySQL database to store registration information and encrypting passwords using a bcrypt algorithm; A data loading module (420), the data loading module (420) is used to obtain an MRI image data set and pre-process the MRI image data set; An inference module (430), wherein the inference module (430) is used to update the weight of the optimized segmentation model based on the SGD optimization algorithm, iterate a preset number of times and save the model weight; and use the test data to perform a test to obtain multiple test weights, and select the test weight corresponding to the best test effect as the optimal weight; A visualization module (440), the visualization module (440) is used to display 3D morphology and 2D slices, and supports reviewing historical diagnosis results; the historical diagnosis results at least include images and corresponding annotations.

Images