Accurate breast tumor segmentation method based on multi-scale and multi-task fusion UNet model

Abstract

The invention provides an accurate breast tumor segmentation method based on a multi-scale and multi-task fusion UNet model. The method comprises the following steps: (1) obtaining an original tumor medical image and a corresponding tumor region label, and adjusting the size of the original tumor medical image and the corresponding tumor region label; (2) performing Z-Score standardization and random rotation on the tumor image for image enhancement, and dividing a training set, a verification set and a test set according to a proportion of 7: 2: 1; (3) segmenting a target, namely a region of interest; (4) establishing an MTF-UNet model, wherein the MTF-UNet model comprises a multi-branch encoder, a branch self-attention network ADF module and a tumor element prediction auxiliary branch (EPB); (5) training an MTF-UNet model; and (6) adjusting hyper-parameters of the model to further optimize the performance of the model. According to the method, the segmentation of the small target can be enhanced, the generalization ability of the model is improved, the diversity and comprehensiveness of the fused features can still be kept, and the data demand is reduced on the premise that the segmentation performance is not reduced.

Description

Technical Field

[0001] The present invention belongs to artificial intelligence image processing, specifically, is an accurate breast tumor segmentation method based on a multi-scale, multi-task fusion UNet model. Background Art

[0002] Breast tumors are the most common type of cancer in women. Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) plays a vital role in improving breast tumor diagnosis. It provides high-resolution images that allow comprehensive assessment of tumor characteristics. However, due to the inherent similarity between breast regions and segmentation targets in breast MRI, as well as the small size and irregularity of tumor lesions, professionals need to spend a lot of time to avoid misdiagnosis. Auxiliary segmentation models have become valuable tools to aid breast tumor analysis, diagnosis, and treatment selection.

[0003] The development of convolutional neural networks (CNNs), such as AlexNet, has made significant progress in the field of deep learning, but faces the challenge of spatial information loss. Fully convolutional networks (FCNs) address this problem by incorporating skip connections to better integrate features, but it still leads to blurred boundaries. UNet adopts an encoder-decoder architecture and feature concatenation to improve accuracy and boundary continuity in image segmentation. However, when applied to breast tumor segmentation, UNet faces the challenge of breast MRI image problems, and segmentation accuracy still has much room for improvement.

[0004] In order to address the challenges encountered in breast tumor segmentation, researchers have conducted extensive and in-depth research. In order to address the irregular tumor shape and uncertain location, multi-scale strategies are widely adopted. In addition, in order to better distinguish between tumor regions and similar regions, some researchers have adopted magnetic resonance imaging (MRI) enhanced imaging with a multi-stage strategy to improve the performance of tumor segmentation. On the other hand, in order to address the problems of class imbalance and uncertainty, various attention-based modules have emerged. At the same time, some researchers have proposed multi-task models for segmentation and classification that consider the diagnosis of doctors in clinical environments.

[0005] However, each of the above methods has some problems:

[0006] (1) In MRI images, there are some targets with a small pixel ratio, referred to as small targets. In multi-scale models, they usually rely on convolution kernels of different sizes to capture multi-scale feature information. Since small targets have a small coverage area, after multiple convolution operations, the information related to the small targets is blurred or lost. How to enhance the segmentation of small targets is a difficult problem.

[0007] (2) As far as the use of attention mechanism is concerned, most researchers only focus on channel attention and spatial attention, without effectively considering the relationship between different fusion branches, which makes the model unable to fully utilize the complementarity between different features, resulting in incomplete extracted features, affecting the accuracy and stability of the model and limiting the generalization ability of the model.

[0008] (3) Some models only consider feature fusion between different stages of the same encoder, facing significant semantic gaps in the fusion process and ignoring the feature diversity brought by multiple encoders.

[0009] (4) Multi-task models that use classification to assist segmentation and multi-modal models that use MRI data at different time points or intensities to enhance the contrast between tumors and surrounding tissues have improved segmentation performance to some extent, but their data requirements are beyond the capabilities of most researchers. Summary of the invention

[0010] In response to the problems existing in the existing models, the present invention provides an accurate breast tumor segmentation method based on a multi-scale, multi-task fusion UNet model, which can enhance the segmentation of small targets, improve the generalization ability of the model, enable the features to maintain diversity and comprehensiveness after fusion, and reduce data requirements without reducing the segmentation performance.

[0011] The accurate breast tumor segmentation method based on the multi-scale, multi-task fusion UNet model includes the following steps:

[0012] Step 1: Obtain the original tumor medical image to be segmented and the corresponding tumor region label, and adjust them to a uniform size; such as 256×256 size.

[0013] Step 2: Perform Z-Score normalization and random rotation of the tumor image at an angle of -10° to 10° for image enhancement, and divide the training set, validation set, and test set into a ratio of 7:2:1. The specific formula for Z-Score normalization is as follows:

[0014]

[0015] Here x ij is the pixel value of the current position (i, j), μ is the mean of the pixel values ​​at all positions, and σ is the standard deviation of the pixel values ​​at all positions. The Z-Score standardization method is used to process the original image. Through this process, the pixel values ​​in the image will be converted into label values ​​with values ​​0 and 1. Among them, the pixel value of the background area is 0, representing the non-tumor area, and the pixel value of the tumor area is 1; so that the pixel value of the image is converted to a distribution with a mean of 0 and a standard deviation of 1, thereby eliminating the impact of different images due to differences in imaging conditions;

[0016] By randomly rotating the image in the range of -10° to 10°, the diversity of image data is increased, thereby improving the generalization ability and robustness of the model at different angles.

[0017] Step 3: Obtain the number of tumor pixels and background pixels according to the segmentation target as the target of the auxiliary segmentation task. The specific steps are as follows:

[0018] To calculate the number of pixels in the tumor area, the pixel values ​​in the entire image are accumulated to obtain the sum s. Similarly, the number of pixels in the background area is obtained by calculating the total number of pixels minus the difference t between the sum s.

[0019] Through such calculation, the number of pixels in the tumor area and the background area is obtained.

[0020] Step 4: Establish the MTF-UNet model. The MTF-UNet model mainly includes a multi-branch encoder, a branch self-attention network ADF module, and a tumor element prediction auxiliary branch (EPB). The specific steps are as follows:

[0021] (401) In the multi-branch encoder, each part of the encoder of ResNet34, VGG19, and ResUNet++ is divided into branches for feature fusion.

[0022] The specific approach is as follows: three classic neural network models are selected as encoder branches: ResNet34, VGG19 and ResUNet++. Instead of relying on convolution kernels of different sizes to obtain multi-scale contextual information, the same convolution kernel size but different convolution layers are selected for the above three encoder branches. The purpose is to better obtain feature specificity and diversity, and to effectively obtain contextual information and shallow, medium and deep features, and avoid the interference of large convolution kernels on the segmentation of small targets.

[0023] Each branch is divided into four stages. In the first stage, in order to enable the model to learn more specific features, the branch self-attention network ADF module is not used to fuse the features of each branch. On the contrary, in the other three stages, the branch self-attention network ADF module is used to fuse the features of each branch.

[0024] In order to allow each branch to obtain feature information from different levels and scales, this technical solution applies a branch self-attention network ADF module, which dynamically adjusts the fusion weight of each branch, avoiding the fixed or empirically set weights commonly used in traditional methods. This adaptive weight adjustment takes into account the contribution of each branch to the final result, effectively considers the relationship between different fusion branches, and makes full use of the complementarity between different features.

[0025] The features of different branches are then fused so that the model can better understand the feature information of different levels and scales in the image, improve the accuracy and performance of the segmentation task, and enable the model to better understand and distinguish different targets and backgrounds in the image.

[0026] (402) Apply the branch self-attention network ADF module to learn the fusion weights of each branch. The specific steps are as follows:

[0027] In the ADF module, we first generate a tensor A∈R (batchsizes×1×3×3) , which is initialized to all 1. Then, two convolution kernels of size 1×3, stride 1, and padding (1,0) are used to convert A from one channel to h / 4 channels, and then to h channels, where h represents the number of channels of each branch fusion feature. At this time, A∈R (batchsizes×h×3×3) ; Next, use the Softmax function along the rows of A, that is:

[0028] w i1 +w i2 +w i3 =1 (i=1,2,3)

[0029] Then, the features F from each branch encoder are 1i 、F 2i and F 3i The corresponding branch weight w given by A ij Multiply, and then add the weighted features from each branch. This process produces a fused multi-scale feature. The specific fusion formula is as follows:

[0030] F 1i '=w 11 ×F 1i +w 12 ×F 2i (1)

[0031] F 2i '=w 21 ×F 1i +w 22 F 2i +w 23 ×F 3i (2)

[0032] F 3i '=w 32 ×F 2i +w 33 ×F 3i (3)

[0033] After obtaining the fused features, for the auxiliary branches VGG19 and ResNet34, the fused features are directly input into the next convolution module specific to each branch; for the main path based on ResUNet++, the fused features from each auxiliary branch are first connected with the fused features from the main path to obtain feature information from all paths; then, 1×1 convolution is used for dimensionality reduction, and then the SE module is used to obtain channel attention; finally, these features are input into the next encoding module of the main path.

[0034] (403) Add a tumor element prediction auxiliary branch (EPB) in the bottleneck layer of the encoder to perform multi-task auxiliary segmentation. The specific method is as follows:

[0035] The tumor element prediction auxiliary branch (EPB) consists of convolutional layers and fully connected layers. In EPB, the features in the bottleneck of MTF-UNet are first passed through three 3×3 convolutional layers with a stride of 2 and a padding of 1. In addition, the number of channels is halved in each convolutional layer; next, the features are reshaped into a one-dimensional vector along the batchsize dimension and passed through four fully connected layers; in order to prevent overfitting and enhance the generalization ability of the model, Dropout layers are added to the first two fully connected layers, with a dropout rate of 0.1. In addition, the number of neurons in the fully connected layer undergoes the following transformations: 1024 to 512, 512 to 256, 256 to 128, 128 to 64, and finally 64 to 2. The specific formula is as follows:

[0036] x′=Conv3(Conv2(Conv1(x)))

[0037] x″=Reshape(x′)

[0038] y=FC5(FC4(FC3(Dropout2(FC2(Dropout1(FC1(x″)))))))

[0039] The above Conv is a 3×3 convolution operation, and FC is a fully connected neural network.

[0040] Traditional auxiliary tasks often require additional data requirements. Adding EPB for multi-task assisted segmentation provides additional supervision signals by analyzing the number of tumor pixels, which eliminates the need for additional data and helps the model learn better feature representations to obtain more accurate segmentation results.

[0041] Step 8: Train the MTF-UNet model and adjust its output image to a single channel to meet the requirements of the binary segmentation task.

[0042] The specific approach is as follows: the number of output feature channels of the model is set to 1, because this is a segmentation task involving only two categories. To achieve this goal, a Sigmoid activation function is used in the last layer of the model. In this way, the output value of each pixel will be limited to between 0 and 1.

[0043] According to the probability threshold, the position in the output image greater than 0.5 is marked as the tumor area, while the position less than or equal to 0.5 is marked as the background area. In this way, the output image is converted into the form required for the binary segmentation task.

[0044] Through this step, a trained MTF-UNet model is obtained, which can accurately perform binary classification segmentation on the input image and reliably separate the tumor area from the background area in the image.

[0045] Step 9: Adjust the model's hyperparameters based on the results of the validation set to further optimize the model's performance. The specific steps are as follows:

[0046] 1. Adjusting Hyperparameters: During the training process, I improved the performance of the model by adjusting the model’s hyperparameters, such as learning rate, batch size, regularization parameters, etc. I used cross-validation to select the best combination of hyperparameters.

[0047] 2. Train the model: Use the adjusted hyperparameters to retrain the model. Input the training set into the model for training, and adjust the model weights and parameters based on the evaluation results of the validation set.

[0048] 3. Model evaluation: Use the adjusted model parameters to predict the test set and evaluate the performance of the model on the test set. Use various indicators such as accuracy, precision, recall, F1 score, etc. to evaluate the performance of the model.

[0049] Through the above steps, the model's hyperparameters are adjusted according to the results of the validation set to further optimize the model's performance. Finally, the model is applied to the test set using the best training parameters to verify the model's performance on real data. This ensures the model's generalization ability on unseen data to verify the model's true performance.

[0050] The present invention establishes a tumor segmentation model MTF-UNet that utilizes multi-scale and multi-level feature fusion, and introduces an adaptive attention module ADF based on branch relationships to enhance the learning of tumor edge features and improve the accuracy of tumor localization. At the same time, the performance of the segmentation model is further improved by proposing a tumor element prediction auxiliary branch (EPB) that determines the distribution and approximate size of the tumor by analyzing the number of tumor pixels, thereby eliminating the need for additional data. Experimental results show that the model has good performance on multiple public datasets, and the auxiliary task branch can be effectively transferred to other models. In summary, the model shows good performance not only on breast tumors, but also on other types of tumors, providing clinicians with a more accurate tool to assist in tumor diagnosis and treatment planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the MTF-UNet network structure of the present invention.

[0052] Figure 2 Schematic diagram of the MTF-UNet encoder branch of the present invention.

[0053] Figure 3 It is a schematic diagram of the structure of the multi-branch self-attention network ADF module of the present invention.

[0054] Figure 4 It is a schematic diagram of the EPB structure of the tumor element prediction branch of the present invention.

[0055] Figure 5 This is a comparison chart of the segmentation results of different models of the present invention on a private breast tumor dataset.

[0056] Figure 6 This is a comparison chart of the segmentation results of different models of the present invention on the public dataset BUSI.

[0057] Figure 7 This is a comparison chart of the segmentation results of different models of the present invention on the public dataset Kvasir-SEGD. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the technical means adopted by the present invention are further described below in combination with specific embodiments and with reference to the accompanying drawings.

[0059] See also Figure 1-Figure 7 It can be seen that the present invention is an accurate breast tumor segmentation method based on a multi-scale, multi-task fusion UNet model. The present invention is specifically aimed at medical image segmentation tasks, which includes the following steps:

[0060] Step 1: Obtain the original tumor medical image to be segmented and the corresponding tumor region label, and resize them to 256×256 size;

[0061] Step 2: Perform Z-Score normalization and random rotation of the tumor image at an angle of -10° to 10° for image enhancement, and divide the training set, validation set, and test set into a ratio of 7:2:1. The specific formula for Z-Score normalization is as follows:

[0062]

[0063] Here x ij is the pixel value at the current position (i, j), μ is the mean of the pixel values ​​at all positions, and σ is the standard deviation of the pixel values ​​at all positions;

[0064] Step 3: Obtain the number of tumor pixels and background pixels according to the segmentation target as the target of the auxiliary segmentation task. The specific steps are as follows:

[0065] For the two-class segmentation task, the Z-Score normalization method is used to process the original image in step 2. Through this process, the pixel values ​​in the image will be converted into label values ​​with values ​​0 and 1, where the pixel value of the background area is 0, representing the non-tumor area, and the pixel value of the tumor area is 1; in order to calculate the number of pixels in the tumor area, the pixel values ​​in the entire image are accumulated to obtain the sum s. Similarly, the number of pixels in the background area is obtained by calculating the total number of pixels minus the difference t of the sum s.

[0066] The number of pixels in the tumor area and the background area is obtained by calculation;

[0067] Step 4: Establish the MTF-UNet model. The MTF-UNet model mainly includes a multi-branch encoder, a branch self-attention network ADF module, and a tumor element prediction branch (EPB). See the specific flowchart for details. Figure 1 , Figure 3 ,as well as Figure 4 ;

[0068] Step 5: In the encoder, the encoder parts of ResNet34, VGG19, and ResUNet++ are divided into branches for feature fusion.

[0069] The specific approach is as follows: select three classic neural network models as encoder branches: ResNet34, VGG19 and ResUNet++, and divide each branch into four stages (see Figure 2 ).

[0070] In the first stage, in order to enable the model to learn more specific features, the branched self-attention network ADF module is not used to fuse the features of each branch. On the contrary, in the other three stages, the branched self-attention network ADF module is used to fuse the features of each branch.

[0071] In order to allow each branch to obtain feature information from different levels and scales, the features of different branches are fused by applying the branch self-attention network ADF module, allowing the model to better understand the feature information of different levels and scales in the image.

[0072] The above processing method can improve the accuracy and performance of the segmentation task, enabling the model to better understand and distinguish different objects and backgrounds in the image.

[0073] Step 6: Apply the branch self-attention network ADF module to learn the fusion weights of each branch. For the specific network structure, see Figure 3 ; The specific steps are as follows.

[0074] In the ADF module, we first generate a tensor A∈R (batchsizes×1×3×3) , which is initialized to all 1. Then, two convolution kernels with a size of 1×3, a step size of 1, and padding of (1,0) are used to convert A from one channel to h / 4 channels, and then to h channels, where h represents the number of channels of each branch fusion feature; at this time, A∈R (batchsizes×h×3×3) Next, the Softmax function is applied along the rows of A, i.e.

[0075] w i1 +w i2 +w i3 =1(i=1,2,3)

[0076] Then, the features F from each branch encoder are 1i 、F 2i and F 3i The corresponding branch weight w given by A ij Multiply, and then add the weighted features from each branch. This process produces fused multi-scale features. The specific fusion formula is as follows:

[0077] F 1i '=w 11 ×F 1i +w 12 ×F 2i (1)

[0078] F 2i '=w 21 ×F 1i +w 22 F 2i +w23 ×F 3i (2)

[0079] F 3i '=w 32 ×F 2i +w 33 ×F 3i (3)

[0080] After obtaining the fused features, for the auxiliary branches VGG19 and ResNet34, the fused features are directly input into the next convolutional module specific to each branch. However, for the main path based on ResUNet++, the fused features from each auxiliary branch are first concatenated with the fused features from the main path to obtain feature information from all paths. Then, 1×1 convolution is used for dimensionality reduction, and then, the SE module is used to obtain channel attention. Finally, these features are input into the next encoding module of the main path.

[0081] Step 7: Add a tumor element prediction auxiliary branch (EPB) to the bottleneck layer of the encoder to perform multi-task auxiliary segmentation. The specific network structure is shown in Figure 4 , the specific steps are as follows:

[0082] The tumor element prediction auxiliary branch (EPB) consists of convolutional layers and fully connected layers. In the EPB, the features in the bottleneck of MTF-UNet are first passed through three 3×3 convolutional layers with a stride of 2 and a padding of 1. In addition, the number of channels is halved in each convolutional layer. Next, the features are reshaped into a one-dimensional vector along the batchsize dimension and passed through four fully connected layers. In order to prevent overfitting and enhance the generalization ability of the model, Dropout layers with a dropout rate of 0.1 are added to the first two fully connected layers respectively. In addition, the number of neurons in the fully connected layer undergoes the following transformations: 1024 to 512, 512 to 256, 256 to 128, 128 to 64, and finally 64 to 2; the specific formula is as follows:

[0083] x′=Conv3(Conv2(Conv1(x)))

[0084] x″=Reshape(x′)

[0085] y=FC5(FC4(FC3(Dropout2(FC2(Dropout1(FC1(x″)))))))

[0086] The above Conv is a 3×3 convolution operation, and FC is a fully connected neural network.

[0087] Step 8: Train the MTF-UNet model and adjust its output image to a single channel to meet the requirements of the binary segmentation task.

[0088] The specific approach is as follows: the number of output feature channels of the model is set to 1, because this is a segmentation task involving only two categories. To achieve this goal, the Sigmoid activation function is used in the last layer of the model. In this way, the output value of each pixel will be limited to between 0 and 1.

[0089] According to the probability threshold, the position in the output image greater than 0.5 is marked as the tumor area, while the position less than or equal to 0.5 is marked as the background area. In this way, the output image is converted into the form required for the binary segmentation task.

[0090] Through this step, a trained MTF-UNet model is obtained, which can accurately perform binary classification segmentation on the input image and reliably separate the tumor area from the background area in the image.

[0091] Step 9: Adjust the model's hyperparameters based on the results of the validation set to further optimize the model's performance. The specific steps are as follows:

[0092] 1. Adjust hyperparameters: During the training process, the performance of the model can be improved by adjusting the model's hyperparameters, such as learning rate, batch size, regularization parameters, etc. Use cross-validation to select the best hyperparameter combination.

[0093] 3. Retrain the model: Use the adjusted hyperparameters to retrain the model. Input the training set into the model for training, and adjust the model weights and parameters based on the evaluation results of the validation set.

[0094] 4. Model evaluation: Use the adjusted model parameters to predict the test set and evaluate the performance of the model on the test set. Use various indicators such as accuracy, precision, recall, F1 score, etc. to evaluate the performance of the model.

[0095] Through the above steps, the model's hyperparameters are adjusted according to the results of the validation set to further optimize the model's performance. Finally, the model is applied to the test set using the best training parameters to verify the model's performance on real data. Ensure the model's generalization ability on unseen data to verify the model's true performance.

[0096] Loss function: The loss function of this article is mainly based on segmentation loss and generation loss. The overall loss is as follows:

[0097] Loss = L seg +L gen

[0098] Here, L seg represents the segmentation loss, L gen Indicates element generation loss.

[0099] For segmentation loss, in order to effectively alleviate data imbalance, a combination of the commonly used BceLoss and DiceLoss is used in breast tumor segmentation, as shown in formulas (4), (5), and (6):

[0100] L seg =F bce +L dice (4)

[0101]

[0102]

[0103] Here, N represents the total number of image pixels, y i represents the label, and p i Represents the predicted value.

[0104] In order to increase the importance of the segmentation target, a weighted MSE loss based on the classic mean square error loss function (MSE loss) in the generative model is proposed for the element prediction loss function, as shown in formula (7):

[0105]

[0106] Here, g i ∈R (2×1) is the vector label, q i ∈R (2×1) is the prediction vector, w∈R (2×1) is a weight vector; in the breast tumor segmentation task, the tumor target pixels are assigned a weight of 20, while the background pixels are assigned a weight of 1; in the BUSI segmentation task, the tumor target pixels are assigned a weight of 5, while the background pixels are assigned a weight of 1; in the Kvasir-SEG segmentation task, the tumor target pixels are assigned a weight of 10, while the background pixels are assigned a weight of 1.

[0107] Experimental analysis:

[0108] In order to ensure the effectiveness of each module and comprehensively evaluate the impact of various factors on the model performance, a series of ablation experiments were conducted. In the experiments, the effectiveness of the ADF fusion module, the transferability of the auxiliary branch of tumor element prediction, and the mutual enhancement of each module in the MTF-UNet model were evaluated respectively.

[0109] Table 1: Ablation experiment comparison table

[0110]

[0111] In order to verify the effectiveness of the ADF module, this embodiment compares the method of replacing the ADF module with different forms of branch addition or connection. In addition, in order to verify the significance of applying row-wise Softmax to the fusion matrix before branch fusion, this embodiment designs an ADF module without Softmax. The comparison of segmentation accuracy of different multi-branch fusion methods on the breast tumor dataset is shown in Table 2. As can be seen from Table 2, the MTF model has significant improvements over the baseline ResUNET++ model without adding auxiliary prediction branches. Among them, the accuracy is improved by 0.99%, the IOU index is improved by 0.75%, the F1 is improved by 0.55%, and the MIOU is improved by 0.38%. In addition, it can be inferred from the comparative analysis that the multi-branch fusion strategy of this embodiment is effective. Compared with the baseline model ResUNet++, all fusion methods are better than it, especially in terms of IOU_label.

[0112] Table 2: Comparison of segmentation accuracy of different multi-branch fusion methods on breast tumor dataset

[0113]

[0114] In order to demonstrate the transferability and generalization of the element prediction branch of this embodiment, the improvements of adding the auxiliary branch of tumor element prediction in the baseline models ResUnet++, UNet, CCBANet, and UTNet are compared in Table 3. In addition, in order to better adapt to each model framework, the auxiliary branch of tumor element prediction on each model was slightly modified. As shown in Table 4, most of the evaluation indicators of the models with the EPB module added are significantly higher than their respective baseline models, namely ResUNet++, UNet, UTNet, and CCBANet. Their accuracies are improved by 0.21%, 0.03%, 0.12%, and 1.55%, respectively. The F1 scores are improved by 0.18%, 0.73%, 0.20%, and 0.42%, respectively. The IOU_label scores are improved by 0.20%, 0.80%, 0.30%, and 0.61%, respectively. The MIOU scores are improved by 0.10%, 0.40%, 0.15%, and 0.31%, respectively.

[0115] Table 3 Comparative analysis of the generalization ability of auxiliary branches for tumor element prediction

[0116]

[0117] In addition, in order to prevent the potential mutual inhibition between different modules, the experimental results of integrating one module at a time and all modules together are given in Table 4. From the results in Table 4, it can be seen that not only the single module is effective, but also the performance of the integrated framework MTF-UNet is better than the model of a single module. This verifies the mutually beneficial interaction between modules in the whole model.

[0118] Table 4 Interaction analysis between MTF-UNet modules

[0119]

[0120] In order to verify the effectiveness of the model, it is compared with nine state-of-the-art models on three different types of data sets (MRI breast tumor data, BUSI data set, Kvasir-SEGD data set). The comparison models include: UTNet, ResUNet++, CCBANet, Attention-UNet, UNet, UNext, ACC-UNet, UNet++, ResUNet. Compared with other methods, the method of this embodiment has better performance under different indicators on different data sets.

[0121] The MRI breast tumor research data set contains a total of 82 dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) data. These data are used for the research purpose of breast tumors. Each data point is carefully annotated by a professional doctor. The plane resolution ranges from 0.95 to 1.1 mm, and the spacing between slices is usually 0.5 mm. In addition, the data used for analysis in this embodiment has a coronal sagittal size of 336×336, and most axial slices contain 300 slices.

[0122] The BUSI dataset is a collection of breast ultrasound images for classification and segmentation purposes. It was collected in 2018 and includes images from 600 female patients aged between 25 and 75 years old. The dataset consists of 780 images, and the average size of each image is 500×500 pixels. The Kvasir-SEG dataset contains 1000 polyp images and their corresponding annotation results.

[0123] The image resolutions in the Kvasir-SEG dataset vary, ranging from 332×487 to 1920×1072 pixels. As an open resource, the dataset provides corresponding segmentation masks for gastrointestinal polyp images, and these labels are carefully annotated and verified by experienced gastroenterologists. In order to minimize the impact of image enhancement on model performance, both public datasets use the same data augmentation method used in the MRI breast tumor segmentation task. At the same time, in order to ensure the fairness of the experiment and ensure the adaptability of the model to different segmentation tasks, only the weights of the auxiliary tasks used to predict the target area elements are modified to 5 and 10, and the same training strategy is maintained. The performance comparison between the model designed by the present invention and other models is shown in Tables 5, 6 and 7.

[0124] Table 5 Comparison of segmentation accuracy of different models on MRI breast tumor dataset

[0125]

[0126] Table 6 Comparison of segmentation accuracy of different models on Kvasir-SEGD dataset

[0127]

[0128] Table 7 Comparison of segmentation accuracy of different models on BUSI dataset

[0129]

[0130] By analyzing Table 5, Table 6 and Table 7, it can be found that the model of the present embodiment performs best in terms of generalization. This observation is derived from the performance of CCBANet and ResUNet++ models on three data sets. As can be seen from Table 5, on the breast tumor data set, the performance of ResUNet++ is second only to the model proposed in the present embodiment, ranking second. However, on the other two public data sets, ResUNet++ has a large performance gap compared with the model of the present embodiment. On the other hand, the model of the present embodiment performs second best on the Kvasir-SEGD data set, and performs best on the breast tumor data set and the BUSI data set. Similarly, it can be seen that on the Kvasir-SEGD data set, the results of the model of the present embodiment are second only to the CCBANet model, but the performance of the CCBANet model on the other two data sets is far inferior to the model of the present embodiment. Therefore, the model of the present embodiment shows excellent results on different types of data and has strong generalization.

[0131] pass Figure 5 , Figure 6 and Figure 7 It can be seen that these visualization results can effectively present the performance comparison of different models on different data sets and other related experimental results.

[0132] This embodiment creates a tumor segmentation model of MTF-UNet, which uses a multi-scale and multi-level feature fusion method to perform multi-scale and multi-level feature fusion to enhance the learning of tumor edge features and improve the accuracy of tumor localization. At the same time, the model also introduces a tumor element prediction branch (EPB) without the need for additional data, further improving performance. Experimental results show that the model performs well on multiple public datasets, and the auxiliary task branches can be effectively transferred to other models. In addition, the model performs well not only on breast tumors, but also on other types of tumors, providing clinicians with a more accurate auxiliary tool for tumor diagnosis and treatment planning.

[0133] The specific embodiments described above further illustrate the purpose, technical solutions, beneficial effects, loss functions and experimental analysis of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An accurate breast tumor segmentation method based on a multi-scale, multi-task fusion UNet model, characterized by The following steps are involved: Step 1: Obtain the original tumor medical image to be segmented and the corresponding tumor region label, and adjust them to a uniform size; Step 2: Perform Z-Score normalization and random rotation on the tumor images for image enhancement, and divide the training set, validation set, and test set into a ratio of 7:2:1; Step 3: Obtain the number of tumor pixels and background pixels according to the segmentation target as the target of the auxiliary segmentation task; the segmentation target is the region of interest; Step 4: Establish the MTF-UNet model. The MTF-UNet model includes a multi-branch encoder, a branch self-attention network ADF module, and a tumor element prediction auxiliary branch EPB. The multi-branch encoder includes ResNet34, VGG19, and ResUNet++ encoders. The steps to establish the MTF-UNet model are as follows; (401) In the multi-branch encoder, the encoders of ResNet34, VGG19, and ResUNet++ are used as ResNet34 branch, VGG19 branch, and ResUNet++ branch, respectively, to perform feature fusion; (402) Apply the branch self-attention network ADF module to learn the fusion weights of each branch, where the branch self-attention network ADF module generates a tensor A∈R initialized to all 1s (batchsizes×1×3×3) , and then use two convolution kernels to convert A from one channel to h / 4 channels, and then to h channels. Use the Softmax function along the row of A, multiply the features from each branch encoder by the corresponding branch weight given by A, and then add them together to obtain the fused multi-scale features; (403) Add a tumor element prediction auxiliary branch EPB in the bottleneck layer of the multi-branch encoder to perform multi-task auxiliary segmentation; Step 5: Train the MTF-UNet model and adjust its output image to a single channel to meet the requirements of the binary segmentation task; Step 6: Adjust the model's hyperparameters based on the results of the validation set to further optimize the model's performance.

2. The accurate breast tumor segmentation method based on the multi-scale, multi-task fusion UNet model according to patent claim 1 is characterized in that: The step (401) uses the encoders of ResNet34, VGG19, and ResUNet++ as branches for feature fusion in the multi-branch encoder. The specific process is as follows: based on the relationship between the receptive field and the convolution layer and the concept of skip connection, features of different scales are merged from different encoder branches; since different encoder branches have the same convolution kernel size but different convolution layers, feature specificity and diversity can be better obtained, and context information and shallow, medium and deep features can be effectively obtained; for this reason, encoders of ResNet34, VGG19, and ResUNet++ with hierarchical convolution times, stable model effects, and simple structures are selected as three branches of the MTF-UNet model; and in order to better fuse features of different convolution times, each branch is divided into four stages. In the first stage, in order to enable the model to learn more specific features, the branch self-attention network ADF module is not used to fuse the features of each branch; In the other three stages, the branch self-attention network ADF module is used to fuse the features of each branch, so that each branch can obtain feature information from different levels and scales; By applying the branched self-attention network ADF module, the features of different branches are fused, allowing the MTF-UNet model to better understand the feature information of different levels and scales in the image.

3. The accurate breast tumor segmentation method based on the multi-scale, multi-task fusion UNet model according to patent claim 1 is characterized in that: The step (402) applies the branch self-attention network ADF module to learn the fusion weights of each branch. The specific process is as follows: (1) Generate a tensor A∈R in the branched self-attention network ADF module (batchsizes×1×3×3) , which is initialized to all 1s; (2) Use two convolution kernels of size 1×3, stride 1, and padding (1,0) to convert A from one channel to h / 4 channels, and then to h channels, where h represents the number of channels of each branch fusion feature. At this time, A∈R (batchsizes ×h×3×3) ; (3) Use the Softmax function along the rows of A, that is: w i1 +w i2 +w i3 =1(i=1,2,3); (4) The feature F from each branch encoder of ResNet34, VGG19, and ResUNet++ is 1i 、F 2i and F 3i The corresponding branch weight w given by A ij Multiply, and then add the weighted features from each branch; this process produces fused multi-scale features; the specific fusion formula is as follows: F 1i ‘=w 11 ×F 1i +w 12 ×F 2i (1) F 2i ‘=w 21 ×F 1i +w 22 F 2i +w 23 ×F 3i (2) F 3i ‘=w 32 ×F 2i +w 33 ×F 3i (3) After obtaining the fused features, for the VGG19 branch and the ResNet34 branch, the fused features are directly input into the next convolutional module specific to each branch; (5) The ResUNet++ branch is used as the main path, and the VGG19 branch and the ResNet34 branch are used as auxiliary paths. First, the fused features from each auxiliary path are connected with the fused features from the main path to obtain feature information from all paths; Then, 1×1 convolution is used for dimensionality reduction; next, the SE module of the branched self-attention network ADF module is used to obtain channel attention; finally, these features are input into the next encoding module of the main path, namely the tumor element prediction auxiliary branch EPB.

4. The accurate breast tumor segmentation method based on the multi-scale, multi-task fusion UNet model according to patent claim 1 is characterized in that: The step (403) adds a tumor element prediction auxiliary branch EPB to the bottleneck layer of the multi-branch encoder, with the purpose of reducing the burden of data collection, solving the class imbalance problem in target segmentation, and performing multi-task auxiliary segmentation to provide additional supervision signals to help the model learn better feature representation and obtain more accurate segmentation results, wherein the tumor element prediction auxiliary branch EPB includes a convolutional layer and a fully connected layer, and the multi-task auxiliary segmentation includes the following steps: (1) Pass the features in the bottleneck of MTF-UNet through three 3×3 convolutional layers with a stride of 2 and padding of 1; (2) Reduce the number of channels by half in each convolutional layer; (3) The features are reshaped into one-dimensional vectors along the batchsize dimension and passed through four fully connected layers. In order to prevent overfitting and enhance the generalization ability of the model, a Dropout layer is added to the first two fully connected layers with a dropout rate of 0.

1. In addition, the number of neurons in the fully connected layer undergoes the following transformations: 1024 to 512, 512 to 256, 256 to 128, 128 to 64, and finally 64 to 2. The specific formula is as follows: x′=Conv3(Conv2(Conv1(x))) x″=Reshape(x′) y=FC5(FC4(FC3(Dropout2(FC2(Dropout1(FC1(x″))))))) The above Conv is a 3×3 convolution operation, and FC is a fully connected neural network.

Images