A Medical Image Segmentation Method Based on Auxiliary Learning Tasks and Re-Segmentation Constraints

Abstract

The present invention relates to a medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint. The method includes the following steps in sequence: (1) preprocess three-dimensional human brain nuclear magnetic resonance data to obtain a training set and a test set; (2) construct a segmentation network based on an auxiliary learning task and a re-segmentation constraint; (3) input the training set into the segmentation network for training to obtain a trained segmentation network; (4) input the test set into the trained segmentation network, and the segmentation network outputs a segmentation result. By introducing an additional image reconstruction task branch, the present invention helps the segmentation network learn complementary medical image features, thereby helping the model better understand the internal structure of medical images; inputting the reconstruction result into the segmentation network again, comparing the obtained re-segmentation result with the real segmentation map, and providing an additional supervision signal for the training of the segmentation network from the semantic level to further improve the accuracy of the image segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, computer vision has shown extremely high recognition effects in the field of natural image applications and has also received extensive attention in the field of medical image segmentation. Generally, the purpose of segmenting medical images is to make the human tissue structure or pathological structure clearer and more intuitive, or to model relevant tissues through the segmentation results for subsequent auxiliary diagnosis operations. However, the data in medical images is slightly different from the natural image format. In addition to two-dimensional structured data, image data mainly based on MRI or CT is usually three-dimensional structured, which contains the scanning results of the entire organ tissue. In terms of image content, the boundaries of various objects in natural images are relatively obvious. However, the medical images show the human tissue structure and are obtained through professional imaging instruments, and may have characteristics such as unclear edge contours of tissues and complex image brightness changes.

[0003] Currently, with the rapid iterative update of deep learning algorithms, researchers have made a series of improvements to natural image segmentation models and applied these models to the field of medical image segmentation. Compared with traditional medical image segmentation methods, there has been a significant improvement in segmentation accuracy. Therefore, traditional medical image segmentation methods have gradually been replaced by deep learning methods. The deep learning method does not require artificial feature acquisition like the traditional method and does not produce differences brought by prior knowledge, so it shows excellent performance in the field of medical image segmentation. Against the background of the increasing demand for intelligent medical tasks, existing medical image segmentation methods using deep learning often require training by modeling large-scale labeled data, while medical image data is smaller in scale than general data. Therefore, existing medical image segmentation models usually have difficulty fully extracting relevant discriminative features for characterization and segmentation. Therefore, these limitations mentioned above have led to the fact that existing deep learning-based medical image segmentation work still needs to be further improved in terms of the accuracy of segmentation performance. Summary of the Invention

[0004] The purpose of the present invention is to provide a medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint, which improves the image segmentation accuracy of the main segmentation task by constructing an auxiliary task of image reconstruction, and at the same time, further constrains the network through the secondary segmentation of the reconstructed image during the model training stage to further improve the accuracy of the segmentation result.

[0005] To achieve the above object, the present invention adopts the following technical solutions: A medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint, the method comprising the following steps in sequence:

[0006] (1) Preprocess the three-dimensional human brain nuclear magnetic resonance data to obtain a training set and a test set;

[0007] (2) Construct a segmentation network based on an auxiliary learning task and a re-segmentation constraint;

[0008] (3) Input the training set into the segmentation network for training to obtain a trained segmentation network;

[0009] (4) Input the test set into the trained segmentation network, and the segmentation network outputs to obtain a segmentation result.

[0010] The step (1) specifically includes the following steps:

[0011] (2a) The three-dimensional human brain nuclear magnetic resonance data includes four modalities: T1, T1c, T2, and FLAIR. Merge the three-dimensional human brain nuclear magnetic resonance data of the four modalities. The original size of the data of these four modalities is 240*240*155, and generate four-channel three-dimensional data with a size of 4*240*240*155, where 4 represents the number of modalities, 155 represents the number of two-dimensional slices contained in each three-dimensional human brain nuclear magnetic resonance data, and 240*240 respectively represent the height and width of the image;

[0012] (2b) Convert the merged three-dimensional human brain image data from the nii format to the numpy format;

[0013] (2c) Normalize the converted data using the zero-mean normalization method;

[0014] (2d) Divide the normalized image into a training set and a test set in a ratio of 7:3 according to random partitioning;

[0015] (2e) Randomly crop the training set to obtain training set data with a size of 4*128*128*128.

[0016] In step (2), the segmentation network includes a first encoding module, a second encoding module, a first decoding module, a second decoding module, and a third decoding module;

[0017] Both the first encoding module and the second encoding module are composed of four convolutional blocks and three max-pooling downsampling layers. The four convolutional blocks include the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block. The first convolutional block includes a first convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a second convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the second convolutional block includes a third convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a fourth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the third convolutional block includes a fifth convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a sixth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the fourth convolutional block includes a seventh convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, an eighth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the three max-pooling downsampling layers include a first max-pooling downsampling layer, a second max-pooling downsampling layer, and a third max-pooling downsampling layer;

[0018] Both the first decoding module, the second decoding module, and the third decoding module are composed of three transposed convolutional blocks and three upsampling layers. The three transposed convolutional blocks include a first transposed convolutional block, a second transposed convolutional block, and a third transposed convolutional block. The first transposed convolutional block includes a ninth convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a tenth convolutional layer, a fourth batch normalization layer, and a fourth rectified linear unit activation layer; the second transposed convolutional block includes an eleventh convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a twelfth convolutional layer, a fourth batch normalization layer, and a fourth rectified linear unit activation layer; the third transposed convolutional block includes a thirteenth convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a fourteenth convolutional layer, a fourth batch normalization layer, a fourth rectified linear unit activation layer, and a fifteenth convolutional layer; the three upsampling layers include a first upsampling layer, a second upsampling layer, and a third upsampling layer.

[0019] Step (3) specifically includes the following steps:

[0020] (3a) Input the training set into the first encoding module batch by batch. The first encoding module encodes the input data to obtain a first feature map;

[0021] (3b) Input the first feature map into the first decoding module and the second decoding module in parallel to implement the forward propagation of the segmentation network. The first decoding module outputs a reconstruction result, and the second decoding module outputs a segmentation result;

[0022] (3c) Input the reconstruction result into the second encoding module to obtain a second feature map;

[0023] (3d) Input the second feature map into the third decoding module to implement the forward propagation of the network and obtain the re-segmentation result;

[0024] (3e) Compare the segmentation result with the corresponding ground truth segmentation map, and calculate the segmentation loss through the Dice loss function; compare the re-segmentation result with the corresponding ground truth segmentation map, and calculate the re-segmentation loss through the Dice loss function. The calculation formula of the Dice loss function is:

[0025]

[0026] where X is the ground truth segmentation map; when calculating the segmentation loss, Y is the segmentation result, and when calculating the re-segmentation loss, Y is the re-segmentation result; compare the reconstruction result obtained in step (3b) with the training set data to be segmented input into the segmentation network, and calculate the reconstruction loss through the cross-entropy loss function;

[0027] (3f) Perform weighted summation on the segmentation loss, re-segmentation loss, and reconstruction loss obtained in step (3e) to obtain the total loss result, and use the gradient descent algorithm to perform backpropagation training on the segmentation network;

[0028] (3g) After the number of training times of the segmentation network reaches the set number of training times, obtain the trained segmentation network.

[0029] The convolution kernel size of the first convolutional layer is 3×3×3, and the number of convolution kernels is 32; the convolution kernel size of the second convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the third convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the fourth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the fifth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the sixth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the seventh convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eighth convolutional layer is 3×3×3, and the number of convolution kernels is 512;

[0030] The sizes of the first max-pooling downsampling layer, the second max-pooling downsampling layer, and the third max-pooling downsampling layer are all 2×2×2;

[0031] The convolution kernel size of the ninth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the tenth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eleventh convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the twelfth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the thirteenth convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the fourteenth convolutional layer is 3×3×3, and the number of convolution kernels is 64; the size of the fifteenth convolutional layer is 3×3×3, and the number of convolution kernels is 4.

[0032] The first convolutional block of the first encoding module serves as the input port of the segmentation network. The output result of the first convolutional block of the first encoding module is input into the first max-pooling downsampling layer of the first encoding module. The output result of the first max-pooling downsampling layer of the first encoding module is input into the second convolutional block of the first encoding module. The output result of the second convolutional block of the first encoding module is input into the second max-pooling downsampling layer of the first encoding module. The output result of the second max-pooling downsampling layer of the first encoding module is input into the third convolutional block of the first encoding module. The result of the third convolutional block of the first encoding module is input into the third max-pooling downsampling layer of the first encoding module. The output result of the third max-pooling downsampling layer of the first encoding module is input into the fourth convolutional block of the first encoding module. The output result of the fourth convolutional block of the first encoding module is input in parallel into the first upsampling layer of the first decoding module and the first upsampling layer of the second decoding module. The output result of the first upsampling layer of the first decoding module and the output result of the third convolutional block of the first encoding module are concatenated to obtain a first concatenated result. The output result of the first upsampling layer of the second decoding module and the output result of the third convolutional block of the first encoding module are concatenated to obtain a second concatenated result. The first concatenated result is input into the first transposed convolutional block of the first decoding module. The second concatenated result is input into the first transposed convolutional block of the second decoding module. The output result of the first transposed convolutional block of the first decoding module is input into the second upsampling layer of the first decoding module. The output result of the first transposed convolutional block of the second decoding module is input into the second upsampling layer of the second decoding module. The output result of the second upsampling layer of the first decoding module and the output result of the second convolutional block of the first encoding module are concatenated to obtain a third concatenated result. The output result of the second upsampling layer of the second decoding module and the output result of the second convolutional block of the first encoding module are concatenated to obtain a fourth concatenated result. The third concatenated result is input into the second transposed convolutional block of the first decoding module. The fourth concatenated result is input into the second transposed convolutional block of the second decoding module. The output result of the second transposed convolutional block of the first decoding module is input into the third upsampling layer of the first decoding module. The output result of the second transposed convolutional block of the second decoding module is input into the third upsampling layer of the second decoding module. The output result of the third upsampling layer of the first decoding module and the output result of the first convolutional block of the first encoding module are concatenated to obtain a fifth concatenated result. The output result of the third upsampling layer of the second decoding module and the output result of the first convolutional block of the first encoding module are concatenated to obtain a sixth concatenated result. The fifth concatenated result is input into the third transposed convolutional block of the first decoding module. The sixth concatenated result is input into the third transposed convolutional block of the second decoding module. The first decoding module outputs a reconstruction result, and the second decoding module outputs a segmentation result. The reconstruction result is input into the first convolutional block of the second encoder. The output result of the first convolutional block of the second encoder is input into the first max-pooling downsampling layer of the second encoder.The output result of the first max pooling downsampling layer is input into the second convolutional block of the second encoder. The output result of the second convolutional block is input into the second max pooling downsampling layer of the second encoder. The output result of the second max pooling downsampling layer of the second encoder is input into the third convolutional block of the second encoder. The output result of the third convolutional block of the second encoder is input into the third max pooling downsampling layer of the second encoder. The output result of the third max pooling downsampling layer of the second encoder is input into the fourth convolutional block of the second encoder. The output result of the fourth convolutional block of the second encoder is input into the first upsampling layer of the third decoder. The output of the first upsampling layer of the third decoder and the output of the third convolutional block of the second encoder are concatenated, and the concatenated result is input into the first deconvolutional block of the third decoder. The output result of the first deconvolutional block of the third decoder is input into the second upsampling layer of the third decoder. The output of the second upsampling layer of the third decoder and the output of the second convolutional block of the second encoder are concatenated, and the concatenated result is input into the second deconvolutional block of the third decoder. The output result of the second deconvolutional block of the third decoder is input into the third upsampling layer of the third decoder. The output of the third upsampling layer of the third decoder and the output of the first convolutional block of the second encoder are concatenated, and the concatenated result is input into the third deconvolutional block of the third decoder to obtain the re-segmentation result.

[0033] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, by introducing an additional image reconstruction task branch, the present invention helps the segmentation network learn complementary medical image features, thereby helping the model better understand the internal structure of medical images; Second, the reconstruction result is input into the segmentation network again, and the obtained re-segmentation result is compared with the real segmentation map, providing an additional supervision signal for the training of the segmentation network from the semantic level to improve the accuracy of the further image segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of the method of the present invention;

[0035] Figure 2 is a schematic structural diagram of the segmentation network in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0036] As Figure 1 shown, a medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint, the method includes the following steps in sequence:

[0037] (1) Preprocess the three-dimensional human brain nuclear magnetic resonance data to obtain a training set and a test set;

[0038] (2) Construct a segmentation network based on an auxiliary learning task and a re-segmentation constraint;

[0039] (3) Input the training set into the segmentation network for training to obtain a trained segmentation network;

[0040] (4) Input the test set into the trained segmentation network, and the segmentation network outputs the segmentation result.

[0041] The specific steps of step (1) include the following steps:

[0042] (2a) The three-dimensional human brain magnetic resonance data includes four modalities: T1, T1c, T2, and FLAIR. Merge the three-dimensional human brain magnetic resonance data of the four modalities. The original size of the data of these four modalities is 240*240*155, and generate four-channel three-dimensional data with a size of 4*240*240*155, where 4 represents the number of modalities, 155 represents the number of two-dimensional slices contained in each three-dimensional human brain magnetic resonance data, and 240*240 respectively represent the height and width of the image;

[0043] (2b) Convert the merged three-dimensional human brain image data from the nii format to the numpy format;

[0044] (2c) Normalize the converted data using the zero-mean normalization method;

[0045] (2d) According to the random division, divide the normalized image into a training set and a test set in a ratio of 7:3;

[0046] (2e) Randomly crop the training set to obtain training set data with a size of 4*128*128*128.

[0047] In step (2), as Figure 2 shown, the segmentation network includes a first encoding module, a second encoding module, a first decoding module, a second decoding module, and a third decoding module;

[0048] Both the first encoding module and the second encoding module are composed of four convolutional blocks and three max-pooling downsampling layers. The four convolutional blocks include the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block. The first convolutional block includes a first convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a second convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the second convolutional block includes a third convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a fourth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the third convolutional block includes a fifth convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a sixth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the fourth convolutional block includes a seventh convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, an eighth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the three max-pooling downsampling layers include a first max-pooling downsampling layer, a second max-pooling downsampling layer, and a third max-pooling downsampling layer;

[0049] Both the first decoding module, the second decoding module, and the third decoding module are composed of three transposed convolutional blocks and three upsampling layers. The three transposed convolutional blocks include a first transposed convolutional block, a second transposed convolutional block, and a third transposed convolutional block. The first transposed convolutional block includes a ninth convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a tenth convolutional layer, a fourth batch normalization layer, and a fourth rectified linear unit activation layer; the second transposed convolutional block includes an eleventh convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a twelfth convolutional layer, a fourth batch normalization layer, and a fourth rectified linear unit activation layer; the third transposed convolutional block includes a thirteenth convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a fourteenth convolutional layer, a fourth batch normalization layer, a fourth rectified linear unit activation layer, and a fifteenth convolutional layer; the three upsampling layers include a first upsampling layer, a second upsampling layer, and a third upsampling layer.

[0050] Step (3) specifically includes the following steps:

[0051] (3a) Input the training set into the first encoding module batch by batch. The first encoding module encodes the input data to obtain a first feature map;

[0052] (3b) Input the first feature map into the first decoding module and the second decoding module in parallel to implement the forward propagation of the segmentation network. The first decoding module outputs a reconstruction result, and the second decoding module outputs a segmentation result;

[0053] (3c) Input the reconstruction result into the second encoding module to obtain a second feature map;

[0054] (3d) Input the second feature map into the third decoding module to implement the forward propagation of the network and obtain the re-segmentation result;

[0055] (3e) Compare the segmentation result with the corresponding ground truth segmentation map, and calculate the segmentation loss through the Dice loss function; compare the re-segmentation result with the corresponding ground truth segmentation map, and calculate the re-segmentation loss through the Dice loss function. The calculation formula of the Dice loss function is:

[0056]

[0057] where X is the ground truth segmentation map; when calculating the segmentation loss, Y is the segmentation result, and when calculating the re-segmentation loss, Y is the re-segmentation result; compare the reconstruction result obtained in step (3b) with the training set data to be segmented input into the segmentation network, and calculate the reconstruction loss through the cross-entropy loss function;

[0058] (3f) Perform weighted summation on the segmentation loss, re-segmentation loss, and reconstruction loss obtained in step (3e) to obtain the total loss result, and use the gradient descent algorithm to backpropagate and train the segmentation network;

[0059] (3g) After the number of training times of the segmentation network reaches the set number of training times, obtain the trained segmentation network.

[0060] The convolution kernel size of the first convolutional layer is 3×3×3, and the number of convolution kernels is 32; the convolution kernel size of the second convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the third convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the fourth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the fifth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the sixth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the seventh convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eighth convolutional layer is 3×3×3, and the number of convolution kernels is 512;

[0061] The sizes of the first max-pooling downsampling layer, the second max-pooling downsampling layer, and the third max-pooling downsampling layer are all 2×2×2;

[0062] The convolution kernel size of the ninth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the tenth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eleventh convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the twelfth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the thirteenth convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the fourteenth convolutional layer is 3×3×3, and the number of convolution kernels is 64; the size of the fifteenth convolutional layer is 3×3×3, and the number of convolution kernels is 4.

[0063] The first convolutional block of the first encoding module serves as the input port of the segmentation network. The output result of the first convolutional block of the first encoding module is input into the first max-pooling downsampling layer of the first encoding module. The output result of the first max-pooling downsampling layer of the first encoding module is input into the second convolutional block of the first encoding module. The output result of the second convolutional block of the first encoding module is input into the second max-pooling downsampling layer of the first encoding module. The output result of the second max-pooling downsampling layer of the first encoding module is input into the third convolutional block of the first encoding module. The result of the third convolutional block of the first encoding module is input into the third max-pooling downsampling layer of the first encoding module. The output result of the third max-pooling downsampling layer of the first encoding module is input into the fourth convolutional block of the first encoding module. The output result of the fourth convolutional block of the first encoding module is input in parallel into the first upsampling layer of the first decoding module and the first upsampling layer of the second decoding module. The output result of the first upsampling layer of the first decoding module and the output result of the third convolutional block of the first encoding module are concatenated to obtain a first concatenated result. The output result of the first upsampling layer of the second decoding module and the output result of the third convolutional block of the first encoding module are concatenated to obtain a second concatenated result. The first concatenated result is input into the first transposed convolutional block of the first decoding module. The second concatenated result is input into the first transposed convolutional block of the second decoding module. The output result of the first transposed convolutional block of the first decoding module is input into the second upsampling layer of the first decoding module. The output result of the first transposed convolutional block of the second decoding module is input into the second upsampling layer of the second decoding module. The output result of the second upsampling layer of the first decoding module and the output result of the second convolutional block of the first encoding module are concatenated to obtain a third concatenated result. The output result of the second upsampling layer of the second decoding module and the output result of the second convolutional block of the first encoding module are concatenated to obtain a fourth concatenated result. The third concatenated result is input into the second transposed convolutional block of the first decoding module. The fourth concatenated result is input into the second transposed convolutional block of the second decoding module. The output result of the second transposed convolutional block of the first decoding module is input into the third upsampling layer of the first decoding module. The output result of the second transposed convolutional block of the second decoding module is input into the third upsampling layer of the second decoding module. The output result of the third upsampling layer of the first decoding module and the output result of the first convolutional block of the first encoding module are concatenated to obtain a fifth concatenated result. The output result of the third upsampling layer of the second decoding module and the output result of the first convolutional block of the first encoding module are concatenated to obtain a sixth concatenated result. The fifth concatenated result is input into the third transposed convolutional block of the first decoding module. The sixth concatenated result is input into the third transposed convolutional block of the second decoding module. The first decoding module outputs a reconstruction result, and the second decoding module outputs a segmentation result. The reconstruction result is input into the first convolutional block of the second encoder. The output result of the first convolutional block of the second encoder is input into the first max-pooling downsampling layer of the second encoder.The output result of the first max-pooling downsampling layer is input into the second convolutional block of the second encoder. The output result of the second convolutional block is input into the second max-pooling downsampling layer of the second encoder. The output result of the second max-pooling downsampling layer of the second encoder is input into the third convolutional block of the second encoder. The output result of the third convolutional block of the second encoder is input into the third max-pooling downsampling layer of the second encoder. The output result of the third max-pooling downsampling layer of the second encoder is input into the fourth convolutional block of the second encoder. The output result of the fourth convolutional block of the second encoder is input into the first upsampling layer of the third decoder. The output of the first upsampling layer of the third decoder and the output of the third convolutional block of the second encoder are concatenated. The concatenated result is input into the first transposed convolutional block of the third decoder. The output result of the first transposed convolutional block of the third decoder is input into the second upsampling layer of the third decoder. The output of the second upsampling layer of the third decoder and the output of the second convolutional block of the second encoder are concatenated. The concatenated result is input into the second transposed convolutional block of the third decoder. The output result of the second transposed convolutional block of the third decoder is input into the third upsampling layer of the third decoder. The output of the third upsampling layer of the third decoder and the output of the first convolutional block of the second encoder are concatenated. The concatenated result is input into the third transposed convolutional block of the third decoder to obtain the re-segmentation result.

[0064] Embodiment 1

[0065] This invention conducts research on medical image segmentation using a total of 285 cases of 3D MRI data provided by the Brain Tumor Segmentation (BraTS) 2018 Challenge. This dataset consists of four MR sequences, and the size of the 3D brain tumor images for each patient is 240x240x155 voxel sizes. The labels for tumor segmentation include background (label 0), necrotic and non-enhancing tumor (label 1), peritumoral edema (label 2), and GD-enhancing tumor (label 4). This invention adopts a random division method and divides the dataset into a training set and a test set according to a ratio of 7:3, and evaluates the effectiveness of the segmentation algorithm by calculating the segmentation accuracy of the test set. The segmentation accuracy is measured by the Dice score metric, where ET, WT, and TC refer to the enhanced tumor region (label 1), the whole tumor (labels 1, 2, and 4), and the tumor core (labels 1 and 4), respectively. After adding image reconstruction, the multi-task learning model promotes the sharing of features between different tasks to improve the overall learning performance of the network. As a result, the segmentation performance of the three regions of WT, ET, and TC has been improved by 1.06%, 0.11%, and 0.17%, respectively. By introducing the reconstruction result and inputting it again into the model branch, the overall brain tumor segmentation results of the model have been improved by 1.44%, 0.58%, and 1.89%, respectively. This shows that by encouraging and constraining the two segmentation results to be sufficiently similar during the training process, additional supervision signals can be generated at the semantic level to guide the training of the model, so as to learn more feature information related to the segmentation target, thereby further optimizing the segmentation performance of the network.

[0066] Table 1 shows the impact of the image reconstruction task branch on the brain tumor segmentation performance:

[0067] Table 1

[0068]

[0069] Table 2 shows the impact of this invention on the brain tumor segmentation performance:

[0070] Table 2

[0071]

[0072] Through the comparative analysis of the experiments, it can be further seen that this invention has a high classification accuracy.

[0073] In summary, by introducing an additional image reconstruction task branch, this invention helps the segmentation network learn complementary medical image features, thereby helping the model better understand the internal structure of medical images; inputting the reconstruction result again into the segmentation network, comparing the obtained re-segmentation result with the real segmentation map, and providing additional supervision signals for the training of the segmentation network at the semantic level to further improve the accuracy of the image segmentation result.

Claims

1. A medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint, characterized in that: The method comprises the following steps in sequence: (1) Preprocess three-dimensional human brain magnetic resonance data to obtain a training set and a test set; (2) Construct a segmentation network based on an auxiliary learning task and a re-segmentation constraint; (3) Input the training set into the segmentation network for training to obtain a trained segmentation network; (4) Input the test set into the trained segmentation network, and the segmentation network outputs a segmentation result; In step (2), the segmentation network includes a first encoding module, a second encoding module, a first decoding module, a second decoding module, and a third decoding module; Both the first encoding module and the second encoding module are composed of four convolutional blocks and three max-pooling downsampling layers. The four convolutional blocks include a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block. The first convolutional block includes a first convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a second convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the second convolutional block includes a third convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a fourth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the third convolutional block includes a fifth convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, a sixth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the fourth convolutional block includes a seventh convolutional layer, a first batch normalization layer, a first rectified linear unit activation layer, an eighth convolutional layer, a second batch normalization layer, and a second rectified linear unit activation layer; the three max-pooling downsampling layers include a first max-pooling downsampling layer, a second max-pooling downsampling layer, and a third max-pooling downsampling layer; Both the first decoding module, the second decoding module, and the third decoding module are composed of three transposed convolutional blocks and three upsampling layers. The three transposed convolutional blocks include a first transposed convolutional block, a second transposed convolutional block, and a third transposed convolutional block. The first transposed convolutional block includes a ninth convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a tenth convolutional layer, a fourth batch normalization layer, and a fourth rectified linear unit activation layer; the second transposed convolutional block includes an eleventh convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a twelfth convolutional layer, a fourth batch normalization layer, and a fourth rectified linear unit activation layer; the third transposed convolutional block includes a thirteenth convolutional layer, a third batch normalization layer, a third rectified linear unit activation layer, a fourteenth convolutional layer, a fourth batch normalization layer, a fourth rectified linear unit activation layer, and a fifteenth convolutional layer; the three upsampling layers include a first upsampling layer, a second upsampling layer, and a third upsampling layer.

2. The medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint according to claim 1, characterized in that: The specific steps of step (1) are as follows: (2a) The three-dimensional human brain magnetic resonance data includes four modalities: T1, T1c, T2, and FLAIR. The three-dimensional human brain magnetic resonance data of the four modalities are merged. The original size of the data of these four modalities is 240*240*155, and a four-channel three-dimensional data with a size of 4*240*240*155 is generated. Here, 4 represents the number of modalities, 155 represents the number of two-dimensional slices contained in each three-dimensional human brain magnetic resonance data, and 240*240 respectively represent the height and width of the image; (2b) Convert the merged three-dimensional human brain image data from the nii format to the numpy format; (2c) Normalize the converted data using the zero-mean normalization method; (2d) According to the random division, divide the normalized image into a training set and a test set in a ratio of 7:3; (2e) Randomly crop the training set to obtain training set data with a size of 4*128*128*128.

3. The medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint according to claim 1, characterized in that: The step (3) specifically includes the following steps: (3a) Input the training set into the first encoding module batch by batch. The first encoding module encodes the input data to obtain a first feature map; (3b) Input the first feature map into the first decoding module and the second decoding module in parallel to implement the forward propagation of the segmentation network. The first decoding module outputs a reconstruction result, and the second decoding module outputs a segmentation result; (3c) Input the reconstruction result into the second encoding module to obtain a second feature map; (3d) Input the second feature map into the third decoding module to implement the forward propagation of the network and obtain a re-segmentation result; (3e) Compare the segmentation result with the corresponding real segmentation map, and calculate the segmentation loss through the dice loss function; compare the re-segmentation result with the corresponding real segmentation map, and calculate the re-segmentation loss through the dice loss function. The calculation formula of the dice loss function is: where X is the real segmentation map; when calculating the segmentation loss, Y is the segmentation result, and when calculating the re-segmentation loss, Y is the re-segmentation result; compare the reconstruction result obtained in step (3b) with the training set data to be segmented input into the segmentation network, and calculate the reconstruction loss through the cross-entropy loss function; (3f) Perform weighted summation on the segmentation loss, re-segmentation loss, and reconstruction loss obtained in step (3e) to obtain a total loss result, and use the gradient descent algorithm to backpropagate and train the segmentation network; (3g) After the number of training times of the segmentation network reaches the set number of training times, obtain a trained segmentation network.

4. The medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint according to claim 1, characterized in that: The convolution kernel size of the first convolutional layer is 3×3×3, and the number of convolution kernels is 32; the convolution kernel size of the second convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the third convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the fourth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the fifth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the sixth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the seventh convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eighth convolutional layer is 3×3×3, and the number of convolution kernels is 512; The sizes of the first max-pooling downsampling layer, the second max-pooling downsampling layer, and the third max-pooling downsampling layer are all 2×2×2; The convolution kernel size of the ninth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the tenth convolutional layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eleventh convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the twelfth convolutional layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the thirteenth convolutional layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the fourteenth convolutional layer is 3×3×3, and the number of convolution kernels is 64; the size of the fifteenth convolutional layer is 3×3×3, and the number of convolution kernels is 4.

5. The medical image segmentation method based on an auxiliary learning task and a re-segmentation constraint according to claim 1, characterized in that: The first convolutional block of the first encoding module serves as the input port of the segmentation network. The output result of the first convolutional block of the first encoding module is input into the first max-pooling downsampling layer of the first encoding module. The output result of the first max-pooling downsampling layer of the first encoding module is input into the second convolutional block of the first encoding module. The output result of the second convolutional block of the first encoding module is input into the second max-pooling downsampling layer of the first encoding module. The output result of the second max-pooling downsampling layer of the first encoding module is input into the third convolutional block of the first encoding module. The result of the third convolutional block of the first encoding module is input into the third max-pooling downsampling layer of the first encoding module. The output result of the third max-pooling downsampling layer of the first encoding module is input into the fourth convolutional block of the first encoding module. The output result of the fourth convolutional block of the first encoding module is input in parallel into the first upsampling layer of the first decoding module and the first upsampling layer of the second decoding module. The output result of the first upsampling layer of the first decoding module and the output result of the third convolutional block of the first encoding module are concatenated to obtain a first concatenated result. The output result of the first upsampling layer of the second decoding module and the output result of the third convolutional block of the first encoding module are concatenated to obtain a second concatenated result. The first concatenated result is input into the first transposed convolutional block of the first decoding module. The second concatenated result is input into the first transposed convolutional block of the second decoding module. The output result of the first transposed convolutional block of the first decoding module is input into the second upsampling layer of the first decoding module. The output result of the first transposed convolutional block of the second decoding module is input into the second upsampling layer of the second decoding module. The output result of the second upsampling layer of the first decoding module and the output result of the second convolutional block of the first encoding module are concatenated to obtain a third concatenated result. The output result of the second upsampling layer of the second decoding module and the output result of the second convolutional block of the first encoding module are concatenated to obtain a fourth concatenated result. The third concatenated result is input into the second transposed convolutional block of the first decoding module. The fourth concatenated result is input into the second transposed convolutional block of the second decoding module. The output result of the second transposed convolutional block of the first decoding module is input into the third upsampling layer of the first decoding module. The output result of the second transposed convolutional block of the second decoding module is input into the third upsampling layer of the second decoding module. The output result of the third upsampling layer of the first decoding module and the output result of the first convolutional block of the first encoding module are concatenated to obtain a fifth concatenated result. The output result of the third upsampling layer of the second decoding module and the output result of the first convolutional block of the first encoding module are concatenated to obtain a sixth concatenated result. The fifth concatenated result is input into the third transposed convolutional block of the first decoding module. The sixth concatenated result is input into the third transposed convolutional block of the second decoding module. The first decoding module outputs a reconstruction result, and the second decoding module outputs a segmentation result. The reconstruction result is input into the first convolutional block of the second encoder. The output result of the first convolutional block of the second encoder is input into the first max-pooling downsampling layer of the second encoder.The output result of the first max pooling downsampling layer is input into the second convolutional block of the second encoder. The output result of the second convolutional block is input into the second max pooling downsampling layer of the second encoder. The output result of the second max pooling downsampling layer of the second encoder is input into the third convolutional block of the second encoder. The output result of the third convolutional block of the second encoder is input into the third max pooling downsampling layer of the second encoder. The output result of the third max pooling downsampling layer of the second encoder is input into the fourth convolutional block of the second encoder. The output result of the fourth convolutional block of the second encoder is input into the first upsampling layer of the third decoder. The output of the first upsampling layer of the third decoder and the output of the third convolutional block of the second encoder are concatenated. The concatenated result is input into the first transposed convolutional block of the third decoder. The output result of the first transposed convolutional block of the third decoder is input into the second upsampling layer of the third decoder. The output of the second upsampling layer of the third decoder and the output of the second convolutional block of the second encoder are concatenated. The concatenated result is input into the second transposed convolutional block of the third decoder. The output result of the second transposed convolutional block of the third decoder is input into the third upsampling layer of the third decoder. The output of the third upsampling layer of the third decoder and the output of the first convolutional block of the second encoder are concatenated. The concatenated result is input into the third transposed convolutional block of the third decoder to obtain the re-segmentation result.

Images