Three-dimensional medical image segmentation method based on transformer and convolutional neural network

Abstract

The application provides a three-dimensional medical image segmentation method based on a Transformer and a convolutional neural network, and comprises the following steps: step S1: performing class-by-class edge extraction on a label image of an original medical image; step S2: performing random flipping, rotation and scaling processing on the original image, the label image and an edge image of the label image of the original medical image; and step S3: inputting the processed image into a medical image segmentation model to segment various types of tissues and organs; the medical image segmentation model is obtained by training an image data generated by performing edge extraction on a label image of a training set data and then performing random flipping, rotation and scaling processing on an original image, a label image and an edge image of the label image based on a network model of a combination of a Transformer and a convolutional neural network in a morphological structure learning mode. The technical scheme can enhance the accuracy of a final segmentation result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a three-dimensional medical image segmentation method based on Transformer and convolutional neural networks. Background Technology

[0002] As a key step in medical image analysis, automatic segmentation of lesions and organs has become a current research hotspot. This is not only to cope with the increasing number of organ-related diseases, but also because automatic segmentation can distinguish the structural information of different feature data with greatly reduced human interference, thereby enabling better functional and differential analysis.

[0003] Researching 3D segmentation technology for medical images to achieve automated interpretation can provide doctors with more accurate diagnostic basis. Current medical image segmentation methods can be mainly divided into two categories: (1) medical image segmentation methods based on traditional image processing; (2) medical image segmentation methods based on machine learning. Medical image segmentation methods based on traditional image processing mainly consist of feature extraction and classifiers. They do not rely on training samples. Their advantages are fast algorithm running time, short cycle, and visualization of the algorithm process, which allows for a more intuitive view of the intermediate processing results. Medical image segmentation methods based on machine learning mainly include those based on traditional machine learning algorithms and deep learning algorithms. Given the powerful feature representation capabilities of deep learning algorithms, most current image segmentation methods are based on deep learning methods. Deep learning-based medical image segmentation is a data-driven algorithm. This algorithm needs to learn the regularity information between data from a training dataset with a large number of annotations to determine the parameters of the segmentation model. Its advantage is that feature extraction in deep learning is autonomously learned through a large amount of training, reducing human labor costs. Compared with traditional image processing methods, deep learning has higher generalization and the final segmentation effect is usually more accurate.

[0004] While most algorithms have achieved a certain level of accuracy in segmenting medical images, a common drawback of many neural networks, including U-Net, is their lack of interpretability. Because these networks are built by interconnecting numerous convolutional layers, visualizing the features they are learning becomes challenging. This makes the neural network a "black box," posing a challenge in trying to find the root cause of misclassification. Furthermore, CNNs are heavily influenced by dense pixel values, which are not robust to information about object shape. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a three-dimensional medical image segmentation method based on Transformer and convolutional neural networks to enhance the accuracy of the final segmentation results.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a three-dimensional medical image segmentation method based on Transformer and convolutional neural networks, comprising the following steps:

[0007] Step S1: Extract the edges of the label map of the original medical image class by class;

[0008] Step S2: Randomly flip, rotate, and scale the original medical image, label image, and edge image of the label image;

[0009] Step S3: Input the processed image into the medical image segmentation model to segment various tissues and organs.

[0010] In a preferred embodiment: step S1 specifically includes the following steps:

[0011] Step S11: Separate the categories of the label map of the medical image channel by channel using one-hot encoding, with each channel representing a category;

[0012] Step S12: Extract the edges of each separated category channel and then compress them into one channel to obtain the final edges of various tissues and organs.

[0013] In a preferred embodiment, step S2 specifically includes the following steps:

[0014] The image is flipped or rotated using a random number in the interval [0, 1] generated by Python's built-in random function, and the processed image is then scaled to the specified size.

[0015] In a preferred embodiment: step S3 specifically includes the following steps:

[0016] The medical image segmentation model is obtained by training the training set data, extracting the edges of the label image, and then randomly flipping, rotating, and scaling the original image, label image, and label image edge images. The resulting image data is then trained on a network model based on morphological structure learning that combines Transformer and convolutional neural networks.

[0017] In a preferred embodiment: the network model based on Transformer and convolutional neural network combined with morphological structure learning, on the basis of the TransUNet network model architecture, combines the morphological structure module to learn the shape information of the segmentation target, and uses max pooling and average pooling to learn richer features; the network includes modular morphological structure modules, Transformer layers, residual module layers, max pooling and average pooling layers, batch normalization layers and ReLU activation function, cascaded upsampling layers and skip connections; after the image is processed by the network, a predicted segmentation result is generated with the same number of channels and image size as the original image in terms of the number of categories and size, respectively. The predicted segmented image is used to calculate the loss function output value with the real segmented image.

[0018] In a preferred embodiment: three loss functions are used for the segmentation network based on the combination of Transformer and convolutional neural network morphological structure learning: DICE loss function, cross-entropy loss function, and binary classification cross-entropy loss function, which are expressed as follows:

[0019]

[0020]

[0021] in, This represents the value of each pixel in the truth map under One-Hot encoding for each class. This represents the value of each pixel in the prediction image under One-Hot encoding for each class. This represents an image with height m, width n, and number of categories. The domain of all pixels, with shape . .

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The morphological structure module used in this invention makes full use of the shape information of the segmentation target and uses the extracted shape information to improve the accuracy of the final segmentation result;

[0024] 2. This invention employs max pooling and average pooling layers to further extract features from the results of the convolutional neural network used for feature extraction, and uses these features as input to the morphological structure module, laying the foundation for subsequent accurate segmentation of tissues and organs.

[0025] The segmentation network used in this invention, which combines Transformer and convolutional neural network with morphological structure learning, makes full use of the shape information of the segmentation target in medical images, and combines max pooling and average pooling layers to achieve high-precision segmentation of three-dimensional medical images. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall process of a preferred embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0029] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0030] like Figure 1 As shown, the training process of the 3D medical image segmentation method based on the combination of Transformer and convolutional neural network morphological structure learning provided in this embodiment includes the following steps:

[0031] First, the label image of the medical image is separated into different categories channel by channel using one-hot encoding, with each channel representing a category. Edge extraction is then performed on each separated category channel, compressing it into a single channel to obtain the final edges of various tissues and organs. Next, the original medical image, label image, and edge images of the label image are randomly flipped, rotated, and scaled. The built-in Python function `random` is used to randomly generate a number in the interval [0, 1] as the probability of image flipping and rotation, and the processed image is scaled to a specified size. Finally, the processed image is input into a segmentation network based on a combination of Transformer and convolutional neural networks for morphological structure learning to train the model, generating a 3D medical image segmentation model.

[0032] The model testing process uses the trained 3D medical image segmentation model to achieve accurate segmentation of the target.

[0033] This embodiment employs the TransUNet network model architecture, which combines Transformer and convolutional neural networks for morphological structure learning. The difference from the original network lies in the use of a morphological structure module to learn the shape information of the segmentation target, and the use of max pooling and average pooling to learn richer features. The network includes modular morphological structure modules, Transformer layers, residual module layers, max pooling and average pooling layers, batch normalization layers and ReLU activation functions, cascaded upsampling layers, and skip connections. After image processing, the network generates predicted segmentation results with the same number of channels and image size as the original image, including the number of categories and image size. These predicted segmented images are used to calculate the loss function output value with the ground truth segmented image.

[0034] The implementation of a 3D medical image segmentation model, trained using a segmentation network combining Transformer and Convolutional Neural Networks with morphological structure learning, is as follows: First, the model's parameters are initialized by loading a pre-trained model on ImageNet. Then, the original and labeled medical images are read, pre-processed as described above, and input into the network to obtain the segmentation results predicted by the network. The loss function value between the predicted result and the corresponding ground truth image is then calculated. This network uses three loss functions: DICE loss function, cross-entropy loss function, and binary cross-entropy loss function, expressed as follows:

[0035]

[0036]

[0037] in, This represents the value of each pixel in the truth map under One-Hot encoding for each class. This represents the value of each pixel in the prediction image under One-Hot encoding for each class. This represents an image with height m, width n, and number of categories. The domain of all pixels, with shape . ;

[0038] The network parameters are then iteratively updated via backpropagation until training is complete, generating the network model. During model testing, the test dataset is input into the network model generated during training to obtain predicted images of various tissues and organs.

[0039] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of 3D medical image segmentation methods based on the combination of Transformer and convolutional neural networks and morphological structure learning under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A three-dimensional medical image segmentation method based on Transformer and convolutional neural networks, characterized in that, Includes the following steps: Step S1: Extract the edges of the label map of the original medical image class by class; Step S2: Randomly flip, rotate, and scale the original medical image, label image, and edge image of the label image; Step S3: Input the processed image into the medical image segmentation model to segment various tissues and organs; Step S3 specifically includes the following steps: The medical image segmentation model is obtained by training the training set data, extracting the edges of the label image, and then randomly flipping, rotating, and scaling the original image, label image, and label image edge images. The resulting image data is then trained on a network model based on Transformer and convolutional neural network combined with morphological structure learning. This network model, based on Transformer and convolutional neural networks and incorporating morphological structure learning, builds upon the TransUNet network architecture by combining a morphological structure module to learn the shape information of the segmentation target. It employs max pooling and average pooling to learn richer features. The network includes modular morphological structure modules, Transformer layers, residual module layers, max pooling and average pooling layers, batch normalization layers and ReLU activation functions, cascaded upsampling layers, and skip connections. After image processing, the network generates predicted segmentation results with the same number of channels and image size as the original image, including the number of categories and image size. These predicted segmented images are then compared with the ground truth labeled images to calculate the output value of the loss function.

2. The three-dimensional medical image segmentation method based on Transformer and convolutional neural network according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11: Separate the categories of the label map of the medical image channel by channel using one-hot encoding, with each channel representing a category; Step S12: Extract the edges of each separated category channel and then compress them into one channel to obtain the final edges of various tissues and organs.

3. The three-dimensional medical image segmentation method based on Transformer and convolutional neural networks according to claim 1, characterized in that: Step S2 specifically includes the following steps: The image is flipped or rotated using a random number in the interval [0,1] generated by Python's built-in random function, and the processed image is then scaled to the specified size.

Images