Bone tumor medical image segmentation method based on improved mixed attention mechanism and TransUnet model

Abstract

The invention discloses a bone tumor medical image segmentation method based on an improved mixed attention mechanism and a TransUnet model, and belongs to the technical field of medical image segmentation. Comprising the steps of data extraction; labeling the data; preprocessing the data; a segmentation network model is established and trained to obtain an ONC-TransUnet model, and the ONC-TransUnet model is utilized to perform segmentation work of medical images such as bone tumor medical images and the like. According to the method, the mixed attention mechanism is introduced into the field of medical image segmentation and is improved, and the channel attention mechanism and the space attention mechanism are combined with the residual structure, so that the model can extract details and boundaries of the image more meticulously. And meanwhile, the improved mixed attention module is added into the key position of the TransUnet model, so that the model not only can well extract details and boundaries of the image, but also can well capture and understand a context relationship in the image, and more accurate segmentation of the bone tumor medical image is realized. Meanwhile, the ONC-TransUnet model also has good segmentation generalization for other medical image data sets.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and specifically to a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model. Background Art

[0002] Bone tumor is a common, highly malignant tumor originating from the bone marrow. Its characteristic is that the proliferating tumor cells directly form bone or bone-like tissue, so it is also called osteogenic sarcoma. Patients may feel pain in the early stages, and as the tumor grows, it may also cause joint deformities, further aggravating functional impairment. If not treated in time, the tumor will spread and metastasize rapidly, invading surrounding tissues and organs, and even metastasizing to important organs such as the lungs, seriously threatening the patient's life.

[0003] However, due to the low incidence of bone tumors, the majority of primary care physicians do not have sufficient knowledge of the diagnosis and treatment of the disease, and their film reading ability and level vary, which seriously restricts the primary care physicians' ability to diagnose the disease. At the same time, the judgment and segmentation of bone tumor boundaries are usually done by clinicians manually outlining or annotating on CT images based on their own anatomical knowledge and experience, but this process is time-consuming, labor-intensive, and highly subjective. Therefore, objective and accurate medical image segmentation is crucial for patients' pre-diagnosis, surgical implementation, and prognosis recovery.

[0004] With the rapid development of computer technology, computers can assist people in some image processing and automatic segmentation and positioning of medical images, such as tissue volume quantification, disease diagnosis, regional positioning, anatomical structure research, radiotherapy planning and computer-assisted surgery. However, there are still certain challenges for specific medical image segmentation. Taking bone tumor medical image segmentation as an example, bone tumors have various types in histology, with different shapes and sizes, the osteogenesis position in the lesion is random, and the tumor imaging image is greatly affected by contrast agents. Secondly, there is a lack of model training data sets, and there is currently no public deep learning model training data set. Finally, there is a class imbalance phenomenon in the medical image segmentation task, that is, the target area in the image, such as lesions, organs, etc., accounts for a small proportion of the entire image, resulting in the background area dominating in the training stage, reducing the optimization quality of the model. Based on this, the present invention aims to improve the lack of accuracy of the boundary of the deep neural network model for bone tumor medical image segmentation, and to enable the model to have a certain generalization ability, and to further improve the accuracy of other medical image segmentation. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to achieve accurate segmentation of bone tumor medical images based on the improved hybrid attention mechanism and TransUnet model. While ensuring the improvement of the accuracy of bone tumor segmentation, generalization to other data sets is achieved. Based on this, we proposed the ONC-TransUnet model to achieve this goal. The specific content of the invention is as follows:

[0006] S1, extract bone tumor CT data. Use code to convert the original CT image format from DICOM to JPG and PNG formats acceptable to the model. Select coronal and sagittal CT images with obvious bone tumor features as the data set

[0007] S2: Label the data and establish the Osteosarcoma+ dataset. Use the open source annotation tool LabelMe to label the bone tumor CT image data source.

[0008] S3, preprocess the dataset

[0009] S4 improves the hybrid attention mechanism by combining the channel attention mechanism and the spatial attention mechanism with the original input in a residual manner to obtain an improved attention mechanism module.

[0010] S5, using the TransUnet model as a prototype, adds the improved hybrid attention mechanism module to the key position of the feature extraction stage in the TransUnet model to build a new segmentation network model.

[0011] S6, train the ONC-TransUnet segmentation network model using the Osteosarcoma+ and Synapse datasets;

[0012] S7, input the image to be segmented into the ONC-TransUnet model for image segmentation, obtain the segmented image and evaluate the image segmentation effect.

[0013] Furthermore, in step 2, the specific process is as follows: first, the clinician demonstrated the annotation of bone tumor CT medical images and explained the relevant knowledge of bone tumor tissue morphology; second, learn the relevant medical knowledge of bone tumors and conduct trial annotation, then summarize the difficulties and difficulties in the annotation process, communicate with hospital doctors, and hold meetings to discuss the problems; third, annotate the bone tumor CT medical image data, and the doctor checks and optimizes the annotation results to complete the annotation task of bone tumor CT image data. The annotated bone tumor CT medical image data set is named Osteosarcoma+, with a total of 1,207 images annotated and named with I+ serial numbers (I0001-I01207).

[0014] Furthermore, the specific processing process in step 3 is: from all the image samples in the Osteosarcoma+ dataset and the Synapse dataset, the image samples are adjusted to a resolution of 224*224.

[0015] The image samples selected from the Osteosarcoma+ dataset and the Synapse dataset are divided into training sets and test sets according to the set ratios.

[0016] Perform random data enhancement on the training data: specific enhancement methods include random rotation, random Gaussian blur, random cropping, etc.

[0017] Furthermore, in step 4, the specific process is as follows: In this embodiment, the specific process of step 4 is as follows: the improved hybrid attention mechanism module includes two branches, the first branch is the channel attention mechanism module, and the second branch is the spatial attention mechanism module. The input feature map is extracted by the channel attention module and the spatial attention module respectively, and then the Add operation is performed with the original input feature, so as to obtain a feature map with both the channel information of the channel attention and the spatial information of the spatial attention, and the details of the original map will not be lost. The specific formula is:

[0018] F Out =Add(F Input , F Sout , F Cout )

[0019] Furthermore, in step 5, the specific process is: adding the improved hybrid attention mechanism module to the feature extraction module of TransUnet, and the specific structure is: the input image first passes through the Stem module of the Resnet50 module to roughly extract image features, and then passes through the improved hybrid attention module (newCBAM1). The generated feature map will be used as the input of the first Skip and the maximum pooling module. After that, the feature map passes through a maximum pooling module, and the generated feature map enters the Block1 module of the Resnet50 module and the improved hybrid attention module (newCBAM1) in turn, and the generated feature map will be used as the input of the second Skip and the Block2 module of the Resnet50 module. After that, the feature map enters the Block2 module of the Resnet50 module and the improved hybrid attention module (newCBAM1) in turn, and the generated feature map will be used as the input of the third Skip and the Block3 module of the Resnet50 module. After that, the feature map enters the Block3 module of the Resnet50 module to obtain the output of the Resnet50 module. The output feature map enters the Transformer structure through PatchEmbedding and position encoding.

[0020] Furthermore, in step 6, the specific process is as follows: In order to alleviate the problem that the background proportion of the sample is too large and the training difficulty is increased, the training adopts a mixed loss function to improve the model's detection ability. The loss function formula is:

[0021] Loss = w1*CE+w2*Dice

[0022] Where w1 and w2 are weight factors, CE is the cross entropy loss function, and Dice is the Dice loss function. The training settings are w1=0.5, w2=0.5.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] (1) A bone tumor dataset was established;

[0025] (2) By improving the hybrid attention mechanism, the model’s ability to extract details is enhanced and the boundary segmentation is more refined;

[0026] (3) A hybrid attention mechanism is added to the key positions of the model, so that the model can better capture and understand the contextual relationship in the image, while not losing the extraction of image details and boundaries. After adding the improved hybrid attention mechanism module to the key positions of the model, the average Dice coefficient on the Osteosarcoma+ dataset is 2.68% higher than that of the original TransUnet model, and the average HD_95 is 4.10mm lower than that of the original TransUnet model. On the Synapse dataset, compared with the TransUnet model, the average Dice coefficient of the ONC-TransUnet model is increased by 2.72%, and the average HD_95 is reduced by 4.76mm. Compared with other models, ONC-TransUnet has the highest average DSC of 78.11%. This shows that the ONC-TransUnet model has better segmentation effects on the whole organ and boundaries, and has better generalization on the Synapse dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model in an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of the structure of an improved hybrid attention mechanism module in an embodiment of the present invention;

[0029] Figure 3 It is a schematic diagram of the network model structure in the feature extraction phase of the bone tumor medical image segmentation method based on the improved hybrid attention mechanism and the TransUnet model in an embodiment of the present invention;

[0030] Figure 4Schematic diagram of the overall network model structure of the bone tumor medical image segmentation method based on the improved hybrid attention mechanism and TransUnet model in an embodiment of the present invention;

[0031] Figure 5 This is a comparison diagram of the segmentation results of the ONC-TransUnet model based on the improved hybrid attention mechanism and the TransUnet model in an embodiment of the present invention and the segmentation results of the TransUnet model, wherein (a) is the model input image, (b) on the left is the segmentation result of the TransUnet model, (b) in the middle is the segmentation result of the ONC-TransUnet model, and (b) on the right is the true label. DETAILED DESCRIPTION

[0032] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment. This embodiment provides a technical solution; a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model, comprising the following steps:

[0033] 1. Extract bone tumor CT data. Use code to convert the original CT image format from DICOM to JPG and PNG formats acceptable to the model. Select coronal and sagittal CT images with obvious bone tumor features as the data set;

[0034] 2. Label the data set. Use the open source annotation tool LabelMe to label the bone tumor CT image data source;

[0035] 3. Preprocess the data;

[0036] 4. Improve the hybrid attention mechanism by combining the channel attention mechanism and the spatial attention mechanism with the original input in a residual manner to obtain an improved hybrid attention mechanism module;

[0037] 5. Using the TransUnet model as a prototype, the improved hybrid attention mechanism module is added to the key position of the feature extraction stage in the TransUnet model. A new segmentation network model is constructed and named ONC-TransUnet model;

[0038] 6. Train the ONC-TransUnet segmentation network model using the Osteosarcoma+ and Synapse datasets;

[0039] 7. Input the image to be segmented into the ONC-TransUnet model for image segmentation, obtain the segmented image and evaluate the image segmentation effect.

[0040] In this embodiment, the specific process of step 2 is as follows: first, the clinician demonstrated the annotation of bone tumor CT medical images and explained the relevant knowledge of bone tumor tissue morphology; second, learn the medical knowledge related to bone tumors and conduct trial annotation, then summarize the difficulties and difficulties in the annotation process, communicate with hospital doctors, and hold a meeting to discuss the problem; third, annotate the bone tumor CT medical image data, and the doctor checks and optimizes the annotation results to complete the annotation task of the bone tumor CT image data. The annotated bone tumor CT medical image data set is named Osteosarcoma+, with a total of 1207 images annotated and named with I+ serial numbers (I0001-I01207).

[0041] In this embodiment, the specific process of step 3 is as follows: In order to verify the generalization of the model for medical image segmentation, we select the multi-organ segmentation dataset Synapse to evaluate the results. From all image samples in the Osteosarcoma+ dataset and the Synapse dataset, the image samples are adjusted to a resolution of 224*224.

[0042] The image samples selected from the Osteosarcoma+ dataset and the Synapse dataset were divided into training sets and test sets according to the set ratio. Since the bone tumor CT image data is relatively small, we randomly selected 15 images from the Osteosarcoma+ dataset as the test set and the remaining 1192 images as the training set. Each set is divided into an image set and a corresponding label set.

[0043] Perform random data enhancement on the training data: specific enhancement methods include random rotation, random Gaussian blur, random cropping, etc.

[0044] In this embodiment, the specific process of step 4 is as follows: the improved hybrid attention mechanism module includes two branches, the first branch is the channel attention mechanism module, and the second branch is the spatial attention mechanism module. The input feature map is extracted by the channel attention module and the spatial attention module respectively, and then the Add operation is performed with the original input feature, so as to obtain a feature map with both the channel information of the channel attention and the spatial information of the spatial attention, and the details of the original map will not be lost. The specific formula is:

[0045] F Out =Add(F Input , F Sout , F Cout )

[0046] This process of the present invention can be seen Figure 2 .

[0047] In this embodiment, the specific process of step 5 is: adding the improved hybrid attention mechanism module to the feature extraction module of TransUnet, and the specific structure is: the input image first passes through the Stem module of the Resnet50 module to perform rough extraction of image features, and the resulting feature map size is 112*112 and the number of channels is 64. Then, through the improved hybrid attention module (newCBAM1), the size and number of channels of the generated feature map remain unchanged. The generated feature map will serve as the input of the first Skip and maximum pooling modules. After that, the feature map passes through a maximum pooling module, and the size of the feature map becomes 55*55, and the number of channels remains unchanged. The generated feature map enters the Block1 module of the Resnet50 module, and the size of the feature map is 56*56 and the number of channels is 256. Then, through the improved hybrid attention module (newCBAM1), the size and number of channels of the generated feature map remain unchanged. The generated feature map will serve as the input of the Block2 module of the second Skip and Resnet50 module. After the generated feature map enters the Block2 module of the Resnet50 module, the size of the feature map becomes 28*28, and the number of channels becomes 512. Then, through the improved hybrid attention module (newCBAM1), the generated feature map will be used as the input of the third Skip and the Block3 module of the Resnet50 module. The generated feature map enters the Block3 module of the Resnet50 module to obtain the final output of the Resnet50 module. The size of the feature map obtained is 14*14, and the number of channels is 1024. The output feature map enters the Transformer structure through Patch Embedding and position encoding, and then undergoes the Reshape operation to obtain a feature map with a size of 14*14 and a number of channels of 768. After that, it passes through a convolution layer, a batch normalization layer, and a Relu layer to obtain a feature map with a size of 14*14 and a number of channels of 512. After that, it passes through three upsampling layers and performs Concat operations with the three Skips in the feature extraction module to obtain a feature map with a size of 112*112 and a number of channels of 64. Finally, it passes through an upsampling layer, a convolution layer, a batch normalization layer, and a Relu layer to obtain a feature map with a size of 224*224 and 16 channels. Finally, it passes through a segmentation layer to obtain the final segmentation map. This process of the present invention can be seen Figure 3 , Figure 4 .

[0048] In this embodiment, the specific process of step 6 is: the training process uses the Ubtun operating system, AMD EPYC7542CPU, two NVIDA3090 GPUs, Pytorch version 1.10.2 and Cuda version 11.3, and uses the SGD optimizer; the initial learning rate is set to 0.01; the batch size is set to 24; training 200 Epochs; the VIT model uses R50-Vit_B16; the patch size is set to 16; the dropout rate is set to 0.5; the number of Transformer modules is 12. At the same time, in order to alleviate the problem of excessive background proportion of samples and increased training difficulty, the training adopts a mixed loss function to improve the model's ability to detect and measure. The loss function formula is:

[0049] Loss = w1*CE+w2*Dice

[0050] Where w1 and w2 are weight factors, CE is the cross entropy loss function, and Dice is the Dice loss function. The training settings are w1=0.5, w2=0.5.

[0051] In this embodiment, the specific process of step 7 is: compare the segmented image and various segmentation evaluation indicators obtained by the ONC-TransUnet model with the original TransUnet model, and the evaluation indicators are the evaluation indicators commonly used in the field of medical image segmentation: average dice similarity coefficient (Dice) and average Hausdorff distance _95 (HD_95). The specific segmentation results are shown in Table 1, and the specific segmented images can be seen Figure 5 At the same time, we chose to test the generalization of the ONC-TransUnet model in Synapse and compare it with the segmentation evaluation indicators of mainstream segmentation models. The specific segmentation results are shown in Table 2. For mainstream models, we chose the TransUnet model, Deeplabv3+ model, Unet model, and Unet++ model.

[0052] Table 1

[0053]

[0054] As can be seen from Table 1, compared with the TransUnet model, the average DSC of the ONC-TransUnet model is increased by 2.68%, and the average HD95 is reduced by 4.1 mm. It can be seen that the ONC-TransUnet model has a better segmentation effect on the bone tumor dataset Osteosarcoma+.

[0055] Table 2

[0056]

[0057] As can be seen from Table 2, compared with the TransUnet model, the average DSC of the ONC-TransUnet model increased by 2.72% and the HD95 decreased by 4.76 mm. Compared with other models, the average DSC of ONC-TransUnet is the highest, which is 78.11%. This shows that the ONC-TransUnet model has better segmentation effects on the whole organ and the boundary, and has better generalization on the Synapse dataset.

[0058] In summary, the bone tumor medical image segmentation method based on the improved hybrid attention mechanism and TransUnet model in the above embodiment enhances the detailed features of the edge of the bone tumor image extracted by the model by adding the improved hybrid attention mechanism module at the key position of the feature extraction stage in the original TransUnet model. At the same time, the present invention also has good generalization and can also achieve high-precision segmentation for other medical image data sets.

[0059] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A bone tumor medical image segmentation method based on an improved hybrid attention mechanism and TransUnet model, characterized in that: The following steps are involved: S1. Extract bone tumor CT data; S2, annotate the extracted bone tumor CT data and establish the dataset Osteosarcoma+; S3, preprocessing the Osteosarcoma+ dataset; S4, based on the hybrid attention mechanism, the channel attention mechanism module and the spatial attention mechanism module are combined in a residual residual manner, so that the channel attention mechanism model can extract the details of the image and the boundaries are improved by the hybrid attention mechanism module; S5. Taking the TransUnet structure as the prototype, the improved hybrid attention mechanism module is added to the key position of the feature extraction stage of the TransUnet model to construct a new segmentation network model, named ONC-TransUnet model; S6. Use the Osteosarcoma+ and Synapse datasets to train the ONC-TransUnet segmentation network model. S7. Input the image to be segmented into the ONC-TransUnet model for image segmentation, obtain the segmented image and evaluate the image segmentation effect.

2. According to claim 1, a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model is characterized in that: The specific processing process in S1 is: convert the original CT image format from DICOM to JPG and PNG formats acceptable to the model through code; select coronal and sagittal CT images with obvious bone tumor characteristics as the data set.

3. According to claim 1, a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model is characterized in that: The specific processing process of S2 is as follows: label the dataset; use the open source annotation tool LabelMe to label the bone tumor CT image data source; the labeled bone tumor CT medical imaging dataset is named Osteosarcoma+, with a total of 1,207 images labeled and named with I+ serial numbers (I0001-I01207).

4. According to claim 1, a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model is characterized in that: The specific processing process of step 3 is as follows: from all image samples in the Osteosarcoma+ dataset and the Synapse dataset, adjust the image samples to a resolution of 224*224; The image samples selected from the Osteosarcoma+ dataset and the Synapse dataset are divided into training sets and test sets according to the set ratios; Perform random data enhancement on the training data: the enhancement methods are random rotation, random Gaussian blur, and random cropping.

5. According to claim 1, a bone tumor medical image segmentation method based on an improved hybrid attention mechanism and a TransUnet model is characterized in that: The specific processing process of S4 is as follows: the improved hybrid attention mechanism module contains two branches, the first branch is the channel attention mechanism module, and the second branch is the spatial attention mechanism module. The input feature map is extracted by the channel attention module and the spatial attention module respectively, and then the Add operation is performed with the original input feature to obtain a feature map with both the channel information of channel attention and the spatial information of spatial attention without losing the details of the original map.

6. The bone tumor medical image segmentation method based on the improved hybrid attention mechanism and TransUnet model according to claim 1, characterized in that: The specific processing process of S5: The improved hybrid attention mechanism module is added to the feature extraction module of TransUnet. The specific structure is: the input image first passes through the Stem module of the Resnet50 module to roughly extract image features, and then passes through the improved hybrid attention module newCBAM1; the generated feature map will be used as the input of the first Skip and maximum pooling modules; after that, the feature map passes through a maximum pooling module, and the generated feature map enters the Block1 module of the Resnet50 module and the improved hybrid attention module newCBAM1 in turn, and the generated feature map will be used as the input of the second Skip and the Block2 module of the Resnet50 module; after that, the feature map enters the Block2 module of the Resnet50 module and the improved hybrid attention module newCBAM1 in turn, and the generated feature map will be used as the input of the third Skip and the Block3 module of the Resnet50 module; after that, the feature map enters the Block3 module of the Resnet50 module to obtain the output of the Resnet50 module; the output feature map enters the Transformer structure through Patch Embedding and position encoding.

7. The bone tumor medical image segmentation method based on the improved hybrid attention mechanism and TransUnet model according to claim 1, characterized in that: The specific processing process of step 6 is: use the mixed loss function for training, and the formula of the mixed loss function is: Loss = w1*CE+w2*Dice Among them, w1 and w2 are weight factors, CE is the cross entropy loss function, and Dice is the Dice loss function; the training settings are w1=0.5, w2=0.5.

Images