Multi-modal cervical cancer anterior lesion region segmentation method based on Swin Transform

Abstract

The invention discloses a multi-mode cervical cancer anterior lesion region segmentation method based on a Swin Transform. The invention provides an MSS-Net neural network, and aims to overcome the defects of an existing method and fully utilize the feature extraction capability of Swin Transformer to realize multi-modal fusion of an acetic acid experiment image and an iodine experiment image. The method comprises the following steps: firstly, collecting and marking an HSIL + region in a colposcope image; then, extracting a cervix uteri area by using a target examination algorithm, and eliminating noise of a non-cervix uteri area; then, respectively inputting the pre-processed acetic acid solution image and the pre-processed iodine solution image into a Swin Transform feature extraction network to obtain a multi-level feature map; after the feature maps are subjected to weighted fusion through the multimode attention module, the feature maps are input into the decoder module, and a final segmentation result is generated through cascade up-sampling. And finally, training a neural network MSS-Net by using a Dice coefficient and a cross entropy loss function to obtain a trained model. According to the invention, through multi-modal data fusion and feature extraction capability enhancement, accurate segmentation of the cervical cancer precancerous lesion area is realized.

Description

Technical Field

[0001] The present invention belongs to the field of medical images and provides a multimodal cervical precancerous lesion region segmentation method based on Swin Transformer. Specifically, deep learning technology is used to segment the cervical precancerous lesion region of the cervical image taken under colposcopy. Background Art

[0002] Cervical cancer is one of the most common malignant tumors in the female reproductive system. Its early diagnosis and treatment have a decisive impact on the patient's prognosis. Cervical precancerous lesions refer to the abnormal changes in cervical epithelial cells in the early stages of cancer. Early detection and accurate segmentation of this stage are of great significance for preventing the occurrence of cervical cancer.

[0003] In clinical practice, the three-level screening model for cervical cancer is the basic principle guiding the screening, diagnosis and management of cervical cancer. First, preliminary screening is performed through cytology testing (TCT) and HPV testing. Then, a colposcope is used to locate suspicious areas for targeted biopsy, thereby improving the accuracy of the diagnosis of cervical lesions, and finally confirmed by pathological examination. This examination usually involves using biopsy forceps to perform a biopsy at the squamous-columnar junction of the cervix under the guidance of a colposcope. Pathological diagnosis is the gold standard for confirming cervical cancer. In this process, colposcopy is a key step in diagnosing cervical lesions. Doctors use 5% acetic acid solution and Lugol's iodine solution to observe images such as cervical epithelial and vascular morphology, glandular openings, etc. under a low-power microscope to identify the obvious characteristics of cervical cancer and its precursor lesions and guide biopsy.

[0004] In the field of medical image analysis, traditional lesion area segmentation methods usually rely on manual annotation by experts, which is not only time-consuming and labor-intensive, but also highly subjective, and the annotation results have large individual differences.

[0005] At present, there are some deep learning methods to assist doctors in segmenting precancerous lesions. At present, most of these methods are aimed at images after the acetic acid experiment. However, in clinical practice, doctors also need to combine the results of iodine solution images to determine the final lesion area. This shows that there is a correlation between the two types of images, and doctors also need to label the lesion areas of both types of images.

[0006] Different from other recognition methods, the present invention proposes a multimodal cervical precancerous lesion area segmentation method based on Swin Transformer based on clinical practice. This method uses the newer Swin Transformer model in recent years to extract image features, and implements a feature fusion module based on multimodal attention to perform multimodal feature fusion on two types of images, acetic acid experimental images and iodine experimental images. Finally, a cascaded upsampling decoder is used to achieve a more accurate segmentation of the precancerous lesion area. Summary of the invention

[0007] The purpose of the present invention is to address the shortcomings of existing methods. In order to make full use of the excellent feature extraction capability of Swin Transformer and realize the multimodal fusion of acetic acid experimental images and iodine experimental images, the present invention proposes a multimodal cervical precancerous lesion region segmentation method based on Swin Transformer.

[0008] The neural network MSS-Net proposed in the present invention mainly includes the following steps:

[0009] Step 1: Collect colposcopic images, and annotate the HSIL+ region of the colposcopic images collected by the acetic acid test and the colposcopic images collected by the iodine solution test of each patient;

[0010] Step 2: pre-process the colposcopy image, extract the cervical region in the colposcopy image through the trained target detection algorithm model, and remove the noise interference in the non-cervical region;

[0011] Step 3: Use Swin Transformer as the feature extraction network, input the acetic acid test colposcopy image and iodine solution colposcopy image obtained by step 2 into two feature extraction networks respectively, and obtain the acetic acid image feature map And iodine solution image feature map

[0012] Step 4: Acetic acid image feature map And iodine solution image feature map Corresponding additions are made to obtain the added feature map C i ; Add feature map C i Input into multi-mode attention to obtain the corresponding weight features and and Respectively and corresponding and Weighted multi-mode attention feature map and

[0013] Step 5: Multi-mode attention feature map and Input to the decoder module, i∈[1,4], which obtains the decoded feature map U by cascading upsampling a and U b , the segmentation head finally outputs the segmentation results of the HSIL+ area of ​​the acetic acid solution image and the HSIL+ area of ​​the iodine solution image;

[0014] Step 6. Input the colposcopic image annotated in step 1 into the neural network MSS-Net constructed in steps 2, 3, 4 and 5 for training. The loss function is set as the weighted average loss function of the cross entropy loss function and the Dice loss function, and finally a trained model is obtained. Input the colposcopic image to be detected into the trained neural network MSS-Net to obtain the HSIL+ region segmentation result.

[0015] The beneficial effects of the present invention are as follows:

[0016] Based on clinical practice, the present invention proposes a multimodal cervical precancerous lesion area segmentation method based on Swin Transformer. The method uses the relatively new Swin Transformer model in recent years to extract image features, and implements a feature fusion module based on multimodal attention. Multimodal feature fusion is performed on two types of images, acetic acid experimental images and iodine experimental images. Finally, a cascaded upsampling decoder is used to achieve a more accurate segmentation of the precancerous lesion area. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the structure diagram of the neural network MSS-Net proposed in the present invention;

[0018] Figure 2 It is a Loss curve diagram of an embodiment of the neural network MSS-Net proposed by the present invention;

[0019] Figure 3 A Dice curve diagram of an embodiment of the neural network MSS-Net proposed in the present invention;

[0020] Figure 4 It is a mIoU curve diagram of an embodiment of the neural network MSS-Net proposed in the present invention; DETAILED DESCRIPTION

[0021] The method of the present invention is further described below in conjunction with the accompanying drawings:

[0022] Step 1: Collect colposcopic images, and annotate the HSIL+ region of the colposcopic images collected by the acetic acid test and the colposcopic images collected by the iodine solution test of each patient;

[0023] Step 2: pre-process the colposcopy image, extract the cervical region in the colposcopy image through the trained target detection algorithm model, and remove the noise interference in the non-cervical region;

[0024] Step 3: Use Swin Transformer as the feature extraction network, input the acetic acid test colposcopy image and iodine solution colposcopy image obtained by step 2 into two feature extraction networks respectively, and obtain the acetic acid image feature map And iodine solution image feature map

[0025] Step 4: Acetic acid image feature map And iodine solution image feature map Corresponding additions are obtained to obtain the added feature map C i ; Add feature map C i Input into multi-mode attention to obtain the corresponding weight features and and Respectively and corresponding and Weighted multi-modal attention feature map and

[0026] Step 5: Multi-mode attention feature map and Input to the decoder module, i∈[1,4], which obtains the decoded feature map U by cascading upsampling a and U b , the segmentation head finally outputs the segmentation results of the HSIL+ area of ​​the acetic acid solution image and the HSIL+ area of ​​the iodine solution image;

[0027] Step 6. Input the colposcopic image annotated in step 1 into the neural network MSS-Net constructed in steps 2, 3, 4 and 5 for training. The loss function is set as the weighted average loss function of the cross entropy loss function and the Dice loss function, and finally a trained model is obtained. Input the colposcopic image to be detected into the trained neural network MSS-Net to obtain the HSIL+ region segmentation result.

[0028] Furthermore, the specific operation of the marking operation described in step 1 is as follows:

[0029] Professional physicians use annotation tools to annotate the cervical precancerous lesions (HSIL+) areas in acetic acid solution colposcopic images and iodine solution colposcopic images, and the HSIL+ areas include high-grade squamous intraepithelial lesions (HSIL) and cervical cancer areas; acetic acid solution images and iodine solution images as well as corresponding acetic acid solution image labels and iodine solution image labels are obtained from the colposcopy examination of each patient as training sets.

[0030] The target detection algorithm model training set described in step 2 is produced as follows:

[0031] The colposcopic image data collected in step one is annotated again, and a professional physician uses a rectangular frame in the annotation tool to frame the cervical area centered on the cervical opening and outputs it as a label; the original colposcopic images and labels in step one are sorted out as a training set for the target detection algorithm model.

[0032] The target detection algorithm described in step 2 refers to a deep learning target detection algorithm, preferably Fast R-CNN and YOLO.

[0033] The feature extraction network described in step 3:

[0034] Define the acetic acid test colposcopy image input to the Swin Transform network as X, and the iodine solution colposcopy image as Y; the Swin Transform network is S(·); then the acetic acid image feature map And iodine solution image feature map As shown below:

[0035]

[0036] Among them, i∈[1,4] is determined by the number of BasicLayer of Swin Transformer. In this patent, the number of BasicLayer is four, so four layers of feature maps will be output.

[0037] The detailed process of obtaining the multi-mode attention feature map of the corresponding image described in step 4 is:

[0038] The acetic acid image feature map And iodine solution image feature map Add together to get the added feature map C i :

[0039]

[0040] Add the feature map C i Input into the multi-mode attention, define the multi-mode attention symbol as MSAtten(·); then the weight feature and

[0041]

[0042] Weight features and The corresponding acetic acid image feature maps And iodine solution image feature map Weighted to get the multi-modal attention feature map and

[0043]

[0044] Further, the detailed structure of the multi-modal attention described in step 4 is:

[0045] Multimodal attention consists of a global information aggregation module and two feature weight generation modules;

[0046] The implementation process of the global information aggregation module is as follows:

[0047] First, the input feature map (addition feature map C i ) to obtain a global description of each channel; let I be the input feature map, with a shape of (H, W, C), where H is the height, W is the width, and C is the number of channels. Global average pooling is expressed as:

[0048]

[0049] Among them, Z c is the global average value of the c-th channel, c∈[1,C];

[0050] The implementation process of the feature weight generation module is as follows:

[0051] First, the aggregated global description is passed through a fully connected (FC) layer with a nonlinear activation function to generate the weight of each channel:

[0052] s=σ(W t1 z)

[0053] Among them, W t1 is the weight matrix for dimensionality reduction, σ is the nonlinear activation function (usually ReLU), z is the vector containing the global average of all channels, and s is the feature map output by the first fully connected layer;

[0054] Then, another fully connected layer is used to increase the dimension:

[0055] W=σ(W t2 s)

[0056] Among them, W t2 is the dimension-increased weight matrix, σ is the nonlinear activation function, s is the vector containing the global average value of all channels, and W is the output result of the feature weight generation module, that is, the weight feature.

[0057] The overall process of channel attention is that the input feature map I is first input into the global information aggregation module to obtain the global description Z, and Z is respectively input into two feature weight generation modules to obtain weight features W1 and W2.

[0058] Furthermore, the decoder module described in step 5:

[0059] Assumption F d is the upsampled feature map in the decoder, F i is the input multi-modal attention feature map and The specific steps are as follows:

[0060] Upsampling:

[0061] F d =Upsample(F i )

[0062] cascade:

[0063] F concat =Concat(F d ,F i )

[0064] The multi-modal attention feature map obtained in step 4 and Input to the decoder module through four cascade upsampling steps to obtain the decoded feature map U a and U b ; The segmentation head consists of a CNN layer with a convolution kernel of 3*3, and the input channel is a multi-mode attention feature or The corresponding number of channels is c, and the output channel is the number of classifications.

[0065] Example:

[0066] This study collected a dataset consisting of cervical images of 266 patients. All images were taken at the Cervical Clinic of the Sir Run Run Hospital Affiliated to Zhejiang University School of Medicine between November 2020 and July 2021. To ensure the quality and applicability of the images, we screened out all cases that were blurred, obscured by leucorrhea, or had obvious bleeding when collecting data. Among them, the iodine solution images and acetic acid solution images are 1:1. There are 134 groups of HSIL+ cases in the dataset, and the other 132 groups are LSIL- cases. According to steps one and two, a training set for training MSS-Net and target detection algorithms was made. In this embodiment, the target detection algorithm selected is the YOLO algorithm. When all images are input into the MSS-Net network, they are resized to 224*224 to adapt to the input size of the network. This study was approved by the Ethics Committee of the Sir Run Run Hospital, Zhejiang University.

[0067] This network uses a pre-trained model, and loads the pre-trained weights Swin-T of the Swin Transformer model into the feature extraction network of MSS-Net. The training environment of the embodiment is: operating system Windows11 22H2 version, CPU is AMDRyzen 5 5600X 6-Core Processor, memory size is 24GB, graphics card is Nvidia GeForce RTX 2080TiMSS-Net is implemented by the Pytorch coding framework. The initial learning rate is set to 0.0001, the optimizer uses adma, momentum is set to 0.9, and batchsize is set to 12. The loss value is obtained by weighted average of the cross entropy loss function and the Dice loss function, the weight ratio is 1:1, and the Epoch is set to 100.

[0068] The Dice coefficient and mIoU are selected as evaluation indicators.

[0069]

[0070] Figure 2 , Figure 3 and Figure 4 They are respectively the Loss, Dice and mIoU curves of this embodiment.

[0071] Table 1 compares the segmentation effects of this embodiment and other segmentation networks under the same experimental conditions. From the experimental data, the present invention is expected to assist physicians in performing regional segmentation of cervical precancerous lesions to further guide biopsy.

[0072] Table 1 Comparison of model effects

[0073] Dice(%) mIoU(%) UNet 80.99 68.33 DeepLabV3+ 81.01 70.98 MSS-Net 82.25 72.01

[0074] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A multimodal cervical precancerous lesion region segmentation method based on Swin Transformer, characterized in that: Here are the steps: Step 1: Collect colposcopic images, and annotate the HSIL+ region of the colposcopic images collected by the acetic acid test and the colposcopic images collected by the iodine solution test of each patient; Step 2: pre-process the colposcopy image, extract the cervical region in the colposcopy image through the trained target detection algorithm model, and remove the noise interference in the non-cervical region; Step 3: Use Swin Transformer as the feature extraction network, input the acetic acid test colposcopy image and iodine solution colposcopy image obtained by step 2 into two feature extraction networks respectively, and obtain the acetic acid image feature map And iodine solution image feature map Step 4: Acetic acid image feature map And iodine solution image feature map Corresponding additions are obtained to obtain the added feature map C i ; Add feature map C i Input into multi-mode attention to obtain the corresponding weight features and and Respectively and corresponding and Weighted multi-modal attention feature map and Step 5: Multi-mode attention feature map and Input to the decoder module, i∈[1,4], which obtains the decoded feature map U by cascading upsampling a and U b , the segmentation head finally outputs the segmentation results of the HSIL+ area of ​​the acetic acid solution image and the HSIL+ area of ​​the iodine solution image; Step 6. Input the colposcopic image annotated in step 1 into the neural network MSS-Net constructed in steps 2, 3, 4 and 5 for training. The loss function is set as the weighted average loss function of the cross entropy loss function and the Dice loss function, and finally a trained model is obtained. Input the colposcopic image to be detected into the trained neural network MSS-Net to obtain the HSIL+ region segmentation result.

2. According to the multimodal cervical precancerous lesion region segmentation method based on Swin Transformer in claim 1, it is characterized in that: The specific operations of the annotation operation described in step 1 are as follows: Professional physicians use annotation tools to annotate the HSIL+ area of ​​cervical precancerous lesions in acetic acid solution colposcopic images and iodine solution colposcopic images, and the HSIL+ area includes high-grade squamous intraepithelial lesions HSIL and cervical cancer areas; acetic acid solution images and iodine solution images as well as corresponding acetic acid solution image labels and iodine solution image labels are obtained from the colposcopy examination of each patient as training sets.

3. According to the multimodal cervical precancerous lesion region segmentation method based on Swin Transformer in claim 2, it is characterized in that: Step 2: Preparation of target inspection algorithm training set and selection of target detection algorithm are as follows: Based on the colposcopy image data collected in step 1, the data is annotated again, and a professional physician uses a rectangular frame in the annotation tool to frame the cervical area centered on the cervical os and output it as a label; the original colposcopy images and labels in step 1 are sorted out as a training set for the target detection algorithm model; The object detection algorithms described here refer to deep learning object detection algorithms, including Fast R-CNN and YOLO.

4. According to the multimodal cervical precancerous lesion region segmentation method based on Swin Transformer in claim 3, it is characterized in that: The feature extraction network described in step 3: Define the acetic acid test colposcopy image input to the Swin Transform network as X, and the iodine solution colposcopy image as Y; the Swin Transform network is S(·); then the acetic acid image feature map And iodine solution image feature map As shown below: Among them, i∈[1,4] is determined by the number of BasicLayer of Swin Transformer.

5. According to the multimodal cervical precancerous lesion region segmentation method based on Swin Transformer in claim 4, it is characterized in that: The detailed process of step 4 is described below: The acetic acid image feature map And iodine solution image feature map Add together to get the added feature map C i : Add the feature map C i Input into the multi-mode attention, define the multi-mode attention symbol as MSAtten(·); then the weight feature and Weight features and The corresponding acetic acid image feature maps And iodine solution image feature map Weighted to obtain a multi-modal attention feature map and 6. The multimodal cervical precancerous lesion region segmentation method based on Swin Transformer according to claim 5, characterized in that: The detailed structure of multi-modal attention described: Multimodal attention consists of a global information aggregation module and two feature weight generation modules; The implementation process of the global information aggregation module is as follows: First, the information of each channel of the input feature map is aggregated through global average pooling to obtain a global description of each channel; let I be the input feature map, with a shape of (H, W, C), where H is the height, W is the width, and C is the number of channels; global average pooling is expressed as: Among them, Z c is the global average value of the c-th channel, c∈[1,C]; The implementation process of the feature weight generation module is as follows: First, the aggregated global description is passed through a fully connected layer with a nonlinear activation function to generate the weights of each channel: s=σ(W t1 With) Among them, W t1 is the weight matrix for dimensionality reduction, σ is the nonlinear activation function, z is the vector containing the global average of all channels, and s is the feature map output by the first fully connected layer; Then, another fully connected layer is used to increase the dimension: W=σ(W t2 s) Among them, W t2 is the dimension-increased weight matrix, σ is the nonlinear activation function, s is the vector containing the global average values ​​of all channels, and W is the output result of the feature weight generation module, i.e., the weight feature; The overall process of channel attention is that the input feature map I is first input into the global information aggregation module to obtain the global description Z, and Z is respectively input into two feature weight generation modules to obtain weight features W1 and W2.

7. The multimodal cervical precancerous lesion region segmentation method based on Swin Transformer according to claim 6, characterized in that: The decoder module described: Assumption F d is the upsampled feature map in the decoder, F i is the input multi-modal attention feature map and The specific steps are as follows: Upsampling: F d =Upsample(F i ) cascade: F concat =Concat(F d ,F i ) The multi-modal attention feature map obtained in step 4 and Input to the decoder module through four cascade upsampling steps to obtain the decoded feature map U a and U b ; The segmentation head consists of a CNN layer with a convolution kernel of 3*3, and the input channel is a multi-mode attention feature. or The corresponding number of channels is c, and the output channel is the number of classifications.

Images