A multimodal cervical precancerous lesion segmentation method based on Swin Transformer

Abstract

This invention discloses a multimodal cervical precancerous lesion region segmentation method based on the Swin Transformer. The invention proposes the MSS-Net neural network to overcome the shortcomings of existing methods, fully utilizing the feature extraction capabilities of the Swin Transformer to achieve multimodal fusion of acetic acid and iodine test images. First, HSIL+ regions in colposcopy images are collected and labeled. Then, a target inspection algorithm is used to extract the cervical region, removing noise from non-cervical regions. Next, the preprocessed acetic acid and iodine solution images are input into the Swin Transformer feature extraction network to obtain multi-level feature maps. After weighted fusion of the feature maps through a multimodal attention module, the data is input into a decoder module, where cascaded upsampling generates the final segmentation result. Finally, the MSS-Net neural network is trained using the Dice coefficient and cross-entropy loss function to obtain the trained model. This invention achieves relatively accurate segmentation of cervical precancerous lesion regions through multimodal data fusion and enhanced feature extraction capabilities.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging and provides a multimodal cervical precancerous lesion region segmentation method based on Swing Transformer. Specifically, it uses deep learning technology to segment cervical precancerous lesion regions in cervical images captured under colposcopy. Background Technology

[0002] Cervical cancer is one of the most common malignant tumors of the female reproductive system, and its early diagnosis and treatment have a decisive impact on the patient's prognosis. Precancerous lesions of the cervix refer to the abnormal changes in cervical epithelial cells during the early stages of cancerous transformation. Early detection and accurate segmentation of this stage are of great significance for preventing the occurrence of cervical cancer.

[0003] In clinical practice, the three-tiered cervical cancer screening model is the fundamental principle guiding the screening, diagnosis, and management of cervical cancer. First, preliminary screening is conducted through TCT (ThinPrep Cytology Test) and HPV testing. Then, colposcopy is used to locate suspicious areas for targeted biopsies, thereby improving the accuracy of cervical lesion diagnosis. Finally, pathological examination confirms the diagnosis. This examination typically involves using biopsy forceps under colposcopic guidance to perform a biopsy at the squamocolumnar junction of the cervix. Pathological diagnosis is the gold standard for confirming cervical cancer. In this process, colposcopy is a crucial step in diagnosing cervical lesions. Doctors use 5% acetic acid solution and Lugol's iodine solution to observe the cervical epithelium, blood vessel morphology, glandular openings, and other images under low magnification to identify the distinctive features of cervical cancer and its precancerous lesions, guiding the biopsy.

[0004] In the field of medical image analysis, traditional lesion region segmentation methods usually rely on manual annotation by experts, which is not only time-consuming and labor-intensive, but also highly subjective, with significant individual differences in the annotation results.

[0005] Currently, some deep learning methods exist to assist physicians in segmenting precancerous lesion areas. Most of these methods are currently designed for images obtained after acetic acid testing; however, in clinical practice, physicians also need to combine iodine solution images to determine the final lesion area. This indicates a correlation between the two types of images, and physicians require lesion area annotation in both types of images.

[0006] Unlike other identification methods, this invention proposes a multimodal cervical precancerous lesion region segmentation method based on Swin Transformer, which is based on clinical practice. This method uses the relatively new Swin Transformer model to extract image features and implements a feature fusion module based on multimodal attention to perform multimodal feature fusion on two types of images: acetic acid test images and iodine test images. Finally, through a cascaded upsampling decoder, a more accurate segmentation of the precancerous lesion region is achieved. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing methods. In order to fully utilize the excellent feature extraction capabilities of Swin Transformer and achieve multimodal fusion of acetic acid test images and iodine test images, this invention proposes a multimodal cervical precancerous lesion region segmentation method based on Swin Transformer.

[0008] The neural network MSS-Net proposed in this invention mainly includes the following steps:

[0009] Step 1: Collect colposcopy images and mark the HSIL+ region in the colposcopy images acquired by the acetic acid test and the iodine solution test for each patient.

[0010] Step 2: Preprocess the colposcopy image by extracting the cervical region from the colposcopy image using a trained target detection algorithm model and removing noise interference from non-cervical regions.

[0011] Step 3: Using the Swin Transformer as the feature extraction network, the acetic acid test colposcopy image and the iodine solution colposcopy image obtained from the preprocessing in Step 2 are input into two feature extraction networks respectively to obtain the acetic acid image feature maps. Image feature map of iodine solution

[0012] Step 4: Acetic acid image feature map Image feature map of iodine solution The corresponding addition yields the addition feature map C. i Addition Feature Map C i The input is fed into a multimodal attention module to obtain the corresponding weighted features. and and Each with the corresponding and Weighted summation is performed to obtain multimodal attention feature maps. and

[0013] Step 5: Convert the multimodal attention feature map and The input is fed into the decoder module, i∈[1,4], which obtains the decoded feature map U through cascaded upsampling. a and U b The segmentation head ultimately outputs the segmentation results of the HSIL+ regions in the acetic acid solution image and the HSIL+ regions in the iodine solution image;

[0014] Step 6: Input the colposcopy image labeled in Step 1 into the neural network MSS-Net constructed in Steps 2, 3, 4, and 5 for training. Set the loss function as a weighted average loss function of cross-entropy loss function and Dice loss function to obtain the trained model. Input the colposcopy image to be detected into the trained neural network MSS-Net to obtain the HSIL+ region segmentation result.

[0015] The beneficial effects of this invention are as follows:

[0016] Based on clinical practice, this invention proposes a multimodal cervical precancerous lesion region segmentation method based on Swin Transformer. This method uses the relatively new Swin Transformer model to extract image features and implements a feature fusion module based on multimodal attention to perform multimodal feature fusion on two types of images: acetic acid test images and iodine test images. Finally, through a cascaded upsampling decoder, a relatively accurate segmentation of the precancerous lesion region is achieved. Attached Figure Description

[0017] Figure 1 This is a diagram of the MSS-Net neural network structure proposed in this invention;

[0018] Figure 2 The loss curve is shown for an embodiment of the neural network MSS-Net proposed in this invention.

[0019] Figure 3 The Dice curve is shown for an embodiment of the neural network MSS-Net proposed in this invention.

[0020] Figure 4 This is a graph showing the mIoU curve of an embodiment of the neural network MSS-Net proposed in this invention; Detailed Implementation

[0021] The method of the present invention will be further described below with reference to the accompanying drawings:

[0022] Step 1: Collect colposcopy images and mark the HSIL+ region in the colposcopy images acquired by the acetic acid test and the iodine solution test for each patient.

[0023] Step 2: Preprocess the colposcopy image by extracting the cervical region from the colposcopy image using a trained target detection algorithm model and removing noise interference from non-cervical regions.

[0024] Step 3: Using the Swin Transformer as the feature extraction network, the acetic acid test colposcopy image and the iodine solution colposcopy image obtained from the preprocessing in Step 2 are input into two feature extraction networks respectively to obtain the acetic acid image feature maps. Image feature map of iodine solution

[0025] Step 4: Acetic acid image feature map Image feature map of iodine solution The corresponding addition yields the addition feature map C. i Addition Feature Map C i The input is fed into a multimodal attention module to obtain the corresponding weighted features. and and Each with the corresponding and Weighted summation is performed to obtain multimodal attention feature maps. and

[0026] Step 5: Convert the multimodal attention feature map and The input is fed into the decoder module, i∈[1,4], which obtains the decoded feature map U through cascaded upsampling. a and U b The segmentation head ultimately outputs the segmentation results of the HSIL+ regions in the acetic acid solution image and the HSIL+ regions in the iodine solution image;

[0027] Step 6: Input the colposcopy image labeled in Step 1 into the neural network MSS-Net constructed in Steps 2, 3, 4, and 5 for training. Set the loss function as a weighted average loss function of cross-entropy loss function and Dice loss function to obtain the trained model. Input the colposcopy image to be detected into the trained neural network MSS-Net to obtain the HSIL+ region segmentation result.

[0028] Furthermore, the annotation operation described in step one is performed as follows:

[0029] Professional physicians used annotation tools to label the cervical precancerous lesion (HSIL+) areas in the colposcopy images with acetic acid solution and iodine solution. The HSIL+ areas include high-grade squamous intraepithelial lesion (HSIL) and cervical cancer areas. Acetic acid solution images and iodine solution images, along with corresponding acetic acid solution image labels and iodine solution image labels, were obtained from the colposcopy examination of each patient as a training set.

[0030] The training set for the object detection algorithm model described in step two is prepared as follows:

[0031] The colposcopy image data collected in step one is labeled again. A professional physician uses a rectangle tool to outline the cervical region centered on the cervix and outputs labels. The original colposcopy images and labels from step one are then compiled as the training set for the target detection algorithm model.

[0032] The object detection algorithm described in step two refers to a deep learning object detection algorithm, with Fast R-CNN and YOLO being preferred.

[0033] The feature extraction network described in step three:

[0034] Define the colposcopy image of the acetic acid test input to the Swing Transform network as X, and the colposcopy image of the iodine solution as Y; the Swing Transform network as S(·); then the acetic acid image feature map Image feature map of iodine solution The following is an expression:

[0035]

[0036] Where i∈[1,4] is determined by the number of BasicLayer in Swin Transformer. In this patent, the number of BasicLayer is four, so four feature maps will be output.

[0037] Step four describes the detailed process of obtaining the multimodal attention feature map of the corresponding image:

[0038] Acetic acid image feature map Image feature map of iodine solution Adding them together yields the additive feature map C. i :

[0039]

[0040] Add feature map C i The input is fed into a multimodal attention function, and the multimodal attention symbol is defined as MSAtten(·); then the weight features... and

[0041]

[0042] Weight features and Corresponding to the acetic acid image feature maps Image feature map of iodine solution Weighting is performed to obtain multimodal attention feature maps. and

[0043]

[0044] Furthermore, the detailed structure of the multimodal attention described in step four:

[0045] Multimodal attention consists of a global information aggregation module and two feature weight generation modules;

[0046] The implementation process of the global information aggregation module is as follows:

[0047] First, the input feature maps (adding feature maps C) are processed by global average pooling. i The information from each channel of the input feature map is aggregated to obtain a global description of each channel. Let I be the input feature map with shape (H, W, C), where H is the height, W is the width, and C is the number of channels. Global average pooling is represented as:

[0048]

[0049] Among them, Z c Let $c$ be the global average value of the $c$-th channel, where $c \in [1, C]$.

[0050] The feature weight generation module implementation process is as follows:

[0051] First, the aggregated global description is used to generate the weights for each channel through a fully connected (FC) layer containing a non-linear activation function:

[0052] s=σ(W t1 z)

[0053] Among them, W t1 σ is the weight matrix for dimensionality reduction, σ is the non-linear activation function (usually ReLU), z is a vector containing the global average of all channels, and s is the feature map output by the first fully connected layer.

[0054] Then, use another fully connected layer to increase the dimensionality:

[0055] W = σ(W t2 s)

[0056] Among them, W t2 σ is the weight matrix for increasing dimensionality, s is the non-linear activation function, s is the vector containing the global average value of all channels, and W is the output of the feature weight generation module, i.e., the weight feature.

[0057] The overall process of channel attention is as follows: the input feature map I is first input into the global information aggregation module to obtain the global description Z, and Z is then input into two feature weight generation modules to obtain the weight features W1 and W2.

[0058] Furthermore, the decoder module described in step five:

[0059] Assume F d It is the upsampled feature map in the decoder, F i It is the input multimodal attention feature map and The specific steps are as follows:

[0060] Upsampling:

[0061] F d =Upsample(F i )

[0062] cascade:

[0063] F concat =Concat(F d ,F i )

[0064] The multimodal attention feature map obtained in step four and The input to the decoder module undergoes four cascaded upsampling steps to obtain the decoded feature map U. a and U b The segmentation head consists of CNN layers with 3x3 kernels, and the input channels are multimodal attention features. or The corresponding number of channels, c, is the number of categories in the output channel.

[0065] Example:

[0066] This study compiled a dataset of cervical images from 266 patients. All images were taken at the Cervical Surgery Clinic of Sir Run Run Shaw Hospital, affiliated with Zhejiang University School of Medicine, between November 2020 and July 2021. To ensure image quality and applicability, all cases with blurred images, leukorrhea obscuring the images, or significant bleeding were excluded during data collection. Iodine solution images and acetic acid solution images were compared at a 1:1 ratio. The dataset included 134 cases of HSIL+ and 132 cases of LSIL-. Training sets for training MSS-Net and the object detection algorithm were created according to steps one and two. In this embodiment, the YOLO algorithm was selected for object detection. All images were resized to 224*224 before being input into the MSS-Net network to fit the network's input size. This study was approved by the Ethics Committee of Sir Run Run Shaw Hospital, Zhejiang University.

[0067] This network uses a pre-trained model, loading the pre-trained weights (Swin-T) of the Swin Transformer model into the feature extraction network of MSS-Net. The training environment for this example is: Windows 11 version 22H2 operating system, AMD Ryzen 5 5600X 6-Core Processor CPU, 24GB of memory, and an Nvidia GeForce RTX 2080Ti graphics card. MSS-Net is implemented using the PyTorch encoding framework. The initial learning rate is set to 0.0001, the optimizer uses ADMA, momentum is set to 0.9, and the batch size is set to 12. The loss value is obtained by a weighted average of the cross-entropy loss function and the Dice loss function, with a weight ratio of 1:1, and the epoch is set to 100.

[0068] Dice coefficient and mIoU were selected as evaluation indicators.

[0069]

[0070] Figure 2 , Figure 3 and Figure 4 The figures shown are the Loss, Dice, and mIoU curves for this embodiment.

[0071] Table 1 compares the segmentation performance of this embodiment with other segmentation networks under the same experimental conditions. Based on the experimental data, this invention shows promise in assisting physicians in segmenting precancerous cervical lesions to further guide biopsies.

[0072] Table 1 Comparison of Model Performance

[0073] Dice (%) mIoU (%) UNet 80.99 68.33 DeepLabV3+ 81.01 70.98 MSS-Net 82.25 72.01

[0074] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A multimodal cervical precancerous lesion region segmentation method based on Swing Transformer, characterized in that, The steps are as follows: Step 1: Collect colposcopy images and mark the HSIL+ region in the colposcopy images acquired by the acetic acid test and the iodine solution test for each patient. Step 2: Preprocess the colposcopy image by extracting the cervical region from the colposcopy image using a trained target detection algorithm model and removing noise interference from non-cervical regions. Step 3: Using the Swin Transformer as the feature extraction network, the acetic acid test colposcopy image and the iodine solution colposcopy image obtained from the preprocessing in Step 2 are input into two feature extraction networks respectively to obtain the acetic acid image feature maps. Image feature map of iodine solution Step 4: Acetic acid image feature map Image feature map of iodine solution The corresponding addition yields the addition feature map C. i Addition Feature Map C i The input is fed into a multimodal attention module to obtain the corresponding weighted features. and and Each with the corresponding and Weighted summation is performed to obtain multimodal attention feature maps. and i∈[1,4], the specific process is as follows: Acetic acid image feature map Image feature map of iodine solution Adding them together yields the additive feature map C. i : Add feature map C i The input is fed into a multimodal attention function, and the multimodal attention symbol is defined as MSAtten(·); then the weight features... and Weight features and Corresponding to the acetic acid image feature maps Image feature map of iodine solution Weighting is performed to obtain multimodal attention feature maps. and Step 5: Convert the multimodal attention feature map and The input is fed into the decoder module, i∈[1,4], which obtains the decoded feature map U through cascaded upsampling. a and U b The segmentation head ultimately outputs the segmentation results of the HSIL+ regions in the acetic acid solution image and the HSIL+ regions in the iodine solution image; The decoder module described is implemented as follows: Assume F d It is the upsampled feature map in the decoder, F i It is the input multimodal attention feature map and The specific steps are as follows: Upsampling: F d =Upsample(F i ) cascade: F concat =Concat(F d ,F i ) The multimodal attention feature map obtained in step four and The input to the decoder module undergoes four cascaded upsampling steps to obtain the decoded feature map U. a and U b ; The segmentation head consists of CNN layers with 3x3 kernels, and the input channels are multimodal attention features. or The corresponding number of channels, c, is the number of categories in the output channel; Step 6: Input the colposcopy image labeled in Step 1 into the neural network MSS-Net constructed in Steps 2, 3, 4, and 5 for training. Set the loss function as a weighted average loss function of cross-entropy loss function and Dice loss function to obtain the trained model. Input the colposcopy image to be detected into the trained neural network MSS-Net to obtain the HSIL+ region segmentation result.

2. The multimodal cervical precancerous lesion region segmentation method based on Swing Transformer according to claim 1, characterized in that, The annotation operation described in step one is as follows: Professional physicians used annotation tools to label the HSIL+ regions of cervical precancerous lesions in colposcopy images with acetic acid solution and colposcopy images with iodine solution. The HSIL+ regions include high-grade squamous intraepithelial lesion HSIL and cervical cancer regions. Acetic acid solution images and iodine solution images, along with corresponding acetic acid solution image labels and iodine solution image labels, were obtained from the colposcopy examination of each patient as a training set.

3. The multimodal cervical precancerous lesion region segmentation method based on Swing Transformer according to claim 2, characterized in that, Step two, regarding the creation of the training set for the object inspection algorithm and the selection of the object detection algorithm, are as follows: The colposcopy image data collected in step one is labeled again. A professional physician uses a rectangle tool to outline the cervical region centered on the cervix and outputs it as a label. The original colposcopy images and labels from step one are then organized and used as the training set for the target detection algorithm model. The object detection algorithm described refers to deep learning-based object detection algorithms, including Fast R-CNN and YOLO.

4. The multimodal cervical precancerous lesion region segmentation method based on Swing Transformer according to claim 3, characterized in that, The feature extraction network described in step three: Define the colposcopy image of the acetic acid test as X, and the colposcopy image of the iodine solution as Y, input into the Swing Transform network as S(·); then the acetic acid image feature map... Image feature map of iodine solution The following is an expression: Where i∈[1,4] is determined by the number of BasicLayers in the Swin Transformer.

5. A multimodal cervical precancerous lesion region segmentation method based on Swing Transformer as described in claim 4, characterized in that, Detailed structure of the described multimodal attention mechanism: Multimodal attention consists of a global information aggregation module and two feature weight generation modules; The implementation process of the global information aggregation module is as follows: First, global average pooling is used to aggregate the information of each channel of the input feature map, obtaining a global description of each channel. Let I be the input feature map with shape (H, W, C), where H is the height, W is the width, and C is the number of channels. Global average pooling is represented as: Among them, Z c Let $c$ be the global average value of the $c$-th channel, where $c \in [1, C]$. The feature weight generation module implementation process is as follows: First, a fully connected layer containing a non-linear activation function is used to generate weights for each channel from the aggregated global description: s=σ(W t1 With) Among them, W t1 σ is the weight matrix for dimensionality reduction, z is the non-linear activation function, z is the vector containing the global average of all channels, and s is the feature map output by the first fully connected layer. Then, use another fully connected layer to increase the dimensionality: W=σ(W t2 s) Among them, W t2 σ is the weight matrix of the increased dimension, s is the non-linear activation function, s is the vector containing the global average value of all channels, and W is the output of the feature weight generation module, i.e. the weight feature. The overall process of channel attention is as follows: the input feature map I is first input into the global information aggregation module to obtain the global description Z, and Z is then input into two feature weight generation modules to obtain the weight features W1 and W2.

Images