A colposcope cervical lesion segmentation method, system, device and medium

Abstract

The application discloses a colposcope cervical lesion segmentation method, system, device and medium based on an aceto-white attention map, and the method comprises the following steps: acquiring a training data set and a test data set; a cervical lesion segmentation model based on a full convolutional neural network, the model comprising two inputs of an aceto-stained colposcope image and an aceto-white attention map; iteratively training the cervical lesion segmentation model based on the training data set; evaluating the cervical lesion segmentation model after training by using the test data set, and the cervical lesion segmentation model meeting the performance requirement can be used for subsequent colposcope cervical lesion area prediction. The application constructs an aceto-white attention map through a pre-aceto-stained colposcope image and a post-aceto-stained colposcope image, and filters interference information such as background in a convolutional coding block output feature map through a spatial attention mode, so that the precision of the full convolutional neural network segmentation model in the colposcope cervical lesion area segmentation task is effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and image segmentation, and more specifically to a colposcopic method, system, equipment, and medium for segmenting cervical lesions based on acetic acid reaction attention maps. Background Technology

[0002] Cervical cancer is one of the three major gynecological malignancies that seriously threaten women's lives and health, severely impacting the physical health and quality of life of women of childbearing age. The WHO states that only when cervical cancer screening coverage exceeds 80% can the incidence and mortality rates of cervical cancer be effectively reduced. In my country, the cervical cancer screening coverage rate is only 25.7%. Therefore, finding a simple, objective, and real-time cervical cancer screening technology that can improve screening coverage is a key focus for obstetricians and gynecologists and policymakers in China.

[0003] The internationally recognized cervical cancer screening method currently uses a three-step approach: cytology and / or HPV testing, colposcopy, and histopathological examination, progressing step by step. Colposcopy plays a crucial role in bridging these three steps. It is a non-invasive, repeatable examination method that does not affect cervical lesions; it can also guide targeted biopsy sampling at suspicious lesion sites, which is important and irreplaceable.

[0004] However, the descriptive system and evaluation criteria for colposcopy images are relatively complex, making its widespread clinical application difficult. The accuracy of colposcopy heavily relies on the physician's subjective experience. In my country, there is a severe shortage of highly skilled professional colposcopy doctors, and the diagnostic efficiency of colposcopy falls far short of meeting the substantial clinical demand. In primary care hospitals, many suspicious lesions detected in early screening encounter bottlenecks in the diagnostic stage, namely, primary care physicians are unable to locate suspicious lesions and guide targeted biopsy sampling through colposcopy, leading to missed diagnoses. Therefore, there is an urgent need for an automated method to assist colposcopy doctors in segmenting cervical lesions.

[0005] In recent years, fully convolutional neural networks (WCNNs), as a representative of deep learning technology, have shown great potential in lesion segmentation tasks in medical imaging such as computed tomography (CT), magnetic resonance imaging (MRI), and gastrointestinal endoscopy. Some studies have also attempted to apply WCNNs to the task of cervical lesion segmentation in colposcopy. During colposcopy, doctors repeatedly compare changes in the cervical epithelium before and after acetic acid application to identify lesions and improve the accuracy of biopsies. Currently, most existing colposcopy cervical lesion segmentation models based on WCNNs are developed based on colposcopy images after acetic acid staining, failing to fully utilize the changes in colposcopy images before and after acetic acid staining. This results in poor adaptability of the models to the task of colposcopy cervical lesion segmentation and difficulty in accurately segmenting cervical lesion regions.

[0006] Therefore, proposing a colposcopy cervical lesion segmentation model based on a fully convolutional neural network that can fully utilize the changes in colposcopy images before and after acetic acid staining to achieve automatic and fine segmentation of the colposcopy cervical lesion region is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a colposcopy-based method, system, device, and medium for cervical lesion segmentation based on acetic acid whitening reaction attention maps. An acetic acid whitening reaction attention map is constructed using colposcopy images before and after acetic acid staining. Spatial attention is used to filter background and other interference information in the encoder output feature map of the fully convolutional neural network segmentation model, thereby improving the accuracy of the colposcopy-based cervical lesion region segmentation model. To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A colposcopy-based method for segmenting cervical lesions based on acetic acid whitening reaction attention diagrams includes the following steps:

[0009] S1: Obtain colposcopy images before and after acetic acid staining, and gold standard annotations of cervical lesion areas based on the colposcopy images after acetic acid staining. Normalize the colposcopy images before and after acetic acid staining to obtain training and test datasets.

[0010] S2: Input the colposcopy image after acetic acid staining as the main branch into the pre-constructed cervical lesion segmentation model; Based on the colposcopy image before and after acetic acid staining, obtain the acetic acid whitening reaction attention map, and input the acetic acid whitening reaction attention map as an auxiliary branch into the pre-constructed cervical lesion segmentation model;

[0011] S3: Iteratively train the cervical lesion segmentation model based on the training dataset, construct a supervised loss function using the lesion region prediction results output by the cervical lesion segmentation model and the gold standard annotation of the cervical lesion region, and iteratively optimize the cervical lesion segmentation model using the supervised loss function until the model converges, and save the model parameters;

[0012] S4: Use the test dataset to evaluate the cervical lesion segmentation model after training. Evaluate the cervical lesion segmentation model whose performance meets the requirements for subsequent colposcopy cervical lesion region prediction. Input the colposcopy image to be detected before and after acetic acid staining into the cervical lesion segmentation model to obtain the cervical lesion region prediction result.

[0013] Optionally, the step of obtaining the acetic acid whitening reaction attention map in step S2 includes: first, registering the colposcopy images before and after acetic acid staining using the BSpline method; converting the registered colposcopy images before and after acetic acid staining from the RGB color space to the LAB color space; and obtaining the acetic acid whitening reaction attention map by subtracting the A channel of the post-acetic acid staining colposcopy image from the registered pre-acetic acid staining colposcopy image.

[0014] Optionally, the acetic acid reaction attention map mentioned in step S2 can be used as an auxiliary branch input for the cervical lesion segmentation model as follows: the acetic acid reaction attention map is input into the spatial attention weight generation module to generate an attention weight map; the attention weight map is downsampled to the corresponding size using the nearest neighbor interpolation method; the original attention weight map and the downsampled attention weight map are element-wise multiplied with the corresponding convolutional coding block output feature map in the cervical lesion segmentation model to filter out background and other interference information in the feature map.

[0015] Optionally, the spatial attention weight generation module consists of two 3*3 convolutional layers, a BatchNorm layer, a ReLU layer, and a Sigmoid layer connected in series. After the acetowhite reaction attention map is input into the spatial attention weight generation module, it is mapped into an attention weight map with values ​​in the range [0,1].

[0016] Some embodiments of this application also provide a colposcopy-based cervical lesion segmentation system based on acetic acid whitening reaction attention maps. The system includes: a data acquisition module for acquiring pre-acetic acid-stained colposcopy images, post-acetic acid-stained colposcopy images, acetic acid whitening reaction attention maps, and gold standard annotations of cervical lesion regions based on post-acetic acid-stained colposcopy images, obtaining a training dataset and a test dataset; a data input module for inputting the post-acetic acid-stained colposcopy images as the main branch into a pre-constructed cervical lesion segmentation model, and inputting the acetic acid whitening reaction attention maps as auxiliary branches into the pre-constructed cervical lesion segmentation model; a training module for iteratively training the cervical lesion segmentation model based on the training dataset; and a testing module for evaluating the trained cervical lesion segmentation model using the test dataset, and evaluating cervical lesion segmentation models whose performance meets the requirements for subsequent colposcopy-based cervical lesion region prediction.

[0017] Some embodiments of this application also provide a colposcopy cervical lesion segmentation device, the device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the method described above.

[0018] Some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the colposcopic cervical lesion segmentation method.

[0019] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a colposcopy-based method for segmenting cervical lesions based on an acetic acid whitening reaction attention diagram, with the following beneficial effects:

[0020] This invention constructs an acetic acid whitening reaction attention map using colposcopy images before and after acetic acid staining, and introduces this attention map into a colposcopy cervical lesion segmentation model using spatial attention. This filters out background and other interference information in the encoder output feature map of the fully convolutional neural network segmentation model, enhancing the segmentation model's ability to perceive the target lesion area. This invention fully utilizes the changes in colposcopy images before and after acetic acid staining, effectively improving the accuracy of the fully convolutional neural network segmentation model in the task of colposcopy cervical lesion segmentation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a colposcopic cervical lesion segmentation method based on an acetic acid whitening reaction attention diagram;

[0023] Figure 2 The structure diagrams are shown for convolutional coding blocks 1-5 and convolutional decoding blocks 1-3.

[0024] Figure 3 Here is a structural diagram of convolutional decoding block 4;

[0025] Figure 4 This is a schematic diagram of the acetic acid whitening reaction process.

[0026] Figure 5 The structure diagram of the spatial attention weight generation module.

[0027] Figure 6 This is a schematic diagram of a colposcopic cervical lesion segmentation system.

[0028] Figure 7 This is a schematic diagram of a colposcope for segmenting cervical lesions. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 1 The image shows a colposcopy-based method for segmenting cervical lesions based on an acetic acid whitening reaction attention diagram, provided by the present invention. This method specifically includes the following steps:

[0031] S1: Construct training and testing datasets. The data includes colposcopy images before and after acetic acid staining, and gold standard annotations of cervical lesion regions based on colposcopy images after acetic acid staining. Normalize the colposcopy images before and after acetic acid staining, and divide the data into training and testing datasets in a 7:3 ratio.

[0032] The colposcopy images before and after acetic acid staining were stored in RGB format. The values ​​of each color channel were divided by 255 to normalize them to 0-1.

[0033] The gold standard annotation of cervical lesion areas based on acetic acid-stained colposcopy images is stored in a binary map format, with pixels within the lesion area represented by 1 and pixels outside the lesion area represented by 0.

[0034] Data including pre-stained colposcopy images, post-stained colposcopy images, and gold standard annotations of cervical lesion regions based on post-stained colposcopy images were divided into training and testing datasets in a 7:3 ratio. The training dataset was used to train the cervical lesion segmentation model, and the testing dataset was used to evaluate the trained cervical lesion segmentation model.

[0035] S2: Construct a cervical lesion segmentation model based on a fully convolutional neural network. The cervical lesion segmentation model includes two inputs: a colposcopy image after acetic acid staining as the main input of the cervical lesion segmentation model, and an acetic acid whitening reaction attention image as an auxiliary branch input of the cervical lesion segmentation model.

[0036] like Figure 1 As shown, the constructed cervical lesion segmentation model is mainly a symmetrical U-shaped network, consisting of convolutional coding blocks 1-5 and convolutional decoding blocks 1-4 connected in series; the structures of convolutional coding blocks 1-5 and convolutional decoding blocks 1-3 are as follows. Figure 2 As shown, it consists of two sets of convolutional blocks (including 3*3 convolutional layers, BatchNorm layers, and ReLU layers) alternately connected in series; the structure of convolutional decoding block 4 is as follows. Figure 3As shown, it consists of two sets of convolutional blocks (including 3*3 convolutional layers, BatchNorm layers and ReLU layers), 3*3 convolutional layers, 1*1 convolutional layers and Sigmoid layers connected in series.

[0037] Max pooling layers are used after convolutional encoding blocks 1-4 to reduce the feature map size back to its original size. After convolutional coding block 5 and convolutional decoding blocks 1-3, a deconvolutional layer is used to enlarge the feature map size to twice its original size. After the feature map passes through convolutional decoding block 4, the cervical lesion segmentation result map is output. The size of the cervical lesion segmentation result map is consistent with the size of the input colposcopy image of the cervical lesion segmentation model.

[0038] By using skip connections, the output feature maps of convolutional coding blocks 1-4 are concatenated with the output feature maps of convolutional coding block 5 and convolutional decoding blocks 1-3 to achieve the function of feature information fusion and supplementation.

[0039] like Figure 4 As shown, the steps for generating the acetic acid whitening reaction attention diagram are as follows:

[0040] ① First, the colposcopy images before and after acetic acid staining are registered using the BSpline method. The colposcopy images before acetic acid staining are then adjusted to obtain the registered colposcopy images before acetic acid staining.

[0041] ② Convert the registered colposcopy images before and after acetic acid staining from the RGB color space to the LAB color space, respectively;

[0042] ③Based on the LAB color space, the A channel of the colposcopy image after acetic acid staining and the A channel of the colposcopy image before acetic acid staining after registration are extracted respectively, and the difference between the two is used to obtain the acetic acid whitening reaction attention image.

[0043] The method for using the acetic acid whitening reaction diagram as an auxiliary branch input in the cervical lesion segmentation model is as follows:

[0044] ① Input the acetic acid whitening reaction attention map into the spatial attention weight generation module to generate an attention weight map; for example Figure 5 As shown, the spatial attention weight generation module consists of two 3*3 convolutional layers, a BatchNorm layer, a ReLU layer, and a Sigmoid layer, which are alternately connected in series. After the acetowhite reaction attention map is input into the spatial attention weight generation module, it is mapped into an attention weight map with values ​​in the range [0,1].

[0045] ② Downsample the attention weight map to the corresponding size using the nearest neighbor interpolation method. size);

[0046] ③ Multiply the original attention weight map and the downsampled attention weight map element-wise with the output feature maps of the corresponding convolutional coding blocks 1-4 in the cervical lesion segmentation model to filter out background and other interference information in the feature maps.

[0047] S3: Iteratively train the cervical lesion segmentation model based on the training dataset. Construct a supervised loss function using the lesion region prediction results output by the cervical lesion segmentation model and the gold standard annotation of the cervical lesion region. Iteratively optimize the cervical lesion segmentation model using this loss function until the model converges and save the model parameters.

[0048] The Hybrid_Loss loss was calculated using the lesion region prediction results output by the cervical lesion segmentation model and the gold standard annotation of the cervical lesion region. Within each training cycle, the cervical lesion segmentation model was iteratively optimized using the Hybrid_Loss loss. The relevant training parameters are shown in the table below.

[0049]

[0050]

[0051] Observe the convergence of the Hybrid_Loss loss. Stop model training when the Hybrid_Loss loss curve converges or reaches the set maximum number of training cycles, and save the model parameters of the trained cervical lesion segmentation model.

[0052] S4: Use the test dataset to evaluate the cervical lesion segmentation model after training. The cervical lesion segmentation model whose performance meets the requirements can be used for subsequent colposcopy cervical lesion region prediction. Input the colposcopy images to be detected before and after acetic acid staining into this model to obtain the cervical lesion region prediction results.

[0053] Figure 6 A colposcopic cervical lesion segmentation system is shown, the system comprising:

[0054] The data acquisition module is used to acquire colposcopy images before and after acetic acid staining, acetic acid whitening reaction attention diagrams, and gold standard annotations of cervical lesion areas based on colposcopy images after acetic acid staining, to obtain training datasets and test datasets.

[0055] The data input module is used to input the acetic acid-stained colposcopy image as the main branch into the pre-constructed cervical lesion segmentation model, and to input the acetic acid whitening reaction attention image as an auxiliary branch into the pre-constructed cervical lesion segmentation model.

[0056] The training module iteratively trains the cervical lesion segmentation model based on the training dataset;

[0057] The testing module is used to evaluate the trained cervical lesion segmentation model using the test dataset. The cervical lesion segmentation model whose performance meets the requirements is used for subsequent colposcopy cervical lesion region prediction.

[0058] It is not difficult to see that the embodiments of this application are system embodiments corresponding to the method embodiments. The implementation details of the embodiments of this application have been described in the method embodiments, and will not be repeated here to avoid repetition.

[0059] Furthermore, this application also provides a colposcopic cervical lesion segmentation device, the structure of which is as follows: Figure 7 As shown, the device includes a memory 90 for storing computer-readable instructions and a processor 100 for executing the computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor is triggered to execute the colposcopic cervical lesion segmentation method.

[0060] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0061] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0062] In another aspect, embodiments of this application also provide a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer-readable instructions, which may be executed by a processor to implement the steps of the methods and / or technical solutions of the various embodiments of this application.

[0063] Furthermore, this application also provides a computer program stored in a computer device, which causes the computer device to execute the method for executing the control code.

[0064] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0065] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0066] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A colposcopic cervical lesion segmentation method based on aceto-white reaction attention map, characterized in that, The method comprises the following steps: S1: obtaining a pre-acetic acid stained colposcopy image, a post-acetic acid stained colposcopy image, and a gold standard annotation of a cervical lesion area based on the post-acetic acid stained colposcopy image, performing a normalization operation on the pre-acetic acid stained colposcopy image and the post-acetic acid stained colposcopy image to obtain a training data set and a test data set; S2: inputting the post-acetic acid stained colposcopy image into a pre-constructed cervical lesion segmentation model as a main branch; obtaining an aceto-white reaction attention map according to the pre-acetic acid stained colposcopy image and the post-acetic acid stained colposcopy image, and inputting the aceto-white reaction attention map into the pre-constructed cervical lesion segmentation model as an auxiliary branch, wherein the cervical lesion segmentation model is constructed based on a full convolutional neural network; wherein The step of obtaining the aceto-white reaction attention map comprises: firstly, performing registration on the pre-acetic acid stained colposcopy image and the post-acetic acid stained colposcopy image by a BSpline method; converting the registered pre-acetic acid stained colposcopy image and the post-acetic acid stained colposcopy image from an RGB color space to an LAB color space; and obtaining the aceto-white reaction attention map by subtracting an A channel of the registered pre-acetic acid stained colposcopy image from an A channel of the post-acetic acid stained colposcopy image; The step of inputting the aceto-white reaction attention map into the pre-constructed cervical lesion segmentation model as an auxiliary branch comprises: inputting the aceto-white reaction attention map into a spatial attention weight generation module to generate an original attention weight map; down-sampling the attention weight map to a corresponding size by a nearest neighbor interpolation method to obtain a down-sampled attention weight map; and multiplying the original attention weight map and the down-sampled attention weight map with feature maps output by corresponding convolutional encoding blocks in the cervical lesion segmentation model to filter interference information in the feature maps; The spatial attention weight generation module is composed of two 3*3 convolutional layers, a BatchNorm layer, a ReLU layer, and a Sigmoid layer connected in an alternating manner, and the aceto-white reaction attention map is mapped into an attention weight map with a value range of [0, 1] after being input into the spatial attention weight generation module; S3: iteratively training the cervical lesion segmentation model based on the training data set, constructing a supervision loss function based on a lesion area prediction result output by the cervical lesion segmentation model and the gold standard annotation of the cervical lesion area, and iteratively optimizing the cervical lesion segmentation model based on the supervision loss function until the model converges, and saving model parameters; S4: evaluating the cervical lesion segmentation model after training using the test data set, and using the cervical lesion segmentation model with an evaluation performance meeting a requirement for subsequent colposcopy cervical lesion area prediction, inputting a pre-acetic acid stained colposcopy image and a post-acetic acid stained colposcopy image to be detected into the cervical lesion segmentation model to obtain a cervical lesion area prediction result.

2. A colposcopic cervical lesion segmentation system based on aceto-white reaction attention map, characterized in that, The system comprises: a data acquisition module configured to obtain a pre-acetic acid stained colposcopy image, a post-acetic acid stained colposcopy image, an aceto-white reaction attention map, and a gold standard annotation of a cervical lesion area based on the post-acetic acid stained colposcopy image to obtain a training data set and a test data set; The data input module is configured to input the post-acetic acid staining colposcopic image as a main branch into the pre-constructed cervical lesion segmentation model and input the aceto-white reaction attention map as an auxiliary branch into the pre-constructed cervical lesion segmentation model. The data input module is configured to: First, the pre-acetic acid staining colposcopic image and the post-acetic acid staining colposcopic image are registered by a BSpline method; the registered pre-acetic acid staining colposcopic image and the post-acetic acid staining colposcopic image are converted from an RGB color space to an LAB color space; and the A channel of the post-acetic acid staining colposcopic image is subtracted from the A channel of the registered pre-acetic acid staining colposcopic image to obtain the aceto-white reaction attention map; The aceto-white reaction attention map is input into a spatial attention weight generation module to generate an original attention weight map; the attention weight map is down-sampled to a corresponding size by a nearest neighbor interpolation method to obtain a down-sampled attention weight map; and the original attention weight map and the down-sampled attention weight map are multiplied by the feature map output by the corresponding convolutional encoding block in the cervical lesion segmentation model to filter the interference information in the feature map; The spatial attention weight generation module is composed of two 3*3 convolutional layers, a BatchNorm layer, a ReLU layer and a Sigmoid layer which are connected in an alternating manner; and the aceto-white reaction attention map is mapped into an attention weight map with a value range of [0, 1] after being input into the spatial attention weight generation module; The training module is configured to iteratively train the cervical lesion segmentation model based on the training data set; The test module is configured to evaluate the cervical lesion segmentation model after training using the test data set, and the cervical lesion segmentation model with a performance meeting the requirements is used for subsequent colposcopic cervical lesion area prediction.

3. A colposcopic cervical lesion segmentation device based on aceto-white reaction attention map, characterized by, The device comprises: one or more processors; and a memory storing computer program instructions which, when executed, cause the processor to perform the method of claim 1.

4. A computer readable medium having stored thereon computer program instructions executable by a processor to implement the method of claim 1.

Images