A semi-supervised cervical cancer image classification method based on virtual adversarial training network

Abstract

The present invention discloses a semi-supervised cervical cancer image classification method based on a virtual adversarial training network, comprising: collecting a cervical cancer image dataset to obtain a training dataset and a test dataset for the semi-supervised network; performing data augmentation on the labeled and unlabeled cervical cancer images in the training dataset, including performing translation, rotation, and scaling transformations on the cervical cancer images; building a backbone network including AlexNet and a hybrid attention mechanism; building a virtual adversarial training network, taking a given input and adding perturbations, so that the model gives different outputs; penalizing the model's sensitivity due to the perturbations; obtaining relevant features of the cervical cancer images through the virtual adversarial training network and feeding them into a classifier for classification, and finally outputting the category to which the test cervical cancer image belongs. The present invention can fully utilize both labeled and unlabeled data, and perform classification through the model, thereby assisting doctors in better diagnosis and treatment.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to a method for classifying cervical cancer images using a deep learning method based on an attention mechanism and a virtual adversarial training network (VATN), in particular, a semi-supervised cervical cancer image classification method based on a virtual adversarial training network. Background Art

[0002] Public reports in 2018 revealed over 500,000 new cases of cervical cancer and over 300,000 deaths. The disease is typically caused by persistent infection with one or more of a dozen strains of human papillomavirus (HPV). Early detection of HPV-induced precancerous lesions (precursors of cancer) and, when necessary, appropriate treatment can reduce suffering and premature death. However, in low- and middle-income countries (LMICs), there is a severe shortage of clinical and gynecological services and expertise, as well as insufficient access to effective, low-cost cervical screening programs. This shortage is highly correlated with the highest reported mortality rates in these regions.

[0003] Cervical cancer images are an essential component of modern medical imaging. Automated medical image analysis using advanced artificial intelligence technologies, such as deep learning, has become a key trend in this field. Li et al. used a convolutional neural network (CNN) that combined global context information and an attention mechanism to classify cervical cells (Li J, Dou Q, Yang H, et al. Cervical cell multi-classification algorithm using global context information and attention mechanism [J]. Tissue and Cell, 2022, 74: 101-677). Hyeon et al. proposed an automatic classification model that uses a pretrained convolutional neural network for feature extraction and compared the classification results of four classifiers: logistic regression, random forest, ensemble learning, and support vector machine (Hyeon J, Choi HJ, Lee BD, et al. Diagnosing cervical cell images using a pre-trained convolutional neural network as a feature extractor [C] / / 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). IEEE, 2017: 390-393). However, annotating medical data is an expert-oriented, expensive, and time-consuming task. In most cases, obtaining ground truth labels is demanding, requiring experienced clinicians to combine multimodal clinical records and consult with other experts. Compared to the limited amount of annotated images, clinical centers store a large amount of raw cervical cancer images, motivating methods to go beyond traditional supervised learning by incorporating more widely available unlabeled data. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a cervical cancer image classification method based on a semi-supervised virtual adversarial training network with an attention mechanism. It uses labeled data and unlabeled data to perform classification through a model to assist doctors in better diagnosis and treatment.

[0005] In order to solve the above technical difficulties, the present invention adopts the following technical solutions.

[0006] A semi-supervised cervical cancer image classification method based on a virtual adversarial training network, wherein the virtual adversarial training network comprises a backbone network and a virtual adversarial training module, and the method comprises the following steps:

[0007] Step 1. Collect a dataset of cervical cancer images, including the following cervical cancer images: cells with a high nuclear-cytoplasmic ratio that meet the HSIL standard, abnormal keratinized cells in HSIL or SCC, abnormal naked nuclei in HSIL or SCC, koiled cells, non-koiled cells that meet the LSIL standard, abnormal glandular cells, abnormal metaplastic cells, normal naked nuclei, normal middle and surface cells, pathogenic microbial cells, normal parabasal cells, normal endocervical cells, normal endometrial glandular cells, typical squamous metaplastic cells, and typical parakeratotic cells. Relevant experts annotate the cervical cancer images, and select 30% of the cervical cancer images to hide the annotations or select unannotated cervical cancer images, and finally obtain the training dataset and test dataset for the semi-supervised network.

[0008] Step 2: Data augmentation is performed on the labeled and unlabeled cervical cancer images in the training dataset, including translation transformation as shown in formula (1), rotation transformation as shown in formula (2), and scaling transformation as shown in formula (3) to increase the size of the dataset:

[0009]

[0010]

[0011]

[0012] Where x, y, w are pixel points, and α is the rotation angle;

[0013] Step 3: Build a backbone network consisting of AlexNet and a hybrid attention mechanism. This involves adding an attention mechanism to AlexNet to correct the features of cervical cancer images. This correction preserves valuable features of cervical cancer images. The attention mechanism operates as follows: squeeze-like global average pooling and two full connections using excitation.

[0014] Step 4: Build a virtual adversarial training network: Take pairs of data points that are very close in the input space but very far apart in the model output space; then train the model to make their outputs close to each other; to do this, take a given input and add perturbations so that the model gives different outputs; then, the model penalizes sensitivity due to the perturbations; thus, building a complete virtual adversarial training network;

[0015] Step 5: The cervical cancer image is trained through a virtual adversarial network to obtain relevant features, which are then sent to the classifier for classification, and finally the category to which the tested cervical cancer image belongs is output.

[0016] Specifically, the squeeze-type global average pooling described in step 3 includes:

[0017] Compress a c-channel, hxw feature map into a c-channel 1x1, and the result represents global information:

[0018]

[0019] Where H, W are the length and width of the image respectively, u c (i, j) is the image, z c Represents global information.

[0020] Specifically, the excitation two full connections described in step 3 include:

[0021] The result of squeeze global pooling, a C-dimensional vector, is fully connected to obtain a C / r-dimensional vector and then reulu activation is performed. A full connection is then performed on the C / r-dimensional vector on the reulu activation to convert the C / r-dimensional vector back to a C-dimensional vector. Sigmoid activation is then performed to make the value between 0 and 1, resulting in the weight matrix:

[0022] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) (5)

[0023] W1, W2 are weight matrices, and σ is the sigmoid activation function.

[0024] Specifically, the step 4 includes:

[0025] Step 4.1. Use Represents a labeled dataset, using Represents an unlabeled dataset; let x * Represents x l or x ul ; The objective function is:

[0026] L qadv :=D[q(y|x * ),p(y|x * +r qadv ,θ)]

[0027] wherer qadv :=argmax r:‖r‖≤∈ D[q(y|x * ),p(y|x * +r,θ)] (6)

[0028] Among them, D[p,q] is a non-negative metric function that measures the distance between distributions p and q; r adv is the adversarial perturbation, and p(y|x,θ) approximately represents it.

[0029] Step 4.2. Use Replace q(y|x * ), the loss function becomes:

[0030]

[0031] Add LDS to the entire objective function:

[0032]

[0033] loss=l(D l ,θ)+αR vadv (D l ,D ul ,θ)

[0034] Among them, l(D l ,θ) is the negative log-likelihood function of labeled data;

[0035] The entire regularization process has only two hyperparameters ∈ and α, and α is fixed to 1.

[0036] Furthermore, in said step 4, step 4.3 includes:

[0037] Notice If h(y|y l ) is replaced by p(y|x l ,θ), then g is always equal to 0, and the second-order information must be used;

[0038] If the first derivative is 0, then D[p(y|x * ,θ),p(y|x * +r,θ)] second-order Taylor expansion:

[0039]

[0040] in, is the Hessian matrix, denoted as H; then D[p(y|x * ,θ),p(y|x * +r,θ)] is recorded as and

[0041] in u is the characteristic unit vector corresponding to the largest eigenvalue of H;

[0042] Use the energy iteration method to calculate the eigenvector:

[0043]

[0044] Step 4.4. Use label smoothing to replace the traditional ont-hot to obtain more pseudo labels Information:

[0045]

[0046] Where y represents the distribution probability of the original label, α is a small hyperparameter, generally taken as 0.1; N is the total number of categories, that is,

[0047]

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] 1. This paper introduces a virtual adversarial training loss module, an effective technique for local distribution smoothness. It uses pairs of data points that are very close in input space but very close in model output space. The model is then trained to make their outputs close to each other. To do this, it takes a given input and finds a perturbation that results in a very different output from the model. The model then penalizes its sensitivity to the perturbation.

[0050] 2. The present invention introduces the AlexNet network module and the attention mechanism. Adding the attention mechanism on the basis of the AlexNet network can correct the features. The corrected features can retain valuable features and eliminate worthless features, focusing on the attention extraction of the channel. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0052] Figure 2 This is a label-smoothed virtual adversarial training network graph of the present invention.

[0053] Figure 3 Schematic diagram of the virtual adversarial training network structure of the attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] This paper provides a semi-supervised cervical cancer image classification method based on a virtual adversarial training network. Computer studies have shown that unlabeled images often contain valuable information that is not present in labeled images. This paper utilizes unlabeled and labeled cervical cancer images to perform classification through model training, assisting doctors in better diagnosis and treatment.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings.

[0056] like Figure 1As shown, the present invention provides a semi-supervised cervical cancer image classification method based on a virtual adversarial training network, wherein the virtual adversarial training network includes a backbone network module and a virtual adversarial training module. The method includes the following steps:

[0057] Step 1: Collect a dataset of cervical cancer images, including "cells with a high nuclear-cytoplasmic ratio that meet HSIL standards", "abnormal keratinized cells in HSIL or SCC", "abnormal naked nuclei in HSIL or SCC", "knocking out cells", "non-knocking out cells that meet LSIL standards", "abnormal glandular cells", "abnormal metaplastic cells", "normal naked nuclei", "normal mid-surface cells", "pathogenic microbial cells", "normal parabasal cells", "normal endocervical cells", "normal endometrial glandular cells", "typical squamous metaplastic cells", and "typical parakeratotic cells". Relevant experts annotated the cervical cancer images, and selected 30% of the cervical cancer images to hide the annotations or select unannotated cervical cancer images, and finally obtain the training dataset and test dataset for the semi-supervised network;

[0058] Step 2: Data augmentation is performed on the labeled and unlabeled cervical cancer images in the training dataset, including translation transformation as shown in formula (1), rotation transformation as shown in formula (2), and scaling transformation as shown in formula (3) to increase the size of the dataset:

[0059]

[0060]

[0061]

[0062] Where x, y, w are pixel points and α is the rotation angle.

[0063] Step 3: Build the backbone network. This module includes AlexNet and a hybrid attention mechanism. Adding the attention mechanism to AlexNet corrects the features of cervical cancer images, preserving valuable features. This includes channel attention extraction. This includes the following operations: squeeze-like global average pooling and two full connections using excitation.

[0064] Step 3.1. First, perform a squeeze operation, which is similar to global average pooling. A c-channel hxw feature map is compressed into a c-channel 1x1. The result represents global information.

[0065]

[0066] Where H, W are the length and width of the image respectively, u c (i, j) is the image, zc Represents global information

[0067] Step 3.2. Perform another excitation operation, including two full connections. Perform a full connection on the result of squeeze global pooling (which can be viewed as a C-dimensional vector) to obtain a C / r-dimensional vector, which is then re-activated. Perform another full connection on the C / r-dimensional vector to convert it back to a C-dimensional vector. Then perform a sigmoid activation (to keep the value between 0 and 1) to obtain the weight matrix.

[0068] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) (5)

[0069] W1, W2 are weight matrices, and σ is the sigmoid activation function

[0070] Step 3.3. The AlexNet network module has 8 layers, including 5 convolutional layers and 3 fully connected layers.

[0071] The first layer: convolution layer 1, the input is a 96×96×3 image, the number of convolution kernels is 96,

[0072] The second layer is convolutional layer 2. The input is the feature map of the previous convolution layer. The number of convolutions is 256. The two GPUs in the paper each have 128 convolution kernels. The convolution kernel size is: 5×5×48; pad=2, stride=1; then LRN is performed, and finally max_pooling is performed, pool_size=(3,3), stride=2;

[0073] The third layer: Convolution 3, the input is the output of the second layer, the number of convolution kernels is 384, kernel_size = (3 × 3 × 256), padding = 1, the third layer does not perform LRN and Pooling

[0074] The fourth layer: Convolution 4, the input is the output of the third layer, the number of convolution kernels is 384, kernel_size = (3 × 3), padding = 1, and the same as the third layer, without LRN and Pool

[0075] Layer 5: Convolution 5, with the output of layer 4 as input, 256 kernels, kernel_size = (3×3), padding = 1. Then, max_pooling is performed directly, with pool_size = (3, 3) and stride = 2.

[0076] The sixth, seventh, and eighth layers are fully connected layers. The number of neurons in each layer is 4096, and the final output softmax is 16. RELU and Dropout are used in the fully connected layers.

[0077] The AlexNet network module has a total of 8 layers, 5 convolutional layers, and 3 fully connected layers, as shown in the following table

[0078] Layer Name Kernel Size Kernel Num Stride Padding Input Size Output Size Conv1 11×11 48 (×2 GPU) 4 [1,2] 224×224×3 55×55×96 Maxpool1 3×3 2 0 55×55×96 27×27×96 Conv2 5×5 128 (×2 GPU) 1 [2,2] 27×27×96 27×27×256 Maxpool2 3×3 2 0 27×27×256 13×13×256 Conv3 3×3 192 (×2 GPU) 1 [1,1] 13×13×256 13×13×384 Conv4 3×3 192 (×2 GPU) 1 [1,1] 13×13×384 13×13×384 Conv5 3×3 128 (×2 GPU) 1 [1,1] 13×13×384 13×13×256 Maxpool3 3×3 2 0 13×13×256 6×6×256 FC1 2048 6×6×256 4096 FC2 2048 4096 4096 FC3 1000 4096 20

[0079] Step 4: Building a virtual adversarial training module is an effective technique for local distribution smoothness. It takes pairs of data points that are very close in the input space but very close in the model output space. The model is then trained to make their outputs close to each other. To do this, a given input is taken and a perturbation is found where the model gives a very different output. The model is then penalized for sensitivity due to the perturbation, e.g. Figure 2 shown.

[0080] Step 4.1. Use represents a labeled dataset, and represents an unlabeled dataset. Let x * Represents x l or x ul The objective function is now given by

[0081] L qadv :=D[q(y|x * ),p(y|x * +r qadv ,θ)]

[0082] wherer qadv :=argmax r:‖r‖≤∈ D[q(y|x * ),p(y|x * +r,θ)] (6)

[0083] Among them, D[p,q] is a non-negative metric function that measures the distance between distributions p and q, such as cross entropy, which can be called r adv It is the adversarial perturbation, that is, the adversarial party wants to find the quantity to use for attack, and the adversarial party wants to minimize the impact of this quantity on the model. p(y|x,θ) approximately represents.

[0084] Step 4.2. Some samples have no labels, and the labeled samples p(y|x,θ) are similar to q(y|x), so Replace q(y|x * ), the loss function becomes:

[0085]

[0086] So, add LDS to the entire objective function:

[0087]

[0088] loss=l(D l ,θ)+αR vadv (D l ,D ul ,θ)

[0089] Among them, l(D l ,θ) is the negative log-likelihood function of labeled data.

[0090] The entire regularization process has only two hyperparameters ∈ and α, and α is fixed to 1.

[0091] Step 4.3. Note If h(y|y l ) is replaced by p(y|x l ,θ), then g is always equal to 0, and the second-order information must be used.

[0092] If the first derivative is 0, then D[p(y|x * ,θ),p(y|x * +r,θ)] second-order Taylor expansion:

[0093]

[0094] in, is the Hessian matrix, denoted as H. Then D[p(y|x * ,θ),p(y|x * +r,θ)] is recorded as and

[0095] in u is the characteristic unit vector corresponding to the largest eigenvalue of H.

[0096] To calculate the eigenvectors, an energy iteration method is used, which is calculated as:

[0097]

[0098] Step 4.4. When performing multi-classification, one-hot labels are often used to calculate cross entropy loss. However, when using simple cross entropy loss, only the loss of the correct label position is considered, while the loss of the wrong label position is ignored. In order to better give pseudo labels to unlabeled samples for better model training, label smoothing is used to replace the traditional one-hot label to obtain more pseudo labels. information.

[0099]

[0100] Where y represents the distribution probability of the original label, α is a small hyperparameter (usually 0.1), and N is the total number of categories, that is,

[0101]

[0102] Step 5: The cervical cancer image is trained through a virtual adversarial network to obtain relevant features, and then the features are sent to the classifier for classification. Finally, the category of the tested cervical cancer image is output, such as Figure 3 shown.

Claims

1. A semi-supervised cervical cancer image classification method based on virtual adversarial training network, characterized in that: The virtual adversarial training network includes a backbone network and a virtual adversarial training module, and the method includes the following steps: Step 1. Collect a dataset of cervical cancer images, including the following cervical cancer images: cells with a high nuclear-cytoplasmic ratio that meet the HSIL standard, abnormal keratinized cells in HSIL or SCC, abnormal naked nuclei in HSIL or SCC, koiled cells, non-koiled cells that meet the LSIL standard, abnormal glandular cells, abnormal metaplastic cells, normal naked nuclei, normal middle and surface cells, pathogenic microbial cells, normal parabasal cells, normal endocervical cells, normal endometrial glandular cells, typical squamous metaplastic cells, and typical parakeratotic cells. Relevant experts annotate the cervical cancer images, and select 30% of the cervical cancer images to hide the annotations or select unannotated cervical cancer images, and finally obtain the training dataset and test dataset for the semi-supervised network. Step 2: Perform data augmentation on the labeled and unlabeled cervical cancer images in the training dataset, including performing translation transformation on the cervical cancer images as shown in formula (1), rotation transformation as shown in formula (2), and scaling transformation as shown in formula (3) to increase the size of the dataset: Where x, y, w are pixel points, and α is the rotation angle; Step 3: Build a backbone network consisting of AlexNet and a hybrid attention mechanism. This involves adding an attention mechanism to AlexNet to correct the features of cervical cancer images. This correction preserves valuable features of cervical cancer images. The attention mechanism operates as follows: squeeze-like global average pooling and two full connections using excitation. Step 4: Build a virtual adversarial training network: Take pairs of data points that are very close in the input space but very far apart in the model output space; then train the model to make their outputs close to each other; to do this, take a given input and add perturbations so that the model gives different outputs; then, the model penalizes sensitivity due to the perturbations; thus, building a complete virtual adversarial training network; Step 5: The cervical cancer image is trained through a virtual adversarial network to obtain relevant features, which are then sent to the classifier for classification, and finally the category to which the tested cervical cancer image belongs is output.

2. The semi-supervised cervical cancer image classification method based on virtual adversarial training network according to claim 1 is characterized in that: The squeeze-type global average pooling described in step 3 includes: Compress a c-channel, hxw feature map into a c-channel 1x1, and the result represents global information: Where H, W are the length and width of the image respectively, u c (i, j) is the image, z c Represents global information.

3. The semi-supervised cervical cancer image classification method based on virtual adversarial training network according to claim 1 is characterized in that: The excitation described in step 3 is fully connected twice, including: The result of squeeze global pooling, a C-dimensional vector, is fully connected to obtain a C / r-dimensional vector and then reulu activation is performed. A full connection is then performed on the C / r-dimensional vector on the reulu activation to convert the C / r-dimensional vector back to a C-dimensional vector. Sigmoid activation is then performed to make the value between 0 and 1, resulting in the weight matrix: s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) (5) W1, W2 are weight matrices, and σ is the sigmoid activation function.

4. The semi-supervised cervical cancer image classification method based on virtual adversarial training network according to claim 1 is characterized in that: The step 4 includes: Step 4.

1. Use Represents a labeled dataset, using Represents an unlabeled dataset; let x * Represents x l or x ul ; The objective function is: L qadv :=D[q(y∣x * ),p(y∣x * +r qadv ,θ)] where r qadv :=argmax r:‖r‖≤∈ D[q(y∣x * ),p(y∣x * +r,θ)] (6) Among them, D[p,q] is a non-negative metric function that measures the distance between distributions p and q; r adv is the adversarial perturbation, and p(y|x,θ) approximately represents it.

5. The semi-supervised cervical cancer image classification method based on virtual adversarial training network according to claim 1 is characterized in that: The step 4 includes: Step 4.

2. Use Replace q(y|x * ), the loss function becomes: Add LDS to the entire objective function: loss=l(D l ,θ)+αR vadv (D l ,D ul ,i) Among them, l(D l ,θ) is the negative log-likelihood function of labeled data; The entire regularization process has only two hyperparameters ∈ and α, and α is fixed to 1.

6. The semi-supervised cervical cancer image classification method based on virtual adversarial training network according to claim 1 is characterized in that: In said step 4, step 4.3 includes: Notice If h(y|y l ) is replaced by p(y|x l ,θ), then g is always equal to 0, and the second-order information must be used; If the first derivative is 0, then D[p(y|x * ,θ),p(y|x * +r,θ)] second-order Taylor expansion: in, is the Hessian matrix, denoted as H; then D[p(y|x * ,θ),p(y|x * +r,θ)] is recorded as and in u is the characteristic unit vector corresponding to the largest eigenvalue of H; Use the energy iteration method to calculate the eigenvector: r vadv ≈∈d。 7. The semi-supervised cervical cancer image classification method based on virtual adversarial training network according to claim 1 is characterized in that: The step 4 includes: Step 4.

4. Use label smoothing to replace the traditional ont-hot to obtain more pseudo labels Information: Where y represents the distribution probability of the original label, α is a small hyperparameter, which is 0.1; N is the total number of categories, that is,

Images