Semi-supervised keratitis IVCM image classification method, system and device based on median enhanced attention mechanism and medium

Abstract

The invention discloses a semi-supervised keratitis IVCM image classification method, system and device based on a median enhanced attention mechanism and a medium, and the method comprises the steps: carrying out the class prediction of IVCM images with labels, weak enhancement and no labels after strong enhancement, and obtaining corresponding class score vectors; according to the category score vector with the label, obtaining label loss; according to the category score vector without the label after weak enhancement and a confidence coefficient threshold value of the current training round, defining a pseudo label, and obtaining a maximum category probability of the pseudo label; calculating no-label loss according to the category score vector of the no-label after strong enhancement and the maximum category probability of the pseudo-label; training the MEWRN network in combination with the label loss and the label-free loss; systems, devices, and media for implementing the method; the accuracy of the classification model is improved by adopting the label-free data and the label data; a median-enhanced attention mechanism is introduced into the MEWRN, so that the extraction effect of image features is enhanced, and accurate classification is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of IVCM image classification, and in particular relates to a semi-supervised keratitis classification method, system, device and medium based on a median enhanced attention mechanism. Background Art

[0002] At present, in vivo confocal microscopy (IVCM) has become a common technique in the analysis and diagnosis of infectious keratitis. As a non-invasive imaging method, IVCM can provide high-definition corneal images and intuitively display the lesion area. However, the current processing of IVCM images of infectious keratitis mainly relies on fully supervised learning methods, which have high requirements on the quality and quantity of labeled data. Traditional feature extraction methods, such as local binary patterns combined with support vector machines, are highly dependent on human intervention, resulting in a long processing time, poor adaptability, and a single feature extraction. Although the introduction of convolutional neural networks (CNNs) has improved the above problems to a certain extent and reduced human intervention by automatically extracting features, most CNN methods still focus on extracting low-level information and have insufficient ability to distinguish complex IVCM images. Especially in the diagnosis of bacterial and fungal keratitis, these methods often find it difficult to extract key features with significant distinguishing capabilities, resulting in the need to improve the accuracy and reliability of diagnosis.

[0003] Lv et al. (Lv, Jian et al. "Deep learning-based automated diagnosis of fungal keratitis with in vivo confocal microscopy images." Annals of Translational Medicine 8 (2020): n.pag.) proposed an intelligent system for automatic diagnosis of fungal keratitis based on deep residual network (Deep Residual Learning for Image Recognition, ResNet). This method uses ResNet101 to classify IVCM images into two categories to detect whether it is fungal keratitis. However, due to the large depth of the network, its training and optimization process is relatively complicated. At the same time, the extracted features are still mainly concentrated on low-level information, which makes it difficult to fully characterize the characteristics of IVCM images. In addition, the small amount of annotated data also limits the generalization ability of the model, which makes the diagnosis results have certain limitations when dealing with more complex actual scenarios.

[0004] The current application of IVCM images still faces some challenges in actual diagnosis:

[0005] 1. Data annotation is difficult: IVCM images lack obvious specificity, which makes annotation difficult. At the same time, existing methods do not fully utilize unlabeled data, and the scarcity of training data significantly limits the performance and generalization ability of deep learning algorithms.

[0006] 2. Limited feature extraction capabilities: Although deep learning methods have made some progress in IVCM image analysis, the features extracted by the model are mostly low-level information, making it difficult to cope with complex pathological scenarios. Its ability to express the global and local pathological features of the differences between bacterial and fungal keratitis in IVCM images is limited, resulting in significant restrictions on the diagnostic performance and clinical application effects of the classification model. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a keratitis IVCM image classification method, system, device and medium based on semi-supervised learning. By adopting a semi-supervised learning method, a large amount of unlabeled data is fully utilized, combined with a small amount of labeled data, and the accuracy of the classification model is effectively improved while significantly reducing the demand for labeled samples. At the same time, by introducing a median-enhanced attention mechanism in the MEWRN network, the extraction effect of image features is enhanced, the ability to capture the characteristics of bacterial and fungal keratitis is provided, and more accurate classification is achieved.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism includes the following steps:

[0010] Step 1: Obtain IVCM images of keratitis to form a semi-supervised learning dataset and divide it into a training set and a test set; the semi-supervised learning dataset includes a labeled dataset and an unlabeled dataset; the labeled data are IVCM images that have been manually labeled; and the unlabeled data are IVCM images that have not been manually labeled;

[0011] Step 2: Use the MEWRN network to predict the category of the preprocessed labeled IVCM image and obtain the category score vector s of the labeled IVCM image i , and calculate its probability distribution p i , according to the probability distribution p i Calculate the cross entropy loss function and get the labeled loss

[0012] Step 3: Perform a weak enhancement operation on the preprocessed unlabeled IVCM image, and then use the MEWRN network to predict the category of the weakly enhanced unlabeled IVCM image to obtain the category score vector of the weakly enhanced unlabeled IVCM image. And calculate its probability distribution Select the category c with the largest probability as the predicted category of the weakly enhanced unlabeled IVCM image, and obtain the maximum category probability of the weakly enhanced unlabeled IVCM image

[0013] Step 4: According to the minimum value τ of the confidence threshold set min and the maximum value τ max , calculate the confidence threshold τ' of the current training round; when the maximum category probability in the weakly enhanced unlabeled IVCM image Higher than the confidence threshold τ' of the current training round, the weakly enhanced unlabeled IVCM image is defined as a pseudo-label;

[0014] Step 5: Perform a strong enhancement operation on the preprocessed unlabeled IVCM image, and then use the MEWRN network to predict the category of the strongly enhanced unlabeled IVCM image to obtain the category score vector s of the strongly enhanced unlabeled IVCM image j ; According to the category score vector s of the strongly enhanced unlabeled IVCM image j Maximum class probability in unlabeled IVCM image after weak enhancement defined as pseudo label Get the unlabeled loss

[0015] Step 6: According to the labeled loss and the unlabeled loss Calculate total loss The training set is input into the MEWRN network for iterative training to obtain a trained MEWRN network model;

[0016] Step 7: Use the trained MEWRN network model to predict the category of the labeled test set and obtain the classification results.

[0017] The preprocessing operations of the labeled IVCM image and the unlabeled IVCM image are both: dividing the IVCM image into m×n sub-regions, and performing normalization processing and contrast stretching on each sub-region in turn.

[0018] The MEWRN network includes a low-level feature extraction unit, a channel attention mechanism unit, a spatial attention mechanism unit and a classifier; the low-level feature extraction unit includes an initial convolution layer, three residual blocks, and a ReLU activation function; the convolution kernel size of the initial convolution layer is 3×3, the step size is 2, and the number of output channels is 16; each residual block includes two convolution layers with a convolution kernel size of 3×3, and the number of output channels of the last residual block is 128; the channel attention mechanism unit includes three pooling layers and a multi-layer perceptron; the three pooling layers are respectively a global maximum pooling layer, a global median pooling layer and a global average pooling layer; the multi-layer perceptron The device comprises two convolutional layers with a convolution kernel size of 1×1 and a Sigmoid activation function. The number of output channels of the first convolutional layer is 32, the dimensionality reduction ratio is 4, and the number of output channels of the second convolutional layer is 128; the spatial attention mechanism unit comprises an initial deep convolutional layer, six deep convolutional layers and a feature matching convolutional layer; the convolution kernel size of the initial deep convolutional layer is 5×5, and the number of output channels is 128; the convolution kernel sizes of the six deep convolutional layers are 1×3, 3×1, 1×7, 7×1, 1×13, and 13×1 respectively; the convolution kernel size of the feature matching convolutional layer is 1×1; the classifier is composed of a fully connected layer.

[0019] The specific operations of category prediction in step 2, step 3 and step 5 are:

[0020] Step (1): The IVCM image is sequentially input into the initial convolutional layer and three residual blocks, and then nonlinearly mapped through the ReLU activation function to obtain a feature map F with a size of 14×14;

[0021] Step (2): According to the median enhanced attention mechanism, the feature map with a size of 14×14 is subjected to global maximum pooling, global average pooling, and global median pooling operations respectively to obtain three pooled feature maps;

[0022] Step (3): Input each pooled feature map into a shared multi-layer perceptron, first increase the dimension of each pooled feature map and then decrease it, and then compress the output value to [0, 1] through the Sigmoid activation function to obtain three attention feature maps;

[0023] Step (4): Add the three attention feature maps element-wise to obtain the channel attention feature map F c :

[0024] Step (5): Transform the channel attention feature map F c Perform element-wise multiplication with the feature map F of size 14×14 to obtain the weighted channel attention feature map F′;

[0025] Step (6): Based on the spatial attention mechanism, the weighted channel attention feature map F′ is input into the initial deep convolutional layer for preliminary feature extraction; then it is input into six deep convolutional layers of different sizes to extract features in different directions and ranges; the output feature maps of the six deep convolutional layers of different sizes are element-wise added to obtain the spatial attention feature map F s ;

[0026] Step (7): Transform the spatial attention feature map F s Input feature matching convolution layer, multiply the output feature map with the weighted channel attention feature map F′ element-wise to obtain the median enhanced attention feature map F″;

[0027] Step (8): Perform a global average pooling operation on the median enhanced attention feature map F″, compress its height and width dimensions to 1×1, and obtain a one-dimensional feature; input the one-dimensional feature into the classifier to obtain the category score vector of the IVCM image.

[0028] The weak enhancement operation in step 3 is: performing random cropping, padding and random horizontal flipping operations on the preprocessed unlabeled IVCM image in sequence.

[0029] The calculation formula of the confidence threshold τ' of the current training round in step 4 is as follows:

[0030]

[0031] Among them, α represents the hyperparameter that controls the growth rate, epoch represents the current training round, and totalepoch represents the total training round.

[0032] The strong enhancement operation in step 5 is: performing random cropping, padding and random horizontal flipping operations on the preprocessed unlabeled IVCM image in sequence, then setting the enhancement degree, performing three random enhancement operations, and obtaining a strongly enhanced unlabeled IVCM image; the three random enhancement operations are any three of 14 enhancement processes, wherein the 14 enhancement processes include adjusting image brightness, adjusting image contrast, adjusting image saturation, adjusting image hue, image rotation, cropping, padding, image flipping, translation, scaling, perspective transformation, Gaussian blurring, adding noise, histogram equalization, and sharpening.

[0033] The present invention also provides a semi-supervised keratitis IVCM image classification system based on median enhanced attention mechanism, based on the above method, including:

[0034] Data acquisition and preprocessing module: used to acquire IVCM images of keratitis, form a semi-supervised learning dataset and divide it into a training set and a test set, and then preprocess it;

[0035] Labeled category prediction module: used to use the MEWRN network to perform category prediction on the preprocessed labeled IVCM image and obtain the category score vector s of the labeled IVCM image i , and calculate the probability distribution p i , according to the probability distribution p i Calculate the cross entropy loss function and get the labeled loss

[0036] Unlabeled category prediction module: It is used to perform weak enhancement and strong enhancement operations on the preprocessed unlabeled IVCM images, and then use the MEWRN network to perform category prediction on the weakly enhanced and strongly enhanced unlabeled IVCM images, respectively, to obtain the category score vector s of the strongly enhanced unlabeled IVCM image j And the category score vector of the unlabeled IVCM image after weak enhancement Class score vector Convert to probability distribution Select the category c with the largest probability as the predicted category of the weakly enhanced unlabeled IVCM image, and obtain the maximum category probability of the weakly enhanced unlabeled IVCM image According to the confidence threshold τ' of the current training round, the pseudo label of the unlabeled IVCM image after weak enhancement is defined; according to the category score vector s of the unlabeled IVCM image after strong enhancement j Maximum class probability in unlabeled IVCM image after weak enhancement defined as pseudo label Get the unlabeled loss

[0037] Network model training module: based on labeled loss and the unlabeled loss Calculate total loss The training set is input into the MEWRN network for iterative training to obtain a trained MEWRN network model;

[0038] Result category prediction module: Use the trained MEWRN network model to predict the category of the labeled test set and obtain the classification results.

[0039] The present invention also provides a semi-supervised keratitis IVCM image classification device based on median enhanced attention mechanism, comprising:

[0040] Memory: used to store a computer program for implementing the semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism;

[0041] Processor: used to implement the above-mentioned semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism when executing the computer program.

[0042] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned semi-supervised keratitis IVCM image classification method based on the median enhanced attention mechanism.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) Compared with the traditional fully supervised learning method, the improved FixMatch semi-supervised learning method of the present invention is to fully combine a large amount of easily accessible unlabeled data for training when the amount of labeled data is small. The MEWRN network model of the present invention not only reduces the dependence on manually labeled data and significantly reduces the data preparation cost, but also shows higher classification accuracy and robustness in the task of distinguishing bacterial from fungal keratitis.

[0045] (2) The present invention constructs a MEWRN network by introducing a median-enhanced channel attention mechanism into the WideResNet network and combining it with a multi-size convolutional spatial attention mechanism. The network effectively captures subtle feature differences in IVCM images through the channel attention mechanism. At the same time, the median enhancement operation significantly improves the adaptability of the network in noise interference scenarios, so that the model can still maintain efficient feature extraction capabilities under low-quality image input. The spatial attention mechanism extracts spatial information through multi-size convolution kernels, which can capture different scales of information from local details to larger range features, so that the spatial attention mechanism can focus on different spatial positions more accurately. In addition, the WideResNet network gives full play to the advantages of channel attention and spatial attention in global information processing, further improving the robustness and accuracy of classification tasks.

[0046] (3) Compared with the fixed confidence threshold in the traditional FixMatch framework, the improved FixMatch semi-supervised learning method of the present invention sets a dynamic confidence threshold, which reduces the confidence threshold of pseudo-label generation in the early stage of training to better utilize unlabeled data and increases the confidence threshold of pseudo-label generation in the later stage of training, thereby improving the quality of pseudo-labels and making IVCM image classification more accurate.

[0047] In summary, the present invention fully utilizes unlabeled data and reduces the workload of manual labeling by combining semi-supervised learning, median-enhanced attention mechanism and dynamic confidence threshold. At the same time, it significantly improves the generalization performance and classification effect of the MEWRN network model under the condition of limited labeled data, fully meeting the dual needs of efficient data utilization and diagnostic accuracy in actual clinical environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of the semi-supervised keratitis IVCM image classification method based on the median enhanced attention mechanism provided by the present invention.

[0049] Figure 2 A schematic diagram of the structure of the MEWRN network provided by the present invention.

[0050] Figure 3 The confusion matrix of the semi-supervised keratitis IVCM image classification based on the median enhanced attention mechanism provided by the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, a semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism includes the following steps:

[0053] Step 1: Obtain IVCM images of keratitis taken by IVCM equipment to form a semi-supervised learning data set; the semi-supervised learning data set includes a labeled data set and an unlabeled data set; the labeled data are IVCM images that have been manually labeled; the unlabeled data are IVCM images that have not been manually labeled; the semi-supervised learning data are divided to obtain a training set and a test set;

[0054] In this embodiment, the microbial culture results of keratitis patients are collected, and the IVCM images confirmed by microbial culture are screened and calibrated by doctors as labeled data sets, and the labeled data sets are divided into three categories of data sets: bacterial keratitis label (BK), fungal keratitis label (FK) and healthy cornea label (normal); at the same time, a large number of uncalibrated IVCM images are collected as unlabeled data sets; the initial size of the acquired IVCM image is a grayscale image of 384×484, and the IVCM image of the initial size is cropped to remove the 384×100 area at the bottom of the grayscale image, that is, the patient's personal information and IVCM equipment parameters, to obtain an IVCM image of size 384×384, which constitutes a semi-supervised learning data set, and the semi-supervised learning data set is divided to obtain a training set and a test set; the cropped labeled data set and the unlabeled data set are respectively stored in the labeled folder and the unlabeled folder, and the labeled data sets of the corresponding categories are respectively placed in the subfolders of the three categories of bacterial keratitis (BK), fungal keratitis (FK) and healthy cornea (normal) under the labeled folder.

[0055] Step 2: Preprocess the IVCM image in step 1;

[0056] The IVCM image is divided into m×n sub-regions, and each sub-region is normalized and contrast stretched in turn; specifically:

[0057] Step 2.1: Read the semi-supervised learning dataset in PILImage format and convert the original PILImage format into a tensor type that can be quickly processed by pytorch;

[0058] Step 2.2: Divide the converted IVCM image evenly into 6×6 rectangular sub-regions, each sub-region is 64×64 in size;

[0059] Step 2.3: Normalize each sub-region and map the pixel value to the interval [0,1], the formula is as follows:

[0060]

[0061] Among them, I ij (x,y) represents the pixel value of the pixel point. Indicates the pixel value of the normalized pixel;

[0062] Step 2.4: Perform contrast stretching on the normalized pixels of each sub-region, that is, set the pixels whose pixel values ​​of the IVCM image are lower than half of the average value to zero, and magnify the pixel values ​​of the remaining pixels by 1.5 times to highlight the texture features in the IVCM image, thereby enhancing the expression of the lesion area; the formula is as follows:

[0063]

[0064] in, Represents the pixel value after contrast stretching of each sub-region, Represents the pixel value of each sub-region before contrast stretching, Represents the normalized average pixel value of each sub-region;

[0065] Step 2.5: The stretched IVCM image size is adjusted from the original 384×384 to 224×224 to reduce the computational complexity.

[0066] Step 3: Use the MEWRN network to predict the category of the preprocessed labeled IVCM image and obtain the category score vector s of the labeled IVCM image i ; and use the Softmax function to calculate its probability distribution p i , according to the probability distribution p i Calculate the cross entropy loss function and get the labeled loss

[0067] The probability distribution p iThe calculation formula is as follows:

[0068]

[0069] Among them, p i Represents the probability distribution of the i-th sample p i =(p i,1 ,…,p i,c ,…,p i,C ), where p i,c It represents the probability that the i-th sample is predicted to be the c-th category, where C is the number of categories.

[0070] The label loss The calculation formula is as follows:

[0071]

[0072] Among them, N represents the number of labeled samples, C represents the number of categories, and y i,c Represents an indicator function. When the true label of the i-th sample is the c-th class, y i,c =1, otherwise y i,c =0.

[0073] like Figure 2 As shown, the MEWRN network includes a low-level feature extraction unit, a channel attention mechanism unit, a spatial attention mechanism unit and a classifier; the low-level feature extraction unit includes an initial convolution layer, three residual blocks, and a ReLU activation function; the convolution kernel size of the initial convolution layer is 3×3, the step size is 2, and the number of output channels is 16; each residual block includes two convolution layers with a convolution kernel size of 3×3, and the number of output channels of the last residual block is 128; the channel attention mechanism unit includes three pooling layers and a multi-layer perceptron; the three pooling layers are respectively a global maximum pooling layer (MaxPool), a global median pooling layer (MedianPool) and a global average pooling layer (AvgPool); the multilayer perceptron (MLP) includes two convolutional layers with a convolution kernel size of 1×1 and a Sigmoid activation function. The number of output channels of the first convolutional layer is 32, the dimensionality reduction ratio is 4, and the number of output channels of the second convolutional layer is 128; the spatial attention mechanism unit includes an initial deep convolutional layer, six deep convolutional layers and a feature matching convolutional layer; the convolution kernel size of the initial deep convolutional layer is 5×5, and the number of output channels is 128; the convolution kernel sizes of the six deep convolutional layers are 1×3, 3×1, 1×7, 7×1, 1×13, and 13×1 respectively; the convolution kernel size of the feature matching convolutional layer is 1×1; the classifier is composed of a fully connected layer.

[0074] The specific operation of the category prediction of the preprocessed labeled IVCM image is as follows:

[0075] Step 3.1: The preprocessed labeled IVCM image is input into the initial convolutional layer and three residual blocks in sequence, and then nonlinear mapping is performed through the ReLU activation function to enhance the expression ability of the network, and finally a feature map F with a size of 14×14 is obtained;

[0076] Step 3.2: According to the Median-enhanced Attention Mechanism (MFAM), the feature map of size 14×14 in step 3.1 is subjected to global maximum pooling, global average pooling, and global median pooling operations respectively to obtain three pooled feature maps;

[0077] Step 3.3: Input each pooled feature map into a shared multi-layer perceptron (MLP), first increase the dimension of each pooled feature map and then decrease it, and then compress the output value to [0,1] through the Sigmoid activation function to obtain three attention feature maps;

[0078] Step 3.4: Add the three attention feature maps element-wise to get the channel attention feature map F c , the formula is as follows:

[0079] F c =σ(MLP(AvgPool(F)))+σ(MLP(MaxPool(F)))+σ(MLP(MedianPool(F)))

[0080] Among them, σ represents the Sigmoid activation function;

[0081] Step 3.5: Channel attention feature map F c Multiply the feature map F of size 14×14 in step 3.1 by element level to obtain the weighted channel attention feature map F′, as follows:

[0082] F′=F c ⊙F

[0083] Among them, ⊙ represents element-wise multiplication;

[0084] Step 3.6: Based on the spatial attention mechanism, the weighted channel attention feature map F′ is input into the initial deep convolutional layer for preliminary feature extraction; then it is input into six deep convolutional layers of different sizes to extract features in different directions and ranges; the output feature maps of the six deep convolutional layers of different sizes are element-wise added to obtain the spatial attention feature map F s , the formula is as follows:

[0085]

[0086] Among them, D i (F') represents depthwise convolution operations of different sizes, and n represents the number of convolutions;

[0087] Step 3.7: Transform the spatial attention feature map F s Input the feature matching convolution layer, multiply the output feature map by the weighted channel attention feature map F′ element-wise to obtain the median enhanced attention feature map F″. The formula is as follows:

[0088] F″=Conv1x1(F s )⊙F′

[0089] Among them, Conv1x1 represents the convolution operation with a convolution kernel of 1×1;

[0090] Step 3.8: Perform a global average pooling operation on the median enhanced attention feature map F″, compress its height and width dimensions to 1×1, and obtain a one-dimensional feature; input the one-dimensional feature into the classifier to obtain the category score vector s of the labeled IVCM image i .

[0091] Step 4: Perform a weak enhancement operation on the unlabeled IVCM image preprocessed in step 2, and then use the MEWRN network to predict the category of the weakly enhanced unlabeled IVCM image. Repeat steps 3.1 to 3.8 to obtain the category score vector of the weakly enhanced unlabeled IVCM image. Calculate probability distribution using Softmax function And select the category c with the largest probability as the predicted category of the weakly enhanced unlabeled IVCM image, and get the maximum category probability of the weakly enhanced unlabeled IVCM image

[0092] The weak enhancement operation is: performing random cropping, padding and random horizontal flipping operations on the unlabeled IVCM image preprocessed in step 2 in sequence; specifically:

[0093] The preprocessed unlabeled IVCM image is cropped from 224×224 to 196×196, and the cropped area is filled with image boundary pixels; this operation retains the main content of the data, provides a slight change to the model, meets the requirements of weak enhancement, and improves the model's adaptability to directional changes without significantly affecting the semantic content of the data.

[0094] Step 5: Set the minimum confidence threshold τ min and the maximum value τ max , calculate the confidence threshold τ' of the current round; when the maximum category probability in the weakly enhanced unlabeled IVCM image When the confidence threshold τ' is higher than the current training round, the weakly enhanced unlabeled IVCM image is defined as a pseudo-label. By screening out low-confidence predictions, the incorrect pseudo-labels can be prevented from affecting the model training.

[0095] Dynamically calculate the confidence threshold according to the current training round and set the minimum confidence threshold τ min is 90%, the maximum confidence threshold τ max is 95%; as the current training round epoch increases, the confidence threshold will be gradually increased in a nonlinear manner, so as to gradually increase the requirements for the quality of pseudo labels during the training process; the calculation formula of the confidence threshold τ' of the current training round is as follows:

[0096]

[0097] Among them, α represents the hyperparameter that controls the growth rate. A larger α can make the threshold approach the maximum confidence threshold τ faster. max ;totalepoch represents the total number of training rounds.

[0098] Step 6: Perform a strong enhancement operation on the unlabeled IVCM image preprocessed in step 2, and then use the MEWRN network to predict the category of the strongly enhanced unlabeled IVCM image. Repeat steps 3.1 to 3.8 to obtain the category score vector s of the strongly enhanced unlabeled IVCM image. j ; According to the category score vector s of the strongly enhanced unlabeled IVCM image j , use the Softmax function to calculate the probability distribution p j , according to the probability distribution p j The maximum class probability in the unlabeled IVCM image after weak enhancement defined as the pseudo label in step 5 Calculate the unlabeled loss Maintain consistent predictions for the same image under different augmentation conditions, improving generalization capabilities on unlabeled data;

[0099] The unlabeled loss The calculation formula is as follows:

[0100]

[0101] Where M represents the number of unlabeled samples. Represents a confidence filtering operation.

[0102] The strong enhancement operation is specifically as follows: the unlabeled IVCM image preprocessed in step 2 is subjected to random cropping, filling and random horizontal flipping operations in sequence, and then the enhancement degree is set to perform three random enhancement operations to obtain a strongly enhanced unlabeled IVCM image; the enhancement degree is set to 5, indicating a medium enhancement effect; the three random enhancement operations are any three of 14 enhancement processes, wherein the 14 enhancement processes include adjusting the brightness of the image, adjusting the contrast of the image, adjusting the saturation of the image, adjusting the hue of the image, image rotation, cropping, filling, image flipping, translation, scaling, perspective transformation, Gaussian blurring, adding noise, histogram equalization, sharpening, etc.

[0103] Step 7: According to the labeled loss and the unlabeled loss Calculate total loss The training set is input into the MEWRN network according to the total loss Perform iterative training to obtain a trained MEWRN network model;

[0104] According to step 3, there is label loss The unlabeled loss in step 6 Calculate total loss The formula is as follows:

[0105]

[0106] Here, λ represents a hyperparameter. In this embodiment, the parameter is set to 1, so that unlabeled data can also be effectively utilized, and the network weights are continuously optimized through back propagation, thereby improving the overall performance of the model.

[0107] Step 8: Use the trained MEWRN network model to predict the category of the labeled test set and obtain the test results, such as Figure 3 As shown, it is possible to achieve classification diagnosis of keratitis IVCM images and complete accurate identification of bacterial and fungal keratitis.

[0108] Figure 3 is the confusion matrix of semi-supervised keratitis IVCM image classification based on median enhanced attention mechanism, which is Figure 3 It can be seen that the probability of correctly predicting the corresponding category by the classification method of the present invention reaches 90%, indicating that the classification accuracy of the classification method of the present invention is relatively high.

[0109] The present invention also provides a semi-supervised keratitis IVCM image classification system based on median enhanced attention mechanism, comprising:

[0110] Data acquisition and preprocessing module: used to acquire IVCM images of keratitis, form a semi-supervised learning dataset and divide it into a training set and a test set, and then preprocess it;

[0111] Labeled category prediction module: used to use the MEWRN network to perform category prediction on the preprocessed labeled IVCM image and obtain the category score vector s of the labeled IVCM image i , and calculate the probability distribution p i , according to the probability distribution p i Calculate the cross entropy loss function and get the labeled loss

[0112] Unlabeled category prediction module: It is used to perform weak enhancement and strong enhancement operations on the preprocessed unlabeled IVCM images, and then use the MEWRN network to perform category prediction on the weakly enhanced and strongly enhanced unlabeled IVCM images, respectively, to obtain the category score vector s of the strongly enhanced unlabeled IVCM image j And the category score vector of the unlabeled IVCM image after weak enhancement Class score vector Convert to probability distribution Select the category c with the largest probability as the predicted category of the weakly enhanced unlabeled IVCM image, and obtain the maximum category probability of the weakly enhanced unlabeled IVCM image According to the confidence threshold τ' of the current training round, the pseudo label of the unlabeled IVCM image after weak enhancement is defined; according to the category score vector s of the unlabeled IVCM image after strong enhancement j Maximum class probability in unlabeled IVCM image after weak enhancement defined as pseudo label Get the unlabeled loss

[0113] Network model training module: based on labeled loss and the unlabeled loss Calculate total loss The training set is input into the MEWRN network for iterative training to obtain a trained MEWRN network model;

[0114] Result category prediction module: Use the trained MEWRN network model to predict the category of the labeled test set and obtain the classification results.

[0115] The present invention also provides a semi-supervised keratitis IVCM image classification device based on median enhanced attention mechanism, comprising:

[0116] Memory: used to store a computer program for implementing the semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism;

[0117] Processor: used to implement the above-mentioned semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism when executing the computer program.

[0118] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned semi-supervised keratitis IVCM image classification method based on the median enhanced attention mechanism.

[0119] Simulation experiment

[0120] 1. Experimental conditions

[0121] The central processing unit used in the present invention is 16vCPU Intel(R)Xeon(R)Platinum 8481C, the graphics card is NVIDIA RTX4090D (24G), the memory is 80G, the operating system is ubuntu20.04, and the experiment is carried out using python software.

[0122] 2. Experimental content

[0123] Comparative Example 1: Only labeled datasets were used and the existing WideResNet28 network model was used to conduct supervised keratitis IVCM image classification experiments.

[0124] Comparative Example 2: Only labeled datasets were used and the MEWRN network model was used to conduct a supervised keratitis IVCM image classification experiment.

[0125] Comparative Example 3: A mixed dataset of labeled data and unlabeled data was used, and the existing FixMatch semi-supervised method and WideResNet28 network model were used to conduct a keratitis IVCM image classification experiment.

[0126] Comparative Example 4: A mixed dataset of labeled data and unlabeled data was used, and the FixMatch semi-supervised method and MEWRN network model were used to conduct a keratitis IVCM image classification experiment.

[0127] Example 5: A mixed data set of labeled data and unlabeled data was used to conduct a keratitis IVCM image classification experiment using the improved FixMatch semi-supervised method and the MEWRN network model of the present invention.

[0128] Table 1 shows the test indicators of the above five experimental results, among which, As shown in Table 1, compared with the supervised method based on the WideResNet28 network, the classification effect of the present invention is significantly improved, especially after the use of unlabeled data, the classification effect is significantly improved. Therefore, the method of the present invention can achieve more accurate keratitis IVCM image classification when there is insufficient labeled data and a large amount of unlabeled data can be obtained.

[0129] Table 1 shows the test indicators of the five experimental results.

[0130] experiment Accuracy (acc) Precision Recall F1 Comparative Example 1 0.8507 0.8735 0.8566 0.8627 Comparative Example 2 0.8640 0.8668 0.8698 0.8680 Comparative Example 3 0.9057 0.9080 0.8846 0.8935 Comparative Example 4 0.9149 0.9095 0.9019 0.9053 Example 5 0.9287 0.9188 0.9215 0.9201

Claims

1. A semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism, characterized in that: The steps include: Step 1: Obtain IVCM images of keratitis to form a semi-supervised learning dataset and divide it into a training set and a test set; the semi-supervised learning dataset includes a labeled dataset and an unlabeled dataset; the labeled data are IVCM images that have been manually labeled; and the unlabeled data are IVCM images that have not been manually labeled; Step 2: Use the MEWRN network to predict the category of the preprocessed labeled IVCM image and obtain the category score vector s of the labeled IVCM image i , and calculate its probability distribution p i , according to the probability distribution p i Calculate the cross entropy loss function and get the labeled loss Step 3: Perform a weak enhancement operation on the preprocessed unlabeled IVCM image, and then use the MEWRN network to predict the category of the weakly enhanced unlabeled IVCM image to obtain the category score vector of the weakly enhanced unlabeled IVCM image. And calculate its probability distribution Select the category c with the largest probability as the predicted category of the weakly enhanced unlabeled IVCM image, and obtain the maximum category probability of the weakly enhanced unlabeled IVCM image Step 4: According to the minimum value τ of the confidence threshold set min and the maximum value τ max , calculate the confidence threshold τ' of the current training round; When the maximum class probability of the unlabeled IVCM image after weak enhancement Higher than the confidence threshold τ' of the current training round, the weakly enhanced unlabeled IVCM image is defined as a pseudo-label; Step 5: Perform a strong enhancement operation on the preprocessed unlabeled IVCM image, and then use the MEWRN network to predict the category of the strongly enhanced unlabeled IVCM image to obtain the category score vector s of the strongly enhanced unlabeled IVCM image j ; According to the category score vector s of the strongly enhanced unlabeled IVCM image j The maximum class probability of the unlabeled IVCM image after weak enhancement defined as the pseudo label Get the unlabeled loss Step 6: According to the labeled loss and the unlabeled loss Calculate total loss The training set is input into the MEWRN network for iterative training to obtain a trained MEWRN network model; Step 7: Use the trained MEWRN network model to predict the category of the labeled test set and obtain the classification results.

2. According to claim 1, a semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism is characterized in that: The preprocessing operations of the labeled IVCM image and the unlabeled IVCM image are both: dividing the IVCM image into m×n sub-regions, and performing normalization processing and contrast stretching on each sub-region in turn.

3. According to the semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism according to claim 1, it is characterized in that: The MEWRN network includes a low-level feature extraction unit, a channel attention mechanism unit, a spatial attention mechanism unit and a classifier; the low-level feature extraction unit includes an initial convolution layer, three residual blocks, and a ReLU activation function; the convolution kernel size of the initial convolution layer is 3×3, the step size is 2, and the number of output channels is 16; each residual block includes two convolution layers with a convolution kernel size of 3×3, and the number of output channels of the last residual block is 128; the channel attention mechanism unit includes three pooling layers and a multi-layer perceptron; the three pooling layers are respectively a global maximum pooling layer, a global median pooling layer and a global average pooling layer; the multi-layer perceptron The device comprises two convolutional layers with a convolution kernel size of 1×1 and a Sigmoid activation function. The number of output channels of the first convolutional layer is 32, the dimensionality reduction ratio is 4, and the number of output channels of the second convolutional layer is 128; the spatial attention mechanism unit comprises an initial deep convolutional layer, six deep convolutional layers and a feature matching convolutional layer; the convolution kernel size of the initial deep convolutional layer is 5×5, and the number of output channels is 128; the convolution kernel sizes of the six deep convolutional layers are 1×3, 3×1, 1×7, 7×1, 1×13, and 13×1 respectively; the convolution kernel size of the feature matching convolutional layer is 1×1; the classifier is composed of a fully connected layer.

4. According to claim 1, a semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism is characterized in that: The specific operations of category prediction in step 2, step 3 and step 5 are: Step (1): The IVCM image is sequentially input into the initial convolutional layer and three residual blocks, and then nonlinearly mapped through the ReLU activation function to obtain a feature map F with a size of 14×14; Step (2): According to the median enhanced attention mechanism, the feature map with a size of 14×14 is subjected to global maximum pooling, global average pooling, and global median pooling operations respectively to obtain three pooled feature maps; Step (3): Input each pooled feature map into a shared multi-layer perceptron, first increase the dimension of each pooled feature map and then decrease it, and then compress the output value to [0, 1] through the Sigmoid activation function to obtain three attention feature maps; Step (4): Add the three attention feature maps element-wise to obtain the channel attention feature map F c : Step (5): Transform the channel attention feature map F c Perform element-wise multiplication with the feature map F of size 14×14 to obtain the weighted channel attention feature map F′; Step (6): Based on the spatial attention mechanism, the weighted channel attention feature map F′ is input into the initial deep convolutional layer for preliminary feature extraction; then it is input into six deep convolutional layers of different sizes to extract features in different directions and ranges; The output feature maps of the six deep convolutional layers of different sizes are element-wise added to obtain the spatial attention feature map F s ; Step (7): Transform the spatial attention feature map F s Input feature matching convolution layer, multiply the output feature map with the weighted channel attention feature map F′ element-wise to obtain the median enhanced attention feature map F″; Step (8): Perform a global average pooling operation on the median enhanced attention feature map F″, compress its height and width dimensions to 1×1, and obtain a one-dimensional feature; input the one-dimensional feature into the classifier to obtain the category score vector of the IVCM image.

5. According to claim 1, a semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism is characterized in that: The weak enhancement operation in step 3 is: performing random cropping, padding and random horizontal flipping operations on the preprocessed unlabeled IVCM image in sequence.

6. A semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism according to claim 1, characterized in that: The calculation formula of the confidence threshold τ' of the current training round in step 4 is as follows: Among them, α represents the hyperparameter that controls the growth rate, epoch represents the current training round, and totalepoch represents the total training round.

7. A semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism according to claim 1, characterized in that: The strong enhancement operation in step 5 is: performing random cropping, padding and random horizontal flipping operations on the preprocessed unlabeled IVCM image in sequence, then setting the enhancement degree, performing three random enhancement operations, and obtaining a strongly enhanced unlabeled IVCM image; the three random enhancement operations are any three of 14 enhancement processes, wherein the 14 enhancement processes include adjusting image brightness, adjusting image contrast, adjusting image saturation, adjusting image hue, image rotation, cropping, padding, image flipping, translation, scaling, perspective transformation, Gaussian blurring, adding noise, histogram equalization, and sharpening.

8. A semi-supervised keratitis IVCM image classification system based on median enhanced attention mechanism, characterized in that: The method according to any one of claims 1 to 7 comprises: Data acquisition and preprocessing module: used to acquire IVCM images of keratitis, form a semi-supervised learning dataset and divide it into a training set and a test set, and then preprocess it; Labeled category prediction module: used to use the MEWRN network to perform category prediction on the preprocessed labeled IVCM image and obtain the category score vector s of the labeled IVCM image i , and calculate the probability distribution p i , according to the probability distribution p i Calculate the cross entropy loss function and get the labeled loss Unlabeled category prediction module: It is used to perform weak enhancement and strong enhancement operations on the preprocessed unlabeled IVCM images, and then use the MEWRN network to perform category prediction on the weakly enhanced and strongly enhanced unlabeled IVCM images, respectively, to obtain the category score vector s of the strongly enhanced unlabeled IVCM image j And the category score vector of the unlabeled IVCM image after weak enhancement Class score vector Convert to probability distribution Select the category c with the largest probability as the predicted category of the weakly enhanced unlabeled IVCM image, and obtain the maximum category probability of the weakly enhanced unlabeled IVCM image According to the confidence threshold τ' of the current training round, the pseudo label of the unlabeled IVCM image after weak enhancement is defined; according to the category score vector s of the unlabeled IVCM image after strong enhancement j Maximum class probability in unlabeled IVCM image after weak enhancement defined as pseudo label Get the unlabeled loss Network model training module: based on labeled loss and the unlabeled loss Calculate total loss The training set is input into the MEWRN network for iterative training to obtain a trained MEWRN network model; Result category prediction module: Use the trained MEWRN network model to predict the category of the labeled test set and obtain the classification results.

9. A semi-supervised keratitis IVCM image classification device based on median enhanced attention mechanism, characterized in that: include: Memory: used to store a computer program for implementing a semi-supervised keratitis IVCM image classification method based on a median enhanced attention mechanism according to claims 1-7; Processor: used to implement the semi-supervised keratitis IVCM image classification method based on median enhanced attention mechanism of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement claims 1-7, a semi-supervised keratitis IVCM image classification method based on a median enhanced attention mechanism.

Images