Bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning

Abstract

The application provides a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning, and comprises the following steps: a slit lamp image dataset containing bacterial keratitis and fungal keratitis is constructed, and is processed into a proper format; the input image dataset is preprocessed; the preprocessed image dataset is input into a feature extraction network to extract features of different granularities; the features of different granularities are put into a contrast selection attention module for feature attention enhancement; the enhanced features of different granularities are predicted, and a progressive knowledge regularization module is combined to train a model; and the trained network model is used to classify bacterial keratitis and fungal keratitis images. The application imitates the diagnosis thought of ophthalmic experts from details to the whole and from seeing the small to knowing the big, and improves the accuracy and interpretability of automatic diagnosis.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and recognition, and more particularly relates to a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning. BACKGROUND

[0002] Infectious keratitis is a common and destructive ophthalmic disease. It is of great significance to diagnose the type of infectious keratitis in time and effectively and to take the correct drug selection for the treatment of the disease. Clinically, due to the long time required for the diagnosis of infectious keratitis by means of microbial smear and culture and the low detection rate, most ophthalmologists currently make initial diagnosis results by observing the corneal pathological pictures taken by a slit lamp microscope. However, the visual differences between different types of infectious keratitis are not obvious, and especially the indistinguishable nature between bacterial keratitis and fungal keratitis often leads to the difficulty of this diagnosis method relying on the experience and knowledge of ophthalmologists to guarantee a high accuracy. The use of computer image processing technology to achieve automatic diagnosis of bacterial keratitis and fungal keratitis can improve the efficiency while maintaining a high diagnostic accuracy.

[0003] In recent years, with the emphasis on deep learning in the field of intelligent medicine, some domestic and foreign scholars have begun to use image processing technology to automatically diagnose infectious keratitis. These studies can be broadly classified into two categories. One is based on strong supervision learning, such as the article "Deep sequential feature learning in clinical image classification of infectious keratitis" by Xu et al. (Engineering, 2021, 7(7): 1002-1010), which requires prior annotation of image lesion centers. Then, a deep model is used to classify four types of infectious keratitis (bacterial, fungal, herpes simplex virus, and others) on 2284 corneal images from 867 patients. The authors first serialized the image features, sorted them spatially, and sequentially fed them into the long short-term memory network (LSTM). Finally, the classification results showed that the proposed deep sequential model outperformed three classic neural network architectures. This type of study requires ophthalmologists to manually annotate the relevant lesion area, which is not only time-consuming and labor-intensive, but also prone to errors. At the same time, the second type of weakly supervised learning based on only using class labels has attracted attention. The article "Deep learning for discrimination between fungal keratitis and bacterial keratitis: DeepKeratitis" by Ghosh et al. (Cornea, 2022, 41(5): 616) classified bacterial and fungal keratitis on 2167 images from 194 patients using a pre-trained deep model. However, this method simply puts the images directly into the neural network for training and learning, without considering the characteristics of the disease images. The article "Concept-based explanation for fine-grained images and its application in infectious keratitis classification" by Fang et al. (Proceedings of the 28th ACM international conference on Multimedia, 2020: 700-708) improved the performance of traditional deep models by combining the features of the visual concepts they mined. This method achieved satisfactory classification results, but the visual concepts mined may not be absolutely correct, lacking explainability.

[0004] Although the deep learning-based infectious keratitis image recognition has achieved good classification performance, due to the insufficient data, large intra-class difference and small inter-class difference of different categories of keratitis images, and the interference of irrelevant factors in the images, the current method has certain deficiencies, and therefore a new method needs to be designed to realize high-precision infectious keratitis, especially bacterial keratitis and fungal keratitis image classification. SUMMARY

[0005] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning, to imitate the diagnosis thinking of ophthalmologists from details to the whole and see the microcosm in the macrocosm, improve the accuracy and interpretability of automatic diagnosis, and achieve the purpose of reducing the misdiagnosis rate of doctors.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning, comprising the following steps:

[0008] Obtain a slit lamp image dataset containing bacterial keratitis and fungal keratitis, and pre-process the slit lamp image dataset;

[0009] Construct an image classification model, the image classification model comprising a feature extraction module, a contrast selection attention module, a feature adjustment module, a classifier module and a progressive knowledge regularization module; the feature extraction module is used to extract features of different granularities of the pre-processed slit lamp image data; the contrast selection attention module selectively enhances part of the feature channels by comparing the feature differences between bacterial and fungal; the feature adjustment module is used to process the enhanced features to the same dimension; the classifier module is based on different granularity features for classification prediction;

[0010] Train the image classification model based on progressive knowledge regularization, update the image classification model based on the loss function until convergence; the loss function includes a multi-granularity prediction loss And a progressive knowledge regularization loss The multi-granularity prediction loss is used to correct the classification effect of the model; by measuring the difference between the real class label and the model prediction, the cross-entropy loss is calculated based on the prediction results of different granularity features and the image label; the progressive knowledge regularization loss is used to enhance the causal relationship between the simple features generated by the shallow layer of the network model and the advanced semantic features generated by the deep layer;

[0011] Classify bacterial or fungal keratitis images using the trained network model.

[0012] As a preferred technical solution, the preprocessing is specifically:

[0013] Crop the images in the slit lamp image dataset to a set size, randomly horizontally and vertically flip the cropped images, and map the image data to between 0 and 1 using image standardization operations;

[0014] Separate the three primary color channels of the image after the standardization operation, apply the limited contrast adaptive histogram equalization method to the B channel, and then fuse the three channels.

[0015] Perform CutMix augmentation on the three-channel fused image.

[0016] As a preferred technical solution, the feature extraction module uses a ResNet50 network, which is composed of 5 stages, each stage containing convolution and pooling operations, and different convolution operations have different perception domains to extract feature maps at different stages.

[0017] As a preferred technical solution, the contrast selection attention module divides the feature maps output by the feature extraction module into two categories according to the class label, i.e., the bacterial feature set F BK and the fungal feature set F FK , then calculates the average value of each to represent the average feature of the category, specifically as follows:

[0018]

[0019] Where, C, H, and W are the size of the image's channel, width, and height, respectively, j ∈ {BK, FK}, BK and FK are bacterial keratitis images and fungal keratitis images, respectively, and m is the number of BK or FK samples in the training batch.

[0020] Adjust the average class feature size to Express the similarity between the two average class features in each channel with a similarity vector, and convert it to a channel weight through SoftMax, as shown in the following formula:

[0021]

[0022]

[0023] Where, z ∈ [0, 1] C There is a weight in each channel, sim is the cosine similarity operation between feature channels, and z iThe similarity value of the channel subscript index i;

[0024] Since the smaller the similarity is, the more the visual feature represented by the channel has the characteristics of distinguishing between two categories, in order to make such a channel obtain a greater weight, z is processed as follows:

[0025] z = 1 - z

[0026] The values of the first C / 2 smallest values in the channel weight are selected and set to 0, that is, the channels without discriminative characteristics in the feature do not have an effect on feature enhancement, and z after selection is denoted as z select , and then each channel of the feature is multiplied by a weight in the form of dot product, and finally the enhanced weight value is added to the original feature value to obtain the enhanced feature F', which is represented by the following formula:

[0027] F' i = F i + (1 - z select )·F i

[0028] Where i is the subscript of the different granularities extracted;

[0029] The feature adjustment module processes the enhanced feature into the same dimension, and the following formula is used:

[0030] V i = ReLU(BN(Conv3(ReLU(BN(Conv1(F' i ))))

[0031] Where Conv3 and Conv1 are convolution operations with a kernel size of 3x3 and 1x1, respectively, and a step size of 1, BN is batch normalization, ReLU is ReLU activation operation, and i∈{1,2,3} represents the feature index of different granularities.

[0032] As a preferred technical solution, the features processed by the feature adjustment module are spliced to obtain a comprehensive feature, and the multi-granularity features and the spliced comprehensive feature are input into the classifier module, respectively, to obtain prediction results based on different granularity features, and the classifier module is composed of two fully connected layers, BatchNorm and ELU operation.

[0033] As a preferred technical solution, in the step of training the image classification model based on progressive knowledge regularization, the multi-granularity prediction loss and the progressive knowledge regularization loss are as follows:

[0034]

[0035]

[0036]

[0037] where y is the sample class label, taking value 1 if belonging to the class, otherwise taking value 0, is the probability of the model's prediction result on the c-th class based on the i-th stage feature, and λ i is the weight parameter of each prediction result, and T is the temperature coefficient, which can change the attention degree to negative samples in the training process;

[0038] Progressive knowledge regularization loss To enhance the causal relationship between the simple features generated by the shallow layers of the network model and the advanced semantic features generated by the deep layers, the robustness of the model as a whole is improved by using the prediction results of the deep layers to guide the prediction of the shallow layers, and the formula is as follows:

[0039]

[0040] Finally, the parameters are adjusted by backpropagation when training the image classification model, wherein α and β are the coefficients for balancing each loss.

[0041] As a preferred technical solution, if the current sample has undergone CutMix operation during preprocessing, then The loss is adjusted as follows:

[0042]

[0043] where y A and y B are the true labels of the two images fused by CutMix, and θ is the ratio of the size of the replacement area to the original image.

[0044] In a second aspect, the present application provides a bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning, which is applied to the bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning, and comprises a data acquisition module, a model construction module, a model training module and an image classification module.

[0045] The data acquisition module is used to acquire a slit lamp image dataset containing bacterial keratitis and fungal keratitis, and to pre-process the slit lamp image dataset.

[0046] ​The model construction module is configured to construct an image classification model, the image classification model comprising a feature extraction module, a contrast selection attention module, a feature adjustment module, a classifier module, and a progressive knowledge regularization module; the feature extraction module is configured to extract features of different granularities of a preprocessed slit lamp image dataset; the contrast selection attention module is configured to selectively enhance some feature channels by contrasting feature differences between bacterial and fungal characteristics; the feature adjustment module is configured to process the enhanced features into the same dimension; and the classifier module is configured to make classification predictions based on different granularity features.

[0047] The model training module is configured to train the image classification model based on progressive knowledge regularization, update the image classification model based on a loss function until convergence, and the loss function comprises a multi-granularity prediction loss and a progressive knowledge regularization loss The multi-granularity prediction loss is used to correct the classification effect of the model; by measuring the difference between the real class label and the model prediction, calculating the cross-entropy loss based on the prediction results of different granularity features and the image label; and the progressive knowledge regularization loss is used to enhance the causal relationship between simple features generated by the shallow layer of the network model and advanced semantic features generated by the deep layer of the network model.

[0048] The image classification module uses the trained network model to classify bacterial or fungal keratitis images.

[0049] In a third aspect, the present application provides an electronic device, which comprises:

[0050] at least one processor; and

[0051] a memory in communication connection with the at least one processor; wherein

[0052] the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning.

[0053] In a fourth aspect, the present application provides a computer readable storage medium storing a program, and the program is executed by a processor to implement the bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning.

[0054] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0055] (1), the application provides a kind of bacterial and fungal keratitis classification method based on multi-granularity feature learning, by imitating the diagnosis thought of ophthalmologist from detail to whole, seeing microcosm know famous, improve the accuracy and interpretability of automatic diagnosis, thereby effectively reduce the misdiagnosis rate of doctor.

[0056] (2), the application is designed for bacterial and fungal keratitis image difficult to distinguish and the limitation of small amount of data, a unique image preprocessing module is designed, which can effectively augment data to highlight the details of the lesion area and reduce the overfitting of the method.

[0057] (3), the application proposes to enhance different granularity features by contrast selection attention mechanism, from the perspective of contrast of two kinds of keratitis features, selectively enhance the features specific to two diseases, thereby enhancing the network model to extract features and classification ability of the region.

[0058] (4), the application proposes a progressive knowledge regularization module to improve the overall robustness of the feature extraction network and enhance the causal relationship between different granularity features generated at different stages of the network. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] Figure 1 The application proposes a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning.

[0061] Figure 2 The network structure diagram in the embodiment of the present application.

[0062] Figure 3 The structure diagram of ResNet50 in the embodiment of the present application.

[0063] Figure 4 The structure diagram of the contrast selection attention module in the embodiment of the present application.

[0064] Figure 5 The structure diagram of the feature adjustment module in the embodiment of the present application.

[0065] Figure 6 The structure diagram of the classifier module in the embodiment of the present application.

[0066] Figure 7A structural schematic diagram of a bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning in an embodiment of the present application is shown.

[0067] Figure 8 A structural schematic diagram of a storage medium in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] In order for those skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments.

[0070] As shown in Figure 1 The present application is a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning. First, a bacterial and fungal keratitis dataset is constructed and divided into a training set and a test set. Second, the training set images are preprocessed and input into a neural network. Then, different granularity feature maps extracted from different stages of the network are enhanced and the network is trained based on this combined with progressive knowledge regularization. Finally, the test set images or new bacterial or fungal keratitis images are input into the network for classification and diagnosis.

[0071] To further elaborate the technical solutions of the present application, the main technical points are described as follows:

[0072] S1, dataset preparation:

[0073] The images of only bacterial keratitis and fungal keratitis constructed by the application come from a university eye center. And there are at least three evidences to prove that the category labels of all samples are absolutely correct: (1) confirming the pathogen type by methods such as microscopic examination of sample smear or microbial culture; (2) finally recovering after relevant drug treatment; (3) all image categories are confirmed again by ophthalmic experts. To prevent network over-learning of a certain case, the number of clinical pictures of each patient is controlled to be less than 5, and after screening out the pictures with poor optical fiber or angle, a total of 1433 slit top images composed of 459 patients are obtained, including 666 bacterial images and 767 fungal images. At the same time, the bilinear interpolation method is used to reduce each slit lamp image to 512*512 in proportion, and the insufficient part is filled with black pixels.

[0074] S2, image preprocessing;

[0075] The image needs to be processed before being sent into the network, and the specific steps in the application are as follows:

[0076] (1) First, the image is cropped to 448*448, and then the image is processed using random horizontal flip and vertical flip, wherein the probability of flip is set to 0.5, and then the image normalization operation proposed in the article "Investigating the impact of data normalization on classification performance" (Applied Soft Computing, 2020, 97: 105524) by Singh D et al. is used to map the image data to 0 to 1;

[0077] (2) Separate the three primary color channels of the image, and apply the contrast limited adaptive histogram equalization method proposed in the article "Contrast limited adaptive histogram equalization" (Graphics gems, 1994: 474-485) by Zuiderveld K et al. to the B channel, and then fuse the three channels;

[0078] (3) Use the method proposed in the article "Cutmix: Regularization strategy to train strong classifiers with localizable features" by Yun S et al. (Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6023-6032) to augment the image. Replace a random region of size 0 in an image (x A ,y A ) with a corresponding region of the same size in another random image (x B ,y B ), and fuse the label of this image with the label of the region corresponding to the area in proportion to the area, and finally generate the augmented image where the probability of generating the CutMix method is set to 0.8.

[0079] S3, classification network and training method

[0080] The entire network structure is shown in Figure 2 The network mainly includes five modules: feature extraction module, contrast selection attention module, feature adjustment module, classifier module and progressive knowledge regularization module. The specific training process includes the following steps:

[0081] (1) Multi-granularity feature extraction

[0082] The ResNet50 network proposed in the article "Deep residual learning for image recognition" by He K et al. (Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778) is used as the feature extraction network. As shown in Figure 3 ResNet50 is composed of 5 stages, each stage contains convolution and pooling operations, and different convolution operations are used to extract feature maps of different stages. Specifically, the feature maps F1, T2, F3 output by the last three stages of the extraction network represent different granularity features, where the shallow layer output has fewer convolution operations and the area perceived by each position in the feature map is relatively small, i.e. it can represent smaller granularity features. Based on this, three features of increasing granularity are extracted as input for subsequent operations;

[0083] (2) Attention enhancement of features

[0084] F1, F2, F3 are respectively passed through the contrastive selection attention module, and part of the feature channels are selectively enhanced by contrasting the feature differences between bacteria and fungi, so that the network gradually pays attention to more discriminative information. Specifically, as shown in Figure 4 the batch feature map output in this stage is divided into two categories according to the class label, i.e. the feature set F BK of the bacteria class and the feature set F FK of the fungi class, and then their average values are calculated to represent the average features of the class. The specific calculation is as follows:

[0085]

[0086] wherein, C, H, W are the size of the channel, width and height of the image respectively, j ∈ {BK, FK}, BK and FK are respectively bacterial or fungal keratitis, and m is the number of BK or FK samples in the training batch;

[0087] The average class features are resized to The similarity vector represents the similarity of the two average class features on each channel, and it is converted to a channel weight through SoftMax, as shown in the following formula:

[0088]

[0089]

[0090] wherein, z ∈ [0, 1] C has a weight on each channel, sim is a cosine similarity operation between feature channels, and z i is the similarity value when the channel index i;

[0091] Since smaller similarity indicates that the visual features represented by the channel have more characteristics to distinguish between the two classes, in order to make such channels get more weight, z is processed as follows:

[0092] z = 1-z (4)

[0093] Select the first C / 2 small values in the channel weight, and set them to 0, i.e. the channels without discriminative features in the feature do not have effect in feature enhancement, and let z after selection be denoted as z select , and then multiply each channel of the feature by a weight through dot product, and finally add the enhanced weight value to the original feature value to obtain the enhanced feature F', which is represented by the following formula:

[0094] F′ i = F i + (1-z select)·F i (5)

[0095] Where i∈{1,2,3} are the subscripts of the different granularities extracted.

[0096] (3) Multi-granularity feature prediction

[0097] First, the enhanced features are processed through the feature adjustment module, the structure of which is as follows: Figure 5 As shown. The goal is to process features into the same dimension, expressed by the following formula:

[0098] V i =ReLU(BN(Conv3(ReLU(BN(Conv1(F′ i (6)

[0099] Wherein, Conv3 and Conv1 are convolution operations with kernel sizes of 3×3 and 1×1 and stride of 1, respectively; BN is batch normalization; ReLU is ReLU activation; and i∈{1,2,3} represents feature indices of different granularities.

[0100] The above features are combined to obtain the comprehensive features:

[0101] V4=Concatenate(V1, V2, V3) (7)

[0102] V respectively i For i∈{1,2,3,4}, a classifier can be used to obtain prediction results based on features of different granularities. This process can be represented by the following formula:

[0103] p i =Classifier i (V i ), i∈{1,2,3,4} (8)

[0104] Among them, Classifier i The classifier consists of two fully connected layers, BatchNorm, and ELU operations, with the specific structure as follows: Figure 6 As shown, p is the output of the fully connected layer, which is the predicted value for bacteria and fungi, respectively;

[0105] (4) Calculate the loss function

[0106] The loss function used in the network structure of this invention is a multi-granularity prediction loss. and progressive knowledge regularization loss Composition, specifically, through The difference between the true class label and the model prediction is measured, and the cross-entropy loss is calculated based on the prediction results of different granularity features and the image label, and the expression is:

[0107]

[0108]

[0109]

[0110] wherein y is a sample class label, if it belongs to the class, the value is 1, otherwise the value is 0, is the probability of the prediction result of the model based on the i-stage feature in the c-class, λ i is the weight parameter of each prediction result (set to [1, 1, 1, 2] in this embodiment), and T is the temperature coefficient (set to 1 in this embodiment), which can change the attention degree to negative samples during the training process;

[0111] In particular, for the current sample, if it has undergone CutMix operation during preprocessing, then The loss is adjusted as:

[0112]

[0113] wherein y A and y B are the true labels of the two images fused by CutMix, and θ is the ratio of the substitute area to the original image size;

[0114] Progressive knowledge regularization loss is used to enhance the causal relationship between the simple features generated by the shallow layers of the network model and the advanced semantic features generated by the deep layers, improve the robustness of the model as a whole by using the prediction results of the deep layers to guide the prediction of the shallow layers, and is expressed by the following formula:

[0115]

[0116] Finally, the parameters are adjusted by backpropagation during the training of the network, wherein α and β are coefficients for balancing each loss (set to [1, 1] in this embodiment);

[0117] S4, training an image classification model;

[0118] The training network uses the Adam optimization algorithm with adaptive learning rate, uses the PyTorch deep learning framework in this embodiment, the training batch size is 12, and there are 100 training cycles.

[0119] ​During the training process, 6 bacterial sample pictures and 6 fungal sample pictures are randomly selected from the provided training set images as the input of this batch, sequentially pass through the preprocessing, feature extraction, feature enhancement, feature adjustment and classifier module, calculate two losses and back propagate the gradient, and save the best model during the training process.

[0120] S5, using the classification network to classify the bacterial keratitis and fungal keratitis images

[0121] The image preprocessing part is modified to only contain the center cropping to 448x448 size operation and the standardization operation, and for the images of the test set or new bacterial and fungal keratitis images, the different granularity-based prediction results are generated after preprocessing and the classification network, and the final prediction result is obtained by adding these prediction results, that is, the diagnosis of bacterial and fungal keratitis can be realized.

[0122] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously.

[0123] Based on the same idea as the bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning in the above embodiment, the present application also provides a bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning, which can be used to execute the above bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning. For the convenience of description, in the structural schematic diagram of the bacterial keratitis and fungal keratitis slit lamp image classification system embodiment based on multi-granularity feature learning, only the parts related to the embodiments of the present application are shown, and those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, which can include more or fewer components than the illustrated, or combine certain components, or different component arrangements.

[0124] Please refer to Figure 7 In another embodiment of the present application, a bacterial keratitis and fungal keratitis slit lamp image classification system 100 based on multi-granularity feature learning is provided, which includes a data acquisition module 101, a model construction module 102, a model training module 103 and an image classification module 104.

[0125] The data acquisition module 101 is used to acquire a slit lamp image dataset containing bacterial keratitis and fungal keratitis, and preprocess the slit lamp image dataset.

[0126] The model construction module 102 is configured to construct an image classification model, the image classification model comprising a feature extraction module, a contrast selection attention module, a feature adjustment module, a classifier module and a progressive knowledge regularization module; the feature extraction module is configured to extract features of different granularities of a preprocessed slit lamp image dataset; the contrast selection attention module is configured to selectively enhance part of the feature channels by contrasting the feature differences between bacterial and fungal characteristics; the feature adjustment module is configured to process the enhanced features into the same dimension; and the classifier module is configured to make classification prediction based on the features of different granularities.

[0127] The model training module 103 is configured to train the image classification model based on the progressive knowledge regularization, and update the image classification model based on a loss function until convergence; the loss function comprises a multi-granularity prediction loss and a progressive knowledge regularization loss The multi-granularity prediction loss is used to correct the classification effect of the model; by measuring the difference between the real class label and the model prediction, the cross-entropy loss is calculated based on the prediction results of the features of different granularities and the image label; and the progressive knowledge regularization loss is used to enhance the causal relationship between the simple features generated by the shallow layer of the network model and the advanced semantic features generated by the deep layer of the network model.

[0128] The image classification module 104 is configured to classify bacterial or fungal keratitis images using the trained network model.

[0129] It should be noted that the bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning of the present application corresponds to the bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning of the present application, and the technical features and advantages described in the above embodiment of the bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning are applicable to the embodiment of the bacterial keratitis and fungal keratitis slit lamp image classification based on multi-granularity feature learning, and the specific content can be referred to the description in the method embodiment of the present application, which will not be repeated here, and hereby declared.

[0130] In addition, in the implementation of the bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning in the above embodiment, the logical division of each program module is only illustrative. In actual applications, the above-mentioned function allocation can be completed by different program modules according to needs, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning is divided into different program modules to complete all or part of the functions described above.

[0131] Please refer to Figure 8 In one embodiment, an electronic device implementing a bacterial keratitis and fungal keratitis slit lamp image classification method based on multi-granularity feature learning is provided. The electronic device 200 can include a first processor 201, a first memory 202, and a bus. It can also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a bacterial keratitis and fungal keratitis slit lamp image classification program 203 based on multi-granularity feature learning.

[0132] The first memory 202 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Further, the first memory 202 can include both an internal storage unit and an external storage device of the electronic device 200. The first memory 202 can be used not only to store application software and various data installed on the electronic device 200, such as the code of the bacterial keratitis and fungal keratitis slit lamp image classification program 203 based on multi-granularity feature learning, but also to temporarily store data that has been or will be output.

[0133] The first processor 201 may, in some embodiments, be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits of the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, which connects various components of the entire electronic device through various interfaces and lines, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202 and calling data stored in the first memory 202.

[0134] Figure 8 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 8 The structure shown does not constitute a limitation on the electronic device 200, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0135] The first memory 202 in the electronic device 200 stores a bacteria keratitis and fungal keratitis slit lamp image classification program 203 based on multi-granularity feature learning, which is a combination of multiple instructions and can realize:

[0136] Obtain a slit lamp image dataset containing bacteria keratitis and fungal keratitis, and pre-process the slit lamp image dataset;

[0137] Construct an image classification model, which includes a feature extraction module, a contrast selection attention module, a feature adjustment module, a classifier module, and a progressive knowledge regularization module; the feature extraction module is used to extract features of different granularities of the pre-processed slit lamp image dataset; the contrast selection attention module selectively enhances part of the feature channels by comparing the feature differences between bacteria and fungi; the feature adjustment module is used to process the enhanced features to the same dimension; the classifier module classifies and predicts based on different granularity features;

[0138] Train the image classification model based on progressive knowledge regularization, update the image classification model based on a loss function until convergence; the loss function includes a multi-granularity prediction loss And a progressive knowledge regularization loss By Measuring the difference between the real class label and the model prediction, calculating the cross-entropy loss based on the prediction results of different granularity features and the image label; the multi-granularity prediction loss The classification effect of the correction model; the progressive knowledge regularization loss For enhancing the causal relationship between the simple features generated by the shallow layers of the network model and the advanced semantic features generated by the deep layers.

[0139] Classify bacterial or fungal keratitis images using the trained network model.

[0140] Further, the modules / units of the electronic device 200, if implemented in the form of software functional units and sold or used as independent products, can be stored in a non-volatile computer-readable storage medium. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. In the embodiments provided in the present application, any reference to memory, storage, database or other medium can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0142] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0143] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A method for classifying bacterial keratitis and fungal keratitis slit lamp images based on multi-granularity feature learning, characterized in that, The method comprises the following steps: obtaining a slit lamp image data set containing bacterial keratitis and fungal keratitis, and preprocessing the slit lamp image data set; constructing an image classification model, the image classification model comprising a feature extraction module, a contrast selection attention module, a feature adjustment module, a classifier module and a progressive knowledge regularization module; the feature extraction module is used to extract features of different granularities of the preprocessed slit lamp image data set; the contrast selection attention module selectively enhances part of the feature channels by comparing the feature differences of bacterial and fungal characteristics; the feature adjustment module is used to process the enhanced features into the same dimension; the classifier module performs classification prediction based on different granularity features; training the image classification model based on progressive knowledge regularization, updating the image classification model based on a loss function until convergence; the loss function comprises a multi-granularity prediction loss and a progressive knowledge regularization loss ; the multi-granularity prediction loss is used to correct the classification effect of the model; by measuring the difference between the real class label and the model prediction, calculating the cross-entropy loss based on the prediction results of different granularity features and the image label; the progressive knowledge regularization loss is used to enhance the causal relationship between the simple features generated by the shallow layer of the network model and the advanced semantic features generated by the deep layer. using the trained network model to classify bacterial or fungal keratitis images; The contrast selection attention module divides the feature maps output by the feature extraction module into two categories according to the category labels: the feature set of the bacterial class. Characteristic sets of fungi Then, the average value of each is calculated to represent the average feature of that category. The specific calculation is as follows: wherein, , are the size of the channels, width and height of the image respectively, , BK, FK are the bacterial keratitis images, fungal keratitis images respectively, is the number of BK or FK samples in the training batch; Resize the average class feature to The similarity of the two average class features on each channel is represented by a similarity vector, which is converted to channel weights by The formula is as follows: wherein, there is a weight on each channel, to compute a cosine similarity operation between feature channels, denotes the similarity weight of the i-th channel; Since the smaller the similarity is, the more the visual feature represented by the channel has the characteristics of distinguishing the two classes, in order to make such a channel obtain a greater weight, the similarity is processed as follows: by processing as follows: Selecting the channel weights small value, set it to 0, that is, the channel without discriminability in the feature has no effect on the feature enhancement, and the selected is expressed as , then multiply each channel of the feature by a weight through dot multiplication, and finally add the enhanced weight value to the original feature value to obtain the enhanced feature , which is expressed by the following formula: wherein, is the index of the different granularity extracted; The feature adjustment module processes the enhanced features into the same dimension, using the following formula: wherein, and are convolution operations with kernel size 3x3 and 1x1, respectively, with a stride of 1, is batch normalization, is a ReLU activation operation, denotes the feature index of different granularities.

2. The method of claim 1, wherein the method is based on multi-granularity feature learning for bacterial keratitis and fungal keratitis slit lamp image classification. The preprocessing is specifically as follows: crop the images in the slit lamp image data set to a set size, randomly horizontally and vertically flip the cropped images, and map the image data to 0 to 1 by using image standardization operation; separate the three primary color channels of the image after the standardization operation, apply the limited contrast adaptive histogram equalization method to the B channel, and then fuse the three channels; perform CutMix augmentation on the image after three-channel fusion. 3.The method of claim 1, wherein the method further comprises: determining a plurality of feature maps based on the plurality of feature layers; and determining a plurality of feature vectors based on the plurality of feature maps. The feature extraction module uses a ResNet50 network, which is composed of 5 stages, each stage containing a plurality of convolution and pooling operations, and different features are extracted from different stages by using different convolution operations with different perception domains.

4. The method of claim 1, wherein the method is based on multi-granularity feature learning for bacterial keratitis and fungal keratitis slit lamp image classification. The features processed by the feature adjustment module are spliced to obtain comprehensive features, and the multi-granularity features and the spliced comprehensive features are input into the classifier module respectively to obtain prediction results based on different granularity features, and the classifier module is composed of two fully connected layers, BatchNorm and ELU operation.

5. The method of claim 1, wherein the method is based on multi-granularity feature learning for bacterial keratitis and fungal keratitis slit lamp image classification. In the step of training the image classification model based on progressive knowledge regularization, the multi-granularity prediction loss and the progressive knowledge regularization loss Specifically as follows: in, These are the sample category labels; a value of 1 indicates the sample belongs to that category, and a value of 0 indicates otherwise. The model is based on The results predicted by stage features Probability of category Here are the weighting parameters for each prediction result, and T is the temperature coefficient, which can change the degree of attention given to negative samples during training. progressive knowledge regularization loss To enhance the causal relationship between simple features generated by the shallow layers and advanced semantic features generated by the deep layers of the network model, the robustness of the model as a whole is improved by using the prediction results of the deep layers to guide the prediction of the shallow layers. The formula is as follows: Finally, the parameters are adjusted by backpropagation with reflection when training the image classification model, wherein is a coefficient for balancing the respective losses. and is a coefficient for balancing the respective losses.

6. The method of claim 5, wherein the method is based on multi-granularity feature learning for bacterial keratitis and fungal keratitis slit lamp image classification. For the current sample, if it has undergone CutMix operation at pre-processing, then The loss is adjusted as: where, and are the true labels of the two images fused by CutMix, is the ratio of the substitute region to the original image size.

7. A bacterial keratitis and fungal keratitis slit lamp image classification system based on multi-granularity feature learning, characterized in that, The method for classifying bacterial keratitis and fungal keratitis slit lamp images based on multi-granularity feature learning according to any one of claims 1-6 comprises a data acquisition module, a model construction module, a model training module and an image classification module; The data acquisition module is used to obtain a slit lamp image data set containing bacterial keratitis and fungal keratitis, and preprocess the slit lamp image data set; The model construction module is used to construct an image classification model, the image classification model comprising a feature extraction module, a contrast selection attention module, a feature adjustment module, a classifier module and a progressive knowledge regularization module; the feature extraction module is used to extract features of different granularities of the preprocessed slit lamp image data set; the contrast selection attention module selectively enhances part of the feature channels by comparing the feature differences of bacterial and fungal characteristics; the feature adjustment module is used to process the enhanced features into the same dimension; the classifier module performs classification prediction based on different granularity features; The model training module is configured to train the image classification model based on progressive knowledge regularization, and update the image classification model based on a loss function until convergence; the loss function comprises a multi-granularity prediction loss and a progressive knowledge regularization loss ; the multi-granularity prediction loss is used to correct the classification effect of the model; by measuring the difference between the real class label and the model prediction, the cross-entropy loss is calculated based on the prediction results of different granularity features and the image label; the progressive knowledge regularization loss is used to enhance the causal relationship between the simple features generated by the shallow layer of the network model and the advanced semantic features generated by the deep layer. The image classification module uses the trained network model to classify bacterial or fungal keratitis images.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor; wherein the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the method for classifying bacterial keratitis and fungal keratitis slit lamp images based on multi-granularity feature learning according to any one of claims 1-6.

9. A computer-readable storage medium storing a program, the program comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The program is executed by the processor to implement the method for classifying bacterial keratitis and fungal keratitis slit lamp images based on multi-granularity feature learning according to any one of claims 1-6.

Images