A cataract fine-grained classification method and system based on a slit lamp image classification model training

Abstract

A cataract fine-grained classification method in unbalanced slit lamp images comprises the following steps: a cataract fine-grained classification network based on a self-attention mechanism is constructed; pictures are divided into fixed-size patches; after a patch embedding layer, the pictures are input into multi-head self-attention to learn from different dimensional spaces so as to better extract the pathological level features of cataracts; the focus loss is dynamically modified according to the sample size to rebalance the weights of different categories; the existing data categories are reasonably divided according to their data volume to construct a fine-grained data set containing multiple cataract categories for model training; and the proposed method is verified for effectiveness by using multiple indexes. The application also comprises a cataract fine-grained classification system in unbalanced slit lamp images. The application can output the specific category and severity of cataracts, and can still obtain better output results when the data volume presents a long-tail distribution.

Description

Technical Field

[0001] This invention pertains to artificial intelligence-based medical image processing and analysis methods and systems, and specifically relates to a method and system for training an unbalanced slit-lamp image classification model based on self-attention mechanism and refocusing loss, thereby achieving fine-grained cataract classification. Background Technology

[0002] Cataracts are a leading cause of blindness in low- and middle-income countries. The World Health Organization estimates that approximately 2.2 billion people worldwide suffer from visual impairment, with cataracts accounting for about 33% of all cases. Furthermore, the number of cataract patients is increasing annually due to rapid population aging. The primary site of cataract damage is the lens, a transparent, biconvex structure that refracts light and focuses it onto the retina. When the internal environment of the eye is subjected to radiation or physical damage, the metabolism of the lens is disrupted, leading to protein denaturation and clouding. Clinically, doctors make diagnoses after obtaining slit-lamp images of the patient's eye—a time-consuming and labor-intensive process. Moreover, the subjective factors that can influence the assessment process can also affect doctors. Additionally, the lack of access to expensive medical equipment in some remote areas prevents many cataract patients from assessing their eye condition for timely treatment. Therefore, establishing a deep learning-based automated cataract diagnosis method and system is essential.

[0003] Currently, the main artificial intelligence-based cataract diagnosis methods include: Naive Bayes (…). Methods such as Bayes (NB), Random Forest (RF), Support Vector Machine (SVM), and Convolutional Neural Network (CNN) are used. However, these methods only classify a single disease, or only classify cataracts without grading them. Furthermore, these methods are all tested on balanced datasets and do not explore how the models perform in real-world scenarios with imbalanced data. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a fine-grained classification method for cataracts based on slit-lamp image classification model training.

[0005] To overcome the problem of deep learning models biasing their weights towards the majority class in imbalanced datasets, and to obtain more granular cataract classification results, this invention provides a simple and effective method that utilizes a refocusing loss function to help the model learn more features of the minority class during training. Furthermore, in the data section, this invention not only provides each category of cataracts but also the specific grade of each category. Therefore, this invention can predict not only the type of cataract in a patient but also the grade of the cataract.

[0006] This invention utilizes a self-attention mechanism to construct a feature extraction network, extracting both global and attention-based features. Furthermore, during training, a refocusing loss is used to guide the network to allocate more attention to learning features of rare classes, further balancing the weights within the network and optimizing the learning process.

[0007] The present invention provides a fine-grained cataract classification method based on a slit-lamp image classification model training, comprising:

[0008] Step 1: Construct a fine-grained cataract classification network framework; build a fine-grained cataract classification network model based on a multi-head self-attention mechanism. In slit-lamp images, the lens of the human eye plays a crucial role in fine-grained cataract classification, yet this part constitutes a small portion of the entire image. By utilizing a multi-head self-attention mechanism to grasp the key parts of the slit-lamp image from a global perspective, and by modeling features and extracting feature relationships within the lens, the network's performance in learning fine-grained cataract features is improved.

[0009] Step 2: Train the fine-grained cataract classification network model; use a customized loss function to balance the weights of the majority and minority classes within the network. Based on learning the differences between cataract classes, further capture the features that cause intra-class differences, thereby improving the classification performance of the fine-grained cataract classification network model for the minority classes; where the majority class refers to the class with a large number of samples, and the minority class refers to the class with a small number of samples.

[0010] Step 3: Dataset preprocessing; high-quality slit lamp images are selected and divided into nine categories according to existing labels; with high-quality slit lamp images as input, the model can learn more intuitively the fine and rich cataract fine-grained features, thereby improving the robustness and generalization ability of the model.

[0011] Step 4: Cataract classification test; Input the slit-lamp image into the trained fine-grained cataract classification network model to test the model's accuracy, especially the performance of fine-grained cataract classification.

[0012] Furthermore, step one specifically includes the following steps:

[0013] Step 1.1: The cataract fine-grained classification network model includes an encoder and a classifier; the encoder includes four feature extraction modules for extracting pathological layer information from slit-lamp images; the classifier consists of a fully connected layer that maps the cataract fine-grained features extracted from the slit-lamp image to their respective cataract fine-grained categories.

[0014] Step 1.2: Before inputting the images into the cataract fine-grained classification network, the slit-lamp images need to be normalized by scaling down the high-resolution images of inconsistent sizes to low-resolution images of uniform size. Then, convolutional layers are used to uniformly divide the normalized images into fixed-size patches, and the original three-channel images are enlarged to a higher-dimensional feature space, representing the fine-grained features of cataracts in the image. This ensures that the model obtains consistent results when processing images of different sizes; and in the higher-dimensional space, the model can learn more fine-grained cataract features, thus improving the final accuracy of the model.

[0015] Step 1.3: The encoder of the fine-grained cataract classification network model is constructed. The encoder contains four feature extraction modules, which are called the first, second, third and fourth feature extraction modules in the order of network depth. Each feature extraction module consists of multiple cascaded sub-encoder Transformer Blocks. The first, second and fourth feature extraction modules each consist of 2 sub-encoders, and the remaining sub-encoders are located in the third feature extraction module.

[0016] The internal structures of the multiple sub-encoders are the same, but the input neurons of the fully connected layers inside the sub-encoders need to be adjusted according to the dimensions of the cataract features. Specifically, sub-encoders located in the same feature extraction module have the same number of input channels in their fully connected layers. Conversely, sub-encoders located in different feature extraction modules have different numbers of input channels in their fully connected layers, and the number of input channels increases as the network deepens.

[0017] Step 1.4: Patch Merging between the feature extraction modules. There are three patch merging steps between the four feature extraction modules. As their name suggests, they overlay adjacent patches on the channels, halving the image's length and width and quadrupling the number of channels. Then, a fully connected layer reduces the number of channels by half, completing the merging process. Therefore, as mentioned in the previous step, the sub-encoders within different feature extraction modules need to have fully connected layers with different numbers of input channels to adapt to different image sizes and feature dimensions, i.e., different input shapes.

[0018] Step 1.5: The classifier of the cataract fine-grained classification model consists of a fully connected layer that maps the number of output neurons of the fourth feature extraction module to the number of cataract fine-grained categories. This fully connected layer has weights and biases, both defined as floating-point variables.

[0019] Furthermore, step two specifically includes:

[0020] Step 2.1: Obtain the number of images for each fine-grained cataract category; because different weights need to be assigned to different categories, and the weights are defined by the number of images in a certain category, the number of images CN for each category is counted. i c represents the c-th type of fine-grained cataract;

[0021] Step 2.2: Define the weight of each cataract fine-grained category, as shown in the following formula;

[0022]

[0023]

[0024] Where pcw is an abbreviation for per_cls_weight, β∈[0,1);

[0025] Step 2.3: Design the category factor for each fine-grained cataract category, using the following formula:

[0026]

[0027] Step 2.4: Generate the final loss function, the formula of which is:

[0028]

[0029] Step 2.5: Use the pre-trained model as the initial model and start multiple rounds of iterative training; during the training process, the learning rate of the initial few rounds of training needs to be set to linear growth, and the learning rate of subsequent rounds needs to be set to non-linear decrease;

[0030] Step 2.6: Calculate the refocusing loss between the model output and the true label in each iteration;

[0031] Step 2.7: Minimize the loss function using the AdamW optimizer and save the optimal model.

[0032] Furthermore, step three specifically includes:

[0033] Step 3.1: Screening slit-lamp images; the screening requirements include: the eye structure is clearly visible, especially the lens region, which plays a decisive role in the fine-grained classification of cataracts; the lens region cannot be completely obscured by the slit-lamp light source, otherwise fine-grained features of cataracts cannot be extracted.

[0034] Step 3.2: Categorize unbalanced cataract images; merge categories with a small number of slit-lamp images and similar clinical symptoms and treatments.

[0035] Step 3.3: With existing labels, organize and classify the dataset after processing the imbalanced images. The final fine-grained cataract categories include: nuclear type II, nuclear type III, ≥ nuclear type IV, mild cortical type, intermediate cortical type, severe cortical type, fully mature, posterior cystic, and healthy. These fine-grained cataract categories correspond to the final output of the fine-grained cataract classification network. Divide the data into training, validation, and test sets in a certain proportion, and ensure that the distribution of each fine-grained cataract class is consistent across these datasets.

[0036] Furthermore, the patching described in step 1.2 satisfies the condition that the convolution kernel size K = patch size P, and the relationship between the patch size P, the image size N×N, and the window size W satisfies the following condition. n is an integer greater than 1; the number of convolution kernels satisfies 2n m m is an integer greater than or equal to 6.

[0037] Furthermore, in step 1.3, the height N and width N of the initial image in the four feature extraction modules result in the following output image shapes: Where f represents the f-th feature extraction module.

[0038] Furthermore, in step 1.3, each sub-encoder performs window self-attention mechanism and offset window self-attention mechanism calculations to extract features within the window and features of the connections between windows. The operations required for the window-internal self-attention mechanism are: LayerNorm normalization, calculation of window attention offset WA, random deactivation of neurons (Dropout), and calculation of Multilayer Perceptron (MLP). The window attention offset WA and the MLP layers contain weights and biases, both defined as floating-point variables. The window-to-window self-attention mechanism is implemented by moving the window, cyclically moving the entire window to the upper left corner, and then performing the window-internal self-attention mechanism calculation to extract features between windows. The formula for self-attention calculation is as follows:

[0039]

[0040] Building upon this, the attention mechanism is further extended to a multi-head self-attention mechanism, as shown in the following formula:

[0041] MSA(Q,K,V)=Concat(head1,…,headh )·W O ,

[0042] where head i = Attention(QW i Q ,KW i K VW i V (2)

[0043] MSA stands for Multi-Head Self-Attention Mechanism (MSA), W O W Q W K W V All of these are learnable parameters, defined as floating-point variables;

[0044] In addition, residual connections are used after each random dropout to make the training process more stable.

[0045] Furthermore, within the sub-encoder described in step 1.4, the input size of the corresponding LayerNorm, Window Attention Shift (WA), and Multilayer Perceptron (MLP) must correspond to the image size; simultaneously, the feature dimension dim must be divisible by the number of attention heads h.

[0046] Furthermore, the iterative training described in step 2.5, specifically the warm-up phase with linearly increasing learning rate, is mainly to prevent the model from getting stuck in a local optimum. Therefore, the number of training rounds should be controlled to within twenty rounds. During the nonlinear learning rate decay process, a sine function is selected.

[0047] Furthermore, step four specifically includes:

[0048] Step 4.1: Test the saved optimal model on the test set. The test metrics include precision, recall, F1-score, and average accuracy.

[0049] Step 4.2: For the nine fine-grained categories of cataracts, calculate the accuracy of predicting each category and compare it with other more effective methods for handling imbalanced data, and demonstrate the effectiveness of the method through experiments;

[0050] Step 4.3: Shuffle the source dataset and randomly sample it to test the accuracy of this fine-grained cataract classification on randomly distributed data.

[0051] This invention also provides a system for implementing a fine-grained cataract classification method based on slit-lamp image model training, comprising:

[0052] The cataract fine-grained classification network framework construction module is used to build a cataract fine-grained classification network framework.

[0053] The cataract fine-grained classification network training module is used to train the cataract fine-grained classification network.

[0054] The dataset preprocessing module is used to preprocess datasets;

[0055] The cataract classification test module is used to classify and test cataracts.

[0056] The present invention also provides a cataract fine-grained classification device based on slit-lamp images, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement a cataract fine-grained classification method based on slit-lamp images according to the present invention.

[0057] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a fine-grained cataract classification method based on slit-lamp images according to the present invention.

[0058] This invention can output not only the specific category of cataracts but also their severity, especially when the data volume has a long tail distribution, it still achieves good output results. The working process of this invention is as follows: A network based on a self-attention mechanism is constructed. Images are divided into fixed-size patches, which are then embedded into a PatchEmbedding layer before entering a multi-head self-attention layer. Learning from different dimensions of space allows for better extraction of pathological features of cataracts. ReFocal Loss (RFL) is a multi-class balance loss based on Focal Loss (FL) and ReWeight. The ReFocal Loss dynamically adjusts its loss according to the sample size to rebalance the weights of different categories. Existing data categories are reasonably divided according to their data volume to construct a fine-grained dataset containing multiple cataract categories for model training. Multiple metrics are used to verify the effectiveness of the proposed method, and its robustness under single categories and different distributions is explored. This invention proposes a fine-grained cataract classification method for unbalanced slit-lamp images based on self-attention mechanism and refocusing loss, which alleviates the problem of weight bias towards the majority class in the model and can output fine-grained cataract types.

[0059] The advantages of this invention are: by utilizing the refocusing loss until the weights are biased towards the majority class during the self-attention learning process, it can not only output the specific category of cataracts, but also the severity. Especially when the data volume is long-tailed, this invention can still obtain good output results. It is simple and effective in practical applications. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the cataract classification network framework used in this invention.

[0061] Figure 2 This is a schematic diagram of the multi-head self-attention mechanism of the present invention.

[0062] Figure 3 This is a schematic diagram of data preprocessing according to the present invention.

[0063] Figure 4 This is a flowchart of the model reasoning process of the present invention.

[0064] Figure 5 This is an overall flowchart of the method of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] Example 1

[0067] Slit lamps are commonly used in the clinical diagnosis of cataracts, providing detailed information about the eye, particularly the opacity of the lens. There are various methods for acquiring slit lamp images, one of which is using a digital slit lamp microscope. The slit lamp is connected to a digital camera, resulting in images with higher contrast and resolution, and expanding the functionality of the lens area, allowing ophthalmologists to determine the patient's condition from these images. This application describes a fine-grained classification of cataracts based on slit lamp images.

[0068] Reference Figure 1 and Figure 2 A fine-grained cataract classification method based on slit-lamp images is proposed. The method includes four processes: construction of a fine-grained cataract classification network framework, training of the fine-grained cataract classification network, dataset preprocessing, and cataract classification testing.

[0069] The method includes the following steps:

[0070] Step 1: Construct a fine-grained classification network model framework for cataracts. The process is as follows:

[0071] Step 1.1: The fine-grained cataract classification network model includes an encoder and a classifier. The encoder includes four feature extraction modules, which are used to extract feature information from slit-lamp images of cataracts. The classifier consists of a fully connected layer, which can map the global slit-lamp features, especially cataract-related features, extracted by the encoder to nine fine-grained cataract categories.

[0072] Step 1.2: Before inputting the image into the cataract fine-grained classification network, it needs to be patched, that is, the image is divided into fixed-size blocks. Specifically, this is achieved using a convolutional layer with 3 input channels (In Channels), 128 output channels (Out Channels), a 4×4 kernel size, and a padding of 4. Simultaneously, the original image (3648×2432) is scaled down to 224×224. After this operation, the image is divided into... There are 7x7 pixel patches. Since all subsequent self-attention calculations are performed within the window, we define a window as consisting of 7x7 pixel patches. Therefore, the original image is divided into... Each window is used. Previously, the 3 RGB channels of the image were magnified to a 128-channel feature space to represent the features of each patch. This was done because a 3-channel image cannot adequately measure image features, as it only contains color information. Magnifying it to a higher dimension allows for the learning of more abstract semantic features of cataracts; this is image featureization.

[0073] Step 1.3: As Figure 2 As shown, the fine-grained cataract classification method constructed in this embodiment includes an encoder and a classifier. The encoder consists of four feature extraction modules. The first, second, and fourth feature extraction modules each contain two sub-encoder Transformer Blocks. The third feature extraction module contains 18 sub-encoder Transformer Blocks. At this point, the total number of sub-encoder Transformer Blocks is 24.

[0074] Specifically, within each sub-encoder Transformer Block, LayerNorm_1, WA, Dropout_1, LayerNorm_2, MLP, Dropout_2, LayerNorm_3, SWA, Dropout_3, LayerNorm_4, MLP, and Dropout_4 are executed sequentially. The LayerNorm normalization layer matches the dimension of the features and has no weights or biases, as its purpose is to normalize features and prevent gradient vanishing or exploding. The window self-attention (WA) and offset window self-attention (SWA) are implemented as a fully connected Linear(dim, 3*dim) operation, with weights W_1 and biases b_1, both defined as floating-point variables. The attention calculation formula within the window is as follows:

[0075]

[0076] Where Q, K, and V are the three different layers of features obtained after the features pass through the learnable weights of the fully connected layer, and d k This represents the feature dimension, and the formula is used to measure the correlation between different patches within a window. The multi-head self-attention mechanism builds upon this by dividing patch features into several different features: Q1, Q2, ..., Q... h ,K1,K2,…,K h ,V1,V2,…,V h Where h represents the number of attention heads, set to 16, the formula is as follows:

[0077] MSA(Q,K,V)=Concat(head1,…,head h )·W O ,

[0078] where head i = Attention(QW i Q ,KW i K VW i V (2)

[0079] After completing the intra-window attention calculation, we need to calculate the inter-window attention. Keeping the original window partitioning unchanged, we cyclically move the entire image towards the top left corner. Patches that were previously adjacent between windows are now within a single window. This addresses the inability to measure feature similarity between windows. Subsequent self-attention calculations are performed in the same way as the intra-window attention calculations. Furthermore, the probability of random dropout is set to 0.1, with no weights or biases, to prevent overfitting.

[0080] The Multilayer Perceptron (MLP) consists of two linear mappings: a fully connected layer Linear(feature, 4*feature) and a fully connected layer Linear(4*feature, feature). The feature represents the feature dimension of the output feature of the previous operation, and its weights are W_2, b_2, W_3, and b_3, respectively, all defined as floating-point variables. After each fully connected operation, a GELU activation function is applied.

[0081] Additionally, after each random deactivation dropout, the residual connection is added to the previous features.

[0082] Step 1.4: The feature extraction modules are connected by Patch Merging. To obtain more local feature information, after calculating the self-attention mechanism within and between windows, adjacent 2×2 patches are stacked by channel. At this point, the image length and width become half of the original, and the number of channels (C) becomes four times the original (4C). Then, a fully connected layer is used to map the 4C channels to 2C. The weight of this fully connected layer is W_4, unbiased, and W_4 is defined as a floating-point variable. After this, after each feature extraction module, the image length and width become half of the original. The image channels are doubled. The initial channel size is 128. After 4 modules (3 patch fusion operations), the feature dimension becomes 1024.

[0083] Step 1.5: The classifier consists of a fully connected layer with 1024 input neurons and 9 output neurons, corresponding to 9 fine-grained cataract categories. The weights are W_5 and the biases are b_5, both defined as floating-point variables.

[0084] Step 2: The training process for the cataract fine-grained classification network is as follows:

[0085] Step 2.1: First, obtain the number of images for each fine-grained cataract category in the constructed dataset. In imbalanced datasets, assigning different weights to different categories is a good approach to handling imbalanced data. We generate a Class Factor (CF) based on the number of samples in each category. This CF represents the level of attention the model should give to a particular class during training. The level of attention gradually decreases as the CF increases. Because the Class Factor (CF) is an exponential parameter, a larger value, even with a base less than 1, will further reduce the overall loss value, thus decreasing the model's attention to that class. Based on this, we propose the Refocusing Loss (RFL), the steps of which are as follows:

[0086] Step 2.2: First, we need to define the weights for each fine-grained cataract category, using the following formula:

[0087]

[0088]

[0089] Where CN = [CN1, CN2, ..., CN] C [] represents the number of samples per class, C represents the number of classes, pcw is an abbreviation for per_cls_weight, β∈[0,1) controls the rate at which the weight increases exponentially, and is a predefined hyperparameter less than 1, which we generally set to 0.9999. At this point, the class correlation factor we obtain increases with CN. i The increase and decrease are contrary to our design philosophy.

[0090] Step 2.3: To meet our expectations and ensure that the weighting factors fall within a controllable range (i.e., [0,1]), we design the categorical factors for the fine-grained cataract categories as follows:

[0091]

[0092] Step 2.4: The final loss function is as follows:

[0093]

[0094] Where p t This indicates that the model ultimately outputs the probability value of the true label class. Similarly, CF... t This represents the class weight of the actual label class.

[0095] At this point, the class weight factor we obtain increases with the increase in the number of images of that class. This is because when performing gradient backpropagation, the larger the class weight factor, the larger the penalty term it receives. In this case, the model will pay more attention to other classes, but due to the numerical advantage, the model will inevitably learn the rich features of that class during learning. Conversely, for rare classes, the class weight factor is smaller, and the penalty is smaller. The model can pay more attention to these classes, thus achieving a balance between common and rare classes, which is more beneficial for fine-grained classification of cataracts.

[0096] Step 2.5: Next, we will use the pre-trained model on the ImageNet-1K dataset as the initial model and begin training for a total of 300 epochs. The first 20 epochs are a warm-up phase with a linearly increasing learning rate until it reaches 0.0001. The following 20-300 epochs are a training phase with a sinusoidally decaying learning rate until it decreases to 1×10⁻⁶. -6 This accelerates model convergence;

[0097] Step 2.6: For each iterative model, calculate the probability values ​​of 9 fine-grained cataract classifications and calculate the refocusing loss with the true category;

[0098] Step 2.7: Minimize the loss function using the AdamW optimizer to optimize the model training and save the best cataract classification model.

[0099] Step 3: Preprocessing the dataset as described above, the process is as follows:

[0100] Step 3.1: For blurry or unclear slit-lamp images, especially those where the slit-lamp light source completely obscures the lens, we cannot make a judgment based on the lesion location. Considering the quality of the input images and the richness of features for the deep learning model, we exclude these images.

[0101] Step 3.2: Regarding the imbalanced cataract data, initially, the class with the most data had over 2000 images, while the class with fewer data had only about 10. In this situation, considering the severity of the disease and the obviousness of symptoms, we merged classes with fewer slit-lamp images within a given category. For example, in the original data, nuclear type V and nuclear type IV cataracts had relatively few images, and these two types of cataracts are very severe in causing blindness, with very obvious symptoms, and both require surgical treatment. Therefore, we merged these two classes into one class (≥ nuclear type IV). Another example: in the original dataset, the number of images for mild, moderate, and severe posterior capsule cataracts were 6, 3, and 12 respectively. Compared to the 2242 images for nuclear type II, the classification performance of the model for this rare class would be further reduced after posterior capsule subdivision. Furthermore, posterior capsule cataracts have relatively obvious symptoms in the early stages, and the treatment methods are similar; therefore, we uniformly classified them as "posterior capsule type". Therefore, our final cataract fine-grained classification is as follows: nuclear type II, nuclear type III, ≥ nuclear type IV, mild cortical type, moderate cortical type, severe cortical type, fully mature, posterior cystic, and healthy. The cataract fine-grained classification corresponds to the final output of the cataract fine-grained classification network.

[0102] Step 3.3: The data is then divided into a training set, a validation set, and a test set in a 6:2:2 ratio, with the distribution of each class remaining consistent across these data sets.

[0103] Step four, regarding the aforementioned cataract fine-grained classification test, the steps are as follows:

[0104] Step 4.1: Input the test set into the trained cataract fine-grained classification network model. We calculate the precision, recall, F1-Score and mean accuracy of the final results to evaluate the overall performance of the model and the model's cataract fine-grained classification ability under the effect of refocusing loss.

[0105] Step 4.2: In particular, for the nine fine-grained categories of cataracts, we calculated the prediction accuracy for each category and compared it with other methods. This comparison method can better reflect the problem of refocusing loss alleviating severe model weight shift in the case of data imbalance, thereby achieving fine-grained classification of cataracts. It can also obtain good prediction results for rare categories such as posterior cystic and ≥ nuclear grade IV, which have only about 10 images. At the same time, while focusing more attention on rare categories, we also ensured that the classification effect of common categories (such as nuclear grade II and cortical intermediate) is almost the same as before.

[0106] Step 4.3: Finally, to further verify the performance of this method in fine-grained cataract classification under imbalanced slit-lamp image conditions, we randomly shuffled the source data and then performed sampling with replacement ten times, sampling 532 images each time, generating a total of 10 randomly distributed datasets for model testing. Under this comparison method, we demonstrated the robustness of our fine-grained cataract classification method, showing its ability to better classify various cataract categories. Therefore, our fine-grained cataract classification method based on slit-lamp images is quite effective.

[0107] Example 2

[0108] This invention also provides a system for implementing a fine-grained cataract classification method based on slit-lamp image model training as described in Embodiment 1, comprising:

[0109] The cataract fine-grained classification network framework construction module is used to build a cataract fine-grained classification network framework.

[0110] The cataract fine-grained classification network training module is used to train the cataract fine-grained classification network.

[0111] The dataset preprocessing module is used to preprocess datasets;

[0112] The cataract classification test module is used to classify and test cataracts.

[0113] The above modules correspond to steps one through four of embodiment one.

[0114] Example 3

[0115] The present invention also provides a cataract fine-grained classification device based on slit-lamp images, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a cataract fine-grained classification method based on slit-lamp images according to Embodiment 1.

[0116] Example 4

[0117] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the fine-grained cataract classification method based on slit-lamp images described in Embodiment 1.

[0118] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A fine-grained cataract classification method based on a slit-lamp image classification model training, comprising: Step 1: Construct a fine-grained classification network framework for cataracts; Construct a fine-grained cataract classification network model based on a multi-head self-attention mechanism; By utilizing a multi-head self-attention mechanism to grasp the lens portion of the human eye structure in slit-lamp images from a global perspective, and performing feature modeling and feature relationship extraction in lens images, the performance of the network in learning fine-grained features of cataracts can be improved. Step 2: Train a fine-grained cataract classification network model. A customized loss function is used to balance the weights of the majority and minority classes within the network. Building upon the inter-class differences in cataracts, the model captures features that cause intra-class differences, thereby improving the classification performance of the fine-grained cataract classification network model for the minority class. The majority class refers to the class with the most samples, and the minority class refers to the class with the fewest samples. Specifically, this includes: Step 2.1: Obtain the number of images for each fine-grained cataract category; because different weights need to be assigned to different categories, and the weights are defined by the number of images in a certain category, it is necessary to count the number of images in each category. , This indicates the number of fine-grained categories of cataracts; Step 2.2: Define the weight of each category, using the following formula; ； ； in for abbreviation, ; Step 2.3: Design the category factor for each fine-grained cataract category, using the following formula: ； Step 2.4: Generate the final loss function, the formula of which is: ； Step 2.5: Use the pre-trained model as the initial model and start multiple rounds of iterative training; during the training process, the learning rate of the initial few rounds of training needs to be set to linear growth, and the learning rate of subsequent rounds needs to be set to non-linear decrease; Step 2.6: Calculate the refocusing loss between the model output and the true label in each iteration; Step 2.7: Minimize the loss function using the AdamW optimizer and save the optimal model; Step 3: Dataset preprocessing; high-quality slit lamp images are selected and divided into nine categories according to existing labels; with high-quality slit lamp images as input, the model can intuitively learn fine-grained features of cataracts, improving the robustness and generalization ability of the model. Step 4: Cataract classification test; Input the slit-lamp image into the trained fine-grained cataract classification network model to test the model's accuracy, thereby testing the performance of fine-grained cataract classification.

2. The cataract fine-grained classification method based on slit-lamp images as described in claim 1, characterized in that, Step one specifically includes: Step 1.1: The cataract fine-grained classification network model includes an encoder and a classifier; the encoder includes four feature extraction modules for extracting fine-grained feature information of the pathological layer of the slit-lamp image; the classifier consists of a fully connected layer, which maps the cataract fine-grained features extracted from the slit-lamp image to their respective corresponding cataract fine-grained categories. Step 1.2: Before inputting the images into the cataract fine-grained classification network, the slit-lamp images need to be normalized by scaling down the high-resolution images of inconsistent sizes to low-resolution images of consistent sizes. Then, convolutional layers are used to uniformly divide the normalized images into patches of fixed size, and the original three-channel images are enlarged into a higher-dimensional feature space, representing the fine-grained features of the image related to cataracts. In this way, the model can obtain consistent results when processing images of different sizes. Furthermore, in the higher-dimensional space, the model can learn more fine-grained features of cataracts, thus improving the final accuracy of the model. Step 1.3: Construct the encoder of the cataract fine-grained classification network model. The encoder contains four feature extraction modules, which are called the first, second, third and fourth feature extraction modules in order of increasing network depth. Each feature extraction module consists of multiple cascaded sub-encoder Transformer Blocks. Multiple sub-encoders have the same internal structure, but the input and output neurons of the fully connected layers inside the sub-encoders need to be adjusted according to the dimensions of the cataract features. Specifically, sub-encoders located in the same feature extraction module have the same number of input channels in their fully connected layers. Conversely, sub-encoders located in different feature extraction modules have different numbers of input channels in their fully connected layers, and the number of input channels increases as the network deepens. Step 1.4: Patch Merging between the various feature extraction modules. There are three patch merging steps between the four feature extraction modules. Adjacent patches are superimposed on the channels. At this time, the length and width of the image become half of the original, and the number of channels becomes four times the original. Then, a fully connected layer is used to reduce the number of channels by half, thus completing the merging process. Therefore, as mentioned in the previous step, the sub-encoders inside different feature extraction modules need to set fully connected layers with different numbers of input channels to adapt to different image sizes and feature dimensions, i.e., different shapes of input. Step 1.5: The classifier of the fine-grained cataract classification network model consists of a fully connected layer that maps the number of output neurons of the fourth feature extraction module to the number of fine-grained cataract categories. This fully connected layer has weights and biases, both defined as floating-point variables.

3. The cataract fine-grained classification method based on slit-lamp images as described in claim 2, characterized in that, The patch mentioned in step 1.2 satisfies the condition that the kernel size K = the patch size P, and that the patch size P is also equal to the image size. The relationship between the window size W and the window size W satisfies , The number of kernels is an integer greater than 1; the number of kernels satisfies , It is an integer greater than or equal to 6.

4. The fine-grained cataract classification method based on slit-lamp images as described in claim 2, characterized in that, The four feature extraction modules described in step 1.3, with the initial image height N and width N, output image shapes for each feature extraction module are as follows: ;in Representing the Feature extraction module; In step 1.3, each sub-encoder performs window self-attention and offset window self-attention calculations to extract features within the window and features of connections between windows. The operations required for the window-internal self-attention mechanism include: LayerNorm normalization, calculation of window attention offset WA, random deactivation of neurons (Dropout), and calculation of the Multilayer Perceptron (MLP). The window attention offset WA and the MLP layers contain weights and biases, both defined as floating-point variables. The window-to-window self-attention mechanism is implemented by moving the window cyclically towards the top left corner, and then performing the window-internal self-attention calculation to extract features between windows. The formula for self-attention calculation is as follows: ； Building upon this, the attention mechanism is further extended to a multi-head self-attention mechanism, as shown in the following formula: ； ； in This indicates the multi-head self-attention mechanism (MSA). , , , All of these are learnable parameters, defined as floating-point variables; In addition, residual connections are used after each random dropout to make the training process more stable.

5. The cataract fine-grained classification method based on slit-lamp images as described in claim 2, characterized in that, Within the sub-encoder described in step 1.4, the input sizes of the corresponding LayerNorm, Window Attention Shift (WA), and Multilayer Perceptron (MLP) must correspond to the image size; simultaneously, the feature dimensions... The number of heads that must be able to be noticed Divisible by.

6. The cataract fine-grained classification method based on slit-lamp images as described in claim 1, characterized in that, The iterative training described in step 2.5, with its linear learning rate growth warm-up phase, is mainly to prevent the model from getting stuck in local optima. Therefore, the number of training rounds should be controlled to within twenty rounds. During the nonlinear learning rate decay process, a sine function is selected.

7. The cataract fine-grained classification method based on slit-lamp images as described in claim 1, characterized in that: Step three specifically includes: Step 3.1: Screening slit-lamp images; the screening requirements include: the eye structure is clearly visible, especially the lens region, which plays a decisive role in the fine-grained classification of cataracts; the lens region cannot be completely obscured by the slit-lamp light source, otherwise fine-grained features of cataracts cannot be extracted. Step 3.2: Categorize unbalanced cataract images; merge categories with a small number of slit-lamp images and similar clinical symptoms and treatments. Step 3.3: With existing labels, organize and classify the dataset after processing the imbalanced images. The final fine-grained cataract categories include: nuclear type II, nuclear type III, ≥ nuclear type IV, mild cortical type, intermediate cortical type, severe cortical type, fully mature, posterior cystic, and healthy. These fine-grained cataract categories correspond to the final output of the fine-grained cataract classification network. Divide the data into training, validation, and test sets in a certain proportion, and ensure that the distribution of each fine-grained cataract class is consistent across these datasets.

8. The cataract fine-grained classification method based on slit-lamp images as described in claim 1, characterized in that: Step four specifically includes: Step 4.1: Test the saved optimal model on the test set. The test metrics include precision, recall, F1-score, and average accuracy. Step 4.2: Calculate the accuracy of predicting each of the nine fine-grained categories of cataracts, and demonstrate the effectiveness of the method through experiments; Step 4.3: Shuffle the source dataset and randomly sample it to test the accuracy of this fine-grained cataract classification on randomly distributed data.

9. A system for implementing the fine-grained cataract classification method based on slit-lamp images as described in claim 1, characterized in that: include: The cataract fine-grained classification network framework construction module is used to build a cataract fine-grained classification network framework. The cataract fine-grained classification network training module is used to train the cataract fine-grained classification network. The dataset preprocessing module is used to preprocess datasets; The cataract classification test module is used to classify and test cataracts.

Images