Cornea conus image classification method based on improved deep learning self-attention mechanism

Abstract

The invention provides a keratoconus image classification method based on an improved deep learning self-attention mechanism, and the method comprises the steps: obtaining a four-refraction report graph through a three-dimensional anterior segment analysis and diagnosis system, carrying out the image preprocessing of the four-refraction report graph through employing a method for extracting a region of interest, and obtaining a four-refraction report graph data set; constructing a network model based on a multi-directional axis attention mechanism, and inputting the four-refraction report graph data set into the constructed network model based on the multi-directional axis attention mechanism for training to obtain a trained network model based on the multi-directional axis attention mechanism; and carrying out image preprocessing and data enhancement on the four-refraction report graph to be processed, and inputting the four-refraction report graph to be processed into the trained network model based on the multi-directional axis attention mechanism to carry out cornea conus classification. According to the method, unique directional feature information in the corneal topographic map is efficiently captured, the information of the whole image can be comprehensively considered, and the diagnosis accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of corneal diseases and image processing, and in particular to a keratoconus image classification method. Background Art

[0002] Keratoconus is an ophthalmic disease that often occurs in adolescence, characterized by significant expansion and thinning of the central or near-central area of ​​the cornea, leading to severe vision loss or even blindness. Before undergoing refractive surgery such as Laser-Assisted In Situ Keratomileusis (LASIK) and Photorefractive Keratectomy (PRK), it is crucial to accurately assess the corneal condition. Therefore, improving the accuracy of keratoconus diagnosis has become an urgent problem to be solved. Pentacam HR is a powerful tool often used in clinical practice that can help ophthalmologists detect and find keratoconus.

[0003] At present, keratoconus is mainly diagnosed by doctors observing corneal topography. On the one hand, doctors' subjective interpretation of corneal topography may vary due to personal experience, skill level and judgment criteria. This may lead to inconsistency in diagnostic results and increased misdiagnosis rates. On the other hand, although corneal topography provides morphological information on the corneal surface, it may be difficult to accurately identify early or subtle changes in keratoconus based on the doctor's subjective judgment alone. Especially in the subclinical stage, the signs of keratoconus may not be obvious, which increases the difficulty of diagnosis. There is currently a lack of unified diagnostic criteria for keratoconus, and doctors may lack clear guidance when interpreting corneal topography. This may lead to differences in diagnostic results between different doctors, and even inconsistent diagnostic results may occur at different time points for the same doctor.

[0004] The attention mechanism was first introduced in the field of natural language processing and machine translation and achieved remarkable results. Subsequently, in the field of computer vision, researchers have also been committed to exploring the integration of attention mechanisms into visual processing and convolutional neural networks, aiming to optimize network performance. Various attention mechanisms, such as spatial attention, channel attention, and self-attention, have been proven to effectively improve network performance. In the image classification task of deep learning, there have also been many explorations on the application of attention mechanisms. For example, the Swin transformer uses self-attention based on shifted windows to replace the self-attention mechanism in the transformer, solving the problem of computational complexity in the transformer.

[0005] In the existing technology, it is difficult for the network model to fully capture the directional characteristics contained in the corneal topography, and it is difficult to effectively integrate various characteristic information such as the anterior surface axial curvature map, the anterior surface height map, the posterior surface height map, and the corneal thickness map. As an important ophthalmic diagnostic tool, the corneal topography contains rich directional characteristics, such as the inclination angle and curvature change of the corneal surface. However, when processing such data, the existing network model often cannot accurately capture these directional characteristics, which affects the accuracy of the diagnostic results.

[0006] At the same time, the morphological characteristics of the cornea are not limited to a single image, but need to comprehensively consider multiple image information such as the anterior surface axial curvature map, the anterior surface height map, the posterior surface height map, and the corneal thickness map. However, existing network models have difficulties in integrating this information and cannot effectively combine them, thus limiting the model's comprehensive understanding of corneal morphology.

[0007] In addition, spatial information dependency is also an important factor in corneal diagnosis. The morphology and characteristics of the cornea are often closely related to its position and relationship in space. However, existing network models are insufficient in extracting and utilizing these spatial information dependencies, which may cause the model to ignore this important information during the diagnosis process. Summary of the invention

[0008] In view of the fact that the existing technology is difficult to accurately capture the complex and diverse directional features in corneal topography, specifically the technical problem that it is difficult to identify and analyze subtle curvature changes and height fluctuations on the corneal surface in different axial directions, the present invention proposes a keratoconus image classification method based on an improved deep learning self-attention mechanism. The present invention introduces and optimizes the axial attention mechanism, and integrates it with the multi-head self-attention mechanism, so as to efficiently capture the unique directional feature information in the corneal topography, and can comprehensively consider the information of the entire image to improve the accuracy of diagnosis.

[0009] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0010] A keratoconus image classification method based on an improved deep learning self-attention mechanism comprises the following steps:

[0011] S1: Use the three-dimensional anterior segment analysis and diagnosis system to obtain four refraction report images, use the method of extracting regions of interest to preprocess the four refraction report images, divide and enhance the data after preprocessing, and label them to obtain a four refraction report image dataset;

[0012] S2: Construct a network model based on the multi-directional axis attention mechanism, including an image feature embedding module, a multi-axis encoder, and an MLP classification module connected in sequence;

[0013] S3: Input the four refraction report map data set into the constructed network model based on the multi-directional axis attention mechanism for training, and obtain the trained network model based on the multi-directional axis attention mechanism;

[0014] S4: After the image preprocessing and data enhancement in the same way as step S1, the four refractive report images to be processed are input into the trained network model based on the multi-directional axis attention mechanism for keratoconus classification to obtain the classification results of the four refractive report images to be processed.

[0015] The method described in step S1 for obtaining four-metropia report maps using a three-dimensional anterior segment analysis and diagnosis system is as follows: using a PENTACAM system, using a rotating Scheimpflug camera scanning method to quickly scan around the eyeball, generating a Pentacam corneal topography map, and extracting four-metropia report maps from the Pentacam corneal topography map.

[0016] The method for image preprocessing described in step S1 is: cropping and denoising the image in the four-refractive report image: using the OpenCV library in combination with edge detection and Hough transform, and using the contour search method to determine whether the aspect ratio of the circumscribed rectangle of the target contour in the four-refractive report image is close to the circle for circle detection, according to the circle detection result, obtaining the center coordinates and radius information of the detected circle, and generating an ROI mask, multiplying the generated ROI mask with the topographic heat map corresponding to the four-refractive report image, and retaining the part containing corneal information; normalizing and standardizing the cropped and denoised four-refractive report image; the division and data enhancement operations are: dividing the preprocessed four-refractive report image into a training set, a validation set and a test set in a ratio of 8:1:1, using a rotation operation to perform data enhancement on the images in the training set, and keeping the data of the validation set and the test set with the original data volume unchanged;

[0017] The labeling method is to label the images marked as keratoconus by the doctor as the KC group, and the images of normal cornea as the control group.

[0018] The multi-axis encoder in step S2 includes multiple shallow coding layers and multiple deep coding layers connected in sequence. Each shallow coding layer adopts a multi-directional axial attention mechanism based on diagonal axis simulation to obtain multi-directional features, and each deep coding layer adopts a multi-head self-attention mechanism to obtain global features.

[0019] The plurality of shallow coding layers are 2 layers, and each shallow coding layer includes a multi-directional axial attention module I based on diagonal axis simulation, an overlapping downsampling module I, and a multi-directional axial attention module II based on diagonal axis simulation, which are connected in sequence;

[0020] The multiple deep coding blocks are 2 layers, including a first deep coding layer and a second deep coding layer. The first deep coding layer includes an overlapping downsampling module II and a multi-head self-attention module I connected in sequence, and the second deep coding layer includes a block merging module and a multi-head self-attention module II connected in sequence.

[0021] The processing processes in the multi-directional axial attention module I and the multi-directional axial attention module II are both: the input feature tensor is subjected to a convolution operation to obtain a feature tensor A, and the feature tensor A is then residually connected with the input feature tensor to obtain a feature tensor X. After the feature tensor X is subjected to multi-directional axial attention calculation, the multi-directional axial attention calculation result is residually connected with the feature tensor X to obtain a feature tensor B. The feature tensor B is subjected to linear operations and a multi-layer perceptron in turn, and the feature tensor B is residually connected with the output result of the multi-layer perceptron to output the final feature tensor.

[0022] The method for calculating multi-directional axial attention is: using the attention score calculation formula to perform horizontal axis attention calculation and vertical axis attention calculation on the feature tensor X respectively, performing attention calculation in the simulated diagonal axis direction, and connecting the results of the horizontal axis attention calculation, the vertical axis attention calculation and the simulated diagonal axis direction attention calculation to obtain the multi-directional axis attention calculation result.

[0023] The formula for calculating multi-directional axial attention is:

[0024]

[0025] out=LayerNorm[Dropout(∑ ad attn_outputs[ad])]

[0026] Where ad represents the horizontal axis, vertical axis or diagonal axis, Q ad , K ad and V ad Represents query, key and value respectively, d k is the dimension of the key, softmax() is the activation function, LayerNorm[] is the layer normalization function, Dropout[] is the Dropout operation function, attn_outputs[ad] is the attention calculation output along each axis, and out is the output of the multi-directional axial attention calculation.

[0027] The image feature embedding module divides the input image into n small blocks, and the input feature tensor X corresponds to the input image; the method for simulating the attention calculation in the diagonal axis direction is: reorganizing the feature tensor X by dividing the feature tensor X into four quadrants, each of which contains small block, all the The small block and all the The small blocks are exchanged, and then the first quadrant and the second quadrant are spliced, and the third quadrant and the fourth quadrant are spliced ​​to obtain a reorganized feature tensor. The attention calculation of the horizontal axis is performed on the reorganized feature tensor to obtain the result of the attention calculation in the simulated diagonal axis direction.

[0028] The training method described in step S3 is: inputting the training set into the network model based on the multi-directional axis attention mechanism, dividing the input image into n small blocks through the image feature embedding module and generating an embedded representation; using the multi-axis encoder to extract the local and global information contained in the embedded representation; using the MLP classification module to classify according to the local and global information, and output the final predicted classification result; comparing the output predicted classification result with the label corresponding to the image, using L1 regularization combined with binary classification CrossEntropyLoss as the loss function to calculate the loss, passing the calculated loss to the model, updating the parameters in the model, and using the verification set to verify the model and adjust the hyperparameters until the model converges.

[0029] Beneficial effects of the present invention:

[0030] 1. By introducing a multi-directional axis attention mechanism and fusing it with a multi-head self-attention mechanism, the present invention can more comprehensively capture the local and global features in corneal topography images, thereby significantly improving the accuracy of keratoconus classification.

[0031] 2. Although a complex attention mechanism is introduced, the present invention successfully reduces the computational complexity by optimizing the attention calculation method and improving the basic network structure design in a targeted manner. This enables the model to process a large amount of image data more efficiently while maintaining high performance, thereby improving the processing speed and efficiency in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0034] Figure 2 This is a network structure diagram proposed by the present invention.

[0035] Figure 3 Schematic diagram of the training process of the model of the present invention.

[0036] Figure 4 Schematic diagram of data preprocessing and data enhancement concepts.

[0037] Figure 5 Schematic diagram simulating the concept of "diagonal axis" attention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] A keratoconus image classification method based on improved deep learning self-attention mechanism, such as Figure 1 As shown, the steps include:

[0040] S1: Use the three-dimensional anterior segment analysis and diagnosis system to obtain four refractive index reports, use the method of extracting regions of interest to preprocess the four refractive index reports, and obtain a four refractive index report data set. The preprocessed four refractive index reports are divided and data enhanced, and then labeled to obtain a four refractive index report data set.

[0041] The three-dimensional anterior segment analysis and diagnosis system adopts the PENTACAM system. The PENTACAM system uses the rotating Scheimpflug camera scanning principle to quickly scan around the eyeball to generate a Pentacam corneal topography map. Key information is extracted from the Pentacam corneal topography map to obtain a four-mirror report map. The four-mirror report map is a topographic heat map that focuses on reflecting the refractive properties and morphological changes of the cornea. This chart contains the anterior surface axial curvature map, the anterior surface height map, the posterior surface height map and the corneal thickness. Figure 4 The four parts are represented by four circular topographic maps. Each of the four parts analyzes a specific aspect of the cornea. This information is important for assessing corneal health, diagnosing corneal diseases, and guiding ophthalmic surgery. Medical research requires a comprehensive judgment based on the four parts.

[0042] The image preprocessing method is as follows: first, the image in the four-refractive report map is cropped and denoised, specifically, the contour search method is used to detect the circle by judging whether the aspect ratio of the circumscribed rectangle of the contour is close to the circle, and the OpenCV library is used in combination with edge detection and Hough transform to complete this process. Then, according to the circle detection result, the center coordinates and radius information of the detected circle are obtained to generate an ROI mask. In actual operation, it can be completed by creating a mask map of the same size as the image, setting the detected circle area to white and the rest to black. Then, the generated ROI mask is multiplied by the topographic heat map corresponding to the four-refractive report map, and the background area of ​​the topographic heat map is set to white, leaving only the part containing corneal information. Finally, the cropped and denoised four-refractive report map is normalized and standardized.

[0043] The preprocessed four-refractive report map data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. By performing data enhancement operations on the training set, the validation set and the test set data keep the original data volume unchanged, and the capacity of the original data set is expanded. A richer data set can improve the generalization ability of the model and reduce the risk of overfitting. The method of the data enhancement operation is: the specific operation is to generate new training samples by rotating the images in the corneal topography data set (rotating the four circular areas containing corneal information by the same angle respectively), which can increase the diversity of samples and maintain the consistency of color and space in the image. Using a richer data set can improve the generalization ability of the model and reduce the risk of overfitting. The images in the experimental data set are divided into two categories: images marked as keratoconus by doctors are called KC group, and images of normal corneas are marked as control group.

[0044] S2: Construct a network model based on the multi-directional axis attention mechanism, including an image feature embedding module, a multi-axis encoder, and an MLP classification module connected sequentially.

[0045] First, the VIT framework is constructed, and the self-attention mechanism is replaced by a fusion attention mechanism of a hierarchical multi-directional axial attention mechanism and a multi-head self-attention mechanism based on the VIT network.

[0046] The image feature embedding module is used to divide the input image into multiple small patches (patches) and flatten these patches to form features suitable for model processing. The image is divided into small patches (the size is specified by the parameter) through the convolution layer, and each small patch is flattened into a one-dimensional vector to form a new embedding representation.

[0047] The multi-axis encoder includes two shallow coding layers and two deep coding layers connected in sequence. Figure 2Transformer layer 1 and transformer layer 2 shown; deep encoding layers such as Figure 2 Transformer layer 3 and transformer layer 4 are shown. In each shallow coding layer, a multi-directional axial attention mechanism based on diagonal axis simulation is used to obtain multi-directional features, and each deep coding layer uses a multi-head self-attention mechanism to obtain global features.

[0048] Transformer layer 1 and transformer layer 2 both include two shallow coding blocks, both of which are 2-layer structures. The shallow coding block is a multi-directional axial attention module based on diagonal axis simulation; transformer layer 1 includes a multi-directional axial attention module Ⅰ1 based on diagonal axis simulation, an overlapping down-sampler module (Overlapping Down-sampler) Ⅰ1, and a multi-directional axial attention module Ⅱ1 based on diagonal axis simulation, which are connected in sequence; transformer layer 2 includes a multi-directional axial attention module Ⅰ2 based on diagonal axis simulation, an overlapping down-sampler module Ⅰ2, and a multi-directional axial attention module Ⅱ2 based on diagonal axis simulation, which are connected in sequence; the multi-directional axial attention module Ⅰ2 based on diagonal axis simulation is connected to the multi-directional axial attention module Ⅱ1 based on diagonal axis simulation; the network structure diagram is shown in FIG. Figure 2 As shown in the figure, the right part shows the proposed multi-directional axis attention module based on diagonal axis simulation, which mainly includes deep separable convolution (DWConv), multi-directional axis attention operation and multi-layer perceptron. The multi-directional axis attention module combines multi-directional axial attention and multi-layer perceptron to enhance feature representation. The feature tensor processed by attention and multi-layer perceptron is output.

[0049] The processing processes in the multi-directional axial attention module I and the multi-directional axial attention module II are both: the input feature tensor is subjected to a convolution operation to obtain a feature tensor A, and the feature tensor A is then residually connected with the input feature tensor to obtain a feature tensor X. After the feature tensor X is subjected to multi-directional axial attention calculation, the multi-directional axial attention calculation result is residually connected with the feature tensor X to obtain a feature tensor B. The feature tensor B is subjected to linear operations and a multi-layer perceptron in turn, and the feature tensor B is residually connected with the output result of the multi-layer perceptron to output the final feature tensor C.

[0050] The multi-directional axial attention calculation: The input feature tensor of the multi-directional axial attention mechanism is X, represented by a matrix X0 with a dimension of n×m. The attention score of each element of the matrix X0 is obtained by performing attention calculation along the horizontal axis: For each row of the input matrix X0, it is regarded as a sequence, and a query Q about the horizontal axis is generated through a one-dimensional convolution row , key K row and V row, where the horizontal axis attention score is calculated:

[0051]

[0052] attn-output row =Attention row ×V row

[0053] Among them, d k is the dimension of the query or key, Attention row is the horizontal axis attention weight, attn-output row is the horizontal attention score, usually C / num_heads; C is the number of channels, indicating the dimension of the input feature vector, num_heads indicates the number of attention heads, the number of attention heads of shallow coding blocks is 3 and 6; the number of attention heads of deep coding blocks is 12 and 24.

[0054] Calculate the vertical axis attention score: transpose the input matrix X0 and repeat the above steps to get the vertical axis attention score attn-output colum ;

[0055] Calculate the diagonal axis attention score: The diagonal axis attention score is calculated by simulating the attention in the diagonal axis direction. Specifically: Figure 5 For example, the input feature tensor X corresponds to the input image. The input feature tensor X is divided into a series of fixed-size patches, taking 16×16 pixels as an example. You can modify it as needed when using it. Just make sure that the number of patches divided is 2. 2n , n=1,2,3…, at this time, the feature tensor X is divided into 16 patches. Consider the feature tensor X as a 4×4 grid layout, and the entire image is evenly divided into four quadrants, each containing 4 patches. Next, a specific inter-quadrant patch reorganization operation is performed. Specifically, all 4 patches in the second quadrant (upper left corner) are exchanged with all 4 patches in the third quadrant (lower left corner). This means that the patch originally located in the upper left corner of the image is now moved to the lower left corner, and the patch originally located in the lower left corner is moved to the upper left corner. This exchange operation keeps the number and arrangement order of patches in each quadrant unchanged, but swaps patches in different quadrants, and then splices the first and second quadrants and the third and fourth quadrants to obtain the reorganized feature tensor. Subsequently, a horizontal axis attention calculation is performed on the operated image matrix, thereby effectively simulating the diagonal axis attention mechanism and obtaining the diagonal axis attention score attn-output Diagonal, the steps for patch exchange between quadrants are:

[0056] The first step is to segment the image and divide the feature tensor X into four quadrants:

[0057] Quadrant 1: top_left = X[:,∶mid_h,∶mid_w,∶]

[0058] Quadrant 2: top_right=X[:,:mid_h,mid_w:,:]

[0059] Quadrant 3: bottom_left=X[:,mid_h:,:mid_w,:]

[0060] Quadrant 4: bottom_right=X[:,mid_h:,mid_w:,:]

[0061] Among them, mid_h and mid_w respectively represent the height and width of the quadrant to be divided, mid_h=height / 2, mid_w=width / 2, height represents the height, and width represents the width.

[0062] Step 2: Exchange quadrants: Exchange the second and third quadrants:

[0063] top_left,bottom_left=bottom_left,top_left

[0064] The third step is patch reorganization between quadrants:

[0065] First, concatenate the upper half (the first and second quadrants) into:

[0066] top=torch.cat([top_left,top_right],dim=2)

[0067] Among them, torch.cat represents the splicing operation, and dim=2 means splicing along the width dimension.

[0068] Then, the lower half (the third and fourth quadrants) are concatenated as follows:

[0069] bottom=torch.cat([bottom_left,bottom_right],dim=2)

[0070] Finally, concatenate the top and bottom halves together to get the reorganized tensor:

[0071] recombined=torch.cat([top,bottom],dim=1)

[0072] dim=1 means stitching along the height dimension.

[0073] Transformer layer 3 and transformer layer 4 both include two deep encoding blocks, with 6 layers and 2 layers respectively. The deep encoding block is a multi-head self-attention module. Transformer layer 3 includes overlapping downsampling module II and multi-head self-attention module I connected in sequence. The input of overlapping downsampling module II is connected to the output of multi-directional axial attention module II2 based on diagonal axis simulation. Transformer layer 4 includes patch merging module and multi-head self-attention module II connected in sequence. The input of patch merging module is connected to the output of head self-attention module I. Patch merging module is used to merge features of multiple patches, reduce the dimension of features, and enhance the abstract ability of the model. Output the merged feature tensor with reduced dimension. Multi-head self-attention module II is connected to MLP classification module, and multi-head self-attention is used to consider the global context information of input to further enhance feature expression. Output the feature tensor that integrates global context information.

[0074] The first two shallow encoding blocks use a multi-directional axial attention mechanism to replace the multi-head self-attention mechanism, and the last two deep encoding blocks continue to use the multi-head self-attention mechanism. The multi-axis encoder is used to extract features from the input feature embedding representation to obtain local and global features;

[0075] The MLP classification module is used to output the classification results. In the MLP classification module, after all the encoding layers are processed, the extracted local and global features will pass through the fully connected layer, which is responsible for mapping the high-dimensional feature vector to a lower-dimensional space, further abstracting and compressing the feature information. Finally, the data enters the classification layer, which is a fully connected layer containing a Softmax activation function. It is classified according to the feature tensor extracted by the fully connected layer and outputs the final predicted classification result, which represents the category probability of each sample. This hierarchical structure enables the model to gradually extract, abstract and finally classify complex image information from the input image.

[0076] S3: Input the four-refractive report image dataset into the constructed network model based on the multi-directional axis attention mechanism for training, and obtain a trained network model based on the multi-directional axis attention mechanism.

[0077] Training steps: The training set is input into the network model based on the improved self-attention machine. The input image (686×686 pixels) is uniformly adjusted to 224×224 pixels through the image feature embedding module. The input image is then divided into small blocks (16×16 pixels) and each small block is converted into a fixed-length feature vector through a linear embedding layer. At the same time, position embedding is added to retain the relative position information between image small blocks, thereby generating a feature-oriented embedding representation that can be used for processing by the improved encoding module.

[0078] In the shallow encoding block of the multi-axis encoder, the feature tensor is subjected to multi-directional axial attention calculation to extract the local detail features of the image. By integrating the attention mechanism of the horizontal axis, vertical axis and diagonal axis, the intrinsic relationship between the four parts of the four-refraction report map in terms of imageology can be revealed. In the deep encoding block of the multi-axis encoder, the entire image is encoded by multi-head self-attention calculation to generate a feature tensor containing global information. This feature tensor containing global information effectively captures the key global features of the image, such as the overall shape and color distribution, and provides important information support for the subsequent keratoconus classification task. In order to effectively fuse local and global features, a feature fusion strategy is adopted. The output of the multi-directional axis attention calculation is spliced ​​with the output of the multi-head self-attention calculation through the block merging module to obtain a comprehensive feature vector, which contains both local and global information. Next, the fused comprehensive feature vector is subjected to global average pooling in the block merging module to reduce the feature dimension. Although this process reduces the dimension, it still represents the key information of the high-dimensional features, thereby reducing the amount of calculation while retaining the most important information. After being processed by multiple coding layers, the features of the improved coding module are finally output;

[0079] The MLP classification module is used to perform classification based on local and global information, and output the final predicted classification results.

[0080] Compare the output prediction classification result with the label corresponding to the image, and use L1 regularization combined with binary classification CrossEntropyLoss as the loss of the current model. The formula is as follows:

[0081]

[0082] Where N is the number of samples, that is, the total number of samples in the training data; y i is the true label (0 or 1) of the i-th sample, p i is the model's predicted probability that the i-th sample belongs to the positive class, W j is the weight parameter in the model, λ is the regularization coefficient, and P is the total number of weight parameters.

[0083] The calculated loss is passed to the model, the parameters in the model are updated, and the model is verified using the validation set to adjust the hyperparameters until the model converges.

[0084] In order to reduce the optimization difficulty of the model and accelerate the training of the model during model training, the training method of transfer learning is used. The specific implementation steps are: first use the processed data set to train the transformer basic network. The transformer basic network is the basic network for image classification, which contains the necessary structure for image classification, including image feature embedding, encoder and MLP classification head. Then, based on the parameters trained by the transformer basic network, the multi-head self-attention mechanism is replaced with the multi-directional axial attention module proposed in the present invention. The multi-head self-attention mechanism is hierarchically combined with the attention mechanism, and the parameters of the entire network are updated through training. During the network training process, the Nvidia GeForceRTX3090 GPU is used to train the network, epoch=50, batchsize=32, and learning rate=1e-2. Stop training when the network converges.

[0085] S4: The four refractive report images to be processed are subjected to the same image preprocessing and data enhancement as step S1 and then input into the trained network model based on the multi-directional axis attention mechanism for keratoconus classification.

[0086] In order to demonstrate the generalization performance of the present invention, the model was applied to a previously untouched test set for performance evaluation. The experimental results are shown in Table 1. The performance of the model of the present invention on complex medical data sets is significantly improved. Specifically, the network model of the present invention achieves an accuracy of 95.7% and a recall of 92.3% for the task of diagnosing keratoconus. Compared with the basic ViT model (accuracy of 90.2%), the performance has been significantly improved. This progress is not only reflected in numerical indicators, but more importantly, it is expected to improve the early detection rate of keratoconus in practical applications, thereby improving the treatment effect and prognosis of patients. By optimizing resource utilization and improving diagnostic efficiency, the present invention provides strong technical support for the early screening and treatment of keratoconus.

[0087] Table 1 Network model comparison experimental results

[0088]

[0089] This method is based on an improved self-attention mechanism and integrates it into the visual Transformer (ViT) model, which is particularly suitable for the field of image processing. The present invention introduces and optimizes the multi-directional axial attention mechanism and integrates it with the multi-head attention mechanism to construct a deep learning network specifically for the classification task of keratoconus. The multi-directional axial attention mechanism is used to efficiently capture the unique directional feature information in the corneal topography while maintaining a low computational cost. The global attention mechanism is responsible for comprehensively and deeply extracting the global features in the image, ensuring that the model can comprehensively consider the information of the entire image and improve the accuracy of diagnosis.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A keratoconus image classification method based on improved deep learning self-attention mechanism, characterized in that: Includes steps: S1: Use the three-dimensional anterior segment analysis and diagnosis system to obtain four refraction report images, use the method of extracting regions of interest to preprocess the four refraction report images, divide and enhance the data after preprocessing, and label them to obtain a four refraction report image dataset; S2: Construct a network model based on the multi-directional axis attention mechanism, including an image feature embedding module, a multi-axis encoder, and an MLP classification module connected in sequence; S3: Input the four refraction report map data set into the constructed network model based on the multi-directional axis attention mechanism for training, and obtain a trained network model based on the multi-directional axis attention mechanism; S4: After the image preprocessing and data enhancement in the same way as step S1, the four refractive report images to be processed are input into the trained network model based on the multi-directional axis attention mechanism for keratoconus classification to obtain the classification results of the four refractive report images to be processed.

2. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 1, characterized in that: The method described in step S1 for obtaining four-metropia report maps using a three-dimensional anterior segment analysis and diagnosis system is as follows: using a PENTACAM system, using a rotating Scheimpflug camera scanning method to quickly scan around the eyeball, generating a Pentacam corneal topography map, and extracting four-metropia report maps from the Pentacam corneal topography map.

3. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 2, characterized in that: The method for image preprocessing described in step S1 is: cropping and denoising the image in the four-metropia report image: using the OpenCV library in combination with edge detection and Hough transform, and using the contour search method to determine whether the aspect ratio of the circumscribed rectangle of the target contour in the four-metropia report image is close to a circle for circle detection, according to the circle detection result, obtaining the center coordinates and radius information of the detected circle, and generating a ROI mask, multiplying the generated ROI mask with the topographic heat map corresponding to the four-metropia report image, and retaining the part containing corneal information; normalizing and standardizing the cropped and denoised four-metropia report image; The division and data enhancement operations are as follows: the preprocessed four refraction report images are divided into a training set, a validation set and a test set in a ratio of 8:1:1, and the images in the training set are enhanced by a rotation operation, and the data of the validation set and the test set are preserved with the original data volume unchanged; The labeling method is to label the images marked as keratoconus by the doctor as the KC group, and the images of normal cornea as the control group.

4. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 3, characterized in that: The multi-axis encoder in step S2 includes multiple shallow coding layers and multiple deep coding layers connected in sequence. Each shallow coding layer adopts a multi-directional axial attention mechanism based on diagonal axis simulation to obtain multi-directional features, and each deep coding layer adopts a multi-head self-attention mechanism to obtain global features.

5. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 4, characterized in that: The plurality of shallow coding layers are 2 layers, and each shallow coding layer includes a multi-directional axial attention module I based on diagonal axis simulation, an overlapping downsampling module I, and a multi-directional axial attention module II based on diagonal axis simulation, which are connected in sequence; The multiple deep coding blocks are 2 layers, including a first deep coding layer and a second deep coding layer. The first deep coding layer includes an overlapping downsampling module II and a multi-head self-attention module I connected in sequence, and the second deep coding layer includes a block merging module and a multi-head self-attention module II connected in sequence.

6. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 5, characterized in that: The processing processes in the multi-directional axial attention module I and the multi-directional axial attention module II are both: the input feature tensor is subjected to a convolution operation to obtain a feature tensor A, and the feature tensor A is then residually connected with the input feature tensor to obtain a feature tensor X. After the feature tensor X is subjected to multi-directional axial attention calculation, the multi-directional axial attention calculation result is residually connected with the feature tensor X to obtain a feature tensor B. The feature tensor B is subjected to linear operations and a multi-layer perceptron in turn, and the feature tensor B is residually connected with the output result of the multi-layer perceptron to output the final feature tensor.

7. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 6, characterized in that: The method for calculating multi-directional axial attention is: using the attention score calculation formula to perform horizontal axis attention calculation and vertical axis attention calculation on the feature tensor X respectively, performing attention calculation in the simulated diagonal axis direction, and connecting the results of the horizontal axis attention calculation, the vertical axis attention calculation and the simulated diagonal axis direction attention calculation to obtain the multi-directional axis attention calculation result.

8. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 7, characterized in that: The formula for calculating multi-directional axial attention is: out=LayerNorm[Dropout(∑ ad attn_outputs[ad])] Where ad represents the horizontal axis, vertical axis or diagonal axis, Q ad , K ad and V ad Represents query, key and value respectively, d k is the dimension of the key, softmax() is the activation function, LayerNorm[] is the layer normalization function, Dropout[] is the Dropout operation function, attn_outputs[ad] is the output of the attention calculation along each axis, and out is the output of the multi-directional axial attention calculation.

9. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 8, characterized in that: The image feature embedding module divides the input image into n small blocks, and the input feature tensor X corresponds to the input image; the method for simulating the attention calculation in the diagonal axis direction is: reorganizing the feature tensor X by dividing the feature tensor X into four quadrants, each of which contains small block, all the The small block and all the The small blocks are exchanged, and then the first quadrant and the second quadrant are spliced, and the third quadrant and the fourth quadrant are spliced ​​to obtain a reorganized feature tensor. The attention calculation of the horizontal axis is performed on the reorganized feature tensor to obtain the result of the attention calculation in the simulated diagonal axis direction.

10. The keratoconus image classification method based on improved deep learning self-attention mechanism according to claim 9, characterized in that: The training method in step S3 is: inputting the training set into a network model based on a multi-directional axis attention mechanism, dividing the input image into n small blocks and generating an embedded representation through an image feature embedding module; and extracting local and global information contained in the embedded representation using a multi-axis encoder; The MLP classification module is used to classify according to local and global information, and the final predicted classification results are output; the output predicted classification results are compared with the labels corresponding to the images, and L1 regularization combined with binary classification CrossEntropyLoss is used as the loss function to calculate the loss. The calculated loss is passed to the model, the parameters in the model are updated, and the model is verified using the validation set to adjust the hyperparameters until the model converges.

Images