Meibomian gland deletion grading method based on convolutional neural network and Transform technology

Abstract

The invention relates to the technical field of meibomian gland image processing, in particular to a meibomian gland deletion grading method based on a convolutional neural network and a Transform technology. Obtaining a meibomian gland image data set, and introducing a fast forward algorithm to carry out restoration processing on the meibomian gland image data set so as to reduce the gland boundary ambiguity in the meibomian gland image caused by tear reflection and improve the sensitivity of the segmentation model to boundary features; constructing a multi-granularity meibomian gland segmentation model, inputting the processed meibomian gland image data set into the multi-granularity meibomian gland segmentation model, and training the multi-granularity meibomian gland segmentation model based on a convolutional neural network and a Transform technology; according to the meibomian gland image segmentation method, the meibomian gland image segmentation efficiency is improved, the robustness and consistency of the segmentation result are enhanced, and technical support is provided for rapid diagnosis and accurate evaluation of meibomian gland dysfunction.

Description

Technical Field

[0001] The present invention relates to the technical field of meibomian gland image processing, and in particular to a meibomian gland deficiency grading method based on convolutional neural network and Transformer technology. Background Art

[0002] Meibomian glands are located in the tarsal plate of the eyelids. They are a type of sebaceous gland that is mainly responsible for synthesizing and secreting lipids to form the lipid layer of tears, effectively preventing tear evaporation and protecting the physiological environment of the eye. The function of the meibomian glands is closely related to their morphology. Factors such as blockage, inflammation, and abnormal hormone levels can lead to morphological changes and decreased oil secretion function, which in turn lead to meibomian gland dysfunction. Meibomian gland morphological loss is one of the important diagnostic indicators of meibomian gland dysfunction. The percentage of gland atrophy (the proportion of gland loss area to the total eyelid area) is an important parameter for assessing its severity, which can be classified using a standardized grading scale.

[0003] Early non-contact meibomian gland infrared photography technology is an important clinical examination method for evaluating the morphology of meibomian glands. The morphology, number, opening position and patency of meibomian glands can be observed through meibomian gland infrared images. Therefore, the study of meibomian gland infrared images has important research significance for the diagnosis and treatment of ocular surface diseases. However, when collecting meibomian gland infrared images, due to the influence of illumination, angle and human physiological factors, it is very easy to have image blur, uneven illumination and low contrast between glandular and non-glandular areas, which makes it difficult for doctors to make a diagnosis. In order to solve this problem, Koh et al. proposed an automatic gland segmentation method using Gabor filtering to extract imaging gland features. However, this method is only applicable to the analysis of upper eyelid images and is prone to false detection. Arita et al. used Wallis filter to enhance low-contrast areas in image processing, but it is still easy to cause false detection for meibomian gland images with poor imaging quality.

[0004] In recent years, the application of deep learning has provided new approaches to the diagnostic efficiency and accuracy of meibomian gland dysfunction. Researchers have been exploring various deep learning methods and architectures to better achieve the grading of meibomian gland deficiency, thereby accelerating the diagnostic efficiency of the disease. For example, although some researchers have used a simplified version of CNN to examine the morphology of the meibomian glands and tried to link their findings with clinical parameters, their research was based on a relatively limited dataset, which contained only 60 right eye images, of which 40 images were allocated for training and 20 for validation. Such a sample size may not be sufficient to prevent overfitting, which may lead to biased results. In contrast, Setu et al. introduced a unique deep learning model that aims to objectively evaluate morphometric parameters through segmentation of meibomian gland images to identify key glandular features such as number, length, width, and curvature, but for diagnosis, two model trainings are required.

[0005] In general, in the field of meibomian gland image segmentation, although deep learning-based methods have made significant progress, not only in improving segmentation accuracy, but also in promoting the improvement of diagnostic accuracy to a certain extent, there are still some limitations that need to be overcome. First, the current meibomian gland segmentation algorithm still has a lot of room for improvement in accuracy and result optimization, especially when dealing with complex backgrounds, glandular morphological changes or blurred boundaries, the segmentation performance has not yet reached the ideal level. Second, compared with traditional medical segmentation methods, deep learning-based technologies often only focus on a single target (such as the gland area) and do not fully integrate the synchronous segmentation of the eyelid area and the gland area. This fragmented processing may lead to reduced efficiency and increase the complexity of subsequent analysis and processing. In addition, the existing methods are also insufficient in model generalization and robustness. When faced with different image quality or acquisition conditions, the segmentation effect may be unstable. To solve these problems, it is necessary to further improve the model structure, integrate multi-task learning strategies, and optimize the feature extraction and regional synchronous segmentation capabilities, so as to improve the diagnostic efficiency and reliability. Summary of the invention

[0006] The present invention aims to solve the technical difficulties of the meibomian gland segmentation task and provides a meibomian gland loss grading method based on convolutional neural network and Transformer technology.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is:

[0008] The present invention discloses a method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology, comprising the following steps:

[0009] S1: Obtain a meibomian gland image dataset and introduce a fast march algorithm to repair the meibomian gland image dataset to reduce the fuzziness of the gland boundary in the meibomian gland image caused by tear reflection and improve the sensitivity of the segmentation model to boundary features;

[0010] S2: Construct a multi-granularity meibomian gland segmentation model, input the processed meibomian gland image dataset into the multi-granularity meibomian gland segmentation model, and train the multi-granularity meibomian gland segmentation model based on convolutional neural network and Transformer technology;

[0011] S3: Use the trained segmentation model to segment the meibomian gland images, calculate the gland missing rate, and perform grading.

[0012] Furthermore, in step S2, the processed meibomian gland image dataset is input into the multi-granularity meibomian gland segmentation model and the multi-granularity meibomian gland segmentation model is trained based on the convolutional neural network and the Transformer technology, including the following steps:

[0013] In the encoding stage, a convolutional neural network is used to extract features from the input meibomian gland image, reducing the image resolution by downsampling while increasing the number of channels to capture local features;

[0014] The meibomian gland image of size W×H is divided into several 16×16 non-overlapping image blocks, and the features of the image blocks are mapped to C dimensions through a linear embedding layer; after linear transformation, the image blocks are input into the corresponding Transformer module to capture global image features; where W is the width, H is the height, and C is the number of input channels.

[0015] Also includes:

[0016] For Transformer, position encoding needs to be added, and the process is as follows:

[0017]

[0018] Where E∈R (p2·C)×D Embedding projection for patches; E pos is the position embedding; z0 is the input of the Transformer module, which is a vector containing all image block features and position information; p is the size of each image block; R is the real number field; E is the linear embedding layer; C is the feature dimension after linear embedding; D is the hidden layer dimension of the Transformer model; is the feature of the first image block; is the feature of the second image block; is the feature of the Nth image block;

[0019] Moreover, the encoder consists of L layers of multi-head self-attention and multi-layer perceptron, so the output expression of the layer is as follows:

[0020] Z l′ =MSA(LN(Z l-1 ))+Z l-1

[0021] Z l =MLP(LN(Z l′ ))+Z l′

[0022] Where LN(·) represents the layer normalization operator; Z l is the encoded image representation; MSA stands for the multi-head self-attention mechanism, which is used to capture the relationship between image blocks; MLP stands for multi-layer perceptron, which is used to perform nonlinear transformation on features; Z l-1 Represents the output of the encoder layer l-1, that is, the output of the previous layer; Z l′ Represents the output of the multi-head self-attention module in the encoder layer l′.

[0023] Furthermore, in step S2, the processed meibomian gland image dataset is input into the multi-granularity meibomian gland segmentation model and the multi-granularity meibomian gland segmentation model is trained based on the convolutional neural network and the Transformer technology, and the following steps are also included:

[0024] In the process of skip connection, multiple fusion dense skip connections are introduced;

[0025] During the meibomian gland image sampling process, the feature maps generated by downsampling at each stage are saved, and the adjacent feature maps are aligned in size to have the same resolution through interpolation or pooling;

[0026] Two 3x3 convolutional layers are used to process the aligned feature maps to extract image features. The feature extraction process is repeated, and each time the low-level and high-level features are fused to obtain three feature maps of different sizes.

[0027] After obtaining the feature map, the channel attention mechanism aggregates the global information of each channel through global pooling, compresses the spatial features of each channel into a single value, and forms a global representation of each channel; the aggregated features are then passed through a fully connected layer or a small neural network to calculate the importance of each channel, thereby obtaining the weight of each channel;

[0028] The calculated weights are normalized using an activation function to generate a scaling factor for each channel, indicating the relative importance of the channel in the current task;

[0029] The calculated channel weights are multiplied by the original feature map channel by channel, and the feature map of each channel is weighted and adjusted so that more important channels are strengthened and unimportant channels are suppressed.

[0030] Through the above process, the attention mechanism can adaptively adjust the contribution of different channels, thereby improving the performance of the model in the task, enhancing the model's ability to focus on key information, and improving the accuracy and robustness of feature extraction.

[0031] In order to achieve accurate segmentation of eyelids and glands, the decoder is designed as two independent branches. The branches share the features extracted by the encoder, but are trained using different labels and loss functions, thereby independently generating segmentation results for eyelids and glands.

[0032] At each stage of decoding, the upsampled feature map is fused with the feature map of the corresponding stage of the encoder through skip connections; the fused feature map is further processed by a convolutional layer to enhance the ability to distinguish between categories; through this iterative process, the resolution of the feature map is gradually restored to be consistent with the input image;

[0033] In the final stage of the decoder, a 1×1 convolutional layer is used to convert the high-dimensional feature map into a category score map. The score of each pixel represents the probability of it belonging to a different category, and finally an accurate segmentation result is generated.

[0034] Also includes:

[0035] Calculate the loss function and optimize the neuron weight parameters;

[0036] When training the multi-granularity meibomian gland segmentation model, the error between the predicted quality score and the target quality score output by the multi-granularity meibomian gland segmentation model is continuously reduced, and the Adam optimization algorithm is used to automatically update the learning rate and continuously optimize the weight parameters of the multi-granularity meibomian gland segmentation model;

[0037] Among them, the batch-size in the Adam automatic update learning rate optimization algorithm is set to 4, the learning rate is initialized to 0.00009, and the weight decay factor is set to 0.0001; the activation function uses the ReLU activation function; and the loss function is obtained by a linear combination of two different loss functions.

[0038] The loss function calculation formula is as follows:

[0039] Loss=λ1×WCE+λ2DiceLoss(X,Y)

[0040] Where Loss represents the final loss function value; λ1 represents the first weighting coefficient, which is used to control the weight of the weighted cross entropy loss in the total loss; λ2 represents the second weighting coefficient, which is used to control the weight of the Dice loss in the total loss; WCE represents the weighted cross entropy loss; DiceLoss(X,Y) represents the Dice loss function; X represents the prediction result of the model; Y represents the target result of the model;

[0041] Among them, WCE represents the weighted cross entropy loss function, which is expressed as follows:

[0042]

[0043] In the formula, w i is the weight of the category to which sample i belongs; y i is a binary value corresponding to the true label; is the probability predicted by the model; DiceLoss(X,Y) represents the Dice loss function.

[0044] Among them, the expression of DiceLoss(X,Y) is:

[0045]

[0046] Among them, X and Y represent the prediction results and the true labels, X and Y represent their regions respectively, and |X∩Y| represents the intersection of the two regions.

[0047] DSC and MIoU are used to evaluate the segmentation results, and the calculation formula is:

[0048]

[0049] Where DSC is the Dice similarity coefficient; X and Y represent the area of ​​the segmentation result and the label, respectively, and |X∩Y| represents the area of ​​the overlap between the segmentation result and the label;

[0050]

[0051] Where MIoU is the mean intersection over union ratio; TP represents the number of correctly segmented target pixels, that is, the number of pixels that are actually positive examples and predicted as positive examples; FP represents the number of incorrectly segmented target pixels, that is, the number of pixels that are actually negative examples but predicted as positive examples; FN represents the number of incorrectly segmented target pixels, that is, the number of pixels that are actually positive examples but predicted as negative examples; k represents the number of segmented targets.

[0052] Furthermore, the trained segmentation model is used in step S3 to segment the meibomian gland image, and the gland loss rate is calculated and graded, specifically:

[0053] The remaining 20% ​​of the data in the meibomian gland image dataset is input into the trained network model for quality prediction; binary images of the gland and eyelid area are predicted;

[0054] The glandular absence rate is calculated and graded using the binary image of the glandular area compared to the binary image of the upper eyelid area. The calculation formula is as follows:

[0055]

[0056] Among them, Density is the glandular loss rate; Smg i is the area of ​​the meibomian gland, in pixels; St is the area of ​​the tarsal plate; ∑smg i It is the sum of the areas of all meibomian glands on each tarsal plate.

[0057] The meibomian gland image dataset is randomly divided into two parts, the first part is 80% and the second part is 20%; wherein 80% of the images are used for training and 20% of the images are used for testing.

[0058] The present invention solves the technical defects existing in the background technology, and the present invention has the following beneficial effects: by introducing convolutional neural network (CNN) and Transformer technology, the advantages of both are fully utilized to achieve efficient and accurate image segmentation. CNN has a strong local feature extraction capability and is suitable for processing detailed features of glands and eyelid areas, while Transformer is superior in modeling global dependencies and long-distance relationships, and can effectively capture the complex structural information between glands and eyelids. By combining CNN and Transformer, the present invention not only improves the accuracy of gland segmentation, but also innovatively realizes the synchronous segmentation of eyelid areas and gland areas. This method not only improves the segmentation efficiency, but also enhances the robustness and consistency of the segmentation results as a whole, providing technical support for the rapid diagnosis and accurate evaluation of meibomian gland dysfunction (MGD). BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, drawings of other embodiments can be obtained based on these drawings without paying creative work.

[0060] Figure 1 This is a flowchart of the overall method of this meibomian gland loss grading method;

[0061] Figure 2 This is a schematic diagram of the segmentation network structure implemented in the present invention;

[0062] Figure 3 This is a schematic diagram of the segmentation result of the present invention;

[0063] Figure 4 A schematic diagram of the flow chart for implementing segmentation and rating in the present invention. DETAILED DESCRIPTION

[0064] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0066] The present invention discloses a method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology. Figure 1 , 4 As shown, the following steps are included:

[0067] S1: Obtain a meibomian gland image dataset and introduce a fast march algorithm to repair the meibomian gland image dataset to reduce the fuzziness of the gland boundary in the meibomian gland image caused by tear reflection and improve the sensitivity of the segmentation model to boundary features;

[0068] It should be noted that in order to improve the accuracy of meibomian gland image segmentation, the original data set was preprocessed, focusing on solving the problem of blurred gland boundaries due to tear reflection, and improving the sensitivity of the segmentation model to boundary features, laying the foundation for accurate identification and segmentation of the glands. Specifically, for the images in the meibomian gland image data set, they are randomly divided into two parts, of which 80% of the image samples are used for training and the remaining 20% ​​are used for testing. This ensures that the model can be effectively verified on both training and test data. Regarding the problem of blurred gland boundaries caused by tear reflection. The highlight interference in the reflective area will have an adverse effect on the feature extraction and gland boundary recognition of the segmentation algorithm, so the fast forward algorithm is introduced to repair these highlight areas. The algorithm can accurately locate the reflective area and fill in the missing information of the bright spot area through interpolation or texture compensation, thereby reconstructing a more natural and continuous gland boundary.

[0069] S2: Construct a multi-granularity meibomian gland segmentation model, input the processed meibomian gland image dataset into the multi-granularity meibomian gland segmentation model, and train the multi-granularity meibomian gland segmentation model based on convolutional neural network and Transformer technology;

[0070] S3: Use the trained segmentation model to segment the meibomian gland images, calculate the gland missing rate, and perform grading.

[0071] Furthermore, in step S2, the processed meibomian gland image dataset is input into the multi-granularity meibomian gland segmentation model and the multi-granularity meibomian gland segmentation model is trained based on the convolutional neural network and the Transformer technology, including the following steps:

[0072] In the encoding stage, a convolutional neural network is used to extract features from the input meibomian gland image, reducing the image resolution by downsampling while increasing the number of channels to capture local features;

[0073] The meibomian gland image of size W×H is divided into several 16×16 non-overlapping image blocks, and the features of the image blocks are mapped to C dimensions through a linear embedding layer; after linear transformation, the image blocks are input into the corresponding Transformer module to capture global image features; where W is the width, H is the height, and C is the number of input channels.

[0074] Also includes:

[0075] For Transformer, position encoding needs to be added, and the process is as follows:

[0076]

[0077] In the formula, Embedding projection for patches; E pos is the position embedding; z0 is the input of the Transformer module, which is a vector containing all image block features and position information; p is the size of each image block; R is the real number field; E is the linear embedding layer; C is the feature dimension after linear embedding; D is the hidden layer dimension of the Transformer model; is the feature of the first image block; is the feature of the second image block; is the feature of the Nth image block;

[0078] Moreover, the encoder consists of L layers of multi-head self-attention and multi-layer perceptron, so the output expression of the layer is as follows:

[0079] Z l′ =MSA(LN(Z l-1 ))+Z l-1

[0080] Z l =MLP(LN(Z l′ ))+Z l′

[0081] Where LN(·) represents the layer normalization operator; Z l is the encoded image representation; MSA stands for the multi-head self-attention mechanism, which is used to capture the relationship between image blocks; MLP stands for multi-layer perceptron, which is used to perform nonlinear transformation on features; Z l-1 Represents the output of the encoder layer l-1, that is, the output of the previous layer; Z l′ Represents the output of the multi-head self-attention module in the encoder layer l′.

[0082] It should be noted that in the encoding stage, a convolutional neural network (CNN) is first used to extract features from the input image, and the image resolution is reduced by downsampling, while the number of channels is increased to capture local features. Next, the image of size W×H is divided into multiple 16×16 non-overlapping blocks, and the features of these blocks are mapped to C dimensions through a linear embedding layer. After linear transformation, the image blocks are input into a series of Transformer modules to capture global image features, and position encoding needs to be added to the Transformer.

[0083] Furthermore, in step S2, the processed meibomian gland image dataset is input into the multi-granularity meibomian gland segmentation model and the multi-granularity meibomian gland segmentation model is trained based on the convolutional neural network and the Transformer technology, and the following steps are also included:

[0084] In the process of skip connection, multiple fusion dense skip connections are introduced;

[0085] During the meibomian gland image sampling process, the feature maps generated by downsampling at each stage are saved, and the adjacent feature maps are aligned in size to have the same resolution through interpolation or pooling;

[0086] Two 3x3 convolutional layers are used to process the aligned feature maps to extract image features. The feature extraction process is repeated, and each time the low-level and high-level features are fused to obtain three feature maps of different sizes.

[0087] After obtaining the feature map, the channel attention mechanism aggregates the global information of each channel through global pooling, compresses the spatial features of each channel into a single value, and forms a global representation of each channel; the aggregated features are then passed through a fully connected layer or a small neural network to calculate the importance of each channel, thereby obtaining the weight of each channel;

[0088] The calculated weights are normalized using an activation function to generate a scaling factor for each channel, indicating the relative importance of the channel in the current task;

[0089] The calculated channel weights are multiplied by the original feature map channel by channel, and the feature map of each channel is weighted and adjusted so that more important channels are strengthened and unimportant channels are suppressed.

[0090] Through the above process, the attention mechanism can adaptively adjust the contribution of different channels, thereby improving the performance of the model in the task, enhancing the model's ability to focus on key information, and improving the accuracy and robustness of feature extraction.

[0091] It should be noted that in the process of skip connection, multiple fused dense skip connections are first introduced. During the initial image downsampling process, the feature maps generated by downsampling at each stage are saved. For adjacent feature maps, the sizes are aligned and interpolated or pooled to make them have the same resolution. Then, two 3x3 convolutional layers are used to process these aligned feature maps to extract richer features. This process is repeated, and each time the low-level and high-level features are fused to finally obtain three feature maps of different sizes. After obtaining the feature map, the channel attention mechanism first aggregates the global information of each channel through global pooling (such as global average pooling or global maximum pooling), compresses the spatial features of each channel into a single value, and forms a global representation of each channel. These aggregated features are then passed through a fully connected layer or a small neural network to calculate the importance of each channel, thereby obtaining the weight of each channel. Then, these calculated weights are normalized using an activation function (such as Sigmoid) to generate a scaling factor for each channel, indicating the relative importance of the channel in the current task. Finally, these calculated channel weights are multiplied channel by channel with the original feature map, and the feature map of each channel is weighted and adjusted so that more important channels are strengthened and unimportant channels are suppressed. Through this process, the attention mechanism can adaptively adjust the contribution of different channels, thereby improving the performance of the model in the task, enhancing the model's ability to focus on key information, and further improving the accuracy and robustness of feature extraction.

[0092] In order to achieve accurate segmentation of eyelids and glands, the decoder is designed as two independent branches. The branches share the features extracted by the encoder, but are trained using different labels and loss functions, thereby independently generating segmentation results for eyelids and glands.

[0093] At each stage of decoding, the upsampled feature map is fused with the feature map of the corresponding stage of the encoder through skip connections; the fused feature map is further processed by a convolutional layer to enhance the ability to distinguish between categories; through this iterative process, the resolution of the feature map is gradually restored to be consistent with the input image;

[0094] In the final stage of the decoder, a 1×1 convolutional layer is used to convert the high-dimensional feature map into a category score map. The score of each pixel represents the probability of it belonging to a different category, and finally an accurate segmentation result is generated.

[0095] It should be noted that the decoder plays an indispensable role in image detail restoration and accurate segmentation, especially in scenarios that require high-precision edge detection, such as medical image processing. The design of the dual-branch decoder not only significantly improves segmentation accuracy, but also greatly improves processing efficiency, making it an ideal choice for complex segmentation tasks.

[0096] Also includes:

[0097] Calculate the loss function and optimize the neuron weight parameters;

[0098] When training the multi-granularity meibomian gland segmentation model, the error between the predicted quality score and the target quality score output by the multi-granularity meibomian gland segmentation model is continuously reduced, and the Adam optimization algorithm is used to automatically update the learning rate and continuously optimize the weight parameters of the multi-granularity meibomian gland segmentation model;

[0099] Among them, the batch-size in the Adam automatic update learning rate optimization algorithm is set to 4, the learning rate is initialized to 0.00009, and the weight decay factor is set to 0.0001; the activation function uses the ReLU activation function; and the loss function is obtained by a linear combination of two different loss functions.

[0100] It should be noted that the loss function is calculated and the neuron weight parameters are optimized. When training the network model, the error between the predicted quality score and the target quality score output by the network model is continuously reduced, and the weight parameters of the network model are continuously optimized. The activation function uses the ReLU activation function, and the Adam (Adaptive moment estimation) optimization algorithm is used to automatically update the learning rate. The batch-size is set to 4, the learning rate is initialized to 0.00009, and the weight decay factor is set to 0.0001. The loss function is obtained by linearly combining four L1 loss functions.

[0101] The loss function calculation formula is as follows:

[0102] Loss=λ1×WCE+λ2DiceLoss(X,Y)

[0103] Where Loss represents the final loss function value; λ1 represents the first weighting coefficient, which is used to control the weight of the weighted cross entropy loss in the total loss; λ2 represents the second weighting coefficient, which is used to control the weight of the Dice loss in the total loss; WCE represents the weighted cross entropy loss; DiceLoss(X,Y) represents the Dice loss function; X represents the prediction result of the model; Y represents the target result of the model;

[0104] Among them, WCE represents the weighted cross entropy loss function, which is expressed as follows:

[0105]

[0106] In the formula, w i is the weight of the category to which sample i belongs; y i is a binary value corresponding to the true label; p xiis the probability predicted by the model; DiceLoss(X,Y) represents the Dice loss function.

[0107] Among them, the expression of DiceLoss(X,Y) is:

[0108]

[0109] Among them, X and Y represent the prediction results and the true labels, X and Y represent their regions respectively, and |X∩Y| represents the intersection of the two regions.

[0110] In order to verify the superior performance of the model, two general indicators DSC (Dice similarity coefficient) and MIoU (mean intersection over union) are used to evaluate the segmentation results. The calculation formula is:

[0111]

[0112] Where DSC is the Dice similarity coefficient; X and Y represent the area of ​​the segmentation result and the label, respectively, and |X∩Y| represents the area of ​​the overlap between the segmentation result and the label;

[0113]

[0114] Where MIoU is the mean intersection over union ratio; TP represents the number of correctly segmented target pixels, that is, the number of pixels that are actually positive examples and predicted as positive examples; FP represents the number of incorrectly segmented target pixels, that is, the number of pixels that are actually negative examples but predicted as positive examples; FN represents the number of incorrectly segmented target pixels, that is, the number of pixels that are actually positive examples but predicted as negative examples; k represents the number of segmented targets.

[0115] Furthermore, the trained segmentation model is used in step S3 to segment the meibomian gland image, and the gland loss rate is calculated and graded, specifically:

[0116] The remaining 20% ​​of the data in the meibomian gland image dataset is input into the trained network model for quality prediction; binary images of the gland and eyelid area are predicted;

[0117] The glandular absence rate is calculated and graded using the binary image of the glandular area compared to the binary image of the upper eyelid area. The calculation formula is as follows:

[0118]

[0119] Among them, Density is the glandular loss rate; Smg i is the area of ​​the meibomian gland, in pixels; St is the area of ​​the tarsal plate; ∑smg i The area of ​​each meibomian gland (Smgi )(in pixels), and then the sum of the areas of all glands on each tarsal plate was compared with the area of ​​the tarsal plate (St).

[0120] The meibomian gland image dataset is randomly divided into two parts, the first part is 80% and the second part is 20%; wherein 80% of the images are used for training and 20% of the images are used for testing.

[0121] In order to solve the accuracy and efficiency problems of existing meibomian gland deficiency grading methods, the present invention proposes a meibomian gland deficiency grading method based on convolutional neural network and Transformer technology. First, for meibomian gland images with poor effect, the restoration technology is used to pre-process them to enhance the image quality, reduce the interference of reflection, noise and other factors on the segmentation results, and ensure that the boundaries of the gland area are clearer and more definite. Figure 2 , 3 As shown. Then, a multi-decoder deep learning model is proposed, which can simultaneously segment the gland and eyelid regions in the meibomian gland image. Through the collaborative work of multiple decoders, the segmentation results of the gland and eyelid regions are generated respectively, thereby ensuring the independent and accurate recognition of these two key regions. Finally, the segmentation results of the gland and eyelid regions are calculated by ratio, and the relationship between different regions is compared and analyzed to classify the possible missing conditions in the image, so as to accurately judge the functional status and health status of the meibomian glands.

[0122] In summary, the present invention not only improves the accuracy of gland segmentation, but also innovatively realizes the synchronous segmentation of the eyelid area and the gland area. This method not only improves the segmentation efficiency, but also enhances the robustness and consistency of the segmentation results as a whole, providing technical support for the rapid diagnosis and accurate evaluation of meibomian gland dysfunction (MGD).

[0123] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0124] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0125] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0126] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0127] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0128] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology, characterized in that: The following steps are involved: S1: Obtain a meibomian gland image dataset and introduce a fast march algorithm to repair the meibomian gland image dataset to reduce the fuzziness of the gland boundary in the meibomian gland image caused by tear reflection and improve the sensitivity of the segmentation model to boundary features; S2: Construct a multi-granularity meibomian gland segmentation model, input the processed meibomian gland image dataset into the multi-granularity meibomian gland segmentation model, and train the multi-granularity meibomian gland segmentation model based on convolutional neural network and Transformer technology; S3: Use the trained segmentation model to segment the meibomian gland images, calculate the gland missing rate, and perform grading.

2. According to claim 1, a method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology is characterized in that: In step S2, the processed meibomian gland image dataset is input into the multi-granularity meibomian gland segmentation model and the multi-granularity meibomian gland segmentation model is trained based on the convolutional neural network and the Transformer technology, including the following steps: In the encoding stage, a convolutional neural network is used to extract features from the input meibomian gland image, reducing the image resolution by downsampling while increasing the number of channels to capture local features; The meibomian gland image of size W×H is divided into several 16×16 non-overlapping image blocks, and the features of the image blocks are mapped to C dimensions through a linear embedding layer; after linear transformation, the image blocks are input into the corresponding Transformer module to capture global image features; where W is the width, H is the height, and C is the number of input channels.

3. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 2, characterized in that: Also includes: Adding position encoding to Transformer, the process is as follows: In the formula, Embedding projection for patches; E pos is the position embedding; z0 is the input of the Transformer module, which is a vector containing all image block features and position information; p is the size of each image block; R is the real number field; E is the linear embedding layer; C is the feature dimension after linear embedding; D is the hidden layer dimension of the Transformer model; is the feature of the first image block; is the feature of the second image block; is the feature of the Nth image block; Moreover, the encoder consists of L layers of multi-head self-attention and multi-layer perceptron, so the output expression of the layer is as follows: Z l′ =MSA(LN(Z l-1 ))+Z l-1 WITH l =MLP(LN(Z l′ ))+Z l′ Where LN(·) represents the layer normalization operator; Z l is the encoded image representation; MSA stands for the multi-head self-attention mechanism, which is used to capture the relationship between image blocks; MLP stands for multi-layer perceptron, which is used to perform nonlinear transformation on features; Z l-1 Represents the output of the encoder layer l-1, that is, the output of the previous layer; Z l′ Represents the output of the multi-head self-attention module in the encoder layer l′.

4. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 1, characterized in that: In step S2, the processed meibomian gland image dataset is input into the multi-granularity meibomian gland segmentation model and the multi-granularity meibomian gland segmentation model is trained based on the convolutional neural network and Transformer technology, and further includes the following steps: In the process of skip connection, multiple fusion dense skip connections are introduced; During the meibomian gland image sampling process, the feature maps generated by downsampling at each stage are saved, and the adjacent feature maps are aligned in size to have the same resolution through interpolation or pooling; Two 3x3 convolutional layers are used to process the aligned feature maps to extract image features. The feature extraction process is repeated, and each time the low-level and high-level features are fused to obtain three feature maps of different sizes. After obtaining the feature map, the channel attention mechanism aggregates the global information of each channel through global pooling, compresses the spatial features of each channel into a single value, and forms a global representation of each channel; the aggregated features are then passed through a fully connected layer or a small neural network to calculate the importance of each channel, thereby obtaining the weight of each channel; The calculated weights are normalized using an activation function to generate a scaling factor for each channel, indicating the relative importance of the channel in the current task. The calculated channel weight is multiplied by the original feature map channel by channel, and the feature map of each channel is weighted and adjusted so that the target channel is strengthened and the non-target channel is suppressed; According to the above process, the attention mechanism can adaptively adjust the contribution of different channels, thereby improving the performance of the model in the task, enhancing the model's ability to focus on key information, and improving the accuracy and robustness of feature extraction.

5. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 4, characterized in that: In order to achieve accurate segmentation of eyelids and glands, the decoder is designed as two independent branches. The branches share the features extracted by the encoder, but are trained using different labels and loss functions, thereby independently generating segmentation results for eyelids and glands. At each stage of decoding, the upsampled feature map is fused with the feature map of the corresponding stage of the encoder through skip connections; the fused feature map is further processed by a convolutional layer to enhance the ability to distinguish between categories; through this iterative process, the resolution of the feature map is gradually restored to be consistent with the input image; In the final stage of the decoder, a 1×1 convolutional layer is used to convert the high-dimensional feature map into a category score map. The score of each pixel represents the probability of it belonging to a different category, and finally an accurate segmentation result is generated.

6. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 4, characterized in that: Also includes: Calculate the loss function and optimize the neuron weight parameters; When training the multi-granularity meibomian gland segmentation model, the error between the predicted quality score and the target quality score output by the multi-granularity meibomian gland segmentation model is continuously reduced, and the Adam optimization algorithm is used to automatically update the learning rate and continuously optimize the weight parameters of the multi-granularity meibomian gland segmentation model; Among them, the batch-size in the Adam automatic update learning rate optimization algorithm is set to 4, the learning rate is initialized to 0.00009, and the weight decay factor is set to 0.0001; the activation function uses the ReLU activation function; and the loss function is obtained by a linear combination of two different loss functions.

7. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 6, characterized in that: The loss function calculation formula is as follows: Loss=λ1×WCE+λ2DiceLoss(X,Y) In the formula, Loss represents the final loss function value; λ1 represents the first weighting coefficient, which is used to control the weight of weighted cross entropy loss in the total loss; λ2 represents the second weighting coefficient, which is used to control the weight of Dice loss in the total loss; WCE represents weighted cross entropy loss; DiceLoss(X,Y) represents the Dice loss function; X represents the prediction result of the model; Y represents the target result of the model; Among them, WCE represents the weighted cross entropy loss function, which is expressed as follows: In the formula, w i is the weight of the category to which sample i belongs; y i is a binary value corresponding to the true label; is the probability predicted by the model; DiceLoss(X,Y) represents the Dice loss function; Among them, the expression of DiceLoss(X,Y) is: Where X and Y represent the predicted results and the true labels, X and Y represent their regions respectively, and |X∩Y| represents the intersection of the two regions.

8. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 7, characterized in that: DSC and MIoU are used to evaluate the segmentation results, and the calculation formula is: Where DSC is the Dice similarity coefficient; X and Y represent the area of ​​the segmentation result and the label, respectively, and |X∩Y| represents the area of ​​the overlap between the segmentation result and the label; Where MIoU is the mean intersection over union ratio; TP represents the number of correctly segmented target pixels, that is, the number of pixels that are actually positive examples and predicted as positive examples; FP represents the number of incorrectly segmented target pixels, that is, the number of pixels that are actually negative examples but predicted as positive examples; FN represents the number of incorrectly segmented target pixels, that is, the number of pixels that are actually positive examples but predicted as negative examples; k represents the number of segmented targets.

9. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 1, characterized in that: In step S3, the trained segmentation model is used to segment the meibomian gland image, and the gland loss rate is calculated and graded, specifically: The remaining 20% ​​of the data in the meibomian gland image dataset is input into the trained network model for quality prediction; binary images of the gland and eyelid area are predicted; The glandular absence rate is calculated and graded using the binary image of the glandular area compared to the binary image of the upper eyelid area. The calculation formula is as follows: Among them, Density is the glandular loss rate; Smg i is the area of ​​the meibomian gland, in pixels; St is the area of ​​the tarsal plate; ∑smg i It is the sum of the areas of all meibomian glands on each tarsal plate.

10. The method for grading meibomian gland deficiency based on convolutional neural network and Transformer technology according to claim 1, characterized in that: The meibomian gland image dataset is randomly divided into two parts, the first part is 80% and the second part is 20%; wherein 80% of the images are used for training and 20% of the images are used for testing.

Images