Myopic maculopathy classification method based on knowledge distillation and self-supervised learning

Abstract

The present invention discloses a method and system for classifying myopic maculopathy based on knowledge distillation and self-supervised learning, which relates to the technical field of medical image classification, including a self-supervised learning stage and a myopic maculopathy classification stage. The self-supervised learning stage includes an image mask module, a knowledge distillation and feature encoding module, and an image reconstruction module; the original fundus color image is subjected to high-pass filtering to generate a reconstruction target, so that the representation learning gives priority to high-frequency components, namely, image details and edge information, so that the network pays more attention to the high-frequency details of the image to achieve image reconstruction. The present invention realizes efficient application in the classification of fundus image lesions by optimizing distillation technology and reconstruction strategy; extracts knowledge from a larger model to improve the training efficiency of the model; enhances the model's capture of image details, and has good classification performance, laying a foundation for the subsequent quantitative analysis of lesions.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image classification, and in particular to a myopic maculopathy classification method based on knowledge distillation and self-supervised learning. Background Art

[0002] Image classification is a core task in computer vision, which aims to assign images to one or more predefined categories based on their content. Traditional image classification algorithms generally include several stages, such as underlying feature learning, feature encoding, spatial constraints, classifier design, and model fitting. Simply put, feature extraction is to extract key information from an image that can represent its content, such as color, texture, shape, etc. The classifier recognizes and classifies the image based on these features. Common classifiers include support vector machines, decision trees, random forests, etc. With the rise of deep learning technology, many deep neural networks have been widely used in image classification algorithms. In 2015, a deep convolutional neural network ResNet was proposed. By introducing residual connections, it solved the gradient vanishing and gradient exploding problems in deep neural network training and became a basic model for image classification. The Vision Transformer (ViT) proposed in 2020 is an architecture that applies the Transformer model to computer vision tasks. It has demonstrated performance that can compete with or even surpass traditional convolutional neural networks in image classification tasks, but it still needs further optimization and improvement in scenarios with small data sets and limited computing resources.

[0003] Medical image classification refers to the automatic classification of medical images using image classification technology, including lung nodule classification, skin lesion classification, fundus disease classification, etc. The results can assist doctors in comprehensive analysis of lesions, improve diagnostic accuracy and promote good intervention strategies. However, in actual applications, medical images still have limitations such as insufficient data quality and quantity, difficulty in obtaining high-quality labels, and insufficient generalization ability of classification models. Medical image classification is still very challenging.

[0004] ResNet successfully solved the gradient vanishing problem of deep convolutional neural networks by introducing residual connections, and became the basic network for image classification. EfficientNet uses a compound scaling method to evenly expand the width, depth and resolution of the network, providing efficient classification performance and accuracy. The visual transformer ViT applies the Transformer model to image classification tasks, relying on the self-attention mechanism instead of the traditional convolution operation, and performs well in large-scale data sets and high computing resource environments. In addition to the above traditional supervised learning methods, researchers also use self-supervised learning methods to design suitable self-supervised tasks to utilize the information of the data itself, thereby learning meaningful feature representations. For example, momentum contrastive learning MoCo learns feature representations by calculating the similarity between the encoder's embedded features and the momentum encoder's dynamic representation dictionary; the bidirectional encoding structure network BEiT based on ViT successfully uses mask image modeling for model training by learning to predict visual tags; SimMIM, a simple framework for mask image modeling, and the mask autoencoder MAE both propose to train models by directly reconstructing the pixel values ​​of occluded image blocks.

[0005] In the task of medical image classification, the use of self-supervised learning method MAE can reduce the dependence on a large amount of labeled data, improve the quality of feature representation, and help solve practical problems such as lack of labeled data and difficulty in labeling. However, there are still the following shortcomings:

[0006] Training models requires large datasets and high computing resources. Medical image datasets are usually small and expensive to annotate. MAE relies on large-scale data for self-supervised training, and insufficient data may limit its performance. Knowledge distillation technology can be used to transfer knowledge from powerful but complex models to smaller models. Convolutional Neural Networks with Knowledge Distillation BYOT proposed a new training framework called self-distillation, which enhances the performance of the network by performing knowledge distillation within the network instead of using a separate teacher-student model. BYOT divides the network into several parts and extracts the knowledge of the deeper parts into the shallower parts. However, the effect of BYOT self-distillation is highly dependent on the performance of the original model. If the performance of the original model is poor, self-distillation may not significantly improve its performance, and may even solidify the original errors and defects. Contrastive Representation Distillation Network CRD incorporates contrastive learning into knowledge distillation, and improves the performance of the student network by obtaining more structural knowledge represented by the teacher network. This method uses multiple teacher networks for integration, which significantly increases the consumption of computing resources and training time. At the same time, in order to give full play to the advantages of integrated distillation, additional experiments and tuning are required, which increases the complexity and computational overhead of the model.

[0007] The reconstruction target has limitations. Since the emergence of MAE, most mask image modeling methods have used original pixels as the reconstruction target. The training target needs to perfectly reconstruct the mask patch, so it contains a lot of information redundancy and lacks the capture of high-level semantics. The mask feature prediction model MaskFeat simply takes the hand-designed HOG features as the target, but the target is still a descriptor based on local statistics, which may also have the same problem as the pixel prediction-based method. The mask image pre-training model MILAN based on language-assisted representation uses the image encoder of the contrastive learning text-image pre-training model CLIP, and proposes a more efficient decoder architecture and semantic-aware sampling mechanism based on the latent features generated by the complete input image as the reconstruction target, but this method increases the computational complexity of the model, resulting in longer training time or requiring greater computing resources. Summary of the invention

[0008] In view of the problems existing in the existing myopic maculopathy classification method based on knowledge distillation and self-supervised learning, the present invention is proposed.

[0009] Therefore, the problem to be solved by the present invention is how to accelerate the training process of the classification model while ensuring high-precision classification.

[0010] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0011] In a first aspect, an embodiment of the present invention provides a method for classifying myopic maculopathy based on knowledge distillation and self-supervised learning, which comprises:

[0012] The self-supervised learning stage and the myopic maculopathy classification stage;

[0013] The self-supervised learning stage is used to divide the input image into non-overlapping patches, extract features by constructing a teacher model and a student model, and predict the complete image through visible patches;

[0014] The myopic macular lesion classification stage is used to fine-tune the classification network using a cross entropy loss function and a center loss function to obtain a classifier network, send the complete image to be predicted to the trained classifier network, and obtain the output of the lesion classification result;

[0015] The self-supervised learning stage includes an image mask module, a knowledge distillation and feature encoding module, and an image reconstruction module; the encoder module in the image mask module uses a visual converter ViT as a backbone network to convert the input image into a sequence as the input of ViT, and converts the input image into a sequence. Divide into regular non-overlapping N=H×W / p 2 Patch blocks , where p 2 is the number of pixels in the patch block, H×W is the resolution of the input image, C is the number of channels, and the patch block Using Linear Projection Layer Mapped to a D-dimensional embedding vector , based on the embedding vector, add a learnable embedding vector , aggregating information from the entire sequence and adding learnable position embeddings to the patch embedding sequence , and get the ViT input sequence, expressed as:

[0016] ,

[0017] Randomly sample the patch sequence and mask the remaining patch blocks, that is, construct a random mask based on the obtained patch sequence ,set up Indicates that the corresponding patch block is masked, K represents the number of unmasked patch blocks, then the final encoder input sequence is ;

[0018] The ViT input sequence models the remote information through the self-attention mechanism. For a given L-layer ViT, only some visible patches in the input sequence are included, i.e. It is fed into the encoder and mapped to latent features, represented as:

[0019] ;

[0020] ;

[0021] in, represents the layer of ViT, LN(·) represents the layer normalization operation, in the multi-head self-attention MSA layer, each patch embedding vector calculates the self-attention map and updates itself by aggregating other patch embedding vectors. The output of the feature encoder includes the global feature representation and patch-level feature representation .

[0022] As a preferred solution of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning described in the present invention, wherein: the knowledge distillation and feature encoding module uses a teacher-student model to initialize the teacher model and the student model respectively, the teacher model is initialized using the ViT model pre-trained by ImageNet, and the student model is randomly initialized;

[0023] Extract features from the middle three layers of the student model and input them into a small projection layer whose output is required to mimic the features from the corresponding layers of the teacher model. The projection layer is implemented through a fully connected layer.

[0024] Set the extracted l-th layer features of the student model and the teacher model to be ,in is the total number of patches, C is the number of channels; the middle three layers of the teacher model and the student model are taken as the layers for knowledge distillation, FC() is used to represent the projection network function, and the feature output dimension of the student model is aligned with the dimension of the teacher model through the fully connected layer. The distillation loss function is expressed as:

[0025] ;

[0026] Among them, S and T represent the student model and the teacher model respectively. Represents the total number of pixels, i represents the pixel index, represents the Manhattan distance.

[0027] As a preferred solution of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning described in the present invention, the image reconstruction module is used to enhance the model's ability to reconstruct high-frequency details, and a Gaussian filter is used to smooth the image to obtain the low-frequency components of the image. Assuming that the input image is ,in represents the pixel coordinates in the image, and the Gaussian filter kernel is expressed as:

[0028] ;

[0029] in, is the standard deviation of the Gaussian distribution, which is used to control the smoothness of the filter. The steps of extracting low-frequency components of an image using a Gaussian filter include: specifying the Gaussian kernel radius r and the standard deviation , thereby generating a Gaussian filter kernel of the corresponding size; convolve the Gaussian filter kernel with the input image to obtain the low-frequency component of the image ; Subtract the low-frequency component of the image from the original image to obtain the high-frequency component of the image , expressed as:

[0030] ;

[0031] ;

[0032] The pixel value of the input image is represented in vector form, that is, assuming that the input pixel value is , then the high-frequency reconstruction target of the image is , calculate the reconstruction loss of the model, expressed as:

[0033] ;

[0034] in, represents the predicted pixel value, M represents the set of mask pixels, is the total number of pixels, i is the mask pixel index; the final loss function used in the self-supervised learning stage is expressed as:

[0035] ;

[0036] in, To reconstruct the loss function, is the distillation loss function for feature alignment.

[0037] As a preferred solution of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning of the present invention, wherein: after the myopic maculopathy classification stage is fully trained in the self-supervised learning stage, the encoder part of the student model is taken out for fine-tuning; the loss function is a joint loss function including the cross entropy loss and the center loss function, which is expressed as:

[0038] ;

[0039] in, is the cross entropy loss function, is the center loss function, is a hyperparameter used to control the balance between the cross entropy loss and the center loss function; in the last layer of the classification network, the fully connected layer outputs a vector with a length of P, the number of categories, Q is the total number of output vectors, and the cross entropy loss function is used to guide the training of the network, that is, to minimize the cross entropy loss function between the prediction result and the label. :

[0040] ;

[0041] in, is the indicator function that the true label of sample i is category j, that is, when the true category of the sample is j, otherwise ; is the probability that the model predicts that sample i is of category j. After being processed by the softmax function, the sum of the predicted probabilities of all categories is 1.

[0042] As a preferred solution of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning described in the present invention, the central loss function is introduced to enhance the feature discrimination ability and generalization ability of the model. The central loss function minimizes the distance between the sample feature and the center of the category to which it belongs, so that the sample features of the same category are more compact in the feature space. The central loss function is expressed as:

[0043] ;

[0044] in, represents the Euclidean distance, The feature representation obtained after the sample is input into the model, Is the category to which the sample belongs The category center; comprehensively utilizing the advantages of the cross entropy loss and the center loss function, the joint loss function is introduced, which is expressed as:

[0045] ;

[0046] By combining the loss function, the classification performance is guaranteed, the discriminability and stability of features are improved, and different feature distributions within and between classes can be better handled.

[0047] In a second aspect, an embodiment of the present invention provides a myopic maculopathy classification system based on knowledge distillation and self-supervised learning, which includes: an image mask module, which is used to divide an input image into non-overlapping patch blocks, and use a masking ratio of n% to retain a small random subset of them as visible blocks for use by the encoder, and the remaining patch blocks are used as prediction targets for the decoder; a knowledge distillation and feature encoding module, which is used to initialize a teacher model and a student model, and input the visible patch blocks into the encoders of the teacher model and the student model for feature extraction, respectively, to obtain deep semantic features belonging to the teacher model and the student model, and minimize the distance between the intermediate layer feature mappings of the teacher model and the student model; an image reconstruction module, which is used to obtain the visible patch blocks through the encoders of the teacher model and the student model; The complete image is predicted by the block, and the low-frequency component of the input image is extracted by using a Gaussian filter based on a Gaussian kernel. The high-frequency component of the input image is obtained by subtracting the original image from its low-frequency component as the reconstruction target of the model. The deep semantic features of the student model and the reconstruction target of the model are used as input, and the mean absolute error loss of the two is calculated, which is optimized together with the distillation loss to guide the model training; the myopic macular lesion classification stage includes taking out the encoder of the student model as the encoder of the myopic macular lesion classification task after the training in the self-supervised learning stage is completed, and the classification network is fine-tuned and trained using the cross entropy loss function and the center loss function to obtain the classifier network; the fundus color photo image is sent to the trained classifier network to obtain the output of the lesion classification result.

[0048] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the processor executes the computer program, it implements any step of the above-mentioned method for classifying myopic maculopathy based on knowledge distillation and self-supervised learning.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the above-mentioned method for classifying myopic maculopathy based on knowledge distillation and self-supervised learning is implemented.

[0050] The beneficial effect of the present invention is that the knowledge distillation structure designed by the present invention first follows the training process of MAE, allowing the encoder to perform feature encoding on visible patches, and using a lightweight decoder for pixel reconstruction. Subsequently, the model seeks to match the intermediate feature representation, that is, to minimize the distance between the intermediate feature maps of the teacher model and the student model, which allows the complex teacher model to propagate the input features forward only through the first few layers, thereby reducing the amount of calculation. The original fundus color image is high-pass filtered to generate a reconstruction target, so that the representation learning gives priority to high-frequency components, that is, image details and edge information, so that the network pays more attention to the high-frequency details of the image to achieve the reconstructed image. The present invention achieves efficient application in the classification of fundus image lesions by optimizing distillation technology and reconstruction strategies. The present invention is applied to the classification task of myopic macular lesions with a small number of images and unbalanced lesion categories, and good classification performance is obtained, indicating the effectiveness of the knowledge distillation structure and image reconstruction optimization strategy designed by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0052] Figure 1 Schematic diagram of the overall structure of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning.

[0053] Figure 2 Schematic diagram of the image mask module of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning.

[0054] Figure 3 Schematic diagram of the knowledge distillation and feature encoding module of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning.

[0055] Figure 4Schematic diagram of the teacher-student model for myopic maculopathy classification method based on knowledge distillation and self-supervised learning.

[0056] Figure 5 Schematic diagram of the image reconstruction module of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0060] Example 1

[0061] Reference Figure 1~Figure 5 , which is the first embodiment of the present invention, provides a myopic maculopathy classification method based on knowledge distillation and self-supervised learning, including:

[0062] Phase 1: Self-supervised learning phase, including image mask module, knowledge distillation and feature encoding module, and image reconstruction module;

[0063] Stage 2: Myopic macular lesion classification stage, using the encoder part trained in the self-supervised learning stage to classify the lesions.

[0064] S1. The main steps of the self-supervised learning stage include:

[0065] The input image is divided into non-overlapping patches, and a small random subset of them is retained as visible patches for the encoder with a masking ratio of 75% (i.e., the proportion of patches removed), while the remaining patches are used as prediction targets for the decoder.

[0066] Initialize the teacher model and the student model, input the visible patch blocks into the encoders of the teacher model and the student model for feature extraction, obtain the deep semantic features belonging to the teacher model and the student model respectively, and minimize the distance between the feature maps of the middle layer of the teacher model and the student model;

[0067] The low-frequency components of the input image are extracted using a Gaussian filter based on a Gaussian kernel, and the high-frequency components of the input image are obtained by subtracting the original image from its low-frequency components as the reconstruction target of the model. The deep semantic features of the student model and the reconstruction target of the model are used as input, and the mean absolute error loss of the two is calculated and optimized together with the distillation loss to guide the model training.

[0068] S1.1. Image mask module

[0069] The encoder module uses the visual converter ViT as the backbone network. First, the input image is converted into a sequence as the input of ViT through the image mask module. The structure of the image mask module is shown in the figure. Figure 2 As shown, the input image Divide into regular non-overlapping N=H×W / p 2 Patch blocks , where p 2 is the number of pixels in the patch block, H×W is the resolution of the input image, C is the number of channels, and then the patch block Using Linear Projection Layer Mapped to a D-dimensional embedding vector , based on these embedding vectors, add a learnable embedding vector , which is used to aggregate information from the entire sequence and add a learnable position embedding to the patch embedding sequence , then the ViT input sequence is obtained, expressed as:

[0070] ,

[0071] Randomly sample the patch sequence and mask the remaining patch blocks, that is, construct a random mask based on the obtained patch sequence ,set up Indicates that the corresponding patch block is masked, K represents the number of unmasked patch blocks, then the final encoder input sequence is Random sampling with high masking ratio largely eliminates redundancy, creating a task that cannot be solved by reasoning from visible neighboring patches, while the highly sparse input creates opportunities for designing efficient encoders.

[0072] S1.2. Knowledge distillation and feature encoding module

[0073] Unlike convolution-based feature extractors, ViT models long-range information through a self-attention mechanism. The encoder of ViT consists of a cascade of transformer blocks, each of which consists of a multi-headed self-attention (MSA) layer and a multi-layer perceptron (MLP) block. The structure of the feature encoding module is shown in the attached figure. Figure 3 shown.

[0074] For a given L-layer ViT, only some visible patches in the input sequence are included, i.e. It is fed into the encoder and mapped to latent features, represented as:

[0075] ,

[0076] ,

[0077] in, represents the layer of ViT, LN(·) represents the layer normalization operation, in the MSA layer, each patch embedding vector calculates the self-attention map and updates itself by aggregating other patch embedding vectors. The output of the feature encoder includes the global feature representation , and patch-level feature representation .

[0078] Combining knowledge distillation with the MAE framework can effectively obtain small and fast models with similar performance to those powerful but bulky models. If the existing knowledge distillation methods are applied to the fine-tuned MAE model, good performance cannot be achieved. Therefore, knowledge distillation is directly applied in the self-supervised training stage. Since labels are not used in MAE self-supervised training, the method of extracting logits in knowledge distillation is difficult to help learn semantically meaningful representations. Therefore, the method of extracting intermediate layer features is used for knowledge distillation.

[0079] S1.3. Teacher-Student Model

[0080] As attached Figure 4As shown in the figure, knowledge distillation requires the use of a teacher-student model. The teacher model and the student model are initialized separately. The teacher model is initialized using the ViT model pre-trained on ImageNet, and the student model is randomly initialized. Specifically, features are first extracted from the middle three layers of the student model; these features are input into a small projection layer, and the output is required to simulate the features from the corresponding layers of the teacher model. In practical applications, the projection layer is implemented through a simple fully connected layer, which solves the problem of feature dimension mismatch between the teacher model and the student model.

[0081] Set the extracted l-th layer features of the student model and the teacher model to be ,in is the total number of patches, C is the number of channels; the middle three layers of the teacher model and the student model are taken as the layers for knowledge distillation, FC() is used to represent the projection network function, and the feature output dimension of the student model is aligned with the dimension of the teacher model through the fully connected layer. The distillation loss function is expressed as:

[0082] ,

[0083] Among them, S and T represent the student model and the teacher model respectively. Represents the total number of pixels, i represents the pixel index, represents the Manhattan distance.

[0084] S1.4. Image reconstruction module

[0085] The structure of the image reconstruction module is shown in the attached figure. Figure 5 As shown. In order to enhance the model's ability to reconstruct high-frequency details, a new high-frequency target generator is designed, that is, to filter out the low-frequency components of the original image. First, the image is smoothed using a Gaussian filter F(·) to obtain the low-frequency components of the image. A Gaussian filter is a linear filter whose filter kernel is defined by a Gaussian function. Assume that the input image is ,in The Gaussian filter kernel representing the pixel coordinates in the image is expressed as:

[0086] ,

[0087] in, is the standard deviation of the Gaussian distribution, which is used to control the smoothness of the filter;

[0088] The steps of extracting low-frequency components of an image using a Gaussian filter include: specifying the Gaussian kernel radius r and the standard deviation , thereby generating a Gaussian filter kernel of the corresponding size; convolve the Gaussian filter kernel with the input image to obtain the low-frequency component of the image ; Subtract the low-frequency component of the image from the original image to obtain the high-frequency component of the image , expressed as:

[0089] ,

[0090] ,

[0091] The pixel value of the input image is represented in vector form, that is, assuming that the input pixel value is , then the high-frequency reconstruction target of the image is , calculate the reconstruction loss of the model, expressed as:

[0092] ,

[0093] in, represents the predicted pixel value, M represents the set of mask pixels, is the total number of pixels, i is the mask pixel index;

[0094] The final loss function used in the self-supervised learning phase of stage 1 is the reconstruction loss and feature alignment distillation loss The joint loss of , the two work together to guide network training:

[0095] ,

[0096] in, To reconstruct the loss function, is the distillation loss function for feature alignment.

[0097] S2. The main steps of the myopic maculopathy classification stage are:

[0098] After the self-supervised learning model is trained, the encoder of the student model is taken out as the encoder of the myopic maculopathy classification task, and the classification network is fine-tuned using the cross entropy loss function and the center loss function to obtain the classifier network;

[0099] The fundus color image is sent to the trained classifier network to obtain the output of the lesion classification result.

[0100] S2.1 Utilization including reconstruction loss and feature alignment distillation loss After the joint loss collaborative guidance network is fully trained, the encoder part of the student model is taken out and fine-tuned for the myopic maculopathy classification task in stage 2.

[0101] The loss function is a joint loss function that includes the cross entropy loss and the center loss function, expressed as:

[0102] ,

[0103] in, is the cross entropy loss function, is the center loss function, Is a hyperparameter used to control the balance between the cross entropy loss and the central loss function;

[0104] In the last layer of the classification network, the fully connected layer outputs a vector with a length of P, the number of categories, Q is the total number of output vectors, and the cross entropy loss function is used to guide the training of the network, that is, to minimize the cross entropy loss function between the prediction result and the label. :

[0105] ,

[0106] in, is the indicator function that the true label of sample i is category j, that is, when the true category of the sample is j, otherwise ; is the probability that the model predicts that sample i is of category j. After being processed by the softmax function, the sum of the predicted probabilities of all categories is 1.

[0107] The center loss function is introduced to enhance the feature discrimination and generalization ability of the model to make up for the shortcomings of cross entropy loss that ignores the relative position relationship between sample features and is insensitive to the differences between samples in the same category. The center loss function minimizes the distance between the sample feature and the center of its category, making the sample features of the same category more compact in the feature space. The formula of the center loss function is:

[0108] ,

[0109] in, represents the Euclidean distance, The feature representation obtained after the sample is input into the model, Is the category to which the sample belongs Category center of

[0110] Taking advantage of the cross entropy loss and the center loss function, a joint loss function is introduced, which is expressed as:

[0111] ,

[0112] By combining the loss function, the model can not only guarantee the classification performance, but also improve the discriminability and stability of the features, so as to better handle the different feature distributions within and between classes. This method enables the model to classify more accurately and improve the generalization ability when facing complex data distribution.

[0113] In summary, the knowledge distillation structure designed by the present invention first follows the training process of MAE, allowing the encoder to feature encode the visible patches and using a lightweight decoder for pixel reconstruction. Subsequently, the model seeks to match the intermediate feature representation, that is, to minimize the distance between the intermediate feature maps of the teacher model and the student model, which allows the copied teacher model to propagate the input features forward only through the first few layers, thereby reducing the amount of calculation. The original fundus color image is high-pass filtered to generate the reconstruction target, so that the representation learning gives priority to high-frequency components, that is, image details and edge information, so that the network pays more attention to the high-frequency details of the image to achieve the reconstructed image. The present invention achieves efficient application in the classification of fundus image lesions by optimizing distillation technology and reconstruction strategies. The present invention is applied to the classification task of myopic macular lesions with a small number of images and unbalanced lesion categories, and good classification performance is obtained, which shows the effectiveness of the knowledge distillation structure and image reconstruction optimization strategy designed by the present invention.

[0114] Example 2

[0115] Reference Figure 1 Based on the first embodiment, this embodiment further provides a myopic maculopathy classification system based on knowledge distillation and self-supervised learning, including:

[0116] The image mask module is used to divide the input image into non-overlapping patches, and retain a small random subset of them as visible patches for the encoder with a masking ratio of n%, while the remaining patches are used as prediction targets for the decoder;

[0117] The knowledge distillation and feature encoding module is used to initialize the teacher model and the student model, input the visible patch blocks into the encoders of the teacher model and the student model for feature extraction, obtain the deep semantic features belonging to the teacher model and the student model respectively, and minimize the distance between the feature mappings of the middle layer of the teacher model and the student model;

[0118] The image reconstruction module is used to predict the complete image through the visible patch blocks. The low-frequency components of the input image are extracted using a Gaussian filter based on a Gaussian kernel. The high-frequency components of the input image are obtained by subtracting the original image from its low-frequency components as the reconstruction target of the model. The deep semantic features of the student model and the reconstruction target of the model are used as inputs. The mean absolute error loss of the two is calculated and optimized together with the distillation loss to guide the model training.

[0119] The myopic macular lesion classification stage includes taking out the encoder of the student model as the encoder of the myopic macular lesion classification task after the image reconstruction module is trained in the self-supervised learning stage, and fine-tuning the classification network using the cross entropy loss function and the center loss function to obtain the classifier network; sending the fundus color image to the trained classifier network to obtain the output of the lesion classification result.

[0120] This embodiment also provides a computer device, which is suitable for the case of a myopic maculopathy classification method based on knowledge distillation and self-supervised learning, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the myopic maculopathy classification method based on knowledge distillation and self-supervised learning proposed in the above embodiment.

[0121] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0122] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the myopic maculopathy classification method based on knowledge distillation and self-supervised learning as proposed in the above embodiment.

[0123] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0124] Example 3

[0125] On the basis of the previous two embodiments, this embodiment provides a myopic maculopathy classification method based on knowledge distillation and self-supervised learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0126] Constructing a self-supervised learning dataset: The EyePACS dataset is used as the training dataset for the self-supervised learning phase 1. The dataset consists of 35,126 fundus images with different DR levels. These images are from 44,351 patients, using different imaging devices and under various imaging conditions, so the image diversity in the dataset is large. Image labels are not used in the self-supervised learning phase.

[0127] Constructing a dataset for the classification task of myopic maculopathy: Based on the classification dataset of myopic maculopathy in the Myopic Maculopathy Analysis Challenge (MMAC), the image size is 800×800 pixels. The training set consists of 1,143 color fundus images, of which 990 are from Data Center 1 and 1,153 are from Data Center 2. The validation set contains 248 color fundus images, of which 215 are from Data Center 1 and 33 are from Data Center 2. When reading the data, online random cropping, brightness transformation, and left-right flipping are used for data enhancement. Evaluation indicators include ACC, quadratic weighted kappa coefficient ( ), F1 score (F1), specificity (Spe), and the average of the latter three. ACC represents the proportion of samples correctly classified by the model to the total number of samples; the quadratic weighted kappa coefficient is an indicator used to measure the consistency and accuracy of the classification model; the F1 score is a comprehensive indicator of the performance of the classification model, taking into account both the precision and recall rates; and the specificity Spe represents the ability of the model to correctly identify negative samples. ACC, F1 score, and specificity Spe are defined by true positive (TP), true negative (TN), false positive (FP), and false negative (FN).

[0128] ,

[0129] ,

[0130] ,

[0131] The quadratic weighted kappa coefficient is defined as follows, where Represents the number of class i predicted as class j, is the expected number of times the i-th class is divided into the j-th class when all labels are randomly assigned, is the corresponding square weight, and P is the total number of categories.

[0132] ,

[0133] In the comparative experiment, the method of the present invention was compared with other excellent self-supervised learning-based networks, including Moco, DINO, MaskFeat, iBot, CAE, SimMIM, and MAE. As shown in Table 1, the ACC index and the average of the Kappa coefficient, F1 score, and specificity Spe of the present invention have been comprehensively improved compared with these networks, with ACC reaching 85.4% and the average of the other indicators being 91.6%. Accuracy ACC measures the overall correct prediction ability of the model; the Kappa coefficient is an indicator used to measure the consistency between the classifier's prediction results and the actual labels; the F1 score comprehensively considers the balance between precision and recall; specificity focuses on the prediction ability of negative examples, so the average is used to combine the advantages of multiple indicators, which can comprehensively reflect the performance of the classifier in different aspects. Moco and DINO are based on contrastive self-supervised learning methods, which focus more on the global features of the image. In contrast, the generative self-supervised learning method pays more attention to local features and can better capture the details of the lesions. Due to this difference, the ability of contrastive self-supervised learning to capture details is relatively insufficient, so its average index may be slightly lower.

[0134] Table 1 Comparative experimental results of myopic maculopathy classification

[0135] ,

[0136] In order to verify the effectiveness of the distillation structure, image reconstruction strategy optimization and loss function proposed in the present invention, corresponding ablation experiments were carried out. As can be seen from Table 2, the baseline network MAE is added with a distillation structure, and the ACC increases by 3.2%, and the average of the other indicators increases by 2.4%, indicating that the student network has fully learned the knowledge transferred by the teacher network and improved the training efficiency. On this basis, the image reconstruction target is set to the high-frequency part of the image, and the ACC increases by 1.7%, and the average index increases by 0.3%, which enhances the model's capture of the details of the image and can better reconstruct the image. Subsequently, the classification loss function of the myopic maculopathy classification stage is set to the central loss function, and the ACC increases by 0.8%, and the average of the other indicators increases by 0.9%. Thanks to the enhancement of intra-class compactness and the improvement of feature discrimination ability, the model can better distinguish different categories. If the central loss function is used directly in the myopic maculopathy classification stage of the baseline network MAE, the average index is improved by 0.8%, which fully demonstrates the effectiveness of the module proposed in the present invention.

[0137] Table 2 Ablation experiment results of each module of the present invention

[0138] ,

[0139] Based on the distillation structure, image reconstruction strategy optimization and loss function proposed in the present invention, the network's ability to extract features is enhanced, the efficiency of model training is improved, and the model can better distinguish different features. The present invention has conducted a comprehensive verification experiment in the classification task of myopic macular degeneration based on fundus color photos, and the classification performance is good, indicating that the method of the present invention has good applicability in fundus image classification.

[0140] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A myopic maculopathy classification method based on knowledge distillation and self-supervised learning, characterized by: include, The self-supervised learning stage and the myopic maculopathy classification stage; The self-supervised learning stage is used to divide the input image into non-overlapping patches, extract features by constructing a teacher model and a student model, and predict the complete image through visible patches; The myopic macular lesion classification stage is used to fine-tune the classification network using a cross entropy loss function and a center loss function to obtain a classifier network, send the complete image to be predicted to the trained classifier network, and obtain the output of the lesion classification result; The self-supervised learning stage includes an image mask module, a knowledge distillation and feature encoding module, and an image reconstruction module; the encoder module in the image mask module uses a visual converter ViT as a backbone network to convert the input image into a sequence as the input of ViT, and converts the input image into a sequence. Divide into regular non-overlapping N=H×W / p 2 Patch blocks , where p 2 is the number of pixels in the patch block, H×W is the resolution of the input image, C is the number of channels, and the patch block Using Linear Projection Layer Mapped to a D-dimensional embedding vector , based on the embedding vector, add a learnable embedding vector , aggregating information from the entire sequence and adding learnable position embeddings to the patch embedding sequence , and get the ViT input sequence, expressed as: ， Randomly sample the patch sequence and mask the remaining patch blocks, that is, construct a random mask based on the obtained patch sequence ,set up Indicates that the corresponding patch block is masked, K represents the number of unmasked patch blocks, then the final encoder input sequence is ; The ViT input sequence models the remote information through the self-attention mechanism. For a given L-layer ViT, only some visible patches in the input sequence are included, i.e. It is fed into the encoder and mapped to latent features, represented as: ； ； in, represents the layer of ViT, LN(·) represents the layer normalization operation, in the multi-head self-attention MSA layer, each patch embedding vector calculates the self-attention map and updates itself by aggregating other patch embedding vectors. The output of the feature encoder includes the global feature representation and patch-level feature representation .

2. The myopic maculopathy classification method based on knowledge distillation and self-supervised learning according to claim 1, characterized in that: The knowledge distillation and feature encoding module uses a teacher-student model to initialize the teacher model and the student model respectively. The teacher model is initialized using the ViT model pre-trained on ImageNet, and the student model is randomly initialized. Extract features from the middle three layers of the student model and input them into a small projection layer whose output is required to mimic the features from the corresponding layers of the teacher model. The projection layer is implemented through a fully connected layer. Set the extracted l-th layer features of the student model and the teacher model to be ,in is the total number of patches, C is the number of channels; the middle three layers of the teacher model and the student model are taken as the layers for knowledge distillation, FC() is used to represent the projection network function, and the feature output dimension of the student model is aligned with the dimension of the teacher model through the fully connected layer. The distillation loss function is expressed as: ； Among them, S and T represent the student model and the teacher model respectively. Represents the total number of pixels, i represents the pixel index, represents the Manhattan distance.

3. The myopic maculopathy classification method based on knowledge distillation and self-supervised learning according to claim 2, characterized in that: The image reconstruction module is used to enhance the model's ability to reconstruct high-frequency details. It uses a Gaussian filter to smooth the image and obtain the low-frequency components of the image. Assume that the input image is ,in represents the pixel coordinates in the image, and the Gaussian filter kernel is expressed as: ， in, is the standard deviation of the Gaussian distribution, which is used to control the smoothness of the filter; The steps of extracting low-frequency components of an image using a Gaussian filter include: specifying the Gaussian kernel radius r and the standard deviation , thereby generating a Gaussian filter kernel of the corresponding size; convolve the Gaussian filter kernel with the input image to obtain the low-frequency component of the image ; Subtract the low-frequency component of the image from the original image to obtain the high-frequency component of the image , expressed as: ； ； The pixel value of the input image is represented in vector form, that is, assuming that the input pixel value is , then the high-frequency reconstruction target of the image is , calculate the reconstruction loss of the model , expressed as: ； in, represents the predicted pixel value, M represents the set of mask pixels, is the total number of pixels, i is the mask pixel index; The final loss function used in the self-supervised learning phase is expressed as: ； in, To reconstruct the loss function, is the distillation loss function for feature alignment.

4. The myopic maculopathy classification method based on knowledge distillation and self-supervised learning according to claim 3, characterized in that: Through the training in the self-supervised learning stage, the encoder part of the student model is extracted for fine-tuning; The loss function is a joint loss function that includes the cross entropy loss and the center loss function, expressed as: ； in, is the cross entropy loss function, is the center loss function, is a hyperparameter; In the last layer of the classification network, the fully connected layer outputs a vector with a length of P, the number of categories, Q is the total number of output vectors, and the cross entropy loss function is used to guide the training of the network, that is, to minimize the cross entropy loss function between the prediction result and the label. : ； in, is the indicator function that the true label of sample i is category j, that is, when the true category of the sample is j, otherwise ; is the probability that the model predicts that sample i is of category j.

5. The myopic maculopathy classification method based on knowledge distillation and self-supervised learning according to claim 4, characterized in that: The central loss function is introduced to enhance the feature discrimination and generalization capabilities of the model. The center loss function minimizes the distance between the sample feature and the center of the category to which it belongs, making the sample features of the same category more compact in the feature space. The center loss function is expressed as: ； in, represents the Euclidean distance, The feature representation obtained after the sample is input into the model, Is the category to which the sample belongs Category center of Taking advantage of the cross entropy loss and the center loss function, a joint loss function is introduced, which is expressed as: ； in, is the indicator function that the true label of sample i is category j, is the probability that the model predicts that sample i is of category j, represents the predicted pixel value, is the input pixel value, is a hyperparameter.

6. A myopic maculopathy classification system based on knowledge distillation and self-supervised learning, based on the myopic maculopathy classification method based on knowledge distillation and self-supervised learning according to any one of claims 1 to 5, characterized in that: include, The image mask module is used to divide the input image into non-overlapping patches, and retain a small random subset of them as visible patches for the encoder with a masking ratio of n%, while the remaining patches are used as prediction targets for the decoder; The knowledge distillation and feature encoding module is used to initialize the teacher model and the student model, input the visible patch blocks into the encoders of the teacher model and the student model for feature extraction, obtain the deep semantic features belonging to the teacher model and the student model respectively, and minimize the distance between the feature mappings of the middle layer of the teacher model and the student model; The image reconstruction module is used to predict the complete image through the visible patch block, extract the low-frequency component of the input image using a Gaussian filter based on a Gaussian kernel, and subtract the original image from its low-frequency component to obtain the high-frequency component of the input image as the reconstruction target of the model. The deep semantic features of the student model and the reconstruction target of the model are used as input, and the mean absolute error loss of the two is calculated. It is optimized together with the distillation loss to guide the model training; The myopic macular lesion classification stage includes taking out the encoder of the student model as the encoder of the myopic macular lesion classification task after the training in the self-supervised learning stage is completed, and fine-tuning the classification network using the cross entropy loss function and the center loss function to obtain the classifier network; sending the fundus color image to the trained classifier network to obtain the output of the lesion classification result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the myopic maculopathy classification method based on knowledge distillation and self-supervised learning as described in any one of claims 1 to 5 are implemented.

Images