A retinal OCT image lesion classification method based on an unsupervised heterogeneous distillation framework

Abstract

The present application relates to a kind of retinal OCT image lesion classification method based on unsupervised heterogeneous distillation framework, belong to image classification technical field.The method contains a teacher network and a student network, and the initial parameters of the teacher network are used as the initial parameters of the teacher network by the parameters of pre-training on natural image, and only a small amount of normal retinal OCT image is needed when training the method, and the parameters of the teacher network are not updated during training, and the features of the teacher network are input into the student network, and the shallow features generated by the teacher network are learned, in the test stage, whether the image to be measured is lesion image is judged by comparing the difference between the features generated by the teacher network and the student network, so as to alleviate the problem that a large number of medical image annotations are needed for training deep learning network.

Description

Technical Field

[0001] This invention relates to a method for classifying retinal OCT image lesions based on an unsupervised heterogeneous distillation framework, belonging to the field of image classification technology. Background Technology

[0002] According to the World Health Organization, approximately 300 million people worldwide suffered from eye diseases in 2010, including 39 million who went blind. About 80% of vision impairment is preventable. Among all the factors that can lead to vision loss, retinal diseases are a significant one. Common retinal diseases include diabetic macular edema, retinal occlusion, and glaucoma.

[0003] Prevention and early intervention of fundus diseases can avoid blindness and vision impairment. Optical coherence tomography (OCT), a novel medical imaging technique, is used for the diagnosis and treatment of ophthalmic diseases. OCT imaging technology has the advantages of being non-invasive and non-surgical, making it suitable for fundus tissue imaging. It can create a three-dimensional model of the retina from acquired two-dimensional scan slices, greatly facilitating diagnosis for doctors. By obtaining scan images near the macula and optic disc using OCT technology, a preliminary assessment of retinal morphology can be made. Because OCT technology can obtain deeper and more detailed information about retinal structures, the qualitative and diagnostic assessment of eye diseases is more accurate.

[0004] Retinal OCT image classification methods can be divided into traditional methods based on handcrafted features and deep learning methods based on convolutional neural networks (CNNs). Traditional methods mainly include edge detection methods, thresholding methods, chromatic aberration methods, and superpixel methods. These methods are primarily based on handcrafted features for image classification and are easily affected by image quality and noise. In contrast, convolutional neural networks can automatically extract features from images. Many CNN-based variants have been proposed for classifying retinal OCT images. Although CNN-based methods have better performance than handcrafted feature-based methods, CNN-based OCT image classification methods require a large amount of medical annotations for model training to improve model performance.

[0005] Unlike annotation of natural images, lesion annotation in medical images must be performed by experienced physicians. Classical deep learning methods for retinal OCT image classification require a large number of annotated retinal OCT images. However, annotating a large number of retinal OCT images would place a huge burden on physicians, making unsupervised methods with high accuracy the focus of current medical image analysis. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for classifying retinal OCT image lesions, which solves the problem that existing technologies require a large amount of medical annotation to achieve high model performance.

[0007] The technical solution of this invention is:

[0008] A retinal OCT image lesion classification method based on an unsupervised feature distillation framework is proposed. This method uses a teacher network and a student network. The teacher network uses parameters pre-trained on natural images as its initial parameters. Only a small number of normal retinal OCT images are needed for training the method, and the teacher network does not update its parameters during training. The student network takes the features of the teacher network as input and learns the shallow features generated by the teacher network. In the testing phase, the difference between the features generated by the teacher network and the student network is compared to determine whether the image to be tested is a lesion image. This can alleviate the problem of needing a large number of medical image annotations for training deep learning networks.

[0009] This method specifically includes the following steps:

[0010] S1: The input retinal OCT image is preprocessed. The teacher network extracts features from the preprocessed OCT image, resulting in four sets of features at different scales. These four sets of features represent the first-stage features of the teacher network. The second phase of teacher networks The third stage characteristics of teacher networks The fourth stage characteristics of teacher networks

[0011] S2, the student network with the fourth-stage characteristics of the teacher network obtained in step S1. As input, three sets of features at different scales are generated, which represent the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks The student network is a hybrid network composed of convolutional neural networks and transformer modules;

[0012] S3 optimizes the parameters of the student network. The optimization objective is to improve the first-stage characteristics of the student network. The first stage characteristics of teacher networks The second phase of student networks is closer to the characteristics of the network. The second phase of the teacher network The third stage characteristics of closer student networks The third stage characteristics of teacher networks A closer match was achieved, ultimately resulting in an optimized student network;

[0013] S4. The teacher network is used to extract features from the retinal OCT image to be tested, resulting in four sets of features at different scales. These four sets of features at different scales represent the first-stage features of the teacher network. The second phase of teacher networks The third stage characteristics of teacher networks The fourth stage characteristics of teacher networks

[0014] S5, using the fourth-stage features of the teacher network obtained in step S4. As input to the optimized student network in step S3, three sets of features at different scales are generated. These three sets of features at different scales represent the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks

[0015] S6, Calculate the first-stage features of the teacher network obtained in step S4. The first-stage features of the student network obtained in step S5 The similarity is then used to calculate the first-stage lesion score, Score1; similarly, the second-stage features of the teacher network are calculated. The second-stage features of the student network obtained in step S5 The similarity is used to calculate the second-stage lesion score (Score2); the third-stage features of the teacher network are then calculated. The third-stage features of the student network obtained in step S5 The similarity is used to calculate the third-stage lesion score Score3, and the first-stage lesion score Score1, the second-stage lesion score Score2, and the third-stage lesion score Score3 are added together to obtain the lesion score Score of the image to be tested.

[0016] In step S1, the preprocessing of the input retinal OCT image specifically involves compressing the input OCT image to a resolution of (H, W), where H ranges from 112 to 448 pixels, and W is the same as H.

[0017] In step S1, the teacher network is a convolutional neural network (also known as a CNN) pre-trained on the ImageNet large-scale dataset. Preferably, the teacher network can be a classification convolutional neural network such as ResNet, DenseNet, and VGGNet.

[0018] The teacher network uses weights pre-trained on the ImageNet dataset as initialization, and the parameter weights of the teacher network are not updated during the training phase.

[0019] The teacher network inherits the structure of the feature extraction blocks in the four stages of the classic classification convolutional neural network. However, the teacher network removes the last fully connected layer in the classic convolutional classification network. The four stages of the teacher network's feature extraction blocks generate four sets of features at different scales as described in step S1, which are denoted as the first-stage features of the teacher network. Characteristics of the second phase of the teacher network Characteristics of the third stage of teacher networks Characteristics of the fourth stage of teacher networks The four feature extraction blocks are the first stage featurer, the second stage featurer, the third stage featurer, and the fourth stage featurer, respectively.

[0020] In S1, the teacher network generates the four sets of features at different scales, wherein the extraction methods for the first-stage features, second-stage features, third-stage features, and fourth-stage features of the teacher network are as follows:

[0021] The preprocessed OCT image with a resolution of (H, W) is input into the first-stage feature generator of the teacher network to obtain the first-stage features of the teacher network. Among the features The dimensions are (H / 4, W / 4, 64); the second-stage feature extractor of the teacher network extracts the features from the first stage. Extracting and compressing the second-stage features into the teacher network in The dimensions are (H / 8, W / 8, 128); furthermore, the third-stage feature extractor of the teacher network will extract the second-stage features. Extracting and compressing the third-stage features of the teacher network in The dimensions are (H / 16, W / 16, 256); finally, the fourth-stage feature extractor of the teacher network extracts the features from the third stage. Extracting and compressing the fourth-stage features of the teacher network in The feature dimensions are (H / 32, W / 32, 512);

[0022] In step S2, the three sets of features at different scales generated by the student network are, in turn, the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks The method for generating three sets of features at different scales is as follows:

[0023] The student network exhibits characteristics of the fourth stage of the teacher network. The input is the fourth-stage feature, which is then processed by the student network. The student network consists of three stages to generate multi-scale features, and then processes the fourth-stage features. The decoding proceeds sequentially to a scale consistent with the three-stage features of the teacher network, where the three-stage features of the teacher network are respectively the first-stage features of the teacher. Second stage characteristics and third stage characteristics

[0024] The student network will incorporate the fourth-stage characteristics of the teacher network. Processing as the third stage feature of student networks in Dimensions H3, W3, and C3 represent the height, width, and number of channels of the feature, respectively; furthermore, the student network will incorporate the third-stage features. Generate second-stage features in Dimensions Finally, the student network will feature the second phase. Generate first-stage features in Dimensions

[0025] In step S2, the student network is a hybrid network consisting of a convolutional neural network and a transformer module. All three stages of the student network are composed of modular units, which include CNN sub-blocks (also called convolutional sub-blocks) and transformer sub-blocks. The modular units are as follows:

[0026] The module unit contains two parallel sub-blocks, namely the CNN sub-block and the transformer sub-block;

[0027] First, the module unit adjusts the dimensions of the input features using a 1×1 convolution. The adjusted new features are divided into two feature groups, denoted as the first group of features in the hybrid module and the second group of features in the hybrid module, respectively. The first group of features in the convolution module is then processed by CNN sub-blocks to generate feature F. Conv The second set of features from the hybrid module generates feature F after passing through the transformer sub-block. Tran Final feature F Conv and feature F Tran The features F are generated by stacking them side by side. Tran-Conv Feature F Tran-conv The number of channels is adjusted using 1×1 convolutions;

[0028] The specific structure of the CNN sub-block in the module unit is as follows: the convolution sub-block contains two consecutive convolution kernels of size 3×3, wherein the stride of the convolution is 1×1 and the padding is 1;

[0029] like Figure 3 As shown, the transformer sub-block in the module unit is a multi-scale sparse transformer module, with the following specific structure:

[0030] The multi-scale sparse transformer module consists of three parts: a feature aggregation module, a multi-head attention mechanism (MCA), and a multilayer perceptron (MLP). Assume the features input to the multi-scale sparse transformer module are input features F, and feature F has a dimension of... Among them (H) i W i ) represents the resolution of feature F in the i-th stage of the student network, C i This represents the dimension of the i-th stage channel in the student network, where 4C1 = 2C2 = C3, H1 = 2H2 = 4H3 and W1 = 2W2 = 4W3; preferably, the values ​​of H1 and W1 range from 56 to 224 pixels, and the value of C3 ranges from 256 to 1024.

[0031] The feature aggregation module generates two types of features. The first type of feature is a local feature F. local The second type of feature generated is the region feature F. region , where F local The dimension is and F region The dimension is R Ci×(Hj·j) ;

[0032] Specifically, the local feature F local (The first feature) is obtained in the following way:

[0033]

[0034] j = 1, ..., N l N l =H i ·W i ,

[0035] Among them, features These are the features after the input feature F of the multi-scale sparse transformer module has been deformed. The dimension is f i express The components of the feature, f i Dimensions Nl =H i ·W i The number of components. Represents location embedding features;

[0036] Specifically, the region feature F region (The second feature) is obtained in the following way:

[0037] First, the input features Divide the blocks into disjoint feature block sequences F using blocks of size (p, p). p Feature sequence F p The dimension is in Indicates the number of sequence blocks and This represents the i-th block; preferably, the value of p ranges from 1 to 8 pixels.

[0038] Then, the disjoint feature block sequences F p Transformed into sequence blocks by matrix E The dimension of matrix E is Furthermore, a learnable location embedding E pos With feature sequence blocks Adding the components of the generated region features The dimension is and E pos The dimension is Components of regional features The calculation is as follows:

[0039]

[0040] j = 1, ..., N p ,

[0041] The multi-head attention mechanism in the multi-scale sparse transformer module is as follows:

[0042] First, the local feature F local matrix Linear mapping to Queriy values ​​(Queriy values ​​are denoted as Q) j ), where the matrix Dimensions Q j The dimension is The dimension and N representing the single-head attention mechanism i This represents the number of single-head attention mechanisms in the i-th stage;

[0043] At the same time F region matrix Linear mapping to key value (key value denoted as...) ),in The dimension is The dimension is F region Also in matrix Linear mapping to Value (Value denoted as...) ),in The dimension is Dimensions Query value (Q) j ), key value and value It can be defined as follows:

[0044]

[0045]

[0046]

[0047] The multi-head attention mechanism calculates the Query value (Q) using the single-head attention mechanism. j ), key value and value The relationship is as follows:

[0048]

[0049] further, The single-head attention is combined to obtain the multi-head attention mechanism (MCA), which is represented as follows:

[0050]

[0051]

[0052] Furthermore, the two multi-head attention mechanisms are merged. Specifically, a multi-head attention mechanism with a block size of p1 is used. and a multi-head attention mechanism with a block size of p2 The features Z are combined and obtained as shown in the following formula:

[0053]

[0054] Finally, the feature Z is enhanced using a regularization layer (LN) and a multilayer perceptron (MLP) to obtain the final feature. feature It can be expressed by the following formula:

[0055]

[0056] Where p1 and p2 represent the block size, MLP represents multilayer perceptron and LN represents layer regularization;

[0057] In step S3, the parameters of the student network are optimized. The optimization goal of the student network is to make the three sets of features of the student network more similar to the three sets of features of the teacher network. The specific method is as follows:

[0058] The characteristics of the teacher network representing the k-th stage (h,w) position. Features of the student network representing the k-th stage (h,w) position; L k (h,w) represents the loss of teacher network features and student network features at position (h,w) in stage k, L k The (h,w) loss is mainly composed of a weighted sum of the cosine function cosine and the squared loss mse, and the specific mathematical formula is as follows:

[0059]

[0060] Where α represents the weight, preferably, the value range is between (0,1);

[0061] Ultimately, the students' network suffered a complete loss. It is the sum of the losses from the three stages, as shown in the following formula:

[0062]

[0063] Among them (H) k W k ) represents the resolution size of the k-th stage feature, and K represents the total number of stages for the student.

[0064] In step S6, the feature similarity between the teacher network and the student network is calculated to obtain the lesion score of the image under test. The lesion score is specifically implemented as follows:

[0065] The characteristics of the teacher network representing the k-th stage (h,w) position. The characteristics of the student network representing the k-th stage (h,w) position; Representative characteristics and characteristics similarity, Represents the lesion score;

[0066] First-stage characteristics of computational teacher networks The first-stage features of the student network obtained in step S5 The similarity is used to calculate the first-stage lesion score, Score1, as shown below.

[0067]

[0068] Computing the second-stage characteristics of teacher networks The second-stage features of the student network obtained in step S5 The similarity is used to calculate the second-stage lesion score, Score2, as shown below.

[0069]

[0070] The third-stage characteristics of computational teacher networks The third-stage features of the student network obtained in step S5 similarity,

[0071]

[0072] The final image lesion score is calculated by summing the student's features from the three stages and the lesion scores from the teacher network from the three stages. The specific formula is as follows.

[0073]

[0074] Beneficial effects

[0075] Compared with related technologies, the method of the present invention has the following advantages:

[0076] 1. The method described is a framework based on unsupervised heterogeneous knowledge distillation for retinal OCT image lesion classification. This method achieves high lesion classification performance by learning the feature distribution of only a small number of normal samples. In this framework, the teacher network is a general-purpose classification network based on a convolutional neural network, while the student network is a hybrid network based on CNN and transformer. The student and teacher networks are heterogeneous structures; during training, only normal samples are used to allow the student network to learn the features of the teacher network. During testing, lesion detection is achieved by calculating the feature differences between the student and teacher networks.

[0077] 2. To fully leverage the heterogeneous advantages of the unsupervised heterogeneous knowledge distillation framework, a multi-scale sparse transformer is designed to improve the lesion classification performance of the classification method. This multi-scale sparse transformer, while capable of modeling long-distance feature dependencies, also alleviates the problems of high computational cost and high memory consumption associated with transformer methods.

[0078] 3. In the classification method described above, using the feature output of the teacher network as the feature input of the student network is equivalent to introducing the idea of ​​autoencoder feature compression and feature reconstruction into the distillation method. This feature compression and feature recovery structure can alleviate the over-detection problem of unsupervised distillation method in retinal OCT image lesions. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of the process in the method and embodiments of the present invention;

[0080] Figure 2 This is a schematic diagram of a retinal OCT image lesion classification method based on an unsupervised heterogeneous distillation network provided in an embodiment of the present invention;

[0081] Figure 3 This is a schematic diagram of the multi-scale sparse transformer module structure in an embodiment of the present invention. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] This invention presents a method for classifying retinal OCT image lesions based on an unsupervised heterogeneous distillation framework. This method employs two structurally dissimilar sub-networks, a teacher network and a student network, as the distillation framework. To further differentiate the student and teacher networks, a multi-scale sparse transformer module is proposed to model long-distance feature relationships and mitigate the inherently high computational cost of the transformer. Furthermore, the output of the teacher model is used as the input to the student network, thus introducing feature compression and feature recovery into the distillation framework. This effectively preserves features from normal OCT images while removing features from abnormal OCT images, ultimately achieving lesion classification in the OCT images.

[0084] Figure 1 This is a schematic diagram of the process in the method and embodiments of the present invention, such as... Figure 1 As shown, it includes the following 6 steps:

[0085] S1: The input retinal OCT image is preprocessed. The teacher network extracts features from the preprocessed OCT image, resulting in four sets of features at different scales. These four sets of features represent the first-stage features of the teacher network. The second phase of teacher networks The third stage characteristics of teacher networks The fourth stage characteristics of teacher networks

[0086] S2, the student network with the fourth-stage characteristics of the teacher network obtained in step S1. As input, three sets of features at different scales are generated, which represent the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks The student network is a hybrid network composed of convolutional neural networks and transformer modules;

[0087] S3 optimizes the parameters of the student network. The optimization objective is to improve the first-stage characteristics of the student network. The first stage characteristics of teacher networks The second phase of student networks is closer to the characteristics of the network. The second phase of the teacher network The third stage characteristics of closer student networks The third stage characteristics of teacher networks A closer match was achieved, ultimately resulting in an optimized student network;

[0088] S4. The teacher network is used to extract features from the retinal OCT image to be tested, resulting in four sets of features at different scales. These four sets of features at different scales represent the first-stage features of the teacher network. The second phase of teacher networks The third stage characteristics of teacher networks The fourth stage characteristics of teacher networks

[0089] S5, using the fourth-stage features of the teacher network obtained in step S4. As input to the optimized student network in step S3, three sets of features at different scales are generated. These three sets of features at different scales represent the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks

[0090] S6, Calculate the first-stage features of the teacher network obtained in step S4. The first-stage features of the student network obtained in step S5 The similarity is then used to calculate the first-stage lesion score, Score1; similarly, the second-stage features of the teacher network are calculated. The second-stage features of the student network obtained in step S5 The similarity is used to calculate the second-stage lesion score (Score2); the third-stage features of the teacher network are then calculated. The third-stage features of the student network obtained in step S5 The similarity is used to calculate the third-stage lesion score Score3, and the first-stage lesion score Score1, the second-stage lesion score Score2, and the third-stage lesion score Score3 are added together to obtain the lesion score Score of the image to be tested.

[0091] Step 1: Preprocess the input retinal OCT image, and then the teacher network extracts features from the preprocessed OCT image and generates four sets of features at different scales in sequence;

[0092] Step 1.1: Preprocess the input retinal OCT image by compressing it to a resolution of (256, 256).

[0093] Step 1.2: The teacher network extracts features from the preprocessed OCT image and generates four sets of features at different scales. The teacher network is a convolutional neural network pre-trained on the ImageNet large-scale dataset. The teacher network uses the ResNet series of networks, and the parameters of the ResNet network pre-trained on the ImageNet large-scale dataset are used as the weights to initialize the network. During the training phase of the method, the parameters of the teacher network are no longer updated. The teacher network removes the last fully connected layer in the ResNet classification network and only inherits the structure of the four-stage feature extraction blocks of the ResNet classification network. The four-stage feature extraction blocks are the first-stage featurer, the second-stage featurer, the third-stage featurer, and the fourth-stage featurer.

[0094] Furthermore, such as Figure 2 The four feature extraction blocks shown in (a) generate four sets of features at different scales generated by the teacher network in step 1.2, which are denoted as the first-stage features of the teacher network. Characteristics of the second phase of the teacher network The third stage characteristics of teacher networks Characteristics of the fourth stage of teacher networks The methods for extracting the first-stage, second-stage, third-stage, and fourth-stage features of the teacher network are as follows:

[0095] After inputting the preprocessed OCT image with a resolution of (256, 256) into the first-stage feature generator of the teacher network, the dimension of the feature is transformed to (64, 64, 64), thus obtaining the first-stage feature of the teacher network. The second-stage feature extractor of the teacher network compresses the first-stage feature extraction into the second-stage features of the teacher network. Its dimensions are (32, 32, 128); furthermore, the third-stage feature extractor of the teacher network compresses the second-stage feature extraction into the third-stage features of the teacher network. Its dimensions are (16, 16, 256); finally, the fourth-stage feature extractor of the teacher network compresses the third-stage feature extraction into the fourth-stage features of the teacher network. The feature dimensions are (8, 8, 512).

[0096] Step 2: As Figure 2 As shown in (a), the three sets of features at different scales generated by the student network are, in order, the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks The specific implementation method is as follows:

[0097] The student network takes the fourth-stage features of the teacher network as input, and then processes these features. The student network consists of three stages to generate multi-scale features. The student model sequentially decodes the features to a scale consistent with the three stages of the teacher network; the fourth-stage features of the teacher network are transformed into features through the first stage of the student model. Where the dimension is Similarly, the second-stage generative features of student networks in Dimensions Third-stage generation characteristics of student networks in Dimensions

[0098] In the aforementioned steps, the student network is a hybrid network composed of convolutional neural networks and transformer modules. All three stages of the student network are composed of modular units, which include CNN sub-blocks (also called convolutional sub-blocks) and transformer sub-blocks. These modular units are:

[0099] The module unit contains two parallel sub-blocks, namely the CNN sub-block and the transformer sub-block;

[0100] First, the module unit adjusts the dimensions of the input features using a 1×1 convolution. The adjusted new features are divided into two feature groups, denoted as the first group of features in the hybrid module and the second group of features in the hybrid module, respectively. The first group of features in the convolution module is then processed by CNN sub-blocks to generate feature F. Conv The second set of features from the hybrid module generates feature F after passing through the transformer sub-block. Tran Final feature F Conv and feature F Tran The features F are generated by stacking them side by side. Tran-Conv Feature F Tran-Conv The number of channels is adjusted using 1×1 convolutions;

[0101] The specific structure of the CNN sub-block in the module unit is as follows: the convolution sub-block contains two consecutive convolution kernels of size 3×3, wherein the stride of the convolution is 1×1 and the padding is 1;

[0102] The transformer sub-block in the module unit is a multi-scale sparse transformer module, with the following specific structure:

[0103] The multi-scale sparse transformer module consists of three parts: a feature aggregation module, a multi-head attention mechanism (MCA) computation, and a multilayer perceptron (MLP). Assume the features input to the multi-scale sparse transformer module are input features F, and feature F has a dimension of... Among them (H) i W i ) represents the resolution of feature F in the i-th stage of the student network, C i Let H1 = 2C2 = C3, H1 = 2H2 = 4H3 and W1 = 2W2 = 4W3; H1 = 64, W1 = 64 and C3 = 512;

[0104] The feature aggregation module generates two types of features. The first type of feature is a local feature F. local The second type of feature generated is the region feature F. region F local The dimension is and F region The dimension is

[0105] Specifically, the local feature F local (The first feature) is obtained in the following way:

[0106]

[0107] j = 1, ..., N lN l =H i ·W i ,

[0108] Among them, features These are the features after the input feature F of the multi-scale sparse transformer module has been deformed. The dimension is f i express The components of the feature, f i Dimensions N l =H i ·W i The number of components. Represents location embedding features;

[0109] Specifically, the region feature F region (The second feature) is obtained in the following way:

[0110] First, the input features Divide the blocks into disjoint feature block sequences F using blocks of size (p, p). p Feature sequence F p The dimension is in Indicates the number of sequence blocks and This represents the i-th block;

[0111] Then, the disjoint feature block sequences F p Transformed into sequence blocks by matrix E The dimension of matrix E is Furthermore, a learnable location embedding E pos With feature sequence blocks Adding the components of the generated region features The dimension is and E pos The dimension is Components of regional features The calculation is as follows:

[0112]

[0113] j = 1, ..., N p ,

[0114] The multi-head attention mechanism in the multi-scale sparse transformer module is as follows:

[0115] First, the local feature F local matrix Linear mapping to Queriy values ​​(Queriy values ​​are denoted as Q)j ), where the matrix Dimensions Q j The dimension is The dimension and N representing the single-head attention mechanism i This represents the number of single-head attention mechanisms in the i-th stage;

[0116] The total number of student network channels at different stages is: C1 = 64, C2 = 128, C3 = 256.

[0117] Among them, the number N of multi-head attention mechanisms i In different stages i, the values ​​are: N1 = 2, N2 = 4, N3 = 8.

[0118] At the same time F region matrix Linear mapping to key value (key value denoted as...) ),in The dimension is The dimension is F region Also in matrix Linear mapping to Value (Value denoted as...) ),in The dimension is Dimensions Query value (Q) j ), key value and value It can be defined as follows:

[0119]

[0120]

[0121]

[0122] The multi-head attention mechanism calculates the Query value (Q) using the single-head attention mechanism. j ), key value and value The relationship is as follows:

[0123]

[0124] further, Single-head attentions are combined to obtain multi-head attention (MCA), which is represented as follows:

[0125]

[0126]

[0127] Furthermore, the two multi-head attention mechanisms are merged. Specifically, a multi-head attention mechanism with a block size of p1 is used. and a multi-head attention mechanism with a block size of p2 The features Z are combined and obtained as shown in the formula below;

[0128]

[0129] Finally, the feature Z is enhanced using a regularization layer (LN) and a multilayer perceptron (MLP) to obtain the final feature. feature It can be expressed by the following formula:

[0130]

[0131] Where p1 and p2 represent the block size, MLP represents multilayer perceptron and LN represents layer regularization; in the first stage of the student network, p1=4 and p2=8; in the second stage of the student network, p1=2 and p2=4; in the third stage of the student network, p1=1 and p2=2.

[0132] Step 3: Optimize the parameters of the student network. The optimization goal of the student network is to make the three sets of features of the student network more similar to the three sets of features of the teacher network. The specific method is as follows:

[0133] The characteristics of the teacher network representing the k-th stage (h,w) position. Features of the student network representing the k-th stage (h,w) position; L k (h,w) represents the loss of teacher network features and student network features at position (h,w) in stage k, L k The (h,w) loss is mainly composed of a weighted sum of the cosine function cosine and the squared loss mse, and the specific mathematical formula is as follows:

[0134]

[0135] Ultimately, the complete loss of the student network is the sum of the losses from the three stages, as shown in the following formula:

[0136]

[0137] Among them (H) k W k) represents the resolution size of the k-th stage feature, K=3 represents the total number of stages for the student; H1=2H2=4H3, W1=2W2=4W3, H1=64, W1=64;

[0138] Step 4: Use the teacher network to extract features from the retinal OCT image to be tested, obtaining four sets of features at different scales. These four sets of features at different scales represent the first-stage features of the teacher network. The second phase of teacher networks The third stage characteristics of teacher networks The fourth stage characteristics of teacher networks

[0139] Step 5: Fourth-stage characteristics of the teacher network obtained in Step 4 As input to the optimized student network from step 3, three sets of features at different scales are generated. These three sets of features at different scales represent the third-stage features of the student network. The second phase characteristics of student networks The first phase characteristics of student networks

[0140] Step 6: Calculate the feature similarity between the teacher network and the student network to finally obtain the lesion score of the image under test. The lesion score is implemented as follows:

[0141] The characteristics of the teacher network representing the k-th stage (h,w) position. The characteristics of the student network representing the k-th stage (h,w) position; Representative characteristics and characteristics similarity, Represents the lesion score;

[0142] First-stage characteristics of computational teacher networks The first-stage characteristics of the student network obtained in step 5 The similarity is used to calculate the first-stage lesion score, Score1, as shown below.

[0143]

[0144] Computing the second-stage characteristics of teacher networks The second-stage features of the student network obtained in step 5 The similarity is used to calculate the second-stage lesion score, Score2, as shown below.

[0145]

[0146] The third-stage characteristics of computational teacher networks The third-stage characteristics of the student network obtained in step 5 similarity,

[0147]

[0148] The final image lesion score is calculated by summing the student's features from the three stages and the lesion scores from the teacher network from the three stages. The specific formula is as follows.

[0149]

[0150] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework, characterized in that... The steps of this method include: S1. The input retinal OCT image is preprocessed. The teacher network extracts the features of the preprocessed retinal OCT image to obtain four sets of features at different scales. The four sets of features at different scales are the first-stage features of the teacher network, the second-stage features of the teacher network, the third-stage features of the teacher network, and the fourth-stage features of the teacher network. S2, the student network takes the fourth-stage features of the teacher network obtained in step S1 as input and generates three sets of features at different scales. The three sets of features at different scales are the third-stage features of the student network, the second-stage features of the student network, and the first-stage features of the student network. S3. Optimize the parameters of the student network. The optimization goal of the student network is to make the first-stage features of the student network closer to the first-stage features of the teacher network, the second-stage features of the student network closer to the second-stage features of the teacher network, and the third-stage features of the student network closer to the third-stage features of the teacher network, so as to obtain the optimized student network. S4. The teacher network is used to extract features from the retinal OCT image to be tested, resulting in four sets of features at different scales. These four sets of features at different scales are the first-stage features, the second-stage features, the third-stage features, and the fourth-stage features of the teacher network. S5. Using the fourth-stage features of the teacher network obtained in step S4 as input to the student network optimized in step S3, three sets of features at different scales are generated. The three sets of features at different scales are the third-stage features of the student network, the second-stage features of the student network, and the first-stage features of the student network. S6. Calculate the similarity between the first-stage features of the teacher network obtained in step S4 and the first-stage features of the student network obtained in step S5, and use it to calculate the first-stage lesion score Score1. Calculate the similarity between the second-stage features of the teacher network obtained in step S4 and the second-stage features of the student network obtained in step S5, and use it to calculate the second-stage lesion score Score2. Calculate the similarity between the third-stage features of the teacher network obtained in step S4 and the third-stage features of the student network obtained in step S5, and use it to calculate the third-stage lesion score Score3. Add the first-stage lesion score Score1, the second-stage lesion score Score2, and the third-stage lesion score Score3 to obtain the final lesion score Score of the image to be tested. The teacher network uses pre-trained weights on the ImageNet dataset as initialization, and its parameters are not updated during training. The teacher network is a general-purpose classification network based on a convolutional neural network, while the student network is a hybrid network based on CNN and transformer. The student and teacher networks are heterogeneous. During training, only normal samples are used to allow the student network to learn the features of the teacher network. During testing, lesion detection is achieved by calculating the feature differences between the student and teacher networks.

2. The retinal OCT image lesion classification method based on an unsupervised feature distillation framework according to claim 1, characterized in that: In step S1, the preprocessing of the input retinal OCT image specifically involves compressing the input retinal OCT image to a resolution of (H, W); the value of H ranges from 224 to 448 pixels, and W is the same as H.

3. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 1 or 2, characterized in that: In step S1, the teacher network is a ResNet classification convolutional neural network, a DenseNet classification convolutional neural network, or a VGGNet classification convolutional neural network.

4. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 1 or 2, characterized in that: In step S1, the method for extracting features from the teacher network is as follows: after inputting the preprocessed retinal OCT image with a resolution of (H, W) into the first-stage feature extractor of the teacher network, the dimension of the feature is transformed into (H / 4, W / 4, 64) to obtain the first-stage features of the teacher network. The second-stage feature extractor of the teacher network compresses the first-stage feature extraction into the second-stage features of the teacher network, with dimensions (H / 8, W / 8, 128). The third-stage feature extractor of the teacher network compresses the second-stage feature extraction into the third-stage features of the teacher network, with dimensions (H / 16, W / 16, 256). The fourth-stage feature extractor of the teacher network compresses the third-stage feature extraction into the fourth-stage features of the teacher network, with dimensions (H / 32, W / 32, 512).

5. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 1, characterized in that: In step S2, the convolutional neural network in the student network contains two consecutive convolutional kernels with a size of [missing value]. The convolution, where the stride of the convolution is... The padding is 1; the Transformer module is a multi-scale sparse transformer module.

6. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 5, characterized in that: The multi-scale sparse transformer module includes a feature aggregation module, a multi-head attention mechanism, and a multilayer perceptron; The feature aggregation module includes local features. and regional characteristics ; Local features for: Among the features The input features of the multi-scale sparse transformer module Characteristics after deformation The dimension is , express The components of the features, Dimensions , The number of components. Represents location embedding features; Regional characteristics The calculation method is as follows: First, features The size used is ( , The blocks are divided into disjoint feature block sequences. ,in( ) represents the feature of the i-th stage. resolution, This represents the dimension of the channel in the i-th stage. Indicates the number of blocks and This represents the i-th block; Then, disjoint feature block sequences matrix Mapping transformation into sequence blocks , where the matrix The dimension is Learnable position embedding With feature sequence blocks Adding the components of the generated region features Components of regional features The calculation is as follows: 。 7. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 6, characterized in that: The input Query, Key, and Value values ​​in the multi-head attention mechanism of the multi-scale sparse transformer module are calculated as follows: First, the local features matrix A linear mapping to the query value, denoted as . , where the matrix Dimensions , The dimension is , Dimensions representing single-head attention mechanisms and This represents the number of single-head attention mechanisms in the i-th stage; at the same time The amount matrix A linear mapping to a key value, where the key value is denoted as... ,in The dimension is , The dimension is ; The amount Also in matrix Linear mapping to Value, denoted as ,in The dimension is , Dimensions ; Query value key value and value Defined as follows: 。 8. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 7, characterized in that: The multi-head attention mechanism is used to calculate the local and regional information of the single-head attention mechanism, as shown below: ， The single-head attention is merged together to obtain the multi-head attention mechanism MCA, which is represented as follows: ， The two-scale multi-head attention mechanisms are merged together, specifically, with a block size of [missing information]. Multi-head attention mechanism and has a block size of Multi-head attention mechanism To obtain features by merging them together The formula is shown below: ， Finally, the features The final features are obtained by enhancing features using a regularization layer (LN) and a multilayer perceptron. ,feature It can be expressed by the following formula: ， in and The value represents the size of the block, MLP represents a multilayer perceptron, and LN represents layer regularization.

9. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 6, characterized in that: In step S3, the parameters of the student network are optimized. The optimization goal of the student network is to make the three sets of features of the student network more similar to the three sets of features of the teacher network. Specifically: Represents the kth stage ( Characteristics of the teacher network at different locations Represents the kth stage ( Characteristics of student networks at different locations; Indicates the k-th stage ( Loss of network characteristics of teachers and students in the location. The loss is mainly composed of a weighted average of the cosine function cosine and the squared loss mse, and the specific mathematical formula is as follows: in This represents the weight, and its value ranges between (0,1). Ultimately, the students' network suffered a complete loss. It is the sum of the losses from the three stages, as shown in the following formula: in( ) represents the resolution size of the feature in the k-th stage. This represents the total number of stages a student has completed.

10. A method for classifying retinal OCT image lesions based on an unsupervised feature distillation framework according to claim 6, characterized in that: Represents the kth stage ( Characteristics of the teacher network at different locations Represents the kth stage ( Characteristics of student networks based on location; Representative characteristics and characteristics similarity, Represents the lesion score; First-stage characteristics of computational teacher networks The first-stage features of the student network obtained in step S5 The similarity is used to calculate the first-stage lesion score, Score1, as shown below. , Computing the second-stage characteristics of teacher networks The second-stage features of the student network obtained in step S5 The similarity is used to calculate the second-stage lesion score, Score2, as shown below. , The third-stage characteristics of computational teacher networks The third-stage features of the student network obtained in step S5 similarity, , The final lesion score is obtained by summing the lesion scores from the three stages, as shown in the following formula. 。

Images