Diabetic retinopathy feature decoupling method for ultra-wide-angle fundus image

Abstract

The invention discloses a diabetic retinopathy feature decoupling method for an ultra-wide-angle fundus image, and relates to the technical field of deep learning medical image processing, and the method sequentially comprises the steps of data preprocessing, data enhancement, model construction, model training and prediction, and model application. According to the method, features of an ultra-wide-angle fundus image are decoupled into diagnosis-related features and diagnosis-unrelated features of diabetic retinopathy, and an encoder of a trained feature decoupling network can directly extract the diagnosis-related features and the diagnosis-unrelated features of the ultra-wide-angle fundus image features; the decoder can directly generate ultra-wide-angle eye fundus image features under specific conditions through diagnosis-related features and diagnosis-unrelated features; the diabetic retinopathy grading performance of a self-supervised training model can be remarkably improved, the method exceeds a supervised training model, and the effectiveness of the decoupled diabetic retinopathy diagnosis related features is proved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning medical image processing, and in particular to a method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images. Background Art

[0002] Compared with traditional color fundus imaging technology, ultra-wide-angle fundus imaging technology has the advantages of fast imaging speed, wide coverage, no need for mydriasis, and large depth of field. Since the successful commercial application of ultra-wide-angle imaging technology many years ago, a large number of ophthalmology clinics and professional ophthalmology hospitals have introduced this technology. Today, this technology has become the mainstream method for fundus disease examination in China.

[0003] Diabetic retinopathy: Diabetic retinopathy is a complication of diabetes that affects the eyes. It is caused by damage to the blood vessels in the light-sensitive tissue at the back of the eye. At first, diabetic retinopathy may not cause symptoms or only mild vision problems, but it can lead to blindness. Anyone with type 1 or type 2 diabetes can develop it. The longer you have diabetes and the worse your blood sugar control, the greater your chance of developing this eye complication.

[0004] Feature decoupling: Feature decoupling is a method of designing a neural network architecture and training process to make the features learned by the network independent or uncorrelated. The main purpose of feature decoupling is to improve the generalization ability and interpretability of the model. In practical applications, feature decoupling helps deep learning models better understand and process complex data, such as images, speech, or text.

[0005] The current decoupled ultra-wide-angle fundus image features of diabetic retinopathy mainly involve the following specific technical issues:

[0006] 1. It is necessary to develop a set of image preprocessing methods and image enhancement methods that are beneficial to the decoupling of ultra-wide-angle fundus image features;

[0007] 2. It is necessary to develop a deep neural feature decoupling model that can successfully decouple the ultra-wide-angle fundus image features while ensuring that the decoupled features have clear meanings;

[0008] 3. Apply the decoupled features to specific computer-aided diabetic retinopathy diagnosis tasks to improve the performance of related computer-aided diagnosis systems.

[0009] Feature decoupling is an important research direction in the field of representation learning, but due to the difficulty in obtaining ultra-wide-angle fundus images and the complexity of the decoupling task of diabetic retinopathy, related research reports are relatively rare. In order to better illustrate the technical features of the present invention, the following will describe the related patent documents of feature decoupling that have been published and are close to the present invention.

[0010] Feature decoupling aims to separate the confounding factors in the features, improving the performance of related downstream artificial intelligence tasks while improving the interpretability of the model. You Yanan et al. (CN202311499603.6, a fine-grained classification method and system based on essential causal reasoning) decouple the extracted essential features through a preset feature decoupling module. Specifically, by mapping the feature vectors in the high-dimensional non-orthogonal space to another orthogonal space of the same dimension to disentangle the features, fine-grained prediction and classification are finally completed; Cao Xiaochun et al. (CN202011447934.1, a face privacy protection method and device based on feature decoupling) cleverly use multiple encoders and the idea of ​​generative adversarial to decouple facial features into identity features and appearance features, and on this basis, manipulate the features and generate real features. The protection of facial privacy information is now realized; Zou Bin et al. (CN202310523139.3, SAR true and false target identification and target recognition method based on decoupling and reconstruction learning) decoupled and reconstructed synthetic aperture radar images by designing a special decoupling loss function under the modeling paradigm of the codec, thereby realizing the identification and recognition of true and false targets in synthetic aperture radar images; Wang Rui et al. (CN202210148651.X, a text-image pair generation method and device based on feature decoupling) decoupled the independent information of image features in the latent space through cross-modal text information, and realized controllable image generation technology by manipulating generation on the basis of obtaining meaningful decoupled features.

[0011] The above decoupling methods are all researched and developed based on remote sensing images or face images. Feature decoupling, as one of the important technologies for realizing trusted artificial intelligence, is of great significance for building a reliable computer-aided diagnosis system for diabetic retinopathy in the future. Therefore, it is urgent to research and develop feature decoupling methods in the field of medical image processing. Summary of the invention

[0012] Abstract: Based on the problems existing in the prior art, the present invention aims to provide a method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images. The trained feature decoupling network has an encoder that can directly extract diagnosis-related features and diagnosis-irrelevant features of ultra-wide-angle fundus image features, and the encoder can directly produce ultra-wide-angle fundus image features under specific conditions through diagnosis-related features and diagnosis-irrelevant features.

[0013] The present invention is achieved through the following technical solutions:

[0014] In a first aspect, the present application provides a method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images, comprising sequentially performing data preprocessing, data enhancement, model construction, model training and prediction, and model application.

[0015] Furthermore, the data preprocessing includes the following steps:

[0016] A. Perform center cropping on all ultra-wide-angle fundus images to obtain cropped ultra-wide-angle fundus images;

[0017] B. applying a contrast adaptive histogram equalization algorithm to all the cropped ultra-wide-angle fundus images to obtain an ultra-wide-angle fundus image processed by the contrast adaptive histogram equalization;

[0018] C. performing image scaling on all ultra-wide-angle fundus images processed by contrast adaptive histogram equalization;

[0019] D. Randomly divide the training data set and the test data set, calculate the statistical features of the training data set, the validation data set and the RGB three-way in each fold division data set, and select the fold division result with the closest statistical features as the final training data set and validation data set.

[0020] Furthermore, the data enhancement method includes an image geometric transformation enhancement method and an image pixel transformation enhancement method.

[0021] Furthermore, the model construction adopts multivariate Gaussian distribution to model the decoupled features, and based on this, a feature decoupling loss function is designed to ensure the independence of the modeled features. At the same time, a diabetic retinopathy continuous development loss function is also designed based on this to ensure the directionality of the modeled features.

[0022] Among them, the continuous development loss function of diabetic retinopathy is calculated by a continuous development model of diabetic retinopathy; the continuous development model of diabetic retinopathy regards the development of diabetic retinopathy as a continuous process, and the next level of diabetic retinopathy characteristics include all the characteristics of the previous level of diabetic retinopathy.

[0023] Furthermore, the feature decoupling loss function L d for:

[0024]

[0025] Among them, μ c ,μ s , They represent the mean and variance of class-related features and class-irrelevant features respectively, and J represents the dimension of the feature.

[0026] Furthermore, the diabetic retinopathy continuous progression loss function L chn for:

[0027]

[0028] Where Y represents the different diabetic retinopathy grades in a batch; w represents the unique characteristics of each diabetic retinopathy grade relative to its previous diabetic retinopathy grade in the continuous development model of diabetic retinopathy, and this characteristic is still described by Gaussian distribution.

[0029] Furthermore, the model training and prediction includes the following steps:

[0030] A. Build a model training data enhancement pipeline;

[0031] B. Initialize the feature decoupling model;

[0032] C. Initialize the optimizer and learning rate scheduling strategy;

[0033] D. The maximum number of iterations of model training is 100. Under the cyclic iteration of the pipeline, the training data set is input into the feature decoupling model in turn for forward propagation. Then the total loss function of the feature decoupling model is calculated and the error is back-propagated. Finally, the optimizer is used to update the weight bias parameters of the feature decoupling model. The above training process is repeated until the maximum number of model iterations is reached.

[0034] E. Build a model prediction data enhancement pipeline;

[0035] F. Fix the parameters of the trained feature decoupling model, send the test data set into the feature decoupling model in sequence to perform feature decoupling, visualize the decoupling data, and evaluate the decoupling performance of the decoupling model.

[0036] Furthermore, the total loss function L of the feature decoupling model is:

[0037]

[0038] Among them, α, β, γ c , γ s , are all hyperparameters.

[0039] Furthermore, the model application includes the following steps:

[0040] A. Train the feature extraction backbone network;

[0041] B. Train feature decoupling model;

[0042] C. Train a linear classifier.

[0043] D. Evaluation and verification.

[0044] Furthermore, the training feature extraction backbone network includes the following steps:

[0045] 1) Build a model training data enhancement pipeline;

[0046] 2) Construct feature extraction backbone network;

[0047] 3) Initialize the optimizer and learning rate scheduling strategy;

[0048] 4) The maximum number of iterations of model training is 800. Under the cyclic iteration of the pipeline, the training data set is sequentially input into the feature extraction backbone network and the contrastive learning prediction head for forward propagation, then the contrastive loss is calculated and the error is back-propagated. Finally, the optimizer is used to update the weight bias parameters of the feature extraction backbone network. The above training process is repeated until the maximum number of model iterations is reached;

[0049] 5) Take the model parameters of the last iteration as the final weight bias parameters of the feature extraction backbone network to complete the training of the feature extraction backbone network.

[0050] Furthermore, the contrast loss uses a supervised contrast loss function:

[0051]

[0052] where z i represents the contrast feature of the anchor sample, z j(i) Represents the comparative features of other samples of the same category as the anchor sample in the same batch, z a It represents the contrast features of other samples of different categories from the anchor sample in the same batch, and τ is the temperature coefficient.

[0053] Furthermore, the method for training the feature decoupling model is: fixing all network weight bias parameters of the feature extraction backbone network, applying the feature extraction backbone network to the feature decoupling network training step, extracting image features in the ultra-wide-angle fundus image output by the training data enhancement pipeline, and then using the extracted image features to train the feature decoupling model according to the training method of the feature decoupling model to complete the training of the feature decoupling model, and taking the model weight bias parameters of the last iteration as the final weight parameters of the feature decoupling model.

[0054] Furthermore, training a linear classifier includes the following steps:

[0055] 1) Continue to use the training data augmentation pipeline with training feature decoupling;

[0056] 2) Construct a linear classifier;

[0057] 3) Continue to use the optimizer and learning rate scheduling strategy for training feature decoupling;

[0058] 4) The maximum number of iterations for model training is 20. Under the cyclic iteration of the pipeline, the training data set is sequentially input into the feature extraction backbone network and the feature decoupling model encoder for forward propagation to obtain the diabetic retinopathy diagnosis class-related features of the ultra-wide-angle fundus image. Then, the obtained class-related features are sent to the linear classifier and the cross-entropy loss is calculated. Finally, the weight bias parameters of the linear classifier are updated through back propagation and the optimizer. The above training process is repeated until the maximum number of model iterations is reached to complete the training of the linear classifier.

[0059] Furthermore, the cross entropy loss function is:

[0060]

[0061] Where M and N are the number of categories and the batch size, respectively. ic and p ic The distribution is the one-hot encoding of the true category of the sample and the predicted probability of the true category of the sample.

[0062] Furthermore, the evaluation and verification includes the following steps:

[0063] 1) Use visualization methods to verify the decoupling performance of the feature decoupling model;

[0064] 2) Verify the quality of the decoupled features on the test dataset and report the diagnostic accuracy and quadratic weighted Kappa of diabetic retinopathy for three different diabetic retinopathy computer-aided diagnosis systems based on supervised training, self-supervised training, and self-supervised training and feature decoupling.

[0065] In a second aspect, the present application provides a computer program, which can implement the steps of any one of the methods described above.

[0066] In a third aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of any one of the methods described above.

[0067] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods described above.

[0068] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0069] (1) The present application decouples the features of ultra-wide-angle fundus images into features related to diagnosis of diabetic retinopathy and features not related to diagnosis. The trained feature decoupling network has an encoder that can directly extract features related to diagnosis and features not related to diagnosis of ultra-wide-angle fundus images, and a decoder that can directly generate features of ultra-wide-angle fundus images under specific conditions through features related to diagnosis and features not related to diagnosis.

[0070] (2) The diabetic retinopathy feature decoupling method for ultra-wide-angle fundus images in this application can significantly improve the diabetic retinopathy grading performance of the self-supervised training model, surpassing the supervised training model, proving the effectiveness of the decoupled diabetic retinopathy diagnosis-related features. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0072] Figure 1 This is a feature visualization diagram before decoupling in Example 1 of the present invention;

[0073] Figure 2 This is a visualization diagram of class-related features after decoupling in Example 1 of the present invention;

[0074] Figure 3 This is a visualization diagram of the class-irrelevant features after decoupling in Example 1 of the present invention;

[0075] Figure 4 This is a visualization result of generating Severe-class class-related features and Severe-class ultra-wide-angle fundus image features by using Mild-class class-related features in Example 1 of the present invention through a continuous progression model of diabetic retinopathy. The blue circle represents the real Severe-class features, and the blue cross represents the Severe-class features generated by the decoupling feature manipulation technology.

[0076] Figure 5 This is a visualization result of exchanging the class-related features of the Mild class and the Severe class and generating corresponding ultra-wide-angle fundus image features in Example 1 of the present invention. The green and blue circles represent real image features, and the green and blue crosses represent image features generated by the decoupling feature exchange technology.

[0077] Figure 6 This is a schematic diagram of the codec structure of the feature decoupling network in Example 1 of the present invention;

[0078] Figure 7 This is an overall overview diagram of the characteristic decoupling model in Example 1 of the present invention;

[0079] Figure 8 The continuous progression model of diabetic retinopathy in Example 1 of the present invention, wherein the diabetic retinopathy features of the latter level include all the diabetic retinopathy features of the previous level, and the above features are all diabetic retinopathy diagnosis-related features decoupled from the feature decoupling model;

[0080] Fig. 9 This is an overview diagram of applying the feature decoupling model to a general diabetic retinopathy computer-aided diagnosis system in Example 1 of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0082] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] The terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0084] Example 1

[0085] This embodiment provides a method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images, and selects ultra-wide-angle fundus images collected in real medical scenarios as implementation scenarios for decoupling diabetic retinopathy features. All ultra-wide-angle fundus images are annotated with diabetic retinopathy grades by professional ophthalmologists. The specific implementation steps include five main parts: data preprocessing, data enhancement, model construction, model training and prediction, and model application.

[0086] S1. Data preprocessing

[0087] A. Performing center cropping on all ultra-wide-angle fundus images at a ratio of 0.6 in the horizontal direction and 0.8 in the vertical direction to obtain cropped ultra-wide-angle fundus images;

[0088] B. Applying a contrast adaptive histogram equalization algorithm to all cropped ultra-wide-angle fundus images, where the algorithm's key hyperparameters cropping limit and block are 5 and 8 respectively, to obtain an ultra-wide-angle fundus image processed by contrast adaptive histogram equalization;

[0089] C. Perform image scaling on all ultra-wide-angle fundus images processed by contrast adaptive histogram equalization, and use Lanczos interpolation algorithm to scale the resolution of all ultra-wide-angle fundus images to 512×512;

[0090] D. Based on the 10-fold cross-validation algorithm, the training data set and the test data set are randomly divided in a ratio of 9:1. The statistical features of the training data set, the validation data set and the RGB three channels in each fold division data set are calculated, and the fold division result with the closest statistical features is selected as the final training data set and validation data set.

[0091] S2. Data enhancement

[0092] Image rotation: Apply the image rotation data enhancement algorithm, with a rotation angle of -45° to 45° and a rotation probability of 0.5;

[0093] Image random scaling and cropping: The image random scaling and cropping data enhancement algorithm is applied, with a maximum scaling ratio of 1.0, a minimum scaling ratio of 0.08, an image cropping resolution of 448×448, and an image interpolation algorithm of bicubic interpolation;

[0094] Image scaling: The image scaling data enhancement algorithm is applied, the scaling resolution is 448×448, and the image scaling interpolation algorithm is the bicubic interpolation algorithm.

[0095] B. Image pixel transformation enhancement method

[0096] Gaussian noise: The Gaussian noise data enhancement algorithm is applied to the image with a probability of 0.2.

[0097] Color dithering: Apply image color dithering data enhancement algorithm, color dithering probability is 0.8, brightness adjustment range is 0.2 to 1.8, contrast adjustment range is 0.2 to 1.8, saturation adjustment range is 0.2 to 1.8, hue adjustment range is -0.2 to 0.2;

[0098] Grayscale: Apply the image grayscale data enhancement algorithm with a grayscale probability of 0.2;

[0099] Standardization: The image RGB three-channel data standardization data enhancement algorithm is applied. The mean of the three channels of the image is R=0.485, G=0.456, B=0.406, and the variance is R=0.229, G=0.224, B=0.225.

[0100] S3, model construction

[0101] The decoupling network of diabetic retinopathy features of ultra-wide-angle fundus images is mainly composed of an encoder and a decoder. The structure of the encoder and decoder is shown in the figure Figure 6 shown.

[0102] The idea of ​​constructing the feature decoupling model is as follows: Assuming that the features related to and unrelated to the diagnosis of diabetic retinopathy in the image features conform to the Gaussian distribution, the idea of ​​variational inference can be used to directly construct a feature decoupling model based on the variational autoencoder. Considering the complexity of the features related to and unrelated to the diagnosis of diabetic retinopathy, this technology uses multivariate Gaussian distribution to model the features to be decoupled. The model overview is as follows: Figure 7 As shown, a feature decoupling loss function is designed based on this to ensure the independence of the modeled features. At the same time, a diabetic retinopathy continuous development loss function is designed based on this to ensure the directionality of the modeled features. The calculation of the diabetic retinopathy continuous development loss function relies on the diabetic retinopathy continuous development model proposed by this technology, such as Figure 8 As shown, specifically, the continuous disease progression model regards the progression of diabetic retinopathy as a continuous process. The next level of diabetic retinopathy features include all the features of the previous level of diabetic retinopathy. Under the assumption of this technology, the continuous disease progression process is computable. Based on the above two feature decoupling loss functions, this technology only needs to ensure that the other modeling processes of the model meet the initial Gaussian distribution assumption of the model, then the feature decoupling model proposed by this technology can successfully complete the decoupling task of diabetic retinopathy features.

[0103] Among them, the feature decoupling loss function L d for:

[0104]

[0105] Among them, μ c ,μ s , They represent the mean and variance of class-related features and class-irrelevant features respectively, and J represents the dimension of the feature.

[0106] Loss function L for continuous progression of diabetic retinopathy chn for:

[0107]

[0108] Where Y represents the different diabetic retinopathy grades in a batch; It represents the unique characteristics of each diabetic retinopathy grade relative to its previous diabetic retinopathy grade in the continuous progression model of diabetic retinopathy, and this characteristic is still described by Gaussian distribution.

[0109] Reconstruction loss function L R for:

[0110]

[0111] Where N represents the number of image features, V i represents the image features, V i R Represents the image features reconstructed by the feature decoupling model decoder;

[0112] The KL loss function is:

[0113]

[0114] Where J represents the dimension of the feature.

[0115] S4. Model training and prediction

[0116] A. Build a data enhancement pipeline for model training: random scaling and cropping, horizontal image flipping, color jittering and standardization. The pipeline uses a data balanced sampling algorithm with a batch size of 128.

[0117] B. Initialize the feature decoupling model. The output feature dimensions of the 4 linear layers of the encoder are 256, 256, 128, 128, respectively, and the output feature dimensions of the 6 linear layers of the decoder are 32, 64, 64, 128, 256, 256, respectively. The probability of all Dropout layers is 0.3. The feature dimension of class-related features and class-independent features is 32. The hyperparameters of each loss of the total loss function of the feature decoupling model are α=1.0, β=1.0, γ c =0.001,γ s =0.005,

[0118] C. Initialize the optimizer and learning rate scheduling strategy. The initialized optimizer is the SGD optimizer, the learning rate is 0.01, and the weight decay rate is 0.001. The initialized learning rate scheduling strategy is the linear warm-up cosine annealing algorithm. The first 10% of iterations use the linear warm-up learning rate scheduling strategy, and the last 90% of iterations use the cosine annealing learning rate scheduling strategy.

[0119] D. The maximum number of iterations for model training is 100. Under the cyclic iteration of the pipeline, the training data set is input into the feature decoupling model in turn for forward propagation. Then the total loss function of the feature decoupling model is calculated and the error is back-propagated. Finally, the optimizer is used to update the weight bias parameters of the feature decoupling model. The above training process is repeated until the maximum number of model iterations is reached.

[0120] E. Build the model prediction data enhancement pipeline: image scaling and normalization, pipeline turns off the data balanced sampling algorithm, and the batch size is 128.

[0121] F. Fix the parameters of the trained feature decoupling model, send the test data set into the feature decoupling model in sequence to perform feature decoupling, visualize the decoupling data, and evaluate the decoupling performance of the decoupling model.

[0122] The total loss function L of the feature decoupling model is:

[0123]

[0124] Among them, α, β, γ c , γ s , are all hyperparameters.

[0125] S5. Model application

[0126] The feature decoupling model is applied to the computer-aided diagnosis system of diabetic retinopathy based on ultra-wide-angle fundus images (such as Fig. 9 Specifically, a feature extraction backbone network is designed to extract the features of ultra-wide-angle fundus images. The extracted image features are then sent to a feature decoupling network to decouple features related to the diagnosis of diabetic retinopathy and features not related to the diagnosis. Finally, the obtained diagnosis-related features are used to train a simple linear classifier. At the same time, the diabetic retinopathy grading performance of the linear classifier on the test data set is compared before and after decoupling to verify the effectiveness of the decoupled features.

[0127] A. Training feature extraction backbone network:

[0128] 1) Build a data enhancement pipeline for model training: random scaling and cropping, image rotation, image horizontal flipping, Gaussian noise, grayscale, color jitter and standardization, turn off the pipeline data balancing sampling algorithm, and set the batch size to 128;

[0129] 2) Construct a feature extraction backbone network: Select ResNet18 as the feature extraction backbone network, and the backbone network output feature dimension is 512 dimensions; construct a contrastive learning prediction head, the prediction head structure is a simple two-layer linear neural network, the first linear layer increases the dimension of 512-dimensional features to 2048 dimensions, and the second layer reduces the dimension of 2048-dimensional features to 512 dimensions. There is a ReLU activation function between the two linear layers;

[0130] 3) Initialize the optimizer and learning rate scheduling strategy. The initialized optimizer is the SGD optimizer, the learning rate is 0.3, and the weight decay rate is 0.00001. The initialized learning rate scheduling strategy is the linear warm-up cosine annealing algorithm. The first 10% of iterations use the linear warm-up learning rate scheduling strategy, and the last 90% of iterations use the cosine annealing learning rate scheduling strategy;

[0131] 4) The maximum number of iterations of model training is 800. Under the cyclic iteration of the pipeline, the training data set is sequentially input into the feature extraction backbone network and the contrastive learning prediction head for forward propagation, then the contrastive loss is calculated and the error is back-propagated. Finally, the optimizer is used to update the weight bias parameters of the feature extraction backbone network. The above training process is repeated until the maximum number of model iterations is reached;

[0132] 5) Take the model parameters of the last iteration as the final weight bias parameters of the feature extraction backbone network to complete the training of the feature extraction backbone network.

[0133] The contrast loss uses a supervised contrast loss function:

[0134]

[0135] where z i represents the contrast feature of the anchor sample, z j(i) Represents the comparative features of other samples of the same category as the anchor sample in the same batch, z a It represents the contrast features of other samples of different categories from the anchor sample in the same batch, and τ is the temperature coefficient.

[0136] B. Training feature decoupling model

[0137] All network weight bias parameters of the feature extraction backbone network are fixed, and the feature extraction backbone network is applied to the feature decoupling network training step to extract image features in the ultra-wide-angle fundus image output by the training data enhancement pipeline. Then, the feature decoupling model is trained using the extracted image features according to the training method of the feature decoupling model to complete the training of the feature decoupling model, and the model weight bias parameters of the last iteration are taken as the final weight parameters of the feature decoupling model.

[0138] C. Training a linear classifier

[0139] Fixed all network weight bias parameters of the feature extraction backbone network and feature decoupling model encoder.

[0140] 1) Continue to use the training data augmentation pipeline with training feature decoupling;

[0141] 2) Constructing a linear classifier: Using a linear layer as a linear classifier, the sample data is mapped from the feature space to the diabetic retinopathy grading space;

[0142] 3) Continue to use the optimizer and learning rate scheduling strategy for training feature decoupling;

[0143] 4) The maximum number of iterations for model training is 20. Under the cyclic iteration of the pipeline, the training data set is sequentially input into the feature extraction backbone network and the feature decoupling model encoder for forward propagation to obtain the diabetic retinopathy diagnosis class-related features of the ultra-wide-angle fundus image. Then, the obtained class-related features are sent to the linear classifier and the cross-entropy loss is calculated. Finally, the weight bias parameters of the linear classifier are updated through back propagation and the optimizer. The above training process is repeated until the maximum number of model iterations is reached to complete the training of the linear classifier.

[0144] The cross entropy loss function is:

[0145]

[0146] Where M and N are the number of categories and the batch size, respectively. ic and p ic The distribution is the one-hot encoding of the true category of the sample and the predicted probability of the true category of the sample.

[0147] D. Evaluation and Verification

[0148] 1) Use visualization technology to verify the decoupling performance of the feature decoupling model;

[0149] 2) Verify the quality of the decoupled features on the test dataset and report the diagnostic accuracy and quadratic weighted Kappa of diabetic retinopathy for three different diabetic retinopathy computer-aided diagnosis systems based on supervised training, self-supervised training, and self-supervised training and feature decoupling.

[0150] According to international diagnostic and treatment standards, diabetic retinopathy can be divided into five levels: normal, mild, moderate, severe and proliferative. The technical effects of this embodiment are demonstrated and described from the following two aspects:

[0151] 1. Decoupling performance

[0152] In the field of feature decoupling, dimension reduction visualization technology is usually used to measure the decoupling performance of feature decoupling networks, including feature visualization before and after decoupling, feature operation generation visualization, and feature exchange generation visualization. The above technologies can reflect whether the decoupled features are independent and have practical significance. This technology decouples the features of ultra-wide-angle fundus images into diagnosis-related features and diagnosis-irrelevant features of diabetic retinopathy. The encoder of the trained feature decoupling network can directly extract the diagnosis-related features and diagnosis-irrelevant features of the ultra-wide-angle fundus image features, and the decoder can directly generate the ultra-wide-angle fundus image features under specific conditions through the diagnosis-related features and diagnosis-irrelevant features.

[0153] ① Feature visualization before and after decoupling: The ultra-wide-angle fundus image features in the dataset are directly visualized using the T-SNE algorithm. The results are shown in Figures 1 to 3 .

[0154] ② Feature manipulation to generate visualization: Obtain the diagnosis-related features and diagnosis-irrelevant features of any type of ultra-wide-angle fundus image of diabetic retinopathy, generate other types of diagnosis-related features from the diagnosis-related features according to the continuous development model of diabetic retinopathy, and finally generate new diabetic retinopathy image features through the feature decoupling network decoder and visualize the results through the T-SNE algorithm. The results are shown in the figure. Figure 4 shown.

[0155] ③ Feature exchange generation visualization: Obtain two types of ultra-wide-angle fundus image features of diabetic retinopathy with different grades, send the two types of features to the feature decoupling network encoder, obtain two different groups of diagnosis-related features and diagnosis-irrelevant features, exchange the two groups of diagnosis-related features and generate the exchanged ultra-wide-angle fundus image features, and directly visualize the generated features through the T-SNE algorithm. The results are shown in the figure. Figure 5 shown.

[0156] The above three visualization results can effectively demonstrate the decoupling performance of the feature decoupling network proposed in this application.

[0157] 2. Downstream Computer Vision Task Performance

[0158] The technical solution of this application selects the diabetic retinopathy grading task based on ultra-wide-angle fundus images to evaluate the impact of decoupled features on the performance of downstream computer vision tasks. Computer-aided diabetic retinopathy grading systems widely use the accuracy (Accuracy) and quadratic weighted kappa (quadraticweighted kappa, referred to as QWK) of the system on the test data set to measure the disease diagnosis performance of the system. Assume that the confusion matrix (Confusion Matrix) composed of the model prediction results and the true category information of the samples on the validation data set is shown in Table 1 below.

[0159] Table 1

[0160]

[0161] ①Accuracy (ACC)

[0162]

[0163] Accuracy is one of the indicators for evaluating classifier performance. It represents the ratio of the number of samples correctly predicted by the classifier to the total number of samples, that is, the proportion of correctly classified samples to the total number of samples. The higher the value, the better.

[0164] ②Quadratic Weighted Kappa (QWK)

[0165]

[0166] Where W i,j is the quadratic weight matrix:

[0167]

[0168] Where E i,j The expected matrix is:

[0169]

[0170] Quadratic weighted Kappa can effectively avoid the impact of medical data imbalance on classifier performance evaluation. Its value range is between -1 and 1, and the larger the better.

[0171] ACC and QWK are calculated using the system diagnosis grading results and doctor diagnosis grading results of the samples in the test dataset. The specific results are shown in Table 2 below.

[0172] Table 2 Objective performance evaluation results of the accuracy (ACC) and quadratic weighted Kappa (QWK) of each comparison method on the test data set

[0173]

[0174] As shown in Table 1, the diabetic retinopathy analysis system based on feature decoupling and network construction achieved the highest accuracy and quadratic weighted Kappa, reaching 0.841 and 0.891 respectively. It also shows that the proposed feature decoupling technology can significantly improve the diabetic retinopathy grading performance of the self-supervised training model, surpassing the supervised training model, proving the effectiveness of the decoupled diabetic retinopathy diagnosis-related features.

[0175] Example 2

[0176] This embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images in the above-mentioned embodiment 1.

[0177] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0178] The storage includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, for example, a hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory can also include both an internal storage unit of the computer device and an external storage device. In this embodiment, the memory is often used to store an operating system and various application software installed on the computer device, such as running the program code of the diabetic retinopathy feature decoupling method for ultra-wide-angle fundus images, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0179] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the diabetic retinopathy feature decoupling method for ultra-wide-angle fundus images.

[0180] Example 3

[0181] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images in the above-mentioned embodiment 1.

[0182] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor executes the steps of the diabetic retinopathy feature decoupling method for ultra-wide-angle fundus images.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only the specific embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention. For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present application is limited by the attached claims rather than the above description, and it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the present application.

Claims

1. A method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images, characterized in that: It includes data preprocessing, data enhancement, model building, model training and prediction, and model application in sequence.

2. The method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images according to claim 1, characterized in that: The data preprocessing comprises the following steps: A. Perform center cropping on all ultra-wide-angle fundus images to obtain cropped ultra-wide-angle fundus images; B. applying a contrast adaptive histogram equalization algorithm to all the cropped ultra-wide-angle fundus images to obtain an ultra-wide-angle fundus image processed by the contrast adaptive histogram equalization; C. performing image scaling on all ultra-wide-angle fundus images processed by contrast adaptive histogram equalization; D. Randomly divide the training data set and the test data set, calculate the statistical features of the training data set, the validation data set and the RGB three-way in each fold division data set, and select the fold division result with the closest statistical features as the final training data set and validation data set.

3. The method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images according to claim 1, characterized in that: The model construction adopts multivariate Gaussian distribution to model the decoupled features, and based on this, a feature decoupling loss function is designed to ensure the independence of the modeled features. At the same time, a diabetic retinopathy continuous development loss function is also designed based on this to ensure the directionality of the modeled features; Among them, the continuous development loss function of diabetic retinopathy is calculated by a continuous development model of diabetic retinopathy; the continuous development model of diabetic retinopathy regards the development of diabetic retinopathy as a continuous process, and the next level of diabetic retinopathy characteristics include all the characteristics of the previous level of diabetic retinopathy.

4. The method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images according to claim 3, characterized in that: The feature decoupling loss function L d for: in, They represent the mean and variance of class-related features and class-irrelevant features respectively, and J represents the dimension of the feature.

5. The method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images according to claim 3, characterized in that: The diabetic retinopathy continuous progression loss function L chn for: Where Y represents the different diabetic retinopathy grades in a batch; It represents the unique characteristics of each diabetic retinopathy grade relative to its previous diabetic retinopathy grade in the continuous progression model of diabetic retinopathy, and this characteristic is still described by Gaussian distribution.

6. The method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images according to claim 1, characterized in that: The model training and prediction includes the following steps: A. Build a model training data enhancement pipeline; B. Initialize the feature decoupling model; C. Initialize the optimizer and learning rate scheduling strategy; D. The maximum number of iterations of model training is 100. Under the cyclic iteration of the pipeline, the training data set is input into the feature decoupling model in turn for forward propagation. Then the total loss function of the feature decoupling model is calculated and the error is back-propagated. Finally, the optimizer is used to update the weight bias parameters of the feature decoupling model. The above training process is repeated until the maximum number of model iterations is reached. E. Build a model prediction data enhancement pipeline; F. Fix the parameters of the trained feature decoupling model, send the test data set into the feature decoupling model in sequence to perform feature decoupling, visualize the decoupling data, and evaluate the decoupling performance of the decoupling model.

7. The method for decoupling diabetic retinopathy features for ultra-wide-angle fundus images according to claim 6, characterized in that: The total loss function L of the feature decoupling model is: Among them, α, β, γ c , γ s , are all hyperparameters.

8. A computer program, characterized in that The computer program can implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images