A Deep Learning-Based Method and System for Quality Assessment of Ultra-Wide Field-of-View Fundus Images

Abstract

This invention proposes a method and system for quality assessment of ultra-wide field-of-view fundus images based on deep learning, belonging to the field of image processing. The method includes: S1: Inputting the ultra-wide field-of-view fundus image dataset into an image preprocessing module to obtain image data; S2: Training the input prediction module, which consists of three residual networks, each containing four sub-models generated during 4-fold cross-validation; S3: Optimizing the prediction module during training using a binary cross-entropy loss function incorporating label smoothing technology; S4: During inference, enhancing the ultra-wide field-of-view fundus image to be predicted, inputting the enhanced image and the ultra-wide field-of-view fundus image to be predicted into the trained prediction module, and aggregating all prediction results as the final prediction result. This invention fully utilizes ultra-wide field-of-view fundus image data, improves model generalization ability, and accurately assesses image quality.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to a method and system for quality assessment of ultra-wide field-of-view fundus images based on deep learning. Background Technology

[0002] Diabetic retinopathy (DR) is one of the most common complications of diabetes and a leading cause of blindness in adults. With increasing life expectancy, unhealthy lifestyles, and other factors, the number of people with diabetes is expected to continue to rise. Early detection, timely referral, and early treatment are widely considered key measures to prevent vision loss. With advancements in computer vision technology, the application of deep learning in the assisted diagnosis of diabetic retinopathy is becoming increasingly mature. Traditional color fundus photography (CFP) can only capture the optic nerve within a 30-50 degree field of view, while ultra-wide field (UWF) fundus images can provide up to 200 degrees of retinal field of view. This wider field of view makes it easier for deep learning models to capture retinal lesions, thereby improving the accuracy of lesion identification. However, ultra-wide field fundus images may have various quality problems, such as visual field obstruction, blurring, and insufficient contrast, which cannot meet the high-quality data requirements for model training. Assessing the quality of ultra-wide field-of-view fundus images requires significant time and human resources. Deep learning, with its automated and precise feature extraction capabilities and ability to handle diverse quality issues, has become an effective tool for evaluating fundus image quality.

[0003] While deep learning methods can effectively evaluate the quality of ultrawide fundus image data, challenges remain. Training deep learning models requires massive amounts of data. The quality annotation process for ultrawide fundus images requires professional ophthalmologists, which is time-consuming and costly, and the dataset is difficult to scale rapidly. Due to the unique nature of fundus images, some data augmentation methods (such as rotation, scaling, and cropping) cannot generate accurate and effective ultrawide fundus images, making it difficult to increase the sample size through data augmentation. How to enable the model to effectively learn image quality assessment features when data is insufficient is a major challenge for deep learning in ultrawide fundus image data quality assessment. Furthermore, ultrawide fundus images may contain numerous artifacts (such as eyelashes, reflected light, etc.) and noise, which can hinder image quality assessment. These real-world problems are often difficult to simulate, requiring models with strong generalization capabilities. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for quality assessment of ultra-wide field-of-view fundus images based on deep learning, comprising the following steps:

[0005] Step S1: During model training, the ultra-wide field-of-view fundus image dataset is input into the image preprocessing module. Data diversity is increased by horizontally flipping the data, and the data is normalized to improve model stability, resulting in processed image data. ;

[0006] Step S2: Transfer image data The prediction module is trained by inputting the prediction module, which consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process.

[0007] Step S3: During training, the prediction module is optimized using a binary cross-entropy loss function that incorporates label smoothing techniques;

[0008] Step S4: During the inference process, the ultra-wide field-of-view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain test image data. Test image data The ultra-wide field-of-view fundus image to be predicted is input into the trained prediction module, and all prediction results are aggregated as the final prediction result.

[0009] Beneficial effects:

[0010] 1. The method disclosed in this invention uses data augmentation and 4-fold cross-validation techniques during model training, which can maximize the utilization of data when the amount of ultra-wide field-of-view fundus image data is limited, help the model adapt to changes in data distribution, and improve the model's predictive performance.

[0011] 2. The method disclosed in this invention integrates ResNet34d, ResNet50d, and ResNext50d models. Different models can learn different features or patterns from the training data. When the results of multiple models are averaged, their learning outcomes are combined, which enhances the model's ability to generalize to new data.

[0012] 3. The method disclosed in this invention employs label smoothing technology during training to convert hard labels into soft labels, thereby preventing the model from becoming overconfident in its predictions during training. Label smoothing makes the clustering between categories more compact, increases inter-class distance, reduces intra-class distance, and improves the consistency between the model's prediction confidence and actual accuracy, thus enhancing the model's generalization ability.

[0013] 4. The method disclosed in this invention uses test-time enhancement and image size enlargement on ultra-wide field-of-view fundus images during testing. Test-time enhancement can reduce the random error of individual predictions, thereby improving the overall prediction accuracy of the model. Enlarging the image size allows the model to capture more details and compensates for changes in statistical properties caused by scale variations during training, thereby improving prediction performance. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a method for quality assessment of ultra-wide field-of-view fundus images based on deep learning according to the present invention.

[0015] Figure 2 This is a schematic diagram of the image preprocessing module.

[0016] Figure 3 This is a schematic diagram of the prediction process in the prediction module;

[0017] Figure 4 This is a structural block diagram of a deep learning-based ultra-wide field-of-view fundus image quality assessment system according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] Example 1

[0020] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for quality assessment of ultra-wide field-of-view fundus images based on deep learning, comprising the following steps:

[0021] Step S1: During model training, the ultra-wide field-of-view fundus image dataset is input into the image preprocessing module. Data diversity is increased by horizontally flipping the data, and the data is normalized to improve model stability, resulting in processed image data. ;

[0022] Step S2: Transfer image data The prediction module is trained by inputting data. The prediction module consists of three residual networks: ResNet34d, ResNet50d, and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process.

[0023] Step S3: Optimize the prediction module during training using a binary cross-entropy loss function that incorporates label smoothing techniques;

[0024] Step S4: During the inference process, the ultra-wide field-of-view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain test image data. Test image data The ultra-wide field-of-view fundus image to be predicted is input into the trained prediction module, and all prediction results are aggregated as the final prediction result.

[0025] In one embodiment, during model training, step S1 involves inputting an ultra-wide field-of-view fundus image dataset into an image preprocessing module. This module enhances data diversity through horizontal flipping and normalizes the data to improve model stability, resulting in processed image data. Specifically, it includes:

[0026] Step S11: Input the ultra-wide field-of-view fundus image into the image size adjustment module, and use Resize to adjust it to the predetermined size to obtain image data. ;

[0027] Image data of embodiments of the present invention This ensures that the input image is processed at a fixed size;

[0028] Step S12: Use the horizontal flip module to randomly flip the image data. Perform a horizontal flipping process to obtain flipped and enhanced data. ;

[0029] This invention uses the HorizontalFlip module of the alumments library to achieve horizontal flipping, setting the flipping probability to 50%. By randomly changing the orientation of the image, the diversity of the data is enhanced, thereby increasing the generalization ability of the model.

[0030] Step S13: During training, use the hybrid enhancement module to enhance the image data. Linearly blending with another random image yields blended augmented data. ;

[0031] This invention uses the MixUp module of the alumments library to combine image data. Linear blending with another random image, with the blending ratio controlled by parameter alpha=0.5, simulates complex data distributions by blending different images, further improving the robustness of the model;

[0032] Step S14: Use the normalization module to normalize the hybrid augmented data Normalization is performed to adjust the pixel values ​​to a specific distribution, and each channel is standardized to obtain standardized image data. .

[0033] In this embodiment of the invention, the Normalize module of the alumensations library is used to standardize each channel using the mean = [0.485, 0.456, 0.406] and the standard deviation std = [0.229, 0.224, 0.225] to ensure that the image features conform to the expected distribution range, thereby improving the training effect of the model.

[0034] Figure 2 The diagram shows the structure of the image preprocessing module, which is a combined image enhancement pipeline that effectively improves the diversity of data and the generalization ability of the model through image resizing, horizontal flipping, blending enhancement, and normalization operations.

[0035] In one embodiment, step S2 above: transferring image data The prediction module is trained by inputting data. The prediction module consists of three residual networks: ResNet34d, ResNet50d, and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process, specifically including:

[0036] Step S21: During the model training phase, the preprocessed ultrawide field-of-view fundus image dataset is divided into four non-overlapping subsets: Fold 1, Fold 2, Fold 3, and Fold 4, where the number of samples in each subset is approximately equal and the class distribution is similar.

[0037] This step ensures that every part of the dataset can be used for training and validation by dividing it into four subsets, thus increasing the sufficiency of data utilization;

[0038] Step S22: Select one of the four subsets as the validation set and the remaining three subsets as the training set to form a four-fold network. Train the residual networks ResNet34d, ResNet50d and ResNext50d on the four-fold network respectively. Each residual network yields four trained sub-models.

[0039] This step ensures that the model fully learns the data features while providing robust evaluation results, effectively preventing overfitting and improving the model's generalization ability.

[0040] like Figure 3 The diagram shown illustrates the prediction process of the prediction module.

[0041] In one embodiment, step S3 above: optimizing the prediction module during training using the binary cross-entropy function (BCE) that incorporates label smoothing, specifically includes:

[0042] Step S31: Transfer the original label Adjusted to a one-hot code format, for the binary classification task of ultra-wide field-of-view fundus image quality assessment, label 0 is rewritten as [1, 0], and label 1 is rewritten as [0, 1].

[0043] Step S32: Smooth the labels: First, set the label smoothing coefficient. , Used to adjust the smoothness of positive and negative class labels, rewriting the label [1, 0] as [ , Rewrite the labels [0, 1] as [ , ];

[0044] This invention sets By setting a label smoothing coefficient, the absolute certainty of the model regarding the labels can be reduced, thus avoiding overfitting.

[0045] Step S33: Minimize the binary cross-entropy loss function that introduces smooth labels. To optimize the parameters, overfitting to the true labels of the samples during training is avoided, thereby improving the robustness and generalization ability to noisy data. Formulas (1) and (2) are shown below:

[0046] (1)

[0047] (2)

[0048] Where N is the number of samples, It is The original tags of the image. It is The prediction results for the image. It is the label smoothing coefficient.

[0049] In one embodiment, step S4 above involves inputting the ultra-wide field-of-view fundus image to be predicted into the inference image enhancement module for enhancement during the inference process to obtain test image data. Test image data The trained prediction module takes the ultra-wide field-of-view fundus image to be predicted as input, and aggregates all prediction results as the final prediction result, which specifically includes:

[0050] Step S41: Enlarge the ultra-wide field-of-view fundus image to be predicted, and adjust the width and height to a predetermined size;

[0051] Due to preprocessing and image augmentation, the model perceives larger image sizes during training than during testing, with the width and height of the images set to a uniform size of 576 (1.25 times the training image size of 448).

[0052] Step S42: The resized image is horizontally flipped using Test-Time Augmentation (TTA). The ultra-wide field-of-view fundus image to be predicted and the flipped image are input into each trained sub-model for independent prediction. The average of the prediction results of the four sub-models corresponding to each residual network is taken as the prediction result of that residual network. The final prediction result is... It is the average of the prediction results of three residual networks, ResNet34d, ResNet50d, and ResNext50d, where It is the probability of determining image quality;

[0053] Step S43: Evaluate image quality based on a pre-set threshold: Images exceeding the threshold are considered high-quality images. Images with a value less than or equal to the threshold are considered low-quality images.

[0054] Example 2

[0055] like Figure 4 As shown, this embodiment of the invention provides a quality assessment system for ultra-wide field-of-view fundus images based on deep learning, comprising the following modules:

[0056] Image preprocessing module 51 is used during model training. It inputs an ultra-wide field-of-view fundus image dataset, increases data diversity through horizontal flipping for data augmentation, and improves model stability through data normalization to obtain processed image data. ;

[0057] Prediction module 52 is used to convert image data The prediction module is trained by inputting data. The prediction module consists of three residual networks: ResNet34d, ResNet50d, and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process.

[0058] Optimization module 53 is used to optimize the prediction module during training using a binary cross-entropy loss function that incorporates label smoothing techniques;

[0059] The inference image enhancement module 54 is used to input the ultra-wide field-of-view fundus image to be predicted into the inference image enhancement module for enhancement during the inference process, so as to obtain test image data. Test image data The ultra-wide field-of-view fundus image to be predicted is input into the trained prediction module, and all prediction results are aggregated as the final prediction result.

Claims

1. A method for quality assessment of ultra-wide field-of-view fundus images based on deep learning, characterized in that, include: Step S1: During model training, the ultra-wide field-of-view fundus image dataset is input into the image preprocessing module. Data diversity is increased by horizontally flipping the data, and the data is normalized to improve model stability, resulting in processed image data. ; Step S2: Transfer image data The prediction module is trained by inputting the prediction module, which consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process. Step S3: During training, the prediction module is optimized using a binary cross-entropy loss function that incorporates label smoothing techniques. Specifically, this includes: Step S31: Transfer the original label Adjusted to a one-hot code format, for the binary classification task of ultra-wide field-of-view fundus image quality assessment, label 0 is rewritten as [1, 0], and label 1 is rewritten as [0, 1]. Step S32: Smooth the labels: First, set the label smoothing coefficient. , Used to adjust the smoothness of positive and negative class labels, rewriting the label [1, 0] as [ , Rewrite the labels [0, 1] as [ , ]; Step S33: Minimize the binary cross-entropy loss function that introduces smooth labels. To optimize the parameters, overfitting to the true labels of the samples during training is avoided, thereby improving the robustness and generalization ability to noisy data. Formulas (1) and (2) are shown below: (1) (2) Where N is the number of samples, It is The original tags of the image. It is The prediction results for the image. It is the label smoothing coefficient; Step S4: During the inference process, the ultra-wide field-of-view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain test image data. Test image data The ultra-wide field-of-view fundus image to be predicted is input into the trained prediction module, and all prediction results are aggregated as the final prediction result, specifically including: Step S41: Enlarge the ultra-wide field-of-view fundus image to be predicted, and adjust the width and height to a predetermined size; Step S42: Horizontally flip the resized image. Input the ultra-wide field-of-view fundus image to be predicted and the flipped image into each of the trained sub-models for independent prediction. Take the average of the prediction results of the four sub-models corresponding to each residual network as the prediction result of that residual network. The final prediction result is... It is the average of the prediction results of three residual networks, ResNet34d, ResNet50d, and ResNext50d, where It is the probability of determining image quality; Step S43: Evaluate image quality based on a pre-set threshold: Images exceeding the threshold are considered high-quality images. Images with a value less than or equal to the threshold are considered low-quality images.

2. The method for quality assessment of ultra-wide field-of-view fundus images based on deep learning according to claim 1, characterized in that, In step S1: During model training, the ultra-wide field-of-view fundus image dataset is input into the image preprocessing module. Data diversity is increased by horizontal flipping data augmentation, and data normalization is used to improve model stability, resulting in processed image data. , specifically including: Step S11: Input the ultra-wide field-of-view fundus image into the image size adjustment module, and use Resize to adjust it to the predetermined size to obtain image data. ; Step S12: Use the horizontal flip module to randomly flip the image data. Perform a horizontal flipping process to obtain flipped and enhanced data. ; Step S13: During training, the image data is processed using a hybrid enhancement module. Linearly blending with another random image yields blended augmented data. ; Step S14: Use the normalization module to process the hybrid augmented data. Normalization is performed to adjust the pixel values ​​to a specific distribution, and each channel is standardized to obtain standardized image data. .

3. The method for quality assessment of ultra-wide field-of-view fundus images based on deep learning according to claim 2, characterized in that, Step S2: Transfer image data The prediction module is trained by inputting data. This prediction module consists of three residual networks: ResNet34d, ResNet50d, and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process, specifically including: Step S21: During the model training phase, the preprocessed ultrawide field-of-view fundus image dataset is divided into four non-overlapping subsets: Fold 1, Fold 2, Fold 3, and Fold 4, where the number of samples in each subset is approximately equal and the class distribution is similar. Step S22: Select one of the four subsets as the validation set and the remaining three subsets as the training set to form a four-fold network. Train the residual networks ResNet34d, ResNet50d and ResNext50d on the four-fold network respectively. Each residual network yields four trained sub-models.

4. A quality assessment system for ultra-wide field-of-view fundus images based on deep learning, characterized in that, Includes the following modules: The image preprocessing module is used during model training. It takes an ultra-wide field-of-view fundus image dataset as input, increases data diversity through horizontal flipping for data augmentation, and improves model stability through data normalization, resulting in processed image data. ; The prediction module is used to process image data. The prediction module is trained by inputting the prediction module, which consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated during the 4-fold cross-validation process. An optimization module is used to optimize the prediction module during training using a binary cross-entropy loss function that incorporates label smoothing techniques. Specifically, it includes: Step S31: Transfer the original label Adjusted to a one-hot code format, for the binary classification task of ultra-wide field-of-view fundus image quality assessment, label 0 is rewritten as [1, 0], and label 1 is rewritten as [0, 1]. Step S32: Smooth the labels: First, set the label smoothing coefficient. , Used to adjust the smoothness of positive and negative class labels, rewriting the label [1, 0] as [ , Rewrite the labels [0, 1] as [ , ]; Step S33: Minimize the binary cross-entropy loss function that introduces smooth labels. To optimize the parameters, overfitting to the true labels of the samples during training is avoided, thereby improving the robustness and generalization ability to noisy data. Formulas (1) and (2) are shown below: (1) (2) Where N is the number of samples, It is the first The original tags of the image. It is the first The prediction results for the image. It is the label smoothing coefficient; The inference image enhancement module is used to enhance the ultra-wide field-of-view fundus image to be predicted during the inference process, thereby obtaining test image data. Test image data The ultra-wide field-of-view fundus image to be predicted is input into the trained prediction module, and all prediction results are aggregated as the final prediction result, specifically including: Step S41: Enlarge the ultra-wide field-of-view fundus image to be predicted, and adjust the width and height to a predetermined size; Step S42: Horizontally flip the resized image. Input the ultra-wide field-of-view fundus image to be predicted and the flipped image into each of the trained sub-models for independent prediction. Take the average of the prediction results of the four sub-models corresponding to each residual network as the prediction result of that residual network. The final prediction result is... It is the average of the prediction results of three residual networks, ResNet34d, ResNet50d, and ResNext50d, where It is the probability of determining image quality; Step S43: Evaluate image quality based on a pre-set threshold: Images exceeding the threshold are considered high-quality images. Images with a value less than or equal to the threshold are considered low-quality images.

Images