Deep learning-based quality evaluation method and system for ultra-wide-view eye fundus image

Abstract

The invention provides a quality evaluation method and system for an ultra-wide-view eye fundus image based on deep learning, and belongs to the field of image processing, and the method comprises the steps: S1, inputting an ultra-wide-view eye fundus image data set into an image preprocessing module, and obtaining image data # imgabs0 #; s2, the # imgabs1 # is input into a prediction module for training, the prediction module is composed of three residual networks, and each residual network comprises four sub-models generated in the four-fold cross validation process; s3, optimizing a prediction module by using a binary cross entropy loss function introduced with a label smoothing technology in a training process; and S4, in the reasoning process, enhancing a to-be-predicted ultra-wide-view eye fundus image, inputting the enhanced image and the to-be-predicted ultra-wide-view eye fundus image into the trained prediction module, and aggregating all prediction results as a final prediction result. According to the method, ultra-wide-view eye fundus image data are fully utilized, the model generalization ability is improved, and the image quality is accurately evaluated.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a quality assessment method and system for ultra-wide field of view fundus images based on deep learning. Background Art

[0002] Diabetic retinopathy (DR) is one of the most common complications of diabetes and one of the leading causes of blindness in adults. With the extension of human life expectancy, irregular lifestyles and the influence of various other factors, the number of diabetic patients is expected to continue to rise. Early detection, timely referral and early treatment are widely considered to be key measures to prevent vision loss. With the advancement of computer vision technology, the application of deep learning in the auxiliary diagnosis of diabetic retinopathy has become increasingly mature. Traditional color fundus photography (CFP) can only capture the optic nerve within a 30 to 50 degree field of view, while ultra-wide field (UWF) fundus images can provide up to 200 degrees of retinal field of view. This wider field of view makes it easier for deep learning models to capture retinal lesions, thereby improving the accuracy of lesion recognition. However, ultra-wide field of view fundus images may have a variety of quality problems, such as field of view occlusion, blur, insufficient contrast, etc., which cannot meet the high-quality data requirements for model training. Evaluating the quality of ultra-wide field-of-view fundus images requires a lot of time and labor costs. Deep learning has become an effective tool for evaluating the quality of fundus images due to its automation, precise feature extraction capabilities, and ability to handle a variety of quality issues.

[0003] Although deep learning methods can effectively evaluate the quality of ultra-wide fundus image data, there are still challenges that need to be addressed. The training of deep learning models requires a large amount of data. The quality annotation process of ultra-wide fundus images requires professional ophthalmologists. Data annotation takes a long time and is costly, and the data set is difficult to expand quickly. Due to the special properties of fundus images, some data augmentation methods (such as rotation, scaling, cropping, etc.) cannot generate correct and effective ultra-wide field of view fundus images, which makes it difficult for us to increase the sample size through data augmentation. How to enable the model to effectively learn the features of image quality assessment when the amount of data is insufficient is a major challenge for deep learning in the task of ultra-wide fundus image data quality assessment. In addition, ultra-wide field of view fundus images may contain a large number of artifacts (such as eyelashes, reflected light, etc.) and noise. These interference information will cause trouble for the assessment of image quality. Such practical problems are often difficult to simulate, and the model needs to have strong generalization capabilities. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for quality assessment of ultra-wide field of view fundus images based on deep learning, comprising the following steps: Step S1: During model training, the ultra-wide field of view fundus image dataset is input into the image preprocessing module, and the data diversity is increased by horizontal flipping data enhancement. After the data is normalized to improve the model stability, the processed image data is obtained. ; Step S2: Image data Input prediction module for training, the prediction module consists of three residual networks: ResNet34d, ResNet50d and ResNext50d, each residual network contains four sub-models generated in the 4-fold cross-validation process; Step S3: During the training process, the prediction module is optimized using a binary cross entropy loss function that introduces label smoothing technology; Step S4: During the inference process, the ultra-wide field of view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain the test image data , the test image data The ultra-wide field of view fundus image to be predicted is input into a trained prediction module, and all prediction results are aggregated as the final prediction result.

[0005] Beneficial effects: 1. The method disclosed in the present invention uses a data enhancement method and a 4-fold cross-validation technique during model training, which can maximize the use of data when the amount of ultra-wide field of view fundus image data is limited, help the model adapt to changes between data distributions, and improve the prediction performance of the model.

[0006] 2. The method disclosed in the present invention integrates ResNet34d, ResNet50d and ResNext50d models. Different models can learn different features or patterns from training data. When the results of multiple models are averaged, their learning results are combined, which can enhance the model's generalization ability to new data.

[0007] 3. The method disclosed in the present invention uses label smoothing technology during the training process to convert hard labels into soft labels, thereby preventing the model from being overconfident in the prediction results during the training process. Label smoothing makes the clustering between categories tighter, increases the distance between categories, reduces the distance within categories, and improves the consistency between the model prediction confidence and the actual accuracy, thereby enhancing the generalization ability of the model.

[0008] 4. The method disclosed in the present invention uses test enhancement and image size enlargement on the ultra-wide field of view fundus images during testing. Test enhancement can reduce the random error of a single prediction, thereby improving the overall prediction accuracy of the model. Increasing the image size enables the model to capture more details and compensate for changes in statistical properties caused by scale changes during training, thereby improving prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic flow chart of a method for quality assessment of ultra-wide field of view fundus images based on deep learning according to the present invention; Figure 2 It is a structural diagram of the image preprocessing module; Figure 3 It is a schematic diagram of the prediction process of the prediction module; Figure 4 This is a structural block diagram of a quality assessment system for ultra-wide field of view fundus images based on deep learning of the present invention. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0011] Embodiment 1 like Figure 1 As shown, an embodiment of the present invention provides a method for quality assessment of ultra-wide field of view fundus images based on deep learning, comprising the following steps: Step S1: During model training, the ultra-wide field of view fundus image dataset is input into the image preprocessing module, and the data diversity is increased by horizontal flipping data enhancement. After the data is normalized to improve the model stability, the processed image data is obtained. ; Step S2: Image data Input prediction module for training. The prediction module consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated in the 4-fold cross-validation process. Step S3: During the training process, the prediction module is optimized using a binary cross entropy loss function that introduces label smoothing technology; Step S4: During the inference process, the ultra-wide field of view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain the test image data , the test image data The trained prediction module is input with the ultra-wide field of view fundus image to be predicted, and all prediction results are aggregated as the final prediction result.

[0012] In one embodiment, in the above step S1: during model training, the ultra-wide field of view fundus image data set is input into the image preprocessing module, and the data diversity is increased by horizontal flipping data enhancement, and the data is normalized to improve the model stability to obtain the processed image data. , specifically including: Step S11: Input the ultra-wide field of view fundus image into the image size adjustment module, and use Resize to adjust it to a predetermined size to obtain image data ; Image data of the embodiment of the present invention , to ensure that the input image is of a fixed size for subsequent processing; Step S12: Use the horizontal flip module to randomly flip the image data Perform horizontal flip processing to obtain flip enhanced data ; The embodiment of the present invention uses the HorizontalFlip module of the albumentations library to implement horizontal flipping, sets the flipping probability to 50%, and enhances the diversity of data by randomly changing the direction of the image, thereby increasing the generalization ability of the model; Step S13: During training, use the hybrid enhancement module to transform the image data Linearly mix with another random image to get mixed enhanced data ; The embodiment of the present invention uses the MixUp module of the albumentations library to convert image data Linearly mix with another random image, and the mixing ratio is controlled by the parameter alpha=0.5. By mixing different images, complex data distribution is simulated, further improving the robustness of the model; Step S14: Use the normalization module to enhance the mixed data Perform normalization to adjust the pixel values ​​to a specific distribution and standardize each channel to obtain standardized image data. .

[0013] The embodiment of the present invention adopts the Normalize module of the albumentations library and uses mean mean=[0.485, 0.456, 0.406] and standard deviation std=[0.229, 0.224, 0.225] to standardize each channel to ensure that the image features conform to the expected distribution range, so as to improve the training effect of the model.

[0014] Figure 2 The structural diagram of the image preprocessing module is shown. This module is a combined image enhancement pipeline that effectively improves the data diversity and the generalization ability of the model through image resizing, horizontal flipping, hybrid enhancement and normalization operations.

[0015] In one embodiment, the above step S2: converting the image data Input prediction module for training. The prediction module consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated in the 4-fold cross-validation process, including: Step S21: In the training phase of the model, the preprocessed ultra-wide field of view fundus image dataset is divided into four non-overlapping subsets: Fold 1, Fold 2, Fold 3, and Fold 4, wherein the number of samples in each subset is roughly equal and the category distribution is similar; This step ensures that every part of the dataset can be used for training and validation by dividing it into four subsets, increasing the adequacy of data utilization; Step S22: Select one of the four subsets as the validation set, and use the remaining three subsets as the training set to form four folds. Residual networks ResNet34d, ResNet50d, and ResNext50d are trained on the four folds respectively, and each residual network obtains 4 trained sub-models. This step can provide robust evaluation results while ensuring that the model fully learns the data features, effectively preventing overfitting and improving the generalization ability of the model.

[0016] like Figure 3 As shown, a schematic diagram of the prediction process of the prediction module is shown.

[0017] In one embodiment, the above step S3: using a binary cross-entropy loss function (BCE) that introduces label smoothing technology (LabelSmoothing) to optimize the prediction module during the training process, specifically includes: Step S31: The original label Adjust to the form of one-hot encoding. For the binary classification task of ultra-wide field fundus image quality assessment, rewrite label 0 as [1, 0] and rewrite label 1 as [0, 1]; Step S32: Smoothing the label: First, set the label smoothing coefficient , Used to adjust the smoothness of positive and negative labels, rewriting the label [1, 0] to [ , ], rewrite the label [0, 1] to [ , ]; The present invention sets ,By setting the label smoothing coefficient, the absolute certainty of the model on the label can be reduced to avoid overfitting; Step S33: Minimize the binary cross entropy loss function with smooth labels To optimize the parameters, we avoid overfitting the real labels of the samples during training, thereby improving the robustness and generalization ability to noisy data. Formulas (1) to (2) are as follows: (1) (2) Where N is the number of samples, It is The original labels of the images, It is The prediction results for the image, is the label smoothing coefficient.

[0018] In one embodiment, in the above step S4: during the inference process, the ultra-wide field of view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain the test image data , the test image data The ultra-wide field of view fundus image to be predicted is input into the trained prediction module, and all prediction results are aggregated as the final prediction result, including: Step S41: enlarging the ultra-wide field of view fundus image to be predicted, and adjusting the width and height to a predetermined size; Due to preprocessing and image augmentation, the model perceives a larger image size during training than during testing, so the image width and height are set to a uniform size of 576 (1.25 times the image size of 448 during training). Step S42: Use the test-time augmentation (TTA) method to horizontally flip the resized image, input the ultra-wide field of view fundus image to be predicted and the flipped image into each trained sub-model for independent prediction, and take the average of the prediction results of the four sub-models corresponding to each residual network as the prediction result of the residual network. The final prediction result is the average of the prediction results of the three residual networks ResNet34d, ResNet50d and ResNext50d, where is the probability of determining image quality; Step S43: Evaluate image quality according to a preset threshold: Images with values ​​greater than the threshold are judged as high-quality images; Images with a value less than or equal to the threshold are considered low-quality images.

[0019] Embodiment 2 like Figure 4 As shown, an embodiment of the present invention provides a quality assessment system for ultra-wide field of view fundus images based on deep learning, including the following modules: The image preprocessing module 51 is used for model training. The ultra-wide field of view fundus image data set is input into the image preprocessing module, and the data diversity is increased by horizontal flipping data enhancement. The data is normalized to improve the model stability, and the processed image data is obtained. ; Prediction module 52, used to convert image data Input prediction module for training. The prediction module consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated in the 4-fold cross-validation process. An optimization module 53, used for optimizing the prediction module using a binary cross entropy loss function that introduces a label smoothing technique during training; The inference image enhancement module 54 is used to input the ultra-wide field of view fundus image to be predicted into the inference image enhancement module for enhancement during the inference process to obtain the test image data. , the test image data The trained prediction module is input with the ultra-wide field of view fundus image to be predicted, and all prediction results are aggregated as the final prediction result.

Claims

1. A method for quality assessment of ultra-wide field of view fundus images based on deep learning, characterized in that: include: Step S1: During model training, the ultra-wide field of view fundus image dataset is input into the image preprocessing module, and the data diversity is increased by horizontal flipping data enhancement. After the data is normalized to improve the model stability, the processed image data is obtained. ; Step S2: Image data The prediction module is input for training. The prediction module consists of three residual networks: ResNet34d, ResNet50d, and ResNext50d. Each residual network contains four sub-models generated in the 4-fold cross-validation process. Step S3: During the training process, the prediction module is optimized using a binary cross entropy loss function that introduces label smoothing technology; Step S4: During the inference process, the ultra-wide field of view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain the test image data , the test image data The ultra-wide field of view fundus image to be predicted is input into a trained prediction module, and all prediction results are aggregated as the final prediction result.

2. The method for quality assessment of ultra-wide field of view fundus images based on deep learning according to claim 1, characterized in that: Step S1: During model training, the ultra-wide field of view fundus image data set is input into the image preprocessing module, data diversity is increased by horizontal flipping data enhancement, and data is normalized to improve model stability to obtain processed image data. , specifically including: Step S11: Input the ultra-wide field of view fundus image into the image size adjustment module, and use Resize to adjust it to a predetermined size to obtain image data ; Step S12: Use the horizontal flip module to randomly flip the image data Perform horizontal flip processing to obtain flip enhanced data ; Step S13: During training, the image data is transformed into Linearly mix with another random image to get mixed enhanced data ; Step S14: Use a normalization module to normalize the mixed enhancement data Perform normalization to adjust the pixel values ​​to a specific distribution and standardize each channel to obtain standardized image data. .

3. The method for quality assessment of ultra-wide field of view fundus images based on deep learning according to claim 2, characterized in that: Step S2: converting the image data The prediction module is input for training. The prediction module consists of three residual networks: ResNet34d, ResNet50d and ResNext50d. Each residual network contains four sub-models generated in the 4-fold cross-validation process, including: Step S21: In the training phase of the model, the preprocessed ultra-wide field of view fundus image dataset is divided into four non-overlapping subsets: Fold 1, Fold 2, Fold 3, and Fold 4, wherein the number of samples in each subset is roughly equal and the category distribution is similar; Step S22: Select any one of the four subsets as the validation set, and use the remaining three subsets as the training sets to form four folds. Residual networks ResNet34d, ResNet50d, and ResNext50d are trained on the four folds respectively, and each residual network obtains 4 trained sub-models.

4. The method for quality assessment of ultra-wide field of view fundus images based on deep learning according to claim 3, characterized in that: The step S3: optimizing the prediction module using a binary cross entropy loss function that introduces label smoothing technology during the training process, specifically includes: Step S31: The original label Adjust to the form of one-hot encoding. For the binary classification task of ultra-wide field fundus image quality assessment, rewrite label 0 as [1, 0] and rewrite label 1 as [0, 1]; Step S32: Smoothing the label: First, set the label smoothing coefficient , Used to adjust the smoothness of positive and negative labels, rewriting the label [1, 0] to [ , ], rewrite the label [0, 1] to [ , ]; Step S33: Minimize the binary cross entropy loss function with smooth labels To optimize the parameters, we avoid overfitting the real labels of the samples during training, thereby improving the robustness and generalization ability to noisy data. Formulas (1) to (2) are as follows: (1) (2) Where N is the number of samples, It is The original labels of the images, It is The prediction results for the image, is the label smoothing coefficient.

5. The method for quality assessment of ultra-wide field of view fundus images based on deep learning according to claim 4, characterized in that: Step S4: During the inference process, the ultra-wide field of view fundus image to be predicted is input into the inference image enhancement module for enhancement to obtain the test image data. , the test image data The ultra-wide field of view fundus image to be predicted is input into a trained prediction module, and all prediction results are aggregated as the final prediction result, specifically including: Step S41: Enlarging the ultra-wide field of view fundus image to be predicted, and adjusting the width and height to a predetermined size; Step S42: horizontally flip the resized image, input the ultra-wide field of view fundus image to be predicted and the flipped image into each of the trained sub-models for independent prediction, and take the average of the prediction results of the four sub-models corresponding to each residual network as the prediction result of the residual network. The final prediction result is the average of the prediction results of the three residual networks ResNet34d, ResNet50d and ResNext50d, where is the probability of determining image quality; Step S43: Evaluate image quality according to a preset threshold: Images with values ​​greater than the threshold are judged as high-quality images; Images with a value less than or equal to the threshold are considered low-quality images.

6. A quality assessment system for ultra-wide field of view fundus images based on deep learning, characterized in that: Includes the following modules: The image preprocessing module is used for model training. The ultra-wide field of view fundus image data set is input into the image preprocessing module, and the data diversity is increased by horizontal flipping data enhancement. After the data is normalized to improve the model stability, the processed image data is obtained. ; Prediction module, used to transform image data The prediction module is input for training. The prediction module consists of three residual networks: ResNet34d, ResNet50d, and ResNext50d. Each residual network contains four sub-models generated in the 4-fold cross-validation process. An optimization module, used for optimizing the prediction module during training using a binary cross entropy loss function that introduces a label smoothing technique; The inference image enhancement module is used to input the ultra-wide field of view fundus image to be predicted into the inference image enhancement module for enhancement during the inference process to obtain the test image data. , the test image data The ultra-wide field of view fundus image to be predicted is input into a trained prediction module, and all prediction results are aggregated as the final prediction result.

Images