A method, system, device and storage medium for fundus image quality assessment

Abstract

The present invention belongs to the field of image processing, and provides a fundus image quality evaluation method, system, device and storage medium. The quality evaluation model of the fundus image quality evaluation system includes a two-stream network module, a cross-fusion module, a first RCAM module, a second RCAM module and a classification module; the two-stream network module includes a first network and a second network; the high-quality image correlation features extracted by the first network are used to subsequently determine whether the fundus image is a high-quality image, and the low-quality image correlation features extracted by the second network are used to subsequently determine whether the fundus image is a low-quality image. The setting of the two-stream network module enables the present invention to accurately identify intermediate-quality images, avoiding waste caused by the abandonment of this category of images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and relates to a method, system, device and storage medium for fundus image quality assessment. Background Art

[0002] Fundus image quality assessment is a crucial part of medical imaging. Especially in ophthalmic diagnosis, accurate image quality plays a vital role in disease screening and treatment plan formulation. Fundus images are widely used in the diagnosis of various ophthalmic diseases such as retinal diseases, diabetic retinopathy, macular degeneration, glaucoma, etc. Due to differences in different devices, shooting conditions, imaging technologies, and doctor operation levels, there are significant variations in the quality of fundus images. Some low-quality fundus images may affect the identification and accurate diagnosis of lesions due to blurring, noise, or perspective problems, so these images need to be identified and excluded. Another part of the fundus images with relatively low quality still contains valuable pathological information, and their quality can be improved through reasonable image enhancement processing and then used for effective diagnosis.

[0003] Existing fundus image quality assessment methods mainly rely on manual inspection or traditional image quality scoring systems. These methods usually rely on doctors' experience, are inefficient, and are easily affected by human subjective factors. With the continuous development of deep learning technology, automated image quality assessment methods based on Convolutional Neural Networks (CNN) have gradually been applied. These methods can perform quality scoring by automatically learning image features and automatically detect low-quality images, significantly improving the accuracy and efficiency of image quality assessment. However, existing image quality assessment methods do not perform well in evaluating fundus images (i.e., intermediate-state quality images) that are affected by degradation features to a lesser extent, contain pathological structures distinguishable by the naked eye, and can be used for clinical diagnosis through subsequent image enhancement, and are prone to classifying this part of the images as unusable images, thus wasting part of the patient data resources. Summary of the Invention

[0004] Aiming at the problem that existing image quality assessment methods are difficult to define intermediate-state quality images, the present invention proposes a method, system, device and storage medium for fundus image quality assessment.

[0005] The present invention is realized through the following technical solutions:

[0006] A fundus image quality assessment system, comprising:

[0007] A quality assessment model for classifying the quality of fundus images to obtain quality categories;

[0008] The quality assessment model includes:

[0009] The dual-stream network module includes a first network and a second network respectively used to extract high-quality image correlation features and low-quality image correlation features in the fundus image; both the first network and the second network include a plurality of densely connected blocks connected in sequence; the outputs of the densely connected blocks of the first network are concatenated and used as the first output of the first network, and the output of the last densely connected block is used as the second output of the first network; the outputs of the densely connected blocks of the second network are concatenated and used as the first output of the second network, and the output of the last densely connected block is used as the second output of the second network;

[0010] The Cross-Fusing Module (CFM) is used to perform cross-fusion processing on the first output of the first network and the first output of the second network to obtain the first CFM output and the second CFM output;

[0011] The first RCAM module (Residual Channel Attention Module) is used to focus attention on the first CFM output and the second output of the first network to obtain the first classification feature;

[0012] The second RCAM module; is used to focus attention on the second CFM output and the second output of the second network to obtain the second classification feature;

[0013] The classification module is used to obtain the quality category according to the first classification feature and the second classification feature.

[0014] Preferably, the fundus image quality assessment system further includes a Gabor filtering module; the Gabor filtering module is used to perform Gabor filtering on the fundus image to obtain a Gabor filtered image; the input image of the first network is the fundus image, and the input image of the second network is the Gabor filtered image.

[0015] Preferably, the cross-fusion module includes:

[0016] The first channel is used to extract local features from the first output of the first network;

[0017] The first cross-fusion summation module is used to add the output of the first channel to the first output of the second network;

[0018] The first SE (Squeeze-and-Excitation) module (SEM) is used to perform feature selection on the output of the first cross-fusion summation module;

[0019] The first post - processing module is used to adjust the number of feature channels of the output of the first SE module so that the number of feature channels of the output of the first SE module is the same as that of the first output of the first network, and obtain the first CFM output;

[0020] The second channel is used to perform local feature extraction on the first output of the second network;

[0021] The second cross - fusion addition module is used to add the output of the second channel to the first output of the first network;

[0022] The second SE module is used to perform feature selection on the output of the second cross - fusion addition module;

[0023] The second post - processing module is used to adjust the number of feature channels of the output of the second SE module so that the number of feature channels of the output of the second SE module is the same as that of the first output of the second network, and obtain the second CFM output.

[0024] Further, the first channel includes: a first cross - fusion convolutional layer, a first cross - fusion batch normalization (Batch Normalization, BN) layer, and a first cross - fusion ReLU (Rectified Linear Unit) activation function module; the second channel includes: a second cross - fusion convolutional layer, a second cross - fusion batch normalization layer, and a second cross - fusion ReLU activation function module;

[0025] The first output of the first network is processed successively through the first cross - fusion convolutional layer, the first cross - fusion batch normalization layer, and the first cross - fusion ReLU activation function module to obtain the output of the first channel;

[0026] The first output of the second network is processed successively through the second cross - fusion convolutional layer, the second cross - fusion batch normalization layer, and the second cross - fusion ReLU activation function module to obtain the output of the second channel.

[0027] Preferably, the first RCAM module includes:

[0028] The first residual convolutional layer is used to perform a convolutional operation on the first CFM output;

[0029] The first residual addition module is used to perform element - level addition of the output of the first residual convolutional layer and the second output of the first network;

[0030] The second residual convolutional layer is used to perform a convolutional operation on the output of the first residual addition module;

[0031] The first Sigmoid activation function module is used to process the output of the second residual convolutional layer using the Sigmoid activation function to obtain the first attention weight;

[0032] The third residual convolutional layer is used to perform a convolutional operation on the output of the first CFM;

[0033] The first multiplication module is used to multiply the first attention weight by the output of the third residual convolutional layer to obtain the first selected feature;

[0034] The first global pooling (GP) layer is used to perform pooling processing on the output of the first CFM;

[0035] The fourth residual convolutional layer is used to perform a convolutional operation on the output of the first global pooling layer;

[0036] The first residual ReLU activation function module is used to process the output of the fourth residual convolutional layer using the ReLU activation function to obtain the first channel attention score;

[0037] The second multiplication module is used to multiply the first selected feature by the first channel attention score to obtain the first classification feature.

[0038] Furthermore, the second RCAM module includes:

[0039] The fifth residual convolutional layer is used to perform a convolutional operation on the output of the second CFM;

[0040] The second residual addition module is used to perform element-wise addition of the output of the fifth residual convolutional layer and the second output of the second network;

[0041] The sixth residual convolutional layer is used to perform a convolutional operation on the output of the second residual addition module;

[0042] The second Sigmoid activation function module is used to process the output of the sixth residual convolutional layer using the Sigmoid activation function to obtain the second attention weight;

[0043] The seventh residual convolutional layer is used to perform a convolutional operation on the output of the second CFM;

[0044] The third multiplication module is used to multiply the second attention weight by the output of the seventh residual convolutional layer to obtain the second selected feature;

[0045] The second global pooling layer is used to perform pooling processing on the output of the second CFM;

[0046] The eighth residual convolutional layer is used to perform a convolutional operation on the output of the second global pooling layer;

[0047] The second residual ReLU activation function module is used to process the output of the eighth residual convolutional layer using the ReLU activation function to obtain the second channel attention score;

[0048] The fourth multiplication module is used to multiply the second selected feature by the second channel attention score to obtain the second classification feature.

[0049] Preferably, the fundus image quality assessment system further includes a data augmentation module;

[0050] The data augmentation module is used to augment the fundus images of a few quality categories in the training dataset before training the quality assessment model.

[0051] A fundus image quality assessment method uses the above-mentioned fundus image quality assessment system to classify the quality of a fundus image and obtain the quality category.

[0052] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned fundus image quality assessment method is implemented.

[0053] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned fundus image quality assessment method is implemented.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The fundus image quality assessment system described in the present invention is an image quality classification network based on dual-stream cross-fusion. Its quality assessment model includes a dual-stream network module, a cross-fusion module, residual channel attention modules (i.e., the first RCAM module and the second RCAM module), and a classification module, which can accurately identify the categories of image quality. The first network and the second network in the dual-stream network module are respectively used to extract the associated features of high-quality images and the associated features of low-quality images in the fundus image. The associated features of high-quality images extracted by the first network are used to subsequently determine whether the fundus image is a high-quality image, and the associated features of low-quality images extracted by the second network are used to subsequently determine whether the fundus image is a low-quality image. When the former judgment result is a non-high-quality image and the latter judgment result is a non-low-quality image, it is considered that the fundus image is an intermediate-state quality image. Therefore, the setting of the dual-stream network module enables the present invention to accurately identify intermediate-state quality images and avoid waste caused by the abandonment of this category of images. The cross-fusion module performs cross-fusion between the first network and the second network, extracting the useful information in the fundus image to the greatest extent, which can solve the problems of no information exchange and no feature interaction between the first network and the second network, and achieve efficient fusion of information between channels and feature enhancement; the features output by the cross-fusion module couple the useful information of the fundus image itself, aiming to improve the robustness of the quality assessment model classification. The residual channel attention module enhances the feature selection ability with the attention mechanism. This module can complete feature selection, generate the final classification features, and improve the stability of the quality assessment model. The system of the present invention can be used in the quality analysis of premature infant fundus images and intelligent medical image assessment.

[0056] Furthermore, most existing classification models do not pay attention to the structural features of fundus images and lack the ability to represent structural features. They need to assume in advance which form of degradation the fundus image has been affected by and generate degradation features. They are not sensitive enough to fundus images containing multiple degradation features. To solve this problem, the first network of the present invention uses the original fundus image as the input image, while the second network uses the Gabor filter image of the original fundus image as the input image. The Gabor filter image can highlight the structural features of the fundus image, and it will be easier and more sufficient to extract the structural features after being input into the second network, so as to automatically judge the degree of influence of the fundus image by various forms of degradation, judge the amount of mixed degradation features in the fundus image, and be more sensitive to fundus images containing multiple degradation features. Therefore, the present invention can better extract the structural features of fundus images, identify multiple degradation features, and thus improve the accuracy of fundus image quality classification.

[0057] Furthermore, the present invention sets up a data augmentation module, which is used to expand the fundus images of a small number of quality categories, so as to generate diverse training data, significantly improve the generalization ability of the quality assessment model, and enable it to maintain high performance in various practical applications. Brief Description of the Drawings

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0059] Figure 1 is the framework of the quality assessment model of the present invention;

[0060] Figure 2 is the framework of the cross-fusion module of the present invention;

[0061] Figure 3 is the framework of the residual channel attention module of the present invention. Detailed Embodiments

[0062] The following illustrates the embodiments of the present invention through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0063] It should be noted that the process equipment or devices not specifically noted in the following embodiments all adopt conventional equipment or devices in the art.

[0064] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices. Moreover, unless otherwise specified, the numbering of each method step is only a convenient tool for identifying each method step, rather than limiting the arrangement order of each method step or the scope in which the present invention can be implemented. The change or adjustment of their relative relationships, without substantial change in the technical content, should also be regarded as the scope in which the present invention can be implemented.

[0065] The fundus image quality assessment system described in the present invention includes:

[0066] A quality assessment model for classifying the quality of the input fundus image to obtain a quality category;

[0067] Among them, as Figure 1 shown, the quality assessment model includes:

[0068] The dual-stream network module includes a first network and a second network respectively used to extract high-quality image-related features and low-quality image-related features in fundus images; the first network and the second network respectively include a plurality of densely connected blocks connected in sequence; the outputs of each densely connected block in the first network are concatenated in the feature channel dimension and used as the first output of the first network, and the output of the last densely connected block is used as the second output of the first network; the outputs of each densely connected block in the second network are concatenated in the feature channel dimension and used as the first output of the second network, and the output of the last densely connected block is used as the second output of the second network;

[0069] The cross-fusion module is used to perform cross-fusion processing on the first output of the first network and the first output of the second network to obtain a first CFM output and a second CFM output corresponding to the first network and the second network respectively;

[0070] The residual channel attention module includes a first RCAM module and a second RCAM module; the first RCAM module takes the first CFM output and the second output of the first network as inputs, performs attention focusing processing, and obtains a first classification feature; the second RCAM module takes the second CFM output and the second output of the second network as inputs, performs attention focusing processing, and obtains a second classification feature;

[0071] The classification module is used to obtain the quality category according to the first classification feature and the second classification feature.

[0072] In the present invention, both the first network and the second network are densely connected convolutional networks (Densely Connected Convolutional Networks, DenseNet), and the high-quality image-related features and the low-quality image-related features both include degradation features and structural features.

[0073] In a preferred embodiment of the present invention, the fundus image quality assessment system further includes an image acquisition module and a result output module;

[0074] The image acquisition module is used to acquire a fundus image and input the fundus image into the quality assessment model;

[0075] The result output module is used to output the quality category.

[0076] In a preferred embodiment of the present invention, the fundus image quality assessment system further includes a Gabor filtering module; the Gabor filtering module is used to perform Gabor filtering on the fundus image to obtain a Gabor filtered image; the input image of the first network is the fundus image, and the input image of the second network is the Gabor filtered image.

[0077] The first network of the present invention uses the original fundus image as the input image, and the second network uses the Gabor filter image of the original fundus image as the input image. The Gabor filter image can highlight the structural features of the fundus image, and it will be easier and more sufficient to extract the structural features after inputting into the second network.

[0078] As Figure 2 shown, the cross-fusion module of the present invention includes:

[0079] The first channel is used for local feature extraction of the first output of the first network;

[0080] The first cross-fusion addition module is used to add the output of the first channel to the first output of the second network;

[0081] The first SE module is used for feature selection of the output of the first cross-fusion addition module;

[0082] The first post-processing module is used to adjust the number of feature channels of the output of the first SE module so that the number of feature channels of the output of the first SE module is the same as that of the first output of the first network, and the first CFM output is obtained;

[0083] The second channel is used for local feature extraction of the first output of the second network;

[0084] The second cross-fusion addition module is used to add the output of the second channel to the first output of the first network;

[0085] The second SE module is used for feature selection of the output of the second cross-fusion addition module;

[0086] The second post-processing module is used to adjust the number of feature channels of the output of the second SE module so that the number of feature channels of the output of the second SE module is the same as that of the first output of the second network, and the second CFM output is obtained.

[0087] Among them, the first channel includes: the first cross-fusion convolutional layer, the first cross-fusion batch normalization layer, and the first cross-fusion ReLU activation function module; the second channel includes: the second cross-fusion convolutional layer, the second cross-fusion batch normalization layer, and the second cross-fusion ReLU activation function module;

[0088] The first output of the first network is processed successively through the first cross-fusion convolutional layer, the first cross-fusion batch normalization layer, and the first cross-fusion ReLU activation function module to obtain the output of the first channel;

[0089] The first output of the second network is successively processed by a second cross-fusion convolutional layer, a second cross-fusion batch normalization layer, and a second cross-fusion ReLU activation function module to obtain the output of the second channel.

[0090] Among them, the first post-processing module includes: a third cross-fusion convolutional layer, a third cross-fusion batch normalization layer, and a third cross-fusion ReLU activation function module; the second post-processing module includes: a fourth cross-fusion convolutional layer, a fourth cross-fusion batch normalization layer, and a fourth cross-fusion ReLU activation function module;

[0091] The output of the first SE module is successively processed by a third cross-fusion convolutional layer, a third cross-fusion batch normalization layer, and a third cross-fusion ReLU activation function module to obtain the first CFM output;

[0092] The output of the second SE module is successively processed by a fourth cross-fusion convolutional layer, a fourth cross-fusion batch normalization layer, and a fourth cross-fusion ReLU activation function module to obtain the second CFM output.

[0093] As Figure 3 shown, it is a schematic diagram of one of the RCAM modules, applicable to the first RCAM module and the second RCAM module.

[0094] The first RCAM module of the present invention includes:

[0095] A first residual convolutional layer for performing a convolutional operation on the first CFM output;

[0096] A first residual summation module for performing element-wise addition of the output of the first residual convolutional layer and the second output of the first network;

[0097] A second residual convolutional layer for performing a convolutional operation on the output of the first residual summation module;

[0098] A first Sigmoid activation function module for processing the output of the second residual convolutional layer using a Sigmoid activation function to obtain a first attention weight;

[0099] A third residual convolutional layer for performing a convolutional operation on the first CFM output;

[0100] A first multiplication module for multiplying the first attention weight and the output of the third residual convolutional layer to obtain a first selected feature;

[0101] A first global pooling layer for performing pooling processing on the first CFM output;

[0102] A fourth residual convolutional layer for performing a convolutional operation on the output of the first global pooling layer;

[0103] The first residual ReLU activation function module is used to process the output of the fourth residual convolutional layer with the ReLU activation function to obtain the first channel attention score;

[0104] The second multiplication module is used to multiply the first selected feature by the first channel attention score to obtain the first classification feature.

[0105] The second RCAM module includes:

[0106] The fifth residual convolutional layer is used to perform a convolutional operation on the output of the second CFM;

[0107] The second residual summation module is used to perform element-wise addition of the output of the fifth residual convolutional layer and the second output of the second network;

[0108] The sixth residual convolutional layer is used to perform a convolutional operation on the output of the second residual summation module;

[0109] The second Sigmoid activation function module is used to process the output of the sixth residual convolutional layer with the Sigmoid activation function to obtain the second attention weight;

[0110] The seventh residual convolutional layer is used to perform a convolutional operation on the output of the second CFM;

[0111] The third multiplication module is used to multiply the second attention weight by the output of the seventh residual convolutional layer to obtain the second selected feature;

[0112] The second global pooling layer is used to perform pooling processing on the output of the second CFM;

[0113] The eighth residual convolutional layer is used to perform a convolutional operation on the output of the second global pooling layer;

[0114] The second residual ReLU activation function module is used to process the output of the eighth residual convolutional layer with the ReLU activation function to obtain the second channel attention score;

[0115] The fourth multiplication module is used to multiply the second selected feature by the second channel attention score to obtain the second classification feature.

[0116] The output of the cross-fusion module in the present invention generates the final classification feature through residual connection in the residual channel attention module, so as to improve the stability of the quality assessment model. The residual channel attention module effectively avoids the problem of gradient disappearance during the training process of the quality assessment model through residual connection, ensuring the effective training of the quality assessment model; with the help of the attention mechanism, the feature selection ability is enhanced, and this module can complete feature selection and generate the final classification feature.

[0117] The classification module described in the present invention maps the first classification feature and the second classification feature to Bernoulli distributions of different quality categories respectively to obtain Bernoulli probabilities, and obtains the quality category according to the Bernoulli probabilities.

[0118] As Figure 1 shown, the classification module described in the present invention specifically includes:

[0119] A third global pooling layer for performing global pooling processing on the first classification feature;

[0120] A fourth global pooling layer for performing global pooling processing on the second classification feature;

[0121] A first fully connected layer (Fully Connected Layer, FC) that maps the globally pooled first classification feature to a high-quality Bernoulli distribution classifier to obtain a first Bernoulli probability;

[0122] A second fully connected layer for mapping the globally pooled second classification feature to a low-quality Bernoulli distribution classifier to obtain a second Bernoulli probability;

[0123] A quality category judgment module for obtaining the quality category according to the first Bernoulli probability and the second Bernoulli probability.

[0124] In order to better train the quality assessment model, increase the number of fundus images of minority quality categories, and balance the dataset, the present invention can also set a data augmentation module in the fundus image quality assessment system. The data augmentation module is used to augment the fundus images of minority quality categories in the training dataset before training the quality assessment model. The minority quality category is relative to the majority quality category. When the ratio of the number of fundus images of one quality category to the number of fundus images of another quality category is less than 0.5, then this quality category is considered a minority quality category. By augmenting the fundus images of minority quality categories through the data augmentation module, the problem of heavy workload of doctors caused by relying on doctors for augmentation can be avoided.

[0125] In order to perform visual analysis on the degradation features and structural features, the fundus image quality assessment system described in the present invention further includes a Gradient-weighted Class Activation Mapping (GradCAM) visualization processing module; the GradCAM visualization processing module is used to generate a heat map by calculating the gradient of the first classification feature with respect to the output of the third residual convolutional layer and the gradient of the second classification feature with respect to the output of the seventh residual convolutional layer.

[0126] By combining GradCAM visualization analysis, the present invention provides an intuitive explanation of feature distribution, which not only shows the degradation features and structural features that the quality assessment model focuses on, but also helps doctors understand the classification basis of the quality assessment model and improve the credibility of classification results.

[0127] The fundus image quality assessment method of the present invention includes:

[0128] Inputting the fundus image into a pre-trained quality assessment model for quality classification to obtain a quality category;

[0129] Wherein, the quality assessment model is the quality assessment model in the fundus image quality assessment system as described above.

[0130] Example 1

[0131] The fundus image quality assessment system of this embodiment includes: an image acquisition module, a quality assessment model, and a result output module;

[0132] The image acquisition module is used to acquire a fundus image and input the acquired fundus image into the quality assessment model;

[0133] The quality assessment model is used to perform quality classification on the input fundus image to obtain a quality category;

[0134] The result output module is used to output the quality category.

[0135] The present invention uses a dataset labeled with quality categories by doctors to train the quality assessment model. The trained quality assessment model takes the fundus image as input to achieve accurate assessment of the quality of fundus images.

[0136] The dataset labeled with quality categories by doctors is obtained through the following method:

[0137] First, collect original fundus images and perform preliminary cleaning and preprocessing, including archiving fundus images with the same number, unifying the naming format of fundus images, and classifying left and right eye image data to ensure the unity and integrity of fundus images; the number is the shooting date of the fundus image;

[0138] Then, label the fundus images to confirm the quality category of each fundus image to form an initial dataset.

[0139] In this embodiment, the fundus image quality assessment system further includes a data augmentation module, and the data augmentation module is used to augment a small number of fundus images with quality categories in the training dataset before training the quality assessment model.

[0140] Before training the quality assessment model, the present invention designs a data augmentation module to process the initial dataset to improve the performance of the quality assessment model. The core of the data augmentation module is to use a data augmentation algorithm to augment the fundus images of a few quality categories in the initial dataset to balance the number of fundus images of each quality category in the initial dataset.

[0141] In the present invention, the specific steps for the data augmentation module to augment the fundus images of a few quality categories are as follows:

[0142] 1) Analyze the number of fundus images of each quality category in the initial dataset, and identify the few quality categories and the majority quality categories. For example, if the ratio of the number of fundus images of the low-quality category to the number of fundus images of the high-quality category is less than 0.5, then the low-quality category is considered a few quality categories, and the fundus images of the low-quality category need to be augmented. For the identified few quality categories, apply image data augmentation methods to generate more fundus images. Commonly used image data augmentation methods include, but are not limited to, operations such as rotation, scaling, translation, flipping, color transformation, etc., to generate new image data.

[0143] 2) Merge the augmented fundus images of the few quality categories with the initial dataset to form a new and balanced dataset. Have the doctor re-check the new dataset to determine whether the augmented fundus images match the few quality categories to ensure data quality.

[0144] Through the above steps, the present invention achieves the technical effect of effectively increasing the number of fundus images of the few quality categories and balancing the dataset.

[0145] The quality assessment model in this embodiment is an image quality classification network based on dual-stream cross-fusion feature extraction, specifically including a dual-stream network module, a cross-fusion module, a residual channel attention module, a classification module, and a visualization processing module, which are used for the extraction of diverse degradation features and structural features to achieve accurate identification of image quality categories.

[0146] The present invention designs a dual-stream network module and proposes a brand-new quality classification strategy based on the dual-stream network module to assist in the accurate classification of intermediate-state quality fundus images; designs a cross-fusion module to fuse the associated features of high-quality images and the associated features of low-quality images; designs a residual channel attention module to use residual connections to avoid gradient disappearance during the training process of the quality assessment model, and complete feature selection with the help of the attention mechanism to generate the final classification features. The present invention uses a fully connected layer to output a Bernoulli distribution for predicting quality categories, and uses GradCAM to perform visual analysis on degradation features and structural features to provide an interpretable basis for the feature distribution of the fundus image quality classification results.

[0147] The following is a detailed introduction to each module of the quality assessment model.

[0148] (1) Dual-stream network module for assisting in the quality classification of intermediate states

[0149] The present invention introduces a dual-stream network module for assisting in the quality classification of intermediate states. The dual-stream network module includes two networks, namely, the first network (high-quality classifier) and the second network (low-quality classifier), that is, a dual-stream network. The first network and the second network are respectively used to extract high-quality image correlation features and low-quality image correlation features in the fundus image, and the DenseNet network is used as the backbone network of the first network and the second network. Both the high-quality image correlation features and the low-quality image correlation features include degradation features and structural features. The degradation features are such as blur and noise, and the structural features are such as blood vessel structure and fundus pathological structure.

[0150] In this embodiment, the fundus image quality assessment system further includes a Gabor filtering module, which is used to perform Gabor filtering on the fundus image acquired by the image acquisition module to obtain a Gabor filtered image; the input image of the first network is the fundus image, and the input image of the second network is the Gabor filtered image. The Gabor filtered image can highlight the structural features of the fundus image, and it will be easier and more sufficient to extract the structural features after inputting it into the second network.

[0151] The first network and the second network respectively include a plurality of densely connected blocks connected in sequence; the features obtained by splicing the outputs of each densely connected block in the first network in the feature channel dimension are used as the first output of the first network, and the output of the last densely connected block is used as the second output of the first network; the features obtained by splicing the outputs of each densely connected block in the second network in the feature channel dimension are used as the first output of the second network, and the output of the last densely connected block is used as the second output of the second network.

[0152] (2) Cross-fusion module

[0153] The present invention provides a cross-fusion module, which is designed to solve the problems of no information exchange and no feature interaction in the dual-stream network, and is mainly used for cross-fusion between dual-stream networks. By introducing multiple convolutional layers, batch normalization layers, ReLU activation function modules, and channel attention modules, this cross-fusion module realizes the efficient fusion of information between channels and feature enhancement, thereby improving the overall performance of the quality assessment model.

[0154] Specifically, the cross-fusion module includes a first channel, a second channel, a first cross-fusion addition module, a second cross-fusion addition module, a first SE module, a second SE module, a first post-processing module, and a second post-processing module. The first channel includes a first cross-fusion convolutional layer, a first cross-fusion batch normalization layer, and a first cross-fusion ReLU activation function module. The second channel includes: a second cross-fusion convolutional layer, a second cross-fusion batch normalization layer, and a second cross-fusion ReLU activation function module. The first post-processing module includes: a third cross-fusion convolutional layer, a third cross-fusion batch normalization layer, and a third cross-fusion ReLU activation function module; the second post-processing module includes: a fourth cross-fusion convolutional layer, a fourth cross-fusion batch normalization layer, and a fourth cross-fusion ReLU activation function module;

[0155] In this embodiment, both the first cross-fusion convolutional layer and the second cross-fusion convolutional layer are 3 ×3 convolutional layers (Conv 3 ×3), which are used to adjust the number of feature channels so that the output sizes of the first channel and the second channel are the same.

[0156] The inputs of the first channel and the second channel come from the first output of the first network and the first output of the second network of the two-stream network module respectively. First, the input of the first channel is sequentially processed by the first cross-fusion convolutional layer, the first cross-fusion batch normalization layer, and the first cross-fusion ReLU activation function module to extract local features; at the same time, the input of the second channel is sequentially processed by the second cross-fusion convolutional layer, the second cross-fusion batch normalization layer, and the second cross-fusion ReLU activation function module to extract local features. The output of the first channel is added to the first output of the second network in the first cross-fusion addition module, and the output of the second channel is added to the first output of the first network in the second cross-fusion addition module. Then, the features are enhanced with channel attention through two SE modules (the first SE module and the second SE module), where the first SE module performs feature selection on the output of the first cross-fusion addition module; the second SE module performs feature selection on the output of the second cross-fusion addition module. Finally, the output of the first SE module (i.e., the enhanced features) is sequentially processed by the third cross-fusion convolutional layer, the third cross-fusion batch normalization layer, and the third cross-fusion ReLU activation function module to obtain the first CFM output; the output of the second SE module (i.e., the enhanced features) is sequentially processed by the fourth cross-fusion convolutional layer, the fourth cross-fusion batch normalization layer, and the fourth cross-fusion ReLU activation function module to obtain the second CFM output, so as to achieve dimensionality reduction and non-linear transformation, thereby reducing the number of feature channels and extracting higher-level features. After these processes, two groups of features enhanced and fused are output, and the two groups of features belong to the high-quality classifier and the low-quality classifier respectively, and are used as the inputs of the subsequent residual channel attention module. This cross-fusion module significantly improves the performance and efficiency of the neural network through inter-channel information fusion and feature enhancement, and is applicable to various deep learning tasks, such as image classification, object detection, semantic segmentation, etc.

[0157] In this embodiment, both the third cross-fusion convolutional layer and the fourth cross-fusion convolutional layer are 1 1 convolutional layer (Conv 1 1), which is used to adjust the size of the feature channels so that the sizes of the first CFM output and the second CFM output are the same.

[0158] (3) Residual channel attention module

[0159] The present invention provides a residual channel attention module, which is designed to use residual connections to avoid the problem of gradient disappearance during the training of the quality assessment model, and at the same time use the attention mechanism to complete feature selection to generate the final classification features. The residual channel attention module aims to achieve the communication and interaction between deep features (i.e., the second output of the first network and the second output of the second network) and multi-scale features (i.e., the output of the first CFM and the output of the second CFM). Through multiple convolutional layers, global pooling layers, ReLU activation functions, and Sigmoid activation functions, the residual channel attention module effectively fuses deep features and multi-scale features, enhancing the feature expression ability of the quality assessment model.

[0160] Specifically, two RCAM modules are set. The first RCAM module is corresponding to the high-quality classifier, and the second RCAM module is corresponding to the low-quality classifier. Each RCAM module has three parallel branches. The inputs of the RCAM module are two: one is the output from the corresponding CFM module, and the other is the output from the last dense connection block of the DenseNet network (the first network or the second network). The first RCAM module includes: a first residual convolutional layer, a first residual addition module, a second residual convolutional layer, a first Sigmoid activation function module, a third residual convolutional layer, a first multiplication module, a first global pooling layer, a fourth residual convolutional layer, a first residual ReLU activation function module, and a second multiplication module; the second RCAM module includes: a fifth residual convolutional layer, a second residual addition module, a sixth residual convolutional layer, a second Sigmoid activation function module, a seventh residual convolutional layer, a third multiplication module, a second global pooling layer, an eighth residual convolutional layer, a second residual ReLU activation function module, and a fourth multiplication module.

[0161] For the first RCAM module, in the first branch, the output of the first CFM adjusts the number of feature channels through the first residual convolutional layer and the second residual convolutional layer. The output of the first residual convolutional layer is element-wise added to the second output of the first network in the first residual addition module. After being processed by the second residual convolutional layer, it generates the first attention weight through the first Sigmoid activation function module. In the second branch, the output of the first CFM is convolved through the third residual convolutional layer. Subsequently, the first attention weight and the output of the third residual convolutional layer are multiplied in the first multiplication module to achieve feature selection, obtaining the first selected feature. In the third branch, the output of the first CFM is processed through the first global pooling layer, the fourth residual convolutional layer, and the first residual ReLU activation function module to obtain the first channel attention score. Finally, the above first selected feature and the first channel attention score are multiplied in the second multiplication module to obtain the first classification feature.

[0162] For the second RCAM module, in the first branch, the output of the second CFM adjusts the number of feature channels through the fifth residual convolutional layer and the sixth residual convolutional layer. Among them, the output of the fifth residual convolutional layer and the second output of the second network are added element - by - element in the second residual addition module. After being processed by the sixth residual convolutional layer, it passes through the second Sigmoid activation function module to generate the second attention weight. In the second branch, the output of the second CFM is convolved by the seventh residual convolutional layer. Subsequently, the second attention weight and the output of the seventh residual convolutional layer are multiplied in the third multiplication module to achieve feature selection, obtaining the second selected feature. In the third branch, the output of the second CFM is processed by the second global pooling layer, the eighth residual convolutional layer, and the second residual ReLU activation function module to obtain the second channel attention score. Finally, the above - mentioned second selected feature and the second channel attention score are multiplied in the fourth multiplication module to obtain the second classification feature.

[0163] This residual channel attention module effectively avoids the problem of gradient disappearance and enhances the feature selection ability through residual connection and channel attention mechanism, and is applicable to various deep - learning tasks, such as image classification, object detection, semantic segmentation, etc.

[0164] In this embodiment, the first residual convolutional layer, the second residual convolutional layer, the third residual convolutional layer, the fifth residual convolutional layer, the sixth residual convolutional layer, and the seventh residual convolutional layer are all 3 3 convolutional layers, and the fourth residual convolutional layer and the eighth residual convolutional layer are 1 1 convolutional layer.

[0165] (4)Classification module

[0166] The classification module described in the present invention specifically includes:

[0167] The third global pooling layer is used to perform global pooling processing on the first classification feature;

[0168] The fourth global pooling layer is used to perform global pooling processing on the second classification feature;

[0169] The first fully - connected layer maps the globally - pooled first classification feature to a high - quality Bernoulli distribution classifier to obtain the first Bernoulli probability;

[0170] The second fully - connected layer is used to map the globally - pooled second classification feature to a low - quality Bernoulli distribution classifier to obtain the second Bernoulli probability;

[0171] The quality category judgment module is used to obtain the quality category according to the first Bernoulli probability and the second Bernoulli probability.

[0172] The present invention uses a fully connected layer to map the extracted first classification feature and second classification feature onto Bernoulli distributions of different quality categories, obtaining Bernoulli probabilities corresponding to a high-quality classifier and a low-quality classifier respectively. The Bernoulli probability corresponding to the high-quality classifier is a value between 0 and 1, aiming to determine whether a fundus image is a high-quality image. The Bernoulli probability corresponding to the low-quality classifier is a value between 0 and 1, aiming to determine whether a fundus image is a low-quality image.

[0173] The method for classifying the quality of fundus images is as follows: the Bernoulli probability less than 0.5 is set to 0, and the one greater than 0.5 is set to 1; if the high-quality classifier and the low-quality classifier respectively obtain (1, 0), then the predicted image quality level is 0; if the high-quality classifier and the low-quality classifier respectively obtain (0, 0), then the predicted image quality level is 1; if the high-quality classifier and the low-quality classifier respectively obtain (0, 1), then the predicted image quality level is 2. The quality level 1 is the intermediate-state quality. This way can effectively assist in the classification of intermediate-state quality images. Through the design of the dual-stream network, the present invention improves the classification accuracy of intermediate-state quality images. After the intermediate-state quality fundus images are separated, they can be enhanced subsequently and then used by doctors for clinical diagnosis of patients, greatly improving the utilization efficiency of fundus images.

[0174] (5)GradCAM Visualization Processing Module

[0175] The present invention uses GradCAM technology to perform visual analysis on the degradation features and structural features, providing an interpretable basis for the fundus image quality classification result in terms of the fundus image feature distribution.

[0176] The GradCAM visualization processing module of the present invention uses GradCAM technology to visualize the regions in the fundus image that affect the classification decision. The GradCAM visualization processing module generates a heat map by calculating the gradients of the first classification feature with respect to the output of the third residual convolutional layer and the gradients of the second classification feature with respect to the output of the seventh residual convolutional layer, highlighting the image regions that the quality assessment model focuses on when making decisions. This visualization method not only shows the degradation features (such as blurring and noise) and structural features (such as blood vessel structures and fundus pathological structures) that the quality assessment model focuses on, but also helps doctors understand the classification basis of the quality assessment model and improves the credibility of the classification result.

[0177] By combining the Bernoulli distribution output of the fully connected layer and the GradCAM visual analysis, the present invention achieves high-accuracy classification of fundus image quality and provides an intuitive explanation of the feature distribution. It is applicable to the field of medical image analysis, especially the quality classification task of fundus images, and can provide effective auxiliary support in practical applications, having broad application prospects and significant practical value.

[0178] Example 2

[0179] This embodiment provides a method for evaluating fundus image quality, which is implemented based on the fundus image quality evaluation system provided in Embodiment 1 and includes:

[0180] Obtain the fundus image to be evaluated;

[0181] Input the fundus image into a pre-trained quality evaluation model for quality classification to obtain a quality category;

[0182] Output the quality category.

[0183] The present invention conducts a quality evaluation experiment on a batch of fundus images with different qualities, and the results are shown in Table 1.

[0184]

[0185] Table 1 shows the experimental results of different methods in the fundus image quality evaluation task, and the comparison is made from four indicators: accuracy, AUC area, F1 score, and Kappa score. Compared with the methods of Vision Transformer (ViT), DenseNet, Inception-V3, MCF-Net (Multiple Color-space Fusion Network), and FIQuA (Fundus Image Quality Assessment) (Aditya Raj, Multivariate Regression-Based Convolutional Neural Network Model for Fundus Image Quality Assessment, IEEE Access, 2020, (8):57810-57821), the method proposed by the present invention performs excellently in all indicators and achieves the best performance: the accuracy reaches 87.81%, the AUC area is 85.45%, the F1 score is 79.49%, and the Kappa score is 72.10%. Compared with the performance of the sub-optimal methods FIQuA (83.31%) and Inception-V3 (82.95%) in terms of accuracy, the method of the present invention significantly improves the evaluation performance. This is mainly due to the introduction of the cross-fusion module and the residual channel attention module, which makes the feature distributions of different levels more separated, especially improves the distinguishability between level 0 and level 1, thereby effectively reducing confusion and ensuring higher classification accuracy and stability.

[0186] In another embodiment of the present invention, a computer device is provided. The computer device includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention can be used for the operation of the fundus image quality assessment method.

[0187] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in the computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed random access memory (Random Access Memory, RAM), or a non-volatile memory, such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the fundus image quality assessment method in the above embodiments.

[0188] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, optical memory, etc.) that contain computer-usable program code.

[0189] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0190] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still can modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A fundus image quality assessment system, characterized in that: include: A quality assessment model is used to classify the quality of fundus images and obtain a quality category; The quality assessment model includes: A dual-stream network module, comprising a first network and a second network for respectively extracting high-quality image-related features and low-quality image-related features in fundus images; the first network and the second network both comprise a plurality of densely connected blocks connected in sequence; the outputs of the densely connected blocks of the first network are concatenated as the first output of the first network, and the output of the last densely connected block is the second output of the first network; the outputs of the densely connected blocks of the second network are concatenated as the first output of the second network, and the output of the last densely connected block is the second output of the second network; A cross-fusion module, used for performing cross-fusion processing on the first output of the first network and the first output of the second network to obtain a first CFM output and a second CFM output; A first RCAM module is used to focus attention on the first CFM output and the second output of the first network to obtain a first classification feature; A second RCAM module; used for focusing attention on the second CFM output and the second network second output to obtain a second classification feature; The classification module is used to obtain a quality category according to the first classification feature and the second classification feature.

2. The fundus image quality assessment system according to claim 1, characterized in that: It also includes a Gabor filtering module; the Gabor filtering module is used to perform Gabor filtering on the fundus image to obtain a Gabor filter image; the input image of the first network is the fundus image, and the input image of the second network is the Gabor filter image.

3. The fundus image quality assessment system according to claim 1, characterized in that: The cross-fusion module includes: A first channel is used to extract local features of a first output of the first network; A first cross-fusion summing module, used for adding the output of the first channel to the first output of the second network; A first SE module, used for performing feature selection on the output of the first cross-fusion summation module; A first post-processing module, used for adjusting the number of characteristic channels of the output of the first SE module so that the number of characteristic channels of the output of the first SE module is consistent with the number of characteristic channels of the first output of the first network, and obtaining a first CFM output; A second channel is used to extract local features of the first output of the second network; A second cross-fusion summing module is used to perform addition processing on the output of the second channel and the first output of the first network; The second SE module is used to perform feature selection on the output of the second cross-fusion summation module; The second post-processing module is used to adjust the number of feature channels of the output of the second SE module so that the number of feature channels of the output of the second SE module is consistent with the number of feature channels of the first output of the second network, thereby obtaining a second CFM output.

4. The fundus image quality assessment system according to claim 3, characterized in that: The first channel includes: a first cross-fusion convolution layer, a first cross-fusion batch normalization layer and a first cross-fusion ReLU activation function module; the second channel includes: a second cross-fusion convolution layer, a second cross-fusion batch normalization layer and a second cross-fusion ReLU activation function module; The first output of the first network is processed in sequence by a first cross-fusion convolutional layer, a first cross-fusion batch normalization layer, and a first cross-fusion ReLU activation function module to obtain an output of a first channel; The first output of the second network is processed in sequence by a second cross-fusion convolutional layer, a second cross-fusion batch normalization layer, and a second cross-fusion ReLU activation function module to obtain an output of a second channel.

5. The fundus image quality assessment system according to claim 1, characterized in that: The first RCAM module comprises: A first residual convolution layer, used to perform a convolution operation on the first CFM output; A first residual summing module, used for performing feature element-level addition of the output of the first residual convolutional layer and the second output of the first network; A second residual convolution layer, used to perform a convolution operation on the output of the first residual summation module; A first Sigmoid activation function module is used to process the output of the second residual convolution layer using the Sigmoid activation function to obtain a first attention weight; The third residual convolution layer is used to perform a convolution operation on the first CFM output; A first multiplication module, used for multiplying the first attention weight and the output of the third residual convolution layer to obtain a first selected feature; A first global pooling layer, used for performing pooling processing on the first CFM output; A fourth residual convolution layer, used to perform a convolution operation on the output of the first global pooling layer; A first residual ReLU activation function module, used to process the output of the fourth residual convolution layer using a ReLU activation function to obtain a first channel attention score; The second multiplication module is used to multiply the first selected feature by the first channel attention score to obtain a first classification feature.

6. The fundus image quality assessment system according to claim 5, characterized in that: The second RCAM module comprises: A fifth residual convolution layer, used to perform a convolution operation on the second CFM output; A second residual summing module, used for performing feature element-level addition of the output of the fifth residual convolutional layer and the second output of the second network; A sixth residual convolution layer, used for performing a convolution operation on the output of the second residual summation module; A second Sigmoid activation function module is used to process the output of the sixth residual convolution layer using a Sigmoid activation function to obtain a second attention weight; A seventh residual convolution layer, used for performing a convolution operation on the second CFM output; A third multiplication module is used to multiply the second attention weight and the output of the seventh residual convolution layer to obtain a second selected feature; A second global pooling layer, used for performing pooling processing on the second CFM output; An eighth residual convolution layer, used to perform a convolution operation on the output of the second global pooling layer; A second residual ReLU activation function module, used to process the output of the eighth residual convolution layer using the ReLU activation function to obtain a second channel attention score; The fourth multiplication module is used to multiply the second selected feature by the second channel attention score to obtain a second classification feature.

7. The fundus image quality assessment system according to claim 1, characterized in that: It also includes a data augmentation module; The data augmentation module is used to expand the fundus images of a few quality categories in the training data set before training the quality assessment model.

8. A method for evaluating fundus image quality, characterized in that: The fundus image quality assessment system according to any one of claims 1 to 7 is used to classify the quality of the fundus image to obtain a quality category.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the fundus image quality assessment method according to claim 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the fundus image quality assessment method according to claim 8 is implemented.

Images