Reading system and reading method based on fundus image classification

Abstract

The disclosure describes a reading system and a reading method based on fundus image classification. The reading system includes an acquisition module for acquiring fundus images; a first classification module for classifying the fundus images using a first classification model to obtain a first classification result and a classification result type; a grouping module for grouping the fundus images into negative result images, positive result images and images to be reclassified; a first quality control module for obtaining a final classification result and images to be arbitrated based on a quality control model configured using a preset negative prediction rate and a preset positive prediction rate; a second classification module for classifying the images to be reclassified using a second classification model to obtain a final classification result and images to be arbitrated; and an arbitration module for arbitrating the images to be arbitrated to obtain an arbitration classification result. Thus, the classification accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a reading system and a reading method based on fundus image classification. BACKGROUND

[0002] Medical images often contain many details of body structures or tissues. In modern hospitals, most of the treatment information comes from medical images such as fundus images. In the clinic, by understanding these details in the medical images, doctors can help identify related diseases. Medical images have developed into the main method of clinical disease identification. However, the traditional identification of disease information based on medical images mainly relies on the judgment of experienced doctors. In this case, the development of an automatic reading technology that can assist doctors in identifying related diseases has become a hot direction in the field of medical imaging. With the development of artificial intelligence technology, reading technology based on computer vision and artificial intelligence such as machine learning has been developed and applied in medical image identification.

[0003] For example, patent document 1 (CN105513077A) discloses a system for diabetic retinopathy screening, which includes a fundus image acquisition device, an image processing and screening device, and a report output device. The fundus image acquisition device is used to collect or receive the fundus image of the examinee; the image processing and screening device is used to process the fundus image and detect whether there is a lesion therein, and then transmit the detection result to the report output device; the report output device outputs the corresponding detection report based on the detection result.

[0004] However, in actual clinical application, due to the diversity of fundus images, the screening system described in patent document 1 may output incorrect or inaccurate detection reports when processing some fundus images, resulting in a decrease in the classification accuracy of the screening system. SUMMARY

[0005] The present disclosure is proposed in view of the above situation, and aims to provide a reading system and a reading method based on fundus image classification that can improve the classification accuracy.

[0006] To this end, the first aspect of the present disclosure provides a reading system based on fundus image classification, comprising: an acquisition module configured to acquire fundus images; a first classification module configured to classify the fundus images using a first classification model to obtain a first classification result and a classification result type based on the first classification result, the classification result type comprising whether re-classification is needed; a grouping module configured to group the fundus images into negative result images, positive result images and images to be re-classified based on the first classification result and the classification result type; a first quality control module comprising a negative quality control module and a positive quality control module, the negative quality control module configured to obtain a negative quality control result of the negative result images using a first quality control model configured with a first confidence threshold based on a preset negative prediction rate, obtain a final classification result based on the negative quality control result or take the negative result images as first images to be arbitrated, the positive quality control module configured to obtain a positive quality control result of the positive result images using a second quality control model configured with a second confidence threshold based on a preset positive prediction rate, obtain the final classification result based on the positive quality control result or take the positive result images as second images to be arbitrated; a second classification module configured to classify the images to be re-classified using a second classification model trained for the images to be re-classified to obtain a second classification result, obtain the final classification result based on the second classification result or take the images to be re-classified as third images to be arbitrated; and an arbitration module configured to receive the first images to be arbitrated, the second images to be arbitrated or the third images to be arbitrated as images to be arbitrated, arbitrate the images to be arbitrated to obtain an arbitration classification result as the final classification result. In this case, the fundus images are grouped into negative result images, positive result images and images to be re-classified, for the negative result images and the positive result images, the negative result images with lower risk are obtained based on the negative prediction rate and the positive result images with higher risk are obtained based on the positive prediction rate and consistency judgment is performed, and for the images to be re-classified, further classification is performed using the second classification model, and finally arbitration is performed on the images to be arbitrated. Thus, the classification accuracy of the reading system can be improved.

[0007] In addition, in the reading system of the first aspect of the present disclosure, optionally, for the negative result images, if the negative quality control result is consistent with the first classification result, the negative quality control result is taken as the final classification result, otherwise the negative result images are taken as the first images to be arbitrated; for the positive result images, if the positive quality control result is consistent with the first classification result, the positive quality control result is taken as the final classification result, otherwise the positive result images are taken as the second images to be arbitrated; for the images to be re-classified, if the second classification result is consistent with the first classification result, the second classification result is taken as the final classification result, otherwise the images to be re-classified are taken as the third images to be arbitrated.

[0008] In addition, in the reading system according to the first aspect of the present disclosure, optionally, the first confidence threshold is configured based on the preset negative prediction rate using the gold standard data; and the second confidence threshold is configured based on the preset positive prediction rate using the gold standard data. In this way, the confidence threshold can be determined.

[0009] In addition, in the reading system according to the first aspect of the present disclosure, optionally, the reverse solving is to traverse the confidence thresholds in a preset range according to a preset step size and solve the performance indicators to obtain a correspondence between the plurality of confidence thresholds and the plurality of groups of performance indicators, the first confidence threshold is determined based on the correspondence between the plurality of confidence thresholds and the plurality of groups of performance indicators and the preset negative prediction rate, and the second confidence threshold is determined based on the correspondence between the plurality of confidence thresholds and the plurality of groups of performance indicators and the preset positive prediction rate. In this case, the correspondence between the plurality of confidence thresholds and the plurality of groups of performance indicators can be used to conveniently and quickly determine the first confidence threshold and the second confidence threshold.

[0010] In addition, in the reading system according to the first aspect of the present disclosure, optionally, the performance indicators include sensitivity, specificity, positive prediction rate, and negative prediction rate.

[0011] In addition, in the reading system according to the first aspect of the present disclosure, optionally, the first classification result includes no retinal lesions, background stage, pre-proliferative stage, and proliferative stage; the negative result image includes the pre-processed fundus image whose first classification result is no retinal lesions and whose classification result type is no need for re-classification; the positive result image includes the pre-processed fundus image whose first classification result is pre-proliferative stage or proliferative stage and whose classification result type is no need for re-classification; and the image to be re-classified includes the pre-processed fundus image whose classification result type is need for re-classification. In this case, the pre-processed fundus image is divided into the negative result image, the positive result image, and the image to be re-classified, which can facilitate subsequent targeted processing of each type of image. In this way, the classification accuracy of the reading system can be further improved.

[0012] In addition, in the reading system according to the first aspect of the present disclosure, optionally, the preset negative prediction rate is 95% to 99%, and the preset positive prediction rate is 95% to 99%. In this way, the preset negative prediction rate and the preset positive prediction rate can be obtained.

[0013] Additionally, in the reading system according to the first aspect of the present disclosure, optionally, the reading system further comprises a self-checking module configured to perform sampling inspection on the fundus images of the negative quality control results to determine whether the first confidence threshold meets the requirement and perform sampling inspection on the fundus images of the positive quality control results to determine whether the second confidence threshold meets the requirement. In this case, the first confidence threshold and the second confidence threshold can be further confirmed to meet the requirement. Thus, the classification accuracy of the reading system can be improved.

[0014] Additionally, in the reading system according to the first aspect of the present disclosure, optionally, the first classification model is established based on a machine learning algorithm, the machine learning algorithm being at least one of a traditional machine learning algorithm and a deep learning algorithm. Thus, the first classification model can be established based on a machine learning algorithm.

[0015] The second aspect of the present disclosure provides a reading method based on fundus image classification, comprising: obtaining a fundus image; classifying the fundus image by using a first classification model to obtain a first classification result and a classification result type including whether re-classification is needed based on the first classification result; dividing the fundus image into a negative result image, a positive result image and a to-be-reclassified image based on the first classification result and the classification result type; obtaining a negative quality control result of the negative result image by using a first quality control model configured to configure a first confidence threshold based on a preset negative prediction rate, obtaining a final classification result based on the negative quality control result or taking the negative result image as a first to-be-arbitrated image, obtaining a positive quality control result of the positive result image by using a second quality control model configured to configure a second confidence threshold based on a preset positive prediction rate, obtaining the final classification result based on the positive quality control result or taking the positive result image as a second to-be-arbitrated image; classifying the to-be-reclassified image by using a second classification model trained for the to-be-reclassified image to obtain a second classification result, obtaining the final classification result based on the second classification result or taking the to-be-reclassified image as a third to-be-arbitrated image; and taking the first to-be-arbitrated image, the second to-be-arbitrated image or the third to-be-arbitrated image as a to-be-arbitrated image, arbitrating the to-be-arbitrated image to obtain an arbitration classification result as the final classification result. In this case, the fundus image is divided into a negative result image, a positive result image and a to-be-reclassified image, for the negative result image and the positive result image, a negative result image with lower risk is obtained based on the negative prediction rate and a positive result image with higher risk is obtained based on the positive prediction rate and consistency is determined, and for the to-be-reclassified image, the second classification model is used for further classification, and finally the to-be-arbitrated image is arbitrated. Thus, the classification accuracy can be improved.

[0016] According to the present disclosure, a reading method and a reading system for fundus image classification with improved classification accuracy can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0017] The present disclosure will now be explained in further detail by way of example only with reference to the drawings, in which:

[0018] Figure 1 is a scenario diagram illustrating a reading method for fundus image classification according to an example of the present disclosure.

[0019] Figure 2 is a block diagram illustrating a reading system for fundus image classification according to an example of the present disclosure.

[0020] FIG. 3(a) is a schematic diagram illustrating a fundus image according to an example of the present disclosure.

[0021] FIG. 3(b) is a schematic diagram illustrating a fundus image according to an example of the present disclosure.

[0022] Figure 4 is a schematic diagram illustrating a convolution kernel used in a convolutional neural network of a first classification module according to an example of the present disclosure.

[0023] Figure 5 is a block diagram illustrating a reading system for fundus image classification according to an example of the present disclosure.

[0024] Figure 6 is a flowchart illustrating a reading method for fundus image classification according to an example of the present disclosure. DETAILED DESCRIPTION

[0025] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same components are designated by the same reference numerals, and a repeated description of the same components will be omitted. In addition, the drawings are schematic views, and the ratio of the dimensions between the components or the shape of the components and the like can be different from the actual ones. It should be noted that the terms "comprise" and "have" and any variations thereof, such as a process, a method, a system, a product, or an apparatus including or having a series of steps or units, are not necessarily limited to those clearly listed steps or units, but can include or have other steps or units that are not clearly listed or inherent to those processes, methods, products, or apparatuses. All methods described in the present disclosure can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context.

[0026] This disclosure relates to a fundus image classification-based image reading system 200 and a reading method that can improve classification accuracy. The fundus image classification-based image reading system 200 may sometimes be simply referred to as the image reading system 200, and the fundus image classification-based image reading method may sometimes be simply referred to as the image reading method.

[0027] Figure 1 This diagram illustrates an application scenario of the image reading method based on fundus image classification as described in this disclosure.

[0028] In some examples, the image reading method (described later) can be applied to, for example... Figure 1 In application scenario 100, operator 110 can control acquisition device 130 connected to terminal 120 to acquire fundus images of the fundus of human eye 140. After acquisition device 130 completes fundus image acquisition, terminal 120 can submit the fundus images to server 150 via computer network. Server 150 can execute computer program instructions stored on server 150 to implement an image reading method, which receives fundus images and generates a fundus image result report. Server 150 can return the generated fundus image result report to terminal 120. In some examples, terminal 120 can display the result report. In other examples, the result report can be stored as an intermediate result in the memory of terminal 120 or server 150. In still other examples, the fundus images received by the image reading method can be fundus images stored in terminal 120 or server 150.

[0029] In some examples, operator 110 may be a professional, such as an ophthalmologist. In other examples, operator 110 may be a general practitioner trained in image interpretation. Image interpretation training may include, but is not limited to, the operation of the acquisition device 130 and the operation of the terminal 120 involving image interpretation methods. In some examples, terminal 120 may include, but is not limited to, a laptop, tablet, or desktop computer. In some examples, acquisition device 130 may be a camera. The camera may be, for example, a handheld fundus camera or a desktop fundus camera. In some examples, acquisition device 130 may be connected to terminal 120 via a serial port. In some examples, acquisition device 130 may be integrated into terminal 120.

[0030] In some examples, server 150 may include one or more processors and one or more memories. The processor may include a central processing unit, a graphics processing unit, and any other electronic components capable of processing data and executing computer program instructions. The memory may be used to store the computer program instructions. In some examples, server 150 can implement a video viewing method by executing computer program instructions stored in memory. In some examples, server 150 may also be a cloud server.

[0031] The reading system 200 related to the present disclosure will be described in detail below in combination with the accompanying drawings. The reading system 200 related to the present disclosure is used to implement the reading method described above. Figure 2 is a block diagram illustrating the reading system 200 based on fundus image classification related to an example of the present disclosure.

[0032] In some examples, as shown in Figure 2 the reading system 200 can include an acquisition module 210, a preprocessing module 220, a first classification module 230, a grouping module 240, a first quality control module 250, a second classification module 260, and an arbitration module 270. In some examples, the acquisition module 210 can be configured to acquire fundus images, the preprocessing module 220 can be configured to preprocess the fundus images to obtain preprocessed fundus images, the first classification module 230 can be configured to classify the preprocessed fundus images and obtain a first classification result and a classification result type, the grouping module 240 can group the preprocessed fundus images into negative result images, positive result images, and images to be reclassified, the first quality control module 250 can obtain a final classification result, first images to be arbitrated, and second images to be arbitrated based on the negative result images and the positive result images, the second classification module 260 can obtain a final classification result and third images to be arbitrated based on the images to be reclassified, and the arbitration module 270 can be configured to arbitrate the images to be arbitrated to obtain an arbitration classification result as the final classification result. In this case, the fundus images are divided into negative result images, positive result images, and images to be reclassified, for the negative result images and the positive result images, the negative result images with lower risk are obtained based on the negative predictive rate and the positive result images with higher risk are obtained based on the positive predictive rate and consistency judgment is performed, and for the images to be reclassified, the second classification model is used for further classification, and finally the images to be arbitrated are arbitrated. Thus, the classification accuracy of the reading system 200 can be improved.

[0033] FIG. 3(a) is a schematic diagram illustrating a fundus image related to an example of the present disclosure. FIG. 3(b) is a schematic diagram illustrating a fundus image related to an example of the present disclosure.

[0034] In some examples, the acquisition module 210 can be configured to acquire fundus images. In some examples, the fundus images can be color fundus images. The color fundus images can clearly present the rich fundus information such as optic disc, optic cup, macula, and blood vessels. In addition, the fundus images can be images in RGB mode, CMYK mode, Lab mode, or grayscale mode, etc. In some examples, the fundus images can be acquired by the acquisition device 130. In other examples, the fundus images can be fundus images stored in the terminal 120 or the server 150. As examples of fundus images, for example, FIG. 3(a) and FIG. 3(b) are fundus images of different human eyes 140, respectively.

[0035] In some examples, the preprocessing module 220 can be configured to preprocess the fundus image to obtain a preprocessed fundus image. Specifically, the preprocessing module 220 can obtain the fundus image output by the acquisition module 210 and preprocess the fundus image to obtain the preprocessed fundus image.

[0036] In some examples, the preprocessing module 220 can crop the fundus image. Generally, since the fundus image acquired by the acquisition module 210 can have different image formats or sizes, the fundus image needs to be cropped to convert the fundus image into a fixed standard form. The fixed standard form can refer to the same format and consistent size of the image. For example, in some examples, the size of the preprocessed fundus image can be unified to 256x256, 374x374, 512x512, 768x768, or 1024x1024 pixels.

[0037] In some examples, the preprocessing module 220 can perform normalization processing on the fundus image.

[0038] In some examples, the normalization processing can include coordinate centering, scaling normalization, and the like. In this way, the differences between different fundus images can be overcome, and the performance of the first classification model can be improved. In addition, in some examples, the preprocessing module 220 can include noise reduction, grayscale processing, and the like. In this way, the features of the fundus image can be highlighted.

[0039] In some examples, the fundus image can also not be preprocessed, and the fundus image can be directly classified subsequently. That is, the preprocessing module 220 can not be necessary.

[0040] In some examples, the first classification module 230 can be configured to classify the preprocessed fundus image and obtain a first classification result and a classification result type. In some examples, the first classification module 230 can also obtain the confidence of the first classification result. In some examples, the first classification module 230 can output the first classification result according to the retinopathy grading system used by the UK National Retinopathy Screening Program. In some examples, the first classification result can include at least no retinopathy (R0), background stage (R1), pre-proliferative stage (R2), and proliferative stage (R3). In this case, based on the retinopathy grading system that has been maturely applied, the classification accuracy of the reading system 200 can be further improved. In some examples, the first classification result can also include no diabetic macular edema (M0) and macular edema (M1).

[0041] But examples of the present disclosure are not limited thereto, and in other examples, the first classification result can include at least a negative result and a positive result. In some examples, the first classification module 230 can screen out pre-processed fundus images that cannot be classified (e.g., pre-processed fundus images with too poor picture quality to be classified).

[0042] In some examples, the classification result type can include a type of whether re-classification is needed (e.g., re-classification needed and re-classification not needed). In some examples, the classification result type can be obtained based on the first classification result. In some examples, the classification result type can be set as re-classification needed for pre-processed fundus images with the first classification result of background stage, and set as re-classification not needed for other pre-processed fundus images. In some examples, whether re-classification is needed can be determined based on the confidence of the first classification result to obtain the classification result type. For example, pre-processed fundus images with the first classification result lower than a pre-set confidence (e.g., 40% or 50%) are set as re-classification needed, and other pre-processed fundus images are set as re-classification not needed.

[0043] In some examples, the first classification module 230 can use a machine learning algorithm to classify the fundus images to obtain the first classification result. In some examples, the machine learning algorithm can be at least one of a traditional machine learning algorithm and a deep learning algorithm. In this case, a suitable machine learning algorithm can be selected according to actual needs. In some examples, the first classification model can be established based on the machine learning algorithm.

[0044] Figure 4 is a schematic diagram showing a convolution kernel used in a convolutional neural network of the first classification module 230 involved in examples of the present disclosure.

[0045] In some examples, the first classification model established based on the deep learning algorithm can be a convolutional neural network (CNN). In some examples, the convolutional neural network (CNN) can automatically identify features in the fundus images using a 3x3 convolution kernel (see Figure 4 ). But examples of the present disclosure are not limited thereto, and in other examples, the convolution kernel of the convolutional neural network (CNN) can be a 5x5 convolution kernel, a 2x2 convolution kernel, or a 7x7 convolution kernel, etc. In this case, since the convolutional neural network (CNN) has the characteristic of high efficiency in identifying image features, the performance of the reading system 200 can be effectively improved.

[0046] But examples of the present disclosure are not limited thereto, and in other examples, the machine-learned algorithm of the first classification module 230 can be a traditional machine-learned algorithm. In some examples, the traditional machine-learned algorithm can include, but is not limited to, a linear regression algorithm, a logistic regression algorithm, a decision tree algorithm, a support vector machine algorithm, or a Bayesian algorithm, etc. In this case, the fundus features in the fundus image can be extracted by using the image processing algorithm first, and then the fundus features are input into the first classification model established based on the traditional machine-learned algorithm to realize classification of the fundus image.

[0047] In some examples, the first classification model can include a plurality of sub-classification models. Each sub-classification model can be for each type of diabetic retinopathy. Each sub-classification model can receive the pre-processed fundus image and obtain a sub-classification result. In some examples, the first classification module 230 can obtain the first classification result based on the plurality of sub-classification results. Thus, the first classification result can be obtained based on the plurality of sub-classification models.

[0048] Specifically, different sub-classification models can be established and trained for no retinopathy, background stage, pre-proliferative stage, proliferative stage, non-diabetic macular edema, and macular edema to obtain a sub-classification result (e.g., whether it is a background stage or not) and a confidence level for whether it is a certain type of diabetic retinopathy, and then the first classification result can be obtained according to each sub-classification result and the confidence level. For example, the sub-classification result with the highest confidence level can be obtained as the first classification result.

[0049] In some examples, the grouping module 240 can group the pre-processed fundus image into a negative result image, a positive result image, and a to-be-reclassified image. In some examples, the pre-processed fundus image can be grouped into the negative result image, the positive result image, and the to-be-reclassified image based on the first classification result and the classification result type. In this case, grouping the pre-processed fundus image into the negative result image, the positive result image, and the to-be-reclassified image can facilitate subsequent targeted processing of each type of image. Thus, the classification accuracy of the reading system 200 can be further improved.

[0050] Specifically, the negative result image can include a pre-processed fundus image with a first classification result of no retinopathy and a classification result type of no need for reclassification. The positive result image can include a pre-processed fundus image with a first classification result of pre-proliferative stage or proliferative stage and a classification result type of no need for reclassification. The to-be-reclassified image can include a pre-processed fundus image with a classification result type of need for reclassification. In some examples, the pre-processed fundus image that needs reclassification can include a pre-processed fundus image with a first classification result of background stage and a pre-processed fundus image that cannot be classified.

[0051] In some examples, the first quality control module 250 can obtain the final classification result, the first image to be arbitrated and the second image to be arbitrated based on the negative result image and the positive result image. As shown in FIG. 8, in some examples, the first quality control module 250 can include a negative quality control module 251 and a positive quality control module 252. Figure 2

[0052] In some examples, the negative quality control module 251 can receive the negative result image, and obtain a negative quality control result by using a first quality control model configured based on a preset negative prediction rate for the first confidence threshold. Generally, the higher the negative prediction rate, the more sensitive the first quality control model is to negative results (i.e., no retinopathy), and the more uncertain negative result images will be classified as positive results (i.e., there is a certain type of diabetic retinopathy). Specifically, before the reading system 200 is officially released to the formal environment, the preset negative prediction rate can be set according to requirements (e.g., customer requirements or default requirements). In some examples, the preset negative prediction rate can be 95% to 99%. For example, the preset negative prediction rate can be 95%, 96%, 97%, 98% or 99%, etc.

[0053] In some examples, the first confidence threshold can be configured by using the gold standard data. Thus, the first confidence threshold can be determined. In some examples, the first confidence threshold can be inversely solved based on the gold standard data according to the preset negative prediction rate. In some examples, in the inverse solution, the confidence threshold in the preset range (e.g., the preset range can be 90% to 100%) can be traversed according to the preset step size, and the performance indicators can be solved to obtain a plurality of confidence thresholds and a plurality of sets of performance indicators, and the first confidence threshold can be determined based on the correspondence between the plurality of confidence thresholds and the plurality of sets of performance indicators and the preset negative prediction rate. In this case, the correspondence between the plurality of confidence thresholds and the plurality of sets of performance indicators can be used to conveniently and quickly determine the first confidence threshold. In some examples, the performance indicators can include sensitivity, specificity, positive prediction rate and negative prediction rate.

[0054] ​Specifically, since the sensitivity, specificity, positive predictive rate (the positive predictive rate can be the number of true positives / (the number of true positives+the number of false positives)) and negative predictive rate (the negative predictive rate can be the number of true negatives / (the number of true negatives+the number of false negatives)) and other performance indicators corresponding to each confidence threshold are determined on the gold standard data. In some examples, the confidence thresholds in the preset range can be traversed in a preset step size and the related performance indicators can be solved. For example, the preset step size can be 0.01, 0.001 or 0.0001, etc. In this case, a table recording a group of performance indicators corresponding to each confidence threshold can be created based on the correspondence between the plurality of confidence thresholds and the plurality of groups of performance indicators, and the confidence threshold corresponding to the performance indicator of the preset negative predictive rate in the table is the first confidence threshold.

[0055] But examples of the present disclosure are not limited to this, in other examples, the first confidence threshold corresponding to the preset negative predictive rate can be obtained by continuously adjusting the first confidence threshold and testing based on the preset negative predictive rate. For example, the first initial confidence threshold can be set based on the gold standard data and the negative predictive rate based on the first initial confidence threshold is obtained, if the absolute difference between the negative predictive rate and the preset negative predictive rate is greater than the preset value (for example, 1%, 2% or 3%), the first initial confidence threshold is adjusted and the comparison between the negative predictive rate and the preset negative predictive rate is continued, otherwise the first initial confidence threshold is taken as the first confidence threshold.

[0056] In some examples, the first quality control model can be the same as the first classification model. In other examples, the first quality control model can be a model retrained for negative result images. In some examples, the negative quality control result can include part of the first classification result. For example, the negative quality control result can include no retinopathy. In some examples, the confidence can be the probability that the negative result image belongs to a certain negative quality control result. In some examples, the first confidence threshold can include a positive release threshold, a negative release threshold and a result threshold. In some examples, if the first quality control model includes a plurality of first sub-quality control models for each type of diabetic retinopathy, the first confidence threshold can be a plurality of groups. For example, if there are n first sub-quality control models, n groups of first confidence thresholds are needed. In some examples, the first quality control model can output a negative quality control result according to the result threshold. In some examples, the negative quality control result whose first quality control model confidence is between the negative release threshold and the positive release threshold can be arbitrated.

[0057] In some examples, the negative quality control module 251 can obtain a final classification result based on the negative quality control result or take the negative result image as the first image to be arbitrated. In some examples, if the negative quality control result of the negative result image is consistent with the first classification result, the negative quality control result is taken as the final classification result of the negative result image, otherwise the negative result image is taken as the first image to be arbitrated. In this case, by setting a preset negative prediction rate, a negative result image with lower risk can be distinguished and compared with the first classification result.

[0058] In some examples, the first quality control module 250 can include a positive quality control module 252. In some examples, the positive quality control module 252 can receive a positive result image and obtain a positive quality control result by using a second quality control model configured based on a preset positive prediction rate for the second confidence threshold. Generally speaking, the higher the positive prediction rate, the more sensitive the second quality control model is to positive results (i.e., the presence of a certain type of diabetic retinopathy), and uncertain positive result images will be classified as negative results (i.e., no retinopathy). Specifically, before the reading system 200 is officially released to the formal environment, the preset positive prediction rate can be set according to requirements (e.g., customer requirements or default requirements). In some examples, the preset positive prediction rate can be 95% to 99%. For example, the preset positive prediction rate can be 95%, 96%, 97%, 98%, or 99%, etc.

[0059] In some examples, the second confidence threshold can be configured by using gold standard data. Thus, the second confidence threshold can be determined. In some examples, the second confidence threshold can be inversely solved based on the preset positive prediction rate according to the gold standard data. In some examples, the second confidence threshold can be determined based on the correspondence between the plurality of confidence thresholds and the plurality of sets of performance indicators and the preset positive prediction rate. In this case, the correspondence between the plurality of confidence thresholds and the plurality of sets of performance indicators can be used to conveniently and quickly determine the second confidence threshold. For specific content, refer to the description of the inverse solution of the first confidence threshold.

[0060] However, examples of the present disclosure are not limited thereto, and in other examples, the second confidence threshold can be continuously adjusted and tested to obtain the second confidence threshold corresponding to the preset positive prediction rate based on the preset positive prediction rate.

[0061] In some examples, the second quality control model can be the same as the first classification model. In other examples, the second quality control model can be a model retrained for positive result images. In some examples, the positive quality control results can include partial first classification results. For example, the positive quality control results can include pre-proliferative and proliferative. In some examples, the confidence can be a probability that a positive result image belongs to a certain positive quality control result. In some examples, the second confidence threshold can include a positive release threshold, a negative release threshold, and a result threshold. Details can refer to the description of the first confidence threshold.

[0062] In some examples, the positive quality control module 252 can obtain a final classification result based on the positive quality control result or take the positive result image as a second image to be arbitrated. In some examples, if the positive quality control result of the positive result image is consistent with the first classification result, the positive quality control result is taken as the final classification result of the positive result image, otherwise the positive result image is taken as the second image to be arbitrated. In this case, by setting a preset positive prediction rate, positive result images with higher risk can be distinguished and compared with the first classification result.

[0063] As described above, the reading system 200 can include a second classification module 260 (see Figure 2 ). In some examples, the second classification module 260 can obtain a final classification result and a third image to be arbitrated based on the image to be reclassified.

[0064] In some examples, the second classification module 260 can receive the image to be reclassified, and classify the image to be reclassified using a second classification model to obtain a second classification result. The second classification model can be obtained based on deep learning and trained for the image to be reclassified. In some examples, the second classification result can include partial first classification results. For example, the second classification result can include no retinopathy, background, pre-proliferative, and proliferative. In some examples, when training for the image to be reclassified, features related to such images as the image to be reclassified can be extracted and used to train the second classification model together with the image to be reclassified. In some examples, the related features can include microaneurysms, hemorrhages, exudates, cotton wool spots, neovascularization, or maculopathy. In some examples, the related features can include health conditions, age, and medical history records. In some examples, the second classification model can also be trained in combination with color features, texture features, and shape features of the image to be reclassified.

[0065] In some examples, the second classification module 260 can obtain a final classification result based on the second classification result or re-classify the image as a third image to be arbitrated. In some examples, if the second classification result of the image to be re-classified is consistent with the first classification result, the second classification result is taken as the final classification result of the image to be re-classified, otherwise the image to be re-classified is taken as a third image to be arbitrated.

[0066] In some examples, the arbitration module 270 can be configured to arbitrate the image to be arbitrated to obtain an arbitration classification result. In some examples, the arbitration classification result can be taken as the final classification result. In some examples, the image to be arbitrated can be the first image to be arbitrated, the second image to be arbitrated or the third image to be arbitrated. In some examples, the arbitration classification result can be consistent with the first classification result. In some examples, the image to be arbitrated can be judged by an arbitration doctor to obtain the arbitration classification result.

[0067] Figure 5 is a block diagram illustrating a reading system 200 for fundus image classification according to examples of the present disclosure.

[0068] In some examples, as shown in Figure 5 the reading system 200 further comprises a self-checking module 280. In some examples, the self-checking module 280 can be configured to perform sampling inspection on the fundus images of the negative quality control results to determine whether the first confidence threshold meets the requirements. In some examples, the self-checking module 280 can be configured to perform sampling inspection on the fundus images of the positive quality control results to determine whether the second confidence threshold meets the requirements. In some examples, the sampling inspection can be performed by using a sampling method. For example, the sampling inspection can be performed by using random sampling. In some examples, the degree of sampling inspection for the newly released reading system 200 can be more stringent (e.g., increasing the sampling ratio). In this case, it can be further determined whether the first confidence threshold and the second confidence threshold meet the requirements. Thus, the classification accuracy of the reading system 200 can be improved.

[0069] In some examples, as shown in Figure 5 the reading system 200 further comprises an output module 290. In some examples, the output module 290 can be configured to output a result report. In some examples, the output module 290 can determine at least one of the first classification result, the negative quality control result, the positive quality control result, the second classification result, the arbitration classification result, and the final classification result to output a result report of the fundus image. In some examples, the result report can comprise the confidence of each result.

[0070] In the following, examples of the present disclosure will be described in detail with reference to the accompanying drawings. Figure 6The method for reading film based on fundus image classification of the present disclosure is described in detail. The method for reading film based on fundus image classification involved in the present disclosure can be referred to as the method for reading film for short sometimes. The method for reading film is applied in the reading film system 200 described above. Figure 6 is a flow chart showing the method for reading film based on fundus image classification involved in the examples of the present disclosure.

[0071] In some examples, as shown in Figure 6 The method for reading film can include acquiring a fundus image (step S110), pre-processing the fundus image to obtain a pre-processed fundus image (step S120), classifying the pre-processed fundus image and obtaining a first classification result and a classification result type (step S130), dividing the pre-processed fundus image into negative result images, positive result images and images to be re-classified (step S140), obtaining a final classification result, a first image to be arbitrated and a second image to be arbitrated based on the negative result images and the positive result images (step S150), obtaining a final classification result and a third image to be arbitrated based on the images to be re-classified (step S160), arbitrating the images to be arbitrated to obtain an arbitrated classification result as the final classification result (step S170). In this case, the fundus image is divided into negative result images, positive result images and images to be re-classified, for the negative result images and the positive result images, the negative result images with lower risk are obtained based on the negative predictive rate and the positive result images with higher risk are obtained based on the positive predictive rate and consistency judgment is performed, and for the images to be re-classified, the second classification model is used for further classification, and finally the images to be arbitrated are arbitrated. Thus, the classification accuracy can be improved.

[0072] In some examples, in step S110, the fundus image can be acquired. The fundus image can be a color fundus image. The color fundus image can clearly present the rich fundus information such as optic disc, optic cup, macula and blood vessels. The specific description can be referred to the related description of the acquisition module 210 in the reading film system 200.

[0073] In some examples, in step S120, the fundus image can be pre-processed to obtain a pre-processed fundus image. In some examples, the fundus image can be processed by cutting, normalization, noise reduction, grayscale and the like. The specific description can be referred to the related description of the pre-processing module 220 in the reading film system 200.

[0074] In some examples, in step S130, the pre-processed fundus images can be classified by using the first classification model based on deep learning to obtain a first classification result. In some examples, a classification result type can be obtained based on the first classification result. In some examples, the classification result type includes a type of whether re-classification is needed. In some examples, the first classification result can be output according to the retinopathy grading system used by the UK National Retinopathy Screening Program. In some examples, the first classification result can at least include no retinopathy (R0), background stage (R1), pre-proliferative stage (R2), and proliferative stage (R3). In this case, based on the retinopathy grading system that has been maturely applied, the classification accuracy can be further improved. In some examples, pre-processed fundus images that cannot be classified (for example, pre-processed fundus images with too poor picture quality to be classified) can be obtained. In some examples, the first classification model can include a plurality of sub-classification models. Each sub-classification model can be for each type of diabetic retinopathy. Each sub-classification model can receive the pre-processed fundus images and obtain a sub-classification result. In some examples, the first classification module 230 can obtain the first classification result based on a plurality of sub-classification results. In this way, the first classification result can be obtained based on a plurality of sub-classification models. For specific description, please refer to the related description of the first classification module 230 in the reading system 200.

[0075] In some examples, in step S140, the pre-processed fundus images can be divided into negative result images, positive result images, and images to be re-classified based on the first classification result and the classification result type. In this case, by dividing the pre-processed fundus images into negative result images, positive result images, and images to be re-classified, subsequent targeted processing of each type of image can be facilitated. In this way, the classification accuracy can be further improved. Specifically, the negative result images can include pre-processed fundus images with the first classification result of no retinopathy and the classification result type of no need for re-classification. The positive result images can include pre-processed fundus images with the first classification result of pre-proliferative stage or proliferative stage and the classification result type of no need for re-classification. The images to be re-classified can include pre-processed fundus images with the classification result type of need for re-classification. In some examples, the pre-processed fundus images that need re-classification can include pre-processed fundus images with the first classification result of background stage and pre-processed fundus images that cannot be classified. For specific description, please refer to the related description of the grouping module 240 in the reading system 200.

[0076] In some examples, in step S150, a negative quality control result of the negative result image can be obtained by using the first quality control model. In some examples, the first quality control model can be configured based on a preset negative prediction rate to the first confidence threshold. In some examples, if the negative quality control result is consistent with the first classification result, the negative quality control result is taken as the final classification result, otherwise the negative result image is taken as the first to-be-arbitrated image. In some examples, a positive quality control result of the positive result image can be obtained by using the second quality control model. The first quality control model can be configured based on a preset positive prediction rate to the second confidence threshold. In some examples, if the positive quality control result is consistent with the first classification result, the positive quality control result is taken as the final classification result, otherwise the positive result image is taken as the second to-be-arbitrated image. In some examples, the first confidence threshold can be configured by using the gold standard data (i.e., the first confidence threshold is continuously adjusted by using the gold standard data and finally determined). Thus, the first confidence threshold can be determined. In some examples, the second confidence threshold can be configured by using the gold standard data (i.e., the second confidence threshold is continuously adjusted by using the gold standard data and finally determined). Thus, the second confidence threshold can be determined. For specific description, please refer to the related description of the first quality control module 250 in the reading system 200.

[0077] In some examples, in step S160, the to-be-reclassified image can be classified by using the second classification model based on deep learning to obtain a second classification result. In some examples, the second classification model can be trained for the to-be-reclassified image. In some examples, if the second classification result is consistent with the first classification result, the second classification result is taken as the final classification result, otherwise the to-be-reclassified image is taken as the third to-be-arbitrated image. For specific description, please refer to the related description of the second classification module 260 in the reading system 200.

[0078] In some examples, in step S170, the to-be-arbitrated image can be arbitrated to obtain an arbitration classification result and taken as the final classification result. In some examples, the to-be-arbitrated image can be the first to-be-arbitrated image, the second to-be-arbitrated image or the third to-be-arbitrated image. For specific description, please refer to the related description of the arbitration module 270 in the reading system 200.

[0079] In some examples, the reading method further comprises a self-checking step (not shown). In some examples, in the self-checking step, the fundus images of the negative quality control results can be sampled to determine whether the first confidence threshold meets the requirements. In some examples, the fundus images of the positive quality control results are sampled to determine whether the second confidence threshold meets the requirements. In this case, it can be further confirmed whether the first confidence threshold and the second confidence threshold meet the requirements. Thus, the classification accuracy of the reading system 200 can be improved. For specific description, please refer to the related description of the self-checking module 280 in the reading system 200.

[0080] In some examples, the reading method further comprises an output step. In some examples, the output step can be used to output a result report. For specific description, please refer to the related description of the output module 290 in the reading system 200.

[0081] Although the present disclosure is specifically described above in combination with the drawings and embodiments, it should be understood that the above description does not limit the present disclosure in any form. Those skilled in the art can modify and change the present disclosure as needed without departing from the spirit and scope of the present disclosure, and such modifications and changes all fall within the scope of the present disclosure.

Claims

1. A reading system based on fundus image classification, characterized in that, Comprising: an acquisition module configured to acquire an eye fundus image; a first classification module configured to classify the eye fundus image using a first classification model to obtain a first classification result and obtain a classification result type including whether re-classification is needed based on the first classification result; a grouping module configured to group the eye fundus image into a negative result image, a positive result image, and a to-be-reclassified image based on the first classification result and the classification result type; a first quality control module including a negative quality control module and a positive quality control module, the negative quality control module configured to obtain a negative quality control result of the negative result image using a first quality control model configured with a first confidence threshold based on a preset negative predictive rate, obtain a final classification result based on the negative quality control result or take the negative result image as a first to-be-arbitrated image, the positive quality control module configured to obtain a positive quality control result of the positive result image using a second quality control model configured with a second confidence threshold based on a preset positive predictive rate, obtain the final classification result based on the positive quality control result or take the positive result image as a second to-be-arbitrated image; a second classification module configured to classify the to-be-reclassified image using a second classification model trained for the to-be-reclassified image to obtain a second classification result, obtain the final classification result based on the second classification result or take the to-be-reclassified image as a third to-be-arbitrated image; and an arbitration module configured to receive the first to-be-arbitrated image, the second to-be-arbitrated image, or the third to-be-arbitrated image as a to-be-arbitrated image, arbitrate the to-be-arbitrated image to obtain an arbitration classification result as the final classification result.

2. The reading system of claim 1, wherein: for the negative result image, if the negative quality control result is consistent with the first classification result, the negative quality control result is taken as the final classification result, otherwise the negative result image is taken as the first to-be-arbitrated image; for the positive result image, if the positive quality control result is consistent with the first classification result, the positive quality control result is taken as the final classification result, otherwise the positive result image is taken as the second to-be-arbitrated image; for the to-be-reclassified image, if the second classification result is consistent with the first classification result, the second classification result is taken as the final classification result, otherwise the to-be-reclassified image is taken as the third to-be-arbitrated image.

3. The reading system of claim 1, wherein: the first confidence threshold is inversely solved based on the preset negative predictive rate according to gold standard data, and the second confidence threshold is inversely solved based on the preset positive predictive rate according to gold standard data.

4. The reading system of claim 3, wherein: ​ The reverse solving is to traverse the confidence threshold in the preset range according to a preset step size and solve the performance indicators to obtain a plurality of confidence thresholds and a plurality of sets of performance indicators, determine the first confidence threshold based on the correspondence between the plurality of confidence thresholds and the plurality of sets of performance indicators and the preset negative prediction rate, and determine the second confidence threshold based on the correspondence between the plurality of confidence thresholds and the plurality of sets of performance indicators and the preset positive prediction rate.

5. The reading system of claim 4, wherein: The performance indicators include sensitivity, specificity, positive prediction rate, and negative prediction rate.

6. The reading system of claim 1, wherein: The first classification result includes no retinopathy, background stage, pre-proliferative stage, and proliferative stage; The negative result image includes the fundus image whose first classification result is no retinopathy and whose classification result type is no need for re-classification; The positive result image includes the fundus image whose first classification result is pre-proliferative stage or proliferative stage and whose classification result type is no need for re-classification; The image to be re-classified includes the fundus image whose classification result type is need for re-classification.

7. The reading system of claim 1, wherein: The preset negative prediction rate is 95% to 99%, and the preset positive prediction rate is 95% to 99%.

8. The reading system of claim 1, wherein: The reading system further includes a self-checking module configured to perform sampling inspection on the fundus images of the negative quality control result to determine whether the first confidence threshold meets the requirements and perform sampling inspection on the fundus images of the positive quality control result to determine whether the second confidence threshold meets the requirements.

9. The reading system of claim 1, wherein: The first classification model is established based on an algorithm of machine learning, and the algorithm of machine learning is at least one of an algorithm of traditional machine learning and an algorithm of deep learning.

10. A method of reading a film based on fundus image classification, characterized by, including: obtaining a fundus image; classifying the fundus image by using a first classification model to obtain a first classification result and a classification result type including whether re-classification is needed based on the first classification result; dividing the fundus image into a negative result image, a positive result image, and an image to be re-classified based on the first classification result and the classification result type; obtaining a negative quality control result of the negative result image by using a first quality control model configured with a first confidence threshold based on a preset negative prediction rate, obtaining a final classification result based on the negative quality control result or taking the negative result image as a first image to be arbitrated, obtaining a positive quality control result of the positive result image by using a second quality control model configured with a second confidence threshold based on a preset positive prediction rate, and obtaining the final classification result based on the positive quality control result or taking the positive result image as a second image to be arbitrated; classifying the image to be reclassified by using a second classification model trained for the image to be reclassified to obtain a second classification result, and obtaining the final classification result based on the second classification result or taking the image to be reclassified as a third image to be arbitrated; and taking the first image to be arbitrated, the second image to be arbitrated, or the third image to be arbitrated as an image to be arbitrated, arbitrating the image to be arbitrated to obtain an arbitration classification result as the final classification result.

Images