Evaluation system for diabetic retinopathy and quality control grading module thereof
Through the combination of deep learning model and multiple judgment modules, fundus images are graded and sampled, which solves the problem of insufficient evaluation accuracy of diabetic retinopathy screening system in the prior art, and achieves higher evaluation accuracy and system reliability.
Patent Information
- Application Number
- CN202411102483.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-24
- Publication Date
- 2024-12-13
AI Technical Summary
The existing diabetic retinopathy screening system based on fundus images has insufficient assessment accuracy, and may output incorrect or inaccurate test reports, resulting in a decrease in assessment accuracy.
The judgment model based on deep learning and multiple sub-judgment models are used to perform primary quality control and advanced quality control grading of fundus images, and multiple judgments and confirmations are made through the sampling monitoring module and the third judgment module to improve the evaluation accuracy.
Through the combination of deep learning model and multiple judgment module, the evaluation accuracy of diabetic retinopathy in fundus images is improved, the error rate is reduced, and the reliability of the system is further improved through sampling monitoring and recall modules.
Smart Images
Figure CN119132522A_ABST
Abstract
Description
[0001] This application is a divisional application of a patent application with an application date of March 24, 2021, application number 2021103163458, and invention name "Evaluation system and evaluation method for diabetic retinopathy based on fundus images." Technical Field
[0002] The present disclosure generally relates to a diabetic retinopathy evaluation system and a quality control grading module thereof. Background Art
[0003] Medical images often contain many details of body structures or tissues. In modern hospitals, most treatment information comes from medical images such as fundus images. In clinical practice, understanding these details in medical images can help doctors identify related diseases. Medical images have developed into the main method for clinical disease identification. However, traditional disease information identification based on medical images mainly relies on professional physicians to make judgments based on experience. In this case, developing a system that can assist doctors in identifying related diseases has become a hot topic in the field of medical imaging. With the development of artificial intelligence technology, disease identification systems based on computer vision and artificial intelligence such as machine learning have been developed and applied in medical image recognition.
[0004] For example, patent document 1 (CN105513077A) discloses a system for screening diabetic retinopathy, which includes: a fundus image acquisition device, an image processing and screening device, and a report output device. The fundus image acquisition device is used to collect or receive fundus images of the person being examined; the image processing and screening device is used to process the fundus image and detect whether there are lesions therein, and then transmit the detection results to the report output device; the report output device outputs a corresponding detection report based on the detection results.
[0005] However, in actual clinical applications, due to the diversity of fundus images, the screening system described in Patent Document 1 may output erroneous or inaccurate detection reports when evaluating certain fundus images, resulting in a decrease in the evaluation accuracy of the screening system. Summary of the invention
[0006] The present disclosure is proposed in view of the above-mentioned situation, and its purpose is to provide a diabetic retinopathy evaluation system and evaluation method based on fundus images that can improve the evaluation accuracy of diabetic retinopathy in fundus images.
[0007] To this end, the first aspect of the present disclosure provides an evaluation system for diabetic retinopathy based on fundus images, including an acquisition module, a first judgment module, a quality control grading module, a first sampling module, a second judgment module, a sampling monitoring module, and a third judgment module; the acquisition module is used to acquire fundus images; the first judgment module judges whether the fundus image from the acquisition module has diabetic retinopathy based on a deep learning judgment model and outputs a first judgment result, the judgment model includes a plurality of sub-judgment models for receiving the fundus image and acquiring sub-judgment results for each stage of diabetic retinopathy, each of the sub-judgment results includes a negative result and a confidence level of the negative result, or a positive result and a confidence level of the positive result, by comparing the confidence level of the negative result and the confidence level of the positive result in the sub-judgment results to obtain a result with a high confidence level and use it as a sub-model result of the sub-judgment model, the first judgment result includes the sub-model result and the confidence level corresponding to the sub-model result; the quality control grading module divides the fundus image into a primary quality control image or a high quality control image based on the first judgment result of the fundus image The primary quality control image is a fundus image of a sub-model result whose confidence level is less than a preset confidence level and does not exist in the first judgment result, and the advanced quality control image is a fundus image of a sub-model result whose confidence level is less than the preset confidence level and does not exist in the first judgment result; the first sampling module is used to sample the primary quality control image and use the sampled primary quality control image as a sampled fundus image; the second judgment module is used to use the advanced quality control image and / or the sampled fundus image as a quality control image and judge the quality control image to obtain a second judgment result; the sampling monitoring module is used to obtain a second judgment result of the primary quality control image within a preset time period and determine whether the first judgment result of the primary quality control image in at least one dimension is qualified according to at least one dimension. If not qualified, the sampling is stopped, the primary quality control image is used as the quality control image and judged to obtain the second judgment result; and the third judgment module is used to confirm and judge the first judgment result of the first judgment module and the second judgment result of the second judgment module to obtain a final judgment result. In this case, a first judgment result is obtained by using a judgment model based on deep learning, the fundus image is divided based on the confidence in the first judgment result, and different judgment modules are used to make targeted judgments on the divided fundus images and sample the judgment modules. Thus, the accuracy of the assessment of diabetic retinopathy in fundus images can be improved.
[0008] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the sampling monitoring module obtains the second judgment result of the primary quality control image within the preset time period and calculates the sampling error rate according to the at least one dimension, and if the sampling error rate is greater than the preset error rate, the first judgment result of the primary quality control image of the at least one dimension is determined to be unqualified. Thus, it is possible to determine whether the first judgment result is qualified based on the sampling error rate.
[0009] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the stages of diabetic retinopathy include no retinopathy, background stage, pre-proliferative stage and proliferative stage.
[0010] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the dimension includes at least one of the source of the fundus image, the model of the shooting device of the fundus image, the shooting personnel information of the fundus image, the shooting quality of the fundus image, and the judgement personnel information of the fundus image. In this case, the fundus images can be classified based on different dimensions and sampled separately for each category. Thereby, the reliability of sampling can be improved.
[0011] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, in the sampling, the sampling scheme is adjusted according to the dimension or the release time of the evaluation system, thereby improving the reliability of the sampling.
[0012] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the evaluation system further includes a recall module, which is used to perform a recall process on the fundus image of at least one dimension corresponding to the judgment result determined as unqualified by the sampling monitoring module. Thereby, the evaluation accuracy of diabetic retinopathy of the fundus image can be further improved.
[0013] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the first judgment module, the second judgment module and the third judgment module also judge the image quality of the fundus image respectively to obtain the image quality level of the fundus image. In this case, the image quality level of the fundus image is obtained, and the fundus image can be evaluated in combination with the image quality level of the fundus image. Thereby, the evaluation accuracy of diabetic retinopathy of the fundus image can be further improved.
[0014] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the image quality level includes very good, good, average, poor and extremely poor. Thus, the image quality level can be determined.
[0015] In addition, in the evaluation system involved in the first aspect of the present disclosure, optionally, the evaluation system further includes an output module, the output module is used to output a result report, the result report includes user information, medical history, the fundus image, the image quality level of the fundus image, comprehensive evaluation results, comprehensive suggestions, reviewer information and interpretation based on the final judgment result. Thus, the result report can be output.
[0016] A second aspect of the present disclosure provides a method for evaluating diabetic retinopathy based on fundus images, comprising: acquiring a fundus image; judging whether the fundus image has diabetic retinopathy based on a deep learning judgment model and outputting a first judgment result, wherein the judgment model comprises a plurality of sub-judgment models for receiving the fundus image and obtaining sub-judgment results for each stage of diabetic retinopathy, each of the sub-judgment results comprising a negative result and a confidence level of the negative result, or a positive result and a confidence level of the positive result, and obtaining a result with a high confidence level by comparing the confidence level of the negative result with the confidence level of the positive result in the sub-judgment results and using the result as a sub-model result of the sub-judgment model, wherein the first judgment result comprises the sub-model result and the confidence level corresponding to the sub-model result; and dividing the fundus image into a primary quality control image or an advanced quality control image based on the first judgment result of the fundus image, wherein the primary quality control image is the confidence level of the negative result. In the first judgment result, there is no fundus image of a sub-model result corresponding to a sub-model result whose confidence is less than a preset confidence, and the advanced quality control image is a fundus image of a sub-model result corresponding to a sub-model result whose confidence is less than the preset confidence in the first judgment result; the primary quality control image is sampled and the extracted primary quality control image is used as a sampled fundus image; the advanced quality control image and / or the sampled fundus image is used as a quality control image and the quality control image is judged to obtain a second judgment result; the second judgment result of the primary quality control image within a preset time period is obtained and the first judgment result of the primary quality control image in the at least one dimension is determined to be qualified according to at least one dimension, if not qualified, the sampling is stopped, the primary quality control image is used as the quality control image and judged to obtain the second judgment result; and the first judgment result and the second judgment result are confirmed and judged to obtain a final judgment result. In this case, the first judgment result is obtained by using a judgment model based on deep learning, the fundus image is divided based on the confidence in the first judgment result, and the divided fundus image is judged and sampled in a targeted manner. This can improve the accuracy of evaluating diabetic retinopathy in fundus images.
[0017] According to the present disclosure, a system and method for evaluating diabetic retinopathy based on fundus images can be provided, which can improve the evaluation accuracy of diabetic retinopathy in fundus images. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The present disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings, in which:
[0019] Figure 1 is a diagram showing an application scenario of the method for evaluating diabetic retinopathy based on fundus images involved in the example of the present disclosure.
[0020] Figure 2 is a block diagram showing a system for evaluating diabetic retinopathy based on fundus images according to an example of the present disclosure.
[0021] Figure 3 is a schematic diagram showing a fundus image related to the example of the present disclosure.
[0022] Figure 4 It is a schematic diagram showing the division of fundus images by the quality control grading module involved in the example of the present disclosure.
[0023] Figure 5 is a block diagram showing a system for evaluating diabetic retinopathy based on fundus images according to an example of the present disclosure.
[0024] Figure 6 is a flow chart showing a method for evaluating diabetic retinopathy based on fundus images according to an example of the present disclosure. DETAILED DESCRIPTION
[0025] Hereinafter, with reference to the accompanying drawings, the preferred embodiments of the present disclosure are described in detail. In the following description, the same symbols are given to the same parts, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the sizes of the parts to each other or the shapes of the parts, etc. may be different from the actual ones. It should be noted that the terms "including" and "having" in the present disclosure and any variations thereof, such as the processes, methods, systems, products or devices of a series of steps or units included or possessed are not necessarily limited to those steps or units clearly listed, but may include or have other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. All methods described in the present disclosure may be performed in any suitable order, unless otherwise indicated herein or clearly contradictory to the context.
[0026] Figure 1 is a diagram showing an application scenario of the method for evaluating diabetic retinopathy based on fundus images involved in the example of the present disclosure.
[0027] In some examples, the evaluation method (described later) can be applied to Figure 1In the application scenario 100 shown. In the application scenario 100, the operator 110 can control the acquisition device 130 connected to the terminal 120 to acquire the fundus image of the fundus of the human eye 140. After the acquisition device 130 completes the fundus image acquisition, the terminal 120 can submit the fundus image to the server 150 through the computer network. The server 150 can implement the evaluation method by executing the computer program instructions stored on the server 150. Through the evaluation method, the fundus image can be received to evaluate the diabetic retinopathy of the fundus image and generate a result report of the fundus image. The server 150 can return the generated result report of the fundus image to the terminal 120. In some examples, the terminal 120 can display the result report. In other examples, the result report can be stored in the memory of the terminal 120 or the server 150 as an intermediate result. In some examples, the result report can be displayed on the user side for the user to view, and the user can be a patient, for example. In other examples, the fundus image received by the evaluation method can be a fundus image stored in the terminal 120 or the server 150.
[0028] In some examples, the operator 110 may be a professional such as an ophthalmologist. In other examples, the operator 110 may be a trained ordinary person. The content of the training may include, but is not limited to, the operation of the acquisition device 130 and the operation of the terminal 120 involving the evaluation method. In some examples, the terminal 120 may include, but is not limited to, a laptop, a tablet computer, or a desktop computer. In some examples, the acquisition device 130 may be a camera. The camera may be, for example, a handheld fundus camera or a desktop fundus camera. In some examples, the acquisition device 130 may be connected to the terminal 120 via a serial port. In some examples, the acquisition device 130 may be integrated in the terminal 120.
[0029] In some examples, server 150 may include one or more processors and one or more memories. The processor may include a central processing unit, a graphics processing unit, and any other electronic component capable of processing data, and may execute computer program instructions. The memory may be used to store computer program instructions. In some examples, server 150 may implement the evaluation method by executing the computer program instructions on the memory. In some examples, server 150 may also be a cloud server.
[0030] The evaluation system 200 of the present disclosure is described in detail below in conjunction with the accompanying drawings. The evaluation system 200 of the present disclosure is used to implement the above-mentioned evaluation method. Figure 2 is a block diagram showing a system 200 for evaluating diabetic retinopathy based on fundus images according to an example of the present disclosure.
[0031] like Figure 2As shown, the evaluation system 200 may include an acquisition module 210, a first judgment module 220, a quality control grading module 230, a first sampling module 240, a second judgment module 250, a sampling monitoring module 260, and a third judgment module 270. The acquisition module 210 may be used to acquire a fundus image. The first judgment module 220 may judge whether the fundus image has diabetic retinopathy based on a judgment model of deep learning and output a first judgment result. The quality control grading module 230 may classify the fundus image into a primary quality control image or an advanced quality control image based on the first judgment result of the fundus image. The first sampling module 240 may sample the primary quality control image and use the extracted primary quality control image as a sampled fundus image. The second judgment module 250 may be used to use the advanced quality control image and / or the sampled fundus image as a quality control image and judge the quality control image to obtain a second judgment result. The sampling monitoring module 260 may be used to monitor whether the first judgment result of the primary quality control image is qualified. The third judgment module 270 can be used to confirm and judge the judgment results of each judgment module to obtain a final judgment result. In this case, the first judgment result is obtained by using a judgment model based on deep learning, the fundus image is divided based on the confidence in the first judgment result, and different judgment modules are used to make targeted judgments on the divided fundus images and sample the judgment modules. In this way, the accuracy of the assessment of diabetic retinopathy in the fundus image can be improved.
[0032] Figure 3 is a schematic diagram showing a fundus image related to the example of the present disclosure.
[0033] In some examples, the acquisition module 210 can be used to acquire fundus images. In some examples, the fundus images may include fundus images from the user's left eye and right eye. In some examples, the fundus images may be color fundus images. Color fundus images can clearly present rich fundus information such as the optic disc, optic cup, macula, blood vessels, etc. In addition, the fundus image may be an image in RGB mode, CMYK mode, Lab mode, or grayscale mode, etc. In some examples, the fundus image may be acquired by the acquisition device 130. In other examples, the fundus image may be a fundus image stored in the terminal 120 or the server 150. As an example of a fundus image, for example Figure 3 The fundus image of the human eye 140 is shown.
[0034] In some examples, after the acquisition module 210 acquires the fundus image, the fundus image may be preprocessed. In some instances, the preprocessing may include cropping the fundus image. Generally speaking, since the fundus image acquired by the acquisition module 210 may have different image formats or sizes, it is necessary to crop the fundus image to convert the fundus image into an image in a fixed standard form. The fixed standard form may refer to images with the same format and size. For example, in some examples, the size of the fundus image after preprocessing can be unified into a fundus image of 256×256, 374×374, 512×512, 768×768 or 1024×1024 pixels.
[0035] In addition, in some examples, preprocessing may include normalizing the fundus image. In some examples, normalization may include performing operations such as coordinate centering and scaling normalization on the fundus image. Thus, the differences between different fundus images can be overcome and the performance of the evaluation system 200 can be improved. In addition, in some examples, preprocessing may include performing noise reduction, grayscale processing, etc. on the fundus image. Thus, the features of the fundus image can be highlighted. In some examples, the fundus image may not need to be preprocessed, and the fundus image may be directly judged later.
[0036] In some examples, the first determination module 220 may determine whether the fundus image from the acquisition module 210 has diabetic retinopathy and output a first determination result.
[0037] In some examples, the first judgment module 220 can output a first judgment result according to the retinopathy grading system used by the British National Retinopathy Screening Program. Specifically, the fundus image can be processed to determine the stage of diabetic retinopathy to which the fundus image belongs and the corresponding confidence. The stage of diabetic retinopathy can include no retinopathy (R0), background stage (R1), pre-proliferation stage (R2) and proliferative stage (R3). The stage of no retinopathy (R0) can be set as a negative result, and the stage of background stage (R1), pre-proliferation stage (R2) and proliferative stage (R3) can be set as a positive result. In this case, based on the mature retinopathy grading system, the assessment accuracy of diabetic retinopathy of the fundus image can be further improved. In some examples, the stage of diabetic retinopathy can include no diabetic macular edema (M0) and macular edema (M1). The stage of diabetic retinopathy can be set as no retinopathy (R0) and no diabetic macular edema (M0) as a negative result, and the others are set as positive results.
[0038] In some examples, the first judgment module 220 can use a judgment model based on machine learning to judge whether diabetic retinopathy exists in the fundus image. In some examples, the judgment model based on machine learning can be at least one of a judgment model based on traditional machine learning and a judgment model based on deep learning.
[0039] In some examples, the judgment model based on deep learning can be a convolutional neural network (CNN). In this case, since the convolutional neural network (CNN) has the characteristics of high efficiency in image feature recognition, it can effectively improve the performance of the evaluation system 200.
[0040] However, the examples disclosed herein are not limited thereto. In other examples, the judgment model based on machine learning of the first judgment module 220 may be a judgment model based on traditional machine learning. In some examples, the algorithms of traditional machine learning may include but are not limited to linear regression algorithms, logistic regression algorithms, decision tree algorithms, support vector machine algorithms, or Bayesian algorithms, etc. In this case, the fundus features in the fundus image may be first extracted using an image processing algorithm, and then the fundus features may be input into the judgment model based on traditional machine learning to realize the judgment of the fundus image.
[0041] Figure 4 is a schematic diagram showing the division of fundus images by the quality control grading module 230 involved in the example of the present disclosure.
[0042] In some examples, the judgment model may include multiple sub-judgment models for each stage of diabetic retinopathy. Figure 4 As shown, the number of multiple sub-judgment models can be n, specifically, the multiple sub-judgment models can include sub-judgment model 1, sub-judgment model 2, ..., sub-judgment model n. In some examples, each sub-judgment model can be used to receive a fundus image and obtain a sub-judgment result. In some examples, each sub-judgment result can include a negative result and a confidence level of the negative result, or a positive result and a confidence level of the positive result. Among them, the confidence level of the negative result can be the probability of not belonging to a single stage of diabetic retinopathy, and the confidence level of the positive result can be the probability of belonging to a single stage of diabetic retinopathy. Specifically, the fundus image can be input into each sub-judgment model for judgment, and the sub-judgment model can output the probability that the fundus image belongs to the stage of diabetic retinopathy corresponding to the sub-judgment model (that is, a positive result) or the probability that the fundus image does not belong to the stage of diabetic retinopathy corresponding to the sub-judgment model (that is, a negative result).
[0043] In some examples, the confidence of negative results and the confidence of positive results in the sub-judgment results of the fundus image can be compared to obtain a result with high confidence, and the result with high confidence can be used as the sub-model result of the sub-judgment model. For example, if in the sub-judgment results of a fundus image, the confidence of the negative result is 10% and the confidence of the positive result is 90%, then the sub-model result of the sub-judgment model corresponding to the fundus image can be a positive result. In some examples, if the sub-model result is a negative result, it may be that the fundus image does not belong to the stage of diabetic retinopathy corresponding to the sub-judgment model; if the sub-model result is a positive result, it may be that the fundus image belongs to the stage of diabetic retinopathy corresponding to the sub-judgment model. In this case, multiple sub-judgment models can obtain multiple sub-model results and the confidence corresponding to each sub-model result. Figure 4 As shown, the number of multiple sub-model results can be n. Specifically, the multiple sub-model results and the confidence levels corresponding to each sub-model result can include sub-model result 1 and the confidence level corresponding to sub-model result 1, sub-model result 2 and the confidence level corresponding to sub-model result 2, ..., sub-model result n and the confidence level corresponding to sub-model result n.
[0044] In some examples, the first judgment result may include the sub-model result and the confidence level corresponding to the sub-model result (see Figure 4 ). In some examples, the number of sub-model results in the first judgment result may be multiple. In some examples, the stage of diabetic retinopathy to which the fundus image belongs can be determined based on multiple sub-model results and the confidences corresponding to the sub-model results. For example, the sub-model result with a positive result and the highest confidence of the positive result in multiple sub-judgment models can be obtained and the sub-model result with the highest confidence can be used as the stage of diabetic retinopathy to which the fundus image belongs. In some examples, the first judgment result may include the stage of diabetic retinopathy to which the fundus image belongs.
[0045] In some examples, the first judgment module 220 can also be used to judge the image quality of the fundus image to obtain the image quality level of the fundus image. For example, the image quality grading model based on machine learning can be trained using fundus images of different image quality levels so that the image quality grading model has the ability to judge the image quality of the fundus image to obtain the image quality level of the fundus image. In this case, the image quality level of the fundus image is obtained, and the fundus image can be evaluated in combination with the image quality level of the fundus image. Thus, the evaluation accuracy of diabetic retinopathy of the fundus image can be further improved. In some examples, the image quality level of the fundus image includes five levels: very good, good, general, poor and extremely poor. Thus, the image quality level can be determined. In some examples, the first judgment result may include the image quality level of the fundus image. In some examples, the first judgment result may also include the existing lesion characteristics.
[0046] In some examples, the quality control grading module 230 may classify the fundus image into a primary quality control image or an advanced quality control image based on the first judgment result of the fundus image. In some examples, the primary quality control image may be a fundus image for which there is no sub-model result corresponding to a confidence level less than a preset confidence level in the first judgment result. In some examples, the advanced quality control image may be a fundus image for which there is a sub-model result corresponding to a confidence level less than a preset confidence level in the first judgment result. Specifically, Figure 4 As shown, assuming that a fundus image outputs n sub-model results and the confidence corresponding to each sub-model result in n sub-judgment models, the quality control grading module 230 is configured to classify the fundus image as a high-level fundus image if there is a sub-model result whose confidence corresponding to the result is less than the preset confidence, otherwise classify the fundus image as a primary fundus image. In some examples, the preset confidence can be set according to actual conditions. For example, the preset confidence can be 95% to 98%.
[0047] However, the examples disclosed herein are not limited thereto. In other examples, the first judgment result may also be a sub-model result including a positive result among multiple sub-judgment models and a sub-model result with the highest confidence of the positive result. In this case, the primary quality control image may be a fundus image with a confidence corresponding to the positive result greater than a preset confidence, and the advanced quality control image may be a fundus image with a confidence of the positive result less than or equal to the preset confidence. In other examples, the judgment model may also not include multiple sub-judgment models, that is, the stage of diabetic retinopathy to which the fundus image belongs and the confidence corresponding to the stage of diabetic retinopathy to which it belongs can be directly obtained through the judgment model. In this case, the first judgment result may include the stage of diabetic retinopathy to which the fundus image belongs and the confidence corresponding to the stage of diabetic retinopathy to which it belongs, the primary quality control image may be a fundus image with a confidence corresponding to the stage of diabetic retinopathy greater than the preset confidence, and the advanced quality control image may be a fundus image with a confidence corresponding to the stage of diabetic retinopathy less than or equal to the preset confidence.
[0048] In some examples, the first sampling module 240 can sample the primary quality control images and use the extracted primary quality control images as sampled fundus images. Specifically, the first sampling module 240 can sample the primary quality control images according to a preset time (e.g., every hour) or a preset number of images (e.g., 100 images). In some examples, the extracted primary quality control images can be marked to distinguish the extracted primary quality control images and used as sampled fundus images. In some examples, the unextracted primary quality control images can be input into the third judgment module 270 (described later).
[0049] In some examples, the primary fundus images can be sampled based on a sampling scheme. In some examples, the sampling scheme can include uniformly sampling the primary fundus images. Specifically, a sampling method can be performed on all primary fundus images. In some examples, the sampling method can include, but is not limited to, random sampling, stratified sampling, overall sampling, or systematic sampling. In some examples, the sampling method can be based on national standards for sampling inspection. In some examples, national standards for sampling inspection can include, but are not limited to, GB / T2828.1-2012, GB / T2828.5-2011, or GB / T8052-2002.
[0050] In addition, in some examples, the sampling scheme may include classifying the primary fundus images according to one or more different dimensions, and sampling separately for each category (for example, several dimensions can be used as a category). In this case, the primary fundus images can be classified based on different dimensions and sampled separately for each category. Thereby, the reliability of sampling can be improved. In some examples, the dimensions may include at least one of the source of the fundus image, the model of the shooting device of the fundus image, the shooting personnel information of the fundus image, the shooting quality of the fundus image, and the judgement personnel information of the fundus image. In some examples, the judgment model based on machine learning to judge whether diabetic retinopathy exists in the fundus image can be regarded as a judgment person, and if there are multiple judgment models, multiple judgment persons can correspond to them.
[0051] In some examples, in sampling, the sampling scheme can be adjusted according to the dimension or the release time of the evaluation system 200. In some examples, the sampling scheme can have different degrees of sampling. In some examples, the degree of sampling can be general, special, strict or relaxed, etc. In some examples, different degrees of sampling can correspond to different sampling methods or different sampling parameters. In some examples, the sampling parameters can include but are not limited to sampling ratios or AQL (Acceptable Quality Limit, receiving quality limit) values, etc.
[0052] In some examples, a preliminary sampling ratio may be obtained, and the preliminary sampling ratio may be adjusted based on the degree of sampling to determine the sampling ratio. For example, if the degree of sampling is relaxed, the preliminary sampling ratio may be reduced to determine the sampling ratio.
[0053] Generally speaking, AQL refers to the worst average process quality level allowed when a series of batches are submitted for acceptance sampling. If the quality level is expressed as a percentage of unqualified products, the AQL value should not exceed 10%. If the quality level is expressed as the number of unqualified products per 100 units of fundus images to be sampled, the AQL value should not exceed 10. Different degrees of sampling can correspond to different AQL values.
[0054] In some examples, in sampling, the sampling scheme can be adjusted according to the release time of the evaluation system 200. For example, the sampling degree of the newly released evaluation system 200 can be tightened, and after a period of operation, it is determined that the accuracy of the evaluation system 200 meets the requirements, the sampling degree can be adjusted to be relaxed. In some examples, in sampling, the sampling scheme can be adjusted according to the dimension. For example, the sampling degree of the primary fundus image judged by the judge with lower seniority can be tightened.
[0055] In some examples, the second judgment module 250 can be used to judge the quality control image to obtain a second judgment result. The second judgment result can include the stage of diabetic retinopathy. In some examples, the quality control image can be an advanced quality control image from the first judgment module 220 and / or a sampled fundus image from the first sampling module 240. In some examples, the second judgment module 250 can also judge the image quality of the fundus image to obtain the image quality grade of the fundus image. In some examples, the second judgment result can also include the image quality grade of the fundus image. In some examples, the second judgment result can also include the existing lesion characteristics.
[0056] Specifically, the second judgment module 250 can obtain and display basic information of the quality control image, and the basic information may include but is not limited to eye type (e.g., left eye or right eye), creation time, affiliated medical institution, shooting model, and current judgment module (e.g., second judgment module 250), etc. After the judgment personnel judge the quality control image based on the basic information and generate judgment information, the second judgment module 250 can receive the judgment information of the judgment personnel and generate a second judgment result of the quality control image based on the judgment information. In some examples, the basic information may also include medical history, vision information, gender, and age.
[0057] In some examples, the judgment information may include, but is not limited to, the image quality level of the fundus image, the stage of diabetic retinopathy or the existing lesion characteristics, etc.
[0058] As described above, the second judgment module 250 can be used to judge the quality control image to obtain a second judgment result. The quality control image can be an advanced quality control image from the first judgment module 220 and / or a sampled fundus image from the first sampling module 240. Therefore, the sampled fundus image has a second judgment result (that is, the extracted primary quality control image has a second judgment result).
[0059] In some examples, the sampling monitoring module 260 can obtain the second judgment result of the primary quality control image within a preset time period (e.g., the same day, morning, or afternoon), and determine whether the first judgment result of the primary quality control image in at least one dimension is qualified according to at least one dimension (e.g., the judging personnel). In some examples, if it is unqualified, the first sampling module 240 is stopped and the primary quality control image is input to the second judging module 250 for judgment, that is, the sampling of the primary quality control image is stopped, and the primary quality control image is used as the quality control image and judged to obtain the second judgment result.
[0060] However, the examples of the present disclosure are not limited thereto. In other examples, the sampling monitoring module 260 may obtain the second judgment results of the preset number of primary quality control images to determine whether the first judgment result of the primary quality control image of at least one dimension is qualified. For example, the sampling monitoring module 260 may monitor the primary quality control images in the evaluation system 200. When the number of primary quality control images in the evaluation system 200 reaches a preset number, such as 500, the second judgment results of the preset number of primary quality control images are obtained according to at least one dimension to determine the first judgment result of the primary quality control image of at least one dimension.
[0061] In some examples, the sampling monitoring module 260 can obtain the second judgment result of the primary quality control image within a preset time period, and calculate the sampling error rate according to at least one dimension. If the sampling error rate is greater than the preset error rate, the first judgment result of the primary quality control image in at least one dimension is determined to be unqualified. Thus, it is possible to determine whether the first judgment result is qualified based on the sampling error rate. In some examples, the preset error rate can be set according to actual conditions.
[0062] In some examples, if the first judgment result of the primary quality control image is inconsistent with the second judgment result, the first judgment result of the primary quality control image is an erroneous judgment result. In some examples, whether the first judgment result is consistent with the second judgment result can be determined based on the image quality level of the fundus image or the stage of diabetic retinopathy to which it belongs.
[0063] As described above, the sampling monitoring module 260 can obtain the second judgment result of the primary quality control image within a preset time period, and calculate the sampling error rate according to at least one dimension. In some examples, the sampling error rate can be the number of sampling errors divided by the total number of samples. The sampling monitoring module 260 can obtain the number of primary quality control images (e.g., primary quality control images of at least one dimension) used to calculate the sampling error rate, the total number of samples, and the number of sampling errors. The total number of samples can be the number of primary fundus images that have been extracted by the first sampling module 240 in the primary quality control image used to calculate the sampling error rate. The number of sampling errors can be the number of primary quality control images corresponding to the erroneous judgment results in the primary quality control image used to calculate the sampling error rate.
[0064] In some examples, the sampling scheme may be adjusted based on the sampling error rate. For example, if the sampling error rate corresponding to a certain dimension is low, the degree of sampling corresponding to the sampling scheme may be adjusted to be relaxed.
[0065] In some examples, the third judgment module 270 can be used to confirm and judge the first judgment result of the first judgment module 220 and the second judgment result of the second judgment module 250 to obtain a final judgment result. In some examples, the third judgment module 270 can also confirm and judge the first judgment result of the primary quality control image that has not been extracted to obtain a final judgment result.
[0066] In some examples, the third judgment module 270 can check the judgment results of each module and confirm whether to retain the judgment result of the fundus image in the last judgment module (e.g., the first judgment module 220 or the second judgment module 250) as the final judgment result. If not, a re-judgment is performed to obtain the final judgment result. Thus, the judgment results of each judgment module can be uniformly confirmed. In some examples, the final judgment result may include the stage of diabetic retinopathy to which it belongs. In some examples, the third judgment module 270 may also judge the image quality of the fundus image to obtain the image quality level of the fundus image. In some examples, the final judgment result may also include the image quality level of the fundus image. In some examples, the final judgment result may also include the existing lesion characteristics.
[0067] Figure 5 is a block diagram showing a system 200 for evaluating diabetic retinopathy based on fundus images according to an example of the present disclosure.
[0068] In some examples, the evaluation system 200 may also include an output module 280 (see Figure 5 ), the output module 280 can be used to output a result report. In some examples, the result report can include the final judgment result. In some examples, the result report can include user information, medical history, fundus image, image quality level of the fundus image, comprehensive evaluation results, comprehensive suggestions, reviewer information and interpretation based on the final judgment result. Thus, the result report can be output.
[0069] In some examples, user information may include, but is not limited to, name, gender, and age. In addition, in some examples, medical history may include, but is not limited to, a history of diabetes, a history of hypertension, and a history of eye diseases. In addition, in some examples, the fundus image in the result report may include a fundus image of the left eye and a fundus image of the right eye. In addition, in some examples, the comprehensive evaluation result may include an evaluation of the left eye and an evaluation of the right eye. As an example of a comprehensive evaluation result, for example, the comprehensive evaluation result may be that there is no obvious diabetic retinopathy in the fundus of the left eye, and proliferative diabetic retinopathy is visible in the fundus of the right eye. In addition, in some examples, the reviewer information in the result report may be the information of the judge who confirms and judges each result using the third judgment module 270. In some examples, the interpretation based on the final judgment result may include at least one of a medical explanation, common causes, and guidance suggestions.
[0070] In some examples, the evaluation system 200 also includes a recall module 290 (see Figure 5 ), the recall module 290 can be used to perform recall processing on the fundus images of at least one dimension corresponding to the judgment result (such as the first judgment result mentioned above) determined as unqualified by the sampling monitoring module 260. In some examples, the recalled fundus images can be fully inspected, the sampling ratio of at least one dimension corresponding to the recalled fundus images can be increased, and at least one of the software and hardware of the evaluation system 200 can be repaired. In this way, the evaluation accuracy of diabetic retinopathy in fundus images can be further improved.
[0071] The following, combined Figure 6 The method for evaluating diabetic retinopathy based on fundus images of the present disclosure is described in detail. The method for evaluating diabetic retinopathy based on fundus images of the present disclosure may sometimes be referred to as an evaluation method. The evaluation method is applied to the above-mentioned evaluation system 200. Figure 6 is a flow chart showing a method for evaluating diabetic retinopathy based on fundus images according to an example of the present disclosure.
[0072] In some examples, such as Figure 6As shown, the evaluation method may include acquiring a fundus image (step S110), judging whether the fundus image has diabetic retinopathy based on a judgment model of deep learning and outputting a first judgment result (step S120), dividing the fundus image into a primary quality control image or an advanced quality control image based on the first judgment result of the fundus image (step S130), sampling the primary quality control image and using the extracted primary quality control image as a sampled fundus image (step S140), using the advanced quality control image and / or the sampled fundus image as a quality control image and judging the quality control image to obtain a second judgment result (step S150), monitoring whether the first judgment result of the primary quality control image is qualified (step S160), and confirming and judging each judgment result to obtain a final judgment result (step S170). In this case, the first judgment result is obtained using a judgment model based on deep learning, the fundus image is divided based on the confidence in the first judgment result, and the divided fundus image is judged and sampled in a targeted manner. Thus, the evaluation accuracy of diabetic retinopathy in the fundus image can be improved.
[0073] In some examples, in step S110, a fundus image may be acquired. For a detailed description, see the related description of the acquisition module 210.
[0074] In some examples, in step S120, it is possible to judge whether there is diabetic retinopathy in the fundus image based on the judgment model of deep learning and output a first judgment result. In some examples, the judgment model may include multiple sub-judgment models for receiving fundus images and obtaining sub-judgment results for each stage of diabetic retinopathy. In some examples, each sub-judgment result may include a negative result and a confidence of the negative result, or a positive result and a confidence of the positive result. In some examples, a result with high confidence can be obtained by comparing the confidence of the negative result and the confidence of the positive result in the sub-judgment result, and used as a sub-model result of the sub-judgment model. In some examples, if the sub-model result is a negative result, it may be that there is no stage of diabetic retinopathy corresponding to the sub-judgment model in the fundus image. In some examples, if the sub-model result is a positive result, it may be that there is a stage of diabetic retinopathy corresponding to the sub-judgment model in the fundus image. In some examples, the first judgment result may include a sub-model result and a confidence corresponding to the sub-model result. In some examples, in step S120, the image quality of the fundus image may also be judged to obtain the image quality level of the fundus image. In this case, the image quality level of the fundus image is obtained, and the fundus image can be evaluated in combination with the image quality level of the fundus image later. Thus, the evaluation accuracy of diabetic retinopathy of the fundus image can be further improved. In some examples, the image quality level of the fundus image can be five levels: very good, good, average, poor, and extremely poor. Thus, the image quality level can be determined. For a specific description, see the relevant description of the first judgment module 220. In addition, in some examples, the staging of diabetic retinopathy can include no retinopathy, background period, pre-proliferation period, and proliferative period.
[0075] In some examples, in step S130, the fundus image can be divided into a primary quality control image or an advanced quality control image based on the first judgment result of the fundus image. In some examples, the fundus image can be divided into a primary quality control image or an advanced quality control image based on the first judgment result of the fundus image. In some examples, the primary quality control image can be a fundus image in which there is no sub-model result corresponding to a sub-model result with a confidence level less than a preset confidence level in the first judgment result. In some examples, the advanced quality control image can be a fundus image in which there is a sub-model result corresponding to a sub-model result with a confidence level less than a preset confidence level in the first judgment result. For a specific description, see the relevant description of the quality control grading module 230.
[0076] In some examples, in step S140, the primary quality control image can be sampled and the extracted primary quality control image can be used as a sampled fundus image. In some examples, the primary quality control image that has not been sampled can be processed by step S170 (described later). In some examples, in sampling, the sampling scheme can be adjusted according to the dimension or the release duration of the evaluation system 200. Thus, the reliability of sampling can be improved. In some examples, the dimension may include at least one of the source of the fundus image, the model of the shooting device of the fundus image, the shooting personnel information of the fundus image, the shooting quality of the fundus image, and the judgment personnel information of the fundus image. In this case, the fundus images can be classified based on different dimensions and sampled separately for each category. Thus, the reliability of sampling can be improved. For a specific description, see the relevant description of the first sampling module 240.
[0077] In some examples, in step S150, the advanced quality control image and / or the sample fundus image may be used as the quality control image and the quality control image may be judged to obtain the second judgment result. For a detailed description, see the relevant description of the second judgment module 250.
[0078] In some examples, in step S160, it is possible to monitor whether the first judgment result of the primary quality control image is qualified. In some examples, the second judgment result of the primary quality control image within a preset time period can be obtained, and the first judgment result of the primary quality control image of at least one dimension is determined to be qualified according to at least one dimension. If it is unqualified, the sampling in step S140 is stopped and the primary quality control image is transferred to step S160 for judgment, that is, the sampling of the primary quality control image is stopped, and the primary quality control image is used as a quality control image and judged to obtain the second judgment result. In some examples, it can be determined whether the first judgment result of the primary quality control image of at least one dimension is qualified based on the sampling error rate. Specifically, the sampling monitoring module 260 can obtain the second judgment result of the primary quality control image within a preset time period, and calculate the sampling error rate according to at least one dimension. If the sampling error rate is greater than the preset error rate, it is determined that the first judgment result of the primary quality control image of at least one dimension is unqualified. Therefore, it is possible to determine whether the first judgment result is qualified based on the sampling error rate. For a specific description, see the relevant description of the sampling monitoring module 260.
[0079] In some examples, in step S170, each judgment result may be confirmed and judged to obtain a final judgment result. In some examples, the first judgment result and the second judgment result may be confirmed and judged to obtain a final judgment result. In some examples, in step S170, the image quality level of the fundus image may also be obtained. For a specific description, see the relevant description of the third judgment module 270.
[0080] In some examples, the evaluation method can also output a result report. In some examples, the result report can include a final judgment result. In some examples, the result report can include user information, medical history, fundus image, image quality grade of the fundus image, comprehensive evaluation results, comprehensive suggestions, reviewer information, and interpretation based on the final judgment result. Thus, the result report can be output. For a specific description, see the relevant description of output module 280.
[0081] In some examples, the evaluation method can also perform a recall process on the fundus images of at least one dimension corresponding to the unqualified judgment results (such as the first judgment results described above). In some examples, the recalled fundus images can be fully inspected, the sampling ratio of at least one dimension corresponding to the recalled fundus images can be increased, and at least one of the software and hardware of the evaluation system 200 can be repaired. In this way, the evaluation accuracy of diabetic retinopathy in fundus images can be further improved.
[0082] Although the present disclosure is specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the scope of the present disclosure.
Claims
1. A quality control grading module of a diabetic retinopathy evaluation system, characterized in that: Based on a first judgment result of the fundus image obtained by a first judgment module of the evaluation system, the fundus image is divided into a primary quality control image or an advanced quality control image, wherein the first judgment module is used to judge whether the fundus image has diabetic retinopathy based on a deep learning judgment model and output the first judgment result, the judgment model includes a plurality of sub-judgment models for receiving the fundus image and obtaining sub-model results for each stage of diabetic retinopathy, the first judgment result includes the sub-model result and the confidence corresponding to the sub-model result, the primary quality control image is a fundus image in which the sub-model result corresponding to the sub-model result and the confidence less than a preset confidence does not exist in the first judgment result, and the advanced quality control image is a fundus image in which the sub-model result corresponding to the sub-model result and the confidence less than the preset confidence exists in the first judgment result.
2. The quality control grading module according to claim 1, characterized in that: Each of the sub-judgment models receives the fundus image to obtain each sub-judgment result, and each of the sub-judgment results includes a negative result and the confidence of the negative result, or a positive result and the confidence of the positive result. A result with a high confidence is obtained by comparing the confidence of the negative result with the confidence of the positive result in the sub-judgment results, and the result is used as the sub-model result of the sub-judgment model.
3. The quality control grading module according to claim 2, characterized in that: The confidence level of a negative result of the sub-judgment result is the probability of not belonging to a single stage of diabetic retinopathy, and the confidence level of a positive result of the sub-judgment result is the probability of belonging to a single stage of diabetic retinopathy.
4. The quality control grading module according to claim 1, characterized in that: The preset confidence level is 95% to 98%.
5. The quality control grading module according to any one of claims 1 to 4, characterized in that: The judgment model based on deep learning is a convolutional neural network.
6. A diabetic retinopathy evaluation system, characterized in that: It includes an acquisition module, a first judgment module, a quality control grading module according to any one of claims 1 to 5, a first sampling module, a second judgment module, and a third judgment module; the acquisition module is used to acquire a fundus image; the first judgment module is used to judge whether the fundus image has diabetic retinopathy and output a first judgment result; the quality control grading module is used to divide the fundus image into a primary quality control image or an advanced quality control image based on the first judgment result; the first sampling module is used to sample the primary quality control image and use the sampled primary quality control image as a sampled fundus image; the second judgment module is used to use the advanced quality control image and / or the sampled fundus image as a quality control image and judge the quality control image to obtain a second judgment result; the third judgment module is used to confirm and judge the first judgment result of the first judgment module and the second judgment result of the second judgment module to obtain a final judgment result.
7. The evaluation system according to claim 6, characterized in that: It also includes a sampling monitoring module, which is used to obtain a second judgment result of the primary quality control image within a preset time period and determine whether the first judgment result of the primary quality control image in at least one dimension is qualified according to at least one dimension. If it is unqualified, the sampling is stopped, the primary quality control image is used as the quality control image and a judgment is performed to obtain the second judgment result.
8. The evaluation system according to claim 7, characterized in that In the sampling, the sampling plan is adjusted according to the release time of the dimension or the evaluation system.
9. The evaluation system according to claim 8, characterized in that The sampling plan has different degrees of sampling, and the degree of sampling is general, special, strict or relaxed.
10. The evaluation system according to any one of claims 7 to 9, characterized in that The dimension includes at least one of the source of the fundus image, the model of the device for shooting the fundus image, the information of the person shooting the fundus image, the shooting quality of the fundus image, and the information of the person judging the fundus image.