Method for acquiring quality control eye fundus image of screening system and related method and server

Abstract

The invention discloses a method for acquiring quality control eye fundus images of a screening system and a related method and server. The method for obtaining the quality control eye fundus image comprises the steps of obtaining a preset number of eye fundus images as eye fundus image samples, screening misjudgment samples based on screening information of the eye fundus image samples, and dividing the misjudgment samples based on the screening information of the misjudgment samples, classifying the eye fundus image samples based on the divided classification categories to obtain the classification accuracy of each classification category; obtaining at least one batch of fundus images as target fundus images, and taking screening information corresponding to the target fundus images as target information; and classifying the target fundus images based on the target information and the division categories to obtain the division categories to which the target fundus images belong, and taking the target fundus images belonging to the division categories of which the classification accuracy is lower than a specified value as quality control fundus images. Therefore, quality control auditing can be conveniently carried out on the target fundus image belonging to the division category with low classification accuracy.

Description

[0001] This application is a divisional application of a patent application with an application date of 2021-03-24, application number 2021103163439, and invention name "Quality Control Method and Quality Control System for Fundus Image Screening System". Technical Field

[0002] The present disclosure relates to a method for acquiring a quality control fundus image of a screening system and related methods and servers. Background Art

[0003] Fundus images often contain rich tissue details. For example, fundus images often include tissue details such as the inner membrane, retina, macula and blood vessels of the eyeball. In clinical practice, by understanding these tissue details in fundus images, doctors can be assisted in identifying related diseases. In modern hospitals, the use of fundus images to assist in the identification of diseases has developed into a major method for clinical screening. With the development of artificial intelligence technology, related technologies based on computer vision and artificial intelligence such as machine learning have been developed and applied in the identification of diseases based on medical images such as fundus images. For example, an artificial intelligence screening system can automatically identify diseases in fundus images by learning the method of professional ophthalmologists identifying diseases based on fundus images. Thus, clinicians can be assisted in screening fundus images. Patent document 1 (CN108231194A) describes a disease diagnosis system, which includes an image acquisition module for acquiring annotated medical images, a diagnostic model establishment module for constructing a deep neural network based on annotated medical images and optimizing the deep neural network, an output diagnostic model and a disease diagnosis module for obtaining diagnostic results.

[0004] However, the disease diagnosis system disclosed in Patent Document 1 does not include quality control of the diagnosis results. In this case, the reliability of the disease diagnosis system when processing certain fundus images is likely to be low. Summary of the invention

[0005] The present disclosure is proposed in view of the above-mentioned situation, and its object is to provide a quality control method and a quality control system of a fundus image screening system that can improve the reliability of the fundus image screening system.

[0006] To this end, the first aspect of the present disclosure provides a quality control method for a fundus image screening system, the fundus image screening system comprising a plurality of fundus images and screening information corresponding to each fundus image including system screening results, comprising: obtaining a preset number of the fundus images as fundus image samples, the screening information corresponding to the fundus image samples as sample information, comparing the system screening results of the fundus image samples with the correct screening results of the fundus image samples based on the sample information to screen out misjudged samples, dividing the misjudged samples based on the sample information of the misjudged samples, and classifying the fundus image samples based on the divided classification categories to obtain the classification accuracy of each of the divided categories; obtaining a specified number of the fundus images of at least one batch as target fundus images, the system screening results corresponding to the target fundus images The result is taken as the target screening result, and the screening information corresponding to the target fundus image is taken as the target information; the target fundus image is classified based on the target information and the classification category to obtain the classification category to which the target fundus image belongs, and the target fundus image belonging to the classification category with the classification accuracy lower than the specified value is taken as the quality control fundus image; the quality control fundus image is audited to obtain the quality control screening result; the target fundus images of each batch are sampled based on the sampling scheme, and the sampled fundus images in the sampling results are sampled and audited to determine whether each batch is qualified, and the sampling scheme includes at least one of uniform sampling and separate sampling for each batch, and the separate sampling is to classify the target fundus images of each batch according to one or more different dimensions and sample for each category. In the present disclosure, the classification accuracy of the classification category is obtained based on the fundus image samples of the fundus image screening system, and the target fundus images belonging to the classification category with lower classification accuracy in each batch are audited based on the classification accuracy, and each batch is sampled and the sampling results are audited. In this case, quality control review can be performed on target fundus images belonging to the classification categories with low classification accuracy, and sampling review can be further performed on each batch, thereby improving the reliability of the fundus image screening system.

[0007] In addition, in the quality control method involved in the first aspect of the present disclosure, optionally, the dimension includes at least one of the target screening result of the target fundus image, the quality control screening result of the target fundus image, the source of the target fundus image, the shooting device model of the target fundus image, the shooting personnel information of the target fundus image, the shooting quality of the target fundus image, and the user information of the target fundus image. In this case, the target fundus images in each batch can be classified based on different dimensions and sampled separately for each category. Thereby, the reliability of sampling can be improved.

[0008] In addition, in the quality control method involved in the first aspect of the present disclosure, optionally, the batches determined to be unqualified are recalled, and at least one of the following processes is performed: full inspection of the recalled batches, increasing the sampling rate of the recalled batches, and overhauling the hardware and software of the fundus image screening system. In this case, the unqualified batches are further processed and the hardware and software are overhauled in a timely manner. Thereby, the reliability of the fundus image screening system can be improved.

[0009] In addition, in the quality control method involved in the first aspect of the present disclosure, optionally, the quality control audit or the sampling audit is a multi-level audit, thereby further improving the reliability of the fundus image screening system.

[0010] In addition, in the quality control method involved in the first aspect of the present disclosure, optionally, in the multi-level audit, the target fundus image is first subjected to primary screening, and then the difficult target fundus image obtained by the primary screening is subjected to advanced screening, and the advanced screening also performs a random check on the screening results of the primary screening. In this case, the advanced screening performs an advanced screening on the difficult target fundus image obtained by the primary screening and performs a random check on the screening results of the primary screening. Thereby, the reliability of the screening results can be improved.

[0011] In addition, in the quality control method involved in the first aspect of the present disclosure, optionally, each of the divided categories includes at least one of the fundus images in the misjudged samples and the fundus images in the non-misjudged samples, and the classification accuracy is obtained based on the number of fundus images in the non-misjudged samples in each of the divided categories and the ratio of the fundus images in the fundus image samples belonging to the divided category. Thus, the classification accuracy can be obtained.

[0012] A second aspect of the present disclosure provides a quality control system for a fundus image screening system, the fundus image screening system comprising a plurality of fundus images and screening information corresponding to each fundus image including a system screening result, comprising a division module, an acquisition module, a screening module, a quality control review module and a sampling module; the division module acquires a preset number of the fundus images as fundus image samples, and the screening information corresponding to the fundus image samples as sample information, compares the system screening results of the fundus image samples with the correct screening results of the fundus image samples based on the sample information to screen out misjudged samples, divides the misjudged samples based on the sample information of the misjudged samples, and classifies the fundus image samples based on the divided classification categories to obtain the classification accuracy of each of the divided categories; the acquisition module is used to acquire a specified number of the fundus images of at least one batch as target fundus images, and the target fundus images correspond to as the target screening result, and the screening information corresponding to the target fundus image as the target information; the screening module classifies the target fundus image based on the target information and the classification category to obtain the classification category to which the target fundus image belongs, and uses the target fundus image belonging to the classification category whose classification accuracy is lower than the specified value as the quality control fundus image; the quality control review module is used to perform quality control review on the quality control fundus image to obtain the quality control screening result; and the sampling module samples the target fundus images of each batch based on the sampling plan and performs sampling review on the sampled fundus images in the sampling results to determine whether each batch is qualified, and the sampling plan includes at least one of unified sampling and separate sampling for each batch, and the separate sampling is to classify the target fundus images of each batch according to one or more different dimensions and sample for each category. In the present disclosure, the division module obtains the classification accuracy of the division category based on the fundus image samples of the fundus image screening system; the acquisition module obtains the target information of the target fundus images of at least one batch from the fundus image screening system; the screening module obtains the target fundus images of the division category with lower classification accuracy in each batch based on the classification accuracy; the quality control and audit module performs quality control audit on the target fundus images of the division category with lower classification accuracy; the sampling module samples each batch and performs sampling audit on the sampling results. In this case, it is possible to perform quality control audit on the target fundus images of the division category with lower classification accuracy and further perform sampling audit on each batch. As a result, the reliability of the fundus image screening system can be improved.

[0013] In addition, in the quality control system involved in the second aspect of the present disclosure, optionally, the dimension includes at least one of the target screening result of the target fundus image, the quality control screening result of the target fundus image, the source of the target fundus image, the shooting device model of the target fundus image, the shooting personnel information of the target fundus image, the shooting quality of the target fundus image, and the user information of the target fundus image. In this case, the target fundus images in each batch can be classified based on different dimensions and sampled separately for each category. Thereby, the reliability of sampling can be improved.

[0014] In addition, in the quality control system involved in the second aspect of the present disclosure, optionally, the quality control system also includes a recall module, which is used to recall batches determined to be unqualified, and perform at least one of full inspection of the recalled batches, increasing the sampling rate of the recalled batches, and repairing the software and hardware of the fundus image screening system. In this case, the unqualified batches are further processed and the software and hardware are repaired in time. Thereby, the reliability of the fundus image screening system can be improved.

[0015] In addition, in the quality control system involved in the second aspect of the present disclosure, optionally, the quality control audit or the sampling audit is a multi-level audit, thereby further improving the reliability of the fundus image screening system.

[0016] According to the present disclosure, a quality control method and a quality control system of a fundus image screening system are provided, which are capable of improving the reliability of the fundus image screening system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings, in which:

[0018] Figure 1 is an application scenario diagram showing the quality control method of the fundus image screening system involved in the example of the present disclosure.

[0019] Figure 2 is a flowchart showing a quality control method of a fundus image screening system according to an example of the present disclosure.

[0020] Figure 3 is a flow chart showing the process of obtaining classification accuracy involved in the example of the present disclosure.

[0021] Figure 4 is a flowchart showing the acquisition of quality control fundus images involved in the example of the present disclosure.

[0022] Figure 5 is a flow chart showing the three-level review involved in the examples of the present disclosure.

[0023] Figure 6 is a flow chart showing sampling and sampling review based on multiple different dimensions involved in the example of the present disclosure.

[0024] Figure 7 is a flowchart showing the random inspection and recall based on GB / T2828.1-2012 involved in the example of the present disclosure.

[0025] Figure 8 is a block diagram showing a quality control system of a fundus image screening system to which the present disclosure example relates. DETAILED DESCRIPTION

[0026] Hereinafter, the preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same symbols are assigned to the same components, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the dimensions of the components or the shapes of the components may be different from the actual ones.

[0027] It should be noted that the terms "including" and "having" and any variations thereof in the present disclosure, such as a process, method, system, product or device that includes or has a series of steps or units, are not necessarily limited to those steps or units clearly listed, but may include or have other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 is a diagram showing an application scenario of the quality control method of the fundus image screening system 110 involved in the example of the present disclosure.

[0029] In some examples, the quality control method (sometimes also referred to as the quality control method) of the fundus image screening system 110 (described later) involved in the present disclosure can be applied to, for example, Figure 1 In the application scenario 100 shown. In the application scenario 100, the fundus image can be screened by the fundus image screening system 110 to obtain screening information. In some examples, the screening information can be used as target information. The quality control server 120 can implement a quality control method by executing a computer program, and the quality control method can be used to perform quality control on the fundus image screening system 110 based on the target information.

[0030] In some examples, the fundus image screening system 110 may include an acquisition device 112, a terminal 113, and a screening server 114. Although the fundus image screening system 110 of the present embodiment includes the acquisition device 112, the terminal 113, and the screening server 114, it is only for the convenience of explaining the application scenario 100 of the quality control method. In other examples, the fundus image screening system 110 may be a device, module, or interface capable of performing screening based on fundus images.

[0031] In some examples, the acquisition device 112 can be used to acquire a fundus image of the fundus of the human eye 111. In some examples, the terminal 113 can acquire the fundus image by controlling the acquisition device 112. In some examples, the terminal 113 can send the fundus image to the screening server 114. In some examples, the screening server 114 can implement screening of the fundus image and generate screening information by executing a computer program. In some examples, the terminal 113 can obtain screening information of the screening server 114 screening the fundus image. In some examples, the terminal 113 can display the screening information. In some examples, the terminal 113 can store the screening information.

[0032] In some examples, the fundus of the human eye 111 refers to the tissue at the back of the eyeball, which may include the inner membrane, retina, macula, and blood vessels of the eyeball. In addition, in some examples, the acquisition device 112 may be a camera. The camera may be, for example, a handheld fundus camera or a desktop fundus camera. In addition, in some examples, the terminal 113 may include, but is not limited to, a laptop computer, a tablet computer, or a desktop computer. In addition, in some examples, the screening server 114 may include one or more processors and one or more memories. In addition, in some examples, the quality control server 120 may include one or more processors and one or more memories. In addition, in some examples, the screening server 114 and the quality control server 120 may be the same server.

[0033] As described above, the quality control method can be used to perform quality control on the fundus image screening system 110 based on the target information. In addition, the quality control method can also be easily extended to perform quality control on screening systems for other images besides fundus images.

[0034] The quality control method of the present disclosure is described in detail below with reference to the accompanying drawings. Figure 2 is a flowchart showing a quality control method of the fundus image screening system 110 according to an example of the present disclosure.

[0035] In some examples, such as Figure 2As shown, the quality control method may include obtaining the classification accuracy (step S110), obtaining the target information of at least one batch of target fundus images (step S120), obtaining the quality control fundus images (step S130), performing quality control audit on the quality control fundus images (step S140), and sampling and sampling audit on each batch to determine whether each batch is qualified (step S150). In this case, the quality control audit can be performed on the target fundus images belonging to the classification category with a lower classification accuracy, and the sampling audit can be further performed on each batch. As a result, the reliability of the fundus image screening system 110 can be improved.

[0036] Figure 3 is a flow chart showing the process of obtaining classification accuracy involved in the example of the present disclosure.

[0037] In some examples, in step S110, the classification accuracy can be obtained. Figure 3 As shown, obtaining the classification accuracy may include obtaining sample information of fundus image samples (step S111), obtaining correct screening results of fundus image samples (step S112), screening misjudged samples (step S113), classifying misjudged samples (step S114), classifying fundus image samples based on the divided categories (step S115) and obtaining the classification accuracy of each divided category (step S116).

[0038] In some examples, in step S111, sample information of fundus image samples may be obtained. In some examples, the fundus image samples may be fundus images from the fundus image screening system 110, that is, the fundus image screening system 110 may include multiple fundus images. In some examples, a preset number of fundus images may be obtained as fundus image samples. For example, 50,000 to 100,000 fundus images may be obtained from the fundus image screening system 110 and used as fundus image samples.

[0039] In some examples, the fundus images from the fundus image screening system 110 have screening information, that is, the fundus image screening system 110 may include screening information corresponding to each fundus image. In some examples, the screening information may include system screening results. In some examples, the screening information corresponding to the fundus image samples may be used as sample information.

[0040] In some examples, the screening information may also include at least one of the source of the fundus image, the model of the device for shooting the fundus image, the information of the person shooting the fundus image, the doctor information of the fundus image, the shooting quality of the fundus image, or the user information of the fundus image. In some examples, the screening information may also include the characteristics of the fundus image such as microaneurysm, hemorrhage, exudation, cotton wool spot, neovascularization, or macular lesion. In some examples, the shooting quality of the fundus image may be very good, good, fair, poor, and extremely poor.

[0041] In some examples, the system screening results may include two results, negative or positive. However, the examples disclosed herein are not limited thereto. In other examples, the system screening results may be other classification results. For example, for a fundus image screening system 110 based on diabetic retinopathy (DR) screening, the system screening result may be a retinopathy grade in a retinopathy grading system used in the UK National Retinopathy Screening Program. Retinopathy grades may include no retinopathy (R0), background period (R1), preproliferative period (R2), and proliferative period (R3).

[0042] In some examples, in step S112, a correct screening result of the fundus image sample may be obtained. Specifically, a professional ophthalmologist may determine the fundus image sample to obtain a correct screening result of the fundus image sample. For example, a professional ophthalmologist may determine the fundus image sample to obtain a correct screening result. In some examples, the correct screening result may include both negative and positive results.

[0043] In some examples, in step S113, misjudged samples may be screened. In some examples, the system screening results of the fundus image samples and the correct screening results of the fundus image samples may be compared to screen misjudged samples. In some examples, fundus images in which the system screening results and the correct screening results in the fundus image samples are inconsistent may be used as misjudged samples. For example, 2,000 fundus images may be screened out from the fundus image samples and used as misjudged samples.

[0044] In some examples, in step S114, the misjudged samples can be divided. The misjudged samples can be obtained by step S113. For example, the 2000 fundus images in the misjudged samples can be divided into three classification categories: classification category A, classification category B, and classification category C. In some examples, the misjudged samples can be divided based on the sample information of the misjudged samples (such as system screening results). In some examples, the misjudged samples can be clustered and the classification categories can be obtained. Specifically, clustering can be performed on the misjudged samples to obtain the cluster center of each cluster and use each cluster as a classification category. In some examples, clustering can be performed using a clustering algorithm. For example, the clustering algorithm can be the K-Means classification method. In other examples, the misjudged samples can be classified based on predefined classification categories.

[0045] In some examples, in step S115, the fundus image samples can be classified based on the divided classification categories. That is, the fundus images in the fundus image samples can be divided into multiple classification categories. As described above, clustering can be performed on the misjudged samples to obtain the cluster centers of each cluster and use each cluster as a classification category. In this case, the distance from each fundus image sample to each cluster center can be calculated to select the cluster corresponding to the cluster center closest to the cluster center as the cluster category (that is, the classification category) to which the fundus image sample belongs. As described above, the classification categories can be pre-defined or obtained by clustering the misjudged samples. For example, if three classification categories of classification category A, classification category B and classification category C are obtained in step S114, the fundus image samples can be divided into three classification categories of classification category A, classification category B and classification category C.

[0046] In some examples, in step S116, the classification accuracy of each divided category can be obtained. As described above, in step S113, the misjudged samples can be screened, and in step S115, the fundus image samples can be classified based on the divided categories. In this case, in some examples, each divided category can include at least one of the fundus images in the misjudged samples and the fundus images in the non-misjudged samples. Here, the non-misjudged samples can be fundus images other than the misjudged samples in the fundus image samples. In some examples, the classification accuracy can be obtained based on the number of fundus images in the non-misjudged samples in each divided category and the ratio of the fundus images in the fundus image samples belonging to the divided category. For example, if the division category A includes 10 fundus images in the misjudged samples and 990 fundus images in the non-misjudged samples, the classification accuracy of the division category A can be 99%. If the division category B includes 0 fundus images in the misjudged samples and 100 fundus images in the non-misjudged samples, the classification accuracy of the division category B can be 100%. If classification category C includes 98 fundus images in misjudged samples and 0 fundus images in non-misjudged samples, the classification accuracy of classification category C may be 0%.

[0047] As described above, the quality control method may include step S120 (see Figure 2 ). In some examples, in step S120, target information of at least one batch of target fundus images may be acquired.

[0048] In some examples, target information of a specified number of target fundus images of each batch may be obtained. For example, target information of 100, 500, or 1000 target fundus images of each batch may be obtained. For another example, target information of 10%, 30%, or 50% of the target fundus images of each batch may be obtained. However, the examples disclosed herein are not limited thereto, and in other examples, target information of all target fundus images of each batch may be obtained.

[0049] In some examples, the target fundus image may be a fundus image from the fundus image screening system 110, that is, the fundus image from the fundus image screening system 110 may be used as the target fundus image. In some examples, one or more batches of fundus images may be obtained from the fundus image screening system 110 at preset intervals (e.g., by the hour) and used as the target fundus image. In other examples, one or more batches of fundus images pushed by the fundus image screening system 110 may be received and used as the target fundus image. In some examples, the system screening result corresponding to the target fundus image may be used as the target screening result. As described above, the fundus image from the fundus image screening system 110 has screening information. In some examples, the screening information corresponding to the target fundus image may be used as the target information, and the target information may include the target screening result.

[0050] Figure 4 is a flowchart showing the acquisition of quality control fundus images involved in the example of the present disclosure.

[0051] In some examples, in step S130, a quality control fundus image may be acquired. Figure 4 As shown, acquiring a quality control fundus image may include classifying a target fundus image (step S131 ) and acquiring a quality control fundus image based on the classification accuracy and the classification category of the target fundus image (step S132 ).

[0052] In some examples, in step S131, the target fundus image may be classified. In some examples, the target fundus image may be classified based on the target information and the classification category to obtain the classification category to which the target fundus image belongs. The classification category may be obtained in step S110, and the target information may be obtained in step S120.

[0053] For example, it is assumed that three classification categories, namely, classification category A, classification category B, and classification category C, are obtained in step S110, wherein the condition of classification category A may be that the target screening result is positive and the shooting quality is very good, the condition of classification category B may be that the target screening result is negative, and the condition of classification category C may be that the target screening result is positive and the characteristics of the target fundus image include bleeding. If the target information of the target fundus image satisfies the condition of a certain classification category, such as classification category A, the target fundus image may belong to the classification category.

[0054] In some examples, in step S132, a quality control fundus image can be obtained based on the classification accuracy and the classification category of the target fundus image. The classification accuracy can be obtained by step S110. In some examples, the target fundus image belonging to the classification category whose classification accuracy is lower than a specified value can be used as a quality control fundus image. In some examples, the specified value can be 60% to 90%. For example, the specified value can be 60%, 70%, 70%, 80% or 90%, etc.

[0055] For example, if the classification accuracy rates of the above three classification categories, classification category A, classification category B, and classification category C, are 99%, 48%, and 70%, respectively, and the specified value is 60%. In this case, since the classification accuracy rate of classification category B is lower than 60%, the target fundus image corresponding to classification category B can be used as a quality control fundus image. However, the examples disclosed in the present invention are not limited thereto, and in other examples, a quality control fundus image can be obtained according to other standards.

[0056] As described above, the quality control method may include step S140 (see Figure 2). In some examples, in step S140, a quality control review may be performed on the quality control fundus image. In some examples, a quality control review may be performed on the quality control fundus image to obtain a quality control screening result.

[0057] In some examples, the quality control review may be a multi-level review, such as a second-level review, a third-level review, or a fifth-level review, etc. Thus, the reliability of the fundus image screening system 110 can be further improved.

[0058] In some examples, in a multi-level audit, a primary screening may be performed on the target fundus image. In some examples, the primary screening may be performed by a primary-level auditor. In some examples, an advanced screening may be performed on the difficult target fundus image obtained by the primary screening. In some examples, the advanced screening may be performed by a high-level auditor. In some examples, the advanced screening also performs a spot check on the screening results of the primary screening. Thus, the reliability of the screening results can be improved.

[0059] In some examples, the quality control screening results may include negative or positive results. In some examples, the quality control screening results may be screening results obtained at each level of the multi-level audit.

[0060] Figure 5 is a flow chart showing the three-level review involved in the examples of the present disclosure.

[0061] In some examples, the quality control review can be a three-level review. Figure 5 As shown, the three-level review may include obtaining the target fundus image that needs to be reviewed as the fundus image to be reviewed (step S210), screening the fundus image to be reviewed to obtain a primary screening result and a first difficult fundus image (step S220), screening the first difficult fundus image to obtain a secondary screening result and a second difficult fundus image and performing a random check on the primary screening result (step S230), and screening the second difficult fundus image to obtain an arbitration screening result and performing a random check on the secondary screening result (step S240).

[0062] In some examples, in step S210, a target fundus image to be reviewed may be obtained as a fundus image to be reviewed. In some examples, the fundus image to be reviewed may be a quality control fundus image or a sampling fundus image (described later).

[0063] In some examples, in step S220, the fundus image to be reviewed may be screened to obtain a primary screening result and a first difficult fundus image. In some examples, the primary screening result may include a negative or positive result. In some examples, the difficult fundus image to be reviewed may be marked by a primary reader and used as the first difficult fundus image.

[0064] In some examples, in step S230, the first difficult fundus image may be screened to obtain a secondary screening result and a second difficult fundus image and the primary screening result may be sampled. In some examples, the secondary screening result may include a negative or positive result. In some examples, the difficult first difficult fundus image may be marked by a secondary reader and used as a second difficult fundus image. In some examples, the primary screening result may be sampled by a secondary reader.

[0065] In some examples, in step S240, the second difficult fundus image may be screened to obtain an arbitration screening result and the secondary screening result may be sampled. In some examples, the second difficult fundus image may be screened by an arbitrator and the arbitration screening result may be obtained by combining the primary screening result and the secondary screening result. In some examples, the arbitration screening result may include both negative and positive results. In some examples, the secondary screening result may be sampled by an arbitrator.

[0066] As described above, the quality control method may include step S150 (see Figure 2 ). In some examples, in step S150, each batch may be sampled and audited to determine whether each batch is qualified.

[0067] In some examples, each batch of target fundus images may be sampled based on a sampling scheme.

[0068] In some examples, the sampling scheme may include at least one of uniform sampling and individual sampling for each batch. In some examples, the uniform sampling may be a sampling method performed on all fundus images of each batch. In some examples, the sampling method may include, but is not limited to, random sampling, stratified sampling, overall sampling, or systematic sampling.

[0069] In addition, in some examples, separate sampling can be for classifying the target fundus images according to one or more different dimensions, and sampling is performed separately for each category. In some examples, the dimensions may include at least one of the target screening results of the target fundus images, the quality control screening results of the target fundus images, the source of the target fundus images, the model of the shooting equipment of the target fundus images, the shooting personnel information of the target fundus images, the shooting quality of the target fundus images, and the user information of the target fundus images. In this case, the target fundus images in each batch can be classified based on different dimensions and sampled separately for each category. Thereby, the reliability of sampling can be improved.

[0070] In some examples, the sampled fundus images in the sampling results may be sampled and audited to determine whether each batch is qualified.

[0071] As described above, the sampling scheme may include classifying the target fundus images according to one or more different dimensions, and sampling each category separately. The sampled fundus images in the sampling results may be sampled and reviewed to determine whether each batch is qualified. Specifically, the two dimensions of the target screening results and the quality control screening results are used as examples to illustrate sampling the target fundus images according to one or more different dimensions, and sampling and reviewing the sampled fundus images in the sampling results.

[0072] Figure 6 It is a flowchart showing the sampling and sampling review based on the two dimensions of target screening results and quality control screening results involved in the example of the present disclosure. Among them, step S130 and step S140 are steps in the quality control method. Step S130 can screen the target fundus image to obtain the quality control fundus image, and step S140 can perform quality control review on the quality control fundus image. For details, please refer to the relevant description of step S130 and step S140. Steps S151 to S153 are steps for sampling and sampling review based on the two dimensions of target screening results and quality control screening results.

[0073] like Figure 6 As shown, the target fundus image can be divided into a quality control fundus image and other fundus images (i.e., the target fundus image other than the quality control fundus image) through step S130. Since the target fundus image can have a target screening result, the quality control fundus image and other fundus images can also have a target screening result, and the quality control fundus image can also have a quality control screening result through step S130.

[0074] In some examples, sampling and sampling review based on two dimensions of target screening results and quality control screening results may include steps S151 to S153.

[0075] In some examples, in step S151, the target fundus image may be classified based on two dimensions: target screening results and quality control screening results.

[0076] like Figure 6 As shown, in some examples, in step S151, the target fundus image can be divided into six categories based on the target screening results and the quality control screening results: the target screening result is positive and there is no quality control screening result, the target screening result is negative and there is no quality control screening result, the target screening result is positive and the quality control screening result is positive, the target screening result is negative and the quality control screening result is negative, the target screening result is positive and the quality control screening result is negative, and the target screening result is negative and the quality control screening result is positive.

[0077] In some examples, in step S152, sampling may be performed separately for each category. Specifically, different sampling methods or different sampling parameters (e.g., sampling rates) may be used for each category. In some examples, sampling may be performed for each category to obtain a sampling result. The sampling result may include a sampled fundus image.

[0078] In some examples, in step S153, a sampling audit can be performed on the sampled fundus images in the sampling results. In some instances, the sampling audit can be a multi-level audit. The specific content of the multi-level audit can refer to the relevant description of the multi-level audit in step S140. In some examples, it can be determined whether each batch is qualified based on the result of the sampling audit of the sampled fundus images. In some examples, it can be determined whether each batch is qualified based on the result of the sampling audit of the sampled fundus images and using the national standard for sampling inspection. In some examples, the national standard for sampling inspection can include but is not limited to GB / T2828.1-2012, GB / T2828.5-2011 or GB / T8052-2002, etc.

[0079] In some examples, the quality control method also includes recalling the batches determined to be unqualified and performing corresponding processing on the recalled batches (not shown). In some examples, the processing of the recalled batches may include at least one of full inspection of the recalled batches, increasing the sampling rate of the recalled batches, and overhauling the hardware and software of the fundus image screening system 110. In this case, the unqualified batches are further processed and the hardware and software are repaired in a timely manner. Thus, the reliability of the fundus image screening system 110 can be improved. In some examples, the target fundus images in the recalled batches can be re-screened.

[0080] Figure 7 is a flowchart showing the random inspection and recall based on GB / T2828.1-2012 involved in the example of the present disclosure.

[0081] As mentioned above, national standards for sampling inspection may include but are not limited to GB / T2828.1-2012, GB / T2828.5-2011 or GB / T8052-2002, etc. Figure 7It is a process of sampling and recall based on GB / T2828.1-2012. In some examples, the process of sampling and recall may include determining the degree of sampling (step S310), determining the sampling rate (step S320), determining the AQL value (step S330), determining the sample size (step S340), sampling the target fundus image and sampling review (i.e., sampling) (step S350), judging whether the number of unqualified is greater than the upper limit (step S360), executing the recall (step S370), full inspection or increasing the sampling rate (step S380), determining the reason for unqualified (step S390) and re-screening the target fundus images in the unqualified batches (step S410). In order to facilitate the description of the process of sampling and recall, the target fundus image entering the process can be referred to as the fundus image to be sampled. In some examples, the fundus image to be sampled can be all the target fundus images of each batch in step S120. In some examples, the fundus image to be sampled may be the target fundus image of each category in step S152 .

[0082] In some examples, in step S310, the degree of sampling can be determined. In some examples, the degree of sampling can be general, special, strict or relaxed, etc. In some examples, the degree of sampling can be adjusted according to actual conditions. For example, the degree of sampling of the fundus image screening system 110 that has just been released can be stricter, and for example, the degree of sampling of the fundus images to be sampled screened by less experienced ophthalmologists can be stricter. In some examples, different degrees of sampling can correspond to different sampling methods or different sampling parameters. In some examples, sampling parameters can include, but are not limited to, sampling rates or AQL (Acceptable Quality Limit) values, etc.

[0083] In some examples, in step S320, a sampling rate may be determined. In some examples, the sampling rate may be determined based on the degree of sampling and the number of fundus images to be sampled. In some examples, the sampling rate may be the proportion of images that need to be sampled and reviewed in the fundus images to be sampled.

[0084] In some examples, a preliminary sampling rate may be obtained, and the preliminary sampling rate may be adjusted based on the degree of sampling to determine the sampling rate. For example, if the degree of sampling is relaxed, the preliminary sampling rate may be reduced to determine the sampling rate.

[0085] In some examples, a preliminary sampling rate can be obtained by performing a full inspection on a certain sample batch or multiple sample batches. In some examples, the sample batches can be batches from the fundus image screening system 110. Specifically, in some examples, a full inspection can be performed on the sample batches to obtain a failure rate, and a lower limit of the sampling rate can be determined based on the failure rate and the standard passing rate (for example, the lower limit of the sampling rate can be ), and then the preliminary sampling rate can be set based on the lower limit of the sampling rate. However, the examples of the present disclosure are not limited thereto. In other examples, the preliminary sampling rate can also be a fixed value. In some examples, the standard pass rate can be a fixed value. For example, the standard pass rate can be the requirement of the demander for the pass rate of the fundus image screening system 110.

[0086] In some examples, in step S330, an AQL value may be determined. Generally speaking, AQL refers to the worst allowable average quality level of the process when a continuous series of batches are submitted for acceptance sampling. If the quality level is expressed as a percentage of unqualified products, the AQL value should not exceed 10%, and if the quality level is expressed as the number of unqualified products per hundred units of fundus images to be sampled, the AQL value should not exceed 10. In some examples, the AQL value may be a fixed value. In some examples, the AQL value may be adjusted based on relevant information of the fundus images to be sampled. For example, the AQL value of the fundus images to be sampled that have undergone quality control review is higher than the AQL value of the fundus images to be sampled that have not undergone quality control review, and the AQL value of the fundus images to be sampled with negative results is higher than the AQL value of the fundus images to be sampled with positive results.

[0087] In some examples, in step S340, the sample size may be determined. In some examples, the sample size (ie, the number of samples) may be determined based on a sampling rate.

[0088] In some examples, in step S350, the fundus images to be sampled may be sampled and sampled for review. In some examples, the fundus images to be sampled may be sampled based on the sample size and using a sampling method and a sampling result may be obtained. For example, the fundus images to be sampled may be sampled using random sampling. The sampling result includes the sampled fundus images. In some examples, the sampled fundus images in the sampling result may be sampled for review to obtain the number of unqualified images.

[0089] In some examples, in step S360, it can be determined whether the number of unqualified products is greater than the upper limit. In some examples, it can be determined whether the number of unqualified products in each batch is greater than the upper limit. As described above, the fundus images to be sampled can be all target fundus images of each batch in step S120 or the target fundus images of each category in step S152. In this case, based on the number of unqualified products in the fundus images to be sampled, it can be determined whether the number of unqualified products in each batch is greater than the upper limit. If the number of unqualified products is greater than the upper limit, the batch is judged to be unqualified and enters step S370, otherwise the sampling process is completed. In some examples, the upper limit can be obtained based on the AQL value.

[0090] In some examples, in step S370, a recall may be performed. In some examples, a recall may be performed on batches that are determined to be unqualified. For example, the target screening results of the target fundus images of the unqualified batches may be marked as unqualified and relevant personnel such as patients may be notified.

[0091] In some examples, in step S380, a full inspection may be performed or the sampling inspection rate may be increased. In some examples, a full inspection may be performed on the batches determined to be unqualified or the sampling inspection rate may be increased.

[0092] In some examples, in step S390, the reason for failure can be determined. In some examples, relevant information of the sampled fundus images in the failed batch can be analyzed to determine the reason for failure. In some examples, the failed sampled fundus images in the failed batch can be classified to determine which category of sampled fundus images has a high error rate.

[0093] In some examples, the related software and hardware can be repaired based on the reasons for failure. For example, if the failed sampled fundus images are related to the model of the device that captured the sampled fundus images, the device model corresponding to the device model can be repaired.

[0094] In some examples, in step S400, the target fundus images in the unqualified batches may be rescreened. In some examples, after determining the reasons for the unqualified and repairing the relevant software and hardware, the fundus image screening system 110 may rescreen the target fundus images in the unqualified batches. In some examples, the target fundus images that have been rescreened may be resampled and re-audited. In some examples, the new target screening results obtained through the rescreening may be notified to relevant personnel such as patients.

[0095] The quality control system 200 of the present disclosure is described in detail below in conjunction with the accompanying drawings. The quality control system 200 of the present disclosure is used to implement the above-mentioned quality control method. Figure 8 is a block diagram showing a quality control system 200 of the fundus image screening system 110 to which the present disclosure example relates.

[0096] In some examples, such as Figure 8As shown, the quality control system 200 may include a division module 210, an acquisition module 220, a screening module 230, a quality control review module 240, and a sampling module 250. In some examples, in the quality control system 200, the division module 210 may obtain the classification accuracy. The acquisition module 220 may obtain the target information of at least one batch of target fundus images. The screening module 230 may obtain the quality control fundus images. The quality control review module 240 performs a quality control review on the quality control fundus images. The sampling module 250 may perform sampling and sampling review on each batch to determine whether each batch is qualified. In this case, a quality control review can be performed on the target fundus images belonging to the division categories with lower classification accuracy and a sampling review can be further performed on each batch. Thus, the reliability of the fundus image screening system 110 can be improved.

[0097] In some examples, the segmentation module 210 can obtain the classification accuracy. In some examples, the misjudged samples can be segmented based on the sample information of the misjudged samples. In some examples, the misjudged samples can be clustered and the segmentation categories can be obtained. In other examples, the misjudged samples can be classified based on predefined segmentation categories. In some examples, the system screening results of the fundus image samples and the correct screening results of the fundus image samples can be compared based on the sample information to screen the misjudged samples. In some examples, the fundus image samples can be classified based on the segmentation categories to obtain the classification accuracy of each segmentation category. In some examples, the fundus image screening system 110 may include multiple fundus images. In some examples, a preset number of fundus images can be obtained as fundus image samples. The fundus image screening system 110 may include screening information corresponding to each fundus image. In some examples, the screening information may include the system screening results. In some examples, the screening information corresponding to the fundus image samples may be used as sample information. In some examples, each segmentation category may include at least one of the fundus images in the misjudged samples and the fundus images in the non-misjudged samples. In some examples, the classification accuracy can be obtained based on the number of fundus images in the unmisjudged samples in each classification category and the ratio of fundus images in the fundus image samples belonging to the classification category. For a specific description, please refer to the relevant description of step S110, which will not be repeated here.

[0098] In some examples, the acquisition module 220 can acquire target information of at least one batch of target fundus images. In some examples, target information of a specified number of target fundus images of each batch can be acquired. In some examples, the target fundus image can be a fundus image from the fundus image screening system 110, that is, the fundus image from the fundus image screening system 110 can be used as the target fundus image. In some examples, the system screening result corresponding to the target fundus image can be used as the target screening result. As described above, the fundus image from the fundus image screening system 110 has screening information. In some examples, the screening information corresponding to the target fundus image can be used as the target information, and the target information can include the target screening result. For a specific description, please refer to the relevant description of step S120, which will not be repeated here.

[0099] In some examples, the screening module 230 can obtain a quality control fundus image. In some examples, the target fundus image can be classified based on the target information and the classification category to obtain the classification category to which the target fundus image belongs. In some examples, the target fundus image corresponding to the classification category whose classification accuracy is lower than a specified value can be used as a quality control fundus image. For a specific description, please refer to the relevant description of step S130, which will not be repeated here.

[0100] In some examples, the quality control audit module 240 performs a quality control audit on the quality control fundus image. In some examples, the quality control audit can be performed on the quality control fundus image to obtain the quality control screening result. In some examples, the quality control audit can be a multi-level audit. For example, the multi-level audit can be a secondary audit, a tertiary audit, or a quintuple audit, etc. Thus, the reliability of the fundus image screening system 110 can be further improved. In some examples, in the multi-level audit, the target fundus image can be screened at a primary level. In some examples, the primary screening can be performed by a primary-level auditor. In some examples, the difficult target fundus image obtained by the primary screening can be screened at an advanced level. In some examples, the advanced screening can be performed by a high-level auditor. In some examples, the advanced screening also performs a spot check on the screening results of the primary screening. Thus, the reliability of the screening results can be improved. For a specific description, please refer to the relevant description of step S140, which will not be repeated here.

[0101] In some examples, the sampling module 250 can sample and sample each batch to determine whether each batch is qualified. In some instances, the sampling review can be a multi-level review. In some examples, the target fundus images of each batch can be sampled based on the sampling scheme. In some examples, the sampling scheme can include at least one of uniform sampling and individual sampling for each batch. In some examples, the uniform sampling can be a sampling method for all fundus images of each batch. In some examples, the individual sampling can be to classify the target fundus images according to one or more different dimensions, and sample each category separately. In some examples, the dimension can include at least one of the target screening results of the target fundus image, the quality control screening results of the target fundus image, the source of the target fundus image, the shooting equipment model of the target fundus image, the shooting personnel information of the target fundus image, the shooting quality of the target fundus image, and the user information of the target fundus image. In this case, the target fundus images in each batch can be classified based on different dimensions and sampled separately for each category. Thus, the reliability of sampling can be improved. In some examples, the sampled fundus images in the sampling results may be sampled and reviewed to determine whether each batch is qualified. For a specific description, please refer to the relevant description of step S150, which will not be repeated here.

[0102] In some examples, the quality control system 200 also includes a recall module (not shown). The recall module can recall the batches determined to be unqualified and perform corresponding processing on the recalled batches. In some examples, the processing of the recalled batches can include at least one of full inspection of the recalled batches, increasing the sampling rate of the recalled batches, and overhauling the hardware and software of the fundus image screening system 110. In this case, the unqualified batches are further processed and the hardware and software are repaired in time. As a result, the reliability of the fundus image screening system 110 can be improved. In some examples, the target fundus images in the recalled batches can be re-screened.

[0103] Although the present disclosure is specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the scope of the present disclosure.

Claims

1. A method for obtaining a quality control fundus image of a fundus image screening system, wherein the fundus image screening system includes a plurality of fundus images and screening information corresponding to each fundus image including a system screening result, characterized in that: The method comprises obtaining a preset number of fundus images as fundus image samples, and screening information corresponding to the fundus image samples as sample information; comparing the system screening results of the fundus image samples with the correct screening results of the fundus image samples based on the sample information to screen out misjudged samples; dividing the misjudged samples based on the sample information of the misjudged samples, and classifying the fundus image samples based on the divided classification categories to obtain the classification accuracy of each of the divided categories; obtaining at least one batch of fundus images as target fundus images, and screening information corresponding to the target fundus images as target information; And based on the target information and the classification category, the target fundus image is classified to obtain the classification category to which the target fundus image belongs, and the target fundus image belonging to the classification category whose classification accuracy is lower than a specified value is used as a quality control fundus image.

2. The method according to claim 1, characterized in that The misjudged samples are clustered to obtain the classification categories; or the classification categories are predefined.

3. The method according to claim 1, characterized in that The target fundus images are the fundus images of a specified number of at least one batch; or the target fundus images are all the fundus images of at least one batch.

4. The method according to claim 1, characterized in that: Each of the divided categories includes at least one of the fundus images in the misjudged samples and the fundus images in the non-misjudged samples, and the classification accuracy is obtained based on the ratio of the number of fundus images in the non-misjudged samples in each of the divided categories to the fundus images in the fundus image samples belonging to the divided category.

5. The method according to claim 1, characterized in that The prescribed value is 60% to 90%.

6. A quality control method for a fundus image screening system, characterized in that: include: Acquire a quality control fundus image using the method described in any one of claims 1 to 5; The quality control fundus images are quality controlled and audited to obtain quality control screening results; and target fundus images of each batch are sampled and the sampled fundus images in the sampling results are sampled and audited, and whether each batch is qualified is determined based on the results of the sampling audit of the sampled fundus images.

7. The quality control method according to claim 6, characterized in that: The quality control audit or the sampling audit is a multi-level audit; in the multi-level audit, the target fundus image is first subjected to a primary screening, and then an advanced screening is performed on the difficult target fundus images obtained by the primary screening, and the advanced screening also performs a random inspection on the screening results of the primary screening.

8. The quality control method according to claim 6, characterized in that: The target fundus images of each batch are sampled based on a sampling scheme, wherein the sampling scheme includes at least one of uniform sampling and individual sampling for each batch, wherein the uniform sampling is to execute a sampling method for all fundus images of each batch, and the individual sampling is to classify the target fundus images of each batch according to one or more different dimensions and perform sampling for each category.

9. The quality control method according to claim 8, characterized in that: The system screening result corresponding to the target fundus image is used as the target screening result; the dimension includes the target screening result of the target fundus image and the quality control screening result of the target fundus image; or the dimension includes at least one of the target screening result of the target fundus image, the quality control screening result of the target fundus image, the source of the target fundus image, the shooting device model of the target fundus image, the shooting personnel information of the target fundus image, the shooting quality of the target fundus image and the user information of the target fundus image.

10. A quality control server, characterized in that: The quality control method according to any one of claims 6 to 9 is implemented by executing the computer program.

Images