Eye tumor diagnosis and benign and malignant judgment method, system and equipment
The method uses wide-area fundus color photographs and OCT images to construct models for accurate eye tumor diagnosis, reducing misdiagnosis by employing heat maps for precise tumor detection and classification.
Patent Information
- Application Number
- CN202411283590.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-06
AI Technical Summary
Current eye tumor diagnosis relies heavily on human expertise, which can lead to misdiagnosis due to subtle symptoms or similarities with other eye diseases, especially in early stages.
A method using wide-area fundus color photographs and OCT images to construct models for tumor detection and classification, employing heat maps to assist in determining the presence and nature of eye tumors, with models trained and validated through cross-validation and external/prospective validation.
Reduces the risk of misdiagnosis by accurately identifying eye tumors and their malignancy, ensuring high recognition accuracy and operational simplicity, thus improving diagnostic efficiency.
Smart Images
Figure CN119092103A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image machine learning technology, and in particular to a method, system and device for diagnosing eye tumors and judging whether they are benign or malignant. Background Art
[0002] Eye tumors are tumors that occur in the eyelids, conjunctiva, various layers of tissue of the eyeball, and ocular appendages. Tumors can be benign or malignant, but both types may cause damage to eye tissues and their functions.
[0003] The current diagnosis of eye tumors is mainly carried out by doctors based on a comprehensive diagnosis of the patient's symptoms, signs, professional examination results, pathological examinations and other information. This requires the doctor to have strong professional qualities to avoid misdiagnosis and delays in the treatment of the disease. However, in fact, the symptoms of early eye tumors may not be obvious, or may be similar to those of other eye diseases. If the doctor does not have enough experience, these may mislead the doctor in making a preliminary judgment.
[0004] Therefore, based on the above situation, it is necessary to develop a method to provide automated diagnosis, which can assist doctors in treatment and facilitate the diagnosis and treatment of the disease. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a method, system and equipment for diagnosing eye tumors and judging whether the eye tumor is benign or malignant. An eye tumor diagnosis model is constructed using wide-area color fundus photographs, and the wide-area color fundus photographs to be tested are classified to determine whether the eye tumor exists; and an eye tumor benign or malignant judgment model is constructed using OCT images, and the OCT images to be tested are classified to determine whether the eye tumor is benign or malignant.
[0006] In a first aspect, the first object of the present invention is to provide a method for diagnosing fundus tumors and determining whether they are benign or malignant, comprising the following steps: Obtain wide-area fundus color photographs and OCT images, use the wide-area fundus color photographs as first sample data, and use the OCT images as second sample data; classify and label the first sample data according to whether an eye tumor is included, and classify and label the second sample data according to whether the eye tumor is benign or malignant; The eye tumor diagnosis model is trained using the first labeled sample data, and the eye tumor diagnosis model is used to output the classification result of the wide-area fundus color photograph to be tested and the first heat map; the eye tumor benign and malignant judgment model is trained using the second labeled sample data, and the eye tumor benign and malignant judgment model is used to output the classification result of the OCT image to be tested corresponding to each eye tumor; The patient's wide-area fundus color photograph to be tested is input into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is included; according to the first heat map, the OCT images to be tested at each eye tumor of the same patient are obtained, and the OCT images to be tested are input into the eye tumor benign and malignant judgment model and the classification result of the benign and malignant eye tumor is output.
[0007] Furthermore, the method further includes: inputting the patient's wide-area fundus color photograph to be tested into an eye tumor diagnosis model, and ending if the eye tumor diagnosis model outputs a first heat map and a classification result indicating that the eye tumor is not included.
[0008] Furthermore, the trained eye tumor diagnosis model and the eye tumor benign and malignant judgment model are subjected to five-fold cross-validation using an internal data set to train the eye tumor diagnosis model and the eye tumor benign and malignant judgment model; an external validation data set is used to perform external validation; and the model performance parameters are optimized based on the external validation results; and / or a prospective validation data set is used to perform prospective validation; and the model performance parameters are optimized based on the prospective validation results.
[0009] Furthermore, the model performance parameters include: one or more of accuracy, sensitivity, specificity and AUC value.
[0010] Furthermore, the eye tumor benign or malignant judgment model outputs a second heat map for assisting in judging the benign or malignant nature of the eye tumor.
[0011] Further, the generation process of the first heat map or the second heat map includes: Obtaining the classification results and convolutional layer outputs of the eye tumor diagnosis model or the eye tumor benign and malignant judgment model; Generate a gradient matrix based on the classification results and the convolutional layer output; The gradient matrix is scaled to the same resolution as the original input image of the eye tumor diagnosis model or the eye tumor benign and malignant judgment model, and then superimposed with the original input image to generate a first heat map or a second heat map.
[0012] In a second aspect, the second object of the present invention is to provide a system for diagnosing fundus tumors and judging benign and malignant tumors, comprising: Data acquisition module: obtain wide-area fundus color photographs and OCT images, use the wide-area fundus color photographs as first sample data, and use the OCT images as second sample data; classify and label the first sample data according to whether it contains an eye tumor, and classify and label the second sample data according to whether the eye tumor is benign or malignant; A model building module: connected to the data acquisition module, and used to train an eye tumor diagnosis model using the first labeled sample data, wherein the eye tumor diagnosis model is used to output the classification result of the wide-area fundus color photograph to be tested and the first heat map; and to train an eye tumor benign and malignant judgment model using the second labeled sample data, wherein the eye tumor benign and malignant judgment model is used to output the classification result of the OCT image to be tested corresponding to each eye tumor; Diagnosis generation module: connected to the diagnosis generation module, used to input the patient's wide-area fundus color photograph to be tested into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is included; according to the first heat map, the OCT images to be tested at each eye tumor of the same patient are obtained, and the OCT images to be tested are input into the eye tumor benign and malignant judgment model and the classification result of the eye tumor benign and malignant is output.
[0013] In a third aspect, the present invention provides an electronic device, comprising: one or more memories on which an executable program is stored; and one or more processors for executing the executable program in the memory to implement the steps of the method described in the first aspect.
[0014] The embodiments of the present invention have the following technical effects: Based on the above technical solution, the present application constructs an eye tumor diagnosis model using wide-area fundus color photographs, and classifies the wide-area fundus color photographs to be tested to determine whether an eye tumor exists; and constructs a benign or malignant eye tumor judgment model using OCT images, and classifies the OCT images to be tested to determine whether the eye tumor is benign or malignant; wherein the OCT images are obtained according to the eye tumor position in the heat map generated by the eye tumor diagnosis model; the heat map can also assist in the diagnosis of eye tumors; While ensuring the recognition accuracy, the present application provides an effective method for assisting in the diagnosis of eye tumors, reduces the risk of misdiagnosis, is simple to operate, is more conducive to the diagnosis of eye tumors, and can be popularized to improve the public medical level. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 The present invention provides a flowchart of a method for diagnosing eye tumors and judging whether they are benign or malignant.
[0017] Figure 2is a comparison diagram of the first heat map provided by the embodiment of the present invention and the original wide-area fundus color photograph; wherein, Figure 2 a and Figure 2 c in the figure is the original wide-area color fundus photograph; Figure 2 b and Figure 2 d in the figure are the first heat maps generated by figure a and figure c respectively. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0019] Example 1 Figure 1 1 is a flow chart of a method for diagnosing eye tumors and determining whether they are benign or malignant, provided by an embodiment of the present invention. Figure 1 , including: S1. Obtain wide-area fundus color photographs and OCT images, use the wide-area fundus color photographs as first sample data, and use the OCT images as second sample data; classify and label the first sample data according to whether it contains an eye tumor, and classify and label the second sample data according to whether the eye tumor is benign or malignant.
[0020] Wide-area fundus color photography is a color photograph of the fundus taken using a wide-area imaging system to facilitate doctors to screen and diagnose diseases. Compared with images obtained by other technologies, wide-area fundus color photography can obtain a wider range of fundus information in one shot, avoiding image loss in the edge area, which is extremely important for comprehensive assessment of fundus conditions and discovery of potential lesions. Therefore, this application uses wide-area fundus color photography to diagnose the presence or absence of tumors, which can minimize the possibility of missed diagnosis.
[0021] In addition, wide-area fundus color photographs are high-quality color images, and complete fundus image information can be obtained in just a few minutes, which helps improve examination efficiency.
[0022] OCT (Optical Coherence Tomography) imaging, or optical coherence tomography, is a non-contact biological tissue imaging technology with high spatial resolution. Compared with images obtained by magnetic resonance imaging and CT scanning technology, OCT images have better penetration, can clearly display the microstructure and details of fundus tissue, and are more helpful for doctors to make accurate diagnosis of eye diseases. In addition, the imaging process of the OCT system is friendly, non-invasive and lossless, short in time, high in imaging resolution, and inexpensive.
[0023] It is necessary to obtain as much sample data as possible and pre-process the obtained wide-area fundus color photographs and eye OCT images to ensure image quality and improve the accuracy of subsequent models.
[0024] For example, the OCT images and wide-area fundus color photographs were resized to 512 × 512 pixels, and poor-quality images, such as those with low scale area ratio, low illumination, and low blur, were deleted.
[0025] Each image is manually annotated according to the gold standard: the wide-area color fundus photographs containing eye tumors and those not containing eye tumors are classified and annotated with labels; the benign tumors and malignant tumors in the OCT images are classified and annotated with labels.
[0026] In addition, more sample data can be obtained and divided into validation sets and test sets to facilitate model selection, prevent overfitting, and ensure the reliability of the model.
[0027] S2. Use the first labeled sample data to train an eye tumor diagnosis model, and the eye tumor diagnosis model is used to output the classification results of the wide-area fundus color photograph to be tested and the first heat map; use the second labeled sample data to train an eye tumor benign and malignant judgment model, and the eye tumor benign and malignant judgment model is used to output the classification results of the OCT images to be tested corresponding to each eye tumor.
[0028] The diagnosis of tumors in this application is divided into two steps. First, whether there is a tumor is determined based on the wide-area color fundus photograph. The use of wide-area color fundus photographs reduces the possibility of missed detection. Then, the tumor type is determined based on the OCT image.
[0029] Based on the previous diagnostic steps, the model is trained. First, the eye tumor diagnosis model is used. The ResNet, ResNeXt, Wide ResNet, etc. model is used as the basic network model of the eye tumor diagnosis model. The first labeled sample data is input, and the model is trained according to the wide-area fundus color photo and the labeled information to obtain model parameters, such as learning rate, category weight, number of iterations, etc.
[0030] Among them, an internal data set, that is, a data set within a hospital except for the data used for model training, can be used to perform five-fold cross-validation on the eye tumor diagnosis model to train the eye tumor diagnosis model.
[0031] To evaluate the trained model, data (OCT images and wide-area fundus photos) collected from other hospitals can be used as external validation datasets for external validation, and the model performance parameters can be optimized based on the external validation results; The trained eye tumor diagnosis model and the eye tumor benign and malignant judgment model can be prospectively verified using prospective validation data sets, respectively, and the model performance parameters can be optimized based on the prospective validation results; wherein, the prospective validation data set refers to the wide-area fundus color photographs and OCT images of patients collected in this hospital after the model is trained; the model performance parameters include: one or more of accuracy, sensitivity, specificity and AUC value; all of which are prior art and will not be elaborated in this application.
[0032] Similarly, the same training process is required for the eye tumor benign and malignant judgment model. The training process only differs in the input sample data, so it will not be repeated here. The eye tumor benign and malignant judgment model can also output a second heat map for assisting in judging the benign and malignant nature of the eye tumor.
[0033] In addition, see Figure 2 The two first heat maps on the right side of the middle, this application will output the heat maps of the eye tumor diagnosis model and the eye tumor benign and malignant judgment model as an auxiliary means to judge the results. The red area and orange area in the heat map are the tumor areas.
[0034] Exemplarily, the first heat map or the second heat map may be generated by the following steps: The classification result and convolution layer output of the eye tumor diagnosis model or the eye tumor benign and malignant judgment model are obtained; a gradient matrix is generated according to the classification result and the convolution layer output; the gradient matrix is scaled to the same resolution as the original input image of the eye tumor diagnosis model or the eye tumor benign and malignant judgment model, and then superimposed with the original input image to generate a first heat map or a second heat map.
[0035] It should be noted that the heat map can also be generated by other methods, which are all existing technologies and will not be described in detail in this application.
[0036] S3. Input the patient's wide-area fundus color photograph to be tested into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is included; obtain the OCT images to be tested at the positions of each eye tumor of the same patient according to the first heat map, input the OCT images to be tested into the eye tumor benign and malignant judgment model and output the classification result of the benign and malignant eye tumor.
[0037] The patient's wide-area fundus color photograph to be tested is input into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs the first heat map and the classification result is that the eye tumor is not included, the process ends.
[0038] When applying, first take a wide-area color fundus photograph of the patient, input the wide-area color fundus photograph into the eye tumor diagnosis model, and output the tumor classification result: If it is classified as containing a tumor, then according to the positions of the red area and the orange area in the first heat map, an OCT image of the corresponding position is obtained, and the OCT image is input into the eye tumor benign and malignant judgment model to judge the type of each tumor; If it is classified as not containing tumor, it means there is no tumor and the patient is normal.
[0039] It should be noted that if there is more than one fundus tumor, the OCT images corresponding to the positions of the tumors are obtained according to the positions of the red areas in the first heat map, and input into the eye tumor benign and malignant judgment model to obtain the type of tumor at each position. The second heat map can also be output to assist in the judgment of benign and malignant results.
[0040] In summary, firstly, the presence of an eye tumor is determined by using wide-area fundus color photographs and the first heat map is output. The use of wide-area fundus color photographs can minimize missed diagnoses. If a tumor is present, the location of the eye tumor is determined based on the output first heat map, and the corresponding OCT images are obtained. The benign or malignant nature of the tumor is determined by the OCT images. T Imaging can capture more subtle structures to reduce the risk of false positives; The present application combines wide-area fundus color photography and OCT images to diagnose eye tumors, thereby minimizing the risk of misdiagnosis and ensuring the accuracy of the diagnostic results.
[0041] Example 2 Based on the content of Example 1, the present application provides a fundus tumor diagnosis and benign and malignant judgment system, including: Data acquisition module: obtain wide-area fundus color photographs and OCT images, use the wide-area fundus color photographs as first sample data, and use the OCT images as second sample data; classify and label the first sample data according to whether it contains an eye tumor, and classify and label the second sample data according to whether the eye tumor is benign or malignant; A model building module: connected to the data acquisition module, and used to train an eye tumor diagnosis model using the first labeled sample data, wherein the eye tumor diagnosis model is used to output the classification result of the wide-area fundus color photograph to be tested and the first heat map; and to train an eye tumor benign and malignant judgment model using the second labeled sample data, wherein the eye tumor benign and malignant judgment model is used to output the classification result of the OCT image to be tested corresponding to each eye tumor; Diagnosis generation module: connected to the diagnosis generation module, used to input the patient's wide-area fundus color photograph to be tested into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is included; according to the first heat map, the OCT images to be tested at each eye tumor of the same patient are obtained, and the OCT images to be tested are input into the eye tumor benign and malignant judgment model and the classification result of the eye tumor benign and malignant is output.
[0042] For other contents, please refer to Example 1, which will not be elaborated in this application.
[0043] Example 3 Based on the content of Example 1, the present application provides an electronic device, comprising: one or more memories on which an executable program is stored; and one or more processors for executing the executable program in the memory to implement the steps of the method described in Example 1.
[0044] Based on the above technical solution, the present application constructs an eye tumor diagnosis model using wide-area color fundus photographs to determine the presence or absence of a tumor, and constructs an eye tumor benign and malignant judgment model using OCT images to determine the benign and malignant nature of each tumor. While ensuring the recognition accuracy, the present application provides an effective method for auxiliary diagnosis of eye tumors, thereby reducing the risk of misdiagnosis.
[0045] It should be noted that any process or method description in the flowchart or otherwise described herein may be understood to represent a module, fragment or portion of a code comprising one or more executable instructions for implementing steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations, in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0046] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0047] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.
[0048] Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.
Claims
1. A method for diagnosing eye tumors and determining whether they are benign or malignant, characterized in that: The following steps are involved: Obtain wide-area fundus color photographs and OCT images, use the wide-area fundus color photographs as first sample data, and use the OCT images as second sample data; classify and label the first sample data according to whether an eye tumor is included, and classify and label the second sample data according to whether the eye tumor is benign or malignant; The eye tumor diagnosis model is trained using the first labeled sample data, and the eye tumor diagnosis model is used to output the classification result of the wide-area fundus color photograph to be tested and the first heat map; the eye tumor benign and malignant judgment model is trained using the second labeled sample data, and the eye tumor benign and malignant judgment model is used to output the classification result of the OCT image to be tested corresponding to each eye tumor; The patient's wide-area fundus color photograph to be tested is input into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is included; according to the first heat map, the OCT images to be tested at each eye tumor of the same patient are obtained, and the OCT images to be tested are input into the eye tumor benign and malignant judgment model and the classification result of the benign and malignant eye tumor is output.
2. A method for diagnosing fundus tumors and judging whether they are benign or malignant according to claim 1, characterized in that: Also includes: The patient's wide-area fundus color photograph to be tested is input into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is not included, the process ends.
3. A method for diagnosing fundus tumors and judging whether they are benign or malignant according to claim 1, characterized in that: Using an internal data set to perform a five-fold cross validation on the eye tumor diagnosis model and the eye tumor benign and malignant judgment model to train the eye tumor diagnosis model and the eye tumor benign and malignant judgment model; Using an external validation data set to externally validate the trained ocular tumor diagnosis model and ocular tumor benign and malignant judgment model; optimizing model performance parameters based on the external validation results; and / or Prospective validation was performed using a prospective validation data set; model performance parameters were optimized based on the prospective validation results.
4. A method for diagnosing fundus tumors and judging whether they are benign or malignant according to claim 3, characterized in that: The model performance parameters include: One or more of accuracy, sensitivity, specificity and AUC value.
5. The method for diagnosing fundus tumors and judging whether they are benign or malignant according to claim 1, characterized in that: The eye tumor benign and malignant judgment model outputs a second heat map for assisting in judging the benign and malignant nature of the eye tumor.
6. A method for diagnosing fundus tumors and judging whether they are benign or malignant according to claim 5, characterized in that: The generation process of the first heat map or the second heat map includes: Obtaining the classification results and convolutional layer outputs of the eye tumor diagnosis model or the eye tumor benign and malignant judgment model; Generate a gradient matrix based on the classification results and the convolutional layer output; The gradient matrix is scaled to the same resolution as the original input image of the eye tumor diagnosis model or the eye tumor benign and malignant judgment model, and then superimposed with the original input image to generate a first heat map or a second heat map.
7. A system for diagnosing fundus tumors and judging whether they are benign or malignant, characterized in that: include: Data acquisition module: acquiring wide-area fundus color photographs and OCT images, using the wide-area fundus color photographs as first sample data and the OCT images as second sample data; Classifying and labeling the first sample data according to whether it contains an eye tumor, and classifying and labeling the second sample data according to whether the eye tumor is benign or malignant; A model building module: connected to the data acquisition module, and used to train an eye tumor diagnosis model using the first labeled sample data, wherein the eye tumor diagnosis model is used to output the classification result of the wide-area fundus color photograph to be tested and the first heat map; and to train an eye tumor benign and malignant judgment model using the second labeled sample data, wherein the eye tumor benign and malignant judgment model is used to output the classification result of the OCT image to be tested corresponding to each eye tumor; Diagnosis generation module: connected to the diagnosis generation module, used to input the patient's wide-area fundus color photograph to be tested into the eye tumor diagnosis model. If the eye tumor diagnosis model outputs a first heat map and the classification result is that the eye tumor is included; according to the first heat map, the OCT images to be tested at each eye tumor of the same patient are obtained, and the OCT images to be tested are input into the eye tumor benign and malignant judgment model and the classification result of the eye tumor benign and malignant is output.
8. An electronic device, characterized in that: include: One or more memories on which executable programs are stored; One or more processors, configured to execute the executable program in the memory to implement the steps of the method according to any one of claims 1 to 6.