Method and system for detecting small lesions of diabetic retinopathy based on fundus images
By constructing a deep learning model and a voice prompt system, fundus images are automatically identified and classified, solving the problems of time consumption and subjectivity in traditional fundus lesion detection, and realizing real-time, accurate detection and guidance for diabetic retinopathy.
Patent Information
- Application Number
- CN202411162501.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-23
Smart Images

Figure CN118675219B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical detection technology, and in particular to a method and system for detecting small lesions of diabetic retinopathy based on fundus images. Background Art
[0002] The most common early clinical manifestations of diabetic retinopathy include microaneurysm formation and intraretinal hemorrhage. Microvascular damage leads to retinal capillary nonperfusion, cotton-wool spots, increased number of hemorrhages, venous abnormalities, and intraretinal microvascular abnormalities (IRMA). During this stage, increased vascular permeability leads to retinal thickening (edema) and / or exudates, which can cause central vision loss. The proliferative phase results in the proliferation of new vessels on the optic disc, retina and iris, and filtration angle. The diagnosis can be confirmed by funduscopy, or color fundus photography.
[0003] Traditional fundus lesion detection usually relies on manual examination and diagnosis by doctors, which is not only time-consuming and labor-intensive, but may also be affected by the doctor's experience and subjective judgment. There is a lack of real-time feedback and guidance mechanisms, and it is impossible to provide feedback and reminders on the subject's eye behavior. Data processing is usually performed after the examination, and patients need to wait to get the results. Summary of the invention
[0004] The present invention uses a deep learning model to identify lesions, improves the accuracy and consistency of identification, and combines it with a voice reminder system to monitor eye behavior in real time and issue reminders to patients.
[0005] The technical solution provided by the present invention is: a method for detecting small lesions of diabetic retinopathy based on fundus images, the method comprising:
[0006] Build an eye action recognition model to predict specific eye behaviors, remind the subject through voice, and automatically save multiple fundus images after specific eye behaviors occur;
[0007] Acquire multiple fundus images to form an image library;
[0008] Preprocess the fundus images in the image library and use the OpenCV library to detect the black areas of the images;
[0009] Calculate the black area, set the classification threshold, and classify multiple fundus images by the classification threshold to distinguish normal fundus images from abnormal fundus images;
[0010] The abnormal fundus images and the lesion detection results constitute a sample set;
[0011] Build a lesion recognition model, train the lesion recognition model through the sample set, and learn the relationship between the black area and the lesion in the fundus image;
[0012] The specific eye behaviors include one or more of blinking and eye movement.
[0013] Preferably, before building the eye action recognition model to predict specific eye behavior, the following steps are also included:
[0014] A high frame rate camera was used to capture each blink of the subject;
[0015] The fundus imaging device is used to collect fundus image 1 when the subject blinks each time and fundus image 2 when the subject does not blink each time;
[0016] A software trigger was used to ensure that the high frame rate camera and the fundus imaging device started capturing images at the same time.
[0017] Preprocessing the fundus image 1 and the fundus image 2 to form a fundus image set;
[0018] The fundus image 1 and the fundus image 2 are respectively provided with a timestamp 1 and a timestamp 2 for synchronization.
[0019] Preferably, the eye action recognition model is a CNN model constructed by a deep learning framework;
[0020] The prediction of specific eye behaviors includes:
[0021] The fundus image set is divided into a training set and a validation set, and the eye action recognition model is trained. During the training, the cross entropy loss function and the Adam optimizer are used to minimize the loss and update the parameters.
[0022] The trained eye action recognition model is used to predict the newly input fundus image 1 and / or fundus image 2, and the output layer will output the probability of blinking;
[0023] Set the probability threshold. If the output blink probability is greater than the probability threshold, it is judged as a positive class, that is, blinking.
[0024] Preferably, the method of reminding the subject by voice and automatically saving multiple fundus images after the specific eye behavior occurs includes the following steps:
[0025] Get the output of the eye action recognition model;
[0026] When the output of the eye action recognition model is determined to be a blink, the pre-stored reminder text is called;
[0027] The reminder text is converted into audio data using a natural language processing algorithm and output through an external voice output device to remind the subject not to blink;
[0028] Automatically collect and save multiple fundus images after blinking, and send the multiple fundus images to the cloud database.
[0029] Preferably, the acquiring of multiple fundus images to form an image library includes:
[0030] Obtain fundus images of different patients and at different times from the database;
[0031] Obtain the patient's corresponding medical records from the medical record system, and extract the corresponding medical records from the medical records through the natural language processing algorithm;
[0032] Add time tags and patient information tags to each fundus image, and correspond the time tags to the patient's medical records to facilitate analysis of the relationship between lesion changes and disease course;
[0033] An image library is established according to the patient information, and the fundus images related to the patient information are arranged in ascending order according to the time tags.
[0034] Preferably, the preprocessing of the fundus image in the image library and detecting the black area of the image using the OpenCV library includes:
[0035] Perform color correction on the fundus image and remove noise in the fundus image by using a Gaussian filter;
[0036] Use the cv2.threshold() function in OpenCV to set a global threshold and convert the fundus image into a binary image, where black areas correspond to low grayscale values;
[0037] Use OpenCV’s cv2.connectedComponent() or cv2.findContours() function on the binary image to mark and identify independent connected components, which correspond to potential black areas;
[0038] A screening threshold I is set, and connected domains with a number of pixels less than the screening threshold I are removed to screen out connected domains representing real black areas.
[0039] Preferably, the calculating the black area, setting a classification threshold, and classifying the multiple images by the classification threshold includes:
[0040] Use the connectedComponentsWithStats function to find connected components in a binary image;
[0041] Traverse each connected domain, calculate the number of pixels in each connected domain, that is, the area of the connected domain, and then output;
[0042] Set the classification threshold;
[0043] Traverse each connected domain again, compare the area of each connected domain with the classification threshold, and classify the fundus image according to the comparison result;
[0044] The classifying the fundus image according to the comparison result comprises:
[0045] If the area of the connected domain is larger than the set classification threshold, the corresponding fundus image is classified as an abnormal fundus image and a classification label of one is added;
[0046] If the area of the connected domain is smaller than the set classification threshold, the corresponding fundus image is classified as a normal fundus image, and classification label 2 is added;
[0047] Different fundus images are distinguished by classification label one and classification label two.
[0048] Preferably, the abnormal fundus images and lesion detection results are used to form a sample set, including:
[0049] Acquire multiple fundus images with classification label one;
[0050] Connecting to the medical record system, using the patient information tag of the abnormal fundus image, obtaining the case information corresponding to the patient information tag, and extracting the doctor's diagnosis result from the case information, the diagnosis result including the lesion type, lesion location and lesion severity;
[0051] The fundus images with classification label one and the corresponding diagnosis results constitute a sample set;
[0052] The step of building a lesion recognition model, training the lesion recognition model through a sample set, and learning the relationship between the black area in the fundus image and the lesion includes:
[0053] A lesion recognition model is constructed through a multi-layer convolutional neural network;
[0054] Performing standardization processing on abnormal fundus images in the sample set, wherein the standardization processing includes: resizing, normalization, contrast enhancement, and conversion into grayscale images;
[0055] Inputting the normalized abnormal fundus image into the input layer;
[0056] The important information of the connected domain is retained by using the activation function ReLU, wherein the important information of the connected domain includes the edge, shape and area of the connected domain;
[0057] After the convolutional layer, pooling layer, and fully connected layer, the output layer outputs the location and severity of the lesion.
[0058] The present invention also provides a system for detecting small lesions of diabetic retinopathy based on fundus images, and the system is used to execute the method for detecting small lesions of diabetic retinopathy based on fundus images.
[0059] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method for detecting small diabetic retinopathy lesions based on fundus images.
[0060] Beneficial effects of the present invention:
[0061] In the present invention, features related to lesions, such as the area, shape, and distribution of black areas, can be extracted from fundus images through image processing technology. Based on medical knowledge and statistical analysis, the research can determine the classification threshold for distinguishing normal and abnormal fundus images. For example, a threshold for the area of black areas can be set, and images exceeding this threshold are classified as abnormal. Using machine learning algorithms, fundus images are automatically classified to assist doctors in quickly screening out cases that require further examination.
[0062] In the present invention, by analyzing the pre-processed fundus images, combined with time tags and patient medical records, the changing trend of the lesions over time and their relationship with the patient's medical course can be studied. The pre-processed fundus images can be used to train deep learning models and other image analysis algorithms, thereby assisting doctors in the diagnosis and evaluation of diabetic retinopathy. The established image library can not only be used for long-term patient management and scientific research, but also provide rich data support.
[0063] In the present invention, the eye action recognition model predicts the newly input fundus image 1 and fundus image 2, and the output layer will output the probability of blinking. This probability is used to convert into a classification decision. If the output of the eye action recognition model is determined to be blinking, the system will call the pre-stored reminder text, which contains instructions, such as reminding the subject to keep his eyes open or avoid blinking, and convert the reminder text into audio data by using the text-to-speech (TTS) conversion technology in the natural language processing algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1 The present invention is a flow chart of a method for detecting small lesions of diabetic retinopathy based on fundus images. DETAILED DESCRIPTION
[0065] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.
[0066] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.
[0067] Embodiment 1:
[0068] Please combine Figure 1 The present invention provides a method for detecting small lesions of diabetic retinopathy based on fundus images, the method comprising:
[0069] Build an eye action recognition model that can identify and predict specific eye behaviors, such as blinking and eye movement. Use a deep learning framework, especially a convolutional neural network (CNN), to build the model. CNN is very suitable for processing image data and can automatically learn and extract complex features in images. After the model predicts a specific eye behavior, it reminds the subject through voice and automatically saves multiple fundus images after the behavior occurs for further analysis.
[0070] Collect a large number of fundus images to form an image library. These images should cover different eye conditions, including normal and diseased fundus. Make sure that the images in the image library have enough diversity to represent different stages and types of diabetic retinopathy.
[0071] Use the OpenCV library for image preprocessing. This is an open source computer vision library that contains a variety of image processing functions. Detect black areas in the image, which may correspond to abnormal points in the fundus image, such as microaneurysms or hemorrhages. Calculate the area of the detected black area, which is a key parameter for determining whether the image is normal. Set a classification threshold based on the area of the black area. This threshold will be used to distinguish between normal and abnormal fundus images. Use the set threshold to classify the fundus images in the image library into normal fundus images and abnormal fundus images. Put the fundus images classified as abnormal together with the lesion detection results to form a sample set. Ensure that each sample in the sample set is accurately labeled, including information such as lesion type and severity.
[0072] Then, a lesion recognition model is built, which is usually also a deep learning-based model, such as CNN. The model is trained using a sample set to learn the relationship between the black area and the lesion in the fundus image. Through the verification and testing steps in the training process, the model parameters are continuously optimized to improve the accuracy and generalization ability of the model. Specific eye behaviors may include one or more of blinking and eye movement. The recognition of these behaviors helps to capture fundus images at the right time and conduct real-time monitoring.
[0073] By combining eye movement recognition and fundus image analysis, the automated and efficient detection of diabetic retinopathy is achieved, improving the accuracy and efficiency of lesion detection and providing better diagnostic services for diabetic patients.
[0074] Fundus Image 1 and Fundus Image 2 have timestamps 1 and 2, respectively, which are key information for synchronization matching. These timestamps record the exact moment of image acquisition, ensuring that images from different devices can be accurately matched. The accuracy of the timestamps needs to be very high to ensure that the data from the high frame rate camera and the fundus imaging device can be accurately aligned in subsequent analysis. Synchronization of the two devices is achieved by using a shared clock source or synchronization trigger.
[0075] Before using fundus image 1 and fundus image 2 for analysis, they need to be preprocessed. This includes operations such as denoising, contrast enhancement, and brightness adjustment to improve image quality and highlight important features. The preprocessing operations must be consistent to ensure that no bias is introduced or the accuracy of synchronization matching is affected. For example, the same noise filtering and color correction processing is performed on the two images.
[0076] Specialized algorithms are used to match fundus image one and fundus image two based on timestamp one and timestamp two. These algorithms need to take into account the accuracy of the timestamps and be able to handle small time differences that may occur. The matching algorithm should be part of a software system that can automatically read the timestamp information, perform the matching, and save or mark the matched images for further analysis.
[0077] Fundus images can be stored locally, especially when there is no reliable network connection. Local storage needs to ensure the security and integrity of the data. Through IoT technology, fundus images can be transmitted to cloud servers.
[0078] The synchronously matched fundus images will be used to detect diabetic retinopathy. Synchronously matched fundus images can also be used to monitor the patient's fundus condition over a long period of time, track the development of lesions, evaluate treatment effects, or predict disease progression.
[0079] Through these steps, the synchronous matching of fundus image 1 and fundus image 2 can be ensured, thereby providing an accurate and reliable data basis for the detection of diabetic retinopathy. The effectiveness of this method depends largely on the accuracy of the timestamp and the reliability of the image preprocessing and matching algorithm.
[0080] A convolutional neural network (CNN) model was built using a deep learning framework. The fundus image set was divided into a training set and a validation set. The training set was used to train the model, while the validation set was used to evaluate the model performance. During the training process, the cross entropy loss function was used to calculate the difference between the model prediction value and the actual label. The cross entropy loss function is a commonly used loss function in classification problems, which can effectively measure the distance between two probability distributions. The Adam optimizer was used to minimize the loss function and perform parameter updates. Adam is an adaptive learning rate optimization algorithm that combines the advantages of gradient descent and momentum to effectively update network weights.
[0081] The trained eye action recognition model predicts the newly input fundus image 1 and / or fundus image 2, and the output layer will output the probability of blinking. This probability value represents the model's confidence that the current input image is in a blinking state. A probability threshold is set to convert the output blinking probability into a classification decision. If the output blinking probability is greater than the probability threshold, it is judged as a positive class, i.e. blinking.
[0082] If the output of the eye movement recognition model is determined to be blinking, the system will call the pre-stored reminder text. These texts usually contain instructions, such as reminding the subject to keep their eyes open or avoid blinking. The reminder text is converted into audio data through the text-to-speech (TTS) conversion technology in the natural language processing algorithm.
[0083] The generated audio data is output through an external voice output device, so as to remind the subject not to blink in the form of voice. This helps to guide the subject to cooperate with the examination during fundus examination or other occasions that require stable eye status. After reminding the subject not to blink, the multiple fundus images collected can be used to detect lesions.
[0084] Obtain fundus images of different patients at different times from the database, including the patient's multiple examinations and follow-up results. Through the natural language processing algorithm, extract the corresponding medical records from the case system. These records contain important information such as the patient's medical history, symptom changes, and treatment process. Add a time label and patient information label to each fundus image, and correspond the time label to the patient's medical record to facilitate subsequent analysis of the relationship between lesion changes and the course of the disease. Establish an image library based on patient information, and arrange the fundus images related to the patient information in ascending order according to the time label to form an ordered data set.
[0085] Perform color correction on fundus images to ensure color consistency and accuracy of the images. This helps reduce errors in subsequent processing and improves the accuracy of analysis. Remove noise from fundus images through Gaussian filters to improve image quality. Gaussian filters can effectively filter out noise while maintaining image details.
[0086] Use the cv2.threshold() function in OpenCV to set a global threshold and convert the fundus image into a binary image, where black areas correspond to low grayscale values. This step is to simplify the image and make subsequent black area detection easier.
[0087] Use OpenCV’s cv2.connectedComponent() or cv2.findContours() function on the binary image to mark and identify independent connected components, which correspond to the underlying black areas. Connected component marking helps in identifying and separating different objects or regions in the image.
[0088] A screening threshold I is set to remove connected domains with a number of pixels less than the screening threshold I, so as to screen out connected domains representing real black areas, thereby excluding those areas that are too small and unlikely to be real lesions.
[0089] By analyzing the preprocessed fundus images, combined with time tags and patient medical records, we can study the changing trend of the lesions over time and their relationship with the patient's disease course. The preprocessed fundus images can be used to train deep learning models or other image analysis algorithms to assist doctors in the diagnosis and assessment of diabetic retinopathy. The established image library can serve as a valuable resource for long-term patient management and scientific research, providing rich data support.
[0090] In this way, an image library is constructed and managed to improve the quality of fundus images and the accuracy of analysis, providing strong support for the diagnosis and treatment of diabetic retinopathy.
[0091] Use the connectedComponentsWithStats function in the OpenCV library to find connected domains in a binary image, mark the connected domains in the image, and provide statistical data for each connected domain, such as the bounding rectangle and area.
[0092] Traverse each connected domain, use the statistical data provided by the function to calculate the number of pixels in each connected domain, that is, the area of the connected domain, and output the result.
[0093] According to actual needs and experimental data, a classification threshold is set, which will be used to distinguish normal fundus images from abnormal fundus images.
[0094] Traverse each connected domain again and compare the area of each connected domain with the classification threshold. According to the comparison results, classify the fundus image. If the area of the connected domain is larger than the set classification threshold, it means that there is a large black area in the fundus image, which may indicate the presence of a lesion, so it is classified as an abnormal fundus image and the classification label 1 is added. If the area of the connected domain is smaller than the set classification threshold, it means that the black area is small, which may be a normal fundus structure or noise, so it is classified as a normal fundus image and the classification label 2 is added.
[0095] Different fundus images can be clearly distinguished by classification label 1 and classification label 2. This helps in subsequent image management, analysis and processing. The marked fundus images can be more conveniently grouped and analyzed, which is of great significance for research and clinical applications.
[0096] The classified fundus images can be used to train deep learning models or other image analysis algorithms to further identify and evaluate diabetic retinopathy.
[0097] Through regular fundus examinations and image classification, changes in the patient's condition can be monitored and treatment plans can be adjusted in a timely manner.
[0098] Through this process, normal fundus images and abnormal fundus images can be effectively distinguished, providing important reference information for the diagnosis and treatment of diabetic retinopathy.
[0099] Fundus images with classification label 1 are collected from the preprocessing and classification process. These images have been identified by the system as abnormal fundus images with potential pathological features.
[0100] When acquiring these images, their quality and usability need to be ensured for subsequent analysis. This includes checking the image clarity, completeness, and whether there is enough information for subsequent lesion detection and analysis. Patient information labels are used for abnormal fundus images. These labels contain basic patient information such as name, ID number, etc.
[0101] By connecting to the case system, the case information corresponding to each fundus image is obtained using the patient information tag. This case information may include the patient's medical history, symptoms, treatment process, and other relevant medical records. The doctor's diagnosis results are extracted from the case information, especially the description of the type of lesion. The type of lesion may include common manifestations of diabetic retinopathy such as microaneurysms, hemorrhage, and macular edema.
[0102] In addition to lesion type, it is also necessary to extract information about the location and severity of the lesion. This information is crucial for understanding the development of the lesion and formulating a treatment plan.
[0103] The fundus images with classification label one and the diagnosis results extracted from the case system are combined to form a sample set. Each sample in this sample set contains fundus image data and corresponding clinical diagnosis information.
[0104] Ensure that each sample in the sample set is correctly labeled and the information is complete. This includes the annotation of fundus images (such as lesion area) and the organization of diagnostic results (such as coding of lesion type, location and severity).
[0105] The sample set can be used to train deep learning models, enabling them to learn the characteristics of lesions and improve their accuracy in lesion detection and classification. The sample set can also be used in clinical research to help doctors and researchers better understand the development of diabetic retinopathy and explore new treatments. The constructed sample set can be shared with other research institutions as a valuable resource to promote the development of medical image analysis and diabetic retinopathy research.
[0106] The lesion recognition model is built based on a multi-layer convolutional neural network (CNN). This network structure is particularly suitable for image data and can effectively extract image features. The model includes an input layer, multiple convolutional layers, a pooling layer, a fully connected layer, and an output layer. Each layer has its specific function and works together to achieve lesion recognition and location. The abnormal fundus images in the sample set are standardized to ensure the consistency of the input data and the stability of the model training.
[0107] The fundus image is resized to match the model input requirements and normalized so that the pixel values are in a uniform range, such as [0, 1] or [-1, 1]. The contrast is enhanced to highlight the details in the image, and the conversion to grayscale image can reduce the amount of calculation and remove the interference of color information. The normalized abnormal fundus image is input to the input layer of the model to start the training process. ReLU (Rectified Linear Unit) is used as the activation function in each layer of the model to retain important information of the connected domain, including the edge, shape and area of the connected domain. The ReLU function can increase the nonlinearity of the network while avoiding the gradient vanishing problem. Through the combination of multiple convolutional layers and pooling layers, the model can gradually extract high-level features in the fundus image, which are closely related to the identification of lesions, such as specific lesion types and distribution. After the convolutional and pooling layers, the fully connected layer summarizes these high-level features in preparation for the final lesion identification and classification. The output layer outputs the location and severity of the lesion. This usually involves the classification of lesion types, the precise location of lesion areas, and the assessment of lesion severity.
[0108] Define an appropriate loss function to measure the difference between the model prediction and the actual label, and select a suitable optimizer to minimize this loss. Use the sample set to train the model, and test the performance of the model on the validation set to evaluate its generalization ability. The trained model can be applied in clinical practice to assist doctors in the identification and evaluation of fundus lesions and improve the accuracy and efficiency of diagnosis.
[0109] Embodiment 2:
[0110] The present invention also provides a technical solution: a method for detecting small lesions of diabetic retinopathy based on fundus images. The difference from the first embodiment is that the lesion recognition model is a YOLOv5 model. On the basis of YOLOv5, SIoU, shallow convolution detection, better feature fusion of BiFPN and DyHead are introduced. In the original structure, the smallest feature map that the detection head can detect is 1 / 8 of the original input, which means that a considerable number of small targets cannot be detected at all. For the loss function, this embodiment uses SIoU to modify the YOLOv5 loss function. part to solve the problem that slight movement will cause large changes in the loss function.
[0111] The solution of this embodiment achieved good results in the AUC of hard exudate, soft exudate, hemorrhage and microaneurysm. For details, please refer to Table 1:
[0112] Table 1
[0113]
[0114] The embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. The embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wire segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical cable, RF, etc., or any suitable combination of the foregoing.
[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0116] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be deformed or modified in any way.
Claims
1. A method for detecting small lesions of diabetic retinopathy based on fundus images, characterized in that: The method comprises: Build an eye action recognition model to predict specific eye behaviors, remind the subject through voice, automatically save multiple fundus images after specific eye behaviors occur, and send multiple fundus images to a cloud database; Acquire multiple fundus images to form an image library; specifically: acquire fundus images of different patients and at different times from the database; acquire the corresponding case of the patient from the case system, and extract the corresponding medical records from the case through the natural language processing algorithm; add a time label and a patient information label to each fundus image, and correspond the time label to the patient's medical record, so as to analyze the relationship between the change of the lesion and the course of the disease; establish an image library based on the patient information, and arrange the fundus images related to the patient information in ascending order according to the time label; Preprocess the fundus images in the image library and use the OpenCV library to detect the black areas of the images; Calculate the black area, set the classification threshold, and classify multiple fundus images by the classification threshold to distinguish normal fundus images from abnormal fundus images; The abnormal fundus image and the lesion detection result constitute a sample set; the step of obtaining the lesion detection result is: connecting to the case system, using the patient information tag of the abnormal fundus image, obtaining the case information corresponding to the patient information tag, and extracting the doctor's diagnosis result from the case information, the diagnosis result including the lesion type, lesion location and lesion severity; Build a lesion recognition model, train the lesion recognition model through the sample set, and learn the relationship between the black area and the lesion in the fundus image; The specific eye behaviors include one or more of blinking and eye movement.
2. The method for detecting small diabetic retinopathy lesions based on fundus images according to claim 1, characterized in that: Before building an eye action recognition model to predict specific eye behaviors, the following steps are also included: A high frame rate camera was used to capture each blink of the subject; The fundus imaging device is used to collect fundus image 1 when the subject blinks each time and fundus image 2 when the subject does not blink each time; Use software triggers to ensure that the high frame rate camera and fundus imaging device start capturing images at the same time; Preprocessing the fundus image 1 and the fundus image 2 to form a fundus image set; The fundus image 1 and the fundus image 2 are respectively provided with a timestamp 1 and a timestamp 2 for synchronization.
3. The method for detecting small diabetic retinopathy lesions based on fundus images according to claim 2, characterized in that: The eye action recognition model is a CNN model built through a deep learning framework; The prediction of specific eye behaviors includes: The fundus image set is divided into a training set and a validation set, and the eye action recognition model is trained. During the training, the cross entropy loss function and the Adam optimizer are used to minimize the loss and update the parameters. The trained eye action recognition model is used to predict the newly input fundus image 1 and / or fundus image 2, and the output layer will output the probability of blinking; Set the probability threshold. If the output blink probability is greater than the probability threshold, it is judged as a positive class, that is, blinking.
4. The method for detecting small diabetic retinopathy lesions based on fundus images according to claim 3, characterized in that: The method of reminding the subject by voice and automatically saving multiple fundus images after a specific eye behavior occurs includes the following steps: Get the output of the eye action recognition model; When the output of the eye action recognition model is determined to be a blink, the pre-stored reminder text is called; The reminder text is converted into audio data using a natural language processing algorithm and output through an external voice output device to remind the subject not to blink; Automatically acquire and save multiple fundus images after blinking.
5. The method for detecting small diabetic retinopathy lesions based on fundus images according to claim 1, characterized in that: The method of preprocessing the fundus image in the image library and detecting the black area of the image using the OpenCV library includes: Perform color correction on the fundus image and remove noise in the fundus image by using a Gaussian filter; Use the cv2.threshold() function in OpenCV to set a global threshold and convert the fundus image into a binary image, where black areas correspond to low grayscale values; Use OpenCV’s cv2.connectedComponent() or cv2.findContours() function on the binary image to mark and identify independent connected components, which correspond to potential black areas; A screening threshold I is set, and connected domains with a number of pixels less than the screening threshold I are removed to screen out connected domains representing real black areas.
6. The method for detecting small diabetic retinopathy lesions based on fundus images according to claim 5, characterized in that: The calculating the black area, setting the classification threshold, and classifying the multiple images by the classification threshold include: Use the connectedComponentsWithStats function to find connected components in a binary image; Traverse each connected domain, calculate the number of pixels in each connected domain, that is, the area of the connected domain, and then output; Set the classification threshold; Traverse each connected domain again, compare the area of each connected domain with the classification threshold, and classify the fundus image according to the comparison result; The classifying the fundus image according to the comparison result comprises: If the area of the connected domain is larger than the set classification threshold, the corresponding fundus image is classified as an abnormal fundus image and a classification label of one is added; If the area of the connected domain is smaller than the set classification threshold, the corresponding fundus image is classified as a normal fundus image, and classification label 2 is added; Different fundus images are distinguished by classification label one and classification label two.
7. The method for detecting small diabetic retinopathy lesions based on fundus images according to claim 1, characterized in that: The abnormal fundus image and the lesion detection result are used to form a sample set, including: Acquire multiple fundus images with classification label one; The fundus images with classification label one and the corresponding diagnosis results constitute a sample set; The step of building a lesion recognition model, training the lesion recognition model through a sample set, and learning the relationship between the black area in the fundus image and the lesion includes: A lesion recognition model is constructed through a multi-layer convolutional neural network; Performing standardization processing on abnormal fundus images in the sample set, wherein the standardization processing includes: resizing, normalization, contrast enhancement, and conversion into grayscale images; Inputting the normalized abnormal fundus image into the input layer; The important information of the connected domain is retained by using the activation function ReLU, wherein the important information of the connected domain includes the edge, shape and area of the connected domain; After the convolutional layer, pooling layer, and fully connected layer, the output layer outputs the location and severity of the lesion.
8. A small lesion detection system for diabetic retinopathy based on fundus images, characterized in that: The system is used to execute the method for detecting small diabetic retinopathy lesions based on fundus images as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method for detecting small diabetic retinopathy lesions based on fundus images as described in any one of claims 1 to 7.
Citation Information
Patent Citations
An automatic analysis and comparison method of fundus images and a storage device
CN109166117A
Method and device for identifying diabetic retinopathy based on cloud computing
CN113902743A
Fundus image classification model training method, fundus image classification method and fundus image classification system
CN114926460A
Image recognition method based on deep neural network
CN116452571A
Diabetic retinopathy recognition system based on fundus image
US20200160521A1
