Fundus image processing method, device and equipment and storage medium

Abstract

The application relates to the technical field of image detection, and discloses an eye fundus image processing method, device and equipment and a storage medium. The method comprises the following steps: outputting prompt information to a patient terminal, photographing an eye to be detected and generating an eye fundus image when the patient operates according to the prompt information; determining a clipping boundary based on a preset clipping rule, clipping the eye fundus image and obtaining an eye fundus sub-image; identifying a disc edge in the eye fundus sub-image based on a random forest model to obtain an edge image; performing fitting processing on the edge image based on a RANSAC algorithm iteration, using contour matching to combine the fitting result with the most inner points in the iteration process to obtain a smooth disc edge and position a disc center to obtain a center image; receiving doctor's order information output by a doctor according to the center image through a doctor terminal and uploading the doctor's order information to a front-end server, so that the patient terminal can receive the doctor's order information from the front-end server. The application can improve the efficiency of the eye fundus image in the detection process and the recognition process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image detection, in particular to an eye fundus image processing method and device, equipment and storage medium. BACKGROUND

[0002] Diabetic Retinopathy (DR) caused by diabetes is a complication of diabetic microangiopathy, and is one of the most common and most serious complications of diabetes.

[0003] At present, the screening and diagnosis of DR mainly rely on ophthalmologists using fundus imaging technology. Under the instruction of the detection personnel, the patient takes a fundus photograph by using a fundus camera, projects the three-dimensional structure of the retina onto a two-dimensional plane image, and the ophthalmologist identifies the relevant retinal vascular lesion characteristics (such as diabetic microaneurysm, hemorrhagic spot, hard exudate, etc.) by observing and analyzing the fundus photograph, thereby realizing the screening and staging of DR.

[0004] Both the detection process and the identification process require on-site assistance of professionals, which is inefficient. SUMMARY

[0005] The purpose of the present application is to provide an eye fundus image processing method, device, equipment and storage medium, which aims to improve the efficiency of the eye fundus image in the detection process and the identification process.

[0006] The present application provides an eye fundus image processing method, which comprises:

[0007] outputting prompt information to a patient terminal, and taking a to-be-tested eye and generating an eye fundus image when the patient operates according to the prompt information;

[0008] determining a clipping boundary based on a preset clipping rule, clipping the eye fundus image and obtaining an eye fundus sub-image; the clipped image includes an optic disc and an edge region around the optic disc;

[0009] identifying the edge of the optic disc in the eye fundus sub-image based on a random forest model to obtain an edge image;

[0010] performing fitting processing on the edge image based on a RANSAC algorithm iteration, using contour matching to combine the fitting result with the most inliers in the iteration process to obtain a smooth optic disc edge and locate the center of the optic disc, and obtaining a center image;

[0011] receiving doctor's order information output by a doctor according to the center image through a doctor terminal and uploading the information to a front-end server, so that the patient terminal can receive the doctor's order information from the front-end server.

[0012] Further, the outputting prompt information to the patient terminal and taking the to-be-tested eye and generating the eye fundus image when the patient operates according to the prompt information comprises:

[0013] receiving an interaction request of the patient terminal, outputting prompt information to the patient terminal;

[0014] receiving an operation signal generated by the patient according to the prompt information, and capturing the fundus according to the operation signal to obtain a fundus image;

[0015] uploading the fundus image, and performing desensitization, enhancement, and labeling processing on the fundus image.

[0016] Further, the determining of the clipping boundary based on the preset clipping rule, the clipping of the fundus image, and the obtaining of the fundus sub-image include:

[0017] estimating the optic disc radius according to a proportional relationship between the fundus radius and the optic disc radius;

[0018] locating a key pixel point of the optic disc in the fundus image, and clipping the fundus image based on the key pixel point as the center and the estimated optic disc radius to obtain the fundus sub-image.

[0019] Further, the identifying of the optic disc edge in the fundus sub-image based on the random forest model includes:

[0020] sliding a sliding window over the entire fundus sub-image based on a sliding window algorithm, identifying the optic disc edge in the image in the sliding window region based on the random forest model to obtain an edge image.

[0021] Further, before the identifying of the optic disc edge in the fundus sub-image based on the random forest model, includes:

[0022] randomly clipping a plurality of image blocks from the fundus sub-image, extracting a fixed number of channel feature vectors from the image blocks, and generating a plurality of feature maps based on the channels of the feature vectors;

[0023] performing first downsampling processing on the feature maps to obtain a first downsampled image, using the pixels of the first downsampled image as candidate features to obtain first candidate features;

[0024] performing second downsampling processing on the feature maps to obtain a second downsampled image, using the difference between the pixels in the second downsampled image as candidate features to obtain second candidate features;

[0025] using the first candidate features and the second candidate features as training samples, using the fundus area where the segment corresponding to the training sample is located as the expected output, training a plurality of decision trees in the random forest classifier to form a random forest model.

[0026] Further, the iterative fitting processing of the edge image based on the RANSAC algorithm, the use of contour matching to combine the fitting result with the most inliers in the iteration process to obtain a smooth optic disc edge and locate the center of the optic disc, includes:

[0027] The edge image is binarized to obtain a binary image.

[0028] The fitting step is iteratively performed to generate sample estimation models until the sample estimation model with the largest number of inliers appears or a maximum number of iterations is reached.

[0029] The fitting step comprises:

[0030] Based on the least square method, a sample estimation model is generated using a plurality of pixel points randomly selected from the binary image; the sample estimation model is a circle model.

[0031] The number of inliers in the sample estimation model is estimated based on a set threshold.

[0032] The calculation formula of the maximum number of iterations is:

[0033]

[0034] N = P * p / λ min is the maximum number of iterations, P is the confidence level, p is the proportion of inliers in the randomly selected pixel points, and λ is the number of randomly selected pixel points.

[0035] Further, the doctor terminal receives the doctor's order information output according to the center image and uploads it to the front-end server, comprising:

[0036] Output the center image to the doctor terminal;

[0037] Receive the first text information and / or the second text information; the first text information is the text content obtained by analyzing the voice information entered by the doctor for the center image using the sound pickup device, and the second text information is the text content corresponding to the text information generated by the doctor for the center image;

[0038] Retrieve the corresponding pathological diagnosis report template, and generate a pathological diagnosis report according to the first text information and / or the second text information.

[0039] The application also provides an eye fundus image processing device, comprising:

[0040] The first module is configured to output prompt information to the patient terminal, and to capture the eye to be measured and generate an eye fundus image when the patient operates according to the prompt information;

[0041] The second module is configured to determine a clipping boundary based on a preset clipping rule, clip the eye fundus image, and obtain an eye fundus sub-image; the clipped image includes the optic disc and the edge region around the optic disc.

[0042] The third module is configured to identify the optic disc edge in the eye fundus sub-image based on a random forest model, and obtain an edge image.

[0043] The fourth module is configured to perform fitting processing on the edge image based on a RANSAC algorithm iteration, use contour matching to combine the fitting result with the most inliers in the iteration process to obtain a smooth optic disc edge and locate the optic disc center, and obtain a center image;

[0044] The fifth module is configured to receive, by the doctor terminal, doctor order information output by the doctor according to the center image and upload the doctor order information to the front-end server, so that the patient terminal can receive the doctor order information from the front-end server.

[0045] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the fundus image processing method according to any one of the above when executing the computer program.

[0046] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the fundus image processing method according to any one of the above.

[0047] The present application has the beneficial effects that: the patient is prompted to self-collect the fundus image, the eye to be detected is photographed and the fundus image is generated when the patient operates according to the prompt information, the examination time of the patient is reduced, the fundus image is automatically recognized and processed, the optic disc in the fundus image is extracted based on the structure random forest and the RANSAC algorithm, the optic disc can be better positioned and extracted, the false positive in the detection of hard exudation caused by DR is reduced, the doctor can quickly obtain the fundus image and provide relevant doctor order information, the interrogation time of the doctor is reduced, and the efficiency of the fundus image in the detection process and the recognition process is improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the fundus image processing method provided by the embodiment of the present application.

[0049] Figure 2 is Figure 1 is a flowchart of step S101 in

[0050] Figure 3 is Figure 1 is a flowchart of step S102 in

[0051] Figure 4 is Figure 1 is a flowchart before step S102 in

[0052] Figure 5 is Figure 1 is a flowchart of step S104 in

[0053] Figure 6 isFigure 1 the flowchart of step S105 in FIG. 10.

[0054] Figure 7 is a structural schematic diagram of an eye fundus image processing device provided by an embodiment of the present application.

[0055] Figure 8 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the present application will be further described below with reference to the embodiments and drawings.

[0057] In the embodiments of the present application, the meaning of several is indefinite quantity, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If the first, second is described only for the purpose of distinguishing technical features, it cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0058] In the embodiments of the present application, the term "comprising", "containing" or any other variant thereof is intended to cover non-exclusive containing, in addition to containing the listed elements, other elements not explicitly listed can also be contained.

[0059] Diabetic retinopathy (DR) is a complication of diabetic microangiopathy, and is one of the most common and most serious complications of diabetes. It has become the primary cause of blindness in developed countries. The diagnosis and treatment scheme of DR focuses on timely detection, early treatment and effective control, and too late detection can lead to irreversible visual impairment or even loss. The prevalence of diabetes in China is 11.6%, and 1.14 million diabetic patients need to check the fundus every year to achieve the purpose of "timely detection, early treatment and effective control". However, many patients have long diabetes, blood sugar levels have not been well controlled, have not good living habits, and lack of understanding of DR, when visual impairment occurs, the best treatment opportunity has been missed. Therefore, the fundus screening work of diabetic patients and the diagnosis and treatment work of DR patients are of great significance.

[0060] In the related art, the screening and diagnosis of DR mainly rely on ophthalmologists using fundus imaging technology. The screening and diagnosis of DR mainly rely on ophthalmologists using fundus imaging technology. Fundus imaging technology is a kind of fast, simple, efficient and feasible non-invasive diagnosis scheme in reality. The patient takes a fundus photograph under the instruction of the tester through the fundus camera, projects the three-dimensional structure of the retina into a two-dimensional plane image, and the ophthalmologist observes and analyzes the fundus photograph to identify the relevant retinal vascular lesions (such as diabetic microaneurysms, hemorrhagic spots, and hard exudates). Thus, the screening and staging of DR are realized. However, the diagnosis and treatment process of DR still has the following problems:

[0061] Firstly, there are only 35,000 ophthalmologists in China, which is difficult to meet the demand of 114 million diabetic patients for fundus examination every year. However, it is a highly professional task to determine whether DR occurs in the fundus photograph, and general practitioners, internal medicine physicians and even endocrinology physicians are also difficult to do it. Therefore, the contradiction between the huge DR screening workload and the shortage of ophthalmologists is a problem that needs to be solved urgently.

[0062] Secondly, the fundus camera for collecting fundus photographs is a professional medical device, and the patient needs to use it under the guidance of medical staff, which cannot be completed by the patient himself, resulting in low efficiency of the screening process.

[0063] Thirdly, the data volume of the fundus photograph is huge, and it involves patient privacy information, so the storage capacity, security and confidentiality of the fundus photograph database are extremely high, and the local database established by local medical institutions generally cannot meet the requirements.

[0064] Fourthly, the traditional ophthalmologist interprets the fundus photograph of the patient, and writes the medical record and talks to the patient to explain the medical advice and treatment plan, which is low in efficiency.

[0065] Based on this, the embodiment of the application provides a fundus image processing method, device, equipment and storage medium, which provides a prompt for the patient to self-collect the fundus image, automatically identifies and processes the fundus image, so that the doctor can quickly obtain the fundus image and provide related medical advice information, and improves the efficiency of the fundus image in the detection process and the identification process.

[0066] The fundus image processing method, device, equipment and storage medium provided by the embodiment of the application are specifically described by the following embodiments. First, the fundus image processing method of the embodiment of the application is described.

[0067] Figure 1 is an optional flowchart of the fundus image processing method provided by the embodiment of the application, Figure 1 The method in the flowchart can include but is not limited to steps S101-S105.

[0068] Step S101, output prompt information to the patient terminal, in the patient according to the prompt information for operation to take the eye and generate fundus image.

[0069] In step S101, the fundus image acquisition process is that the patient uses the terminal (patient terminal) to obtain the preset prompt information, for example, the patient terminal can be connected with the fundus camera to receive the prompt information sent by the fundus camera, or the patient terminal can log in the preset application program or link to obtain the prompt information. The prompt information is used to instruct the patient to operate the fundus camera to take the eye to be tested. When the patient operates the fundus camera to take the eye to be tested according to the prompt information, the fundus camera generates the fundus image.

[0070] It can be understood that the fundus photography to obtain the fundus image is a way to examine the fundus, which can help us see the organization structure of the fundus. The fundus is like the film of the camera, and it is the only part of the whole body that can be directly and centrally observed by naked eye, which can reflect the dynamics and health status of the whole body blood circulation. Many systemic diseases can be reflected on the fundus, such as fundus hemorrhage, which is a serious complication of diabetes, hypertension, coronary heart disease, and kidney disease.

[0071] The fundus image includes the following structures:

[0072] Optic disc (OD), full name optic disc, also called optic nerve head, retinal macula to the nasal side about 3mm has a diameter of about 1.5mm, clear boundary of light red disc structure, called optic disc, simply called optic disc, like a plate.

[0073] Optic cup (OC), the optic cup is a normal physiological depression of the optic disc, like a cup. The ratio of the diameter of the cup to the disc is called the cup-to-disc ratio, that is, C / D. The normal C / D is about 0.3 and generally does not exceed 0.5.

[0074] Retinal central artery and vein, large blood vessels from the optic disc and radiating. Among them, the artery is bright red, and the vein is dark red. The ratio of the diameter of the artery to the vein is 2:3.

[0075] Macular part, located in the posterior pole of the eyeball, about the size of an optic disc or slightly larger, without blood vessels, with a pinhead-sized light reflection point in the center called the foveal light reflection.

[0076] Retina, the retina is a thin film of neural tissue, which is the innermost layer of the eyeball wall. The normal retina is colorless and transparent, because the choroid under it is full of blood vessels, and the retina is uniformly dark red when the fundus of both eyes is photographed.

[0077] In step S102, the key pixel point of the optic disc in the fundus image is located, the fundus image is cropped with the key pixel point as the center of the cropping region, the boundary and the cropping size of the cropping region are determined based on a preset cropping rule, and the fundus sub-image is obtained by cropping, for example, the cropping rule can be to crop the fundus sub-image with a height of h pixels and a width of w pixels, i.e., the size is h x w, so that only the optic disc region accounting for a small part of the entire fundus image is cropped out, and the defect of reduced recognition efficiency and accuracy caused by directly recognizing the optic disc in the complete fundus image is avoided.

[0078] In step S102, the key pixel point of the optic disc in the fundus image is located, the fundus image is cropped with the key pixel point as the center of the cropping region, the boundary and the cropping size of the cropping region are determined based on a preset cropping rule, and the fundus sub-image is obtained by cropping, for example, the cropping rule can be to crop the fundus sub-image with a height of h pixels and a width of w pixels, i.e., the size is h x w, so that only the optic disc region accounting for a small part of the entire fundus image is cropped out, and the defect of reduced recognition efficiency and accuracy caused by directly recognizing the optic disc in the complete fundus image is avoided.

[0079] In step S103, the edge of the optic disc in the fundus sub-image is recognized based on the random forest model to obtain an edge image.

[0080] In step S103, when the fundus sub-image is classified to recognize the edge of the optic disc, the features of each region of the fundus sub-image are usually extracted first to obtain the feature vectors of all regions of the image, then the feature vectors of all regions are combined to obtain the feature vector of the fundus sub-image, and the feature vector is input into the first layer node of each decision tree of the random forest model. The next node to be traversed is determined according to the feature vector and the split dimension and split condition of each layer node, and the next node is determined. The judgment is continued in the next node, and so on, until the last layer node of the random forest model is determined, so as to determine the classification result of the image in each decision tree. The final classification result of the image is determined according to the classification results of the multiple decision trees.

[0081] The random forest model is trained by the feature vectors extracted from the fundus sub-image to recognize the feature patterns in the fundus sub-image. The random forest model is composed of a series of different decision trees. The output of each decision tree is a local segmentation of the edge of the optic disc target. The output of the random forest model is the average value of the results obtained by each decision tree for local segmentation.

[0082] In step S104, the edge image is fitted based on the RANSAC algorithm, the smooth optic disc edge is obtained by using contour matching combined with the fitting result with the most inliers in the iteration process, and the center of the optic disc is located to obtain a center image.

[0083] In step S104, the located optic disc center is taken as a fitting target, and the edge image is iteratively fitted based on the RANSAC algorithm. The fitting is iteratively performed until the best model is found or the number of iterations reaches a preset threshold. Finally, for all points suitable for the fitting model, the parameters of the specific model are calculated using the least squares method. According to the calculated model parameters, the optimal fitting model is obtained as a fitting result. The determination of the optic disc radius and the fundus radius is completed by combining contour matching. The optic disc radius is estimated according to the certain proportional relationship between the optic disc radius and the fundus radius, so as to locate the center point of the optic disc and obtain the center image.

[0084] The RANSAC (Random Sample Consensus) algorithm is an iterative method based on random sampling, which is commonly used to estimate the model parameters of data containing noise. The RANSAC algorithm is an iterative algorithm for correctly estimating the parameters of a mathematical model from a set of data containing outliers. Outliers generally refer to noise in the data, such as false matches in matching and outliers in estimated curves. In the process of fitting a circle, the RANSAC algorithm can be used to robustly fit a circle.

[0085] In step S105, the doctor receives the doctor's order information output by the center image through the doctor's terminal and uploads it to the front-end server, so that the patient terminal can receive the doctor's order information from the front-end server.

[0086] In step S105, the doctor accesses the system through the doctor's terminal to receive the center image, checks the center image on the doctor's terminal and makes a diagnosis, forms the doctor's order information, and uploads the doctor's order information to the front-end server through the doctor's terminal, so that the patient can receive the doctor's order information from the front-end server through the patient's terminal.

[0087] The steps S101 to S105 shown in the embodiments of the present application provide a prompt for the patient to self-collect the fundus image. When the patient operates according to the prompt information, the eye to be tested is photographed and the fundus image is generated, reducing the examination time of the patient. In combination with automatic recognition and processing of the fundus image, the optic disc in the fundus image is extracted based on the structure random forest and the RANSAC algorithm, which can better locate and extract the optic disc and reduce false positives when detecting hard exudation caused by DR. This enables the doctor to quickly obtain the fundus image and provide relevant doctor's order information, reducing the doctor's inquiry time and improving the efficiency of the fundus image in the detection process and the recognition process.

[0088] Please refer to Figure 2 In some embodiments, step S101 includes but is not limited to steps S201 to S203.

[0089] In step S201, the interactive request of the patient terminal is received, and prompt information is output to the patient terminal.

[0090] Step S202, receiving the operation signal generated by the patient according to the prompt information, and shooting the eye to be tested according to the operation signal to obtain the fundus image.

[0091] Step S203, uploading the fundus image, and performing desensitization, enhancement and labeling processing on the fundus image.

[0092] In step S201, the interaction request of the patient terminal is received, and the prompt information is output to the patient terminal when interacting with the patient terminal. The prompt information is used to instruct the patient to operate the fundus camera to shoot the eye to be tested, so that the patient can operate the fundus camera to shoot the fundus image according to the prompt information. For example, the front-end server and the fundus camera can be preset, the patient sends an interaction request to the front-end server through the patient terminal, and the front-end server receives the interaction request and interacts with the patient terminal sending the interaction request and outputs the prompt information. For another example, only the fundus camera can be preset, the patient sends an interaction request to the fundus camera through the patient terminal, and the fundus camera receives the interaction request and interacts with the patient terminal sending the interaction request and outputs the prompt information.

[0093] In step S202, the preset fundus camera receives the operation signal generated when the patient operates and performs corresponding functions according to the operation signal. When the patient operates according to the prompt information, the fundus image is projected on the photosensitive element of the camera by infrared light or white light imaging, and then image enhancement and correction are performed by image processing software, so that a high-quality fundus image is finally obtained.

[0094] In this embodiment, the fundus camera adopts infrared light imaging mode for shooting. Infrared light imaging is mainly used for checking fundus diseases, diabetic retinopathy, macular degeneration and other various fundus diseases. Infrared light is absorbed by the opaque tissue in the fundus. If there is blood exudation or some diseases in the fundus, these diseases will reflect infrared light. This reflection is emitted from slightly different angles, and the reflected light is usually different at different angles. Therefore, by adjusting the angle of the camera and the intensity of the light and other factors, more information and better resolution can be obtained on the fundus image.

[0095] In step S203, the fundus image is uploaded to the shared database, and after the uploaded fundus image is desensitized, enhanced and labeled, it is stored in the shared database. The shared database can be queried and saved by the user terminal and the doctor terminal after verifying the identity.

[0096] The desensitization process aims to remove sensitive information in the fundus color photograph data, reduce the interference of missing values, noise and irrelevant information in the data, improve the quality of the fundus color photograph data, and enhance the process to facilitate the training and testing of the random forest model. The labeling process aims to complete registration and facilitate the establishment of a cloud-based fundus color photograph sharing database. The sharing database in the module uses cloud platform object storage technology to deploy the sharing database on the cloud platform, saving local storage space of the device. Full consideration is given to the adaptability of the data set design and clinical applicability, with standardized standard control to meet the storage requirements of large capacity, long time, high security and high confidentiality of fundus image data.

[0097] Please refer to Figure 3 In some embodiments, step S102 includes but is not limited to steps S301 to S302.

[0098] Step S301 estimates the optic disc radius according to the proportional relationship between the fundus radius and the optic disc radius.

[0099] Step S302 locates the key pixel point of the optic disc in the fundus image, and cuts the fundus image with the key pixel point as the center and the estimated optic disc radius to obtain a fundus sub-image.

[0100] In step S301, by observing the length of the fundus radius and the length of the optic disc radius in a plurality of fundus images, the proportional range between the fundus radius and the optic disc radius is determined, so that the length of the optic disc radius is estimated according to the length of the fundus radius when the fundus image is obtained. For example, it can be estimated as the radius of the optic disc by using 1 / 7 of the long semi-axis of the fundus.

[0101] In step S302, the key pixel point of the optic disc in the fundus image is located using an adaptive correlation filter, the area covered by the optic disc is determined according to the estimated optic disc radius, and the fundus image is cut with the key pixel point as the center of the fundus sub-image, thereby cutting out the fundus sub-image which can completely cover the size of the optic disc and take the key pixel point as the center.

[0102] The process of locating the key pixel point of the optic disc by the adaptive correlation filter is that the pixel points of the fundus image are input into the adaptive correlation filter as input signals, the adaptive correlation filter with adjustable parameters generates an output signal, which is compared with the expected signal to form an error signal, the filter parameters are adjusted by an adaptive algorithm, and finally the mean square value is minimized. The filter parameters obtained at the previous moment are used to automatically adjust the filter parameters at the current moment to adapt to the unknown or time-varying statistical characteristics of the signal and noise, so as to realize optimal filtering and locate the key pixel point of the optic disc in the fundus image from the filtering result.

[0103] ​​In some embodiments, step S103 comprises: sliding a sliding window across the entire fundus sub-image based on a sliding window algorithm, identifying the optic disc edge in the image within the sliding window region using a random forest model, and obtaining an edge image.

[0104] Specifically, a sliding window with a size of 16x16 is preset, the sliding window is used to slide across the entire fundus sub-image, the fundus sub-image is sampled using the sliding window model, the sampling image content within the sliding window is classified using the trained random forest model, the edge of the optic disc is determined based on the classification result, and an edge image is obtained.

[0105] Please refer to Figure 4 In some embodiments, before step S102, steps S401 to S404 can also be included, but are not limited to.

[0106] Step S401, randomly crop a plurality of image blocks from the fundus sub-image, extract a fixed number of channel feature vectors from the image blocks, and generate a plurality of feature maps based on the channels of the feature vectors.

[0107] Step S402, first down-sampling processing is performed on the feature map to obtain a first down-sampled image, and the pixels of the first down-sampled image are used as candidate features to obtain a first candidate feature.

[0108] Step S403, second down-sampling processing is performed on the feature map to obtain a second down-sampled image, and the difference between the pixels in the second down-sampled image is used as a candidate feature to obtain a second candidate feature.

[0109] Step S404, the first candidate feature and the second candidate feature are used as training samples, the fundus region where the segment corresponding to the training sample is located is used as the expected output, a plurality of decision trees in the random forest classifier are trained, and a random forest model is formed.

[0110] In step S401, the fundus sub-image is cropped from any region to obtain an image block, for example, a fundus sub-image with a size of 224x224 is cropped to obtain an image block with a size of 16x16. The fundus sub-image and the image block are both color pictures. When performing convolution operation on the image block, a corresponding number of channels can be generated according to the number of filters. The feature vector extracted from the image block is used to represent the image block, and the feature map of each channel is generated based on the feature vector. For example, the feature vector v e R 16×16×k where k=9 is the number of channels, including 4 directional channels, 2 amplitude channels and 3 color channels. Based on the channels of the feature vector, 9 feature maps can be generated, and the size of the feature map is 16x16.

[0111] In step S402, the first downsampling processing is to downsample each feature map according to a certain ratio, to reduce the feature map into a first downsampled image, and to take the pixels of the first downsampled image as the candidate features, to obtain a plurality of first candidate features. For example, each feature map with a size of 16x16 is downsampled according to a ratio of n times, and the number of the first candidate features obtained finally is 16x16x9 / n. 2 .

[0112] In step S403, the second downsampling processing is to downsample each feature map according to a ratio different from that of the first downsampling processing, to reduce the feature map into a second downsampled image, and to take the difference between any two pixels in the second downsampled image as a candidate feature, to obtain additional candidate features for each channel, i.e., second candidate features. For example, each feature map with a size of 16x16 is downsampled according to a certain ratio, to obtain a second downsampled image with a size of 3x3, and the difference between two randomly selected pixels in each second downsampled image is taken, to obtain 56 second candidate features for each channel.

[0113] In step S404, the first candidate features and the second candidate features are used to construct training samples. In the training stage of the random forest, the first candidate features and the second candidate features are sent into the random forest classifier for training. The random forest classifier is composed of T decision trees, T={100, 120,..., 200}, and the best value is taken. In the decision-making process, each node randomly selects a feature for classification. The probability determination result of each candidate region by each decision tree is obtained. In the testing stage of the random forest, the trained random forest parameters are used to classify the unused first candidate features and / or second candidate features, to obtain the probability determination result of each candidate region by each decision tree. According to the determination result, the average value is calculated, and the final classification result of the region of the fundus where the segment corresponding to the training sample is located is obtained according to the principle of minority submitting to majority.

[0114] Please refer to Figure 5 In some embodiments, step S104 includes but is not limited to steps S501 to S502, wherein step S502 includes steps S5021 to S5022.

[0115] Step S501, the edge image is binarized to obtain a binary image.

[0116] Step S502, the fitting step is iteratively performed to generate a sample estimation model until the sample estimation model with the largest number of inliers appears or the maximum number of iterations is reached.

[0117] In step S5021, a sample estimation model is generated based on the least square method using a plurality of pixel points randomly selected from the binary image. The sample estimation model is a circle model.

[0118] In step S5022, the number of inliers in the sample estimation model is estimated based on a preset threshold.

[0119] In this embodiment, the calculation formula of the maximum iteration number is:

[0120]

[0121] N is the maximum iteration number, P is the confidence, p is the proportion of inliers in the randomly selected pixel points, and λ is the number of randomly selected pixel points. min

[0122] In step S501, when the edge image is binarized, the gray scale of the pixel point of the edge image is compared with a preset gray scale threshold T n . If the gray scale of the pixel point is less than the gray scale threshold T n , the pixel is replaced by a black pixel, otherwise, the pixel is replaced by a white pixel. Therefore, when the gray scale threshold T n is smaller, more disc edges are saved, and more useless disc edge information is also increased. In actual operation, a suitable threshold can be determined according to prior knowledge, or an algorithm such as the maximum inter-class variance method (OTSU), the maximum entropy threshold segmentation method (KSW entropy algorithm) is used to determine the gray scale threshold T n .

[0123] In step S502, the process of single fitting to obtain a fitting result is as follows: a group of random pixel points are applied from the edge image as samples, and the parameters of the circle are estimated based on these pixel points. For the remaining pixel points in the edge image, the distance of the remaining pixel points to the currently fitted circle is calculated. If the distance of a certain remaining pixel point to the center of the circle is less than a preset threshold (this threshold is usually set in advance), the pixel point is an inlier, and the point is added to the inlier set suitable for circular fitting. Otherwise, the pixel point is an outlier, and the point is added to the outlier set unsuitable for circular fitting. If the number of inliers suitable for circular fitting exceeds the set threshold, it means that the circle estimated by the current random sample fits the inliers well, that is, the current sample is a good model, and another random sample can be selected, and then the remaining data is used to estimate the parameters of the circle.

[0124] Step S502 is iteratively executed until the best model is found or the iteration number reaches the preset threshold. Finally, the specific parameters of the circle are calculated using the least square method for all points suitable for circular fitting, and the final fitting result is obtained according to the calculated parameters of the circle.​

[0125] Referring to Figure 6 In some embodiments, step S105 includes, but is not limited to, steps S601-S603.

[0126] Step S601, output the center image to the doctor terminal.

[0127] Step S602, receive the first text information and / or the second text information.

[0128] Step S603, call the corresponding pathological diagnosis report template, and generate the pathological diagnosis report according to the first text information and / or the second text information.

[0129] In this embodiment, the doctor accesses the system through the doctor terminal, the system outputs the center image to the doctor terminal, so that the doctor can give the corresponding medical order information according to the center image, and through the generation of the first text information and / or the second text information recording the medical order information, the system receives the first text information and / or the second text information, and then generates the pathological diagnosis report using the corresponding pathological diagnosis report template, and uploads it to the shared database for the patient to query.

[0130] The first text information is the text content parsed from the voice information recorded by the sound pickup device generated by the doctor for the center image, and the second text information is the text content corresponding to the text information generated by the doctor for the center image. Specifically, the generation process of the first text information is to record the voice signal generated by the doctor speaking by using the sound pickup device, and convert the voice signal into text information. First, the voice signal is recorded and transmitted to the computer through the microphone, and the signal processing then analyzes the incoming signal to remove background noise, eliminate echo and enhance voice information, etc. Next, feature extraction will convert the processed signal into a series of parameters, which represent the characteristics of the sound, such as energy, frequency and harmonics, etc. Finally, by using a speech recognition model, these features will be mapped to the corresponding text. The speech recognition model is built on machine learning algorithms, which will learn different voice signals and the text they represent through a large amount of training data. When the input features are transmitted into this model, it will determine the most likely corresponding text output according to the previous learning experience. In this way, the voice signal generated by the doctor speaking is converted into text information.

[0131] Referring to Figure 7 The embodiment of the present application also provides an eye fundus image processing device, which can realize the above-mentioned eye fundus image processing method, and the device comprises:

[0132] The first module 701 is configured to output prompt information to the patient terminal, and the eye fundus image is generated by photographing the eye to be measured when the patient operates according to the prompt information.

[0133] The second module 702 is configured to determine a clipping boundary based on a preset clipping rule, clip the fundus image, and obtain a fundus sub-image; the clipping image includes an optic disc and an edge region around the optic disc.

[0134] The third module 703 is configured to identify an optic disc edge in the fundus sub-image based on a random forest model, and obtain an edge image.

[0135] The fourth module 704 is configured to perform fitting processing on the edge image based on a RANSAC algorithm iteration, use contour matching to combine a fitting result with the most inliers in the iteration process to obtain a smooth optic disc edge and locate a center of the optic disc, and obtain a center image.

[0136] The fifth module 705 is configured to receive, by a doctor terminal, doctor order information output by a doctor based on the center image, and upload the doctor order information to a front-end server, so that a patient terminal can receive the doctor order information from the front-end server.

[0137] The specific implementation of the fundus image processing apparatus is basically the same as the specific embodiments of the fundus image processing method described above, and thus will not be described again.

[0138] The embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the fundus image processing method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0139] Please refer to Figure 8 , Figure 8 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0140] The processor 801 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0141] The memory 802 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 802 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 802 and called and executed by the processor 801 to implement the fundus image processing method of the embodiments of the present application.

[0142] The input / output interface 803 is configured to realize information input and output.

[0143] The communication interface 804 is configured to realize communication interaction between the device and other devices, and the communication can be realized through wired mode (for example, USB, network cable and the like) or wireless mode (for example, mobile network, WIFI, Bluetooth and the like).

[0144] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803 and the communication interface 804) of the device.

[0145] The processor 801, the memory 802, the input / output interface 803 and the communication interface 804 are connected to each other through the bus 805 to realize communication connection between the device.

[0146] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned fundus image processing method.

[0147] The memory is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state storage device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0148] The fundus image processing method, device, equipment and storage medium provided by the embodiment of the present application provide a prompt for a patient to self-collect a fundus image, an eye to be detected is photographed and a fundus image is generated when the patient operates according to the prompt information, the examination time of the patient is reduced, the fundus image is combined with automatic recognition and processing, the optic disc in the fundus image is extracted based on a structure random forest and a RANSAC algorithm, the optic disc can be better positioned and extracted, the false positive in detection of hard exudation caused by DR is reduced, so that a doctor can quickly obtain the fundus image and provide related medical advice information, the inquiry time of the doctor is reduced, and the efficiency of the fundus image in the detection process and the recognition process is improved.

[0149] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0150] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0151] The device embodiments described above are merely illustrative, and units described as separate components can or can not be physically separated, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0152] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0153] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0154] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases of only A, only B and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0155] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0156] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0157] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0158] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0159] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An eye fundus image processing method characterized by comprising: The method comprises the following steps: outputting prompt information to a patient terminal, and capturing an eye to be tested and generating a fundus image when the patient operates according to the prompt information; determining a clipping boundary based on a preset clipping rule, clipping the fundus image, and obtaining a fundus sub-image; the clipped picture comprises an optic disc and an edge region around the optic disc; identifying the optic disc edge in the fundus sub-image based on a random forest model to obtain an edge image; iteratively fitting the edge image based on a RANSAC algorithm, using contour matching to combine the fitting result with the most inliers in the iteration process to obtain a smooth optic disc edge and locate the center of the optic disc to obtain a center image; receiving doctor's order information output by the doctor according to the center image through the doctor terminal and uploading it to the front-end server, so that the patient terminal can receive the doctor's order information from the front-end server; before the step of identifying the optic disc edge in the fundus sub-image based on the random forest model, the method comprises the following steps: randomly clipping a plurality of image blocks from the fundus sub-image, extracting a fixed number of feature vectors from the image blocks, and generating a plurality of feature maps based on the channels of the feature vectors; performing first downsampling processing on the feature maps to obtain a first downsampled image, using the pixels of the first downsampled image as candidate features to obtain first candidate features; performing second downsampling processing on the feature maps to obtain a second downsampled image, using the difference between the pixels in the second downsampled image as candidate features to obtain second candidate features; using the first candidate features and the second candidate features as training samples, using the fundus region where the segment corresponding to the training sample is located as the expected output, training a plurality of decision trees in the random forest classifier to form a random forest model; the step of iteratively fitting the edge image based on the RANSAC algorithm, using contour matching to combine the fitting result with the most inliers in the iteration process to obtain a smooth optic disc edge and locate the center of the optic disc, comprises the following steps: performing binaryzation processing on the edge image to obtain a binary image; iteratively performing the fitting step to generate a sample estimation model until the sample estimation model with the most inliers appears or the maximum number of iterations is reached; the fitting step comprises the following steps: generating a sample estimation model based on the least squares method using a plurality of randomly selected pixel points from the binary image; the sample estimation model is a circle model; estimating the number of inliers in the sample estimation model based on a set threshold; the calculation formula of the maximum number of iterations is: ， where N min is the maximum number of iterations, P is the confidence level, p is the proportion of inliers to randomly selected pixels, and λ is the number of randomly selected pixels.

2. The fundus image processing method according to claim 1, characterized by, the step of outputting prompt information to a patient terminal, and capturing an eye to be tested and generating a fundus image when the patient operates according to the prompt information, comprises the following steps: receiving an interaction request from the patient terminal and outputting prompt information to the patient terminal; receiving an operation signal generated by the patient operating according to the prompt information, capturing the eye to be tested according to the operation signal, and obtaining a fundus image; uploading the fundus image and performing desensitization, enhancement and labeling processing on the fundus image.

3. The fundus image processing method according to claim 1, characterized by, the step of determining a clipping boundary based on a preset clipping rule, clipping the fundus image, and obtaining a fundus sub-image, comprises the following steps: estimating the optic disc radius based on the proportional relationship between the fundus radius and the optic disc radius; locating the key pixel point of the optic disc in the fundus image, and clipping the fundus image based on the key pixel point as the center and the estimated optic disc radius to obtain a fundus sub-image.

4. The fundus image processing method according to claim 1, characterized by, The random forest model is used for identifying the optic disc edge in the fundus sub-image, and includes the following steps: The random forest model is used for identifying the optic disc edge in the fundus sub-image, and includes the following steps:

5. The fundus image processing method according to claim 1, characterized by, The doctor terminal receives the doctor's order information based on the center image and uploads it to the front-end server, and includes the following steps: The center image is output to the doctor terminal; The first text information and / or the second text information are received; the first text information is the text content obtained by analyzing the voice information recorded by the sound pickup device and generated by the doctor for the center image, and the second text information is the text content corresponding to the text information generated by the doctor for the center image; The corresponding pathological diagnosis report template is called, and the pathological diagnosis report is generated based on the first text information and / or the second text information.

6. An ocular fundus image processing apparatus characterized by comprising: It includes: The first module is used for outputting prompt information to the patient terminal, and the fundus image is generated by shooting the eye to be tested when the patient operates according to the prompt information; The second module is used for determining the clipping boundary based on the preset clipping rule, clipping the fundus image and obtaining the fundus sub-image; The clipped picture includes the optic disc and the edge region around the optic disc; The third module is used for identifying the optic disc edge in the fundus sub-image based on the random forest model, and obtaining the edge image; The fourth module is used for fitting the edge image based on the RANSAC algorithm, using the contour matching to combine the fitting results with the most inliers in the iteration process to obtain the smooth optic disc edge and locate the center of the optic disc, and obtaining the center image; The fifth module is used for receiving the doctor's order information based on the center image through the doctor terminal and uploading it to the front-end server, so that the patient terminal can receive the order information from the front-end server; Before the third module, it includes: A module for randomly clipping a plurality of image blocks from the fundus sub-image, extracting a fixed number of feature vectors from the image blocks, and generating a plurality of feature maps based on the channels of the feature vectors; A module for performing first downsampling processing on the feature map to obtain a first downsampled image, using the pixels of the first downsampled image as candidate features, and obtaining first candidate features; A module for performing second downsampling processing on the feature map to obtain a second downsampled image, using the difference between the pixels in the second downsampled image as candidate features, and obtaining second candidate features; A module for using the first candidate features and the second candidate features as training samples, using the fundus region where the training samples correspond to as the expected output, training a plurality of decision trees in the random forest classifier, and forming a random forest model; The fourth module includes: A module for performing binaryzation processing on the edge image to obtain a binary image; A module for iteratively performing the fitting step to generate a sample estimation model until the sample estimation model with the most inliers appears or the maximum number of iterations is reached; The fitting step includes: Based on the least square method, a plurality of pixel points randomly selected from the binary image are used to generate a sample estimation model; the sample estimation model is a circle model; The number of inliers in the sample estimation model is estimated based on a set threshold; The calculation formula of the maximum number of iterations is: ， where N min is the maximum number of iterations, P is the confidence level, p is the proportion of inliers to randomly selected pixels, and λ is the number of randomly selected pixels.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the fundus image processing method in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the fundus image processing method in any one of claims 1 to 5.

Images