A dry eye FBUT detection method, system and storage medium based on deep learning

Abstract

The present application relates to a dry eye FBUT detection method, system and storage medium based on deep learning, and the method includes: acquiring fluorescein-stained ocular surface images in real time; extracting open eye image data and ocular surface areas from the fluorescein-stained ocular surface images based on a multi-task prediction model; cropping the open eye image data according to the ocular surface areas, and outputting a recognition sample set; segmenting the tear film rupture area and the pupil area from the recognition sample set based on a fast segmentation model; extracting the pupil center and the optical area from the pupil area; outputting the optical tear film rupture time based on the time record of the appearance of the tear film rupture area in the optical area; outputting the ocular surface tear film rupture moment based on the time record of the appearance of the tear film rupture area in the ocular surface area; identifying the pupil area during the calculation process, and all pictures after the pupil moves continuously can participate in the calculation and be trained for learning, so that the optical area can be stably identified to obtain more accurate results.

Description

Technical Field

[0001] The present application relates to the field of ophthalmology, and in particular to a deep learning-based dry eye FBUT detection method, system and storage medium. Background Art

[0002] Fluorescein staining tear film breakup time is a tear film stability test method that must be performed in a room with room temperature, appropriate humidity, and light-proof environment. Standard test method: Use a sterile dropper to draw about 2 μl of 1% sodium fluorescein solution and drop it into the conjunctival sac, or use a fluorescein test paper moistened with antibiotic eye drops but without excess residual liquid to contact the lower eyelid margin. The patient blinks 3-4 times to spread the fluorescein on the ocular surface, and looks straight ahead. The time from the last blink to the appearance of the first black spot on the cornea is the tear film breakup time.

[0003] In the existing instrument detection technology, the ocular surface image is usually obtained directly in a natural state, and then the tear film rupture position is analyzed through traditional digital image processing technology. The detection process includes adaptive filtering and closing operation to remove the eyelash image. The first frame image after opening the eyes is used as a template, and each subsequent frame is matched with it respectively. Finally, the matched image is subtracted from the first frame image to find the tear film rupture position, and the tear film rupture time is calculated using the frame number of the rupture image, and dry eyes are diagnosed based on this time.

[0004] However, the above existing methods cannot deal well with problems such as eyelash occlusion, deformation, blinking, and light diffusion on the eye surface, so the accuracy of their processing results needs to be improved. Summary of the invention

[0005] In order to improve the accuracy of tear film breakup detection, the present application provides a dry eye FBUT detection method, system and storage medium based on deep learning.

[0006] In the first aspect, the present application provides a dry eye FBUT detection method based on deep learning, which adopts the following technical solution:

[0007] A dry eye FBUT detection method based on deep learning, comprising the following steps:

[0008] Step S1: acquiring fluorescein-stained ocular surface images in real time;

[0009] Step S2: extracting the eye-opening image data and the eye surface area from the fluorescein-stained eye surface image based on the multi-task prediction model of the deep convolutional network; cropping the eye-opening image data according to the eye surface area, and outputting a recognition sample set;

[0010] Step S3: Based on a fast segmentation model of a deep convolutional network, the tear film rupture area and the pupil area are segmented from the recognition sample set; the pupil center and the optical area within a set distance range around the pupil center are extracted from the pupil area;

[0011] Step S4: outputting the optical tear film breakup time according to the time record of the tear film breakup zone appearing in the optical zone; outputting the ocular surface tear film breakup moment according to the time record of the tear film breakup zone appearing in the ocular surface zone.

[0012] By adopting the above technical scheme, the fluorescein staining test method is combined with the calculation method of the multi-task prediction model and the fast segmentation model. Compared with manual viewing and existing instruments, after training, this method can well handle problems such as eyelash occlusion, deformation, blinking, and ocular surface fluorescein diffusion. In addition, the pupil area can be identified during the calculation process. Although the pupil is always moving and the change of ambient light causes the pupil to contract or enlarge, the pictures after the continuous movement of the pupil can participate in the calculation and learning training. The pupil area can be stably identified, and the stable tear film rupture area and optical area can be obtained. Finally, a more accurate tear film rupture time and its distribution can be obtained, which improves the recognition accuracy. The tear film rupture time and its distribution are more medically important and are beneficial for doctors to diagnose ophthalmic problems.

[0013] Preferably, the step S4 further comprises the following steps:

[0014] Step S41: outputting a tear film breakup area distribution table based on overlapping data between the tear film breakup area and the optical zone and an area outside the optical zone;

[0015] Step S42: Acquire tear film breakup standard data;

[0016] Step S43: classifying the overlapping data in the tear film breakup area distribution table according to the tear film breakup standard data and outputting the following classification results: whether there is overlap in the optical zone and whether there is overlap in the area outside the optical zone;

[0017] Step S44: outputting the optical tear film breakup time and the ocular surface tear film breakup moment according to the classification result.

[0018] By adopting the above technical scheme, a tear film rupture area distribution table is obtained according to the situation of the overlapping area, and then the contents in the tear film rupture area distribution table are classified. According to the classification results, the tear film rupture conditions of different areas, namely the optical tear film rupture time and the ocular surface tear film rupture moment, are obtained. The overlapping algorithm does not need to perform collision calculation or range calculation on the edges of other areas, thereby reducing the amount of calculation and improving the calculation efficiency.

[0019] Preferably, the tear film breakup standard data includes an area ratio of the tear film breakup zone in the optical zone and an area ratio of the tear film breakup zone in a region outside the optical zone.

[0020] By adopting the above technical solution, the area ratio is used as a standard, and after setting the area ratio value as the screening data, the situations that can be ignored after calculation can be filtered out, thereby further improving the accuracy of the detection results.

[0021] Preferably, the optical zone is an area within 3.5 mm-4.5 mm around the pupil.

[0022] By adopting the above technical solution, selecting this area as the optical zone can reduce the interference of pupil area instability on data accuracy, and at the same time can reflect the distribution of tear film rupture areas in different areas of the eye, which is of great medical significance.

[0023] Preferably, the step S3 further includes:

[0024] Step S31: segmenting the eye images in the recognition sample set based on a fast segmentation model of a deep convolutional network to obtain a pupil segmentation result image;

[0025] Step S32: extracting a pupil area bounding box from the pupil segmentation result image;

[0026] Step S33: Calculate the center of the pupil area bounding box to obtain the pupil center;

[0027] Step S34: Calculate the optical zone according to the pupil center.

[0028] By adopting the above technical solution, segmentation is performed using a fast segmentation model, and then the segmented image is recognized, the pupil area bounding box is extracted, and then the pupil center is obtained based on the bounding box, and then the optical zone is obtained. Compared with the method of using only a deep convolutional network to calculate the optical zone, it has a lower amount of calculation and can also release some of the computing resources of the deep convolutional network to calculate subsequent data.

[0029] Preferably, the step S3 further includes:

[0030] Step S35: extracting the edges of the tear film breakup area, the optical zone, and the area outside the optical zone;

[0031] Step S36: marking the tear film rupture zone, the optical zone and the area outside the optical zone in the identification sample set according to the extracted edges, wherein the tear film rupture zone, the optical zone and the area outside the optical zone have different marking features.

[0032] By adopting the above technical solution, the edge is extracted and then marked, so that the doctor can see the image results and at the same time estimate whether the calculated results are roughly accurate based on the image results, which is conducive to ensuring the accuracy of the results.

[0033] Preferably, a plurality of pupil images with different positions and sizes are used to train the multi-task prediction model and the fast segmentation model;

[0034] Alternatively, a plurality of pupil images based on different light sources are used to train the multi-task prediction model and the fast segmentation model.

[0035] By adopting the above technical solution and using pupil images that are close to or even real-life for training, doctors can continue to make corrections after seeing the results, making the output of the deep learning network more accurate.

[0036] Preferably, the eye-open image data in step S2 is a set of images in a continuous eye-open state in a time period between adjacent blink image data; the eye-open image data is verified, and if the eye-open image data contains eye-closed image information, the fluorescein-stained ocular surface image is reacquired.

[0037] By adopting the above technical solution, more accurate tear film breakup data can be obtained by acquiring the data of continuous eye opening and verifying it.

[0038] In the second aspect, the present application provides a dry eye FBUT detection system based on deep learning, which adopts the following technical solutions:

[0039] A deep learning-based dry eye FBUT detection system, wherein the system runs an executable program of any one of the deep learning-based dry eye FBUT detection methods described above.

[0040] In a third aspect, the present application provides a storage medium, which adopts the following technical solution:

[0041] A storage medium storing an executable program of any one of the above-described deep learning-based dry eye FBUT detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a method flow diagram of a dry eye FBUT detection method based on deep learning in an embodiment of the present application;

[0043] Figure 2 is a fluorescein-stained ocular surface image in an embodiment of the present application;

[0044] Figure 3 is a point location diagram for cropping an area image outside the optical zone in an embodiment of the present application;

[0045] Figure 4 is an image obtained by cropping the image of the area outside the optical zone in the embodiment of the present application;

[0046] Figure 5 is a schematic diagram of a fast segmentation model in an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of the pupil area, optical area and tear film rupture area being marked on the original image in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following is combined with Figure 1-6 This application is described in further detail.

[0049] The present application embodiment discloses a dry eye FBUT detection method based on deep learning. Figure 1 ,The deep learning based dry eye FBUT detection method includes the following steps:

[0050] Step S1: Real-time acquisition of fluorescein-stained ocular surface images. In actual applications, the eye portion of the fluorescein-stained ocular surface image will appear fluorescent green, such as Figure 2 As shown. In natural light without fluorescein staining, the eyes and skin appear black and white. Black and white images are not easy for deep learning models to recognize faster. Fluorescent green eyes and flesh-colored skin are easier for deep learning models to recognize faster. Doctors use testing equipment to test the eyes. The testing equipment has a built-in display, memory, processor, and color camera. The processor runs multiple algorithms based on deep convolutional networks.

[0051] Step S2: Based on the multi-task prediction model of the deep convolutional network, the eye-opening image data and the eye surface area are extracted from the fluorescein-stained eye surface image. The fluorescein-stained eye surface image obtained in step S1 is used to identify whether the eye is blinking, and the blinking time and the continuous eye-opening time are recorded and counted. The image of the continuous eye-opening time is extracted as the eye-opening image data, and the eye-opening image data includes image information and the time information generated by the image information.

[0052] The eye-open image data is a set of images in a continuous eye-open state in the time period between adjacent blink image data, that is, if the next blink occurs, the next blink time is recorded and a new continuous eye-open time statistics is started, and the video segment with the longest continuous eye-open time is updated repeatedly until the video acquisition is completed. The video includes the required image set, and the image set can be extracted from the video. Among them, the eye-open image data is verified, and if the eye-open image data contains closed eye image information, the fluorescein-stained ocular surface image is reacquired. Only by acquiring continuous eye-open data can more accurate tear film rupture data be obtained.

[0053] Locate the boundary of the area outside the optical zone of the image obtained in step 1 and crop the image in this area; this is used to increase the effective area ratio of the segmentation network input image and reduce the input resolution of the segmentation network, which can make the sample space of the segmentation network smaller, require fewer training samples, and make the segmentation network learn more fully and efficiently, thus achieving the purpose of online real-time reasoning.

[0054] The open-eye image data is cropped according to the ocular surface area, and the recognition sample set is output. When training the model, multiple pupil images with different positions and sizes are used to train the multi-task prediction model, or multiple pupil images based on different light sources are used to train the multi-task prediction model; both training methods can also be used at the same time. When training with pupil images that are close to or even real, doctors can continue to make corrections after seeing the results, making the output of the deep learning network more accurate.

[0055] When implementing step S2, you can use MobileNetV3 as the backbone network. The output has two branches. One branch is a classification task, which outputs two probability values ​​to identify open and closed eyes; the other branch is a regression task, which outputs 8 values ​​of x, y up, down, left, and right coordinates, such as Figure 3 As shown, the eye surface area is then cut out according to these four points, as shown in Figure 4 As shown. The label of open eyes is 1, and the label of closed eyes is 0. The eye surface area label is the normalized value of the upper, lower, left and right coordinates in the original image, with a numerical range of 0 to 1. If it is a closed eye image, the coordinate label value is set to 0. Each image has an open eye label, a closed eye label and an eye surface area label at the same time. The model training method may include: graying the input image and retaining 3 channels; data amplification using preprocessing such as rotation, translation, scaling, grayscale stretching, random blurring, etc.; the loss function of the classification branch uses a binary cross entropy function, and the loss function of the regression branch uses a mean square error function. The model training method may also include: using the training weights of MobileNetV3 on the IMAGENET dataset as the initial weights, and then fine-tuning the data in the model. The model training method may also include: dividing the dataset into a training set and a validation set in a ratio of 4:1, iteratively training for 60 rounds, with an initial learning rate of 0.01, and reducing the learning rate by 10 times in the 20th and 40th rounds respectively. Finally, the training weight with the smallest difference between the loss values ​​of the training set and the validation set is selected as the training result.

[0056] Step S3: Based on the fast segmentation model of the deep convolutional network, the tear film rupture area, the pupil area and the area outside the remaining optical area on the ocular surface are segmented from the recognition sample set. The pupil center and the optical area within a set distance range around the pupil center are extracted from the pupil area. Among them, the optical area is the area within 3.5mm-4.5mm around the pupil. The area around the pupil center of 3.5mm can be selected, the area around the pupil center of 4mm can be selected, and the area around the pupil center of 4.5mm can be selected. Selecting this area as the optical area can reduce the interference of pupil area instability on data accuracy, and at the same time can also reflect the distribution of tear film rupture areas in different areas of the eye, which is of great medical significance. During training, multiple pupil images with different positions and sizes can be used to train the fast segmentation model, or multiple pupil images based on different light sources can be used to train the fast segmentation model; the two training methods can also be used at the same time. Using pupil images that are close to or even real to train, doctors can continue to correct the results after seeing the results, so that the results output by the deep learning network can be more accurate.

[0057] Step S3 also includes: Step S31: segmenting the eye image in the identification sample set based on the fast segmentation model of the deep convolutional network to obtain the pupil segmentation result image. Step S32: extracting the pupil area bounding box from the pupil segmentation result image. Step S33: calculating the center of the pupil area bounding box to obtain the pupil center. Step S34: calculating the optical zone according to the pupil center. Step S35: extracting the edges of the tear film rupture zone, the optical zone, and the area outside the optical zone. Step S36: marking the tear film rupture zone, the optical zone, and the area outside the optical zone in the identification sample set according to the extracted edges, and the tear film rupture zone, the optical zone, and the area outside the optical zone have different marking features, such as marking with lines of different colors or marking with lines of different thicknesses. Continuing to mark after extracting the edges allows doctors to see the image results and can in turn estimate whether the calculated results are roughly accurate based on the image results, which is conducive to ensuring the accuracy of the results.

[0058] The implementation process of step S3 can be as follows: Figure 5 As shown, the DFANet segmentation network is used to perform segmentation detection on the eye image cropped in step S2 in real time, and each pixel is classified into rupture and pupil, that is, it is determined whether the pixel position has rupture and whether it belongs to the pupil; then the pupil area bounding box is extracted in the pupil segmentation result image, and then the center of the bounding box is obtained, which is considered to be the pupil center, and then the circular area with a diameter of 3.5mm-4.5mm with the center as the center of the circle is used as the optical area. Figure 6 As shown, finally, Canny edge extraction is performed on each area and the image is superimposed on the original image to be displayed, with the tear film rupture boundary marked as a green circle, the pupil boundary marked as a red circle, and the optical zone boundary marked as a yellow circle.

[0059] When using the DFANet segmentation network to segment and detect the ring area image in real time, the DFANet segmentation network model is built, the backbone network remains unchanged, and only the final output layer is modified. The input image is grayed; data augmentation uses preprocessing such as rotation, translation, scaling, grayscale stretching, and random blurring. The loss function uses the binary cross entropy function, namely Sigmoid-Cross-Entropy-Loss. The training weights of DFANet on the Cityscapes dataset are used as the initial weights, and then fine-tuning training is performed on the data in the model. The dataset is divided into a training set and a validation set, and iterative training is performed. Finally, the training weight with the smallest difference in loss value between the training set and the validation set is selected as the training result. The DFANet segmentation network contains multiple network layers, conv refers to convolution with a convolution kernel size of 3, enc refers to a convolution layer block, FCattention refers to the attention module that obtains semantic information and category information, C refers to the channel concatenation layer, and xN refers to the upsampling operation of N times, namely upsampling. Through a series of feature extraction, channel fusion, upsampling and other operations, the segmentation probability map is finally output.

[0060] Step S4: Output the optical tear film breakup time according to the time record of the tear film breakup zone appearing in the optical zone. Output the tear film breakup moment on the ocular surface according to the time record of the tear film breakup zone appearing in the area outside the optical zone. More specifically, the time when the tear film starts to break up is obtained according to the time difference between step S2 and step S3. For example, the longest continuous eye-opening time period is cyclically updated in step S2, that is, the first eye-opening moment of the continuous eye-opening time period is set to t0. Then, in step S3, the time when the tear film starts to break up in each area is obtained in real time and is set to t1, and t1 minus t0 is the tear film breakup time in each area.

[0061] Step S4 also includes the following steps: Step S41: Based on the overlapping data of the tear film breakup area, the optical zone, and the area outside the optical zone, output the tear film breakup area distribution table. Step S42: Obtain the tear film breakup standard data. The tear film breakup standard data includes the area ratio of the tear film breakup area in the optical zone and the area ratio of the tear film breakup area in the area outside the optical zone. For example, only 0.1% of the tear film breakup area overlaps in the optical zone, and the remaining 99.9% overlaps in the area outside the optical zone, then the 0.1% area overlapping in the optical zone is ignored. Taking the area ratio as a standard, after setting the area ratio value as the screening data, the situation that can be ignored after calculation can be filtered out, further improving the accuracy of the detection result. Step S43: According to the tear film breakup standard data, the overlapping data appearing in the tear film breakup area distribution table is classified and the following classification results are output: whether there is overlap in the optical zone and whether there is overlap in the area outside the optical zone. Step S44: Output the optical tear film breakup time and the moment of tear film breakup on the ocular surface according to the classification results.

[0062] After the patient aligns the instrument, the image is continuously acquired for 20 seconds, or other time lengths. Each frame of fluorescein-stained ocular surface image acquired within the 20-second duration is processed in real time using a multi-task prediction model and a fast segmentation model. The fluorescein staining test method is combined with the calculation method of the multi-task prediction model and the fast segmentation model. Compared with manual viewing and existing instruments, after training, this method can well handle problems such as eyelash occlusion, deformation, blinking, and ocular surface fluorescein diffusion, and the pupil area is identified during the calculation process. Although the pupil is always moving, the change in ambient light causes the pupil to shrink or enlarge and change, but the images after the pupil moves continuously can participate in the calculation and learning training, and the pupil area can be stably identified. The fast segmentation model is then used for segmentation, and then the segmented image is identified, the pupil area bounding box is extracted, and then the pupil center is obtained according to the bounding box, and then the optical zone is obtained. Compared with the calculation method that only uses a deep convolutional network to calculate the optical zone, it has a lower amount of calculation, and can also release some of the deep convolutional network's computing resources to calculate the subsequent data. The stable tear film breakup zone, optical zone and area outside the optical zone are obtained.

[0063] Then, according to the situation of the overlapping area, the tear film breakup area distribution table is obtained, and then the contents in the tear film breakup area distribution table are classified. According to the classification results, the tear film breakup situation of different areas, namely the optical tear film breakup time and the tear film breakup moment of the eye surface, is obtained. The overlapping algorithm does not need to perform collision calculation or range calculation on the edges of other areas, which reduces the amount of calculation and improves the calculation efficiency. The method finally obtains a more accurate tear film breakup time and its distribution, improves the recognition accuracy, and obtains the tear film breakup time and its distribution, which is more medically important and helps doctors diagnose ophthalmic problems.

[0064] An embodiment of the present application also discloses a deep learning-based dry eye FBUT detection system, in which an executable program of any one of the deep learning-based dry eye FBUT detection methods described above runs.

[0065] An embodiment of the present application also discloses a storage medium storing an executable program of any one of the deep learning-based dry eye FBUT detection methods described above.

[0066] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A dry eye FBUT detection method based on deep learning, characterized by: The steps include: Step S1: acquiring fluorescein-stained ocular surface images in real time; Step S2: extracting the eye-opening image data and the eye surface area from the fluorescein-stained eye surface image based on the multi-task prediction model of the deep convolutional network; cropping the eye-opening image data according to the eye surface area, and outputting a recognition sample set; Step S3: Based on a fast segmentation model of a deep convolutional network, the tear film rupture area and the pupil area are segmented from the recognition sample set; the pupil center and the optical area within a set distance range around the pupil center are extracted from the pupil area; Step S4: outputting the optical tear film breakup time according to the time record of the tear film breakup zone appearing in the optical zone; outputting the ocular surface tear film breakup moment according to the time record of the tear film breakup zone appearing in the ocular surface zone; The step S4 also includes the following steps: Step S41: outputting a tear film breakup area distribution table based on overlapping data between the tear film breakup area and the optical zone and an area outside the optical zone; Step S42: Acquire tear film breakup standard data; Step S43: classifying the overlapping data in the tear film breakup area distribution table according to the tear film breakup standard data and outputting the following classification results: whether there is overlap in the optical zone and whether there is overlap in the area outside the optical zone; Step S44: outputting the optical tear film breakup time and the ocular surface tear film breakup moment according to the classification result.

2. The deep learning-based dry eye FBUT detection method according to claim 1, characterized in that: The tear film breakup standard data includes an area ratio of the tear film breakup zone in the optical zone and an area ratio of the tear film breakup zone in a region outside the optical zone.

3. The deep learning-based dry eye FBUT detection method according to claim 1, characterized in that: The optical zone is the area within 3.5mm-4.5mm around the pupil.

4. The deep learning-based dry eye FBUT detection method according to claim 1, characterized in that: The step S3 also includes: Step S31: segmenting the eye images in the recognition sample set based on a fast segmentation model of a deep convolutional network to obtain a pupil segmentation result image; Step S32: extracting a pupil area bounding box from the pupil segmentation result image; Step S33: Calculate the center of the pupil area bounding box to obtain the pupil center; Step S34: Calculate the optical zone according to the pupil center.

5. The deep learning-based dry eye FBUT detection method according to claim 1, characterized in that: The step S3 also includes: Step S35: extracting the edges of the tear film breakup area, the optical zone, and the area outside the optical zone; Step S36: marking the tear film rupture zone, the optical zone and the area outside the optical zone in the identification sample set according to the extracted edges, wherein the tear film rupture zone, the optical zone and the area outside the optical zone have different marking features.

6. The deep learning-based dry eye FBUT detection method according to claim 1, characterized in that: Using a plurality of pupil images of different positions and sizes to train the multi-task prediction model and the fast segmentation model; Alternatively, a plurality of pupil images based on different light sources are used to train the multi-task prediction model and the fast segmentation model.

7. The deep learning-based dry eye FBUT detection method according to claim 1, characterized in that: The eye-open image data in step S2 is a set of images in a continuous eye-open state in the time period between adjacent blink image data; the eye-open image data is verified, and if the eye-open image data contains eye-closed image information, the fluorescein-stained ocular surface image is reacquired.

8. A dry eye FBUT detection system based on deep learning, characterized by: The system runs an executable program of the deep learning-based dry eye FBUT detection method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: An executable program of the deep learning-based dry eye FBUT detection method as described in any one of claims 1 to 7 is stored.

Images