Real-time tear film break-up time analysis method, device and system
By combining sliding window and segmentation model, the robustness and real-time performance of tear film breakup time detection in dry eye analyzers are addressed, achieving high-precision tear film breakup time analysis, reducing computational load and enhancing the ability to handle eyelash occlusion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU BIGVISION MEDICAL TECH CO LTD
- Filing Date
- 2023-02-17
- Publication Date
- 2026-06-19
Smart Images
Figure CN117408938B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical imaging technology, specifically relating to a method and device system for real-time tear film breakup time analysis. Background Technology
[0002] The main causes of dry eye syndrome include insufficient secretion of tears from the lacrimal glands in the aqueous layer, insufficient secretion of the lipid layer, and insufficient secretion of the mucin layer. Dry eye analyzers are typically used to determine the cause. Current dry eye analyzers utilize tear film breakup time detection modes to detect whether the tested eye has insufficient tear secretion. This is achieved by illuminating the Placido disc onto the ocular surface with a light source to observe the time and location of tear film breakup.
[0003] Tear film breakage location analysis methods based on Placido discs mainly include morphological analysis and deep learning-based recognition model analysis. The typical workflow for morphological analysis of tear film breakage is as follows: selecting a reference image, extracting key regions, image registration, polar coordinate unfolding, processing eyelash-occluded areas, identifying the difference between the floating image and the reference image's circular edge, obtaining the breakage location, and recording the corresponding time. The advantages of morphological analysis are that the internal process of the white-box method involves manual feature extraction, fault location is traceable, and computational resource consumption is relatively low. The disadvantages are that the analysis results are easily affected by the selected reference image and the actual acquired image, leading to insufficient robustness and generalization performance.
[0004] Deep learning-based tear film rupture analysis, a relatively recent development compared to morphological analysis methods, can identify and segment tear film rupture locations in Placido images in an end-to-end manner. The analysis process then utilizes blink detection and pupil center localization methods. The advantages of deep learning-based tear film rupture analysis include higher robustness and the ability to automatically extract features using a data-driven approach, facilitating model accuracy upgrades. Disadvantages include real-time performance significantly influenced by model structure and deployment, and a lack of temporal information capture in existing methods. Summary of the Invention
[0005] To address the aforementioned problems, this invention proposes a real-time tear film breakup time analysis method and device system that effectively utilizes timing information and exhibits high accuracy, real-time performance, and robustness.
[0006] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:
[0007] In a first aspect, the present invention provides a method for real-time tear film breakup time analysis, comprising:
[0008] Real-time acquisition of images with tear film rupture, and recording of image acquisition time and image sequence number;
[0009] Using a sliding window approach, a partial image is obtained by extracting the image with tear film rupture based on the image sequence number;
[0010] After converting each local image into a single-channel grayscale image, they are fed into a pre-trained segmentation model to obtain four types of segmentation results: background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image, and eyelash occlusion region segmentation image.
[0011] If two blinking behaviors are determined based on the Placido highlight ring segmentation map, then all ring breakage region segmentation maps are fused according to the image sequence number and the corresponding sliding window. If a third blinking behavior occurs, a breakage region map is generated.
[0012] Based on the tear film breakup map, tear film breakup time analysis was performed.
[0013] Optionally, the segmentation model includes a first 2x downsampling layer, a second 2x downsampling layer, a third 2x downsampling layer, a fourth 2x downsampling layer, an 8x upsampling layer, a 2x upsampling layer, a first convolutional activation function layer, a second convolutional activation function layer, a third convolutional activation function layer, a fourth convolutional activation function layer, a fifth convolutional activation function layer, a sixth convolutional activation function layer, and an output layer;
[0014] The first convolutional activation function layer and the first 2x downsampling layer are arranged sequentially to form the first unit;
[0015] The second convolutional activation function layer and the second 2x downsampling layer are arranged sequentially to form the second unit;
[0016] The third convolutional activation function layer and the third 2x downsampling layer are arranged sequentially to form the third unit;
[0017] The fourth convolutional activation function layer and the fourth 2x downsampling layer are arranged sequentially to form the fourth unit;
[0018] The first unit, the second unit, the third unit, and the fourth unit are connected in sequence;
[0019] The 8x upsampling layer is positioned between the fourth unit and the fifth convolutional activation function layer;
[0020] The 2x upsampling layer is positioned between the fifth convolutional activation function layer and the sixth convolutional activation function layer;
[0021] The output layer is connected to the sixth convolutional activation function layer.
[0022] Optionally, the training method for the pre-trained segmentation model includes:
[0023] Acquire tear film breakup time videos of multiple individuals, and split the tear film breakup time videos into images with tear film breakup to form a training set;
[0024] All images in the training set are labeled with categories, including background, Placido highlighted rings, broken ring regions, and eyelash-occluded regions.
[0025] Using a sliding window approach, a partial image is obtained by extracting the image with tear film rupture based on the image sequence number;
[0026] After converting each local image into a single-channel grayscale image, they are fed into the segmentation model for training to obtain a pre-trained segmentation model. The output of the pre-trained segmentation model includes four types of segmentation results: background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image, and eyelash occlusion region segmentation image.
[0027] Optionally, the method for obtaining the local map includes the following steps:
[0028] Based on the image sequence number, n sliding windows are selected, and the corresponding images with tear film rupture are extracted in turn based on the window coordinates of each sliding window to obtain a local image.
[0029] Optionally, the number of sliding windows is fixed at 4, and the window coordinates are set as follows: window 1 (0.1, 0.1, 0.6, 0.6), window 2 (0.3, 0.1, 0.6, 0.6), window 3 (0.3, 0.3, 0.6, 0.6), and window 4 (0.1, 0.3, 0.6, 0.6). The four numbers in parentheses represent the proportions of the top-left corner coordinates (horizontal pixel coordinate x, vertical pixel coordinate y, window width w, and window height h) of the sliding window in the original image to the pixel size of the original image.
[0030] Optionally, blinking behavior is detected through the following steps:
[0031] Hough circle detection is used to check if the Placido disk center coordinates exist in the Placido highlight ring segmentation map;
[0032] If the center coordinates of the Placido disk are not detected, it means that the current frame is in the closed-eye state;
[0033] If the center coordinates of the Placido disk are detected, it indicates that the current frame is in an open-eye state;
[0034] In the sequence analysis results, a continuous sequence of opening and closing the eyes is considered as one blink; a continuous sequence of opening and closing the eyes is considered as two blinks.
[0035] Optionally, the method for generating the fracture region map includes the following steps:
[0036] The tear film rupture segmentation results are obtained by sequentially performing opening and closing operations and median filtering on all the segmentation maps of the circular rupture regions.
[0037] Based on the selected four sliding windows, the segmentation results of four adjacent frames are stitched together. The overlapping areas are stitched together using a weighted average method to obtain the complete segmentation result of the tear film breakage location of each original image with tear film breakage.
[0038] The color of each small square in the tear film breakup map is generated based on all tear film breakup locations, with the color intensity representing the breakup time;
[0039] The weight selection rule is as follows: the weights are [0.6, 0.4] in the overlapping areas of sliding window 1 and sliding window 2, sliding window 2 and sliding window 3, sliding window 3 and sliding window 4, and sliding window 4 and sliding window 1, respectively. The weights in the overlapping areas of the four sliding windows are [0.1, 0.2, 0.3, 0.4] according to the order of image acquisition time.
[0040] Optionally, the tear film breakup time analysis includes the following steps:
[0041] Align the center coordinates of the Placido disk in the Placido highlight ring segmentation map with the center of the fracture area map;
[0042] The Placido highlighted ring segmentation image is subjected to multiple iterations of erosion and dilation operations to fill the background between the highlighted rings, thereby obtaining the segmentation image of the inspected region; parts that do not belong to the inspected region are marked in gray in the fracture area image;
[0043] If tear film rupture exists, the acquisition time of the current frame is recorded in the corresponding grid of the rupture area map. Each grid is assigned a value only once and is not updated.
[0044] Calculate the ratio of the area of the fractured region to the area of the inspected region in the image at each time point.
[0045] In a second aspect, the present invention provides a real-time tear film breakup time analysis device, comprising:
[0046] The image acquisition module is used to acquire images with tear film rupture in real time and record the image acquisition time and image sequence number;
[0047] The local image acquisition module is used to extract the local image of the image with tear film rupture by using a sliding window method according to the image sequence number;
[0048] The image segmentation module is used to convert each local image into a single-channel grayscale image and then feed them into a pre-trained segmentation model to obtain four types of segmentation results. The four types of segmentation results are background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image and eyelash occlusion region segmentation image.
[0049] The fusion module is used to fuse all ring breakage region segmentation maps according to the image sequence number and the corresponding sliding window if two blinking behaviors are determined based on the Placido highlighted ring segmentation map. If a third blinking behavior occurs, a breakage region map is generated.
[0050] The analysis module is used to perform tear film breakup time analysis based on the breakup region map.
[0051] Thirdly, the present invention provides a real-time tear film breakup time analysis system, including a storage medium and a processor;
[0052] The storage medium is used to store instructions;
[0053] The processor is configured to operate according to the instructions to perform the method according to any one of the first aspects.
[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0055] 1. This invention significantly reduces the computational load of the segmentation model by constructing a segmentation model. Furthermore, since eyelash occlusion affects the accuracy of tear film breakage recognition, this invention adds an eyelash occlusion region class to improve the segmentation model's ability to distinguish features, thereby improving accuracy.
[0056] 3. This invention uses a sliding window method to acquire local images for analysis, effectively balancing real-time performance and accuracy of details of tear film rupture.
[0057] 4. This invention integrates information from multiple consecutive frames to reduce the image quality of a single frame, thereby improving the stability of the sequence image results. Attached Figure Description
[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:
[0059] Figure 1 This is a schematic flowchart of a real-time tear film breakup time analysis method according to an embodiment of the present invention;
[0060] Figure 2 This is a schematic diagram of the sliding window size and position according to an embodiment of the present invention;
[0061] Figure 3 This is a schematic diagram of the segmentation model according to an embodiment of the present invention;
[0062] Figure 4(a) is an image with tear film rupture used in one embodiment of the present invention;
[0063] Figure 4(b) is a schematic diagram of the tear film breakup label corresponding to Figure 4(a);
[0064] Figure 5 This is the tear film breakage test area according to an embodiment of the present invention;
[0065] Figure 6(a) is a schematic diagram of the tear film breakup area according to an embodiment of the present invention. Figure 1 ;
[0066] Figure 6(b) shows the tear film breakup area corresponding to Figure 6(a). Figure 2 ;
[0067] Figure 6(c) is a tear film breakage trend diagram according to an embodiment of the present invention. Detailed Implementation
[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0069] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
[0070] Example 1
[0071] This invention provides a method for real-time tear film breakup time analysis, such as... Figure 1 As shown, it includes the following steps:
[0072] (1) Acquire images with tear film rupture in real time and record the image acquisition time and image sequence number; that is, image acquisition, record acquisition time and record image sequence number;
[0073] (2) Using a sliding window approach, based on the image sequence number, the image with tear film rupture is extracted to obtain a local image; that is, the local image is extracted using a sliding window.
[0074] (3) After converting each local image into a single-channel grayscale image, they are fed into the pre-trained segmentation model to obtain four types of segmentation results. The four types of segmentation results are the background segmentation image, the Placido highlight ring segmentation image, the ring breakage area segmentation image, and the eyelash occlusion area segmentation image; that is, inputting into the segmentation model to obtain four types of segmentation results.
[0075] (4) If two blinking behaviors are determined based on the Placido highlight ring segmentation map, all ring breakage region segmentation maps are fused according to the image sequence number and the corresponding sliding window; if a third blinking behavior occurs, a breakage region map is generated; if no two blinking behaviors occur, the blink count is continuously recorded; that is, blink detection and tear film breakage segmentation processing are performed.
[0076] (5) Based on the tear film breakup map, complete the tear film breakup time analysis.
[0077] In this embodiment of the invention, a segmentation model is built, which significantly reduces the computational load of the model. Furthermore, since eyelash occlusion affects the accuracy of tear film breakage recognition, this invention adds an eyelash occlusion region class to improve the segmentation model's ability to distinguish features, thereby improving accuracy.
[0078] Since the convolution operations of upsampling layers and large feature layers consume significant computational resources, appropriately reducing the cascading between shallow and deep features will decrease the model's computational load. Therefore, for example... Figure 3 As shown, in a specific embodiment of the present invention, the segmentation model includes a first 2x downsampling layer, a second 2x downsampling layer, a third 2x downsampling layer, a fourth 2x downsampling layer, an 8x upsampling layer, a 2x upsampling layer, a first convolutional activation function layer, a second convolutional activation function layer, a third convolutional activation function layer, a fourth convolutional activation function layer, a fifth convolutional activation function layer, a sixth convolutional activation function layer, and an output layer.
[0079] The first convolutional activation function layer and the first 2x downsampling layer are arranged sequentially to form the first unit;
[0080] The second convolutional activation function layer and the second 2x downsampling layer are arranged sequentially to form the second unit;
[0081] The third convolutional activation function layer and the third 2x downsampling layer are arranged sequentially to form the third unit;
[0082] The fourth convolutional activation function layer and the fourth 2x downsampling layer are arranged sequentially to form the fourth unit;
[0083] The first unit, the second unit, the third unit, and the fourth unit are connected in sequence;
[0084] The 8x upsampling layer is positioned between the fourth unit and the fifth convolutional activation function layer;
[0085] The 2x upsampling layer is positioned between the fifth convolutional activation function layer and the sixth convolutional activation function layer;
[0086] The output layer is connected to the sixth convolutional activation function layer.
[0087] Figure 3 In this model, the first 2x downsampling layer, the second 2x downsampling layer, the third 2x downsampling layer, and the fourth 2x downsampling layer are all represented using Maxpool2x2; the 8x upsampling layer is represented using Upsampling8x8; the 2x upsampling layer is represented using Upsampling2x2; the first convolutional activation function layer, the second convolutional activation function layer, the third convolutional activation function layer, the fourth convolutional activation function layer, the fifth convolutional activation function layer, and the sixth convolutional activation function layer are all represented using (Conv3x3, ReLU); the output layer is represented using Softmax; and each convolutional layer in the segmentation model is a 2D image convolution.
[0088] In the specific implementation process, if the size of the input image is 224×224×1, it becomes 112×112×16 after passing through the first unit; 56×56×32 after passing through the second unit; 28×28×64 after passing through the third unit; 14×14×28 after passing through the fourth unit; 112×112×144 after passing through an 8x upsampling layer; 112×112×16 after passing through a 2x upsampling layer; 224×224×17 after passing through the sixth convolutional activation function layer; and 224×224×4 after passing through the output layer, that is, the output structure is 4 segmentation categories.
[0089] In one specific embodiment of the present invention, the training method of the pre-trained segmentation model includes:
[0090] Acquire tear film breakup time videos of multiple people (with the subjects trying to keep their eyes open as much as possible), and split the tear film breakup time videos into images with tear film breakup to form a training set, as shown in Figure 4(a).
[0091] All images in the training set were labeled with categories, including background, Placido highlighted ring, broken ring region, and eyelash occlusion region. The eyelash occlusion region category was separated to enable the model to be trained to distinguish the feature differences between eyelash occlusion and tear film breakage, thereby improving the robustness to eyelash occlusion interference; see Figure 4(b) for details.
[0092] By using a sliding window approach, local images are extracted from the images with tear film rupture based on the image sequence number to simulate the actual usage process and improve model accuracy. In the specific implementation of this invention, data augmentation can also be achieved by random image contrast, gamma correction, hole punching, local adaptive histogram equalization, rotation, flipping, quality compression, optical changes, and other methods.
[0093] Each local image is converted into a single-channel grayscale image and downsampled to 224*224 before being fed into the segmentation model for training. The pre-trained segmentation model outputs four types of segmentation results: background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image, and eyelash occlusion region segmentation image.
[0094] In one specific embodiment of the present invention, the method for obtaining the local image includes the following steps:
[0095] Based on the image sequence number, n sliding windows are selected, and the corresponding images with tear film rupture are extracted in turn based on the window coordinates of each sliding window to obtain a local image. This invention uses a sliding window method to acquire local images for analysis, effectively balancing real-time performance and accuracy of tear film rupture details.
[0096] In the specific implementation process, such as Figure 2 As shown, the number of sliding windows is fixed at 4, and the window coordinates are set as follows: Window 1 (0.1, 0.1, 0.6, 0.6), Window 2 (0.3, 0.1, 0.6, 0.6), Window 3 (0.3, 0.3, 0.6, 0.6), and Window 4 (0.1, 0.3, 0.6, 0.6). The four numbers in parentheses represent the proportions of the top-left corner coordinates (horizontal pixel coordinate x, vertical pixel coordinate y, window width w, and window height h) of the sliding window in the original image to the pixel size of the original image.
[0097] Since the eyelids cannot reflect incident light from the Placido disk when the eyes are closed, blinking can be detected by obtaining the center coordinates of the Placido disk. In one specific embodiment of the present invention, blinking behavior is detected through the following steps:
[0098] The Hough circle is used to detect whether the Placido disk center coordinates exist in the Placido highlight ring segmentation map. Specifically, for the Placido highlight ring segmentation map, the Hough circle is used to detect and capture the center coordinates of the smallest circle obtained, which can be used to obtain the Placido disk center coordinates.
[0099] If the center coordinates of the Placido disk are not detected, it means that the current frame is in the closed-eye state;
[0100] If the center coordinates of the Placido disk are detected, it indicates that the current frame is in an open-eye state;
[0101] In the sequence analysis results, a continuous sequence of opening and closing the eyes is considered as one blink; a continuous sequence of opening and closing the eyes is considered as two blinks.
[0102] In one specific embodiment of the present invention, the method for generating the fracture region map includes the following steps:
[0103] Opening and closing operations and median filtering are performed sequentially on all the segmented images of the circular tear film rupture regions to obtain the tear film rupture segmentation results. This is because the segmentation model designed in this embodiment of the invention does not have artificially designed mutual constraints between adjacent pixels, which may result in tearing and holes in the segmentation results. In addition, the tear film rupture region only has relatively correct edge division, that is, it is sensitive to the presence or absence of the tear film rupture region but not to the correctness of the edge division.
[0104] Based on the selected four sliding windows, the segmentation results of four adjacent frames are stitched together. The overlapping areas are stitched together using a weighted average method to obtain a complete segmentation result of the tear film rupture position of each original image with tear film rupture. This invention integrates information from multiple consecutive frames to reduce the image with insufficient precision in a single frame, thereby improving the stability of the sequence image results.
[0105] The color of each small square in the tear film breakup map is generated based on all tear film breakup locations, with the color intensity representing the breakup time;
[0106] If a third blink is detected during the tear film breakup segmentation process, the tear film breakup segmentation process is terminated.
[0107] The weight selection rule is as follows: the weights are [0.6, 0.4] in the overlapping areas of sliding window 1 and sliding window 2, sliding window 2 and sliding window 3, sliding window 3 and sliding window 4, and sliding window 4 and sliding window 1, respectively. The weights in the overlapping areas of the four sliding windows are [0.1, 0.2, 0.3, 0.4] according to the order of image acquisition time.
[0108] In one specific embodiment of the present invention, the tear film breakup time analysis includes the following steps:
[0109] Fracture area calibration: Align the center coordinates of the Placido disk of the Placido highlight ring segmentation map with the center of the fracture area map, as shown in Figure 6(a);
[0110] The Placido highlighted ring segmentation image is subjected to multiple iterative erosion and dilation operations to fill the background between the highlighted rings, thereby obtaining the segmentation image of the inspected region. See details [link to documentation]. Figure 5 The parts that are not part of the inspected area are marked in gray on the fracture area map, see Figure 6(b) for details;
[0111] Tear film rupture map: If tear film rupture exists, the acquisition time of the current frame is recorded in the corresponding grid of the tear film rupture map. Each grid is assigned a value only once and is not updated.
[0112] Rupture trend graph: Calculate the ratio of the ruptured area to the inspected area in the image at each time point, see Figure 6(c) for details.
[0113] Example 2
[0114] Based on the same inventive concept as in Embodiment 1, this embodiment of the invention provides a real-time tear film breakup time analysis device, comprising:
[0115] The image acquisition module is used to acquire images with tear film rupture in real time and record the image acquisition time and image sequence number;
[0116] The local image acquisition module is used to extract the local image of the image with tear film rupture by using a sliding window method according to the image sequence number;
[0117] The image segmentation module is used to convert each local image into a single-channel grayscale image and then feed them into a pre-trained segmentation model to obtain four types of segmentation results. The four types of segmentation results are background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image and eyelash occlusion region segmentation image.
[0118] The fusion module is used to fuse all ring breakage region segmentation maps according to the image sequence number and the corresponding sliding window if two blinking behaviors are determined based on the Placido highlighted ring segmentation map. If a third blinking behavior occurs, a breakage region map is generated. If no two blinking behaviors occur, the blink count is continuously recorded. That is, blink detection and tear film breakage segmentation processing are performed.
[0119] The analysis module is used to perform tear film breakup time analysis based on the breakup region map.
[0120] This invention provides a method for real-time tear film breakup time analysis, such as... Figure 1 As shown, it includes the following steps:
[0121] (1) Acquire images with tear film rupture in real time and record the image acquisition time and image sequence number;
[0122] (2) Using a sliding window method, a partial image is obtained by extracting the image with tear film rupture according to the image number;
[0123] (3) After converting each local image into a single-channel grayscale image, they are fed into the pre-trained segmentation model to obtain four types of segmentation results. The four types of segmentation results are background segmentation image, Placido highlight ring segmentation image, ring broken area segmentation image and eyelash occlusion area segmentation image.
[0124] (4) If two blinking behaviors are determined based on the Placido highlight ring segmentation map, all ring breakage region segmentation maps are fused according to the image sequence number and the corresponding sliding window; if a third blink occurs, a breakage region map is generated to complete the tear film breakage analysis; if no two blinking behaviors occur, the blink count is recorded again.
[0125] (5) Based on the tear film breakup map, complete the tear film breakup time analysis.
[0126] In this embodiment of the invention, a segmentation model is built, which significantly reduces the computational load of the model. Furthermore, since eyelash occlusion affects the accuracy of tear film breakage recognition, this invention adds an eyelash occlusion region class to improve the segmentation model's ability to distinguish features, thereby improving accuracy.
[0127] Since the convolution operations of upsampling layers and large feature layers consume significant computational resources, appropriately reducing the cascading between shallow and deep features will decrease the model's computational load. Therefore, for example... Figure 3 As shown, in a specific embodiment of the present invention, the segmentation model includes a first 2x downsampling layer, a second 2x downsampling layer, a third 2x downsampling layer, a fourth 2x downsampling layer, an 8x upsampling layer, a 2x upsampling layer, a first convolutional activation function layer, a second convolutional activation function layer, a third convolutional activation function layer, a fourth convolutional activation function layer, a fifth convolutional activation function layer, a sixth convolutional activation function layer, and an output layer.
[0128] The first convolutional activation function layer and the first 2x downsampling layer are arranged sequentially to form the first unit;
[0129] The second convolutional activation function layer and the second 2x downsampling layer are arranged sequentially to form the second unit;
[0130] The third convolutional activation function layer and the third 2x downsampling layer are arranged sequentially to form the third unit;
[0131] The fourth convolutional activation function layer and the fourth 2x downsampling layer are arranged sequentially to form the fourth unit;
[0132] The first unit, the second unit, the third unit, and the fourth unit are connected in sequence;
[0133] The 8x upsampling layer is positioned between the fourth unit and the fifth convolutional activation function layer;
[0134] The 2x upsampling layer is positioned between the fifth convolutional activation function layer and the sixth convolutional activation function layer;
[0135] The output layer is connected to the sixth convolutional activation function layer.
[0136] Figure 3 In this model, the first 2x downsampling layer, the second 2x downsampling layer, the third 2x downsampling layer, and the fourth 2x downsampling layer are all represented using Maxpool2x2; the 8x upsampling layer is represented using Upsampling8x8; the 2x upsampling layer is represented using Upsampling2x2; the first convolutional activation function layer, the second convolutional activation function layer, the third convolutional activation function layer, the fourth convolutional activation function layer, the fifth convolutional activation function layer, and the sixth convolutional activation function layer are all represented using (Conv3x3, ReLU); the output layer is represented using Softmax; and each convolutional layer in the segmentation model is a 2D image convolution.
[0137] In the specific implementation process, if the size of the input image is 224×224×1, it becomes 112×112×16 after passing through the first unit; 56×56×32 after passing through the second unit; 28×28×64 after passing through the third unit; 14×14×28 after passing through the fourth unit; 112×112×144 after passing through an 8x upsampling layer; 112×112×16 after passing through a 2x upsampling layer; 224×224×17 after passing through the sixth convolutional activation function layer; and 224×224×4 after passing through the output layer, that is, the output structure is 4 segmentation categories.
[0138] In one specific embodiment of the present invention, the training method of the pre-trained segmentation model includes:
[0139] Acquire tear film breakup time videos of multiple people (with the subjects trying to keep their eyes open as much as possible), and split the tear film breakup time videos into images with tear film breakup to form a training set, as shown in Figure 4(a).
[0140] All images in the training set were labeled with categories, including background, Placido highlighted ring, broken ring region, and eyelash occlusion region. The eyelash occlusion region category was separated to enable the model to be trained to distinguish the feature differences between eyelash occlusion and tear film breakage, thereby improving the robustness to eyelash occlusion interference; see Figure 4(b) for details.
[0141] By using a sliding window approach, local images are extracted from the images with tear film rupture based on the image sequence number to simulate the actual usage process and improve model accuracy. In the specific implementation of this invention, data augmentation can also be achieved by random image contrast, gamma correction, hole punching, local adaptive histogram equalization, rotation, flipping, quality compression, optical changes, and other methods.
[0142] Each local image is converted into a single-channel grayscale image and downsampled to 224*224 before being fed into the segmentation model for training. The pre-trained segmentation model outputs four types of segmentation results: background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image, and eyelash occlusion region segmentation image.
[0143] In one specific embodiment of the present invention, the method for obtaining the local image includes the following steps:
[0144] Based on the image sequence number, n sliding windows are selected, and the corresponding images with tear film rupture are extracted in turn based on the window coordinates of each sliding window to obtain a local image. This invention uses a sliding window method to acquire local images for analysis, effectively balancing real-time performance and accuracy of tear film rupture details.
[0145] In the specific implementation process, such as Figure 2 As shown, the number of sliding windows is fixed at 4, and the window coordinates are set as follows: Window 1 (0.1, 0.1, 0.6, 0.6), Window 2 (0.3, 0.1, 0.6, 0.6), Window 3 (0.3, 0.3, 0.6, 0.6), and Window 4 (0.1, 0.3, 0.6, 0.6). The four numbers in parentheses represent the proportions of the top-left corner coordinates (horizontal pixel coordinate x, vertical pixel coordinate y, window width w, and window height h) of the sliding window in the original image to the pixel size of the original image.
[0146] Since the eyelids cannot reflect incident light from the Placido disk when the eyes are closed, blinking can be detected by obtaining the center coordinates of the Placido disk. In one specific embodiment of the present invention, blinking behavior is detected through the following steps:
[0147] The Hough circle is used to detect whether the Placido disk center coordinates exist in the Placido highlight ring segmentation map. Specifically, for the Placido highlight ring segmentation map, the Hough circle is used to detect and capture the center coordinates of the smallest circle obtained, which can be used to obtain the Placido disk center coordinates.
[0148] If the center coordinates of the Placido disk are not detected, it means that the current frame is in the closed-eye state;
[0149] If the center coordinates of the Placido disk are detected, it indicates that the current frame is in an open-eye state;
[0150] In the sequence analysis results, a continuous sequence of opening and closing the eyes is considered as one blink; a continuous sequence of opening and closing the eyes is considered as two blinks.
[0151] In one specific embodiment of the present invention, the method for generating the fracture region map includes the following steps:
[0152] Opening and closing operations and median filtering are performed sequentially on all the segmented images of the circular tear film rupture regions to obtain the tear film rupture segmentation results. This is because the segmentation model designed in this embodiment of the invention does not have artificially designed mutual constraints between adjacent pixels, which may result in tearing and holes in the segmentation results. In addition, the tear film rupture region only has relatively correct edge division, that is, it is sensitive to the presence or absence of the tear film rupture region but not to the correctness of the edge division.
[0153] Based on the selected four sliding windows, the segmentation results of four adjacent frames are stitched together. The overlapping areas are stitched together using a weighted average method to obtain a complete segmentation result of the tear film rupture position of each original image with tear film rupture. This invention integrates information from multiple consecutive frames to reduce the image with insufficient precision in a single frame, thereby improving the stability of the sequence image results.
[0154] The color of each small square in the tear film breakup map is generated based on all tear film breakup locations, with the color intensity representing the breakup time;
[0155] If a third blink is detected during the tear film breakup segmentation process, the tear film breakup segmentation process is terminated.
[0156] The weight selection rule is as follows: the weights are [0.6, 0.4] in the overlapping areas of sliding window 1 and sliding window 2, sliding window 2 and sliding window 3, sliding window 3 and sliding window 4, and sliding window 4 and sliding window 1, respectively. The weights in the overlapping areas of the four sliding windows are [0.1, 0.2, 0.3, 0.4] according to the order of image acquisition time.
[0157] In one specific embodiment of the present invention, the tear film breakup time analysis includes the following steps:
[0158] Align the center coordinates of the Placido disk in the Placido highlight ring segmentation map with the center of the fracture region map, as shown in Figure 6(a);
[0159] The Placido highlighted ring segmentation image is subjected to multiple iterative erosion and dilation operations to fill the background between the highlighted rings, thereby obtaining the segmentation image of the inspected region. See details [link to documentation]. Figure 5 The parts that are not part of the inspected area are marked in gray on the fracture area map, see Figure 6(b) for details;
[0160] If tear film rupture exists, the acquisition time of the current frame is recorded in the corresponding grid of the rupture area map. Each grid is assigned a value only once and is not updated.
[0161] Calculate the ratio of the area of the ruptured region to the area of the inspected region in the image at each time point, as shown in Figure 6(c).
[0162] Example 3
[0163] Based on the same inventive concept as in Embodiment 1, this embodiment of the invention provides a real-time tear film breakup time analysis system, including a storage medium and a processor;
[0164] The storage medium is used to store instructions;
[0165] The processor is configured to operate according to the instructions to execute the method according to any one of Embodiment 1.
[0166] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0167] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0170] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
[0171] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A real-time tear film break-up time analysis method, characterized by, include: Real-time acquisition of images with tear film rupture, and recording of image acquisition time and image sequence number; Using a sliding window approach, a partial image is obtained by extracting the image with tear film rupture based on the image sequence number; After converting each local image into a single-channel grayscale image, they are fed into a pre-trained segmentation model to obtain four types of segmentation results: background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image, and eyelash occlusion region segmentation image. If two blinking behaviors are determined based on the Placido highlight ring segmentation map, then all ring breakage region segmentation maps are fused according to the image sequence number and the corresponding sliding window. If a third blinking behavior occurs, a breakage region map is generated. Based on the tear film breakup map, tear film breakup time analysis was performed. The method for generating the fracture zone map includes the following steps: The tear film rupture segmentation results are obtained by sequentially performing opening and closing operations and median filtering on all the segmentation maps of the circular rupture regions. Based on the selected four sliding windows, the segmentation results of four adjacent frames are stitched together. The overlapping areas are stitched together using a weighted average method to obtain the complete segmentation result of the tear film breakage location of each original image with tear film breakage. The color of each small square in the tear film breakup map is generated based on all tear film breakup locations, with the color intensity representing the breakup time; The weight selection rule is as follows: the weights are [0.6, 0.4] in the overlapping areas of sliding window 1 and sliding window 2, sliding window 2 and sliding window 3, sliding window 3 and sliding window 4, and sliding window 4 and sliding window 1, respectively. The weights in the overlapping areas of the four sliding windows are [0.1, 0.2, 0.3, 0.4] according to the order of image acquisition time.
2. The method for real-time tear film breakup time analysis according to claim 1, characterized in that: The segmentation model includes a first 2x downsampling layer, a second 2x downsampling layer, a third 2x downsampling layer, a fourth 2x downsampling layer, an 8x upsampling layer, a 2x upsampling layer, a first convolutional activation function layer, a second convolutional activation function layer, a third convolutional activation function layer, a fourth convolutional activation function layer, a fifth convolutional activation function layer, a sixth convolutional activation function layer, and an output layer. The first convolutional activation function layer and the first 2x downsampling layer are arranged sequentially to form the first unit; The second convolutional activation function layer and the second 2x downsampling layer are arranged sequentially to form the second unit; The third convolutional activation function layer and the third 2x downsampling layer are arranged sequentially to form the third unit; The fourth convolutional activation function layer and the fourth 2x downsampling layer are arranged sequentially to form the fourth unit; The first unit, the second unit, the third unit, and the fourth unit are connected in sequence; The 8x upsampling layer is positioned between the fourth unit and the fifth convolutional activation function layer; The 2x upsampling layer is positioned between the fifth convolutional activation function layer and the sixth convolutional activation function layer; The output layer is connected to the sixth convolutional activation function layer.
3. A method for real-time tear film breakup time analysis according to claim 1 or 2, characterized in that: The training method for the pre-trained segmentation model includes: Acquire tear film breakup time videos of multiple individuals, and split the tear film breakup time videos into images with tear film breakup to form a training set; All images in the training set are labeled with categories, including background, Placido highlighted rings, broken ring regions, and eyelash-occluded regions. Using a sliding window approach, a partial image is obtained by extracting the image with tear film rupture based on the image sequence number; After converting each local image into a single-channel grayscale image, they are fed into the segmentation model for training to obtain a pre-trained segmentation model. The output of the pre-trained segmentation model includes four types of segmentation results: background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image, and eyelash occlusion region segmentation image.
4. The method for real-time tear film breakup time analysis according to claim 1, characterized in that: The method for obtaining the local image includes the following steps: Select according to image number Each sliding window takes turns extracting the corresponding image with tear film rupture based on its window coordinates, thereby obtaining a local image.
5. The method for real-time tear film breakup time analysis according to claim 4, characterized in that: The number of sliding windows is fixed at 4, and the window coordinates are set as follows: Window 1 (0.1, 0.1, 0.6, 0.6), Window 2 (0.3, 0.1, 0.6, 0.6), Window 3 (0.3, 0.3, 0.6, 0.6), and Window 4 (0.1, 0.3, 0.6, 0.6). The four numbers in parentheses represent the proportion of the top-left corner coordinates (horizontal pixel coordinate x, vertical pixel coordinate y, window width w, and window height h) of the sliding window in the original image to the pixel size of the original image.
6. The method for real-time tear film breakup time analysis according to claim 1, characterized in that: Blinking behavior is detected through the following steps: Hough circle detection is used to check if the Placido disk center coordinates exist in the Placido highlight ring segmentation map; If the center coordinates of the Placido disk are not detected, it means that the current frame is in the closed-eye state; If the center coordinates of the Placido disk are detected, it indicates that the current frame is in an open-eye state; In the sequence analysis results, a continuous sequence of opening and closing the eyes is considered as one blink; a continuous sequence of opening and closing the eyes is considered as two blinks.
7. The method for real-time tear film breakup time analysis according to claim 1, characterized in that: The tear film breakup time analysis includes the following steps: Align the center coordinates of the Placido disk in the Placido highlight ring segmentation map with the center of the fracture area map; The Placido highlighted ring segmentation image is subjected to multiple iterations of erosion and dilation operations to fill the background between the highlighted rings, thereby obtaining the segmentation image of the inspected region; parts that do not belong to the inspected region are marked in gray in the fracture area image; If tear film rupture exists, the acquisition time of the current frame is recorded in the corresponding grid of the rupture area map. Each grid is assigned a value only once and is not updated. Calculate the ratio of the area of the fractured region to the area of the inspected region in the image at each time point.
8. A real-time tear film breakup time analysis device, characterized in that, include: The image acquisition module is used to acquire images with tear film rupture in real time and record the image acquisition time and image sequence number; The local image acquisition module is used to extract the local image of the image with tear film rupture by using a sliding window method according to the image sequence number; The image segmentation module is used to convert each local image into a single-channel grayscale image and then feed them into a pre-trained segmentation model to obtain four types of segmentation results. The four types of segmentation results are background segmentation image, Placido highlight ring segmentation image, ring breakage region segmentation image and eyelash occlusion region segmentation image. The fusion module is used to fuse all ring breakage region segmentation maps according to the image sequence number and the corresponding sliding window if two blinking behaviors are determined based on the Placido highlighted ring segmentation map. If a third blinking behavior occurs, a breakage region map is generated. The analysis module is used to perform tear film breakup time analysis based on the breakup region map; The method for generating the fracture zone map includes the following steps: The tear film rupture segmentation results are obtained by sequentially performing opening and closing operations and median filtering on all the segmentation maps of the circular rupture regions. Based on the selected four sliding windows, the segmentation results of four adjacent frames are stitched together. The overlapping areas are stitched together using a weighted average method to obtain the complete segmentation result of the tear film breakage location of each original image with tear film breakage. The color of each small square in the tear film breakup map is generated based on all tear film breakup locations, with the color intensity representing the breakup time; The weight selection rule is as follows: the weights are [0.6, 0.4] in the overlapping areas of sliding window 1 and sliding window 2, sliding window 2 and sliding window 3, sliding window 3 and sliding window 4, and sliding window 4 and sliding window 1, respectively. The weights in the overlapping areas of the four sliding windows are [0.1, 0.2, 0.3, 0.4] according to the order of image acquisition time.
9. A real-time tear film breakup time analysis system, characterized in that: Including storage media and processor; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the method according to any one of claims 1-7.