Method and device for detecting tear film breakup, and computer program comprising same

Abstract

The present invention relates to a tear film breakup detection method, which may comprise the steps of: collecting an eye image of a user; filtering the eye image of the user so as to generate a preprocessed image; classifying, through a preset model, a frame in which at least one tear film of the user breaks up in the preprocessed image; determining a tear film breakup area in the frame in which the tear film breaks up, and classifying a tear film breakup form; and determining, as a tear film breakup time, the time difference between the frame in which the user first blinks and the frame in which the tear film of the user breaks up.

Description

Method for detecting tear film breakdown, device and computer program including the same

[0001] The present invention relates to a tear film destruction detection method, a device, and a computer program including the same.

[0002] [National Research and Development Project Supporting This Invention]

[0003] [Unique number]711189763

[0004] [Task Number] 2022R1A2C2006109

[0005] [Ministry Name] Ministry of Science and ICT

[0006] [Name of Project Management (Specialist) Institution] National Research Foundation of Korea

[0007] [Research Project Name] Individual Basic Research (Ministry of Science and ICT)

[0008] [Research Project Name] Corneal Wound Repair Using a Tissue Engineering Support Drug Delivery System

[0009] Oil and inflammation control

[0010] [Name of the project performing organization] Catholic University (Sung-i Campus)

[0011] [Research Period] March 1, 2022 - February 28, 2027

[0012] Dry eye is a tear film disorder characterized by abnormalities in the tear film, which maintains moisture in the eyes. The tear film is composed of a mucus layer, an aqueous layer, and a lipid layer. The aqueous layer of the tear film comprises approximately 95% of total tears and is primarily responsible for providing moisture to the eye. The mucus layer forms the innermost layer of the tear film, allowing the aqueous layer to adhere to the ocular surface and retain moisture. The lipid layer, the outermost layer of the tear film, prevents evaporation of the aqueous layer.

[0013] Dry eye is typically diagnosed by staining the tear film with a dye such as fluorescein, observing the disappearance of dry spots on the stained eye, and measuring the time it takes for these spots to appear. This diagnosis of dry eye is based on clinical assessment using a slit-lamp microscope. However, this clinical assessment is subject to some error and quantitative documentation is difficult.

[0014] As technology for image processing and artificial intelligence modeling advances, there is a need for technologies to assist with these clinical judgments and to provide accurate quantitative records.

[0015]

[0016] One embodiment of the present invention relates to a tear film destruction detection method, a device, and a computer program including the same, which can detect tear film destruction through a filtered ocular image.

[0017] One embodiment of the present invention relates to a tear film destruction detection method, a device, and a computer program including the same, which can specify an area where a tear film is destroyed and classify the shape of the tear film destruction.

[0018] One embodiment of the present invention relates to a tear film breakdown detection method, a device, and a computer program including the same, which can calculate the time point at which a tear film is broken from the last blink of a user.

[0019]

[0020] A method for detecting tear film breakdown according to one embodiment of the present invention may include the steps of: collecting an eye image of a user; filtering the eye image of the user to generate a preprocessed image; classifying at least one frame in which the tear film of the user is broken from the preprocessed image using a preset model; determining a tear film breakdown area in the frame in which the tear film is broken and classifying a tear film breakdown shape; and determining a time difference between the frame in which the tear film of the user is first broken and the blink of the user's eye as a tear film breakdown time.

[0021] The above user's eye image may be an image of the user's eye coated with a fluorescent material.

[0022] The step of generating the above preprocessed image can generate the above preprocessed image through filtering that passes the fluorescent material.

[0023] The above preset model may be learned based on images of a normal tear film state and a destroyed tear film state.

[0024] The above-described preset model can learn images in which the tear film is normal and classify frames in which the tear film is destroyed based on the difference between the input value and the output value.

[0025] In the step of classifying the tear film destruction shape, the tear film destruction area can be determined by specifying an area where tear film destruction occurs in a template divided into at least one area on the user's eye image.

[0026] The method may further include a step of determining the user's dry eye type based on the tear film breakdown shape and tear film breakdown time.

[0027] The method may further include a step of determining whether the user's eyes are dry based on the tear film breakdown time.

[0028] The present invention relates to a computer program stored in a computer-readable recording medium that performs a tear film destruction detection method according to one embodiment of the present invention.

[0029] A tear film breakdown detection device according to one embodiment of the present invention comprises at least one memory; and at least one processor, wherein the processor collects an eye image of a user, filters the eye image of the user to generate a preprocessed image, classifies a frame in which at least one tear film of the user is broken in the preprocessed image through a preset model, determines a tear film breakdown area in the frame in which the tear film is broken, classifies a tear film breakdown shape, and determines a time difference between a blink of the user's eye and a frame in which the tear film of the user is first broken as a tear film breakdown time.

[0030]

[0031] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment must include all or only the following effects, and therefore the scope of the disclosed technology should not be construed as being limited thereby.

[0032] The present invention relates to a tear film destruction detection method, a device and a computer program including the same, which can detect tear film destruction through a filtered ocular image.

[0033] The present invention relates to a tear film destruction detection method, a device and a computer program including the same, which can specify an area where a tear film is destroyed and classify the shape of the tear film destruction.

[0034] The present invention relates to a tear film destruction detection method, a device and a computer program including the same, which can calculate the time point at which a tear film is destroyed from the user's last eye blink.

[0035]

[0036] FIG. 1 is a drawing illustrating the physical configuration of a tear film destruction detection device according to one embodiment of the present invention.

[0037] FIG. 2 is a drawing illustrating the functional configuration of a processor according to one embodiment of the present invention.

[0038] FIG. 3 is a diagram comparing eye images depending on whether filtering is performed according to one embodiment of the present invention.

[0039] Figure 4 is a drawing illustrating tear film destruction according to one embodiment of the present invention.

[0040] Figure 5 is a drawing explaining a tear film destruction shape according to one embodiment of the present invention.

[0041] Figure 6 is a drawing illustrating a tear film destruction location according to one embodiment of the present invention.

[0042] FIG. 7 is a drawing illustrating a method for classifying tear film destruction frames according to one embodiment of the present invention.

[0043] FIG. 8 is a drawing illustrating a method for determining tear film breakdown time according to one embodiment of the present invention.

[0044] FIG. 9 is a drawing explaining the sequence of a tear film destruction detection method according to one embodiment of the present invention.

[0045]

[0046] The description of the present invention is merely an example for structural and functional explanation, and therefore, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be modified in various ways and can take various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purposes or effects presented in the present invention do not mean that a specific embodiment must include all of them or only such effects, and therefore, the scope of the present invention should not be construed as being limited thereby.

[0047] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0048] Terms such as "first" and "second" are intended to distinguish one component from another, and the scope of the rights should not be limited by these terms. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

[0049] When a component is said to be "connected" to another component, it should be understood that while it may be directly connected to that other component, there may also be other components intervening. Conversely, when a component is said to be "directly connected" to another component, it should be understood that there are no other intervening components. Similarly, other expressions describing relationships between components, such as "between" and "directly between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0050] Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "comprises" or "have" should be understood to specify the presence of a feature, number, step, operation, component, part or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0051] For each step, the identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps. The steps may occur in a different order than stated unless the context clearly dictates a specific order. That is, the steps may occur in the same order as stated, may be performed substantially simultaneously, or may be performed in the opposite order.

[0052] The present invention can be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also includes those implemented in the form of a carrier wave (e.g., transmission via the Internet). Furthermore, the computer-readable recording medium can be distributed across network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner.

[0053] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted to be consistent with their meaning within the context of the relevant technology, and should not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0054]

[0055] FIG. 1 is a drawing illustrating the physical configuration of a tear film destruction detection device (100) according to one embodiment.

[0056] Referring to FIG. 1, a tear film destruction detection device (100) can be implemented by including a processor (110), a memory (130), a user input / output unit (150), and a network input / output unit (170).

[0057] The processor (110) can execute a procedure that performs an operation of preprocessing an image of the user's eye, classifying a frame in which the user's tear film is destroyed from the preprocessed image, determining a tear film destruction area within the frame, and classifying the tear film destruction shape to determine the tear film destruction time, and can manage the memory (130) that is read or written throughout the process, and can schedule a synchronization time between the volatile memory and the non-volatile memory in the memory (130). The processor (110) can control the overall operation of the tear film destruction detection device (100), and can be electrically connected to the memory (130), the user input / output unit (150), and the network input / output unit (170) to control the data flow therebetween. The processor (110) can be implemented as a CPU (Central Processing Unit) of the tear film destruction detection device (100).

[0058] The memory (130) may include an auxiliary memory device implemented as a non-volatile memory such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive) and used to store all data required for the tear film destruction detection device (100), and may include a main memory device implemented as a volatile memory such as a RAM (Random Access Memory).

[0059] The user input / output unit (150) may include an environment for receiving user input and an environment for outputting specific information to the user. For example, the user input / output unit (150) may include an input device including an adapter such as a touchpad, a touch screen, a virtual keyboard, or a pointing device, and an output device including an adapter such as a monitor or a touch screen. In one embodiment, the user input / output unit (150) may correspond to a computing device accessed via remote access, and in such a case, the tear film breakdown detection device (100) may be implemented as a server.

[0060] The network input / output unit (170) includes an environment for connecting to an external device or system via a network, and may include an adapter for communication such as a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), and a value added network (VAN).

[0061]

[0062] FIG. 2 is a diagram illustrating the functional configuration of a processor (110) according to one embodiment of the present invention. Referring to FIG. 2, the processor (110) may include an image collection unit (210), a preprocessing image generation unit (220), an image classification unit (230), a tear film destruction shape classification unit (240), a tear film destruction time determination unit (250), a dry eye type determination unit (260), and a dry eye determination unit (270).

[0063] The image collection unit (210) can collect an image of the user's eye. Here, the user may be a patient subject to an ocular examination, and the image of the user's eye may be an image of the user's eye coated with a fluorescent substance. The fluorescent substance may be fluorescein, but is not limited thereto. In other words, the image of the user's eye may be an image that can be analyzed using a preset model, and may be an image of the same form as an image generally used to diagnose a patient in a clinical setting.

[0064] The preprocessing image generation unit (220) can filter the user's eye image to generate a preprocessing image. The preprocessing image generation unit (220) can filter the dyed user's eye image to generate a preprocessing image so that it can be analyzed by a clinical or preset model. Here, the preprocessing can be performed according to a known preprocessing method and is not limited to a single method.

[0065] FIG. 3 is a diagram comparing ocular images depending on whether filtering is performed according to an embodiment of the present invention. Referring to FIG. 3, the preprocessing image generation unit (220) can generate a preprocessing image by filtering a fluorescent material to pass through. For example, the preprocessing image generation unit (220) can generate a preprocessing image by filtering the user's ocular image through a yellow filter. FIG. 3 (a) is an image processed by a yellow filter, and FIG. 3 (b) is an image not processed by a yellow filter. The preprocessing image generation unit (220) can provide the effect of inserting a physical yellow filter into a microscope through separate image filtering, without inserting a physical yellow filter into the microscope. In other words, the preprocessing image generation unit (220) can filter the user's ocular image into a form that is easier for a preset model to learn, and can generate a preprocessing image so that a specialist treating the user can easily perform the treatment.

[0066] The image classification unit (230) can classify at least one frame in which the user's tear film is destroyed from the preprocessed image using a preset model. Here, the preset model may be one that has been previously learned based on images of a normal tear film and images of a destroyed tear film. For example, the preset model may be generated according to a known method using a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Generative Adversarial Network (GAN), etc. That is, the preset model can learn the labeled images of a destroyed tear film and images of a normal tear film, and classify the images of a destroyed tear film and the images of a normal tear film. Referring to FIG. 4, for example, the image classification unit (230) can determine that the tear film in region a is destroyed, and classify frames in which the tear film is destroyed, such as a, from among the preprocessed images.

[0067] In another embodiment, a pre-configured model can learn images of a normal tear film and classify frames with a destroyed tear film based on the difference between the input and output values. Here, the pre-configured model can be created as an Auto-Encoder-based model, which learns images of a normal tear film, inputs a specific image, calculates a loss value, and determines an image with a destroyed tear film if the loss value exceeds a threshold.

[0068] Referring to FIGS. 4 to 6, the tear film destruction shape classification unit (240) can determine a tear film destruction area in a frame where the tear film is destroyed and classify the tear film destruction shape. Here, the tear film destruction area can be divided into C (central), S (superior), T (temporal), N (nasal), and I (inferior). More specifically, the tear film destruction shape classification unit (240) can divide the tear film destruction areas of the frames where the tear film is destroyed into five areas, and can label where the tear film destruction of the corresponding frames occurred. For example, the tear film destruction shape classification unit (240) can determine that the tear film is destroyed in C and N of the frame where the tear film is destroyed for the first time in FIG. 4.

[0069] Referring to FIG. 5, the tear film breakdown shape classification unit (240) can classify the tear film breakdown shape in a frame where the tear film is broken. That is, the tear film breakdown shape classification unit (240) can classify the shape of the tear film breakdown of the user according to the type. For example, the tear film breakdown shape classification unit (240) can classify (a) of FIG. 5 as an area breakdown type, (b) of FIG. 5 as a circular breakdown type, (c) of FIG. 5 as a line breakdown type, (d) of FIG. 5 as a dimple breakdown type, and (e) of FIG. 5 as a random breakdown type. Here, the tear film breakdown shape classification unit (240) can determine the tear film breakdown area and classify the tear film breakdown shape through a known object detection model, etc., and is not limited to a specific model, and can determine the tear film breakdown area and classify the tear film breakdown shape through an ensemble of multiple models.

[0070] In one embodiment, the tear film breakdown shape classification unit (240) can determine a tear film breakdown area by specifying an area where tear film breakdown occurs in a template divided into at least one area on the user's eye image. Here, the template divided into at least one area can be divided into five areas as described above. In other words, the template divided into at least one area can be divided into C (central), S (superior, upward direction), T (temporal, toward the ear), N (nasal, toward the nose), and I (inferior, downward direction).

[0071] Referring to FIGS. 7 and 8, the tear film breakdown time determination unit (250) can determine the time difference between the frames in which the user's tear film is first broken from the user's eye blink as the tear film breakdown time. More specifically, the tear film breakdown time determination unit (250) can determine the time difference between the time point (t2) at which the tear film breakdown frames classified by the image classification unit (230) are detected and the time point (t1) at which the user's eye blink occurs as the tear film breakdown time.

[0072] The dry eye type determination unit (260) may determine the user's dry eye type based on the tear film breakdown shape and tear film breakdown time. For example, if the tear film breakdown time is less than a specific time, the dry eye type determination unit (260) may determine the tear film breakdown shape as an area breakdown, a circle breakdown, a line breakdown, a dimple breakdown, or a random breakdown type. For example, if the user's dry eye type is an area breakdown, the dry eye type determination unit (260) may determine that the user's dry eye level is severe. In addition, if the user's dry eye type is a circle breakdown, the dry eye type determination unit (260) may determine that the user's dry eye level is an overall state of reduced wetness. In addition, if the user's dry eye type is a line breakdown, the dry eye type determination unit (260) may determine that the user's dry eye level is a mild or moderate type of moisture secretion deficiency. In addition, the dry eye type determination unit (260) can determine that the user's dry eye type is a state of mild to moderate wetness when the user's dry eye type is dimple destruction. In addition, the dry eye type determination unit (260) can determine that the user's dry eye type is a state of mild to moderate wetness when the user's dry eye type is random destruction when the user's dry eye type is evaporative dry eye.

[0073] The dry eye determination unit (270) can determine whether the user has dry eyes based on the tear film breakdown time. For example, the dry eye determination unit (270) can determine that the patient has mild dry eye if the tear film breakdown time of the user is 5 to 10 seconds, and the dry eye determination unit (270) can determine that the patient has severe dry eye if the tear film breakdown time of the user is 5 seconds or less.

[0074]

[0075] FIG. 9 is a diagram illustrating the sequence of a tear film destruction detection method according to one embodiment of the present invention. Referring to FIG. 9, the tear film destruction detection method can collect an image of the user's eye through an image collection unit (210) (S910).

[0076] The tear film destruction detection method can generate a preprocessed image by filtering the user's eye image through the preprocessed image generation unit (220) (S920).

[0077] The tear film destruction detection method can classify at least one frame in which the tear film of a user is destroyed in a preprocessed image through a preset model via an image classification unit (230) (S930).

[0078] The tear film destruction detection method can determine the tear film destruction area in a frame where the tear film is destroyed and classify the tear film destruction shape through the tear film destruction shape classification unit (240) (S940).

[0079] The tear film destruction detection method can determine the time difference between the frames where the user's tear film is destroyed for the first time from the user's eye blink as the tear film destruction time through the tear film destruction time determination unit (250) (S950).

[0080]

[0081] Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0082]

[0083] The tear film destruction detection method, device and computer program including the same according to an embodiment of the present invention can be used in various fields such as image processing and artificial intelligence modeling.

Claims

1. Step of collecting user's eye image; A step of filtering the user's eye image to generate a preprocessed image; A step of classifying at least one frame in which the tear film of a user is destroyed in the preprocessed image using a preset model; A step of determining a tear film destruction area in the frame where the tear film is destroyed and classifying the tear film destruction shape; and A tear film breakdown detection method comprising the step of determining the time difference between the frames in which the user's tear film is first broken from the user's eye blink as the tear film breakdown time.

2. In paragraph 1, The above user's eye image is, A method for detecting tear film destruction characterized in that the image is a fluorescent material coated image of the user's eye.

3. In paragraph 2, The step of generating the above preprocessed image is: A tear film destruction detection method characterized by generating the preprocessed image through filtering that passes the fluorescent material.

4. In paragraph 1, The above preset model is, A tear film destruction detection method characterized by being learned based on images of a normal tear film state and a destroyed tear film state.

5. In paragraph 1, The above preset model is, A tear film destruction detection method characterized by learning an image in a normal tear film state and classifying a frame in which the tear film is destroyed based on the difference between the input value and the output value.

6. In paragraph 1, In the step of classifying the tear film destruction shape, A tear film breakdown detection method characterized in that the tear film breakdown area is determined by specifying an area where tear film breakdown occurs in a template divided into at least one area on a user's eye image.

7. In paragraph 1, A tear film destruction detection method further comprising a step of determining the user's dry eye type based on the tear film destruction shape and the tear film destruction time.

8. In paragraph 1, A tear film breakdown detection method further comprising a step of determining whether the user's eyes are dry based on the tear film breakdown time.

9. A computer program stored in a computer-readable recording medium for executing a tear film destruction detection method according to any one of paragraphs 1 to 8 on a computer.

10. At least one memory; and Contains at least one processor, The above processor, Collect user's eye images, Filtering the user's eye image to generate a preprocessed image, Classifying at least one frame in which the tear film of the user is destroyed in the preprocessed image using a preset model, In the above tear film destruction frame, the tear film destruction area is determined, and the tear film destruction shape is classified. A computing device that determines the tear film breakup time as the time difference between the frames where the user's tear film is first broken from the user's eye blink.

Images