Tear layer evaluation device, tear layer evaluation program, and tear layer evaluation method

Abstract

In order to improve determination accuracy by subdividing a captured image, and to evaluate a tear layer in consideration of changes in the tear layer over time, captured video data obtained by capturing an image of an eye to be examined is acquired, a plurality of frames to be determined are extracted from the captured video data according to a predetermined extraction rule, and the frames to be determined are extracted from the captured video data according to a predetermined extraction rule. A plurality of sub-region image data of a predetermined size are extracted from a frame to be determined, whether or not each of the sub-region image data is an image showing a predetermined tear state is determined, and if the number of sub-region image data of the image showing the predetermined tear state is determined to exceed a predetermined threshold value, the sub-region image data is determined to be an image showing the predetermined tear state. When the frames to be determined are arranged in a time sequence, the frames to be determined are determined to be candidate images showing the prescribed tear state, and when the frames to be determined, which are determined to be candidate images showing the prescribed tear state, satisfy a prescribed continuous condition, the frames to be determined are determined to be candidate images showing the prescribed tear state, and the frames to be determined are determined to be candidate images showing the prescribed tear state. It is determined that the eye to be examined presented in the captured video data is in a prescribed tear state.

Description

Technical Field

[0001] The present invention relates to a tear layer evaluation device, a tear layer evaluation program, and a tear layer evaluation method for analyzing the state of an eye to be examined. Background Art

[0002] Conventionally, various proposals have been made regarding methods of photographing an eye to be examined and evaluating the state of the eye to be examined by a computer using the photographed image. For example, Patent Document 1 has proposed an ophthalmologic apparatus for diagnosing dry eye syndrome.

[0003] Patent document 1 discloses an ophthalmic device that uses illumination light to perform color photography of an interference pattern formed in the tear layer (lacrimal fluid layer) of an eye to be examined, decomposes the captured color image into multiple color components, determines the interference fringes appearing in each color component based on the signal of each decomposed color component, and calculates the progression of dry eye disease based on the number of interference fringes determined for each color component or the change in the signal level of the determined interference fringes.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent document 1: Japanese Patent Application Publication No. 2005-211173. Summary of the invention

[0007] Problem that the invention aims to solve

[0008] However, as in the ophthalmic device described in the above-mentioned patent document 1, various schemes have been proposed for the technology of determining whether a specified tear state such as dry eye syndrome is present based on a photographed image obtained by photographing the examined eye. However, in general, the method of evaluating an ophthalmic image as a whole has the advantage of short analysis time because the result is obtained by analyzing only one image. On the contrary, there are the following disadvantages: (1) Although the existence of the extraction object can be predicted, it is not known where in the image it exists, so the auxiliary diagnosis ability is poor; (2) It is extremely difficult to extract the structure or characteristics of an object that only exists in a very small part of the image; (3) It is conceivable that noise factors (the area of ​​eyelashes with strong contrast or the outline of the pupil in the interference fringe image) are mistakenly captured as the extraction object, etc.

[0009] In addition, there is a problem that even if a predetermined tear state such as dry eye is evaluated in one ophthalmic image, it cannot be determined that the subject has dry eye based on the single ophthalmic image. This is because there is a problem that the predetermined tear state is evaluated based on a single ophthalmic image without reliability, such as when only one image is detected by mistake by chance or when the diagnosis condition is that the predetermined tear state continues for a certain period of time.

[0010] The present invention is made in view of the above-mentioned problems, and its purpose is to provide a tear layer evaluation device, a tear layer evaluation program and a tear layer evaluation method, which not only subdivide the captured image to improve the judgment accuracy, but also take into account the changes of the tear layer over time.

[0011] Means used to solve problems

[0012] The tear layer evaluation device of the present invention evaluates the tear state of the inspected eye based on the captured video data obtained by photographing the inspected eye, and is characterized by comprising: a captured video data acquisition unit, which acquires the captured video data obtained by photographing the inspected eye; a determination object frame extraction unit, which extracts multiple determination object frames from the captured video data according to a specified extraction rule; a frame unit determination unit, which determines whether the determination object frame is a candidate image showing a specified tear state according to a specified determination condition; and a video unit determination unit, which, when the determination object frames are arranged in time sequence, determines that the inspected eye presented in the captured video data is in the specified tear state when the determination object frame determined to be a candidate image showing the specified tear state satisfies a specified continuity condition.

[0013] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that the frame unit judgment unit sets and stores a threshold value of a relevant number for each type of tear state of the judgment object, and the number is the number of sub-sector image data (Sector Image Data) used to judge whether the judgment object frame is an image showing a specified tear state.

[0014] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that the sub-sector unit determination unit performs extraction of the sub-sector image data for each of the plurality of sub-sector image data in a manner such that at least a portion of the image area overlaps with other sub-sector image data.

[0015] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that it has a display processing unit that generates a display screen to be displayed on a display device, and the display processing unit makes a display representing an image showing a specified tear state overlapped on the determination object frame so that a corresponding area of ​​the determination object frame can be visually identified, and the corresponding area of ​​the determination object frame is the corresponding area of ​​the determination object frame that is the extraction source of the sub-area image data of the image determined by the sub-area unit determination unit as showing a specified tear state.

[0016] In addition, the tear layer evaluation device of the present invention is also characterized in that it has a display processing unit that generates a display screen for display on a display device, and the display processing unit uses a graph to display the time-series change in the number of sub-area image data that is judged to be an image showing a specified tear state while arranging the judgment object frames in time sequence.

[0017] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that, when the display processing unit arranges the determination object frame in time series on the basis of dividing the determination object frame into a plurality of statistical areas, the display processing unit counts the number of the sub-area image data that are determined to be images showing a specified tear state for each statistical area, and uses a graph to display the changes in each statistical area in time series.

[0018] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that, when the video unit judgment unit arranges the judgment object frames in time sequence, the judgment object frames that are judged to be candidate images showing a specified tear state appear continuously for more than a preset specified number of times as the continuity condition.

[0019] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that, when the video unit judgment unit extracts a predetermined number of frames for an arbitrary part on the basis of arranging the judgment object frames in time sequence, the judgment object frames that are judged as candidate images showing a specified tear state appear at a predetermined ratio or above within the extracted range as the continuity condition.

[0020] In addition, in the tear layer evaluation device of the present invention, it is also characterized in that, when the video unit judgment unit arranges the judgment object frames in time sequence, the continuous appearance of the judgment object frame judged as the candidate image showing a specified tear state appears from the point where the judgment object frame appears until the eyelid is closed, and is judged as the continuity condition.

[0021] The tear layer evaluation program of the present invention enables a computer to implement a process of evaluating the tear state of an eye under examination based on captured video data obtained by photographing the eye under examination, and is characterized in that the computer is enabled to implement: a captured video data acquisition function for acquiring captured video data obtained by photographing the eye under examination; a judgment object frame extraction function for extracting a plurality of judgment object frames from the captured video data according to prescribed extraction rules; a frame unit judgment function for judging whether the judgment object frame is a candidate image showing a prescribed tear state according to prescribed judgment conditions; and a video unit judgment function for judging, when the judgment object frames are arranged in time sequence and when the judgment object frame judged to be a candidate image showing a prescribed tear state satisfies prescribed continuity conditions, the eye under examination presented in the captured video data is judged to be in a prescribed tear state.

[0022] The tear layer evaluation method of the present invention utilizes a computer to evaluate the tear state of the inspected eye based on captured video data obtained by photographing the inspected eye, and is characterized by comprising: a captured video data acquisition step for acquiring the captured video data obtained by photographing the inspected eye; a determination object frame extraction step for extracting a plurality of determination object frames from the captured video data based on prescribed extraction rules; a frame unit determination step for determining whether the determination object frame is a candidate image showing a prescribed tear state based on prescribed determination conditions; and a video unit determination step for determining, when the determination object frames are arranged in time sequence and when the determination object frame determined to be a candidate image showing a prescribed tear state satisfies prescribed continuity conditions, that the inspected eye presented in the captured video data is in the prescribed tear state.

[0023] Effects of the Invention

[0024] One or more of the above deficiencies are solved by the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a block diagram showing an example of the configuration of a tear layer evaluation device and peripheral devices corresponding to at least one embodiment of the present invention.

[0026] Figure 2 It is an explanatory diagram showing an example of the configuration of an imaging device corresponding to at least one embodiment of the present invention.

[0027] Figure 3 This is a block diagram showing an example of the configuration of a tear layer evaluation device corresponding to at least one embodiment of the present invention.

[0028] Figure 4 This is a flowchart showing an example of the flow of determination processing corresponding to at least one embodiment of the present invention.

[0029] Figure 5 It is an explanatory diagram showing an example of a determination target frame extracted from captured video data.

[0030] Figure 6 This is an explanatory diagram showing an example of setting of a partition for extracting sub-region image data from an extracted determination target frame.

[0031] Figure 7 This is an explanatory diagram showing an example of superimposing and displaying the determination result in sub-region units on the extracted determination target frame.

[0032] Figure 8 This is a table diagram in which the determination results for a plurality of determination target frames are arranged in time series in units of frames. DETAILED DESCRIPTION

[0033] Hereinafter, examples of embodiments of the present invention will be described with reference to the accompanying drawings. In addition, various constituent elements in the examples of the embodiments described below can be appropriately combined within the scope of no contradiction. In addition, for the content described as an example of a certain embodiment, there is a case where the description is omitted in other embodiments. In addition, for actions or processing that are not related to the characteristic parts of each embodiment, there is a case where the content is omitted. Moreover, the order of various processing constituting the various processes described below can be different within the scope of no contradiction in the processing content.

[0034] [First embodiment]

[0035] Hereinafter, an example of a tear layer evaluation device according to a first embodiment of the present invention will be described with reference to the accompanying drawings. In addition, the tear layer evaluation device according to the present invention can be implemented as a separate tear layer evaluation device that does not need to be connected to other devices via a communication network, but can also be connected to other devices, such as a server device, via a communication network so that part of the processing is performed in other devices.

[0036] Figure 1 This is a block diagram showing an example of the configuration of a tear layer evaluation device and peripheral devices corresponding to at least one embodiment of the present invention.

[0037] The tear layer evaluation device 10 may be a device designed as a dedicated device, or may be a device that can be implemented by a general computer. Figure 1As shown, the tear layer evaluation device 10 has a CPU (Central Processing Unit: Central Processing Unit) 101, a GPU (Graphics Processing Unit: Image Processing Device) 102, and a memory 103 that are generally provided in a general computer. In addition, the tear layer evaluation device 10 is connected to the imaging device 20 and the display device 30 via an interface (I / F) 104. In addition, it can also be a configuration in which input devices such as a mouse and a keyboard, output devices such as a printer, and a communication device for connecting to a communication network are connected via a bus. The processing of each part of the tear layer evaluation device 10 is realized by reading a program for executing the processing of these parts from the memory and executing it in the CPU or GPU that functions as a control circuit (Processing circuit, Processing circuitry). In other words, it is configured so that the processor (processing circuit) can execute each process of each device through the execution of the program. The imaging device 20 is used to capture the eye to be examined, and the display device 30 is used to display content such as an image showing the prediction result superimposed on the eye to be examined.

[0038] Figure 2 The imaging device 20 may be any device that can store a video of the inspected eye showing the imaging object as digital data, and a conventionally known device may be used as appropriate, for example, Figure 2As schematically shown, the photographing device 20 causes the light emitted from the light source 21 composed of a white LED or the like and passing through the aperture to be focused on the anterior ocular segment 25 of the inspected eye of the inspected person through the lens 22, the beam splitter 23, and the objective lens 24 in sequence. The reflected light from the anterior ocular segment 25 passes through the objective lens 24 and the beam splitter 23, and is imaged on the imaging element 27 via the imaging lens 26. The photographing data imaged on the imaging element 27 is converted into still image data and video data by the image processing engine (Image Processing Engine) through prescribed processing. The photographing video data photographed by the photographing device 20 is sent to the tear layer evaluation device 10. In addition, in the following, when the ophthalmic image or sub-area image data is expressed as only "image", it is used as the following expression: in addition to the case of meaning the content represented by "image", the case of also including the data of the image. In addition, the following description will be given as a configuration in which the captured video data is acquired in the camera 20, but the following configuration may also be used: although it is a video, the time interval between frames is long and the continuity of the action is interrupted, that is, a plurality of still image data arranged in time sequence is acquired. However, if the time interval between frames is long, the possibility of not being able to detect the moment becomes high accordingly, so it is preferably controlled to be less than tens of milliseconds as much as possible.

[0039] Figure 3 1 is a block diagram showing an example of the configuration of a tear layer evaluation device according to at least one embodiment of the present invention. Figure 3 As shown, the tear layer evaluation device 10 includes a captured video data acquisition unit 11 , a determination target frame extraction unit 12 , a sub-region unit determination unit 13 , a frame unit determination unit 14 , a video unit determination unit 15 , a display processing unit 16 , and a storage unit 17 .

[0040] The captured video data acquisition unit 11 has a function of acquiring captured video data obtained by capturing the eye to be examined. Any captured video data may be used as long as it is obtained by capturing the eye to be examined, but it is preferred that the captured video data include at least one eyelid opening interval from the time when the eyelid is closed to the time when the eyelid is closed next time. Alternatively, the captured video may be divided into each eyelid opening interval and captured as separate captured video data.

[0041] The judgment object frame extraction unit 12 has a function of extracting a plurality of judgment object frames from the captured video data according to a prescribed extraction rule. Here, the so-called prescribed extraction rule refers to a rule for extracting a frame as a judgment object from a plurality of frame data constituting the captured video data. Generally speaking, video shooting is set to perform shooting at a few frames in 1 second, such as 30fps and 60fps. Although all frames can also be used as judgment object frames, since the processing load of the computer increases, the state change of the inspected eye per frame is small, and it may not be possible to obtain an effect that matches the increase in processing load. Therefore, it is possible to consider adopting an extraction rule such as extracting 1 frame at a certain time interval or extracting 1 frame at a certain number of frames. For example, if 1 judgment object frame is extracted every 0.5 seconds in the captured video data shot at 30fps, 2 judgment object frames are obtained during 1 second.

[0042] The sub-area unit determination unit 13 has the following functions: extracting a plurality of sub-area image data of a predetermined size from the determination target frame, and determining whether each sub-area image data is an image showing a predetermined tear state. Here, the so-called sub-area image data refers to image data extracted by cutting a part of the determination target frame with a predetermined size smaller than the size of the determination target frame (a size determined by the number of vertical and horizontal dots). The determination target frame is obtained by photographing the entire eye to be examined, and the sub-area image data is an image showing a small part of the eye to be examined. The predetermined size for extracting the sub-area image data can be set to various sizes, but if the predetermined size is too large, a lot of information other than the part showing the characteristics of the predetermined tear state will be included, making it difficult to determine the part showing the characteristics that serve as the basis for determination, and if the predetermined size is too small, the characteristics of the predetermined tear state may not be recognized, resulting in false detection, so it is preferably determined to be a size that is easy to determine whether it is a predetermined tear state (high determination accuracy). It can be said that there is a possibility that different sizes are set depending on the type of the predetermined tear state. In addition, the so-called predetermined tear state refers to a specific state that can be used for evaluation of the eye to be examined. Various tear states can be selected as objects, for example, the occurrence of interference fringes in the tear layer, the presence or absence of a break in the tear layer, etc. In addition, as the types of break, it is preferred to distinguish and judge the tear state by area break, line break, dimple break, spot break, etc.

[0043] In the sub-area unit determination unit 13, for each of the extracted multiple sub-area image data, it is determined whether it corresponds to an image showing a predetermined tear state. The determination method of whether it corresponds to an image showing a predetermined tear state may be any method. As an example, it is conceivable to make a determination based on a learned model that has been learned in advance. The learning process of the learned model is performed in the following manner, that is, the sub-area image data is extracted from the learning image data (frame data) obtained by photographing the inspected eye, and the tear state of the inspected eye is predicted for each sub-area image data by machine learning using the correct data related to the state of each sub-area image data. The model used may be, for example, a convolutional neural network (CNN). It is necessary to make the size of the sub-area image data used for learning and the sub-area image data extracted by the sub-area unit determination unit 13 the same size. The sub-area unit determination unit 13 may be dedicated to determining one tear state, or it may be configured to distinguish and determine multiple types of tear states through the same determination process, so that the learning of the model is performed according to the content of the determination process. In addition, it may be configured to determine multiple types of tear states by using multiple learned models that have been learned differently. In such learning to determine tear status in sub-area units, determination accuracy can be improved with a smaller amount of image data than the amount of image data required for learning to determine tear status for an entire ophthalmic image.

[0044] In addition, the learned model used for the determination of the sub-area unit determination unit 13 may be stored in the storage unit 17 of the tear layer evaluation device 10, or may be distributed in a server device, a plurality of terminal devices, etc., which are other devices, and connected to the device where the learned model is set via a communication network for use. That is, as long as the learned model stored in some storage unit is used, the storage unit of the learned model may be set in the tear layer evaluation device itself or in another device.

[0045] In addition, when extracting the sub-region image data in the sub-region unit determination unit 13, it is preferable to extract the sub-region image data without interruption for the determination target frame so as not to erroneously determine the portion showing the characteristic of the tear state. Moreover, for each sub-region image data, the extraction of the sub-region image data may be performed in such a manner that at least a part of the image area overlaps with other sub-region image data. By extracting the sub-region image data with a part being overlapped in this way, the possibility of erroneously determining the portion showing the characteristic of the predetermined tear state existing at the boundary portion of two sub-region image data can be reduced.

[0046] The frame unit determination unit 14 has the following function: determining whether the determination target frame is a candidate image showing a specified tear state according to a specified determination condition. As the specified determination condition, for example, when the number of sub-area image data determined to be an image showing the specified tear state exceeds a specified threshold, the determination target frame is determined to be a candidate image showing the specified tear state. In addition, for the threshold value related to the number of images determined to show the specified tear state in units of sub-areas, a different threshold value may be set for each type of tear state of the determination target. For example, it may be considered to set the threshold value in the following manner, that is, when 5 linear breaks are determined in units of sub-areas, the determination target frame is determined to be a candidate image of linear breaks, and when 1 (or 2) spot breaks are determined in units of sub-areas, the determination target frame is determined to be a candidate image of spot breaks. In addition, the threshold value to be set also varies depending on the extraction size of the sub-area image data, so it is preferable to set it appropriately as the optimal threshold value. The frame unit determination unit 14 compares the frame to be determined with a threshold value set for each type of tear state to be determined, and when the threshold value is exceeded, the determination target frame is determined to be a candidate image showing the tear state corresponding to the threshold value. When determination is performed simultaneously for a plurality of types of tear states, there is a possibility that one determination target frame is determined to be a candidate image showing a plurality of tear states.

[0047] The video unit determination unit 15 has the following function: when the determination object frames are arranged in time sequence, when the determination object frame determined as a candidate image showing a specified tear state satisfies a specified continuity condition, it is determined that the inspected eye shown (presented) in the captured video data is in a specified tear state. Here, the so-called specified continuity condition refers to a condition for determining a time change, and this time change is a time change for determining the tear state of the determination object. As an example, in the video unit determination unit 15, when the determination object frames are arranged in time sequence, it can be considered that the determination object frame determined as a candidate image showing a specified tear state appears continuously for more than a predetermined number of times as a continuity condition. The threshold value regarding the degree of continuity of the determination object frame determined as a candidate image needs to be set to be different for each type of tear state, and also needs to be set to be different according to the time interval of extraction by the determination object frame extraction unit 12. As a specific example, when the type of tear state to be determined is assumed to be linear rupture, and a benchmark is set to determine continuous rupture for 5 seconds as rupture in the captured video data, when the prescribed rule is set to extract 2 frames per second, if more than 10 consecutive frames of the determination object are determined as candidate images of linear rupture, the captured video data is determined to have rupture.

[0048] In addition, the video unit determination unit 15 may be configured to determine, when extracting a preset number of frames for an arbitrary part after arranging the determination target frames in time sequence, that the determination target frames determined as candidate images showing a predetermined tear state appear at a predetermined ratio or more within the extracted range as a continuity condition. For example, when the predetermined number of consecutive frames or more is used as the continuity condition, if there are frames that cannot be determined for some reason even though they are the tear state of the determination target, it may be impossible to correctly determine that the subject's eye shown in the captured video data is in the tear state as a whole. Specifically, when the continuity condition of 10 or more is set, if the determination cannot be made due to one erroneous determination after 7 consecutive frames and 7 consecutive frames are made thereafter, although 15 consecutive frames are actually frames showing the predetermined tear state, the continuity condition of 10 or more is not satisfied due to one erroneous determination, and the determination result of the final captured video data is changed. Therefore, by determining as the continuity condition that the predetermined ratio or more appears within a certain range although not completely continuous, the influence of erroneous determinations such as noise can be excluded. As a specific example, when focusing on any 10 frames, it can be considered that 9 or more frames are determined to be candidate images showing a predetermined tear state (more than 90%) as a continuity condition. This continuity condition is effective in the case of a type of tear state where it is difficult to determine the type of tear state where misalignment occurs at a certain ratio.

[0049] In addition, the video unit determination unit 15 may also determine as a continuity condition that the determination object frame determined as the candidate image appears continuously from the determination object frame determined as the candidate image showing the specified tear state until the eyelid is closed, when the determination object frame is arranged in time sequence. This is because depending on the type of tear state, there is a characteristic that once it occurs, it continues until the eyelid is closed, so it becomes a continuity condition corresponding to this. There are many cases where the rupture that is a sign of dry eye has this characteristic. In addition, the continuity condition that continues until the eyelid is closed can also be used together with the continuity condition that the number of continuous determinations is more than the specified number.

[0050] The display processing unit 16 has a function of generating a display screen to be displayed on a display device. The display device is a display included in the tear layer evaluation device or a display connected to the tear layer evaluation device. It is conceivable to generate a display screen for showing the determination result (evaluation result).

[0051] In addition, the display processing unit 16 may generate a display screen including an image obtained by superimposing a display indicating that the image is a predetermined tear state on the determination target frame so that the corresponding area of ​​the determination target frame, which is the extraction source of the sub-area image data determined as the image showing the predetermined tear state by the sub-area unit determination unit 13, can be visually recognized. Through this processing, the sub-area part determined to be a predetermined tear state such as a rupture can be recognized at a glance. By displaying the image indicating that the image is a predetermined tear state superimposed on the determination target frame in a time series, a plurality of frames are displayed continuously, so that the change of the predetermined tear state over time can be visually recognized. In addition, when multiple types of tear states are determined at the same time, different displays are used for different types of tear states, so that the types of tear states can be distinguished and visually recognized even if they are superimposed and displayed at the same time on the same determination target frame. In addition, as different displays, for example, it is conceivable to make the types of hatching, color depth, color tone, etc. different from each other.

[0052] Furthermore, the display processing unit 16 may display, in a graph, a time-series change in the number of sub-region image data determined as images showing a predetermined tear state when the determination target frames are arranged in time series.

[0053] In addition, the display processing unit 16 may divide the determination target frame into a plurality of statistical regions, and when the determination target frame is arranged in time series, count the number of sub-area image data determined as images showing a predetermined tear state for each statistical region, and display the time-series change of each statistical region in a graph. For example, the inspected eye is classified into five statistical regions, namely, the center, the upper side, the lower side, the right side, and the left side, and on this basis, it is obtained as data in which statistical region the feature of the predetermined tear state increases, and this is displayed in a graph as a time-series change, so that the tendency of the change of each part can be read.

[0054] The storage unit 17 has the function of storing information required for processing by each unit in the tear layer evaluation device 10, and storing various information generated in the processing of each unit. In addition, when the learned model is used in the sub-area unit determination unit 13, the learned model may be stored in the storage unit 17. In addition, it is also possible to configure so that the learned model is stored in a server device that can be connected via a communication network, and the server device has the function of the sub-area unit determination unit 13.

[0055] Next, a description will be given of a flow of a determination process corresponding to at least one embodiment of the present invention. Figure 4 This is a flowchart of an example of a determination process corresponding to at least one embodiment of the present invention. Figure 4In the flowchart shown in FIG. 1 , an example of determining the occurrence of breakup as a tear state is described. Figure 4 As shown, the determination process starts by acquiring the captured video data obtained by capturing the eye to be examined in the captured video data acquisition unit 11 of the tear layer evaluation device 10 (step S101). Next, the tear layer evaluation device 10 extracts a plurality of determination target frames from the acquired captured video data by the determination target frame extraction unit 12 (step S102). Next, the tear layer evaluation device 10 selects the initial determination target frame by the determination target frame extraction unit 12 (step S103). Next, the tear layer evaluation device 10 extracts a plurality of sub-area image data from the image data of the determination target frame by the sub-area unit determination unit 13, and determines whether each sub-area image data is an image showing a feature of rupture (step S104). Next, the tear layer evaluation device 10 determines whether the number of sub-area image data determined to be ruptured is greater than a predetermined threshold value by the frame unit determination unit 14 (step S105). When the number of sub-area image data determined to be ruptured is less than the predetermined threshold value (S105-No), the process moves to step S107. When the number of sub-region image data determined to be broken is equal to or greater than a predetermined threshold value (S105-Yes), the tear layer evaluation device 10 determines the determination target frame as a candidate image of the break through the frame unit determination unit 14 (step S106). Next, the tear layer evaluation device 10 determines whether the determination target frame determined to be broken continues for a predetermined number of times through the video unit determination unit 15 (step S107). When the determination target frame determined to be broken does not continue for a predetermined number of times (S107-No), it is determined whether the determination target frame is the last determination target frame (step S108). When it is not the last determination target frame (S108-No), the next determination target frame is selected (step S109), and the process moves to step S104. The process of steps S104 to S109 is repeatedly performed in this way until the determination target frame determined to be broken continues for a predetermined number of times (S107-Yes) or becomes the last determination target frame (S108-Yes).

[0056] In the process of repeatedly executing the processing, when the determination target frames determined to be broken continue for a predetermined number of times (S107-Yes), the tear layer evaluation device 10 determines that a break occurs in the captured video data through the video unit determination unit 15 (step S111), and ends the determination processing. In the process of repeatedly executing the processing, when the determination target frames determined to be broken do not continue for a predetermined number of times (S107-No), and the processing ends until the last determination target frame (S108-Yes), it is determined that a break does not occur in the captured video data (step S110), and ends the determination processing.

[0057] Next, the determination of the tear state in sub-region units will be described using images. Figure 5 It is an explanatory diagram showing an example of a determination target frame extracted from captured video data. Figure 6 FIG. 1 is an explanatory diagram showing an example of setting of a partition for extracting sub-region image data from an extracted determination target frame. Figure 7 In the example shown, 49 vertical and horizontal partitions (7 x 7) are set for the eye to be examined, and 4 vertical and horizontal partitions (2 x 2) are extracted as one sub-area image data. At this time, the sub-area image data is extracted in a form that overlaps two partitions with other adjacent sub-area image data. Then, for each sub-area image data, it is determined whether it is an image showing the state of tears. Figure 7 FIG. 4 is an explanatory diagram showing an example of superimposing and displaying the determination result in sub-area units on the extracted determination target frame. Figure 7 In the example shown in FIG, as a result of judging the three types of regional rupture, spot rupture, and linear rupture, six sub-area image data are judged to be linear ruptures, and the intervals between the oblique lines of the overlapping parts showing the judgment results are narrow in the repeated parts in the adjacent sub-area image data. Figure 7 Although the portions where the determination results are repeated are visually indicated by intervals between oblique lines, the portions where the determination results are repeated may be indicated by the depth of color.

[0058] Next, the determination of the tear state in units of video will be described. Figure 8 This is a table that arranges the judgment results of multiple judgment target frames in time sequence. The first column indicates the number of the judgment target frame, the second column indicates the frame number, the third column indicates the time (seconds) in the captured video data, the fourth column indicates the judgment result of rupture (tear state), the fifth column indicates the content of the judged type, and the sixth column indicates the detailed content of the judgment result in sub-area units. Figure 8 In the example of , when there are five or more images determined to be linear breakage in units of sub-areas, the determination target frame is determined to be a candidate image for linear breakage. Figure 8In the example, until the 4th determination target frame, it is determined that there is no crack, but in the 5th determination target frame, there are 5 images determined as linear cracks in the sub-area unit, so the 5th determination target frame is determined as a candidate image for cracks. Thereafter, there is a tendency that the number of images determined as linear cracks in the sub-area unit increases gradually from the 6th to the 15th. Since there are more than 5 cracks in the sub-area unit, the candidate images are continuously determined as cracks in the frame unit. In this case, for example, the following continuous condition is adopted, that is, when the determination target frame is determined as a candidate image for cracks for 10 consecutive frames, it is determined that the occurrence of cracks in the captured video data exists. In this case, since the determination target frame No. 14 satisfies the condition of 10 consecutive frames, it is determined that the occurrence of cracks in the captured video data exists based on the candidate image determined as cracks in the determination target frame No. 14.

[0059] As described above, as one aspect of the first embodiment, the apparatus comprises: a captured video data acquisition unit that acquires captured video data obtained by capturing an eye to be examined; a determination target frame extraction unit that extracts a plurality of determination target frames from the captured video data according to a prescribed extraction rule; a sub-area unit determination unit that extracts a plurality of sub-area image data of prescribed sizes from the determination target frame and determines whether each sub-area image data is an image showing a prescribed tear state; a frame unit determination unit that determines that the determination target frame is a candidate image showing the prescribed tear state when the number of sub-area image data determined to be an image showing the prescribed tear state exceeds a prescribed threshold value; and a video unit determination unit that determines that the eye to be examined that appears (presents) in the captured video data is in the prescribed tear state when the determination target frames are arranged in time sequence and the determination target frames determined to be candidate images showing the prescribed tear state satisfy a prescribed continuity condition. Therefore, while the captured image is subdivided to improve the determination accuracy, the tear layer can be evaluated by taking into account the change of the tear layer over time.

[0060] That is, even if it is determined to be a prescribed tear state in units of frames, the determination is an erroneous determination or the determination cannot be made correctly based on the determination result of only one frame, so there is the following problem: it is not necessarily possible to directly determine that the eye under examination shown (present) in the captured video data is in the prescribed tear state, but according to this example, since it is configured to determine that the eye under examination shown in the captured video data is in the prescribed tear state when the continuous condition for determining the prescribed tear state is satisfied, the determination accuracy can be improved. In addition, the determination in units of frames for each determination target frame is also determined using the determination result in units of sub-areas, so that the accuracy of the determination result in units of frames can be improved.

[0061] [Second embodiment]

[0062] In the first embodiment, as a premise for the determination by the frame unit determination unit 14 , a configuration is adopted in which the sub-unit determination unit 13 performs determination in sub-units, but the present invention is not necessarily limited to this.

[0063] That is, instead of making a determination in units of sub-areas, it is also possible to directly determine whether the entire determination target frame is a candidate image showing a predetermined tear state. It is conceivable to use a learned model in the determination at this time, but as long as the learning accuracy in units of frames is improved along with the improvement of the technology in the technical field of machine learning and deep learning, there is a sufficient possibility that the same determination accuracy for the captured video data as in the first embodiment can be achieved even if the entire determination target frame is directly determined whether it is a candidate image showing a predetermined tear state.

[0064] Description of Reference Numerals

[0065] 10 Tear layer evaluation device

[0066] 11. Video data acquisition unit

[0067] 12. Determination target frame extraction unit

[0068] 13. Sub-area unit determination unit

[0069] 14 Frame unit judgment unit

[0070] 15 Video unit judgment department

[0071] 16 Display processing unit

[0072] 17 Storage

[0073] 20. Filming Device

[0074] 21 Light Source

[0075] 22 Lens

[0076] 23 Beam splitter

[0077] 24 Objective lens

[0078] 25 Anterior eye

[0079] 26 Imaging lens

[0080] 27 Camera Components

[0081] 30 Display device

[0082] 101 CPU

[0083] 102 GPU

[0084] 103 Memory

[0085] 104 I / F

Claims

1. A tear layer evaluation device for evaluating the tear state of an eye to be examined based on video data obtained by photographing the eye to be examined, wherein: have: a captured video data acquisition unit, which acquires captured video data obtained by capturing the eye to be examined; a determination target frame extraction unit that extracts a plurality of determination target frames from the captured video data according to a predetermined extraction rule; a frame unit determination unit that determines whether the determination target frame is a candidate image showing a predetermined tear state based on a predetermined determination condition; A video unit determination unit determines that the eye under examination presented in the captured video data is in the specified tear state when the determination object frames are arranged in time sequence and when the determination object frame determined as a candidate image showing a specified tear state satisfies a specified continuity condition.

2. The tear layer evaluation device according to claim 1, wherein: A sub-area unit determination unit is provided, the sub-area unit determination unit extracting a plurality of sub-area image data of a predetermined size from the determination target frame, and determining whether each sub-area image data is an image showing a predetermined tear state, The frame unit determination unit determines that the determination target frame is a candidate image showing a predetermined tear state when the number of sub-area image data determined to be images showing a predetermined tear state exceeds a predetermined threshold value.

3. The tear layer evaluation device according to claim 2, wherein: The frame unit determination unit sets and stores a threshold value regarding the number of sub-area image data for determining that the determination target frame is an image showing a predetermined tear state, for each type of tear state to be determined.

4. The tear layer evaluation device according to claim 2, wherein: The sub-area unit determination section extracts the sub-area image data for each of the plurality of sub-area image data so that at least a part of the image area overlaps with other sub-area image data.

5. The tear layer evaluation device according to claim 2, wherein: A display processing unit is provided, the display processing unit generates a display screen to be displayed on a display device, The display processing unit causes a display representing an image showing a specified tear state to be overlapped and displayed on the determination object frame so that a corresponding area of ​​the determination object frame can be visually identified, and the corresponding area of ​​the determination object frame is a corresponding area of ​​the determination object frame that is an extraction source of sub-area image data determined in the sub-area unit determination unit as an image showing a specified tear state.

6. The tear layer evaluation device according to claim 2, wherein: A display processing unit is provided, the display processing unit generates a display screen to be displayed on a display device, The display processing unit displays, using a graph, a time-series change in the number of sub-area image data determined as images showing a predetermined tear state when the determination target frames are arranged in time series.

7. The tear layer evaluation device according to claim 6, wherein: The display processing unit divides the determination object frame into a plurality of statistical areas and arranges the determination object frame in time series, counts the number of the sub-area image data that are determined to be images showing a specified tear state for each statistical area, and uses a graph to display the time series changes of each statistical area.

8. The tear layer evaluation device according to any one of claims 1 to 7, wherein: The video unit determination unit determines, as the continuity condition, that the determination target frames determined as candidate images showing a predetermined tear state appear continuously for a predetermined number or more when the determination target frames are arranged in time series.

9. The tear layer evaluation device according to any one of claims 1 to 7, wherein: The video unit determination unit extracts a preset number of frames for an arbitrary part after arranging the determination object frames in time sequence, and determines as the continuity condition that the determination object frames determined as candidate images showing a specified tear state appear at a specified ratio or more within the extracted range.

10. The tear layer evaluation device according to any one of claims 1 to 7, wherein: The video unit determination unit arranges the determination object frames in time sequence and determines the continuous appearance of the determination object frame determined as the candidate image during the period from the point where the determination object frame determined as the candidate image showing a specified tear state appears until the eyelid is closed as the continuity condition.

11. A tear layer evaluation program, which enables a computer to evaluate the tear state of an eye under examination based on video data obtained by photographing the eye under examination, wherein: The computer is caused to implement: A function of acquiring video data, which acquires video data obtained by photographing the eye to be examined; A determination object frame extraction function, extracting a plurality of determination object frames from the captured video data according to a prescribed extraction rule; a frame unit determination function for determining whether the determination target frame is a candidate image showing a predetermined tear state according to a predetermined determination condition; The video unit judgment function, when the judgment object frames are arranged in time sequence, and when the judgment object frame judged as a candidate image showing a specified tear state satisfies a specified continuity condition, judges that the inspected eye presented in the captured video data is in a specified tear state.

12. A tear layer evaluation method, which uses a computer to evaluate the tear state of an eye under examination based on video data obtained by photographing the eye under examination, wherein: have: a step of acquiring video data, acquiring video data obtained by photographing the eye to be examined; A determination object frame extraction step, extracting a plurality of determination object frames from the captured video data according to a prescribed extraction rule; a frame unit determination step of determining whether the determination target frame is a candidate image showing a predetermined tear state according to a predetermined determination condition; The video unit judgment step comprises: when the judgment object frames are arranged in time sequence, when the judgment object frame judged to be a candidate image showing a specified tear state satisfies a specified continuity condition, judging that the inspected eye presented in the captured video data is in the specified tear state.

Images