Vertigo diagnostic device and method using ultra-fast lightweight deep running model for tracking eyeball and head position change in video image, recording medium storing program for achieving the same and computer program stored in recording medium

Abstract

To provide a vertigo diagnostic device which automatically removes noise, such as eye blinking and the like, and tracks and quantifies purely eyeball movement so as to allow doctors to use the values in diagnosis.SOLUTION: The present invention relates to: a technique for tracking changes in eyeballs and a head position to diagnose vertigo, and more specifically, a vertigo diagnostic device and method using a deep learning model for tracking movement of the eyeball and the head of a patient from a video including a patient to diagnose vertigo on the basis of a deep learning model, and also relates to; a recording medium storing a program for achieving the vertigo diagnosis; and the computer program stored in the recording medium.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for diagnosing dizziness by tracking changes in eye and head position, and more particularly to an apparatus and method for diagnosing dizziness by tracking eye and head movements based on a deep learning model using a video of a patient, a recording medium storing a program for implementing the same, and a computer program stored on the recording medium. [Background technology]

[0002] The material discussed in this section merely provides background information for the embodiments described herein and does not necessarily constitute prior art.

[0003] Vertigo is a general term for any symptom where you have the impression that you or the things around you are moving even though they are not. This type of dizziness is mostly caused by a malfunction of the peripheral vestibular nervous system. When an abnormality occurs in the peripheral vestibular nervous system, the vestibular-ocular reflex, which means the ability to fix your gaze when your head changes position, disappears, and your eyes move spontaneously without being fixed, causing dizziness. The movement of the eyes caused by abnormalities in the vestibular function is called nystagmus. When dizziness occurs, doctors evaluate the nystagmus to determine whether there is an abnormality in the vestibular function. Nystagmus can be caused not only by abnormalities in the peripheral vestibular nervous system but also by abnormalities in the central nervous system, and doctors can determine whether the dizziness a patient is complaining of is due to a peripheral vestibular system or a central nervous system problem based on the appearance of the nystagmus. In other words, observing nystagmus is an important examination method that provides the greatest diagnostic information when treating vertigo patients. Nystagmus moves in three axes: 1) horizontal (left / right), 2) vertical (up / down), and 3) linear (clockwise / counterclockwise). To date, such observation of nystagmus can be confirmed through Frenzel glasses, infrared photography, video Frenzel glasses, and video nystagmus testing.

[0004] However, Frenzel glasses and video Frenzel glasses are designed to directly observe eye movements, so diagnostic accuracy can depend on the doctor's experience in judging minute eye movements, and video nystagmus machines, which output nystagmus in a graph to complement this, require fast and highly accurate eye tracking technology, so they are expensive and difficult to widely use.

[0005] In particular, the head impulse test, which is the most appropriate way to determine the vestibulo-ocular reflex, which is the movement of the eyes caused by the change in head position, is performed by moving the patient's head quickly from side to side or up and down with the naked eye, and the test results may vary depending on the skill of the person performing the test. In order to make the results objective, the video head impulse test is performed using a device equipped with a gyro sensor that measures the speed of head movement and an eye tracker that calculates the speed of eye movement. In other words, only with such a device can the results of the head impulse test be output, and the presence or absence of abnormalities in the vestibular function can be determined by evaluating the gain, which is evaluated as the ratio of eye movement to head movement. In addition, when noise occurs due to blinking during the observation of eye movement, the current device does not have a function to remove it separately, so the examiner has to ask the patient to open their eyes for a certain period of time during the test, which is inconvenient, and doctors often understand the patient's condition only from the test result paper, so the results may be distorted by such blinking. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Publication No. 10-2018-0101816, 2018.09.14 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, an object of the present invention is to provide a dizziness diagnosis device that uses an algorithm developed by a deep learning model to automatically remove noise such as blinking from eye movement videos captured by commonly used video nystagmus testing equipment as well as eye and head movements captured on video by various devices, and tracks only eye movement, quantifies it, and displays it for use by doctors in diagnosis.

[0008] Another object of the present invention is to provide a dizziness diagnosis device that can perform an important test to distinguish between peripheral and central dizziness with any device capable of video recording, and can enable remote medical care for diagnosing dizziness, by simply taking a video of the head impulse test performed by quickly rotating the patient's head left and right or up and down using a camera such as a webcam or smartphone, instead of performing the video head impulse test using complex devices such as a gyro sensor for tracking and calculating head movement and an eye tracker for tracking and calculating eye movement, which has been used as an objective test method in the past.

[0009] It is still another object of the present invention to provide a recording medium storing a program for implementing the dizziness diagnosis method, and a computer program stored in the recording medium.

[0010] The present specification is not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0011] In accordance with an embodiment of the present invention to achieve the above object, there is provided a dizziness diagnosis device including an eyeball movement learning unit that identifies an eyeball portion from an eyeball video captured by capturing a patient's eyeball, extracts an eyeball image for each frame, recognizes a pupil center, and calculates the pupil center coordinates to learn eyeball movement; an eyeball movement output unit that receives information learned by the eyeball movement learning unit and outputs eyeball movement information from a video captured by the patient; a head movement output unit that outputs head movement information from the video captured by the patient; a calculation unit that receives eyeball movement and head movement information from the eyeball movement and head movement output unit, calculates eyeball movement velocity and head movement velocity, and calculates a gain value; and a diagnosis unit that receives the gain value calculated by the calculation unit and diagnoses dizziness.

[0012] In addition, the eye movement learning unit may include an object segmentation model generation module that divides the eye image divided by frame into areas for the sclera, iris, and pupil to generate an object segmentation model, a blink identification module that receives intermediate layer information of the object segmentation model generation to learn anatomical structures around the eye, identifies blinks, and generates a blink classification model, and an eye gaze tracking model generation module that receives intermediate layer information of the object segmentation model generation to generate a model for the gaze direction.

[0013] In addition, the eye movement learning unit can infer the part of the pupil hidden when the patient closes his / her eyes based on the blink classification model, and recognize the center of the pupil with the eye before the pupil is hidden.

[0014] Also, the eye movement learning unit may further include a graph output module that outputs the information regarding the pupil center in a three-axis (horizontal / vertical / linear) graph.

[0015] In addition, the eye movement output unit can calculate 1) the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the scleral outer periphery of the patient's eye from a video taken of the patient, and 5) receive the coordinates of the pupil center calculated by the eye movement learning unit, convert the two-dimensional coordinates in the direction of the pupil center into three-dimensional coordinates to calculate the direction vector of the eye, and output information about the eye movement.

[0016] In addition, the head movement output unit can calculate the coordinates of 1) the patient's nose, 2) the left end of the outer scleral part of the eyeball, 3) the right end, and 4) the center of the forehead from a video of the patient, convert the two-dimensional coordinates relative to the head direction into three-dimensional coordinates to calculate the head direction vector, and output information about the head movement.

[0017] In addition, the calculation unit calculates the gain value as follows: if the time resolution of the video of the patient is "FPS", the point in time when the head moves significantly is "head peak index", and the point in time when the pupil moves in the opposite direction within 1 second after the head moves significantly is "eye peak index",

number

[0018] The presence or absence of an abnormality in the vestibular nerve function can be determined by using the eye movement outputted from the eye movement output unit and the gain value.

[0019] In another embodiment of the present invention to achieve the above object, there is provided a method for diagnosing dizziness, comprising: (a) identifying an eyeball portion from an eyeball video captured by capturing a patient's eyeball, extracting an eyeball image for each frame, recognizing a pupil center, and calculating the coordinates of the pupil center to learn the movement of the pupil center; (b) outputting information on eyeball movement from the video captured by the patient using information on eyeball movement learned in the process (a); (c) outputting information on head movement from the video captured by the patient; (d) calculating an eyeball movement velocity and a head movement velocity using the information on eyeball movement and head movement output in the processes (b) and (c), and calculating a gain value; and (e) diagnosing dizziness using the gain value calculated in the process (d).

[0020] In addition, the process (a) may include the steps of: (a-1) dividing the eye image divided by frame into regions for the sclera, iris, and pupil to generate an object division model; (a-2) learning anatomical structures around the eye using intermediate layer information from the process (a-1), identifying blinks, and generating a blink classification model; and (a-3) generating a model for the gaze direction using the intermediate layer information from the process (a-1).

[0021] In addition, the step (a) may be a step of recognizing the center of the pupil in the eye before the pupil is hidden by inferring the part of the pupil hidden when the patient closes his / her eyes based on the blink classification model generated in the step (a-2).

[0022] Also, the step (a) may further include the step of outputting the information regarding the pupil center in a three-axis (horizontal / vertical / linear) graph.

[0023] In addition, the step (b) may be a step of: 1) calculating the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the scleral outer portion of the patient's eye from a video of the patient; and 5) calculating a directional vector of the eye by converting two-dimensional coordinates in the direction of the pupil center calculated in the step (a) into three-dimensional coordinates using the coordinate of the pupil center calculated in the step (a), and outputting information on eye movement.

[0024] In addition, the step (c) may be a step of calculating the coordinates of 1) the patient's nose, 2) the left end of the outer sclera of the eyeball, 3) the right end, and 4) the center of the forehead from a video of the patient, converting the two-dimensional coordinates relative to the head direction into three-dimensional coordinates to calculate a head direction vector, and outputting information about the head movement.

[0025] In addition, in the step (d), the time resolution of the video of the patient is defined as "FPS", the point at which the head moves significantly is defined as "head peak index", and the point at which the pupil moves in the opposite direction within 1 second after the head moves significantly is defined as "eye peak index". The gain value is

number

[0026] In addition, the step (e) may be a step of determining an abnormality in vestibular nerve function using the eye movement outputted in the step (b) and the gain value calculated in the step (d).

[0027] In order to achieve the above object, a computer-readable recording medium storing a program for implementing the above dizziness diagnosis method according to still another embodiment of the present invention is provided.

[0028] In order to achieve the above object, a computer program stored on a computer-readable recording medium for implementing the above dizziness diagnosis method according to still another embodiment of the present invention is provided. Effect of the Invention

[0029] As described above, according to the dizziness diagnosis device and method according to the embodiment of the present invention, the algorithm can be applied to existing testing equipment to automatically remove noise such as blinking and output results related purely to eye movement. It can also calculate eye movement and head movement through a deep learning model from a patient's eye video and a video of the patient, making it possible to easily test for dizziness using only video recording without any complicated equipment.

[0030] In addition, according to the dizziness diagnosis device and method according to the embodiment of the present invention, a head impulse test can be performed based on a video of the patient's eyeballs and a video of the patient, making it possible to perform remote medical care in the diagnosis of dizziness, thereby contributing to health.

[0031] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a block diagram illustrating a dizziness diagnosis device according to an embodiment of the present specification. [Diagram 2] 1 is a diagram illustrating a simplified learning process for eye movement according to the present specification; [Diagram 3]This is a diagram showing the process of generating an object segmentation model from an eyeball video, recognizing the pupil center, and outputting the movement of the pupil center in a graph. [Figure 4] 1 is a diagram showing artificial intelligence model generation for pupil center recognition. [Diagram 5] 1 is a diagram showing a method for measuring the speed of eye movement due to head movement. [Figure 6] 1 is a flowchart illustrating a method for diagnosing dizziness according to an embodiment of the present specification. [Figure 7] 13 is a flowchart showing a process of recognizing the pupil center and calculating the pupil center coordinates of an eyeball movement learning unit according to an embodiment of the present specification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The advantages and features of the present invention and the methods for achieving them will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be realized in various different forms, which are provided to fully inform those skilled in the art of the present invention of the scope of the invention, and the present invention is defined by the scope of the claims.

[0034] In addition, the terms used in this specification are for the purpose of describing the embodiments, and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless otherwise specified in the phrase. For example, "include" (or "comprise") and / or "including" (or "comprises") used in the specification can include the mentioned components and steps. The same reference numerals refer to the same components throughout the specification. "And / or" includes each and every combination of one or more of the mentioned items.

[0035] Additionally, unless otherwise defined, all terms (including technical and scientific terms) used herein are used as commonly understood by those of ordinary skill in the art, and terms defined in commonly used dictionaries are not to be overly or overly interpreted unless expressly and specifically defined.

[0036] FIG. 1 is a block diagram showing a dizziness diagnosis device according to an embodiment of the present specification.

[0037] Referring to FIG. 1, a dizziness diagnosis device 10 according to an embodiment of the present specification can learn the movement of a patient's eyeballs using an eyeball video 1. Based on learning information on pupil-centered movement, eyeball movement caused by head movement can be measured to diagnose dizziness. Through this, dizziness can be diagnosed using the eyeball video 1 and the patient-photographed video 2 without using any device. The eyeball video 1 and the patient-photographed video 2 may be videos captured by a video capture device capable of capturing video. Video data is available in various formats such as MP4, MOV, WMV, AVI, and MKV, and the video data according to the present specification is not limited to a specific video file format.

[0038] The dizziness diagnosis device 10 may include an eye movement learning unit 11 , an eye movement output unit 12 , a head movement output unit 13 , a calculation unit 14 , and a diagnosis unit 15 .

[0039] FIG. 2 is a diagram illustrating a simplified learning process for eyeball movement according to the present specification.

[0040] 1 and 2, the eye movement learning unit 11 can identify the eyeball portion from the eyeball video 1 and extract an eyeball image 21 for each video frame. The eye movement learning unit 11 can recognize the pupil center in the eyeball image 21 and calculate the coordinate value of the pupil center to learn the eyeball movement. Image data has various formats such as JPG, PNG, GIF, SVG, etc., and the image data according to the present specification is not limited to a specific image file format.

[0041] The eye movement learning unit 11 may include an object segmentation model generation module 111 , a blink identification module 112 and an eye gaze tracking model generation module 113 .

[0042] The object segmentation model generation module 111 can generate an object segmentation model 22 in which the eyeball is segmented into a sclera, an iris, and a pupil in the eyeball image 21. The eyeball movement learning unit 11 can use information of the object segmentation model 22 to recognize the center of the pupil.

[0043] The object segmentation model generation module 111 can provide information on the areas of the sclera, iris, and pupil that could not be provided by the conventional nystagmus test method using an infrared camera.

[0044] The blink identification module 112 receives information from the intermediate layer of the object segmentation model generation and learns anatomical structures around the eyeball to identify blinks and generate a blink classification model that classifies blink steps.

[0045] The eye movement learning unit 11 can infer the part where the pupil is hidden when the patient closes his / her eyes based on the blink classification model, and can recognize the center of the pupil with the eyeball before the pupil is hidden. Also, it can remove noise caused by blinking, and prevent the result from being distorted by the blinking.

[0046] The gaze tracking model generation module 113 can receive information of the intermediate layer of the object segmentation model generation and output a gaze tracking model, which is a model for the gaze direction. The eye movement learning unit 11 can use the information of the gaze tracking model to recognize the pupil center.

[0047] The eye movement learning unit 11 may further include a graph output module 114. The graph output module 114 may output the coordinate values ​​of the pupil center calculated based on information of the object segmentation model 22, the blink classification model, and the gaze tracking model 23 as a three-axis movement graph 24. The three-axis graph 24 may be output as a horizontal movement graph 241 showing left / right movement of the pupil center, a vertical movement graph 242 showing up / down movement, and a line movement graph (not shown) showing clockwise / counterclockwise movement.

[0048] FIG. 3 shows the process of generating an object segmentation model from an eyeball video, recognizing the pupil center, and outputting the movement of the pupil center in a graph.

[0049] 1 and 3, the object segmentation model generation module 111 can generate an object segmentation model 22 from the patient's eyeball image extracted from the eyeball movement learning unit 11. The object segmentation model can segment the patient's eyeball image into a sclera 30, an iris 31, and a pupil 32. In addition, the object segmentation model generation module 111 can output an object integrated image 33 that integrates the object-segmented images of the sclera 30, the iris 31, and the pupil 32. The eyeball movement learning unit 11 can track the movement of the center of the pupil in real time using the object integrated image 33.

[0050] The eye movement learning unit 11 can output the pupil center movement graph 34 through the graph output module 114 based on the pupil center information recognized by using the object integrated image 33 .

[0051] The pupil center graph 34 may be outputted through pupil center information provided by a deep learning model. Also, the pupil center graph 34 may be more precisely calculated through the center of gravity of the object-segmented pupil 32. The pupil center graph 34 may be outputted as a vertical motion graph 341, a horizontal motion graph 342, and a line motion graph (not shown).

[0052] FIG. 4 is a diagram showing the generation of an artificial intelligence model for pupil center recognition.

[0053] Referring to FIG. 4, in order for artificial intelligence to learn pupil-centered information well, an object segmentation model 22, a blink classification model 40, and an eye-tracking model 41 can be generated through a deep learning model.

[0054] The object segmentation model generation module 111 may generate the object segmentation model 22 from the eyeball image 21 using a convolutional neural network (CNN) deep learning model. The CNN deep learning model is a technology widely known to those skilled in the art, and therefore a detailed description thereof will be omitted.

[0055] The blink classification model 40 and the gaze tracking model 41 can be generated using information from an intermediate layer 42 in the generation process of the object segmentation model 22. By using the information from the intermediate layer 42, it is possible to learn pupil-centered information more efficiently than a conventional deep learning model.

[0056] The blink classification model 40 can classify blinking movements as eyes open (open), eyes closing (closing), or eyes closed (closed) based on information from the intermediate layer 42.

[0057] The gaze tracking model 41 can indicate the gaze direction with a heatmap and keypoints based on the information of the intermediate layer 42.

[0058] According to this specification, the eye movement learning unit 11 can learn information about the pupil center by using at least one model selected from the object segmentation model 22, the blink classification model 40, and the gaze tracking model 41.

[0059] As an example, the eye movement learning unit 11 can learn eye movements using only information from the blink classification model 40 and the gaze tracking model 41. In this case, since information from the object segmentation model 22 is not used, it is not necessary to perform the lengthy calculations of the intermediate layer 42. As a result, it is possible to learn and output information on the pupil center faster than the conventional deep learning model.

[0060] The CNN deep learning model for learning eye movements corresponds to one example, and the learning method is not limited to this example.

[0061] FIG. 5 is a diagram showing a method for measuring the speed of eye movement due to head movement.

[0062] 1 and 5, the eye movement output unit 12 and the head movement output unit 13 can output information on the eye movement and the head movement of the patient from the moving image 2 captured by the patient.

[0063] The eye movement output unit 12 may receive learned information on eye movement from the eye movement learning unit 11. The eye movement output unit 12 may output information on eye movement from the patient-captured video 2 based on the learned information on eye movement, and output the eye movement in a graph 53.

[0064] The eye movement output unit 12 can calculate a direction vector of the eye to output information on the eye movement. In order to calculate the direction vector of the eye, two-dimensional coordinates of at least four or more points can be converted into coordinates in a three-dimensional space using Perspective-n-Point (PnP) pose computation. The PnP pose computation is a technique known to those skilled in the art, and therefore a detailed description thereof will be omitted.

[0065] In order to calculate the direction vector of the eyeball, the eyeball movement output unit 12 can perform PnP pose computation on 1) the coordinates of the upper end, 2) the lower end, 3) the left end, and 4) the right end of the scleral outer periphery, and 5) the two-dimensional coordinates of the pupil center calculated by the eyeball movement learning unit 11.

[0066] Further, the eye movement output unit 12 can output a pupil center display 50 that displays the pupil center according to the eye movement of the patient in the patient-photographed moving image 2.

[0067] The head movement output unit 13 can output information on the head movement from the patient-photographed moving image 2 and output the head movement in the form of a graph 52.

[0068] The head movement output unit 13 can calculate a head direction vector to output information on the head movement. The head movement output unit 13 can calculate a head direction vector from the patient's photographed video 2 using PnP pose computation using coordinates of 1) nose, 2) left end of the sclera outer region, 3) right end, and 4) forehead center. The head movement output unit 13 can also recognize head movement using the movement of the center of the nose in the patient's photographed video 2. Additionally, the head movement output unit 13 can output a nose center display 51 that outputs the center of the patient's nose to the patient's photographed video 2.

[0069] In this specification, in order to output the head movement, the coordinates of 1) the nose, 2) the left end of the scleral outer region, 3) the right end, and 4) the center of the forehead are used from the patient's photographed video 2. However, the head movement does not necessarily have to be output through the four coordinates. As an example, instead of the coordinates of 2) the left end of the scleral outer region and 3) the right end, the coordinates of the upper end and the lower end of the scleral outer region can be used. In addition to the four coordinates, the head movement can be output by adding coordinates of any point present on the patient's face in the patient's photographed video 2. Therefore, it is not necessarily required to use the four coordinates in order to output the head movement, and it should be understood that the head movement can be output by performing PnP pose computation using the coordinates of at least three or more arbitrary points on the patient's face in the patient's photographed video 2.

[0070] The calculation unit 14 can calculate the speed of eye movement and the speed of head movement using the information on eye movement and head movement inputted from the eye movement output unit 12 and the head movement output unit 13. Using this, the calculation unit 14 can calculate a gain value which means the ratio between the speed of head movement and the speed of eye movement.

[0071] According to the conventional technique, to calculate the gain value, it is assumed that the object the patient is looking at is at the center of the camera. In this case, the gain value can be obtained by [Equation 1].

number

[0072] However, the desired temporal resolution cannot be obtained from video, and the prediction of monocular gaze vectors is inaccurate.

[0073] According to the present specification, since eye movements caused by head movements occur simultaneously in the same direction, it is possible to predict an eye gaze vector by taking the average of the changes in both eyes. The eye gaze vector can be used to improve the accuracy of eye direction.

[0074] To calculate the gain value, the time resolution of the patient-recorded video 2 is “FPS”, the point at which the head moves significantly is “head peak index”, and the point at which the pupil moves in the opposite direction within 1 second after the head moves significantly is “eye peak index”, then the gain value can be calculated using [Equation 2].

number

[0075] According to an embodiment of the present specification, the 'eye peak index' may correspond to a value at the point when the velocity value when the eyeball moves away from the head first appears at or above the third quartile.

[0076] The calculation unit 14 can also output a graph 54 showing real-time changes in the head and eyes and a graph 55 showing gain values. At this time, the changes in the left and right eyes and the gain values ​​can be output, respectively.

[0077] The diagnosis unit 15 can diagnose dizziness in a patient by using the gain value calculated by the calculation unit 14.

[0078] The gain value can be used to clinically determine whether there is an abnormality in the vestibular nerve. For people without abnormalities in the vestibular nerve, when the head moves, the eyes move accordingly. This allows the gain value to be calculated to a value close to '1'.

[0079] However, when an abnormality occurs in the vestibular nerve, the eyes cannot move in unison with the movement of the head, causing the gain value to be smaller than "1."

[0080] For example, the gain value of the right eye calculated from the patient-recorded video 2 may be "0.7" and the gain value of the left eye may be "1.23." In this case, the gain value of the right eye is smaller than "1," which may indicate a decrease in the right vestibular nerve function.

[0081] In addition, the diagnosis unit 15 can determine abnormalities in the central vestibular nerve function and abnormalities in the peripheral vestibular nerve function by using the gain value and the directional vector of the eye movement calculated by the eye movement output unit 12.

[0082] FIG. 6 is a flow chart illustrating a method for diagnosing dizziness according to an embodiment of the present specification.

[0083] 1 and 6, eye movement can be learned from an eyeball video 1 obtained by capturing a patient's eyeball (S1). To learn eye movement, the eyeball portion can be identified from the eyeball video 1. Then, an image of the eyeball can be extracted for each frame from the eyeball video 1. The extracted eyeball image can be used to recognize the pupil center and calculate the coordinates of the pupil center to learn eye movement.

[0084] FIG. 7 is a flowchart showing a process of recognizing the pupil center and calculating the pupil center coordinates in the eyeball movement learning unit according to the embodiment of the present invention.

[0085] 1 and 7, an image of an eyeball can be extracted from the eyeball video 1 (S5). An object segmentation model can be generated using the image of the eyeball (S7). In addition, intermediate layer information for generating the object segmentation model can be input to generate a blink classification model (S6) and an eye tracking model (S8). When a patient closes his or her eyes in the eyeball video 1 using the blink classification model, the eyeball before closing the eyes can be used to recognize the pupil center. Thereafter, the pupil center can be recognized and the pupil center coordinates can be calculated using at least one of the object segmentation model, blink classification model, and eye tracking model (S9).

[0086] The eye movement learning step (S1) may further include a step of outputting the coordinates of the pupil center in a graph (S10). In this step, the graph of the pupil center coordinates may be output in a graph of three axes, horizontal, vertical and linear.

[0087] Next, the patient's eye movement and head movement can be outputted in the patient-recorded video 2 (S2). The patient's eye movement can be outputted based on the information on the pupil center learned in the eye movement learning process (S1). Also, the eye movement and head movement information can be outputted in the form of a graph.

[0088] In order to output the eye movement information, the coordinates of 1) the upper end, 2) lower end, 3) left end, and 4) right end of the scleral outer periphery of the patient's eye from the patient-recorded video 2 and the coordinates of the pupil center calculated in the eye movement learning process (S1) can be used to convert two-dimensional coordinates in the direction of the pupil center into three-dimensional coordinates to calculate the eye direction vector.

[0089] In order to output the head movement information, the coordinates of 1) the patient's nose, 2) the left end of the outer scleral area of ​​the eyeball, 3) the right end, and 4) the center of the forehead are used to convert two-dimensional coordinates for the head direction into three-dimensional coordinates to calculate the head direction vector from the patient's video 2.

[0090] Next, a gain value can be calculated and output based on the patient's eye movement and head movement (S3). To calculate the gain value, the eye movement velocity and head movement velocity can be calculated using the eye movement and head movement data.

[0091] The gain value can be calculated using the eye movement speed and head movement speed. If the time resolution of the patient-recorded video 2 is "FPS", the time when the head moves significantly is "head peak index", and the time when the pupil moves in the opposite direction within 1 second after the head moves significantly is "eye peak index", the gain value can be calculated by the above [Equation 2].

[0092] Also, the real-time changes in the head and eyes and the gain values ​​can be output in a graph, and in the process, the changes and gain values ​​of the left and right eyes can be shown, respectively.

[0093] Next, dizziness can be diagnosed based on the gain value (S4). The closer the gain value is to "1", the more normal the condition is, and the more deviated the gain value is, the more abnormal the vestibular nerve function is. Also, based on the gain value and the eye movement, it is possible to determine whether the vestibular nerve function is central or peripheral.

[0094] The dizziness diagnosis method according to the embodiment of the present invention described above can be realized in the form of a recording medium (or a computer program product) executable by a computer, such as a program module stored in a computer-readable medium and executed by a computer.

[0095] Here, the computer-readable medium may include a computer storage medium (e.g., a memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD)). The computer-readable medium may be any available medium that can be accessed by a computer, and includes, for example, both volatile and nonvolatile media, and both separate and non-separate media.

[0096] In addition, the dizziness diagnosis method according to the embodiment of the present invention includes instructions executable in whole or in part by a computer, and a computer program includes programmable machine instructions processed by a processor, and can be realized in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like.

[0097] As described above, the preferred embodiment of the present invention has been described and illustrated using specific terms, but these terms are merely for the purpose of clearly describing the present invention. It is obvious that the embodiments of the present invention and the terms described herein may be modified and changed in various ways without departing from the technical spirit and scope of the following claims. Such modified embodiments should not be understood separately from the spirit and scope of the present invention, but should be considered to fall within the scope of the claims of the present invention. [Explanation of symbols]

[0098] 10 dizziness diagnosis device, 11 eye movement learning unit, 12 eye movement output unit, 13 head movement output unit, 14 calculation unit, 15 diagnosis unit, 111 object segmentation model generation module, 112 blink identification module, 113 gaze tracking model generation module, 114 graph output module.

Claims

1. an eye movement learning unit that identifies the eyeball part from an eyeball video obtained by photographing the patient's eyeball, extracts an eyeball image for each frame, recognizes the pupil center, calculates the pupil center coordinates, and learns the eyeball movement; an eye movement output unit that receives the information learned by the eye movement learning unit and outputs information about the eye movement from a video of a patient; a head movement output unit that outputs head movement information from a video of the patient; a calculation unit that receives eye movement and head movement information from the eye movement and head movement output unit, calculates an eye movement velocity and a head movement velocity, and calculates a gain value; a diagnosis unit which receives the gain value calculated by the calculation unit and diagnoses dizziness; A dizziness diagnosis device comprising:

2. The eye movement learning unit includes: an object segmentation model generating module that generates an object segmentation model by dividing regions of the sclera, iris, and pupil in the eye image divided by frame; a blink identification module that receives information from an intermediate layer of the object segmentation model generation, learns anatomical structures around the eyeball, identifies blinks, and generates a blink classification model; an eye tracking model generation module that receives intermediate layer information of the object segmentation model generation and generates a model for the eye gaze direction; The dizziness diagnosis device according to claim 1, further comprising:

3. The eye movement learning unit includes: Based on the blink classification model, when the patient closes his / her eyes, the part of the pupil that is hidden is inferred and the pupil center is recognized by the eyeball before the pupil is hidden.

3. The dizziness diagnosis device according to claim 2.

4. The eye movement learning unit includes: A graph output module for outputting the information on the pupil center in a three-axis (horizontal / vertical / line) graph, 2. The dizziness diagnosis device according to claim 1.

5. The eye movement output unit includes: From the video of the patient, 1) calculate the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the sclera outer part of the patient's eyeball, and 5) input the coordinates of the pupil center calculated by the eyeball movement learning unit, convert the two-dimensional coordinates in the direction of the pupil center into three-dimensional coordinates to calculate the direction vector of the eyeball, and output information on the eyeball movement.

2. The dizziness diagnosis device according to claim 1.

6. The head movement output unit is From the video of the patient, the coordinates of 1) the patient's nose, 2) the left end of the outer sclera of the eyeball, 3) the right end, and 4) the center of the forehead are calculated, and the two-dimensional coordinates relative to the head direction are converted into three-dimensional coordinates to calculate the head direction vector, and information on the head movement is output.

2. The dizziness diagnosis device according to claim 1.

7. The calculation unit is If the time resolution of the video of the patient is "FPS", the time when the head is moved significantly is "head peak index", and the time when the pupil moves in the opposite direction within 1 second after the head is moved significantly is "eye peak index", the gain value is [0010] It is calculated as:

2. The dizziness diagnosis device according to claim 1.

8. The diagnosis unit includes: determining an abnormality in a vestibular nerve function by using the eye movement outputted by the eye movement output unit and the gain value; 2. The dizziness diagnosis device according to claim 1.

9. (a) identifying an eyeball portion from an eyeball video obtained by photographing a patient's eyeball, extracting an eyeball image for each frame, recognizing a pupil center, calculating a pupil center coordinate, and learning eyeball movement; (b) outputting information on eye movement from a video of a patient using the information on eye movement learned in the step (a); and (c) outputting head movement information from a video of the patient; (d) calculating an eye movement velocity and a head movement velocity using the eye movement and head movement information output in the steps (b) and (c) to calculate a gain value; (e) diagnosing dizziness using the gain value calculated in (d); A method for diagnosing dizziness, comprising:

10. The process (a) comprises: (a-1) generating an object segmentation model by segmenting regions of the sclera, iris, and pupil in the eye image segmented by frame; (a-2) learning anatomical structures around the eyeball using the intermediate layer information of the (a-1) process, and identifying blinks to generate a blink classification model; (a-3) generating a model for the gaze direction using information from the intermediate layer of the (a-1) step; The method for diagnosing dizziness according to claim 9, further comprising:

11. The process (a) comprises: Based on the blink classification model generated in the step (a-2), when the patient closes his / her eyes, the part of the pupil hidden is inferred, and the pupil center is recognized in the eyeball before the pupil is hidden. The method for diagnosing dizziness according to claim 10.

12. The process (a) comprises: The method further includes the step of outputting information about the pupil center in a three-axis (horizontal / vertical / line) graph. The method for diagnosing dizziness according to claim 9.

13. The process (b) comprises: a step of: 1) calculating the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the sclera outer portion of the patient's eye from the video of the patient; and 5) converting the two-dimensional coordinates in the direction of the pupil center calculated in step (a) into three-dimensional coordinates to calculate a direction vector of the eyeball, and outputting information on the eyeball movement. The method for diagnosing dizziness according to claim 9.

14. The process (c) comprises: a process of calculating the coordinates of 1) the patient's nose, 2) the left end of the outer sclera of the eyeball, 3) the right end, and 4) the center of the forehead from the video of the patient, converting the two-dimensional coordinates of the head direction into three-dimensional coordinates to calculate a head direction vector, and outputting information on the head movement. The method for diagnosing dizziness according to claim 9.

15. The step (d) comprises: If the time resolution of the video of the patient is "FPS", the time when the head is moved significantly is "head peak index", and the time when the pupil moves in the opposite direction within 1 second after the head is moved significantly is "eye peak index", the gain value is [0025] This is the process calculated as The method for diagnosing dizziness according to claim 9.

16. The step (e) comprises: determining an abnormality in vestibular nerve function using the eye movement output in step (b) and the gain value calculated in step (d); The method for diagnosing dizziness according to claim 10.

17. A computer-readable recording medium storing a program for implementing the dizziness diagnosis method according to any one of claims 9 to 16.

18. A computer program stored on a computer-readable recording medium for implementing the dizziness diagnosis method according to any one of claims 9 to 16.

Images