Dizziness diagnosis device and method using an ultra-fast, lightweight deep learning model that tracks eye and head position changes in video footage, a recording medium on which a program for realizing the same is stored, and a computer program stored on the recording medium

Abstract

To provide a vertigo diagnostic device which automatically removes noise, such as eye blinking and the like, and tracks and quantifies purely eyeball movement so as to allow doctors to use the values in diagnosis.SOLUTION: The present invention relates to: a technique for tracking changes in eyeballs and a head position to diagnose vertigo, and more specifically, a vertigo diagnostic device and method using a deep learning model for tracking movement of the eyeball and the head of a patient from a video including a patient to diagnose vertigo on the basis of a deep learning model, and also relates to; a recording medium storing a program for achieving the vertigo diagnosis; and the computer program stored in the recording medium.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for diagnosing dizziness by tracking changes in eye and head position, and more particularly to an apparatus and method for diagnosing dizziness by tracking eye and head movements in a video of a patient based on a deep learning model, a recording medium storing a program for realizing the same, and a computer program stored on the recording medium. [Background technology]

[0002] The material described in this section merely provides background information for the embodiments described herein and may not necessarily constitute prior art.

[0003] Vertigo is a general term for any symptom in which one feels as if one is moving even when oneself or the surroundings are stationary. This dizziness is almost always caused by a malfunction of the peripheral vestibular nervous system. When a malfunction occurs in the peripheral vestibular nervous system, the vestibulo-ocular reflex, which is the ability to fix one's gaze in response to changes in head position, disappears, causing the eyes to move spontaneously without being fixed, resulting in dizziness. Eye movement caused by abnormalities in vestibular function is called nystagmus. When dizziness occurs, doctors evaluate the nystagmus to determine whether there is a problem with vestibular function. Nystagmus can be caused by abnormalities in the peripheral vestibular nervous system as well as the central nervous system, and doctors use the appearance of the nystagmus to determine whether the dizziness a patient is experiencing is due to a peripheral vestibular system or central nervous system problem. In other words, observing nystagmus is an important examination tool that provides the greatest diagnostic information when treating patients with vertigo. Nystagmus moves in three axes: 1) horizontal (left / right), 2) vertical (up / down), and 3) axial (clockwise / counterclockwise). To date, nystagmus can be observed using Frenzel glasses, infrared photography, video Frenzel glasses, and video nystagmus testing.

[0004] However, Frenzel glasses and video Frenzel glasses are designed to directly observe eye movements, so the diagnostic accuracy can depend on the doctor's experience in judging subtle eye movements. To complement this, video nystagmus machines output nystagmus as a graph, but they require fast and highly accurate eye tracking technology, which makes them expensive and difficult to widely use.

[0005] In particular, the head impulse test, which most accurately assesses the vestibulo-ocular reflex (VERT), which refers to eye movement caused by changes in head position, involves observing the patient's eye movements while rapidly moving their head left and right or up and down without using the naked eye. Therefore, test results can vary depending on the operator's skill level. To make the results more objective, a video head impulse test is performed using a device equipped with a gyro sensor to measure head movement speed and an eye tracker to calculate eye movement speed. Only with this device can the results of the head impulse test be output, and the presence or absence of vestibular dysfunction can be determined by evaluating the gain, which is the ratio of head movement to eye movement. Furthermore, current devices lack the ability to filter out noise caused by blinking during eye movement observation, which can be inconvenient. Therefore, the examiner must instruct the patient to open their eyes for a certain period of time during the test. Furthermore, because doctors often rely solely on paper test results to determine the patient's condition, blinking can distort the results. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Publication No. 10-2018-0101816, 2018.09.14 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, the object of the present invention is to provide a dizziness diagnosis device that uses an algorithm developed by a deep learning model to automatically remove noise such as blinking from eye movement videos taken with a commonly used video nystagmus testing device as well as eye and head movements taken on video by various devices, and tracks only eye movement, quantifies it, and displays it for use by doctors in diagnosis.

[0008] Another object of the present invention is to provide a dizziness diagnosis device that can perform an important test to distinguish between peripheral and central dizziness using any device capable of video recording, thereby enabling remote medical care for the diagnosis of dizziness. Instead of performing the video head impulse test, which has been used as an objective testing method, by wearing goggles equipped with complex devices such as a gyro sensor for tracking and calculating head movement and an eye tracker for tracking and calculating eye movement, the video of the head impulse test, in which the patient's head is rapidly rotated left and right or up and down, is simply recorded using a camera such as a webcam or smartphone, and the learning algorithm calculates the speed of eye movement and head movement only from the video, replacing the video head impulse test, which is performed using complex devices.

[0009] Another object of the present invention is to provide a recording medium storing a program for implementing the dizziness diagnosis method, and a computer program stored on the recording medium.

[0010] The present specification is not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0011] In order to achieve the above object, according to an embodiment of the present invention, there is provided a dizziness diagnosis device including an eyeball movement learning unit that identifies eyeball parts from an eyeball video captured of a patient's eyeballs, extracts eyeball images for each frame, recognizes the pupil center, calculates the pupil center coordinates, and learns eyeball movement; an eyeball movement output unit that receives information learned by the eyeball movement learning unit and outputs information about eyeball movement from the video captured of the patient; a head movement output unit that outputs information about head movement from the video captured of the patient; a calculation unit that receives information about eyeball movement and head movement from the eyeball movement and head movement output unit, calculates eyeball movement speed and head movement speed, and calculates gain values; and a diagnosis unit that receives the gain values ​​calculated by the calculation unit and diagnoses dizziness.

[0012] In addition, the eye movement learning unit may include an object segmentation model generation module that divides the eye image divided by frame into regions corresponding to the sclera, the iris, and the pupil to generate an object segmentation model; a blink identification module that receives information on an intermediate layer of the object segmentation model generation to learn anatomical structures around the eyeball, identifies blinks, and generates a blink classification model; and an eye tracking model generation module that receives information on the intermediate layer of the object segmentation model generation to generate a model for the gaze direction.

[0013] Furthermore, the eye movement learning unit can infer the part of the pupil hidden when the patient closes his or her eyes based on the blink classification model, and recognize the pupil center in the eye before the pupil is hidden.

[0014] Also, the eye movement learning unit may further include a graph output module that outputs the information about the pupil center in a three-axis (horizontal / vertical / linear) graph.

[0015] In addition, the eyeball movement output unit can calculate 1) the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the outer scleral portion of the patient's eyeball from the video captured of the patient, and 5) receive the coordinates of the pupil center calculated by the eyeball movement learning unit, convert the two-dimensional coordinates relative to the direction of the pupil center into three-dimensional coordinates, calculate the direction vector of the eyeball, and output information about the eyeball movement.

[0016] In addition, the head movement output unit can calculate the coordinates of 1) the patient's nose, 2) the left end of the outer scleral region of the eyeball, 3) the right end, and 4) the center of the forehead from the video of the patient, convert the two-dimensional coordinates relative to the head direction into three-dimensional coordinates, calculate the head direction vector, and output information about the head movement.

[0017] In addition, the calculation unit calculates the gain value as follows: if the time resolution of the video of the patient is "FPS", the point in time when the head moves significantly is "head peak index", and the point in time when the pupil moves in the opposite direction within 1 second after the head moves significantly is "eye peak index",

number

[0018] The presence or absence of abnormality in vestibular nerve function can be determined by using the eye movement output from the eye movement output unit and the gain value.

[0019] In another embodiment of the present invention to achieve the above object, there is provided a method for diagnosing dizziness, including: (a) identifying an eyeball portion from an eyeball video captured by capturing a patient's eyeball, extracting an eyeball image for each frame, recognizing a pupil center, calculating the coordinates of the pupil center, and learning the movement of the pupil center; (b) outputting information on eyeball movement from the video captured by the patient using the information on eyeball movement learned in step (a); (c) outputting information on head movement from the video captured by the patient; (d) calculating a gain value by calculating an eyeball movement velocity and a head movement velocity using the information on eyeball movement and head movement output in steps (b) and (c); and (e) diagnosing dizziness using the gain value calculated in step (d).

[0020] In addition, the step (a) may include the steps of: (a-1) dividing the eyeball image divided by frame into regions corresponding to the sclera, iris, and pupil to generate an object segmentation model; (a-2) learning anatomical structures around the eyeball using information from the intermediate layer of the step (a-1), identifying blinks, and generating a blink classification model; and (a-3) generating a model for the gaze direction using information from the intermediate layer of the step (a-1).

[0021] In addition, the step (a) may be a step of recognizing the center of the pupil in the eyeball before the pupil is hidden by analogizing the part of the pupil hidden when the patient closes his / her eyes based on the blink classification model generated in the step (a-2).

[0022] Also, the step (a) may further include a step of outputting the information about the pupil center in a three-axis (horizontal / vertical / linear) graph.

[0023] In addition, the step (b) may be a step of calculating, from a video of the patient, 1) the coordinates of the upper end, 2) the lower end, 3) the left end, and 4) the right end of the outer sclera portion of the patient's eyeball, and 5) converting the two-dimensional coordinates in the direction of the pupil center calculated in the step (a) into three-dimensional coordinates to calculate a direction vector of the eyeball, and outputting information about the eyeball movement.

[0024] Furthermore, the step (c) may be a step of calculating the coordinates of 1) the patient's nose, 2) the left end of the outer sclera of the eyeball, 3) the right end, and 4) the center of the forehead from a video of the patient, converting the two-dimensional coordinates relative to the head direction into three-dimensional coordinates to calculate a head direction vector, and outputting information about head movement.

[0025] In addition, in the step (d), the time resolution of the video of the patient is defined as "FPS", the point at which the head moves significantly is defined as "head peak index", and the point at which the pupil moves in the opposite direction within 1 second after the head moves significantly is defined as "eye peak index". The gain value is calculated as follows:

number

[0026] In addition, the step (e) may be a step of determining abnormality in vestibular nerve function using the eye movement output in the step (b) and the gain value calculated in the step (d).

[0027] In order to achieve the above object, a computer-readable recording medium storing a program for implementing the above dizziness diagnosis method according to still another embodiment of the present invention is provided.

[0028] In order to achieve the above object, a computer program stored on a computer-readable recording medium for implementing the above dizziness diagnosis method according to still another embodiment of the present invention is provided. [Effects of the Invention]

[0029] As described above, according to the dizziness diagnosis device and method according to the embodiment of the present invention, the algorithm can be applied to existing testing equipment to automatically remove noise such as blinking and output results related purely to eye movement. In addition, the algorithm can calculate eye movement and head movement through a deep learning model from video of the patient's eyes and video footage of the patient, allowing for easy dizziness testing using only video footage without any complicated equipment.

[0030] In addition, according to the dizziness diagnosis device and method according to the embodiment of the present invention, head impulse testing can be performed based on video of the patient's eyes and video of the patient, making remote medical care possible for dizziness diagnosis, thereby contributing to health and well-being.

[0031] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0032] [Figure 1] 1 is a block diagram showing a simplified dizziness diagnosis device according to an embodiment of the present specification. [Figure 2] 1 is a diagram simply illustrating a learning process for eyeball movement according to the present specification; [Figure 3]This is a diagram showing the process of generating an object segmentation model from an eyeball video, recognizing the pupil center, and outputting the movement of the pupil center in a graph. [Figure 4] 1 is a diagram showing the generation of an artificial intelligence model for pupil center recognition. [Figure 5] 1 is a diagram showing a method for measuring the speed of eye movement due to head movement. [Figure 6] 1 is a flowchart illustrating a method for diagnosing dizziness according to an embodiment of the present specification. [Figure 7] 10 is a flowchart showing a process of recognizing the pupil center and calculating the pupil center coordinates in an eyeball movement learning unit according to an embodiment of the present specification. DETAILED DESCRIPTION OF THE INVENTION

[0033] The advantages and features of the present invention, as well as methods for achieving them, will become apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be realized in various different forms, which are provided to fully convey the scope of the invention to those skilled in the art. The present invention is defined by the scope of the claims.

[0034] Furthermore, the terms used in this specification are for the purpose of describing embodiments and are not intended to limit the present invention. Furthermore, in this specification, the singular forms include the plural forms unless otherwise specified in the phrase. For example, the terms "comprise" (or "comprise") and / or "comprising" (or "comprising") used in this specification can include the elements and steps mentioned. The same reference numerals refer to the same elements throughout the specification. "And / or" includes each and every combination of one or more of the mentioned items.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein are used with the meaning commonly understood by those of ordinary skill in the art, and terms defined in commonly used dictionaries are not to be idealized or over-analyzed unless expressly defined otherwise.

[0036] FIG. 1 is a block diagram showing a simplified dizziness diagnosis device according to an embodiment of the present specification.

[0037] Referring to FIG. 1, a dizziness diagnosis device 10 according to an embodiment of the present specification can learn eye movement using an eye video 1. Based on the learning information about pupil-centered movement, eye movement caused by head movement can be measured to diagnose dizziness. As a result, dizziness can be diagnosed using the eye video 1 and the patient-photographed video 2 without using any other device. The eye video 1 and the patient-photographed video 2 may be videos captured by a video-capable camera. Video data can be in various formats, such as MP4, MOV, WMV, AVI, and MKV, and the video data according to the present specification is not limited to a specific video file format.

[0038] The dizziness diagnosis device 10 may include an eye movement learning unit 11 , an eye movement output unit 12 , a head movement output unit 13 , a calculation unit 14 , and a diagnosis unit 15 .

[0039] FIG. 2 is a diagram simply illustrating a learning process for eyeball movement according to the present specification.

[0040] 1 and 2, the eyeball movement learning unit 11 can identify the eyeball portion from the eyeball video 1 and extract an eyeball image 21 for each video frame. The eyeball movement learning unit 11 can recognize the pupil center in the eyeball image 21 and calculate the coordinate value of the pupil center to learn the eyeball movement. Image data comes in various formats such as JPG, PNG, GIF, SVG, etc., and the image data according to the present specification is not limited to a specific image file format.

[0041] The eye movement learning unit 11 may include an object segmentation model generation module 111 , a blink identification module 112 and an eye-tracking model generation module 113 .

[0042] The object segmentation model generation module 111 can generate an object segmentation model 22 that segments the eyeball into a sclera, an iris, and a pupil in the eyeball image 21. The eyeball movement learning unit 11 can use information from the object segmentation model 22 to recognize the pupil center.

[0043] The object segmentation model generation module 111 can provide information on the sclera, iris, and pupil regions that cannot be provided by conventional nystagmus testing methods using an infrared camera.

[0044] The blink identification module 112 receives information from the intermediate layer of the object segmentation model generation and learns anatomical structures around the eyeball to identify blinks, and can generate a blink classification model that classifies blink steps.

[0045] The eye movement learning unit 11 can recognize the center of the pupil by estimating the part of the pupil hidden when the patient closes his / her eyes based on the blink classification model, and can also remove noise caused by blinking, thereby preventing distortion of the results due to blinking.

[0046] The gaze tracking model generation module 113 receives information from the intermediate layer of the object segmentation model generation and outputs a gaze tracking model, which is a model for the gaze direction. The eye movement learning unit 11 can use information from the gaze tracking model to recognize the pupil center.

[0047] The eye movement learning unit 11 may further include a graph output module 114. The graph output module 114 may output the coordinate values ​​of the pupil center calculated based on information from the object segmentation model 22, the blink classification model, and the gaze tracking model 23 as a three-axis movement graph 24. The three-axis graph 24 may be output as a horizontal movement graph 241 showing left / right movement of the pupil center, a vertical movement graph 242 showing up / down movement, and a line movement graph (not shown) showing clockwise / counterclockwise movement.

[0048] Figure 3 shows the process of generating an object segmentation model from an eyeball video, recognizing the pupil center, and outputting the movement of the pupil center in a graph.

[0049] 1 and 3, the object segmentation model generation module 111 can generate an object segmentation model 22 from the patient's eyeball image extracted from the eyeball movement learning unit 11. The object segmentation model can segment the patient's eyeball image into a sclera 30, an iris 31, and a pupil 32. The object segmentation model generation module 111 can also output an object integrated image 33 that integrates the object-segmented images of the sclera 30, the iris 31, and the pupil 32. The eyeball movement learning unit 11 can track the movement of the pupil center in real time using the object integrated image 33.

[0050] The eye movement learning unit 11 can output the pupil center movement graph 34 through the graph output module 114 based on the pupil center information recognized using the object integrated image 33 .

[0051] The pupil center graph 34 may be output using pupil center information provided by a deep learning model. The pupil center graph 34 may also be calculated more precisely using the center of gravity of the object-segmented pupil 32. The pupil center graph 34 may be output as a vertical motion graph 341, a horizontal motion graph 342, and a linear motion graph (not shown).

[0052] FIG. 4 is a diagram showing the generation of an artificial intelligence model for pupil center recognition.

[0053] Referring to FIG. 4, in order for artificial intelligence to learn pupil-centered information well, an object segmentation model 22, a blink classification model 40, and an eye tracking model 41 can be generated through a deep learning model.

[0054] The object segmentation model generation module 111 may generate the object segmentation model 22 from the eyeball image 21 using a convolutional neural network (CNN) deep learning model. The CNN deep learning model is a technology widely known to those skilled in the art, and therefore, a detailed description thereof will be omitted.

[0055] The blink classification model 40 and the gaze tracking model 41 can be generated using information from an intermediate layer 42 in the generation process of the object segmentation model 22. By using the information from the intermediate layer 42, pupil-centered information can be learned more efficiently than in conventional deep learning models.

[0056] The blink classification model 40 can classify blinking movements into eyes open (open), eyes closing (closing), or eyes closed (closed) based on information from the intermediate layer 42.

[0057] The gaze tracking model 41 can show the gaze direction as a heatmap and keypoints based on the information of the intermediate layer 42.

[0058] According to this specification, the eye movement learning unit 11 can learn pupil center information using at least one model selected from the object segmentation model 22, the blink classification model 40, and the gaze tracking model 41.

[0059] For example, the eye movement learning unit 11 can learn eye movements using only information from the blink classification model 40 and the gaze tracking model 41. In this case, information from the object segmentation model 22 is not used, so that the lengthy calculation of the intermediate layer 42 is not required. As a result, information on the pupil center can be learned and output faster than a conventional deep learning model.

[0060] The CNN deep learning model for learning eyeball movements corresponds to one example, and the learning method is not limited to this example.

[0061] FIG. 5 is a diagram showing a method for measuring the speed of eye movement due to head movement.

[0062] 1 and 5, the eye movement output unit 12 and the head movement output unit 13 can output information on the eye movement and the head movement of the patient from the moving image 2 captured by the patient.

[0063] The eye movement output unit 12 may receive information on the learned eye movement from the eye movement learning unit 11. The eye movement output unit 12 may output information on the eye movement from the patient-photographed video 2 based on the learned information on the eye movement, and output the eye movement as a graph 53.

[0064] The eye movement output unit 12 can calculate a direction vector of the eye to output information on the eye movement. To calculate the direction vector of the eye, two-dimensional coordinates of at least four points can be converted into coordinates in three-dimensional space using Perspective-n-Point (PnP) pose computation. The PnP pose computation is a technique known to those skilled in the art, and therefore a detailed description thereof will be omitted.

[0065] In order to calculate the direction vector of the eyeball, the eyeball movement output unit 12 can perform PnP pose computation on 1) the coordinates of the upper end, 2) the lower end, 3) the left end, and 4) the right end of the scleral outer periphery, and 5) the two-dimensional coordinates of the pupil center calculated by the eyeball movement learning unit 11.

[0066] Furthermore, the eye movement output unit 12 can output a pupil center display 50 that displays the pupil center according to the eye movement of the patient in the patient-photographed moving image 2.

[0067] The head movement output unit 13 can output information on head movement from the patient's moving image 2 and output the head movement in the form of a graph 52.

[0068] The head movement output unit 13 can calculate a head direction vector to output information about head movement. The head movement output unit 13 can calculate the head direction vector using PnP pose computation using coordinates of 1) the nose, 2) the left edge of the sclera outer region, 3) the right edge, and 4) the center of the forehead from the patient's photographed video 2. The head movement output unit 13 can also recognize head movement using the movement of the center of the nose in the patient's photographed video 2. Additionally, the head movement output unit 13 can output a nose center display 51 that outputs the center of the patient's nose in the patient's photographed video 2.

[0069] In this specification, the coordinates of 1) the nose, 2) the left edge of the scleral outer periphery, 3) the right edge, and 4) the center of the forehead are used to output the head movement from the patient-recorded video 2. However, the head movement does not necessarily have to be output using these four coordinates. For example, instead of the coordinates of 2) the left edge of the scleral outer periphery and 3) the right edge, the coordinates of the upper and lower edges of the scleral outer periphery can be used. Furthermore, in addition to these four coordinates, the head movement can be output by adding coordinates of any point on the patient's face in the patient-recorded video 2. Therefore, it is not necessary to use these four coordinates to output the head movement. It should be understood that the head movement can be output by performing PnP pose computation using the coordinates of at least three or more arbitrary points on the patient's face in the patient-recorded video 2.

[0070] The calculation unit 14 can calculate the speed of eye movement and the speed of head movement using the information on eye movement and head movement inputted from the eye movement output unit 12 and the head movement output unit 13. Using this, the calculation unit 14 can calculate a gain value which means the ratio between the speed of head movement and the speed of eye movement.

[0071] According to the conventional technique, to calculate the gain value, it is assumed that the object the patient is looking at is at the center of the camera. In this case, the gain value can be obtained by [Equation 1].

number

[0072] However, the desired temporal resolution cannot be obtained from video, and the prediction of monocular gaze vectors is inaccurate.

[0073] According to the present specification, since the eye movements caused by head movements occur simultaneously in the same direction, it is possible to estimate the eye gaze vector by averaging the changes in both eyes. The eye gaze vector can be used to improve the accuracy of the eye direction.

[0074] To calculate the gain value, the time resolution of the patient-recorded video 2 is defined as "FPS," the point at which the head moves significantly is defined as "head peak index," and the point at which the pupil moves in the opposite direction within 1 second after the head moves significantly is defined as "eye peak index." The gain value can be calculated using [Equation 2].

number

[0075] According to an embodiment of the present specification, the 'eye peak index' may correspond to the value at the point when the velocity value when the eyeball moves in the opposite direction to the head is extracted and the value above the third quartile first appears.

[0076] The calculation unit 14 can also output a graph 54 showing real-time changes in the head and eyes and a graph 55 showing gain values. At this time, the changes in the left and right eyes and the gain values ​​can be output, respectively.

[0077] The diagnosing unit 15 can diagnose dizziness in a patient using the gain value calculated by the calculating unit 14.

[0078] The gain value can be used to clinically determine whether there is a problem with the vestibular nerve. For people with normal vestibular nerve function, when the head moves, the eyes move accordingly. This allows the gain value to be calculated to a value close to 1.

[0079] However, if an abnormality occurs in the vestibular nerve, the eyes cannot move in unison with the movement of the head, causing the gain value to be smaller than "1."

[0080] For example, the gain value of the right eye calculated from the patient video 2 may be "0.7" and the gain value of the left eye may be "1.23." In this case, the gain value of the right eye is smaller than "1," which may indicate a decrease in right vestibular nerve function.

[0081] In addition, the diagnosis unit 15 can determine whether there is an abnormality in the central vestibular nerve function or the peripheral vestibular nerve function by using the gain value and the directional vector of the eyeball movement calculated by the eyeball movement output unit 12.

[0082] FIG. 6 is a flowchart illustrating a method for diagnosing dizziness according to an embodiment of the present specification.

[0083] 1 and 6, eyeball movement can be learned from an eyeball video 1 obtained by capturing a patient's eyeball (S1). To learn eyeball movement, the eyeball portion can be identified from the eyeball video 1. Then, an image of the eyeball can be extracted for each frame from the eyeball video 1. The extracted eyeball image can be used to recognize the pupil center and calculate the coordinates of the pupil center to learn eyeball movement.

[0084] FIG. 7 is a flowchart showing the process of recognizing the pupil center and calculating the pupil center coordinates in the eyeball movement learning unit according to the embodiment of the present invention.

[0085] 1 and 7, an image of an eyeball can be extracted from the eyeball video 1 (S5). An object segmentation model can be generated using the image of the eyeball (S7). Furthermore, information on an intermediate layer that generates the object segmentation model can be input to generate a blink classification model (S6) and an eye tracking model (S8). When a patient closes their eyes in the eyeball video 1 using the blink classification model, the pupil center can be recognized using the eyeball before closing the eye. Thereafter, the pupil center can be recognized and pupil center coordinates can be calculated using at least one of the object segmentation model, blink classification model, and eye tracking model (S9).

[0086] The eye movement learning step (S1) may further include a step (S10) of outputting the coordinates of the pupil center as a graph, where the graph of the pupil center coordinates may be output as a graph with three axes: horizontal, vertical, and linear.

[0087] Next, the patient's eye and head movements can be output in the patient-recorded video 2 (S2). The patient's eye movements can be output based on the information about the pupil center learned in the eye movement learning process (S1). In addition, the information about the eye and head movements can be output in the form of graphs.

[0088] In order to output the information on the eyeball movement, the coordinates of 1) the upper end, 2) the lower end, 3) the left end, and 4) the right end of the scleral outer periphery of the patient's eyeball from the patient-captured video 2 and the coordinates of the pupil center calculated in the eyeball movement learning process (S1) are used to convert the two-dimensional coordinates in the direction of the pupil center into three-dimensional coordinates, thereby calculating the direction vector of the eyeball.

[0089] To output the head movement information, the head direction vector can be calculated by converting the two-dimensional coordinates of the head direction into three-dimensional coordinates using the coordinates of 1) the patient's nose, 2) the left end of the outer sclera of the eyeball, 3) the right end, and 4) the center of the forehead from the patient's video 2.

[0090] Next, a gain value can be calculated and output based on the patient's eye and head movements (S3). To calculate the gain value, the eye and head movement velocities can be calculated using the eye and head movement data.

[0091] The gain value can be calculated using the eye movement speed and head movement speed. If the time resolution of the patient-recorded video 2 is "FPS," the point in time when the head moves significantly is "head peak index," and the point in time when the pupil moves in the opposite direction within 1 second after the head moves significantly is "eye peak index," the gain value can be calculated using [Equation 2].

[0092] In addition, the instantaneous changes in the head and eye movements and the gain values ​​can be output in a graph, and in this process, the changes and gain values ​​for the left and right eyes can be displayed respectively.

[0093] Next, dizziness can be diagnosed based on the gain value (S4). The closer the gain value is to "1", the more normal the condition, and the more deviated the value is, the more abnormal the vestibular nerve function. Furthermore, based on the gain value and the eye movement, it is possible to determine whether the vestibular nerve function is central or peripheral.

[0094] The dizziness diagnosis method according to the embodiment of the present invention described above can be realized in the form of a computer-executable recording medium (or a computer program product), for example, a program module stored on a computer-readable medium and executed by a computer.

[0095] Here, the computer-readable medium may include a computer storage medium (for example, a memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD)). The computer-readable medium may be any available medium that can be accessed by a computer, including, for example, both volatile and nonvolatile media, and both separate and non-separate media.

[0096] Furthermore, the dizziness diagnosis method according to the embodiment of the present invention includes instructions executable in whole or in part by a computer, and a computer program includes programmable machine instructions processed by a processor, and can be implemented in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like.

[0097] As mentioned above, the preferred embodiments of the present invention have been described and illustrated using specific terms, but these terms are merely for the purpose of clearly describing the present invention. It is obvious that various modifications and changes can be made to the embodiments of the present invention and the terms used without departing from the technical spirit and scope of the following claims. Such modified embodiments should not be understood separately from the spirit and scope of the present invention, but should be considered to fall within the scope of the claims of the present invention. [Explanation of symbols]

[0098] 10 dizziness diagnosis device, 11 eye movement learning unit, 12 eye movement output unit, 13 head movement output unit, 14 calculation unit, 15 diagnosis unit, 111 object segmentation model generation module, 112 blink identification module, 113 gaze tracking model generation module, 114 graph output module.

Claims

1. an eyeball movement learning unit that identifies the eyeball portion from an eyeball video captured of the patient's eyeball, extracts eyeball images for each frame, recognizes the pupil center, calculates the pupil center coordinates, and learns eyeball movement; an eyeball movement output unit that receives the information learned by the eyeball movement learning unit and outputs information about the eyeball movement from a video image of a patient; a head movement output unit that outputs information about head movement from a video of the patient; a calculation unit that receives information on eyeball movement and head movement from the eyeball movement and head movement output unit, calculates an eyeball movement velocity and a head movement velocity, and calculates a gain value; a diagnosis unit that receives the gain value calculated by the calculation unit and diagnoses dizziness; Including, The eye movement learning unit an object segmentation model generation module that generates an object segmentation model by dividing regions corresponding to the sclera, the iris, and the pupil in the eyeball image divided by frame; a blink identification module that receives information from the intermediate layer of the object segmentation model generation and learns anatomical structures around the eyeball, identifies blinks, and generates a blink classification model; an eye tracking model generation module that receives information on an intermediate layer of the object segmentation model generation module and generates a model for the eye gaze direction; Including, the eyeball movement learning unit infers a part of the pupil that is hidden when the patient closes his / her eyes based on the blink classification model, and recognizes the pupil center with the eyeball before the pupil is hidden. A dizziness diagnostic device characterized by:

2. The eye movement learning unit a graph output module for outputting the information about the pupil center in a three-axis (horizontal / vertical / line) graph; 2. The dizziness diagnosis device according to claim 1.

3. The eye movement output unit From the video of the patient, 1) the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the scleral outer periphery of the patient's eyeball are calculated, and 5) the coordinates of the pupil center calculated by the eyeball movement learning unit are input, and the two-dimensional coordinates relative to the direction of the pupil center are converted into three-dimensional coordinates to calculate the direction vector of the eyeball, and information on the eyeball movement is output.

2. The dizziness diagnosis device according to claim 1.

4. The head movement output unit From the video of the patient, the coordinates of 1) the patient's nose, 2) the left end of the outer sclera of the eyeball, 3) the right end, and 4) the center of the forehead are calculated, and the two-dimensional coordinates relative to the head direction are converted into three-dimensional coordinates to calculate the head direction vector, and information on head movement is output.

2. The dizziness diagnosis device according to claim 1.

5. The calculation unit If the time resolution of the video of the patient is "FPS", the time point when the head moves significantly is "head peak index", and the time when the pupil moves in the opposite direction within 1 second after the head moves significantly is "eye peak index", the gain value is [Equation 1] It is calculated as 2. The dizziness diagnosis device according to claim 1.

6. The diagnostic unit determining an abnormality in vestibular nerve function by using the eye movement output by the eye movement output unit and the gain value; 2. The dizziness diagnosis device according to claim 1.

7. A computer comprising: (a) identifying the eyeball portion from an eyeball video captured of a patient's eyeball, extracting an eyeball image for each frame, recognizing the pupil center, calculating the pupil center coordinates, and learning the eyeball movement; (b) outputting information on eye movement from a video of a patient using the information on eye movement learned in the process (a); (c) outputting head movement information from a video of the patient; (d) calculating a gain value indicating an abnormality in the patient's vestibular nerve function by calculating an eye movement velocity and a head movement velocity using the eye movement and head movement information output in the steps (b) and (c); and Run The step (a) is (a-1) dividing the eyeball image divided by frame into regions corresponding to the sclera, iris, and pupil to generate an object division model; (a-2) a step of learning anatomical structures around the eyeball using information from the intermediate layer in the step (a-1), identifying blinks, and generating a blink classification model; (a-3) generating a model for the gaze direction using information from the intermediate layer in the (a-1) step; Including, The step (a) is a step of recognizing the center of the pupil by analogizing the part of the eyeball where the pupil is hidden when the patient closes his / her eyes based on the blink classification model generated in the step (a-2). A method for operating a dizziness diagnosis device.

8. The step (a) is The method further includes a step of outputting information about the pupil center in a three-axis (horizontal / vertical / line) graph.

8. A method for operating the dizziness diagnosis device according to claim 7.

9. The step (b) is a step of: 1) calculating the coordinates of the upper end, 2) lower end, 3) left end, and 4) right end of the outer sclera portion of the patient's eyeball from the video captured of the patient; and 5) converting the two-dimensional coordinates in the direction of the pupil center into three-dimensional coordinates using the coordinates of the pupil center calculated in step (a) to calculate a direction vector of the eyeball, and outputting information on the eyeball movement.

8. A method for operating the dizziness diagnosis device according to claim 7.

10. The step (c) is a step of calculating the coordinates of 1) the patient's nose, 2) the left edge of the outer sclera of the eyeball, 3) the right edge, and 4) the center of the forehead from the video of the patient, converting the two-dimensional coordinates of the head direction into three-dimensional coordinates to calculate a head direction vector, and outputting head movement information; 8. A method for operating the dizziness diagnosis device according to claim 7.

11. The step (d) is If the time resolution of the video of the patient is "FPS", the time point when the head moves significantly is "head peak index", and the time when the pupil moves in the opposite direction within 1 second after the head moves significantly is "eye peak index", the gain value is [Equation 2] This is the process calculated as 8. A method for operating the dizziness diagnosis device according to claim 7.

12. The step (e) is determining abnormality in vestibular nerve function using the eye movement output in step (b) and the gain value calculated in step (d); 8. A method for operating the dizziness diagnosis device according to claim 7.

13. A computer-readable recording medium storing a program for implementing the method for operating the dizziness diagnosis device according to any one of claims 7 to 12.

14. A computer program stored on a computer-readable recording medium for implementing the method for operating the dizziness diagnosis device according to any one of claims 7 to 12.

Images