A method for identifying nystagmus types based on optical flow

Abstract

The embodiment of the present invention discloses a method for identifying nystagmus type based on optical flow method, including: step 1: collecting nystagmus video of the patient through an infrared camera; step 2: extracting video frames of the nystagmus video every other frame; step 3: removing the closed-eye frames; step 4: using the transfer learning method to segment the iris part of the retained video frame; step 5: calculating the optical flow of continuous video frames, and cropping to obtain an optical flow map; step 6: inputting the obtained optical flow map into the trained ResNet+LSTM network model in sequence, and automatically identifying the nystagmus type through the network model. The present invention combines the optical flow method and deep learning to identify the nystagmus type, and can effectively and accurately identify rotational nystagmus.

Description

Technical Field

[0001] The present invention relates to the medical field, and in particular to a nystagmus type recognition method based on an optical flow method. Background Art

[0002] Benign paroxysmal positional vertigo (BPPV) is the most common peripheral vestibular disease. When the disease occurs, patients may experience severe dizziness, instability and other symptoms. In severe cases, they may experience nausea and vomiting, and dare not open their eyes or walk. Repeated attacks can also cause tension and anxiety, which can seriously threaten people's healthy life and work. BPPV significantly increases the risk of falls in the elderly and increases the socioeconomic burden to a certain extent.

[0003] The main feature of BPPV is short-term vertigo induced when the head moves to a specific position, accompanied by characteristic nystagmus. According to the direction of eye movement, nystagmus is divided into horizontal, vertical, and rotational types. These nystagmus patterns often appear in a mixed manner, such as "horizontal plus rotation", "vertical plus rotation", etc. According to the location of the onset, BPPV types can be mainly divided into posterior semicircular canal BPPV (posterior canal-BPPV, PC-BPPV), horizontal semicircular canal BPPV (horizontal canal-BPPV, HC-BPPV), anterior semicircular canal BPPV (anterior canal-BPPV, AC-BPPV) and mixed BPPV (multiple canals-BPPV, MC-BPPV). Among them, posterior semicircular canal BPPV and horizontal semicircular canal BPPV are the most common. [3] (Von Brevern M, Bertholon P, Brandt T, et al. Benign paroxysmalpositional vertigo: diagnostic criteria[J]. Journal of Vestibular Research,2015, 25(3, 4): 105-117.) In clinical practice, BPPV is mainly diagnosed by the clinical features and patterns of nystagmus induced by movements in specific positions, such as the Dix-Hallpike test and the supine rolling test.

[0004] At present, the clinical nystagmus examination methods for vertigo diseases mainly include:

[0005] (1) Naked eye examination method: The examiner stands in front of the examinee or slightly to one side. The examinee looks in the five basic directions of left, right, up, down and straight ahead, as indicated by the examiner's finger, and observes the eye movement.

[0006] (2) Electronystagmography: This method uses the electric field formed around the eyeball by the potential difference between the cornea (positive potential) and the retina (negative potential). When the eyeball moves, the surrounding electric field changes accordingly. The skin electrodes placed around the eyeball can derive the changes in the electric field, which are then transmitted to the recording device through an amplifier to record the electronystagmography. The main parameters for analyzing the electronystagmography are the slow phase angular velocity and duration of the nystagmus.

[0007] (3) Nystagmus imaging method: Infrared lighting and high-speed cameras are used to collect nystagmus images. Doctors diagnose the type of nystagmus by observing and analyzing the nystagmus images. Currently, some studies use template matching, optical flow, deep learning and other methods to analyze nystagmus images and then determine the type of nystagmus.

[0008] The existing inspection methods have the following disadvantages:

[0009] (1) Naked eye examination: When doctors visually analyze nystagmus, the direction of rotational nystagmus is often difficult to discern. At the same time, some subtle nystagmus can be easily overlooked by the naked eye. These will affect the localization diagnosis of BPPV and subsequent repositioning treatment.

[0010] (2) Electronystagmography: In the experiment, the corneal retinal potential of different people is often different. Calibration is required for each test and it is easily affected by the patient's emotions. Since the bioelectric signal representing nystagmus is extremely weak, it needs to be amplified by an amplifier to detect a clear signal. Interference from other biological signals or electromagnetic interference in the environment can affect the test results. Currently, this technology can only detect horizontal and vertical nystagmus, but not rotational nystagmus. Electronystagmography has very high requirements for equipment and environment, and the equipment is expensive and not easy to popularize.

[0011] (3) Nystagmus imaging method: The most common nystagmus imaging method currently uses a video goggles to collect nystagmus videos, and doctors view the recorded nystagmus videos for diagnosis. Nystagmus videos are numerous and long, and rely on doctors to conduct manual inspections and diagnoses. The inspection process is time-consuming and has poor repeatability. Each diagnosis is affected by the doctor's experience and subjective factors. When the number of patients is large, the doctor's workload will increase significantly, leading to problems such as reduced diagnostic efficiency.

[0012] In nystagmus imaging, how to identify rotational nystagmus is a key issue. Some researchers use iris template matching to track the rotational movement of the eyeball. [4](Tanaka T, Tominaga S. Analysis of rotationalvertigo using video and image processing[C] / / 2011 4th InternationalConference on Biomedical Engineering and Informatics (BMEI). IEEE, 2011, 1:122-127.), some methods detect eye rotation by extracting the outer circle of the iris and flattening it into a rectangular pattern, and then calculating the movement of the feature points of the iris pattern between the previous and next two frames. [5](Ong JKY, Haslwanter T. Measuring torsional eyemovements by tracking stable iris features[J]. Journal of neurosciencemethods, 2010, 192(2): 261-267.) These two methods can measure eye rotation under certain conditions, but the iris will change due to the dilation or contraction of the pupil, and most patients cannot fully open their eyes during clinical examinations, resulting in the inability to match the template. Some researchers have used the optical flow method to track iris feature points to obtain eye rotation information. Reference [2] (Stahl BK, Soares LP, Sangalli VA, et al. DirectFlow: a RobustMethod for Ocular Torsion Measurement[C] / / 2019 IEEE 43rd Annual ComputerSoftware and Applications Conference (COMPSAC). IEEE, 2019, 1: 300-305.) uses the Lucas-Kanade-Pyrm (LKP) optical flow algorithm to track the optical flow of iris feature points in the previous and next frames to measure the eye rotation. However, interference such as pupil dilation and light reflection can cause the tracking effect to decrease or feature points to be lost. Reference [1] (Zhang W, Wu H, Liu Y, et al. Deep learning based torsional nystagmus detection for dizziness and vertigo diagnosis [J]. Biomedical Signal Processing and Control, 2021, 68: 102616.) uses a dense optical flow field to detect whether the eyeball has rotational movement. However, it does not conduct in-depth research on the optical flow movement under different nystagmus conditions. It can only distinguish between the two conditions of rotation and non-rotation, and cannot distinguish between multiple nystagmus patterns well.Reference [6] (Lim EC, Park JH, Jeon HJ, et al. Developing a diagnostic decision support system for benign paroxysmal positional vertigo using a deep-learning model [J]. Journal of clinical medicine, 2019, 8 (5): 633.) used a convolutional neural network to classify nystagmus videos. The experimental results showed that the classification effect of horizontal and vertical nystagmus was good, but the recognition effect of torsional nystagmus was poor. In summary, the existing methods cannot effectively identify rotational nystagmus. Summary of the invention

[0013] The technical problem to be solved by the embodiments of the present invention is to provide a method for identifying nystagmus types based on the optical flow method, so as to collect infrared nystagmus videos, analyze the nystagmus types, and assist doctors in diagnosis.

[0014] In order to solve the above technical problems, an embodiment of the present invention proposes a method for identifying nystagmus type based on an optical flow method, comprising:

[0015] Step 1: Collect the patient’s nystagmus video using an infrared camera , T refers to the total number of frames of the nystagmus video V;

[0016] Step 2: Extract video frames of the nystagmus video every other frame ;

[0017] Step 3: Set a threshold and calculate whether the pupil area exceeds the threshold. If it exceeds the threshold, the video frame is retained; otherwise, the video frame is deleted, and the closed-eye frames are removed.

[0018] Step 4: Use transfer learning method to segment the iris part of the retained video frame;

[0019] Step 5: Calculate the optical flow of continuous video frames and represent it in HSV color space, where the hue of each pixel represents the direction and the intensity represents the size; take the pupil center as the image center and crop to obtain the optical flow map;

[0020] Step 6: Build and train the ResNet+LSTM network model, input the obtained optical flow map into the trained ResNet+LSTM network model in sequence, and automatically identify the type of nystagmus through the network model.

[0021] Furthermore, in step 3, the following steps are used to remove the closed eye frames:

[0022] (1) Perform an opening operation on the frame, that is, erode first and then dilate, to eliminate the bright spots of the pupil and sclera that reflect light;

[0023] (2) Perform a closing operation on the frame, that is, dilate first and then erode to remove eyebrows and eyelashes;

[0024] (3) Calculate the mean of the lowest grayscale values ​​in all frames of this video l and the minimum average gray value m ,use( l + m ) / 2 is used as the binarization threshold to perform binarization on the video frame;

[0025] (4) Calculate the average value of the pupil area ratio after binarization in all frames of this video. t , taking the average value as the threshold;

[0026] (5) Binarize each frame and calculate the proportion of the pupil in the screen. r ;

[0027] (6) If r Greater than t , then the video frame is retained, otherwise it is considered as a closed-eye frame and deleted.

[0028] The beneficial effects of the present invention are:

[0029] (1) A new nystagmus classification scheme based on nystagmus video data is proposed. This scheme only requires the acquisition of infrared video data of the patient's eyes, has low equipment requirements, is highly universal, and can be used in most clinical diagnosis of nystagmus.

[0030] (2) The scheme combines a variety of deep learning-based methods, including iris segmentation and optical flow calculation, which can effectively reduce the impact of eyelid and eyelash occlusion and pupil contraction or dilation on the nystagmus type classification results during nystagmus identification, thereby improving the accuracy of nystagmus type identification.

[0031] (3) Different patients have different degrees of eyelid occlusion, and different nystagmus videos have different values ​​of contrast, brightness, etc. due to the doctor's operation. This scheme adopts an adaptive threshold method based on the same nystagmus video, which can effectively obtain the pupil grayscale value of the nystagmus video and remove the closed eye frames without manually setting the threshold.

[0032] (4) This scheme uses the optical flow method to process nystagmus videos. Even the displacement change of one pixel can be calculated and displayed, which can effectively detect the occurrence of weak nystagmus. Inputting the image processed by optical flow into the ResNet+LSTM network can effectively identify the horizontal, vertical and rotational nystagmus types. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flow chart of a method for identifying nystagmus type based on the optical flow method according to an embodiment of the present invention.

[0034] Figure 2 It is a schematic diagram of a process of removing closed-eye frames according to an embodiment of the present invention.

[0035] Figure 3 Schematic diagram of using a neural network model to segment an iris image according to an embodiment of the present invention.

[0036] Figure 4 It is a schematic diagram of calculating optical flow using a neural network model and converting it into HSV color space representation according to an embodiment of the present invention.

[0037] Figure 5 It is a schematic diagram of the ResNet+LSTM network model structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present invention is further described in detail below in conjunction with the drawings and specific embodiments.

[0039] Please refer to Figure 1 The method for identifying nystagmus type based on the optical flow method of the embodiment of the present invention comprises the following steps:

[0040] Step 1: Input the patient's nystagmus video captured using an infrared camera , T refers to the total number of frames of the nystagmus video V.

[0041] Step 2: Extract video frames of the input video every other frame If frames are taken continuously, the pupil displacement of weak nystagmus will not be obvious enough. By taking frames at intervals, the pupil displacement of weak nystagmus between frames can be increased, which can also reduce the amount of data.

[0042] Step 3: Remove closed eye frames. The method flow is as follows: Figure 2 As shown. Set a threshold and calculate whether the pupil area exceeds the threshold. If it exceeds the threshold, keep the video frame, otherwise delete the video frame. Specifically, do the following processing for each video:

[0043] (1) Perform an opening operation on the frame, that is, erode first and then dilate, to eliminate the bright spots of light reflected from the pupil and sclera.

[0044] (2) Perform a closing operation on the frame, that is, first dilate and then erode to remove eyebrows and eyelashes.

[0045] (3) Calculate the mean of the lowest grayscale values ​​in all frames of this video l and the minimum average gray value m ,use(l + m ) / 2 is used as the binarization threshold to perform binarization on the video frame.

[0046] (4) Calculate the average value of the pupil area ratio after binarization in all frames of this video. t , and use the average value as the threshold.

[0047] (5) Binarize each frame and calculate the proportion of the pupil in the screen. r .

[0048] (6) If r Greater than t , then the video frame is retained, otherwise it is considered as a closed-eye frame and deleted.

[0049] Step 4: Use the transfer learning method to segment the iris part of the video frame. UNet is selected as the network framework, and ResNet34 is used as the network backbone training model. Based on the ImageNet pre-trained model, the model is trained using the labeled iris dataset to obtain the iris segmentation model (Wang C, Wang Y, Zhang K, et al. NIR irischallenge evaluation in non-cooperative environments: Segmentation and localization[C] / / 2021 IEEE International Joint Conference on Biometrics(IJCB). IEEE, 2021: 1-10.). The final effect of iris segmentation is shown in the figure below. Figure 3 shown.

[0050] Step 5: Calculate the optical flow of consecutive video frames and represent it in HSV color space, where the hue of each pixel represents the direction and the intensity represents the size. The optical flow map is cropped with the pupil center as the image center. The cropped optical flow map is as follows: Figure 4 shown.

[0051] Step 6: Input the optical flow map calculated from the nystagmus video into the trained ResNet+LSTM network model in sequence, and the neural network model automatically identifies the type of nystagmus. The structure of the ResNet+LSTM network model is as follows: Figure 5 As shown in the figure, a series of input image data is extracted with ResNet features, and then sequentially input into the LSTM network to finally obtain the nystagmus type classification result. The training data of the ResNet+LSTM network model is the optical flow map obtained in step 5. A series of optical flow maps are calculated for a nystagmus video, and the label corresponding to each video is the nystagmus type of this video.

[0052] The present invention combines optical flow method and deep learning to identify nystagmus types, and can effectively and accurately identify rotational nystagmus.

[0053] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying nystagmus type based on optical flow method, characterized in that: include: Step 1: Collect the patient's nystagmus video V = {f i |i=0,1,……,T}, T refers to the total number of frames of the nystagmus video V; Step 2: Extract video frames of the nystagmus video every other frame Step 3: Set a threshold and calculate whether the pupil area exceeds the threshold. If it exceeds the threshold, the video frame is retained; otherwise, the video frame is deleted, and the closed-eye frames are removed. Step 4: Use the transfer learning method to segment the iris part of the retained video frame; select UNet as the network framework and ResNet34 as the network backbone training model. Based on the ImageNet pre-trained model, use the labeled iris dataset to train the model to obtain the iris segmentation model; Step 5: Calculate the optical flow of continuous video frames and represent it in HSV color space, where the hue of each pixel represents the direction and the intensity represents the size; take the pupil center as the image center and crop to obtain the optical flow map; Step 6: Build and train the ResNet+LSTM network model, input the obtained optical flow map into the trained ResNet+LSTM network model in sequence, and automatically identify the type of nystagmus through the network model. The training data of the ResNet+LSTM network model is the optical flow map obtained in step 5. A series of optical flow maps are calculated for a nystagmus video, and the label corresponding to each video is the nystagmus type of this video.

2. The method for identifying nystagmus type based on optical flow method according to claim 1, characterized in that: In step 3, the following steps are used to remove closed eye frames: (1) Perform an opening operation on the frame, that is, erode first and then expand, to eliminate the bright spots of the pupil and sclera that reflect the light; (2) Perform a closing operation on the frame, that is, dilate first and then erode to eliminate eyebrows and eyelashes; (3) Calculate the mean l of the lowest grayscale value and the lowest average grayscale value m in all frames of this video, use (l+m) / 2 as the binarization threshold, and perform binarization on the video frame; (4) Calculate the average value t of the pupil area ratio after binarization in all frames of this video, and use the average value as the threshold; (5) Binarize each frame and calculate the ratio r of the pupil to the screen; (6) If r is greater than t, the video frame is retained; otherwise, it is considered as a closed-eye frame and deleted.

Images