A video image-based nystagmus type classification method

Abstract

The present invention discloses a nystagmus type classification method based on video images, comprising the following steps: reading a frame of nystagmus video, combining U-net neural network and threshold segmentation method to segment the pupil in the image; performing circle fitting on the segmented pupil image to obtain the pupil boundary, and determining the pupil center and radius; extracting the part of the iris image outside the pupil boundary; repeating the above steps to process each frame of the video, and forming a sequence of the pupil center and radius extracted from each frame; using the iris image extracted from the first frame as a reference template, calculating the cosine similarity value between the iris image extracted from each frame and the reference template, and forming a cosine similarity sequence; inputting the pupil center sequence, the pupil radius sequence, and the cosine similarity sequence as feature vectors into the classification network to obtain the nystagmus type. The present invention realizes automatic classification of nystagmus types based on video images, and can assist doctors in auxiliary diagnosis of diseases such as vertigo.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing technology and information processing technology, and in particular to a nystagmus type classification method based on video images. Background Art

[0002] Nystagmus is a tremor of the eyeball, which is an involuntary, rhythmic reciprocating movement of the eyeball and a manifestation of vertigo symptoms. Its movement is usually horizontal, but there are also vertical, rotational, and mixed rotational and horizontal or vertical nystagmus. Electronystagmus and videonystagmus are both records of this eye movement, which are of great significance for the diagnosis of vertigo diseases in otolaryngology. Electronystagmus is an indirect acquisition of eye movement parameters by guiding weak corneal and retinal potential differences through epidermal electrodes. However, this method cannot determine the direction of nystagmus and has relatively large noise, which has a great impact on the accuracy and reliability of nystagmus recording. The nystagmus recording method based on video acquisition does not require the use of bioelectric signals, but uses an infrared camera to directly obtain images of eye movement and analyzes the pupil movement trajectory by computer, thereby quantitatively analyzing nystagmus. This simple and convenient method has attracted the attention of many researchers and has gradually replaced the original electronystagmus. At present, the identification and classification of nystagmus types mainly rely on the experience of doctors, so human subjective errors cannot be avoided.

[0003] Computer-aided diagnosis has become one of the important means of diagnosing medical diseases. Computer diagnosis can make full use of the information in the collected nystagmus video to perform accurate qualitative and quantitative analysis. At present, the key to computer nystagmus video analysis technology lies in pupil detection and positioning and the measurement of rotational nystagmus. In nystagmus videos, traditional pupil positioning methods mainly include Hough transform and Canny edge detection. The Hough transform method mainly uses the characteristic that the pupil is generally circular, but this method is not suitable for the case where the eyeball is too much blocked by the upper eyelid or the eye is not fully open. In addition, this method locates the eye from the binary edge image, so its performance is directly related to the threshold used in the binary edge image extraction process. The Canny edge detection method uses the large difference between the color of the pupil and the color of the white of the eye, so this method is not suitable for the case where the color of the pupil and the color of the white of the eye are relatively close in infrared images. In addition, for different video images, the threshold required for edge segmentation needs to change in real time. If a fixed threshold is used, it is difficult to obtain the ideal effect. In addition to pupil detection, the measurement of rotational nystagmus is also a major difficulty. At present, the main method is to estimate the nystagmus rotation trajectory using optical flow, and use the estimated nystagmus rotation trajectory to establish an eye rotation intensity confidence map. The visual motion trend and intensity and other eye rotation features are then used to train a deep learning network to complete the determination of the rotational nystagmus image. However, the optical flow method is computationally intensive and time-consuming, and is not applicable in situations where real-time requirements are stringent. In addition, since changing light can be mistakenly identified as optical flow, this method is sensitive to light, which will affect the nystagmus trajectory recognition effect.

[0004] In addition, the current nystagmus type classification algorithm can only distinguish simple nystagmus types, the features based on manual extraction are subjective and limited, and the axial rotation features of the eyeball are difficult to extract. Therefore, accurately locating the pupil and extracting relevant pupil position information, and accurately extracting the rotation features of rotational nystagmus can lay a solid foundation for nystagmus type classification. Summary of the invention

[0005] The purpose of the present invention is to provide a method for classifying nystagmus types based on video images, which can effectively improve the accuracy of pupil positioning and the calculation accuracy and speed of eye rotation angle calculation, and effectively distinguish the types of nystagmus.

[0006] The specific technical solution for achieving the purpose of the present invention is:

[0007] A method for classifying nystagmus types based on video images, characterized in that the method comprises the following steps:

[0008] (1) Read a frame of the nystagmus video, use the U-net neural network to achieve rough segmentation of the pupil in the image, and use the threshold segmentation method to perform fine segmentation to obtain the pupil image; specifically:

[0009] Read the i-th frame image I of the nystagmus video i ;

[0010] The U-net neural network is used to realize the image I i The pupil is roughly segmented and the pupil binary image B is initially obtained. i ;

[0011] According to the pupil binary image B i , using the 4-connected domain method to determine B i The maximum connected domain of the pupil is obtained by fitting a circle based on the outline of the maximum connected domain and using the minimum circumscribed circle algorithm (minEnclosingCircle). i and the radius r i ;

[0012] With the center O i and 1.3 radius r i Set the region of interest and extract the image I i Middle pupil area image R i , and obtain the green channel image G of the pupil area i ;

[0013] Homomorphic filtering and contrast limited adaptive histogram equalization (CLAHE) are used to enhance the green channel image of the pupil area to obtain G' i ;

[0014] Perform threshold segmentation on the enhanced image to obtain the final pupil binary image B' i .

[0015] (2) Performing circle fitting on the subdivided pupil image to obtain the pupil boundary, and determining the center and radius of the circle, i.e., the pupil center and radius; specifically:

[0016] The pupil binary image B' is determined by using the 4-connected domain method i The maximum connected domain of

[0017] For the B' i The outline of the maximum connected domain is fitted with a circle using the minimum circumscribed circle algorithm (minEnclosingCircle) to obtain the pupil boundary;

[0018] Determine the center O' according to the pupil boundary i (x' i ,y' i ) and radius r' i , that is, the pupil center and radius.

[0019] (3) Extract the part of the iris image outside the pupil boundary; specifically:

[0020] With the pupil boundary and the center O' i (x' i ,y' i ) and 1.2 radius r' i The ring formed by the circles extracts the iris image;

[0021] The circle is divided into four parts by horizontal and vertical lines, and the iris image of the lower right 1 / 4 circle is taken, that is, the partial iris image h i .

[0022] (4) Repeat steps (1) to (3) to process each frame of the nystagmus video, and construct a sequence of the pupil center and radius extracted from each frame of the video; specifically:

[0023] Repeat steps (1)-(3) to process each frame of the nystagmus video, obtain the pupil center and radius of each frame, and extract a partial iris image;

[0024] The pupil center O' extracted from each frame image i (x' i ,y' i ) and the radius are recorded as {x'1,x'2,x'3,…}, {y'1,y'2,y'3,…}, {r'1,r'2,r'3,…} respectively.

[0025] (5) Using the partial iris image extracted from the first frame image as a reference template, respectively calculate the cosine similarity value between the partial iris image extracted from each frame image and the reference template, and form a cosine similarity sequence; specifically:

[0026] The partial iris image h1 extracted from the first frame image is used as a reference template;

[0027] Calculate the partial iris image h extracted from each frame of image i The cosine similarity value c with the reference template h1 i , forming a cosine similarity sequence {c1,c2,c3,…}.

[0028] (6) The pupil center sequence, pupil radius sequence, and cosine similarity sequence are input as feature vectors into the classification network training to obtain a classification model, and the nystagmus type is obtained through the model; specifically:

[0029] The pupil center sequence, pupil radius sequence and cosine similarity sequence extracted from batch nystagmus videos with known nystagmus types were used as input feature vectors, and a classification model was obtained by classification network training. The nystagmus type was derived through the model.

[0030] The technical solution provided by the present invention has the following beneficial effects: a nystagmus type classification method based on video images proposed by the present invention, in pupil positioning, firstly uses a deep learning method to perform coarse pupil segmentation, which can not only avoid the interference caused by eyelashes but also solve the problem that different segmentation thresholds are required in traditional methods due to the influence of factors such as differences in video imaging environments, thereby improving the accuracy and efficiency of coarse pupil segmentation. On this basis, using threshold segmentation to perform fine pupil segmentation can further improve the accuracy of pupil positioning; by calculating the degree of change in the cosine similarity value between the partial iris image extracted from each frame and the reference template image, it is judged whether the eyeball rotates, and this parameter can be used to calculate the eyeball rotation angle later; at the same time, the classification method using machine learning improves the accuracy and speed of nystagmus type judgment, and provides a reference basis for assisting doctors to correctly judge the nystagmus type. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 A flow chart of obtaining pupil images provided by the present invention;

[0033] Figure 3 It is a partial result diagram of the embodiments of the present invention. DETAILED DESCRIPTION

[0034] In order to more clearly illustrate the technical means, technical improvements and beneficial effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0035] Example

[0036] See also Figure 1 , Figure 2 and Figure 3 The present invention provides a method for classifying nystagmus types based on video images, comprising the following specific steps:

[0037] S101: Read a frame of the nystagmus video, use a U-net neural network to achieve coarse segmentation of the pupil in the image, and use a threshold segmentation method to perform fine segmentation to obtain a pupil image.

[0038] This step refers to Figure 2 The specific process is as follows:

[0039] Read the i-th frame image I of the nystagmus video i ,See Figure 3(a);

[0040] The U-net neural network is used to realize the image I i The pupil is roughly segmented and the pupil binary image B is initially obtained. i ,See Figure 3 (b);

[0041] The U-net neural network is mainly divided into 4 encoders, i.e., downsampling processes, and 4 corresponding decoders, i.e., upsampling processes. Downsampling consists of two 3x3 convolutional layers and a 2x2 maximum pooling layer, and upsampling consists of an upsampling convolutional layer (deconvolutional layer) combined with feature splicing and two 3x3 convolutional layers. After image downsampling and upsampling, i.e., combining the low-resolution information and high-resolution information of the image for segmentation, the pupil binary image B is initially obtained. i The present invention mainly utilizes this network structure and the trained weight file to perform rough segmentation on the pupil.

[0042] The pupil binary image B obtained by rough pupil segmentation is found using the 4-connected domain labeling method i The maximum connected domain of the maximum connected domain is obtained, and the circle is fitted according to the outline of the maximum connected domain using the minimum circumscribed circle fitting method (minEnclosingCircle), and its center O is preliminarily obtained. i and the radius r i .

[0043] With the center O i and 1.3 radius r i Set the region of interest and extract the image I i Middle pupil area image R i , and obtain the green channel image G of the pupil area i ,See Figure 3 (c);

[0044] Homomorphic filtering and contrast-limited adaptive histogram equalization are used to enhance the green channel image of the pupil area to obtain G' i ,See Figure 3 (d);

[0045] Perform threshold segmentation on the enhanced image to obtain the final pupil binary image B' i ,See Figure 3 (e).

[0046] The threshold is selected between [10, 20] and is set according to the needs of actual applications. The embodiment of the present invention does not limit this. Here, 10 is taken as an example for description.

[0047] S102: performing circle fitting on the sub-segmented pupil image to obtain the pupil boundary, and determining the circle center and radius, that is, the pupil center and radius.

[0048] The pupil binary image B' obtained in the previous step i Continue to use the 4-connected domain marking method to find B' i The maximum connected domain is obtained, and the minimum circumscribed circle is fitted according to the outline of the maximum connected domain (minEnclosingCircle), that is, pupil fitting, see Figure 3 (f) Get the pupil boundary and determine the pupil center O' i (x' i ,y' i ) and radius r' i .

[0049] S103: Extract the part of the iris image outside the pupil boundary.

[0050] The steps are as follows:

[0051] The pupil boundary obtained after the above segmentation and the center O' i (x' i ,y' i ) and 1.2 radius r' i The ring formed by the circles extracts the iris image;

[0052] The circle is divided into four parts by horizontal and vertical lines, and the iris image of the lower right 1 / 4 circle is taken, that is, the partial iris image h i ,See Figure 3 (g).

[0053] S104: Repeat steps S101-S103 to process each frame of the nystagmus video, and construct a sequence of the pupil center and radius extracted from each frame.

[0054] The steps are as follows:

[0055] Repeat steps S101-S103 to process each frame of the nystagmus video, obtain the pupil center and radius of each frame, and extract a partial iris image;

[0056] The pupil center O' extracted from each frame image i (x' i ,y' i ) and the radius are recorded as {x'1,x'2,x'3,…}, {y'1,y'2,y'3,…}, {r'1,r'2,r'3,…}, see Figure 3 (h)-(j).

[0057] S105: Using the partial iris image extracted from the first frame image as a reference template, respectively calculating the cosine similarity value between the iris image extracted from each frame image and the reference template, and forming a cosine similarity sequence.

[0058] The specific steps are:

[0059] The partial iris image h1 extracted from the first frame image is used as a reference template;

[0060] Calculate the partial iris image h extracted from each frame of image i The cosine similarity value c with the reference template h1 i , forming a cosine similarity sequence {c1, c2, c3, ...}, see Figure 3 (k). The cosine similarity is calculated as follows:

[0061] Assume that the reference template h1 is represented as a vector (a1, a2, …, a n ), n = h1 horizontal pixels × h1 vertical pixels, each frame image extracts a portion of the iris image h i Represented as another vector (b1, b2, …, b n ), n=h i Horizontal pixels ×h i vertical pixels, while h1 and h i The calculation formula of cosine similarity cosθ between them is as follows:

[0062]

[0063] S106: Input the pupil center sequence, the pupil radius sequence, and the cosine similarity sequence as feature vectors into a classification network for training to obtain a classification model, and obtain the nystagmus type through the model.

[0064] The classification model in this step is specifically:

[0065] The pupil center sequence, pupil radius sequence and cosine similarity sequence extracted from batch nystagmus videos with known nystagmus types were used as input feature vectors, and a classification model was obtained by classification network training, and the nystagmus type was obtained through the model.

[0066] The classification network may adopt well-known algorithms such as KNN (K-Nearest Neighbor) algorithm, Random Forest (RF) and other machine learning classification algorithms, which will not be introduced in detail here.

[0067] In summary, the video image-based nystagmus type classification method proposed in the present invention uses a deep learning method to perform pupil coarse segmentation in the pupil positioning method, which can not only avoid the interference caused by eyelashes but also solve the disadvantage that different segmentation thresholds are required in different videos due to factors such as differences in video imaging environments in the traditional method, thereby improving the accuracy and efficiency of pupil coarse segmentation; the present invention uses threshold segmentation to perform pupil fine segmentation on the basis of pupil coarse segmentation to further improve the accuracy of pupil positioning; by calculating the degree of change in the cosine similarity value between the partial iris image extracted from each frame and the reference template image, it can be judged whether the eyeball rotates, and the cosine similarity parameter can be further used to calculate the rotation angle later; at the same time, the classification method using machine learning can improve the accuracy and efficiency of judging the type of nystagmus.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for classifying nystagmus types based on video images, characterized in that: The method comprises the following steps: (1) Read a frame of the nystagmus video, use the U-net neural network to achieve rough segmentation of the pupil in the image, and use the threshold segmentation method to perform fine segmentation to obtain the pupil image; (2) Perform circle fitting on the finely segmented pupil image to obtain the pupil boundary and determine the center and radius of the circle, i.e., the pupil center and radius; (3) Extract the part of the iris image outside the pupil boundary; (4) Repeat steps (1) to (3) to process each frame of the nystagmus video, and construct a sequence of the pupil center and radius extracted from each frame of the video; (5) Using the partial iris image extracted from the first frame image as a reference template, calculate the cosine similarity value between the partial iris image extracted from each frame image and the reference template, and form a cosine similarity sequence; (6) The pupil center sequence, pupil radius sequence, and cosine similarity sequence are input as feature vectors into the classification network training to obtain a classification model, and the nystagmus type is obtained through the model.

2. The method for classifying nystagmus types based on video images according to claim 1, characterized in that: The step (1) is specifically as follows: Read the nystagmus video i Frame Image I i ; The image is realized using U-net neural network I i The pupil is roughly segmented and the pupil binary image is initially obtained. B i ; According to the pupil binary image B i , using the 4-connected domain method to determine B i The center of the pupil is initially obtained by fitting a circle based on the outline of the maximum connected domain and the minimum circumscribed circle algorithm. O i And the radius r i ; With the center of the circle O i and 1.3 radius r i Set the region of interest and extract the image I i Middle pupil area image R i , and obtain the green channel image of the pupil area G i ; Homomorphic filtering and contrast-limited adaptive histogram equalization are used to enhance the green channel image of the pupil area to obtain G' i ; Perform threshold segmentation on the enhanced image to obtain the final pupil binary image B' i .

3. The method for classifying nystagmus types based on video images according to claim 1, characterized in that: The step (2) is specifically as follows: Using 4-connected domain method to judge pupil binary image B' i The maximum connected domain of Regarding the B' i Perform circle fitting using the minimum circumscribed circle algorithm on the contour of the maximum connected domain to obtain the pupil boundary; Determine the center of the circle based on the pupil boundary O ' i ( x ' i , y ' i ) and radius r ' i , that is, the pupil center and radius.

4. The method for classifying nystagmus types based on video images according to claim 1, characterized in that: The step (3) of extracting the portion of the iris image outside the pupil boundary is specifically as follows: The pupil boundary and the center of the circle O ' i ( x ' i , y ' i ) and 1.2 radius r ' i The ring formed by the circles of extracts the iris image; Divide the ring into four parts by horizontal and vertical lines, and take the iris image of the lower right 1 / 4 ring, i.e. the partial iris image h i .

5. The method for classifying nystagmus types based on video images according to claim 1, characterized in that: The step (4) is specifically as follows: Repeat steps (1)-(3) to process each frame of the nystagmus video, obtain the pupil center and radius of each frame, and extract a partial iris image; The pupil center extracted from each frame O ' i ( x ' i , y ' i ) and the radius are respectively denoted as { x' 1 , x' 2 , x' 3 , …}, { y' 1 , y' 2 , y' 3 , …}, { r' 1 , r' 2 , r' 3 , …}.

6. The method for classifying nystagmus types based on video images according to claim 1, characterized in that: The step (5) is specifically as follows: Partial iris image extracted from the first frame image h 1 As a reference template; Calculate the partial iris image extracted from each frame of image separately h i With the reference template h 1 The cosine similarity value of c i , forming a cosine similarity sequence { c 1 , c 2 , c 3 , …}.

7. The method for classifying nystagmus types based on video images according to claim 1, characterized in that: The step (6) is specifically as follows: The pupil center sequence, pupil radius sequence and cosine similarity sequence extracted from batch nystagmus videos with known nystagmus types were used as input feature vectors, and a classification model was obtained by classification network training. The nystagmus type was derived through the model.

Images