Method, System, and Storage Medium for Evaluating Eye Movement

Abstract

The present invention relates to a method, a system, and a storage medium for evaluating eye movement. The evaluation method includes: obtaining a first eye position image from the front view of the user at the front position of the eye in a near-infrared light field of 700 to 1200 nm by photographing; obtaining a second eye position image and a third eye position image that are photographed at the front position of the eye and in which the user's eyeball moves to the limit position along the measurement target direction; and comparing the first eye position image with the second eye position image and the third eye position image respectively to calculate the movement angle of the eye movement. In the technical means according to the present invention, by photographing an eye image in a near-infrared light field, the iris and the pupil are effectively distinguished to accurately identify the pupil edge, the movement angle of the eyeball is calculated by comparing different eye position images, and the calculation accuracy of the movement angle is further improved by compensation, so as to accurately calculate the movement angle of the eyeball in each measurement target direction.

Description

Technical Field

[0001] This application belongs to the technical field of eye detection, and particularly relates to a method, apparatus, and storage medium for evaluating the eye movement of the eyeball.

Background Art

[0002] The oculomotor nerve, trochlear nerve, and abducens nerve have the function of controlling the extraocular muscle movement of the eyeball and are called the ocular motor nerves. When the above nerves or nuclei are damaged alone or in combination, ophthalmoplegia or diplopia may occur. When completely damaged, all extraocular muscles may be paralyzed and the eyeball may be fixed and immobile. Extraocular muscle paralysis caused by extraocular muscle injury, infection, or myopathy causes ophthalmoplegia and is clinically collectively referred to as ocular motility disorder. Since ocular motility disorder may also be related to internal medicine diseases such as orbital diseases, diabetes, and neuroinflammation, how to evaluate the rotational ability of the single eye and both eyes has important significance for eye detection.

[0003] In the prior art, the detection of eye movement usually uses a method in which a doctor manually measures. The specific detection method is as follows: 1) For the monocular eye movement examination, cover the contralateral eye, and from the first eye position, let the patient fixate on a penlight and instruct the patient to move along the diagnostic direction of fixation. 2) For the binocular eye movement examination, from the first eye position, let the patient fixate on a penlight and instruct the patient to move along the diagnostic direction of fixation. The binocular movement examination can evaluate the relative position of the two eyes during eye movement and obtain different information from the monocular examination. However, the above detection method lacks objective data and cannot record the changes of the patient each time, and cannot form continuous data.

[0004] In addition, the prior art further discloses a method for automatically detecting eye movement activity. As described in the Chinese invention patent "Eye Movement Activity Detector" (Application No.: 202011260253.4, Publication Date: February 19, 2021), the head of the patient is fixed by a head fixation frame, an indicator lamp is provided in front of the head fixation frame, the indicator lamp emits a visible light beam in a direction away from the eyes of the subject, the visible light beam extends beyond the static visual field of the subject, an imaging lens is provided in front of the head fixation frame, the imaging lens captures an image when the subject's eyeball rotates along the direction of the visible light beam, and the eye movement status of the subject is obtained from the eye image captured by the imaging lens. However, in the above technical means, since no specific distinction and processing are performed on the eye image, the detection accuracy of the eye movement status is insufficient. As described in the Chinese patent "Computer-Based Eye Movement Distance and Binocular Movement Consistency Deviation Detection Device and Method" (Application No.: 201710054692.1, Publication Date: June 13, 2017), the corneal limbus in the reference photo and each test photo transmitted from the camera is extracted by a corneal limbus extraction unit, and the movement distance in each direction of the eyeball is calculated based on the points extracted from the corneal limbus, thereby calculating the binocular movement consistency deviation. However, since the corneal limbus is the transition zone between the cornea and the sclera, the boundary of the extracted corneal limbus is not clear enough, which also affects the accuracy of the final result.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The purpose of this application is to provide a method and system for evaluating eye movement activity that can improve the evaluation accuracy of eye movement activity and eliminate errors in the measurement process. By collecting multiple different eye position images of the user in the near-infrared light field and analyzing the collected multiple different eye position images, the movement angle of the eye movement activity is calculated, and an accurate evaluation of the eye movement activity is realized.

Means for Solving the Problems

[0006] To achieve the above object, in one aspect, the method for evaluating eye movement degree according to the present application is as follows: In a near-infrared light field of 700 to 1200 nm, step S1 of capturing and obtaining a first eye position image from the front view of the user at the front position of the eyeball; Step S2 of capturing an image at the front position of the eyeball and obtaining a second eye position image and a third eye position image in which the user's eyeball moves to the limit position along the measurement target direction; Step S3 of comparing the first eye position image with the second eye position image and the third eye position image respectively, and calculating the activity angle of the eye movement degree, is included.

[0007] Furthermore, the measurement target directions include upward, downward, inward, outward, inward and upward, outward and upward, inward and downward, outward and downward of the eyeball.

[0008] Furthermore, step S3 includes a step of dividing the pupil from the first eye position image, the second eye position image, and the third eye position image using a convolutional neural network; Taking out the center C1 of the first eye position image and the center C2 of the second eye position image or the third eye position image, and connecting the centers C1 and C2 with a straight line; Finding the edge points P1 and P2 before and after the movement of the same point of the pupil of the first eye position image and the second eye position image or the third eye position image along the straight line; Connecting the edge points P1 and P2 with a straight line P1P2; Dividing the straight line P1P2 into two lines, a straight line A and a straight line B, by a straight line that is perpendicular to the straight line P1P2 and passes through the center C1; Obtaining the central angle α corresponding to the straight line A and the central angle β corresponding to the straight line B based on the eyeball radius r, and adding the central angle α and the central angle β to obtain the activity angle, is included.

[0009] Furthermore, the width of the pupil change, the width of the displacement of the eyeball center, and the activity angle are compensated by corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image.

[0010] Furthermore, in the step of compensating the activity angle according to the width of pupil change in the first eye position image, the second eye position image, or the third eye position image, the position of the edge point P2 is compensated by obtaining the width of pupil change in the first eye position image, the second eye position image, or the third eye position image.

[0011] Specifically, calculate the width that changes along the direction orthogonal to the measurement target direction of the pupil diameter in the first eye position image, the second eye position image, or the third eye position image, and proportionally compensate the position of the edge point P2 according to the width.

[0012] Furthermore, in the step of compensating the activity angle according to the width of displacement of the eye center in the first eye position image, the second eye position image, or the third eye position image, obtain the superimposed image formed by different displacements of the eye center in the first eye position image, the second eye position image, or the third eye position image, and translate the superimposed image in the reverse direction to offset the influence of the displacement on the activity angle.

[0013] Furthermore, in the step of compensating the activity angle by corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image, the angle compensated by corneal refractive error compensation and the activity angle are in a linear relationship.

[0014] In one aspect, the apparatus for evaluating eye movement degree according to the present application includes a first acquisition module that acquires a first eye position image from the front view of the user at the front position of the eye in a near-infrared light field of 700 to 1200 nm by photographing; a second acquisition module that photographs at the front position of the eye and acquires a second eye position image and a third eye position image when the user's eye moves to the limit position along the measurement target direction; an image processing module that compares the first eye position image with the second eye position image and the third eye position image respectively, and calculates the activity angle of the eye movement degree.

[0015] In one aspect, the computer-readable storage medium according to the present application stores at least one program code that is loaded and executed by a processor to implement the operations performed by the method for evaluating the eye movement degree.

Advantages of the Invention

[0016] The technical means of the present application has at least the following beneficial effects compared with the prior art.

[0017] By capturing an eye image in a near-infrared light field, the iris and pupil can be effectively distinguished, the pupil edge can be accurately identified, and by calculating the movement angle of the eyeball using the displacement of the pupil edge points, the drawback of large errors in calculating the movement angle of the eyeball using the iris edge in the prior art can be avoided.

[0018] Furthermore, in the present application, by compensating the movement angle using the width of the pupil change before and after eye movement, the width of the displacement of the eye center, and corneal refractive error compensation, the calculation accuracy for the movement angle is further improved, and by accurately calculating the movement angle in each measurement target direction of the eyeball, an accurate determination basis can be provided for eye detection.

Brief Description of the Drawings

[0019] To more clearly explain the technical means of the embodiments of the present application, the drawings that need to be used in the following description of the embodiments of the present application or the prior art will be briefly described.

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0021] Hereinafter, with reference to the drawings and examples, the specific embodiments of the present application will be described in more detail. The following examples are for explaining the present application and do not limit the scope of the present application.

[0022] In one aspect, as shown in FIG. 1, the method for evaluating eye activity according to Embodiment 1 of the present invention includes Step S1 of obtaining a first eye position image taken in front of the user's eye from the user's frontal view in a near-infrared light field of 700 to 1200 nm, Step S2 of obtaining a second eye position image and a third eye position image in which the eyeball photographed in front of the user's eye moves to the limit position along the measurement target direction, Step S3 of comparing the first eye position image with the second eye position image and the third eye position image respectively and calculating the activity angle of the eye activity.

[0023] In a specific step, when obtaining an eye image using an acquisition device, first fix the user's head, for example, realize the fixation of the user's head by means of a forehead rest, a tray, and a canthus fixing point. After being fixed, photograph a first eye position image in which the user's eye looks straight ahead in an infrared light field of 700 to 1200, and a second eye position image and a third eye position image in which the user rotates the eyeball to the limit position according to the indication direction of the indication lamp in each measurement target direction.

[0024] Note that the eye position refers to the position of the eyeball in an ophthalmological examination and is divided into the primary eye position, the secondary eye position, and the tertiary eye position. The primary eye position refers to the eye position when both eyes are looking straight at an infinitely distant point on the horizontal plane. The secondary eye position refers to the eye position when the eyeballs rotate upward, downward, inward, and outward. The tertiary eye position refers to the eye position when the eyeballs rotate obliquely upward, downward, inward, and outward, that is, the eye position when rotating upward, downward, inward, and outward toward the nose and temples. Accordingly, the primary, secondary, and tertiary eye position images refer to the images taken when the eyeballs are in their respective eye positions.

[0025] As shown in Fig. 2, in the wavelength band of 400 to 700 nm of normal visible light, the colors of different parts of the eye, namely the pupil, iris, and sclera, hardly affect imaging, and due to the gradient structure of the corneal limbus at the contact part between the iris and the sclera, the center of the eyeball cannot be accurately identified. Since the absorption peak of human melanin pigment occurs at about 335 nm and it is hardly absorbed at all in the wavelength band exceeding 700 nm, and the reflectance of the iris is quite stable within the near-infrared band exceeding 700 nm, by using a near-infrared light field, the sclera, iris, and pupil boundaries can be well distinguished, thereby better improving the accuracy and stability of the algorithm.

[0026] As shown in Fig. 3, the measurement target directions include upward, downward, inward, outward, inward-upward, outward-upward, inward-downward, and outward-downward of the eyeball. Each eye measures the movement angle in 8 measurement target directions respectively. Each time one eye is measured, the other eye is shielded, and the left and right eyes are respectively instructed to repeat the rotation in each of the above directions using the indicator lamps provided in each measurement target direction, and a plurality of secondary eye position images and tertiary eye position images are obtained. By comparing them with the reference primary eye position image respectively, the movement angle of the eyeball movement is calculated.

[0027] Note that the above movement angle refers to the maximum angle of rotation around the center of the eyeball of the eyeball body before and after eyeball movement.

[0028] (Example 2) As shown in FIG. 4, based on Example 1 of the above method, in this Example 2, the calculation method of the movement angle is further limited. Specifically, the method includes: using a convolutional neural network to segment the pupil from the first eye position image and the second eye position image where the eyeball moves to the limit position downward; extracting the center C1 of the first eye position image and the center C2 of the second eye position image, and connecting the centers C1 and C2 with a straight line; finding the edge points P1 and P2 before and after the movement of the same point of the pupil of the first eye position image and the second eye position image along the straight line; connecting the edge points P1 and P2 with a straight line P1P2; dividing the straight line P1P2 into two straight lines, a straight line A and a straight line B, by a straight line that is perpendicular to the straight line P1P2 and passes through the center C1; obtaining the central angle α corresponding to the straight line A and the central angle β corresponding to the straight line B according to the eyeball radius r, and adding the central angle α and the central angle β to obtain the movement angle θ.

[0029] In the above calculation method, according to simple analytic geometry relationships, by calculating the central angle corresponding to the arc length on the spherical surface, the movement angle before and after the eyeball movement can be obtained. Since the eyeball has a regular spherical structure, for each measurement target direction, such as inward and upward, inward and downward, outward and upward, outward and downward when the eyeball moves obliquely, after obtaining the first, second, and third eye position images, the calculation of the above movement angle can be realized through the image and data processing steps.

[0030] (Example 3) As shown in FIGS. 5 to 7, based on Example 2 of the above method, in this Example 3, by further limiting the compensation method of the movement angle, the calculation accuracy of the movement angle can be improved. In actual measurement, it is found that the changes in the pupil before and after the eyeball movement, the displacement of the eyeball center, and the influence of corneal refraction all bring errors to the calculation of the movement angle. Hereinafter, the compensation methods for the above several errors will be specifically described.

[0031] First, as shown in FIG. 5, when the rotation of the eyeball deviates from the front view, the pupil imaging becomes elliptical, the deformation direction is in the rotation direction, and the imaging in the direction orthogonal to the rotation direction is not affected. Therefore, if there is a change in the diameter of the pupil in the direction orthogonal to the rotation direction, it is a change from the pupil. When the diameter in the front view is smaller than the diameter in the oblique view, the pupil expands when it rotates, and when the diameter in the front view is larger than the diameter in the oblique view, the pupil contracts when it rotates. For example, when the ellipse orthogonal to the rotation direction shrinks, it is due to the contraction of the pupil. Connect the two divided centers of the circle with a straight line, take out the diameters orthogonal to the straight line in the two ellipses, calculate the ratio of pupil contraction using the above two diameters, and compensate the position of the edge point P2 according to the width of the pupil change by proportionally correcting the edge points of the ellipse for calculation on the straight line. Thereby, by removing the influence on the calculation accuracy of the movement angle due to pupil contraction, the movement angle is compensated according to the width of the pupil change in the first eye position image, the second eye position image, or the third eye position image.

[0032] As shown in FIG. 6, when the eye rotates, the eye does not simply float and move in space. Since there are soft tissues all around and the activities of the eye muscles are non-uniform, different displacements of the eye center are caused by the rotation of the eyeball in different directions. The larger the rotation angle of the eyeball, the larger the displacement of the eye center. The influence of the displacement of the eye center on the algorithm corresponds to the occurrence of a deviation when superimposing and aligning the imaging from the front view and the side view. As a result, the calculated movement angle is either too large or too small. The compensation method is to intentionally shift in the opposite direction when superimposing the imaging after knowing how much the eye center has been displaced, so as to offset the influence of the displacement.

[0033] As shown in FIG. 7, the left diagram of FIG. 7 shows the change diagram before and after the rotation of the eyeball in the inner and outer directions photographed from the top of the human body, and the right diagram of FIG. 7 shows the change diagram before and after the rotation of the eyeball in the up and down directions photographed from the top of the human body. Since the displacement of the center of the eyeball in the front-rear direction before and after the rotation of the eyeball does not affect the imaging photographed by the frontal camera, it can be ignored. Only the image projected by the eyeball on the cross-section is obtained, and the following statistical display of the relationship between the direction of the eyeball movement and the center displacement in the cross-section of the eyeball is performed.

[0034] The displacement of the eyeball rotating 30° inward corresponds to a displacement of 0.69 mm inward from the center of the eyeball. The displacement of the eyeball rotating 30° outward corresponds to a displacement of 0.45 mm outward from the center of the eyeball. The displacement of the eyeball rotating 20° upward corresponds to a displacement of 0.43 mm downward from the center of the eyeball. The displacement of the eyeball rotating 20° downward corresponds to a displacement of 0.43 mm upward from the center of the eyeball. Also, the displacement of the center of the eyeball and the angle of eyeball rotation are in a linear rule. When the angle of eyeball rotation is 0° (frontal view), the displacement of the center of the eyeball is 0 mm. According to the relationship between the above-mentioned displacement and angle statistically determined in advance, the position of the eyeball center C2 in the calculation of the activity angle can be corrected, and thereby, the activity angle is compensated by the width of the displacement of the center of the eyeball in the first eye position image, the second eye position image or the third eye position image.

[0035] As shown in FIG. 8, when calculating the activity angle, the edge points of the pupil are affected by the refraction of the corneal crystal, and the edge points of the pupil on the screen are not the actual positions in reality. The edge points of the iris are not affected by this because they do not pass through the cornea. Two types of algorithms, namely the edge points of the pupil and the edge points of the iris, can be used for the same photo to obtain different activity angles. When calculating using the edge points of the pupil, there is a corneal refraction error. As can be seen from experimental measurements, since the difference in the activity angles calculated by the pupil and the iris is in a high-degree linear rule, the calculation formula for the activity angle to compensate for the corneal refraction error is as follows.

[0036] Compensation of activity angle θ = 0.13134 × activity angle θ + 0.52704 The corrected movement angle θ = movement angle θ + compensation for movement angle θ In the above formula, the movement angle θ is the movement angle calculated before corneal refractive error compensation.

[0037] After performing corresponding compensation considering the influence of the change in pupil, the displacement of the eye center, and the influence on the movement angle due to corneal refraction before and after the above eye movement, the error in the movement angle calculation can be removed, and finally an accurate movement angle can be obtained.

[0038] In one aspect, as shown in FIG. 10, the eye movement degree evaluation apparatus according to the present application includes a first acquisition module 1001, a second acquisition module 1002, and an image processing module 1003.

[0039] The first acquisition module 1001 captures and acquires a first eye position image from the front view of the user at the front position of the eye in a near-infrared light field of 700 to 1200 nm. The second acquisition module 1002 captures images at the front position of the eye and acquires a second eye position image and a third eye position image in which the user's eye moves to the limit position along the measurement target direction. The image processing module 1003 compares the first eye position image with the second eye position image and the third eye position image respectively, and calculates the movement angle of the eye movement degree.

[0040] In one aspect, at least one program code for realizing the operations executed by the evaluation method of the eye movement degree is stored in the computer-readable storage medium according to the present application, which is loaded and executed by a processor.

[0041] Regarding the eye movement degree evaluation apparatus in the above embodiment, the specific methods for each module to execute operations have been described in detail in the method embodiments, and the relevant content refers to part of the description of the method embodiments.

[0042] In an exemplary embodiment, there is further provided a computer-readable storage medium including a memory in which at least one program code is stored, which is loaded and executed by a processor to implement the method for evaluating eye movement activity in the above embodiment. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, optical data storage device, or the like.

[0043] Those skilled in the art will understand that all or part of the steps of the above embodiment may be completed by hardware, or may be completed by hardware related to at least one program code, and the program may be stored in a computer-readable storage medium, and the above-described storage medium may be a read-only memory, magnetic disk, optical disk, or the like.

[0044] The above description is only a preferred embodiment of the present application and does not limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included within the protection scope of the present application.

Claims

**Claim 1** In a near-infrared light field of 700 to 1200 nm, step S1 of capturing and obtaining a first eye position image from the front view of the user at a front position of the eyeball; Step S2 of capturing at a front position of the eyeball and obtaining a second eye position image and a third eye position image in which the user's eyeball has moved to a limit position along the measurement target direction; Including step S3 of comparing the first eye position image with the second eye position image and the third eye position image respectively and calculating the activity angle of the eyeball activity. A method for evaluating the activity of the eyeball, characterized in that. **Claim 2** The measurement target direction includes upward, downward, inward, outward, inward-upward, outward-upward, inward-downward, and outward-downward of the eyeball. The method for evaluating the activity of the eyeball according to claim 1, characterized in that. **Claim 3** The step S3 includes: A step of dividing the pupil from the first eye position image, the second eye position image, and the third eye position image using a convolutional neural network; The center C of the first eye position image １ , the center C of the second eye position image or the third eye position image ２ is extracted, and the step of connecting the center C １ and the center C ２ with a straight line The edge points P before and after the movement of the same point of the pupil of the first eye position image and the second or third eye position image along the straight line １ , P ２ finding steps; the edge point P １ and P ２ are connected to a straight line P １ P ２ in a connecting step; the straight line P １ P ２ is perpendicular to and is divided into two lines, line A and line B, by a straight line passing through the center of the circle C １ through the straight line P １ P ２ in a step of dividing it into two lines, line A and line B Obtaining a central angle α corresponding to the straight line A and a central angle β corresponding to the straight line B based on the eyeball radius r, and adding the central angle α and the central angle β to obtain the activity angle. The method for evaluating the activity of the eyeball according to claim 1 or 2, characterized in that. **Claim 4** Compensating the activity angle by the width of the pupil change, the width of the displacement of the eyeball center, and the corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image. The method for evaluating the activity of the eyeball according to claim 3, characterized in that. **Claim 5** In the step of compensating the movement angle according to the width of the pupil change in the first eye position image, the second eye position image, or the third eye position image, the position of the edge point P is compensated by obtaining the width of the pupil change in the first eye position image, the second eye position image, or the third eye position image. ２ ​ The method for evaluating the activity of the eyeball according to claim 4, characterized in that. **Claim 6** Calculate the width that changes along the direction orthogonal to the measurement target direction of the pupil diameter in the first eye position image, the second eye position image, or the third eye position image, and compensate the position of the edge point P ２ proportionally according to the width. The method for evaluating the activity of the eyeball according to claim 5, characterized in that. **Claim 7** In the step of compensating the activity angle by the width of the displacement of the eyeball center in the first eye position image, the second eye position image, or the third eye position image, obtaining a superimposed image formed by different displacements of the eyeball center in the first eye position image, the second eye position image, or the third eye position image, and translating the superimposed image in the reverse direction to offset the influence of the displacement on the activity angle. The method for evaluating the activity of the eyeball according to claim 4, characterized in that. **Claim 8** In the step of compensating the activity angle by corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image, the angle compensated by corneal refractive error compensation and the activity angle are in a linear relationship. The method for evaluating the activity of the eyeball according to claim 4, characterized in that. **Claim 9** A first acquisition module that acquires, by photographing, a first eye position image from the front view of the user at a forward position of the eyeball in a near-infrared light field of 700 to 1200 nm. A second acquisition module that photographs at a forward position of the eyeball and acquires a second eye position image and a third eye position image in which the user's eyeball has moved to a limit position along the measurement target direction. An image processing module that compares the first eye position image with the second eye position image and the third eye position image respectively, and calculates the activity angle of the eyeball activity. An evaluation system for eyeball activity, characterized by the above.

10. At least one program code that is loaded and executed by a processor to implement the evaluation method for eyeball activity according to any one of Claims 1 to 8 is stored. A computer-readable storage medium, characterized by the above.

Images