Eye activity evaluation method, system, and storage medium

Abstract

The present invention relates to a method, a system, and a storage medium for evaluating eye movement. The evaluation method includes: obtaining a first eye position image from the front view of the user at the front position of the eye in a near-infrared light field of 700 to 1200 nm by photographing; obtaining a second eye position image and a third eye position image that are photographed at the front position of the eye and in which the user's eyeball moves to the limit position along the measurement target direction; and comparing the first eye position image with the second eye position image and the third eye position image respectively to calculate the movement angle of the eye movement. In the technical means according to the present invention, by photographing an eye image in a near-infrared light field, the iris and the pupil are effectively distinguished to accurately identify the pupil edge, the movement angle of the eyeball is calculated by comparing different eye position images, and the calculation accuracy of the movement angle is further improved by compensation, so as to accurately calculate the movement angle of the eyeball in each measurement target direction.

Description

[Technical Field]

[0001] The present application relates to the technical field of eye detection, and in particular to a method, device and storage medium for assessing eye activity. [Background technology]

[0002] The oculomotor nerve, trochlear nerve, and abducens nerve control the extraocular muscle movements of the eye and are referred to as oculomotor nerves. Damage to any of these nerves or nerve nuclei, either singly or in combination, can result in ocular akinesia or diplopia. Complete damage can result in paralysis of all extraocular muscles, resulting in the eyeball becoming immobile. Extraocular muscle paralysis caused by extraocular muscle injury, infection, or muscle disease can result in ocular akinesia, clinically referred to as ocular movement disorder. Ocular movement disorders can also be associated with medical conditions such as orbital disease, diabetes, and neuroinflammation. Therefore, how to assess monocular and binocular eye rotation ability is of great importance for ocular detection.

[0003] In the prior art, eye activity detection is typically performed manually by a physician. Specific detection methods include: 1) monocular eye movement testing, in which the contralateral eye is covered, and the patient is instructed to gaze at a flashlight from a first eye position and move along the diagnostic gaze direction; and 2) binocular eye movement testing, in which the patient is instructed to gaze at a flashlight from a first eye position and move along the diagnostic gaze direction. Binocular eye movement testing evaluates the relative position of the two eyes during eye movement and can obtain different information from monocular testing. However, these detection methods lack objective data, cannot record changes in the patient's movements, and cannot form continuous data.

[0004] The prior art also discloses an automatic method for detecting eye activity. For example, the Chinese invention patent "Eye Activity Detector" (Application No. 202011260253.4, Publication Date: February 19, 2021) describes a method in which a patient's head is fixed using a head fixation frame, an indicator lamp is provided in front of the head fixation frame, the indicator lamp emits a visible light beam in a direction away from the subject's eye, the visible light beam extends outside the subject's visual field, and an imaging lens is provided in front of the head fixation frame, which captures images of the subject's eye as it rotates along the direction of the visible light beam. The eye activity status of the subject is obtained from the eye image captured by the imaging lens. However, because the above technical means does not specifically distinguish or process the eye image, the accuracy of detecting eye activity status is insufficient. As described in the Chinese patent "Device and method for detecting eye movement distance and binocular movement consistency deviation based on a computer" (Application number: 201710054692.1, publication date: June 13, 2017), a limbus extraction unit extracts the corneal limbus from the reference photograph and each test photograph sent from the camera, and calculates the movement distance in each direction of the eyeball using the points extracted from the limbus, thereby calculating the binocular movement consistency deviation. However, because the limbus is the transition zone between the cornea and the sclera, the boundary of the extracted limbus is not clear enough, which also affects the accuracy of the final result. Summary of the Invention [Problem to be solved by the invention]

[0005] The present application aims to provide a method and system for assessing eye activity that improves the accuracy of eye activity assessment and eliminates errors in the measurement process. By collecting multiple images of a user's eye positions in a near-infrared light field and analyzing the collected multiple images of different eye positions, the activity angle of the eye activity can be calculated, thereby achieving accurate assessment of eye activity. [Means for solving the problem]

[0006] In order to achieve the above object, in one aspect, a method for evaluating eye activity according to the present application includes: Step S1: capturing and acquiring a first eye position image from a front view of a user at a position in front of the eyeball in a near-infrared light field of 700 to 1200 nm; Step S2: taking a photograph at a position in front of the eyeball and acquiring a second eye position image and a third eye position image in which the user's eyeball moves to a limit position along the measurement target direction; and step S3 of comparing the first eye position image with the second eye position image and the third eye position image, respectively, to calculate an activity angle of eye activity.

[0007] Furthermore, the measurement target direction includes upward, downward, inward, outward, inward-upward, outward-upward, inward-downward, and outward-downward directions of the eyeball.

[0008] Furthermore, step S3 includes a step of segmenting pupils from the first eye position image, the second eye position image, and the third eye position image using a convolutional neural network; extracting a circle center C1 of the first eye-position image and a circle center C2 of the second eye-position image or the third eye-position image, and connecting the circle center C1 and the circle center C2 with a straight line; Finding edge points P1 and P2 along the straight line before and after the movement of the same point of the pupil in the first eye position image and the second eye position image or the third eye position image; connecting the edge points P1 and P2 to a line P1P2; dividing the straight line P1P2 into two straight lines, a straight line A and a straight line B, by a straight line that is perpendicular to the straight line P1P2 and passes through the center C1 of the circle; and obtaining a circular angle α corresponding to the line A and a circular angle β corresponding to the line B based on the eyeball radius r, and adding the circular angle α and the circular angle β to obtain the activity angle.

[0009] Furthermore, the action angle is compensated for by the range of pupil change, the range of displacement of the center of the eyeball, and corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image.

[0010] Furthermore, in the step of compensating the activity angle based on the width of pupil change in the first eye position image, the second eye position image, or the third eye position image, the position of the edge point P2 is compensated by obtaining the width of pupil change in the first eye position image, the second eye position image, or the third eye position image.

[0011] Specifically, the width that varies along a direction perpendicular to the measurement direction of the pupil diameter in the first eye position image, the second eye position image, or the third eye position image is calculated, and the position of the edge point P2 is compensated proportionally using the width.

[0012] Furthermore, in the step of compensating the activity angle based on the range of displacement of the eyeball center in the first eye position image, the second eye position image, or the third eye position image, a superimposed image formed by different displacements of the eyeball center in the first eye position image, the second eye position image, or the third eye position image is obtained, and the superimposed image is translated in the opposite direction to cancel out the effect of the displacement on the activity angle.

[0013] Furthermore, in the step of compensating the activity angle by corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image, the angle compensated by corneal refractive error compensation and the activity angle are in a linear relationship.

[0014] In one aspect, the eye activity evaluation device according to the present application comprises: a first acquisition module that captures and acquires a first eye position image from a front view of the user at a position in front of the eyeball in a near-infrared light field of 700 to 1200 nm; a second acquisition module for acquiring a second eye position image and a third eye position image taken at a position in front of the eyeball and obtained when the user's eyeball moves to a limit position along the measurement target direction; and an image processing module that compares the first eye position image with the second eye position image and the third eye position image, respectively, and calculates an activity angle of eye activity.

[0015] In one aspect, a computer-readable storage medium according to the present application stores at least one program code that, when loaded and executed by a processor, performs the operations performed by the method for assessing eye activity. [Effects of the Invention]

[0016] The technical means of the present application has at least the following beneficial effects compared to the prior art.

[0017] By capturing an image of the eye in a near-infrared light field, the iris and pupil can be effectively distinguished and the pupil edge can be accurately identified. By calculating the eye activity angle using the displacement of the pupil edge point, the drawback of the prior art, which is that the eye activity angle is calculated using the iris edge, which results in a large error, can be avoided.

[0018] Furthermore, in the present application, the activity angle is compensated for using the range of pupil change before and after eye movement, the range of displacement of the eye center, and corneal refractive error compensation, thereby further improving the calculation accuracy of the activity angle, and accurately calculating the activity angle in each measurement target direction of the eye, thereby providing a more accurate basis for judgment through eye detection. [Brief explanation of the drawings]

[0019] In order to more clearly describe the technical solutions of the embodiments of the present application, the following briefly describes the drawings that need to be used in the description of the embodiments of the present application or the prior art.

[0020] [Figure 1] 1 is a flowchart of a method for evaluating eye activity according to the present application. [Figure 2] Images of the eye taken using visible and near-infrared light. [Figure 3] This is a superimposed image of the first-eye-position image and the second-eye-position image or the third-eye-position image taken in each direction of the measurement object. [Figure 4] Calculation diagram of eye activity angle. [Figure 5] This is a diagram showing pupil changes before and after eye movement. [Figure 6] This is a superimposed image of the eye image before and after eye movement. [Figure 7] This is a diagram showing the displacement of the eye center before and after eye movement. [Figure 8] This is a refraction diagram of pupil imaging by corneal crystals. [Figure 9] The activation angle is calculated using the pupil edge and the iris edge. [Figure 10] FIG. 1 is a schematic diagram illustrating the configuration of an eye activity evaluation device. DETAILED DESCRIPTION OF THE INVENTION

[0021] Specific embodiments of the present application will be described in more detail below with reference to the drawings and examples. The following examples are intended to illustrate the present application and are not intended to limit the scope of the present application.

[0022] In one aspect, as shown in FIG. 1, the method for evaluating eye activity according to the first embodiment includes: Step S1 of acquiring a first eye position image captured in front of the user's eye from the user's front view in a near-infrared light field of 700 to 1200 nm; Step S2: acquiring a second eye position image and a third eye position image in which the eyeball photographed in front of the user's eye moves to a limit position along the measurement target direction; and step S3 of comparing the first eye position image with the second eye position image and the third eye position image, respectively, to calculate an activity angle of eye activity.

[0023] In specific steps, when using an acquisition device to acquire eye images, the user's head is first fixed, for example, using a forehead support, a tray, and an eye corner fixation point. After the head is fixed, a first eye position image is taken in which the user's eyes look straight ahead in an infrared light field of 700 to 1200, and second and third eye position images are taken in which the user rotates their eyes to their limit positions according to the directions indicated by the indicator lamps for each measurement target direction.

[0024] Note that eye position refers to the position of the eyeballs during an ophthalmic examination and is divided into the first, second, and third eye positions. The first eye position refers to the eye position when both eyes are looking straight at infinity on a horizontal plane. The second eye position refers to the eye position when the eyeballs rotate up, down, inward, or outward. The third eye position refers to the eye position when the eyeballs rotate diagonally inward-up, inward-down, outward-up, or outward-down, i.e., nasal-supra, nasal-supra, temporal-supra, or temporal-inferior. Accordingly, the first, second, and third eye position images refer to images taken when the eyes are in each eye position.

[0025] As shown in Figure 2, in the normal visible light wavelength range of 400-700 nm, the colors of different parts of the eye—the pupil, iris, and sclera—have little effect on image formation. The gradient structure of the corneal limbus at the interface between the iris and sclera makes it difficult to accurately identify the center of the eye. Human melanin has an absorption peak at approximately 335 nm and exhibits almost no absorption beyond 700 nm. The reflectance of the iris is fairly stable in the near-infrared wavelength range above 700 nm. Therefore, using a near-infrared light field effectively distinguishes the boundaries of the sclera, iris, and pupil, thereby improving the accuracy and stability of the algorithm.

[0026] As shown in Figure 3, the measurement directions include upward, downward, inward, outward, inward-upward, outward-upward, inward-downward, and outward-downward directions of the eyeball. The activity angle of each eye is measured in each of the eight measurement directions. Each time one eye is measured, the other eye is shielded, and indicator lamps provided for each measurement direction are used to instruct the left and right eyes to rotate in each of the above directions repeatedly. Multiple second and third eye position images are obtained, and each is compared with the reference first eye position image to calculate the activity angle of eyeball activity.

[0027] The above-mentioned activity angle refers to the maximum angle of rotation of the eyeball around the center of the eyeball before and after eye movement.

[0028] Example 2 As shown in Figure 4, based on the above-mentioned method embodiment 1, this embodiment 2 further limits the calculation method of the activity angle. Specifically, this method includes: Segmenting the pupil from the first eye position image and the second eye position image in which the eyeball has moved to a downward limit position using a convolutional neural network; extracting a circle center C1 of the first eye-position image and a circle center C2 of the second eye-position image, and connecting the circle center C1 and the circle center C2 with a straight line; Finding edge points P1 and P2 along the straight line before and after the movement of the same point of the pupil in the first eye position image and the second eye position image; connecting the edge points P1 and P2 to a line P1P2; dividing the straight line P1P2 into two straight lines, a straight line A and a straight line B, by a straight line that is perpendicular to the straight line P1P2 and passes through the center C1 of the circle; and obtaining the circular angle α corresponding to the line A and the circular angle β corresponding to the line B based on the eyeball radius r, and adding the circular angle α and the circular angle β to obtain the activity angle θ.

[0029] In the above calculation method, the activity angle before and after eye movement can be obtained by calculating the circular angle corresponding to the arc length on the spherical surface through simple analytical geometric relationships.In addition, since the eyeball has a regular spherical structure, for each measurement target direction, such as inward-upward, inward-downward, outward-upward, and outward-downward, when the eyeball moves diagonally, the above activity angle can be calculated through image and data processing steps after acquiring the first, second, and third eye position images.

[0030] Example 3 As shown in Figures 5 to 7, based on the above-mentioned method in Example 2, Example 3 further restricts the compensation method of the action angle, so that the calculation accuracy of the action angle can be improved.In actual measurement, it is found that the change of pupil before and after eye movement, the displacement of the eyeball center, and the influence of corneal refraction all cause errors in the calculation of the action angle.The following will specifically explain the compensation methods for the above-mentioned several errors.

[0031] First, as shown in Figure 5, when the eyeball rotates away from the frontal gaze, the pupil image becomes elliptical, with the deformation direction in the rotational direction, and the image in the direction perpendicular to the rotational direction is unaffected. Therefore, any change in the pupil diameter in the direction perpendicular to the rotational direction is a change from the pupil. If the diameter in the frontal gaze is smaller than the diameter in the oblique gaze, the pupil expands as the eye rotates. If the diameter in the frontal gaze is larger than the diameter in the oblique gaze, the pupil contracts as the eye rotates. For example, if the ellipse contracts perpendicular to the rotational direction, this is due to pupil contraction. Connect the two divided centers of the ellipses with a line to find the diameter perpendicular to the line. Use these two diameters to calculate the pupil contraction rate. By proportionally correcting the edge point of the ellipse used for calculation on the line, the position of the edge point P2 can be compensated for according to the magnitude of the pupil change. This eliminates the impact of pupil contraction on the accuracy of the activity angle calculation, and the activity angle can be compensated for according to the magnitude of the pupil change in the first, second, or third eye position image.

[0032] As shown in Figure 6, when the eye rotates, it does not float motionlessly in space; rather, due to the surrounding soft tissue and uneven eye muscle activity, different eye rotation directions cause different displacements of the eye center. The larger the angle of eye rotation, the greater the displacement of the eye center. The impact of eye center displacement on the algorithm is equivalent to a misalignment when superimposing and aligning the front and side views, which can cause the calculated activation angle to be too large or too small. The compensation method involves determining the displacement of the eye center and then intentionally shifting it in the opposite direction when superimposing the images to offset the effect of the displacement.

[0033] As shown in Figure 7, the left image in Figure 7 shows the change in the eyeball before and after inward and outward rotation when photographed from the top of the head, and the right image in Figure 7 shows the change in the eyeball before and after upward and downward rotation when photographed from the top of the head. The displacement of the eyeball center before and after rotation and in the front-to-back direction does not affect the image captured by a camera directly in front, so it can be ignored. Only images of the eyeball projected on the cross-section are obtained, and the relationship between the direction of eye movement and the displacement of the eyeball center on the cross-section is statistically displayed.

[0034] A 30° inward rotation of the eyeball corresponds to a 0.69 mm inward displacement from the center of the eyeball. A 30° outward rotation of the eyeball corresponds to a 0.45 mm outward displacement from the center of the eyeball. A 20° upward rotation of the eyeball corresponds to a downward displacement of 0.43 mm from the center of the eyeball. A downward rotation of the eyeball by 20° corresponds to an upward displacement of 0.43 mm from the center of the eyeball. In addition, the displacement of the eyeball center and the angle of eyeball rotation are linear rules, and when the angle of eyeball rotation is 0° (front view), the displacement of the eyeball center is 0 mm. Based on the relationship between the displacement and angle, which has been statistically calculated in advance, the position of the eyeball center C2 in the activity angle calculation can be corrected, thereby compensating for the activity angle according to the range of displacement of the eyeball center in the first eye position image, the second eye position image, or the third eye position image.

[0035] As shown in Figure 8, when calculating the activity angle, the pupil edge point is affected by the refraction of the corneal crystals. The pupil edge point on the screen is not the actual position in reality, and the iris edge point is not affected because it does not pass through the cornea. Using two different algorithms for the pupil edge point and the iris edge point on the same photo can result in different activity angles, and calculation using the pupil edge point will result in corneal refraction error. Experimental measurements show that the difference between the activity angles calculated by the pupil and iris follows a highly linear pattern. Therefore, the calculation formula for the activity angle that compensates for corneal refraction error is as follows:

[0036] Compensation for activity angle θ = 0.13134 × activity angle θ + 0.52704 Corrected activity angle θ = activity angle θ + compensation for activity angle θ In the above equation, the activity angle θ is the activity angle calculated before corneal refractive error compensation.

[0037] After taking into consideration the influence of the pupil change before and after the above eye movement, the displacement of the eyeball center, and the influence of the corneal refraction on the action angle and carrying out corresponding compensation, the error in the action angle calculation can be eliminated, and finally an accurate action angle can be obtained.

[0038] In one aspect, the eye activity evaluation device according to the present application includes a first acquisition module 1001, a second acquisition module 1002, and an image processing module 1003, as shown in FIG.

[0039] The first acquisition module 1001 captures and acquires a first eye position image from the front view of the user at a position in front of the eyeball in a near-infrared light field of 700 to 1200 nm; The second acquisition module 1002 captures a second eye position image and a third eye position image taken at a position in front of the eyeball, and the user's eyeball moves to a limit position along the measurement target direction; The image processing module 1003 compares the first eye position image with the second eye position image and the third eye position image, respectively, and calculates the activity angle of the eye activity.

[0040] In one aspect, a computer-readable storage medium according to the present application stores at least one program code that, when loaded and executed by a processor, performs the operations performed by the method for assessing eye activity.

[0041] Regarding the eye activity evaluation device in the above embodiment, the specific method by which each module performs operations is described in detail in the method embodiment, and for related content, please refer to the description of part of the method embodiment.

[0042] In an exemplary embodiment, a computer-readable storage medium including a memory storing at least one program code that is loaded and executed by a processor to implement the eye activity assessment method of the above embodiment is further provided. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0043] Those skilled in the art will understand that the realization of all or part of the steps in the above embodiments may be completed by hardware, or may be completed by hardware associated with at least one program code, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0044] The above description is only a preferred embodiment of the present application, and does not limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included within the protection scope of the present application.

Claims

1. a first acquisition module that captures and acquires a first eye position image from a front view of the user at a position in front of the eyeball in a near-infrared light field of 700 to 1200 nm; a second acquisition module for acquiring a second eye position image and a third eye position image taken at a position in front of the eyeball and obtained when the user's eyeball moves to a limit position along the measurement target direction; an image processing module that compares the first eye position image with the second eye position image and the third eye position image, respectively, and calculates an activity angle of eye activity; The image processing module includes: Segmenting the pupil from the first eye position image, the second eye position image, and the third eye position image using a convolutional neural network; The circle center C1 of the first eye-position image and the circle center C2 of the second eye-position image or the third eye-position image are extracted, and the circle centers C1 and C2 are connected by a straight line; Find edge points P 1 and P 2 along the straight line before and after the movement of the same point of the pupil in the first eye position image and the second eye position image or the third eye position image; Connect the edge points P 1 and P 2 with a straight line P 1 P 2 ; The straight line P 1 P 2 is divided into two straight lines A and B by a straight line that is perpendicular to the straight line P 1 P 2 and passes through the center C 1 of the circle, According to the eyeball radius r, a central angle α corresponding to the line A and a central angle β corresponding to the line B are obtained, and the central angle α and the central angle β are added to obtain the activity angle. An eye activity evaluation system characterized by:

2. A computer program comprising: By causing a computer to execute the computer program, Step S1: capturing and acquiring a first eye position image from a front view of a user at a position in front of the eyeball in a near-infrared light field of 700 to 1200 nm; Step S2: taking an image at a position in front of the eyeball and acquiring a second eye position image and a third eye position image in which the user's eyeball moves to a limit position along the measurement target direction; Step S3: comparing the first eye position image with the second eye position image and the third eye position image, respectively, and calculating an activity angle of eye activity; Step S3 Segmenting pupils from the first eye position image, the second eye position image, and the third eye position image using a convolutional neural network; extracting a circle center C1 of the first eye-position image and a circle center C2 of the second eye-position image or the third eye-position image, and connecting the circle center C1 and the circle center C2 with a straight line; Finding edge points P1 and P2 along the straight line before and after the movement of the same point of the pupil in the first eye position image and the second eye position image or the third eye position image; connecting the edge points P1 and P2 with a line P1P2; a step of dividing the straight line P 1 P 2 into two straight lines A and B by a straight line that is perpendicular to the straight line P 1 P 2 and passes through the circle center C 1 ; and obtaining a central angle α corresponding to the line A and a central angle β corresponding to the line B according to the eyeball radius r, and adding the central angle α and the central angle β to obtain the activity angle. A computer program characterized by:

3. The measurement target direction includes upward, downward, inward, outward, inward-upward, outward-upward, inward-downward, and outward-downward directions of the eyeball.

3. A computer program according to claim 2.

4. The action angle is compensated for by a range of pupil change, a range of eyeball center displacement, and corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image.

3. A computer program according to claim 2.

5. In the step of compensating the activity angle according to the width of pupil change in the first eye position image, the second eye position image, or the third eye position image, the edge point P is obtained by obtaining the width of pupil change in the first eye position image, the second eye position image, or the third eye position image. ２ Compensating for the position of 5. A computer program according to claim 4.

6. A width that varies along a direction perpendicular to the measurement target direction of the pupil diameter in the first eye position image, the second eye position image, or the third eye position image is calculated, and the edge point P ２ compensate the position of 6. A computer program according to claim 5.

7. In the step of compensating the activity angle according to the range of displacement of the eyeball center in the first eye position image, the second eye position image, or the third eye position image, a superimposed image formed by different displacements of the eyeball center in the first eye position image, the second eye position image, or the third eye position image is obtained, and the superimposed image is translated in the opposite direction to cancel out the influence of the displacement on the activity angle.

5. A computer program according to claim 4.

8. In the step of compensating the activity angle by corneal refractive error compensation in the first eye position image, the second eye position image, or the third eye position image, the angle compensated by corneal refractive error compensation and the activity angle have a linear relationship.

5. A computer program according to claim 4.

Images