Eye motion recognition method and device based on image recognition

Abstract

The invention discloses an eye movement recognition method and device based on image recognition. The method comprises the steps that eye side face images are continuously collected in real time; identifying any frame of the eye side image, and identifying six key points P1, P2, P3, P4, P5 and P6 around the eyes; calculating an included angle a between a connecting line L13 of the P1 point and the P3 point and a connecting line L15 of the P1 point and the P5 point; calculating an included angle b between a connecting line L12 of the P1 point and the P2 point and a connecting line L16 of the P1 point and the P6 point; judging whether the included angle a and the included angle b of the frame of eye side image are qualified or not; calculating the angle difference alpha of the qualified included angle a or the qualified included angle b corresponding to the two adjacent frames of eye side images; if alpha is less than 0, determining that the moment is in an eye closing process; if alpha is greater than 0, determining that the moment is in an eye opening process. The eye closing degree is expressed by using the angles of the upper and lower eyelids instead of the distance between the upper and lower eyelids, so that the detection precision is high and the detection efficiency is good.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and specifically to an eye movement recognition method and device based on image recognition. Background Art

[0002] Eye movement recognition methods have clear applications in the treatment of ophthalmic diseases (such as dry eyes), neurosurgery and cranial nerve rehabilitation (such as facial paralysis), and fatigue status warning in key positions (such as special vehicle driving).

[0003] Existing eye movement recognition methods are usually implemented based on the aspect ratio of the human eye (Eye Aspect Ratio, EAR), referred to as the EAR scheme, which is a method that uses facial key points for detection.

[0004] Specifically, the method first locates the key points of the eye region using a facial key point detector, and then evaluates the eye opening and closing state by calculating the ratio between the height and width of the eye. When the eyes are closed, the EAR decreases significantly and approaches zero; when the eyes are open, the EAR remains relatively constant.

[0005] In addition, there is another eye movement recognition method that uses the dlib library, referred to as the dlib solution. It provides advanced face detection and key point location functions, which can quickly and accurately identify faces in images and locate key points of faces, such as eyes, nose, mouth, etc. Eye closure detection can be achieved by analyzing the distance changes between key points in the eye area. When the average value of these distances is lower than a certain threshold, the eyes can be considered closed.

[0006] The above two methods are both for recognizing the frontal image of the eye, and only analyze the distance difference of the key points.

[0007] However, in some special environments, only the side image of the eye can be collected, and the existing algorithms cannot recognize the eye state based on the side image of the eye. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes an eye movement recognition method and device based on image recognition.

[0009] In order to achieve the above object, the technical solution of the present invention is as follows:

[0010] In one aspect, the present invention discloses an eye movement recognition method based on image recognition, comprising:

[0011] Step S1: continuously and in real time collect side eye images;

[0012] Step S2: Recognize the side view of the eye in any frame and identify six key points P1, P2, P3, P4, P5, and P6 around the eye;

[0013] Among them: P1 point is the eye corner point;

[0014] Points P3 and P5 are the two farthest points where the eyeball and eyelid meet.

[0015] Point P2 is the point closest to the eyelid around the midpoint of the line L13 connecting points P1 and P3;

[0016] Point P6 is the point closest to the eyelid around the midpoint of the line L15 connecting points P1 and P5;

[0017] Point P4 is the intersection of the upper and lower eyelids with the arc of the outer contour of the eye when the upper and lower eyelids are closed;

[0018] Step S3: Calculate the angle a between the line L13 connecting the points P1 and P3 and the line L15 connecting the points P1 and P5;

[0019] Calculate the angle b between the line L12 connecting points P1 and P2 and the line L16 connecting points P1 and P6;

[0020] Step S4: determining whether the angle a and the angle b of the eye side image of the frame are qualified;

[0021] If qualified, repeat steps S2 to S4 to calculate the angle a and angle b of the eye side image of the next frame. When the angle a and angle b of the eye side image of the next frame are qualified, continue to step S5;

[0022] If unqualified, step S5 is not executed;

[0023] Step S5: calculating the angle difference α between the qualified angle a or the qualified angle b corresponding to the side eye images of two adjacent frames;

[0024] If α<0, the eyes are closed at this moment;

[0025] If α>0, the eyes are in the process of opening at that moment.

[0026] On the basis of the above technical solution, the following improvements can be made:

[0027] As a preferred solution, in step S2, a neural convolutional network or an image recognition algorithm is used to recognize the side eye image of any frame.

[0028] As a preferred solution, step S4 specifically comprises: using the angle difference δ to judge whether the angle a and the angle b of the side eye image of the frame are qualified, and the angle difference δ is the difference between the angle a and the angle b;

[0029] If the angle difference δ does not exceed the threshold, the image is qualified, and steps S2 to S4 are repeated to calculate the angle a and angle b of the eye side image of the next frame. If the angle a and angle b of the eye side image of the next frame are qualified, step S5 is continued;

[0030] If the angle difference δ exceeds the threshold, it is unqualified and step S5 is not executed.

[0031] As a preferred solution, in step S4, if the angle a and the angle b are unqualified, the steps S2 to S4 are restarted in sequence to calculate the angle a and the angle b of the next frame of the side eye image and determine whether they are qualified;

[0032] If the angle difference δ exceeds the threshold value repeatedly, an alarm is triggered.

[0033] In addition, on the other hand, the present invention also discloses an eye state detection device based on image recognition, comprising:

[0034] An eye side image acquisition module, used for continuously and in real time acquiring eye side images;

[0035] The key point recognition module is used to recognize the side eye image of any frame and identify six key points around the eyes: P1, P2, P3, P4, P5, and P6;

[0036] Among them: P1 point is the eye corner point;

[0037] Points P3 and P5 are the two farthest points where the eyeball and eyelid meet.

[0038] Point P2 is the point closest to the eyelid around the midpoint of the line L13 connecting points P1 and P3;

[0039] Point P6 is the point closest to the eyelid around the midpoint of the line L15 connecting points P1 and P5;

[0040] Point P4 is the intersection of the upper and lower eyelids with the arc of the outer contour of the eye when the upper and lower eyelids are closed;

[0041] The eyelid angle calculation module is used to calculate the angle a between the line L13 connecting the points P1 and P3 and the line L15 connecting the points P1 and P5;

[0042] Calculate the angle b between the line L12 connecting points P1 and P2 and the line L16 connecting points P1 and P6;

[0043] A qualified judgment module is used to judge whether the angle a and the angle b of the side eye image of the frame are qualified;

[0044] If qualified, the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are repeatedly executed in sequence to calculate the angle a and the angle b of the side eye image of the next frame. When the angle a and the angle b of the side eye image of the next frame are both qualified, the method in the motion key point recognition module is continued to be executed;

[0045] If it fails, the method in the motion key point identification module will not be executed;

[0046] A motion key point recognition module is used to calculate the angle difference α between the qualified angle a or the qualified angle b corresponding to the side eye images of two adjacent frames;

[0047] If α<0, the eyes are closed at this moment;

[0048] If α>0, the eyes are in the process of opening at that moment.

[0049] As a preferred solution, the key point recognition module uses a neural convolutional network or an image recognition algorithm to recognize the side eye image of any frame.

[0050] As a preferred solution, the qualification judgment module specifically includes the following methods:

[0051] The angle difference δ is used to judge whether the angle a and the angle b of the side image of the eye of the frame are qualified, and the angle difference δ is the difference between the angle a and the angle b;

[0052] If the angle difference δ does not exceed the threshold, it is qualified, and the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are repeatedly executed in sequence to calculate the angle a and the angle b of the eye side image of the next frame. When the angle a and the angle b of the eye side image of the next frame are both qualified, the method in the motion key point recognition module is continued to be executed;

[0053] If the angle difference δ exceeds the threshold, it is unqualified and the method in the motion key point identification module is not executed.

[0054] As a preferred solution, in the qualified judgment module, if the angle a and the angle b are unqualified, the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are restarted in sequence to calculate the angle a and the angle b of the next frame of the side eye image and judge whether it is qualified;

[0055] If the angle difference δ exceeds the threshold value repeatedly, an alarm is triggered.

[0056] The present invention discloses an eye movement recognition method and device based on image recognition, which has the following beneficial effects:

[0057] The present invention uses the angle of the upper and lower eyelids to express the degree of eye closure, instead of the distance between the upper and lower eyelids, to detect the eye movement state. It is an innovative detection method with high detection accuracy and good detection efficiency, providing a theoretical basis for subsequent research and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 This is one of the flow charts of the eye movement recognition method provided in an embodiment of the present invention.

[0060] Figure 2 This is a collection diagram of the side image of the eye provided by an embodiment of the present invention.

[0061] Figure 3 An actual schematic diagram of key points P1-P6 of a side eye image provided by an embodiment of the present invention.

[0062] Figure 4 A theoretical schematic diagram of key points P1-P6 of a side eye image provided by an embodiment of the present invention.

[0063] Figure 5 The second flowchart of the eye movement recognition method provided in the embodiment of the present invention.

[0064] Figure 6 A schematic diagram of a head-mounted facial paralysis therapeutic device provided in an embodiment of the present invention.

[0065] Figure 7 A schematic diagram of placing stimulation electrodes on the orbicularis oculi nerve trunk provided in an embodiment of the present invention.

[0066] Figure 8 A schematic diagram of placing stimulation electrodes on the facial nerve trunk provided by an embodiment of the present invention.

[0067] Fig. 9 A schematic diagram of a columnar electrode provided in an embodiment of the present invention.

[0068] Fig.10 A schematic diagram of a filamentary spiral electrode provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The expression of “comprising” an element is an “open” expression, which merely means that a corresponding component or step exists, and should not be interpreted as excluding additional components or steps.

[0072] In order to achieve the purpose of the present invention, in some embodiments of the eye movement recognition method and device based on image recognition, as Figure 1-2 As shown, the eye movement recognition method includes:

[0073] Step S1: continuously and in real time collect side eye images;

[0074] Step S2: Figure 3-4 As shown, the side view of the eye of any frame is recognized, and six key points P1, P2, P3, P4, P5, and P6 around the eye are identified;

[0075] Among them: P1 point is the eye corner point;

[0076] Points P3 and P5 are the two farthest points where the eyeball and eyelid meet.

[0077] Point P2 is the point closest to the eyelid around the midpoint of the line L13 connecting points P1 and P3;

[0078] Point P6 is the point closest to the eyelid around the midpoint of the line L15 connecting points P1 and P5;

[0079] Point P4 is the intersection of the upper and lower eyelids with the arc of the outer contour of the eye when the upper and lower eyelids are closed;

[0080] Step S3: Calculate the angle a between the line L13 connecting the points P1 and P3 and the line L15 connecting the points P1 and P5;

[0081] Calculate the angle b between the line L12 connecting points P1 and P2 and the line L16 connecting points P1 and P6;

[0082] Step S4: determining whether the angle a and the angle b of the eye side image of the frame are qualified;

[0083] If qualified, repeat steps S2 to S4 to calculate the angle a and angle b of the eye side image of the next frame. When the angle a and angle b of the eye side image of the next frame are qualified, continue to step S5;

[0084] If unqualified, step S5 is not executed;

[0085] Step S5: calculating the angle difference α between the qualified angle a or the qualified angle b corresponding to the side eye images of two adjacent frames;

[0086] If α<0, the eyes are closed at this moment;

[0087] If α>0, the eyes are in the process of opening at that moment.

[0088] The angle a and angle b mentioned above are the angles of the upper and lower eyelids.

[0089] P4 is used to check the degree of closure of the upper and lower eyelids, and can be cross-checked with the angles of the upper and lower eyelids.

[0090] Step S4 performs qualification judgment on the angle a and the angle b to ensure accurate calculation and avoid misjudgment.

[0091] Step S5 makes a judgment based on the change in the angles of the upper and lower eyelids between the two preceding and following frames (angle difference α).

[0092] Specifically, α t-1 =a t -a t-1 , or, α t-1 =b t -b t-1 ;

[0093] a t , b t is the angle of the side image of the eye at time t;

[0094] a t-1 , b t-1 is the angle of the side eye image in the frame at time t-1.

[0095] In the above step S2, a neural convolutional network or an image recognition algorithm is used to recognize the side eye image of any frame.

[0096] Further, step S4 specifically includes using the angle difference δ to determine whether the angle a and the angle b of the side view image of the frame are qualified, where the angle difference δ is the difference between the angle a and the angle b;

[0097] If the angle difference δ does not exceed a threshold value (e.g., 5°), the image is qualified, and steps S2 to S4 are repeated to calculate the angle a and angle b of the eye side image of the next frame. If the angle a and angle b of the eye side image of the next frame are both qualified, step S5 is continued;

[0098] If the angle difference δ exceeds the threshold, it is unqualified and step S5 is not executed. The image acquisition device (such as the lens) can be further checked to see if it is seriously contaminated and a repair report can be submitted.

[0099] In a specific embodiment, the threshold value (eg, 5°) should be set according to different groups of people, such as skin color and skin aging degree, to avoid invalid misjudgment.

[0100] Further, in step S4, if the angle a and the angle b are unqualified, the steps S2 to S4 are sequentially executed again to calculate the angle a and the angle b of the next frame of the side view of the eye and determine whether they are qualified;

[0101] If the angle difference δ exceeds the threshold value repeatedly, an alarm is triggered.

[0102] Further, in some embodiments, Figure 5 As shown, the eye movement recognition method of the present invention also includes:

[0103] Step S6: Repeat steps S2 to S5 to obtain an angle difference α between the next frame and the eye side image of the previous frame;

[0104] Step S7: Calculate Ω by the following formula: t , and based on Ω t Determine the speed of eye opening and closing;

[0105] Ω t =(α (t,t+1) -α (t-1,t) ) / △T;

[0106] Where: α (t,t+1) is the angle difference obtained based on the side eye images of two adjacent frames at time t and time t+1;

[0107] α (t-1,t) is the angle difference obtained based on the side eye images of two adjacent frames at time t-1 and time t;

[0108] △T is the sampling time interval between two adjacent frames of eye side images;

[0109] If Ω t >Ω t-1 , it indicates that the eye closing speed is accelerated;

[0110] If Ω t >A, it indicates rapid eye closure;

[0111] If Ω t <A, it indicates slow eye closure, and A is the threshold;

[0112] If Ω t =0, and α (t,t+1) <0, α (t,t+1) <0, it means the eyes are closed;

[0113] If Ω t =0, and α (t,t+1) >0, α (t,t+1) >0, it indicates the eyes are open.

[0114] The present invention uses the rate of change of angle to identify the mode of eye closure: fast eye closure, slow eye closure, long-term eye closure, etc.

[0115] The present invention analyzes and identifies the side image of the eye, and identifies the eye state based on the angle recognition method. The present invention can be applied to fatigue driving, sleep quality monitoring, facial paralysis stimulation treatment, etc. Compared with the traditional method of collecting frontal images of the eye, the present invention can install the image acquisition device (such as a camera) on the side of the eye to reduce the impact on the human body.

[0116] For existing head-mounted facial paralysis therapeutic devices (such as Figure 6 The glasses-like structure shown in the figure) can install image acquisition devices (such as cameras) on both sides of the therapeutic device (specifically, they can be installed at the temples), which further improves the aesthetics and reduces the pressure on the user's eyes.

[0117] The following example illustrates the application of the present invention in facial paralysis stimulation treatment, in which 95% of facial paralysis patients suffer from unilateral facial paralysis, which specifically includes the following steps:

[0118] Step T1: using the eye movement recognition method disclosed in the present invention to collect the side image of the patient's healthy side in real time, and analyze it to obtain the closing pattern of the patient's healthy side, which can be any movement pattern of involuntary blinking, voluntary eye closing, long-term eye closing, and hard blinking;

[0119] The above different motion modes can be divided by using different thresholds A, different eye opening and closing durations, etc.;

[0120] Step T2: generating a stimulation waveform corresponding to the predicted eye movement pattern of the healthy side of the patient;

[0121] Step T3: Send the stimulation waveform to the stimulation electrode on the affected side of the patient.

[0122] The stimulating electrode on the affected side is transcutaneously or implanted to stimulate the orbicularis oculi muscle and / or nerve that controls the eye closing movement.

[0123] The present invention is based on image recognition. After collecting and detecting the movement signals of the muscles controlled by the facial nerve on the healthy side, it processes and outputs a stimulation waveform. By stimulating the facial nerve on the affected side through nerve electrodes, normal muscle movement of the facial paralysis side is achieved, solving the problem that the facial muscles cannot move normally after the facial nerve on the affected side is damaged.

[0124] like Figure 7As shown, the stimulation electrode is placed on the orbicularis oculi nerve trunk; Figure 8 As shown, the stimulating electrode is placed on the trunk of the facial nerve.

[0125] Furthermore, the implantable stimulation electrode may be, but is not limited to, a columnar electrode (e.g. Fig. 9 as shown) or a filamentary spiral electrode (as Fig.10 shown).

[0126] The embodiment of the present invention further discloses an eye state detection device based on image recognition, comprising:

[0127] An eye side image acquisition module, used for continuously and in real time acquiring eye side images;

[0128] The key point recognition module is used to recognize the side eye image of any frame and identify six key points around the eyes: P1, P2, P3, P4, P5, and P6;

[0129] Among them: P1 point is the eye corner point;

[0130] Points P3 and P5 are the two farthest points where the eyeball and eyelid meet.

[0131] Point P2 is the point closest to the eyelid around the midpoint of the line L13 connecting points P1 and P3;

[0132] Point P6 is the point closest to the eyelid around the midpoint of the line L15 connecting points P1 and P5;

[0133] Point P4 is the intersection of the upper and lower eyelids with the arc of the outer contour of the eye when the upper and lower eyelids are closed;

[0134] The eyelid angle calculation module is used to calculate the angle a between the line L13 connecting the points P1 and P3 and the line L15 connecting the points P1 and P5;

[0135] Calculate the angle b between the line L12 connecting points P1 and P2 and the line L16 connecting points P1 and P6;

[0136] A qualified judgment module is used to judge whether the angle a and the angle b of the side eye image of the frame are qualified;

[0137] If qualified, the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are repeatedly executed in sequence to calculate the angle a and the angle b of the side eye image of the next frame. When the angle a and the angle b of the side eye image of the next frame are both qualified, the method in the motion key point recognition module is continued to be executed;

[0138] If it fails, the method in the motion key point identification module will not be executed;

[0139] A motion key point recognition module is used to calculate the angle difference α between the qualified angle a or the qualified angle b corresponding to the side eye images of two adjacent frames;

[0140] If α<0, the eyes are closed at this moment;

[0141] If α>0, the eyes are in the process of opening at that moment.

[0142] Furthermore, the key point recognition module uses a neural convolutional network or an image recognition algorithm to recognize the side eye image of any frame.

[0143] Furthermore, the qualification judgment module specifically includes the following methods:

[0144] The angle difference δ is used to judge whether the angle a and the angle b of the side image of the eye of the frame are qualified, and the angle difference δ is the difference between the angle a and the angle b;

[0145] If the angle difference δ does not exceed the threshold, it is qualified, and the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are repeatedly executed in sequence to calculate the angle a and the angle b of the eye side image of the next frame. When the angle a and the angle b of the eye side image of the next frame are both qualified, the method in the motion key point recognition module is continued to be executed;

[0146] If the angle difference δ exceeds the threshold, it is unqualified and the method in the motion key point identification module is not executed.

[0147] Further, in the qualified judgment module, if the angle a and the angle b are unqualified, the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are restarted in sequence to calculate the angle a and the angle b of the next frame of the side eye image and judge whether it is qualified;

[0148] If the angle difference δ exceeds the threshold value repeatedly, an alarm is triggered.

[0149] In this embodiment, the specific content of the eye movement recognition device based on image recognition is similar to the content of the eye movement recognition method based on image recognition disclosed in the above embodiment, and will not be repeated here.

[0150] In summary, the present invention discloses an eye movement recognition method and device based on image recognition, which has the following beneficial effects:

[0151] The present invention uses the angle of the upper and lower eyelids to express the degree of eye closure, instead of the distance between the upper and lower eyelids, to detect the eye movement state. It is an innovative detection method with high detection accuracy and good detection efficiency, providing a theoretical basis for subsequent research and development.

[0152] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0153] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An eye movement recognition method based on image recognition, characterized in that: include: Step S1: continuously and in real time collect side eye images; Step S2: Recognize the side view of the eye in any frame and identify six key points P1, P2, P3, P4, P5, and P6 around the eye; Among them: P1 point is the eye corner point; Points P3 and P5 are the two farthest points where the eyeball and eyelid meet. Point P2 is the point closest to the eyelid around the midpoint of the line L13 connecting points P1 and P3; Point P6 is the point closest to the eyelid around the midpoint of the line L15 connecting points P1 and P5; Point P4 is the intersection of the upper and lower eyelids with the arc of the outer contour of the eye when the upper and lower eyelids are closed; Step S3: Calculate the angle a between the line L13 connecting the points P1 and P3 and the line L15 connecting the points P1 and P5; Calculate the angle b between the line L12 connecting points P1 and P2 and the line L16 connecting points P1 and P6; Step S4: determining whether the angle a and the angle b of the eye side image of the frame are qualified; If qualified, repeat steps S2 to S4 to calculate the angle a and angle b of the eye side image of the next frame. When the angle a and angle b of the eye side image of the next frame are qualified, continue to step S5; If unqualified, step S5 is not executed; Step S5: calculating the angle difference α between the qualified angle a or the qualified angle b corresponding to the side eye images of two adjacent frames; If α<0, the eyes are closed at this moment; If α>0, the eyes are in the process of opening at that moment.

2. The eye movement recognition method according to claim 1, characterized in that: In step S2, a neural convolutional network or an image recognition algorithm is used to recognize the side eye image of any frame.

3. The eye movement recognition method according to claim 1, characterized in that: Specifically, step S4 comprises using the angle difference δ to determine whether the angle a and the angle b of the eye side image of the frame are qualified, where the angle difference δ is the difference between the angle a and the angle b; If the angle difference δ does not exceed the threshold, the image is qualified, and steps S2 to S4 are repeated to calculate the angle a and angle b of the eye side image of the next frame. If the angle a and angle b of the eye side image of the next frame are qualified, step S5 is continued; If the angle difference δ exceeds the threshold, it is unqualified and step S5 is not executed.

4. The eye movement recognition method according to claim 1, characterized in that: In step S4, if the angle a and the angle b are unqualified, the steps S2 to S4 are executed again in sequence to calculate the angle a and the angle b of the next frame of the side eye image and determine whether they are qualified; If the angle difference δ exceeds the threshold value repeatedly, an alarm is triggered.

5. An eye state detection device based on image recognition, characterized in that: include: An eye side image acquisition module, used for continuously and in real time acquiring eye side images; The key point recognition module is used to recognize the side eye image of any frame and identify six key points around the eyes: P1, P2, P3, P4, P5, and P6; Among them: P1 point is the eye corner point; Points P3 and P5 are the two farthest points where the eyeball and eyelid meet. Point P2 is the point closest to the eyelid around the midpoint of the line L13 connecting points P1 and P3; Point P6 is the point closest to the eyelid around the midpoint of the line L15 connecting points P1 and P5; Point P4 is the intersection of the upper and lower eyelids with the arc of the outer contour of the eye when the upper and lower eyelids are closed; The eyelid angle calculation module is used to calculate the angle a between the line L13 connecting the points P1 and P3 and the line L15 connecting the points P1 and P5; Calculate the angle b between the line L12 connecting points P1 and P2 and the line L16 connecting points P1 and P6; A qualified judgment module is used to judge whether the angle a and the angle b of the side eye image of the frame are qualified; If qualified, the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are repeatedly executed in sequence to calculate the angle a and the angle b of the side eye image of the next frame. When the angle a and the angle b of the side eye image of the next frame are both qualified, the method in the motion key point recognition module is continued to be executed; If it fails, the method in the motion key point identification module will not be executed; A motion key point recognition module is used to calculate the angle difference α between the qualified angle a or the qualified angle b corresponding to the side eye images of two adjacent frames; If α<0, the eyes are closed at this moment; If α>0, the eyes are in the process of opening at that moment.

6. The eye movement recognition device according to claim 5, characterized in that: The key point recognition module uses a neural convolutional network or an image recognition algorithm to recognize the side eye image of any frame.

7. The eye movement recognition device according to claim 5, characterized in that: The qualification judgment module specifically includes the following methods: The angle difference δ is used to judge whether the angle a and the angle b of the side image of the eye of the frame are qualified, and the angle difference δ is the difference between the angle a and the angle b; If the angle difference δ does not exceed the threshold, it is qualified, and the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are repeatedly executed in sequence to calculate the angle a and the angle b of the eye side image of the next frame. When the angle a and the angle b of the eye side image of the next frame are both qualified, the method in the motion key point recognition module is continued to be executed; If the angle difference δ exceeds the threshold, it is unqualified and the method in the motion key point identification module is not executed.

8. The eye movement recognition device according to claim 5, characterized in that: In the qualified judgment module, if the angle a and the angle b are unqualified, the methods in the key point recognition module, the eyelid angle calculation module and the qualified judgment module are restarted in sequence to calculate the angle a and the angle b of the next frame of the side eye image and judge whether it is qualified; If the angle difference δ exceeds the threshold value repeatedly, an alarm is triggered.

Images