An eye movement interaction system and method for a command cabin multi-screen

Abstract

The application discloses an eye movement interaction system and method for a multi-screen command cabin. The system comprises multiple display screens, a multi-screen cooperative display control module, an infrared camera, an interaction control module and a communication module. Each display screen independently displays an operation picture. The infrared camera is used to acquire an eye image of a commander and transmit the eye image to the interaction control module. The interaction control module is used to perform line-of-sight estimation, switch to a display screen and a picture area thereon where the commander gazes and lock the picture area by the multi-screen cooperative display control module, and perform eye movement behavior recognition to acquire an instruction and make the locked display screen execute the instruction by the multi-screen cooperative display control module. The application realizes automatic switching and locking between screens and the function of automatically issuing an instruction according to the intention of the commander by estimating the line of sight of the commander and recognizing the eye movement behavior.

Description

Technical Field

[0001] This invention relates to an eye-tracking interaction system and method, and more particularly to an eye-tracking interaction system and method for use in a multi-screen command cabin. Background Technology

[0002] In modern command and control environments, commanders need to monitor and process large amounts of information in real time. With technological advancements, multi-screen displays have become standard in command and control rooms to provide richer visual information. However, traditional human-computer interaction methods mainly rely on mice, keyboards, or touchscreens, which can lead to low operational efficiency and increased fatigue in high-intensity, high-pressure command environments.

[0003] Eye-tracking interaction technology, as an emerging human-computer interaction method, enables system control and interaction by capturing the commander's eye movements. This technology can free the commander's hands and provide a more natural and intuitive interactive experience. In multi-screen environments, eye-tracking interaction can effectively improve the efficiency of information acquisition and processing, and enhance the timeliness and accuracy of command decisions.

[0004] Currently, the more mature eye-tracking interaction technologies mainly include: 1) Single-screen eye-tracking interaction system: There are some eye-tracking control solutions on the market that are applied to single screens, such as in the medical and aviation fields. They control a single screen or device through eye movements, but they lack solutions for multi-screen environments and cannot fully utilize the advantages of eye-tracking technology.

[0005] 2) Multi-screen display + traditional interaction methods: In multi-screen environments such as command centers, mice, multi-touch, or dedicated control panels are typically used to manage the content and interaction of multiple screens. This traditional method requires frequent focus switching in a multi-screen environment, making the operation complex and time-consuming. It cannot meet the command center's demand for efficient operation, and prolonged use of input devices can lead to fatigue for commanders, greatly affecting their decision-making ability and reaction speed.

[0006] 3) Eye-tracking interaction in virtual reality (VR): Eye-tracking technology has been used to enhance the commander's experience in VR environments, but it is mainly applied to virtual scenes rather than the actual multi-screen environment of a command center. When this virtual reality is applied to real-world complex environments, problems such as reduced accuracy and increased latency occur, which greatly affects the commander's experience and reduces system reliability. Summary of the Invention

[0007] The purpose of this invention is to provide an eye-tracking interaction system and method for multiple screens in a command cabin. By estimating the commander's gaze and recognizing eye movements, it achieves the functions of automatically switching and locking between screens and automatically issuing commands according to the commander's intentions.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An eye-tracking interaction system for a multi-screen command cabin includes multiple display screens, a multi-screen collaborative display control module, an infrared camera, an interaction control module, and a communication module, wherein: Each of the aforementioned display screens independently displays the operation screen; The infrared camera is placed on the table in front of the commander to acquire images of the commander's eyes and transmit the images to the interactive control module. The interactive control module is used to perform gaze estimation based on eye images to switch to the display screen that the commander is gazing at and the image area on the display screen and lock the image area by means of the multi-screen collaborative display control module, and to perform eye movement behavior recognition based on multiple consecutive eye images to obtain instructions and to make the locked display screen execute instructions by means of the multi-screen collaborative display control module.

[0009] An eye-tracking interaction method for multi-screen command cabins, implemented based on the aforementioned eye-tracking interaction system for multi-screen command cabins, includes the following steps: 1) The system starts up and enters the operation state; 2) Each of the aforementioned display screens displays the operation screen, and each operation screen is presented in the commander's field of vision; 3) The infrared camera acquires an image of the commander's eyes and transmits the image to the interactive control module; 4) The interactive control module estimates the line of sight based on the eye image and determines whether the commander is staring at a certain area of ​​the display screen. If not, proceed to the next step. If so, the multi-screen collaborative display control module locks the display screen that the commander is staring at and the area of ​​the screen on the display screen, and the area of ​​the screen is magnified or the key information is highlighted. Proceed to the next step. 5) The interactive control module performs eye movement behavior recognition based on multiple consecutive eye images to determine whether the commander has issued an instruction: if not, return to step 3); if so, obtain the instruction corresponding to this eye movement behavior, and use the multi-screen collaborative display control module to make the locked display screen execute the instruction.

[0010] The advantages of this invention are: This invention can monitor the commander's gaze in real time, achieve automatic switching between screens and lock the currently focused screen through gaze estimation, and automatically recognize the commander's intentions through eye movement behavior recognition, helping the commander to issue instructions quickly and accurately, improve decision-making efficiency and task execution efficiency, and avoid the problems of lag and misoperation in traditional input devices (such as mouse and keyboard). It is highly efficient and reliable, and is suitable for high-pressure operation scenarios with multi-task interaction between the commander and multiple screens in the command cabin. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the eye-tracking interaction system of the present invention. Detailed Implementation

[0012] like Figure 1 As shown, this invention proposes an eye-tracking interaction system for multiple screens in a command cabin, comprising multiple display screens 20, a multi-screen collaborative display control module 10, an infrared camera 30, an interaction control module 40, and a communication module 50, wherein: Each of the 20 display screens displays the operation screen independently; An infrared camera 30 is placed on the table in front of the commander to acquire images of the commander's eyes and transmit the images to the interactive control module 40. The interactive control module 40 is used to perform gaze estimation based on eye images to switch to the display screen 20 that the commander is looking at and the screen area on this display screen 20 and lock the screen area by means of the multi-screen collaborative display control module 10, and to perform eye movement behavior recognition based on multiple consecutive eye images to obtain instructions and make the locked display screen 20 execute instructions by means of the multi-screen collaborative display control module 10.

[0013] like Figure 1 The eye-tracking interaction system of the present invention also includes a communication module 50. When the instruction is to trigger a related device, the interaction control module 40, while enabling the locked display screen 20 to execute the instruction through the multi-screen collaborative display control module 10, sends the instruction to the related device for execution through the communication module 50.

[0014] In this invention, the infrared camera 30 is a multi-view, high-resolution desktop infrared camera, which is a well-known technology in the art. Multiple infrared cameras 30 can be arranged on a table in front of the commander and positioned opposite the commander's eyes. The infrared camera 30 should be able to capture images of both eyes, and the captured eye video consists of a series of consecutive eye images (eye image sequence).

[0015] In this invention, the display screens 20 are existing screens in the art, and each display screen 20 is arranged adjacent to each other according to actual needs for easy operation. Multiple display screens 20 together constitute a desktop information presentation and command environment, each displaying different tasks or information interfaces. For example, the left display screen 20 displays alarm information, the middle display screen 20 displays data monitoring footage, and the right display screen 20 displays the current task status, etc., eliminating the need for the commander to use traditional input devices (such as a mouse or keyboard). Each display screen 20 switches between themselves and highlights or magnifies key information based on the commander's gaze. Furthermore, based on the commander's eye movement behavior, their intention is identified to execute different commands, such as blinking to indicate selection / click, and eye movement to indicate closing / back, etc.

[0016] In actual implementation, the interactive control module 40 is also used to obtain eye movement frequency and eye usage duration based on multiple consecutive eye images, and to determine whether eye fatigue is about to occur based on eye movement frequency and eye usage duration, so as to make timely rest reminders and avoid eye fatigue for commanders.

[0017] Eye use duration can be the sum of all gazing and fixation time. When the eye movement frequency exceeds the eye movement threshold, or the eye use duration exceeds the eye use threshold, it can be judged that eye fatigue is about to occur.

[0018] In actual implementation, once the screen area is locked, it can be set to randomly enlarge key information in the screen area or highlight key information so that the commander can quickly identify key information.

[0019] In this invention, eye movements include prolonged fixation, short-term fixation, blinking, saccades, and smooth left and right movements. Prolonged fixation can be defined as maintaining a fixed position for 5 seconds or more (including 5 seconds), while short-term fixation is defined as maintaining a fixed position for 2 to 4 seconds. Prolonged fixation, short-term fixation, blinking, and smooth left and right movements can each correspond to a command, while saccades are a natural phenomenon of the human eye and do not correspond to any command. Furthermore, short-term fixation can be used for information selection or confirmation (such as confirmation prompts through color changes or slight magnification), while prolonged fixation can be used to trigger operation menus or other advanced functions. For example, when a certain area of ​​the screen is fixed on for a prolonged period, the system can display relevant operation options, allowing selection to be completed through short-term fixation, thus achieving convenient contactless operation.

[0020] In contrast to the above-mentioned gaze, a gaze can be defined as a state where both eyes remain unchanged for less than 1 second (including 1 second).

[0021] In this invention, the multi-screen collaborative display control module 10 and the interactive control module 40 include a microprocessor, and the communication module 50 includes a communication device.

[0022] In practical implementation, this invention can also be combined with interactive methods such as voice or gestures to further improve the flexibility and accuracy of human-computer interaction.

[0023] Based on the above-mentioned eye-tracking interaction system for multiple screens in the command cabin, this invention also proposes an eye-tracking interaction method for multiple screens in the command cabin, comprising the following steps: 1) The system starts up and enters the operation state; 2) Each display screen 20 displays the operation screen, and each operation screen is presented in the commander's field of vision; 3) The infrared camera 30 acquires the commander's eye image and transmits the eye image to the interactive control module 40; 4) The interactive control module 40 estimates the gaze based on the eye image and determines whether the commander is staring at a certain display screen 20. If not, proceed to the next step; if so, the multi-screen collaborative display control module 10 locks the display screen 20 that the commander is staring at and the display area on the display screen 20. This display area can be magnified or key information can be highlighted. Proceed to the next step. 5) The interactive control module 40 performs eye movement behavior recognition based on multiple consecutive eye images to determine whether the commander has issued an instruction: if not, return to step 3); if so, obtain the instruction corresponding to this eye movement behavior, and use the multi-screen collaborative display control module 10 to make the locked display screen 20 execute the instruction.

[0024] In step 5), if the instruction is to trigger the relevant device, the interactive control module 40, while using the multi-screen collaborative display control module 10 to make the locked display screen 20 execute the instruction, sends the instruction to the relevant device for execution via the communication module 50.

[0025] In step 4), a gaze estimation algorithm is used to estimate the gaze. The gaze estimation algorithm includes a pupil detection algorithm and a two-dimensional gaze estimation algorithm, wherein: The pupil detection algorithm includes the following steps: First, the eye image is preprocessed (grayscale value normalization). Then, the well-known Canny edge detection algorithm is used to extract edge information, resulting in multiple contours (which may be the orbital contour, pupil contour, iris contour, etc.). Then, a set of preset pupil geometric feature thresholds (pupil geometric features such as roundness, major and minor axis ratio, contour area range, etc.) are used for preliminary screening. Contours that may be pupil contours are selected from all contours as candidate contours. Then, these candidate contours are fitted with ellipses using the least squares method to obtain candidate fitted contours that are similar to the shape of the pupil. Finally, based on the confidence level, the candidate fitted contour with the highest confidence level is selected as the pupil contour to obtain the pupil center coordinates.

[0026] The two-dimensional line-of-sight estimation algorithm includes the following steps: The pupil center coordinates (u1, v1) and (u2, v2) of the left and right eyes are obtained using a pupil detection algorithm. Then, the left eye pupil center coordinates (u1, v1) to be mapped are input into the left eye gaze estimation fitting curve equation system to obtain the left eye gaze display screen coordinates (x1, y1) corresponding to the left eye pupil center. Similarly, the right eye pupil center coordinates (u2, v2) to be mapped are input into the right eye gaze estimation fitting curve equation system to obtain the right eye gaze display screen coordinates corresponding to the right eye pupil center. (x2, y2), then calculate the average of the coordinates (x1, y1) and (x2, y2) of the left and right eyes' gaze on the display screen to obtain the human eye gaze coordinates (x0, y0), where x0 = (x1 + x2) / 2, y0 = (y1 + y2) / 2. That is, calculate the average of the coordinates of the points where the pupils of both eyes fall on the display screen, and thus determine which display screen the commander is gazing at and which area of ​​the screen based on the human eye gaze coordinates (x0, y0). The steps involved in obtaining the equations for the fitting curves of the left and right eye gaze estimation are as follows: Test subjects are instructed to gaze sequentially at several designated points on each display screen (e.g., the four corners and center of the screen). While gazing at each designated point, eye images of the left and right eyes are obtained. The pupil center coordinates of the left and right eyes are obtained using a pupil detection algorithm. Then, based on the mapping relationship established between the left eye pupil center coordinates and the corresponding designated point coordinates, the fitting coefficients are determined using a curve fitting algorithm (a well-known algorithm), resulting in the left eye gaze estimation fitting curve equation set. Similarly, based on the mapping relationship established between the right eye pupil center coordinates and the corresponding designated point coordinates, the fitting coefficients are determined using a curve fitting algorithm (a well-known algorithm), resulting in the right eye gaze estimation fitting curve equation set.

[0027] In practice, the process of deriving the equations for the left and right eye gaze estimation fitting curves is the same, but the resulting equation sets differ. The equation sets for the left and right eye gaze estimation fitting curves are generally designed as the following quadratic polynomial function equation sets: , In the above formula, x and y are the horizontal and vertical coordinates of the coordinates of the left (or right) eye gazing at the display screen, respectively; u and v are the horizontal and vertical coordinates of the pupil center of the left (or right) eye to be mapped, respectively; and a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, and b5 are fitting coefficients.

[0028] In step 5), an eye-tracking behavior recognition algorithm is used to perform eye-tracking behavior recognition. This algorithm is based on a trained deep learning model to improve accuracy. The input parameters of the deep learning model include eye movement features such as eye velocity, acceleration, and orientation extracted from multiple consecutive eye images (eye image sequence). Then, the deep learning model determines eye movement behavior based on the eye movement features. Eye movement behavior includes long fixation, short fixation, blinking, eye saccades, and left and right smooth movements. The output of the deep learning model includes multiple instructions. Prolonged fixation is defined as the duration during which both eyes remain unchanged for a period of time that is greater than or equal to the second threshold (e.g., 5 seconds). When the duration of fixation of both eyes is greater than the first threshold (e.g., 1 second) and less than the second threshold (e.g., 5 seconds), such as 2 to 4 seconds, it is considered short fixation. Long-term fixation, short-term fixation, blinking, left smooth movement, and right smooth movement each correspond to one instruction; Eye twitching is a natural phenomenon of the human eye and does not correspond to any command. The command specifies one type of eye movement behavior other than saccades as the corresponding end instruction. This type of eye movement behavior can be any other eye movement behavior besides the one mentioned above, such as closing the eyes for a certain period of time. Here, "end" means ending the entire eye movement interaction process.

[0029] In practice, the deep learning model is first trained on a certain number of samples. The training effect can be evaluated based on the loss function. When the loss value tends to stabilize, the training is considered complete, and the trained deep learning model can then be put into use. Of course, in addition to the loss function, other standards can also be used to evaluate the training effect, without any limitations.

[0030] The present invention has the following advantages: This invention overcomes the shortcomings of traditional physical input devices (such as mice and keyboards) in multi-screen environments, such as cumbersome operation and sluggish focus switching, by constructing a cross-screen gaze mapping model and an eye-tracking behavior recognition mechanism. This invention can monitor the commander's gaze trajectory in real time, achieving seamless automatic switching and precise locking of the interactive focus across different display screens. Combined with eye-tracking features, it intelligently recognizes the commander's operational intentions to quickly trigger commands, constructing a highly efficient, non-contact command interaction mode that is "what you see is what you get." This invention not only effectively reduces the cognitive load and physiological fatigue of commanders under high-pressure multi-tasking conditions but also overcomes the operational lag and misoperation defects of traditional physical input devices (such as mice and keyboards), significantly improving the decision-making response speed and mission execution reliability in the command cabin environment.

[0031] 1. Enhanced Command Efficiency and Real-Time Response: Through eye-tracking technology, commanders can directly switch display screens by looking at the screen to quickly locate the required information, eliminating the need for traditional peripherals such as a mouse and keyboard. This significantly reduces manual switching time, improves task execution efficiency and information response speed, and adapts to complex, multi-tasking command scenarios. In emergencies, commanders can switch and issue commands by looking at the screen, shortening response time and improving overall decision-making efficiency. Issuing commands and triggering operations through eye movements (such as blinking and eye movements) is a major technological innovation of this invention, achieving a contactless operation method and improving operational convenience and response speed.

[0032] 2. Natural and efficient human-computer interaction experience: Eye-tracking interaction provides a more natural interaction method, conforms to ergonomic design, and reduces reliance on peripherals such as mice and keyboards for operation. Commanders can complete command execution simply by gazing and simple eye movements (such as blinking and fixation). This invention can efficiently respond to the commander's intentions through gaze estimation and eye movement recognition, achieving natural switching between screens and information focus, as well as command operation and issuance, reducing errors or delays caused by manual operation.

[0033] 3. Accuracy and intuitiveness of multi-screen information processing: The system of this invention can accurately identify the commander's gaze point and automatically ensure that the commander can always see the display screen where the current task or emergency information is located in the multi-screen command environment. The multi-screen collaborative display control module can support multi-task parallel operation without the commander having to manually switch display screens, thus improving task processing efficiency.

[0034] 4. Reduce commander fatigue and improve long-term comfort: Traditional mouse and keyboard interaction methods can easily cause fatigue after prolonged use, especially in complex multi-screen command environments where frequent switching is even more cumbersome. The eye-tracking interaction method provided by this invention reduces the frequency and intensity of hand operations, allowing commanders to maintain a high level of comfort after long periods of work and effectively reducing operational fatigue. Furthermore, this invention can detect commander fatigue levels, effectively mitigating decision-making errors caused by fatigue and other human factors.

[0035] 5. Enhance mission safety and accuracy: In high-pressure command scenarios, this invention can reduce operational errors and ensure mission execution safety through precise gaze estimation and eye movement behavior recognition. It avoids information loss or erroneous instructions caused by accidental touches, which not only improves the accuracy of operation, but also enhances the adaptability to complex command tasks through automatic perception and real-time feedback.

[0036] The above description describes the preferred embodiments of the present invention and the technical principles applied thereto. For those skilled in the art, any obvious changes such as equivalent transformations or simple substitutions based on the technical solutions of the present invention, without departing from the spirit and scope of the present invention, shall fall within the protection scope of the present invention.

Claims

1. An eye-tracking interaction system for multiple screens in a command cabin, characterized in that, It includes multiple display screens, a multi-screen collaborative display control module, an infrared camera, an interactive control module, and a communication module, among which: Each of the aforementioned display screens independently displays the operation screen; The infrared camera is placed on the table in front of the commander to acquire images of the commander's eyes and transmit the images to the interactive control module. The interactive control module is used to perform gaze estimation based on eye images to switch to the display screen that the commander is gazing at and the image area on the display screen and lock the image area by means of the multi-screen collaborative display control module, and to perform eye movement behavior recognition based on multiple consecutive eye images to obtain instructions and to make the locked display screen execute instructions by means of the multi-screen collaborative display control module.

2. The eye-tracking interaction system for multi-screen command cabins as described in claim 1, characterized in that, It also includes a communication module, wherein when the instruction triggers a related device, the interactive control module, while enabling the locked display screen to execute the instruction through the multi-screen collaborative display control module, simultaneously sends the instruction to the related device for execution through the communication module.

3. The eye-tracking interaction system for multi-screen command cabins as described in claim 1, characterized in that, The interactive control module is also used to obtain eye movement frequency and eye usage duration based on multiple consecutive eye images, and to determine whether eye fatigue is about to occur based on eye movement frequency and eye usage duration, so as to make timely rest reminders and avoid eye fatigue for commanders.

4. The eye-tracking interaction system for multi-screen command cabins as described in claim 1, characterized in that, The eye movements include prolonged fixation, short-term fixation, blinking, saccades, and left and right smooth movements. Each of these corresponds to a command, while saccades do not correspond to any command.

5. An eye-tracking interaction method for multiple screens in a command cabin, implemented based on the eye-tracking interaction system for multiple screens in a command cabin as described in any one of claims 1 to 4, characterized in that, Including the following steps: 1) The system starts up and enters the operation state; 2) Each of the aforementioned display screens displays the operation screen, and each operation screen is presented in the commander's field of vision; 3) The infrared camera acquires an image of the commander's eyes and transmits the image to the interactive control module; 4) The interactive control module estimates the line of sight based on the eye image and determines whether the commander is staring at a certain area of ​​the display screen. If not, proceed to the next step. If so, the multi-screen collaborative display control module locks the display screen that the commander is staring at and the area of ​​the screen on the display screen, and the area of ​​the screen is magnified or the key information is highlighted. Proceed to the next step. 5) The interactive control module performs eye movement behavior recognition based on multiple consecutive eye images to determine whether the commander has issued an instruction: if not, return to step 3); if so, obtain the instruction corresponding to this eye movement behavior, and use the multi-screen collaborative display control module to make the locked display screen execute the instruction.

6. The eye-tracking interaction method for multi-screen command cabins as described in claim 5, characterized in that, In step 5), if the instruction is to trigger the relevant device, the interactive control module, while executing the instruction on the locked display screen through the multi-screen collaborative display control module, sends the instruction to the relevant device for execution through the communication module.

7. The eye-tracking interaction method for multi-screen command cabins as described in claim 5, characterized in that, In step 4), a gaze estimation algorithm is used to estimate the gaze. This algorithm includes a pupil detection algorithm and a two-dimensional gaze estimation algorithm, wherein: The pupil detection algorithm includes the following steps: first, preprocessing the eye image; then, using the Canny edge detection algorithm to extract edge information and obtain multiple contours; then, using a preset set of pupil geometric feature thresholds for preliminary screening; selecting contours that may be pupil contours from all obtained contours as candidate contours; then, performing ellipse fitting on each candidate contour using the least squares method to obtain a candidate fitting contour that is similar to the shape of the pupil; finally, based on the confidence level, selecting the candidate fitting contour with the highest confidence level as the pupil contour to obtain the pupil center coordinates. The two-dimensional gaze estimation algorithm includes the following steps: using the pupil detection algorithm to obtain the coordinates of the left eye pupil center and the right eye pupil center; then inputting the left eye pupil center coordinates to be mapped into the left eye gaze estimation fitting curve equation set to obtain the left eye gaze display screen coordinates corresponding to the left eye pupil center; and inputting the right eye pupil center coordinates to be mapped into the right eye gaze fitting curve equation set to obtain the right eye gaze display screen coordinates corresponding to the right eye pupil center; then calculating the average of the left eye gaze display screen coordinates and the right eye gaze display screen coordinates to obtain the human eye gaze coordinates; and thus determining which display screen the commander is gazing at and which area of ​​the screen is being gazed at based on the human eye gaze coordinates.

8. The eye-tracking interaction method for multi-screen command cabins as described in claim 7, characterized in that, The steps for obtaining the equation set of the left eye gaze estimation fitting curve and the equation set of the right eye gaze estimation fitting curve are as follows: The tester is instructed to gaze sequentially at several designated points on each of the aforementioned display screens. While gazing at each designated point, eye images of the left and right eyes are obtained. The pupil detection algorithm is used to obtain the coordinates of the center of the left and right pupils. Then, based on the mapping relationship established by the coordinates of the center of the left and right pupils to be mapped and the coordinates of the corresponding gazed designated points, the fitting coefficients are determined by a curve fitting algorithm, thereby obtaining the equation sets of the left eye gaze estimation fitting curves and the equation sets of the right eye gaze estimation fitting curves.

9. The eye-tracking interaction method for multi-screen command cabins as described in claim 8, characterized in that, The equation set for the left eye gaze estimation fitting curve and the equation set for the right eye gaze estimation fitting curve are quadratic polynomial function equation sets.

10. The eye-tracking interaction method for multi-screen command cabins as described in claim 5, characterized in that, In step 5), an eye-tracking behavior recognition algorithm is used to perform eye-tracking behavior recognition. This algorithm is based on a trained deep learning model, wherein: The input parameters of the deep learning model include eye movement features such as eye speed, acceleration, and direction extracted from multiple consecutive eye images. Then, the deep learning model determines eye movement behavior based on the eye movement features. The eye movement behavior includes long-term fixation, short-term fixation, blinking, eye saccades, and left and right smooth movements. The output of the deep learning model includes multiple instructions. Prolonged fixation is defined as the duration during which both eyes remain unchanged for a period of time that is greater than or equal to the second threshold. When the duration of fixation of both eyes is greater than the first threshold but less than the second threshold, it is considered short-term fixation. Long-term fixation, short-term fixation, blinking, left smooth movement, and right smooth movement each correspond to one instruction; Eye saccades do not correspond to any commands; Specify an end command for one of the eye movement behaviors other than saccades.

Images