An interaction method for ALS patients based on eye-controlled gaze-triggered communication interface

Abstract

The present invention is applicable to the field of information interaction technology, and provides an interaction method for ALS patients based on an eye-controlled gaze-triggered communication interface, comprising the following steps: eye tracking: collecting facial image data of the patient through a camera, extracting facial and eye landmarks, and performing image processing on the eye area to determine the gaze direction; gaze triggering: pre-setting the coordinate area of ​​the interactive control, and determining in real time whether the patient's gaze point falls within the control area. If the gaze point remains within the preset time, the corresponding operation is triggered; execution compensation mechanism: performing real-time compensation for head movement and illumination changes, and correcting the gaze point coordinates; interface design: designing an interactive interface that includes daily communication functions, and broadcasting the control content by voice. The present invention proposes an efficient, convenient, and easy-to-use eye-controlled gaze-triggered communication interface for ALS patients, aiming to reduce the learning cost and operational complexity of patients, and is of great significance to improving the quality of life of ALS patients.

Description

Technical Field

[0001] The present invention belongs to the field of information interaction technology, and in particular relates to an interaction method for ALS patients based on an eye-controlled gaze-triggered communication interface. Background Art

[0002] ALS, also known as amyotrophic lateral sclerosis (ALS), causes patients to gradually lose their ability to move as the disease progresses, eventually becoming paralyzed. However, the patient's eye muscles are usually not affected by the disease, which makes communication through eye movements possible.

[0003] Currently, assistive communication devices for ALS patients are available on the market, such as eye tracking devices. These rely primarily on eye tracking technology, enabling communication functions such as text input and speech synthesis through a display module, eye detection unit, and data processing module. Patients can control the devices through eye movements such as gazing, fixation, or blinking, providing an important communication channel for ALS patients.

[0004] However, existing assistive communication devices still have many shortcomings. In terms of interactive experience, the device's interactive interface is complex and the function settings are cumbersome. Patients need to spend a lot of time learning and adapting, and the operation is difficult. In terms of performance, the response speed and accuracy are poor, and delays or misoperations are prone to occur, which greatly affects communication efficiency. In addition, the environmental adaptability is poor. In complex environments such as light changes and background interference, the performance of the device will be significantly limited. In terms of economic cost, the equipment is expensive, making it difficult to widely use. At the same time, existing eye control detection algorithms, such as those based on pupil center and sclera-iris edge methods, have problems such as insufficient accuracy, large influence of lighting, and large head errors. Their robustness in complex environments is poor, further restricting the improvement of device performance.

[0005] In view of the above-mentioned defects in the prior art, the present invention proposes an interaction method for ALS patients based on an eye-controlled gaze-triggered communication interface. Summary of the Invention

[0006] The purpose of the present invention is to provide an interaction method for ALS patients based on an eye-controlled gaze-triggered communication interface, aiming to solve the problems raised in the above-mentioned background technology.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A method for interacting with ALS patients based on an eye-controlled gaze-triggered communication interface, comprising the following steps:

[0009] Step 1: Eye tracking;

[0010] The camera collects the patient's facial image data, extracts facial and eye landmarks, and processes the eye area to determine the gaze direction;

[0011] Step 2: Gaze trigger;

[0012] Pre-set the coordinate area of ​​the interactive control, and determine in real time whether the patient's gaze point falls within the control area. If the gaze point remains within the preset time, the corresponding action is triggered;

[0013] Step 3: Implement compensation mechanism;

[0014] Compensate for head movement and lighting changes in real time to correct gaze point coordinates;

[0015] Step 4: Interface design;

[0016] Design an interactive interface that includes daily communication functions and announce the control content through voice.

[0017] Furthermore, the specific steps of step 1 are as follows:

[0018] Step 11: facial and eye landmark extraction;

[0019] The patient's facial image data was collected using a camera. The Dlib 68-point facial landmark detector was used to detect and track the patient's facial image captured by the camera to obtain facial landmark information. The focus was then shifted to 12 eye landmarks and the coordinates were extracted as regions of interest.

[0020] Step 12: Gaze direction judgment;

[0021] The extracted eye ROI is Gaussian blurred, and the eye frame is binarized based on the threshold determined by the Otsu algorithm to distinguish the pupil / iris and sclera areas. Based on the sclera region of interest method, the number of white pixels in the sclera area of ​​the left and right eyes is calculated on the binarized image to obtain the left gaze ratio. and right gaze ratio , and then calculate the average of the left and right eye gaze ratios, i.e., the side gaze ratio, to determine the gaze direction of the eyes;

[0022] Left gaze ratio The calculation formula is:

[0023] ;

[0024] Right gaze ratio The calculation formula is:

[0025] ;

[0026] The calculation formula for the side gaze ratio is:

[0027] ;

[0028] in: SG ratio represents the side gaze ratio;

[0029] The top-bottom ratio is calculated based on the eye landmarks. The top-bottom ratio is calculated as follows:

[0030] ;

[0031] in: represents the top-bottom ratio; U left The length of the line between the midpoint of the distance between eye landmarks 43 and 44 and the midpoint of the line between eye landmarks 42 and 45; U right The length of the line between the midpoint of the distance between eye landmarks 37 and 38 and the midpoint of the line between eye landmarks 36 and 39; B left The length of the line between the midpoint of the distance between eye landmarks 47 and 46 and the midpoint of the line between eye landmarks 42 and 45; B right The length of the line between the midpoint of the distance between eye landmarks 41 and 40 and the midpoint of the line between eye landmarks 36 and 39.

[0032] Furthermore, in step 12, the gaze direction is determined by:

[0033] If the real-time side gaze ratio value is greater than the preset central area left boundary threshold, it is determined that the gaze is on the left side of the interface;

[0034] If the real-time side gaze ratio value is less than the preset central area right boundary threshold, it is determined that the gaze is on the right side of the interface;

[0035] If the real-time side gaze ratio value is between the preset central area left boundary threshold and the preset central area right boundary threshold, the central area of ​​the gaze interface is determined;

[0036] If the real-time top-bottom ratio value is less than the demarcation threshold, the bottom sector of the gaze interface is determined; otherwise, the top sector of the gaze interface is determined.

[0037] Furthermore, the specific steps of step 2 are as follows:

[0038] Step 21: Interactive control area setting;

[0039] Pre-set the coordinate area of ​​interactive controls on the screen to clearly define the position and size of each control on the screen;

[0040] Step 22: Gaze trigger determination;

[0041] Determine in real time whether the patient's gaze point (Gaze X, Gaze Y) falls within a certain control area. The judgment conditions are:

[0042] ;

[0043] in: Indicates the left boundary X coordinate of the control on the screen; Indicates the Y coordinate of the upper boundary of the control on the screen; Gaze X indicates the X coordinate of the patient's gaze on the screen; Gaze Y indicates the Y coordinate of the patient's gaze on the screen; Displays the width of the control; The height of the display control;

[0044] If the gaze point remains within the control area for 5 seconds, the corresponding action is triggered.

[0045] Furthermore, the specific steps of step 3 are as follows:

[0046] Step 31: Head motion compensation;

[0047] During the calibration phase, the coordinates of the glabella point are recorded as a reference for measuring the movement of the head from its original position; X and Y Directional movement difference Hm x and Hm y , the formula is:

[0048] ;

[0049] in: 、 is the eyebrow center position during calibration (initial value); g x 、 g y is the coordinate of the eyebrow center detected in the current frame;

[0050] Step 32: Correction of gaze point coordinates;

[0051] Using the calculated head movement difference, the predicted gaze point coordinates ( ) to determine the final gaze position on the screen in the scene ( ), the adjustment formula is:

[0052] ;

[0053] Step 33: Light compensation;

[0054] The light compensation algorithm is used to automatically adjust the binarization threshold according to the histogram distribution of the image and perform illumination normalization on the image.

[0055] Furthermore, in step 4, PyQt5 and Qt Designer are used to design and layout the interface, and the QTextToSpeech module is used to convert the text content into voice output to broadcast the control content.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This invention proposes an efficient, convenient, and easy-to-use eye-controlled gaze-triggered communication interface for ALS patients, aiming to reduce learning costs and operational complexity. First, by utilizing the Dlib 68-point facial landmark detector to accurately capture and track key eye features, the interface effectively extracts facial eye information and tracks gaze, improving the accuracy and robustness of eye movement signal recognition. Second, in terms of interactive design, when a patient gazes at a control on the interface for 5 seconds, the system automatically triggers the control's content and announces it in real-time via voice, accurately expressing the patient's needs and providing an intuitive and easy-to-use daily communication method. Furthermore, the invention utilizes advanced algorithms and hardware technologies to not only improve the speed and accuracy of eye control signal recognition, reduce misoperations, and enhance communication efficiency, but also ensure stable operation in complex environments through mechanisms such as light compensation and head motion compensation. Finally, in terms of cost control, the invention effectively reduces equipment costs while ensuring full functionality through optimized hardware design and software algorithms, thereby alleviating the financial burden on patients' families. In summary, the invention has significant implications for improving the quality of life for ALS patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Flow chart of the method of the present invention.

[0059] Figure 2 Facial landmark information obtained for the Dlib 68-point facial landmark detector.

[0060] Figure 3 These are the main stages of eye region image processing; (a) is the extracted eye region, (b) is the blurred eye region, and (c) is the binarized eye region.

[0061] Figure 4 Schematic diagrams of the top and bottom of the eye gaze interface; (a) is a schematic diagram of the top of the eye gaze interface, and (b) is a schematic diagram of the bottom of the eye gaze interface.

[0062] Figure 5 Interface design for daily communication for ALS patients. DETAILED DESCRIPTION

[0063] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be understood as limiting the scope of implementation of the present invention.

[0064] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0065] The present invention provides an interaction method for ALS patients based on an eye-controlled gaze-triggered communication interface, the flow chart of which is as follows: Figure 1 As shown, the method includes the following steps:

[0066] Step 1: Eye tracking;

[0067] Step 11: facial and eye landmark extraction;

[0068] The patient's facial image data was collected using a 1280×720 pixel camera. The Dlib 68-point facial landmark detector was used to detect and track the patient's facial image collected by the camera to obtain facial landmark information (a total of 68 facial landmark points, see Figure 2 ). Then the focus is shifted to 12 eye landmarks (6 for each eye) and the coordinates of these 12 eye landmarks are extracted as regions of interest (ROIs).

[0069] Step 12: Gaze direction judgment;

[0070] The extracted eye ROI is Gaussian blurred to effectively reduce the noise interference generated during the measurement process; then the eye frame is binarized based on the threshold determined by the Otsu's method. At this time, the pupil / iris (black) and sclera (white) areas are distinguished in the binary frame (see the eye area image processing results). Figure 3 Based on the sclera region of interest (sclera-ROI) method, the number of white pixels in the sclera area of ​​the left and right eyes is calculated on the binarized image to obtain the left gaze ratio. and right gaze ratio , and then calculate the average of the left and right eye gaze ratios (i.e., side gaze ratio) to determine the gaze direction of the eyes.

[0071] Left gaze ratio It is the number of non-zero pixels (white pixels) in the left half of the left eye area divided by the number of all pixels in the left half in the binary image of the left eye. The formula can be expressed as:

[0072] ;

[0073] Right gaze ratio It is the number of non-zero pixels (white pixels) in the right half of the right eye area divided by the number of all pixels in the right half of the right eye area in the binary image of the right eye. The formula can be expressed as:

[0074] ;

[0075] Side gaze ratio ( SG ratio ) is calculated as:

[0076] .

[0077] During the calibration phase, the patient is required to gaze at the left, center, and right areas of the interface (graphical user interface, GUI) in turn, and the system records the corresponding side gaze ratio ( SG ratio ) value, and statistically calculate the left boundary threshold of the central area and the right boundary threshold of the central area (horizontal direction: through SG ratio The user is looking at the left, center, or right side using the calibration threshold. Vertical direction: The polygon shape defined by the eye landmark points is used to determine whether the user is looking at the top or bottom. Threshold statistics: Based on the data collected during the calibration phase. SG ratio The data were collected, the mean and standard deviation were calculated, and the left and right boundary thresholds were set. When actually judging the patient's gaze direction, if the real-time side gaze ratio value is greater than the central area left boundary threshold, it is determined that the patient is looking at the left side of the interface; if the real-time side gaze ratio value is less than the central area right boundary threshold, it is determined that the patient is looking at the right side of the interface; if the real-time side gaze ratio value is between the central area left boundary threshold and the central area right boundary threshold, it is determined that the patient is looking at the center area of ​​the interface.

[0078] Calculate the top-to-bottom ratio based on eye landmarks ( ), the top-bottom ratio is calculated as:

[0079] ;

[0080] in: U left The length of the line between the midpoint of the distance between eye landmarks 43 and 44 and the midpoint of the line between eye landmarks 42 and 45; U right The length of the line between the midpoint of the distance between eye landmarks 37 and 38 and the midpoint of the line between eye landmarks 36 and 39; B left The length of the line between the midpoint of the distance between eye landmarks 47 and 46 and the midpoint of the line between eye landmarks 42 and 45; B rightThe length of the line between the midpoint of the distance between eye landmarks 41 and 40 and the midpoint of the line between eye landmarks 36 and 39. Figure 4 Shown are the top and bottom schematics of the patient gaze interface.

[0081] During the calibration phase, the patient gazes at the top and bottom of the graphical user interface (GUI). The system records the corresponding top-to-bottom ratio and determines a threshold. If the real-time top-to-bottom ratio is less than the threshold, the patient is considered to be gazing at the bottom sector of the interface; otherwise, the patient is considered to be gazing at the top sector of the interface.

[0082] Step 2: Gaze trigger;

[0083] Step 21: Interactive control area setting;

[0084] Pre-set the coordinate area of ​​interactive controls on the screen to clearly define the position and size of each control (such as a button) on the screen;

[0085] Step 22: Gaze trigger determination;

[0086] Determine in real time whether the patient's gaze point (Gaze X, Gaze Y) falls within a certain control area. The judgment conditions are:

[0087] ;

[0088] in: Indicates the left boundary X coordinate of the control on the screen; Indicates the Y coordinate of the upper boundary of the control on the screen; Gaze X indicates the X coordinate of the patient's gaze on the screen; Gaze Y indicates the Y coordinate of the patient's gaze on the screen; Displays the width of the control; Displays the height of the control.

[0089] When the gaze point enters a control area, the system will continuously detect whether the patient's gaze point remains stable in the control area. Set a gaze time of 5 seconds. If the gaze point remains in the control area for 5 seconds, the corresponding action will be triggered. If the patient's gaze point leaves the control area within 5 seconds, the timer will be reset and the gaze time will be calculated again.

[0090] Step 3: Execute the compensation mechanism (executed in real time before the gaze trigger is determined, ensuring that the corrected gaze point coordinates are used for region determination);

[0091] Step 31: Head motion compensation;

[0092] During the calibration phase, the coordinates of the glabella point are recorded and used as a reference for measuring head movements from the original position. During actual use, every head movement beyond that recorded during the calibration phase is fully compensated (up to 15%).

[0093] Calculate the head X and Y Directional movement difference Hm x and Hm y , the calculation formula is:

[0094] ;

[0095] in: 、 is the eyebrow center position during calibration (initial value); g x 、 g y is the coordinate of the eyebrow center detected in the current frame;

[0096] Step 32: Correction of gaze point coordinates;

[0097] The calculated head movement difference is used to adjust the predicted gaze point coordinates to accurately determine the final gaze position on the screen in the scene ( S x , S y ), the adjustment formula is:

[0098] .

[0099] Step 33: Light compensation;

[0100] To minimize the impact of lighting variations on landmark detection, a light compensation algorithm is employed. Histogram analysis provides insights into the overall image lighting conditions: a left-leaning histogram indicates a dark image with pixels concentrated in low grayscale values. a right-leaning histogram indicates a bright image with pixels concentrated in high grayscale values. a uniform histogram indicates relatively uniform image lighting. Based on the histogram distribution, the binarization threshold is dynamically adjusted to accommodate varying lighting conditions. The median of the histogram is calculated: the median is the middle value after sorting all pixel grayscale values. If the histogram median is below a certain threshold (e.g., 128), the image is generally dark, and the binarization threshold should be appropriately lowered. If the histogram median is above a certain threshold (e.g., 128), the image is generally bright, and the binarization threshold should be appropriately increased. This algorithm automatically adjusts the binarization threshold based on the image's histogram distribution and performs lighting normalization on the image, ensuring stable and accurate eye landmark detection under varying lighting conditions, thereby ensuring accurate gaze tracking and gaze triggering.

[0101] Step 4: Interface design;

[0102] Use PyQt5 and Qt Designer for interface design and layout (see Figure 5 ). The QTextToSpeech module is used to convert text content into speech output and announce the control content. The designed interface includes common functional controls that ALS patients may use in daily communication, such as "drink water," "eat," and "get up." When the patient gazes at a control area on the interface for 5 seconds, the control is triggered, and a small speaker will announce the meaning of the control, providing patients with an intuitive and convenient way to communicate.

[0103] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several variations and improvements can be made without departing from the concept of the present invention. These should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for interacting with ALS patients based on an eye-controlled gaze-triggered communication interface, characterized in that: The following steps are involved: Step 1: Eye tracking; The camera collects the patient's facial image data, extracts facial and eye landmarks, and processes the eye area to determine the gaze direction; Step 2: Gaze trigger; Pre-set the coordinate area of ​​the interactive control and determine in real time whether the patient's gaze point falls within the control area. If the gaze point remains within the preset time, the corresponding action is triggered; Step 3: Implement compensation mechanism; Compensate for head movement and lighting changes in real time to correct gaze point coordinates; Step 4: Interface design; Design an interactive interface that includes daily communication functions and announce the control content through voice; The specific steps of step 1 are as follows: Step 11: facial and eye landmark extraction; The patient's facial image data was collected using a camera. The Dlib 68-point facial landmark detector was used to detect and track the patient's facial image captured by the camera to obtain facial landmark information. The focus was then shifted to 12 eye landmarks and the coordinates were extracted as regions of interest. Step 12: Gaze direction judgment; The extracted eye ROI is Gaussian blurred, and the eye frame is binarized based on the threshold determined by the Otsu algorithm to distinguish the pupil / iris and sclera areas; Based on the sclera region of interest method, the number of white pixels in the sclera area of ​​the left and right eyes is calculated on the binarized image to obtain the left gaze ratio. and right gaze ratio , and then calculate the average of the left and right eye gaze ratios, i.e., the side gaze ratio, to determine the gaze direction of the eyes; Left gaze ratio The calculation formula is: ； Right gaze ratio The calculation formula is: ； The calculation formula for the side gaze ratio is: ； in: SG ratio represents the side gaze ratio; The top-bottom ratio is calculated based on the eye landmarks. The top-bottom ratio is calculated as follows: ； in: represents the top-bottom ratio; U left The length of the line between the midpoint of the distance between eye landmarks 43 and 44 and the midpoint of the line between eye landmarks 42 and 45; U right The length of the line between the midpoint of the distance between eye landmarks 37 and 38 and the midpoint of the line between eye landmarks 36 and 39; B left The length of the line between the midpoint of the distance between eye landmarks 47 and 46 and the midpoint of the line between eye landmarks 42 and 45; B right The length of the line between the midpoint of the distance between eye landmarks 41 and 40 and the midpoint of the line between eye landmarks 36 and 39.

2. The ALS patient interaction method based on an eye-controlled gaze-triggered communication interface according to claim 1, characterized in that: In step 12, the gaze direction is determined by: If the real-time side gaze ratio value is greater than the preset central area left boundary threshold, it is determined that the gaze is on the left side of the interface; If the real-time side gaze ratio value is less than the preset central area right boundary threshold, it is determined that the gaze is on the right side of the interface; If the real-time side gaze ratio value is between the preset central area left boundary threshold and the preset central area right boundary threshold, the central area of ​​the gaze interface is determined; If the real-time top-bottom ratio value is less than the demarcation threshold, the bottom sector of the gaze interface is determined; otherwise, the top sector of the gaze interface is determined.

3. The ALS patient interaction method based on an eye-controlled gaze-triggered communication interface according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 21: Interactive control area settings; Pre-set the coordinate area of ​​interactive controls on the screen to clearly define the position and size of each control on the screen; Step 22: Gaze trigger determination; Determine in real time whether the patient's gaze point (Gaze X, Gaze Y) falls within a certain control area. The judgment conditions are: ； in: Indicates the left boundary X coordinate of the control on the screen; Indicates the Y coordinate of the upper boundary of the control on the screen; Gaze X indicates the X coordinate of the patient's gaze on the screen; Gaze Y indicates the Y coordinate of the patient's gaze on the screen; Indicates the width of the control; Indicates the height of the control; If the gaze point remains within the control area for 5 seconds, the corresponding action is triggered.

4. The ALS patient interaction method based on an eye-controlled gaze-triggered communication interface according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 31: Head motion compensation; During the calibration phase, the coordinates of the glabella point were recorded to serve as a reference for measuring head movement from the original position; Calculate the head X and Y Directional movement difference Hm x and Hm y , the formula is: ； in: 、 is the eyebrow center position during calibration (initial value); g x 、 g y is the coordinate of the eyebrow center detected in the current frame; Step 32: Correction of gaze point coordinates; Using the calculated head movement difference, the predicted gaze point coordinates ( ) to determine the final gaze position on the screen in the scene ( ), the adjustment formula is: ； Step 33: Light compensation; The light compensation algorithm is used to automatically adjust the binarization threshold according to the histogram distribution of the image and perform illumination normalization on the image.

5. The ALS patient interaction method based on an eye-controlled gaze-triggered communication interface according to claim 1, characterized in that: In step 4, PyQt5 and Qt Designer are used to design and layout the interface, and the QTextToSpeech module is used to convert the text content into voice output to broadcast the control content.

Images