Mouse control method and device based on eye posture recognition
By integrating eye posture sensors and ambient light sensing technology on head wearable devices, the problems of insufficient sensitivity and high power consumption of eye mouse control in the prior art are solved, convenient and accurate mouse operation is achieved, and disabled people are enhanced Independence.
Patent Information
- Application Number
- CN202411049216.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-15
AI Technical Summary
The existing eye-based mouse control technology has problems such as insufficient head rotation sensitivity, high equipment power consumption, and frequent blinking interference, resulting in high hardware requirements and inconvenient use.
The eye posture sensor on the head wearable device is used to locate the eye pupil position, obtain and process the eye action signal, output control signals, realize the mouse operation function, and transmit it to the electronic device through Bluetooth, combining ambient light sensing and color sensing technology. , reduce blinking interference.
It improves the convenience and accuracy of operation of disabled people, reduces dependence on traditional mice, reduces equipment power consumption and misoperation, and enhances independence.
Smart Images

Figure CN118963567A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mouse control method and device based on eye gesture recognition. Background Art
[0002] Since computers are widely used, they need to be controlled to function. However, some disabled people cannot control computers with their hands due to physical defects. In this scenario, the computer is controlled by the movements of other normal parts of the body, and the movement instructions of other parts need to be converted into mouse instructions to achieve the purpose of controlling the computer. At present, the head and eyeballs are used to replace the traditional mouse to achieve the purpose of controlling the computer. However, the sensitivity of head movement is not enough and it is easy to get tired after a long time. The movement of the eyeball is more flexible, but the existing video recognition method is mostly used, which requires the eyeball to be photographed and then the eyeball movement position in the video is analyzed. The computing power is too large and the equipment power consumption is high, resulting in high hardware requirements. In addition, after the eyeballs move for a period of time, they will become tired and blink frequently, which interferes with eye recognition, including habitual blinking. Summary of the invention
[0003] In view of the above technical defects, the purpose of the present invention is to provide a mouse control method and device based on eye movement, which is suitable for disabled people and is achieved through the following technical solution.
[0004] A mouse control method based on eyeball rotation is provided, wherein an eyeball posture sensor is configured on a head-worn device so that the eyeball posture sensor is located in front of the eye, and the method comprises the following steps:
[0005] Step 1: The eyeball posture sensor locates the pupil position of the eyeball;
[0006] Step 2, obtaining feedback signals of eye movements;
[0007] Step 3, extracting the correct eye movement signal from the feedback signal and sending it to the single chip microcomputer for processing, and outputting the control signal;
[0008] Step 4: The signal is reflected to the screen of the electronic device.
[0009] A control device based on eye movement is configured on a head wearable device. The control device includes a main control chip located on a circuit board, an eye posture sensor, a button, a battery, and a power supply. The eye posture sensor on the circuit board is used to check and measure eye movements, can automatically learn the current environmental functions (including ambient light, distance, etc.) at all times, and transmit the generated electrical signals to the main control module for processing; the main control module is used to convert the electrical signals generated by eye movements into instructions and output them to a port, connect the head wearable device, and then transmit them to the electronic device via Bluetooth; the mouse click, double-click, and scroll wheel functions are realized.
[0010] The beneficial effects of the present invention are:
[0011] 1. Since mouse operation is realized by simple eye movements, people with disabilities can easily master and use this technology, which improves their convenience and autonomy in life;
[0012] 2. Overcome the interference and misoperation caused by frequent blinking, and improve accuracy and practicality. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 is a flow chart of a mouse control method according to an embodiment of the present invention.
[0014] Figure 2 Schematic diagram of a method for locating an eye pupil according to an embodiment of the present invention.
[0015] Figure 3 Schematic diagram of a method for locating an eye pupil when the sensor of an embodiment of the present invention is in other positions.
[0016] Figure 4 It is a schematic diagram of the structure and principle of a head wearable device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0017] The present invention will be further described below in conjunction with the embodiments. The embodiments described are only a part of the present invention. These embodiments are only used to explain the present invention and do not constitute any limitation to the scope of the present invention.
[0018] Disabled people have certain physical defects, and their disabled parts are not the same. This device is designed to help people with limited hand function, high paraplegia, cerebral palsy, muscular dystrophy, multiple sclerosis, amyotrophic lateral sclerosis and spinal cord injury, so that users with different abilities, different diseases and different communication needs can operate the device with their eyes. This invention helps to enhance the independence of disabled people and reduce their dependence on others in daily life.
[0019] like Figure 1 As shown, a mouse control method based on eyeball rotation is provided, wherein an eyeball posture sensor is configured on a head-worn device so that the eyeball posture sensor is located in front of the eye, and the method comprises the following steps:
[0020] S1, eyeball posture sensor locates the pupil position;
[0021] S2, obtaining feedback signals of eye movements;
[0022] S3, extracting the correct eye movement signal from the feedback signal to the single chip microcomputer for processing, and outputting the control signal;
[0023] S4. The signal is reflected to the screen of the electronic device.
[0024] like Figure 2-3 As shown, the step 1 includes a method for locating the pupil of the eye, including:
[0025] Step 11: The eyeball posture sensor obtains the distance value and the angle value of the pupil, and sets them as the distance reference value and the angle reference value;
[0026] First, we need to determine the center coordinates of the pupil. The coordinates of the center of the pupil are usually calculated by finding the geometric center of the points within the boundary, which can be simply understood as the average of the x and y coordinates of all boundary points. The specific formula is:
[0027] [P_{x}=\frac{\sum_{i}^{n}x_i}{n}]
[0028] [P_{y}=\frac{\sum_{i}^{n}y_i}{n}]
[0029] Where (P_{x}) and (P_{y}) are the x and y coordinates of the pupil center, (x_i) and (y_i) are the coordinates of points on the pupil boundary, and (n) is the total number of boundary points.
[0030] Step 12: The eyeball posture sensor actually obtains the distance value and angle value of the pupil, compares them with the distance reference value and the angle reference value, and obtains the distance change value and the angle change value;
[0031] After obtaining the x and y coordinates of the pupil center, the eye posture sensor begins to obtain the position and angle of the eye, and compares these real-time positions and angles with the x and y coordinates of the pupil center to obtain feedback signals of eye movement.
[0032] Step 13: Convert the distance change value and the angle change value into a mouse control signal.
[0033] like Figure 2 and 3 As shown, the method is applicable to both the sensor at the forward position of the eyeball and other positions.
[0034] like Figure 2 As shown, in an embodiment of the present invention, when worn for the first time, the control device is initialized first, and the distance h from the sensor 10 to the eyeball and the angle a formed by the sensor and the surface of the eyeball are calculated as reference values.
[0035] like Figure 3 When the head-mounted device is worn again or each time, the position of sensor 10 changes, and the sensor 10' moves to the position of sensor 10, the control device is initialized, and the distance h' and angle value a of the pupil obtained by the eyeball posture sensor are re-acquired and set as reference values.
[0036] In the embodiment of the present invention, the sensor 10 is an infrared sensor.
[0037] After initialization is completed, the control device starts to output control signals, which are then transmitted to the electronic device via Bluetooth, realizing the mouse click, double-click, and scroll wheel functions.
[0038] During the initialization process, the user's eyeballs are guided to rotate in a circle to obtain the change thresholds of the user's eyeball distance value and angle value.
[0039] After the control device starts working, it determines whether the eye movement is a correct eye movement signal, obtains the distance value and angle value generated by the eye movement in real time, compares them with the change threshold, sets the change value that falls within the change threshold as a valid value, and determines it as a correct eye movement.
[0040] In the embodiment of the present invention, the method further includes filtering out the interference of blinking action on mouse control, specifically: implementing filtering blinking time adjustment, monitoring eye movements in real time, and identifying blinking actions, which is achieved by capturing signals or using sensors.
[0041] When identifying a blinking action, a blink time filtering algorithm is used to dynamically adjust the threshold, effectively reducing the impact of normal blinking on eye switching.
[0042] Head-worn devices use color sensing and ambient light sensing: Through RGB color sensing and ambient light sensing (ALS) technology, the ambient light is detected and the sensitivity of the device is adjusted. It can distinguish between light changes caused by blinking and changes in ambient light, thereby improving recognition accuracy.
[0043] In the embodiment of the present invention, a head-mounted wearable device is configured, such as a helmet, glasses, a headband, etc. A control device is provided on the head-mounted wearable device, and the control device is located in front of the glasses, usually not directly in front of the eyes without affecting the sight line.
[0044] The head-mounted device is equipped with multiple light sensors, such as RGB sensors, ultraviolet and infrared blocking filters.
[0045] Application of RGB sensor and UV and IR blocking filters: Each RGBC channel is equipped with UV and IR blocking filters, which can block specific wavelengths of light and reduce interference caused by non-visual intentions. The RGB sensor can detect the light intensity of the three basic colors of red (R), green (G), and blue (B) in the environment. By analyzing the ratio of these three colors of light, the device can more accurately understand the color of the environment, thereby better distinguishing the changes in environmental color from the interference caused by blinking;
[0046] Digital ambient light and color sensing: Using high-precision digital sensors to sense ambient light and color can improve adaptability to environmental changes and the ability to recognize small light changes caused by blinking. Ambient light and RGB color sensors usually have programmable gain and integration time. This means that they can dynamically adjust the sensitivity (gain) and sampling time (integration time) according to different lighting conditions. Doing so ensures that the sensor can achieve optimal performance in different environments, thereby improving the accuracy and stability of the device under various lighting conditions.
[0047] In an embodiment of the present invention, a control device is configured on a head-mounted device, wherein:
[0048] like Figure 4 As shown, the head wearable device includes: a Bluetooth communication module, a motion sensing gyroscope, a socket and a main control device;
[0049] The eye posture sensor is used to monitor the eye movement / movement trajectory including the movement path and direction. According to the body movement trajectory, the motion sensing gyroscope generates continuous electrical signals, which are processed by the processor to generate control command signals.
[0050] The main control device and the socket are both connected to the Bluetooth communication module, and the plug is connected to the socket of the head wearable device through the matching connection, and then connected to the Bluetooth communication module;
[0051] The Bluetooth communication module is connected to an external intelligent terminal or display terminal via Bluetooth, and the eye control switch realizes single-click, double-click, long-press and drag operations on the intelligent terminal or display terminal through the eye movements;
[0052] The main control device generates a corresponding cursor on the screen according to the collected head movement data. The main control device includes a main control chip located on a circuit board, an eye posture sensor, a button, a battery, and a power supply. The eye posture sensor on the circuit board is used to check and measure eye movements, can automatically learn the current environmental functions at all times, and transmit the generated electrical signals to the main control module for processing; the main control module is used to convert the electrical signals generated by eye movements into instructions and output them to the port, connect the head wearable device, and then transmit them to the electronic device via Bluetooth; realize the mouse click, double-click, and scroll wheel functions.
[0053] The eye gesture sensor is an integration of sensors, which includes an infrared sensor, and the infrared sensor includes an infrared transmitter and an infrared receiver:
[0054] The infrared sensor is used to determine the distance between the control device and the pupil of the eye, locate the position of the pupil, and obtain the distance change and position change of the pupil in real time.
[0055] Start the head-mounted device, initialize the control device first, and use the infrared sensor to calculate the distance and angle value from the current sensor to the pupil as a reference value.
[0056] The position and angle of the infrared sensor can be adjusted to meet the needs of different users.
[0057] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A mouse control method based on eye gesture recognition, characterized in that: Configuring an eye posture sensor on a head-mounted device so that the eye posture sensor is located in front of the eye includes the following steps: Step 1: The eyeball posture sensor locates the pupil position of the eyeball; Step 2, obtaining feedback signals of eye movements; Step 3, extracting the correct eye movement signal from the feedback signal and sending it to the single chip microcomputer for processing, and outputting the control signal; Step 4: The signal is reflected to the screen of the electronic device.
2. The mouse control method according to claim 1, characterized in that: The step 1 includes a method for locating the pupil of the eye, including: Step 11: The eyeball posture sensor obtains the distance value and angle value of the pupil, and sets them as the distance reference value and the angle reference value; Step 12: The eyeball posture sensor actually obtains the distance value and angle value of the pupil, compares them with the distance reference value and the angle reference value, and obtains the distance change value and the angle change value; Step 13: Convert the distance change value and the angle change value into a mouse control signal.
3. The mouse control method according to claim 2, characterized in that: In step 2, the feedback signal of the eye movement is processed by circuit analysis, and the correct eye movement signal is sent to the single-chip microcomputer for processing, which outputs a control signal, cooperates with the head wearable device and reflects the control signal to the screen of the electronic device to realize the mouse click, double-click, and scroll wheel functions.
4. The mouse control method according to claim 2, characterized in that: When you put on the head-mounted device again, initialize the device, re-acquire the pupil distance and angle values obtained by the eye posture sensor, and set them as reference values.
5. The mouse control method according to claim 1, characterized in that: In step 3, determining the correct eye movement signal of the eyeball comprises the following steps: Step 31: First guide the user's eyeball to rotate a circle to obtain the change threshold of the eyeball distance value and angle value; Step 32: Obtain the distance value and angle value generated by the eye movement in real time, compare them with the change threshold, set the change value that falls within the change threshold as a valid value, and judge it as a correct eye movement.
6. The mouse control method according to claim 1, characterized in that: It also includes filtering out the interference of blinking action on mouse control, specifically: implementing blinking time adjustment, real-time monitoring of eye movements, and identifying blinking actions, which is achieved by capturing signals or using sensors. When a blink action is identified, a blink time filtering algorithm is used to dynamically adjust the threshold, effectively reducing the impact of normal blinking on eye switching.
7. The mouse control method according to claim 6, characterized in that: The head-mounted device also uses color sensing and ambient light sensing. Through RGB color sensing and ambient light sensing (ALS) technology, it detects ambient light and adjusts the sensitivity of the device. It can distinguish between light changes caused by blinking and changes in ambient light, thereby improving recognition accuracy.
8. The mouse control method according to claim 7, characterized in that: The head-worn device is equipped with RGB sensors, ultraviolet and infrared blocking filters. Each RGBC channel is equipped with ultraviolet and infrared blocking filters, which can block light of specific wavelengths and reduce interference caused by non-visual intentions. The RGB sensor can detect the light intensity of three basic colors in the environment, and distinguish the environmental color changes from the interference caused by blinking by analyzing the ratio of these three colors of light.
9. A control device based on eye gesture recognition, characterized in that: For implementing the mouse control method according to any one of claims 1 to 8, a control device is configured on a head-worn device, wherein: The head wearable device includes: a Bluetooth communication module, a motion sensing gyroscope, a socket and a main control device; The motion sensing gyroscope is used for the motion trajectory including the motion path and the motion direction. According to the motion trajectory of the body, the motion sensing gyroscope generates a continuous electrical signal, and the processor processes the electrical signal to generate a signal of a control instruction; The main control device and the socket are both connected to the Bluetooth communication module, and the plug is connected to the socket of the head wearable device through the matching connection, and then connected to the Bluetooth communication module; The Bluetooth communication module is connected to an external intelligent terminal or display terminal by Bluetooth, and the eye control switch realizes single-click, double-click, long-press and drag operations on the intelligent terminal or display terminal through the eye movements; The main control device generates a corresponding cursor on the screen according to the collected head movement data. The main control device includes a main control chip located on a circuit board, an eye posture sensor, a button, a battery, and a power supply. The eye posture sensor on the circuit board is used to check and measure eye movements, can automatically learn the current environmental functions at all times, and transmit the generated electrical signals to the main control module for processing; the main control module is used to convert the electrical signals generated by eye movements into instructions and output them to the port, connect the head wearable device, and then transmit them to the electronic device via Bluetooth; realize the mouse click, double-click, and scroll wheel functions.
10. The control device according to claim 9, characterized in that: The eyeball gesture sensor is an integration of sensors, which includes an infrared sensor, wherein: The infrared sensor is used to determine the distance between the control device and the pupil of the eye, locate the position of the pupil, and obtain the distance change and position change of the pupil in real time.
