Eye movement state recognition method based on intelligent glasses, intelligent glasses and storage medium

Abstract

The invention relates to the technology of intelligent glasses, and discloses an eye movement state recognition method based on intelligent glasses, the intelligent glasses and a storage medium, and the method comprises the steps: collecting ambient light data based on an ambient light camera of the intelligent glasses; according to the ambient light data, adjusting exposure parameters and gains of a plurality of infrared cameras of the intelligent glasses; based on the adjusted infrared camera, continuously collecting human eye images; the human eye images collected by the infrared cameras on the left side and the right side are corrected and fused through the human eye images collected by the infrared cameras above the nose bridge; identifying key feature points of the eyes; calculating the movement speed and direction of the eyes according to the position change of the feature points in the continuous frame images; generating eye movement data; and classifying the eye movement data by using a pre-trained eye movement state classification model, and outputting a current eye movement state. The invention further discloses a control device. The objective of the invention is to improve the accuracy of the intelligent glasses in recognizing the eye movement state of the user.

Description

Technical Field

[0001] The present application relates to the technical field of smart glasses, and in particular to an eye movement state recognition method based on smart glasses, a control device, smart glasses, and a computer-readable storage medium. Background Art

[0002] In recent years, smart glasses, thanks to their convenience and proximity to the human eye, have shown great potential in the field of eye movement recognition. They can play an important role in human-computer interaction, VR / AR applications, and medical monitoring, such as enabling eye-controlled operations, optimizing virtual scene displays, and assisting in disease diagnosis.

[0003] However, current smart glasses still face numerous challenges in eye movement recognition technology. For one thing, ambient light interference is a significant issue. Because the cameras in smart glasses are close to the eyes, even subtle changes in ambient light can significantly affect the quality of images captured by the human eye. In bright light environments, such as direct sunlight outdoors, the images captured by the camera are prone to overexposure, obscuring key features like the pupil. In low-light environments, such as dimly lit indoors, images can become blurred due to insufficient light.

[0004] Furthermore, to ensure the overall appearance and wearing comfort of smart glasses, the camera's installation location is subject to numerous restrictions. This results in a limited viewing angle, which may not fully capture the various movements and characteristics of the human eye. For example, certain key eye movement details may not be clearly recorded due to poor camera placement, affecting the accuracy of subsequent feature point positioning and eye movement state recognition.

[0005] These problems make it difficult for smart glasses to accurately capture the user's eye movements.

[0006] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0007] The main purpose of this application is to provide an eye movement state recognition method based on smart glasses, a control device, smart glasses and a computer-readable storage medium, aiming to improve the accuracy of smart glasses in recognizing the user's eye movement state.

[0008] To achieve the above objectives, the present application provides an eye movement state recognition method based on smart glasses, comprising the following steps: An ambient light camera based on smart glasses collects ambient light data; Adjusting exposure parameters and gains of multiple infrared cameras of the smart glasses based on ambient light data; wherein the smart glasses include at least three infrared cameras located on the left and right sides of the glasses frame and above the bridge of the nose; Based on the adjusted infrared camera, human eye images are continuously collected; The human eye images captured by the infrared camera above the bridge of the nose are used to correct and fuse the human eye images captured by the infrared cameras on the left and right sides; Positioning the feature points of the fused human eye image to identify the key feature points of the eye; the key feature points include the pupil center, eye corners and iris edges; According to the position changes of feature points in continuous frame images, the optical flow method is used to calculate the speed and direction of eye movement; Smoothing the calculated eye movement speed and direction to generate eye movement data; Use the pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state.

[0009] To achieve the above objectives, the present application further provides a control device, comprising: The first acquisition module is used to collect ambient light data based on the ambient light camera of the smart glasses; An adjustment module, configured to adjust exposure parameters and gains of multiple infrared cameras of the smart glasses based on ambient light data; wherein the smart glasses include at least three infrared cameras located on the left and right sides of the glasses frame and above the bridge of the nose; The second acquisition module is used to continuously acquire human eye images based on the adjusted infrared camera; The correction and fusion module is used to use the human eye image captured by the infrared camera above the nose bridge to correct and fuse the human eye images captured by the infrared cameras on the left and right sides; A recognition module is used to locate feature points of the fused human eye image and identify key feature points of the eye; the key feature points include the pupil center, eye corners and iris edges; A calculation module is used to calculate the eye movement speed and direction using the optical flow method based on the position changes of feature points in continuous frame images; A generation module is used to smooth the calculated eye movement speed and direction to generate eye movement data; The classification module is used to classify eye movement data using a pre-trained eye movement state classification model and output the current eye movement state.

[0010] To achieve the above-mentioned purpose, the present application also provides a pair of smart glasses, which include: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the eye movement state recognition method based on smart glasses are implemented as described above.

[0011] To achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the eye movement state recognition method based on smart glasses are implemented as described above.

[0012] The eye movement state recognition method, control device, smart glasses and computer-readable storage medium based on smart glasses provided in this application cope with ambient light interference by adjusting the exposure and gain of the infrared camera according to the ambient light data, and use multi-position cameras to collect and correct the fused image to solve the problem of limited installation position, effectively improving the quality of human eye image collection and the viewing angle range, thereby improving the accuracy of smart glasses in recognizing the user's eye movement state. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the steps of an eye movement state recognition method based on smart glasses in one embodiment of the present application; Figure 2 This is a schematic diagram of a control device in an embodiment of the present application; Figure 3 Schematic diagram of the internal structure of smart glasses according to an embodiment of the present application.

[0014] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0015] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be understood as limiting the present application. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application without making any creative efforts shall fall within the scope of protection of the present application.

[0016] In addition, any descriptions of "first," "second," etc., in this application are for descriptive purposes only (e.g., to distinguish identical or similar features) and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one such feature. Furthermore, the technical solutions of various embodiments may be combined with each other, but this must be based on the ability of a person of ordinary skill in the art to implement them. If the combination of technical solutions contradicts or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0017] Reference Figure 1In one embodiment, the eye movement state recognition method based on smart glasses includes: Step S10: collecting ambient light data based on the ambient light camera of the smart glasses; Step S20: adjusting exposure parameters and gains of multiple infrared cameras of the smart glasses based on the ambient light data; wherein the smart glasses include at least three infrared cameras located on the left and right sides of the glasses frame and above the nose bridge; Step S30: continuously collecting human eye images based on the adjusted infrared camera; Step S40: Using the human eye image captured by the infrared camera above the nose bridge, the human eye images captured by the infrared cameras on the left and right sides are corrected and fused; Step S50: Positioning feature points of the fused human eye image to identify key feature points of the eye; the key feature points include the pupil center, eye corners, and iris edges; Step S60: Calculate the eye movement speed and direction using the optical flow method based on the position changes of the feature points in the continuous frame images; Step S70: Smoothing the calculated eye movement speed and direction to generate eye movement data; Step S80: Use the pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state.

[0018] In this embodiment, the execution terminal of the embodiment may be smart glasses, or may be other equipment or devices (such as a control device) that controls the smart glasses.

[0019] As described in step S10, the smart glasses are equipped with an ambient light camera specifically designed to collect ambient light information. Factors such as the intensity and color of ambient light can significantly impact the subsequent acquisition of human eye images. This camera continuously samples ambient light, collecting data such as brightness and color temperature. This data serves as an important basis for subsequent adjustments to the infrared camera parameters to ensure clear and accurate human eye images are captured under varying lighting conditions.

[0020] As described in step S20, an infrared camera is mounted on each side of the eyeglass frame. The two cameras are positioned to capture images of the human eye from different side angles. Images captured from side angles help capture richer, more detailed information about the human eye in the horizontal direction, such as changes in the shape of the corners of the eyes and horizontal rotation of the eyeball. This left-right layout enables separate and comprehensive observation of both eyes, providing basic data from different perspectives for subsequent image correction and fusion.

[0021] An infrared camera is mounted above the bridge of the nose. This centrally located camera captures a clear, frontal image of the eye. This frontal image is crucial for accurately identifying key eye features, such as the pupil center and iris edge. It also serves as a reference image for correcting and fusing the images captured by the left and right cameras to eliminate image deviations caused by viewing angle differences.

[0022] Before adjusting the parameters, it is necessary to first conduct a comprehensive and detailed analysis of the ambient light data collected in step S10. The ambient light data contains multiple key information, such as: (1) Light intensity: Light intensity directly affects the exposure of the infrared camera. When the light intensity is low, it means the environment is dark and the exposure needs to be increased; when the light intensity is high, the environment is bright and the exposure should be reduced.

[0023] (2) Illumination uniformity: Whether the ambient light is evenly distributed in space will also affect image acquisition. If the illumination is uneven, some areas of the collected human eye image may be too bright or too dark, affecting the accuracy of subsequent feature point positioning.

[0024] (3) Color temperature: Different light sources have different color temperatures, which will affect the color performance of the image. Although infrared cameras mainly focus on the infrared band, the difference in color temperature may also indirectly affect the clarity and contrast of the image.

[0025] Exposure parameters mainly include shutter speed, aperture size, etc., which together determine the time and intensity of the camera sensor receiving light.

[0026] Shutter speed controls the length of time the camera sensor is exposed to light. In low-light conditions, increasing the shutter speed allows the sensor more time to receive light, thereby brightening the image. For example, when the ambient light intensity is detected to be below a certain threshold, the shutter speed can be adjusted from 1 / 1000 second to 1 / 500 second. Conversely, in bright light environments, shortening the shutter speed can prevent overexposure.

[0027] The aperture size determines the amount of light that enters the lens. In darker environments, increasing the aperture allows more light to enter the camera; in brighter environments, decreasing the aperture is more appropriate. Aperture size is typically expressed as an F-number; a smaller F-number indicates a larger aperture. For example, in a dark environment, adjust the aperture from F4 to F2.8.

[0028] Gain is the magnification factor for the camera sensor's output signal. When ambient light is low, the sensor receives a weak signal, requiring increased gain to enhance the signal and create a clearer image. However, excessive gain can introduce noise and affect image quality, so precise adjustment is required based on ambient light data.

[0029] Optionally, when the ambient light intensity is lower than a certain standard value, gradually increase the gain value, such as from the initial 1 times to 2 times, 3 times, etc., while observing the image quality to ensure that too much noise is not introduced while enhancing the signal.

[0030] Optionally, in a strong light environment, reduce the gain value to avoid image distortion caused by excessive signal strength, such as reducing the gain from 3 times to 1 times.

[0031] To ensure consistent and comparable images of the human eye, the exposure parameters and gain of multiple infrared cameras must be adjusted synchronously. This can be achieved by developing a unified adjustment strategy based on ambient light data, simultaneously applying the same exposure and gain adjustments to the three infrared cameras located on the left and right sides of the eyeglass frame and above the bridge of the nose. To account for slightly different ambient light effects on cameras in different locations, fine-tuning can be performed on top of the unified adjustment. For example, the camera above the bridge of the nose may be obscured by the eyeglass frame, requiring appropriate increases in its exposure and gain.

[0032] As described in step S30, after adjusting the infrared camera parameters in step S20, the three infrared cameras begin continuously capturing images of the human eye. The infrared cameras illuminate the human eye with infrared light, enabling clear capture of eye details under varying lighting conditions. The continuously captured sequence of human eye images is used for subsequent analysis and processing.

[0033] As described in step S40, representative feature points are extracted from the eye images captured by the infrared cameras above the nose bridge and on the left and right sides. Optional feature point extraction algorithms include SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features). These algorithms can detect stable feature points under different scales and rotations and have good noise immunity.

[0034] The feature points in the image above the bridge of the nose are matched with those in the left and right images to find their corresponding relationship. A feature descriptor is generated for each extracted feature point, describing its local characteristics (a feature descriptor is typically a vector that reflects characteristics such as color and texture). By calculating the distance between the feature descriptors, the feature points in the left and right images that are most similar to the feature points in the image above the bridge of the nose are found, and a corresponding relationship is established between them.

[0035] Based on the feature point matching results, the left and right eye images are corrected to align their perspective and position more closely with the image above the bridge of the nose. Using the matched feature point pairs, the transformation matrix from the left and right images to the image above the bridge of the nose is calculated using methods such as the least squares method. Based on the calculated transformation matrix, the left and right eye images are transformed to align with the image above the bridge of the nose. The transformed images are closer in perspective and position to the image above the bridge of the nose, facilitating subsequent fusion operations.

[0036] The corrected left and right eye images are fused with the image of the eye above the bridge of the nose to produce a complete, clear composite image of the eye. Each pixel is assigned a weight based on the overlapping areas of the images and the matching of feature points. The corresponding pixels of the left and right images and the image above the bridge of the nose are then weighted averaged to produce the fused pixel value. The image is then decomposed into sub-images of varying resolutions, which are then fused separately at each resolution. Finally, the fused sub-images are reconstructed into the final fused image. This method better preserves image detail and features.

[0037] As described in step S50, after obtaining the fused human eye image, a computer vision algorithm is used to locate feature points of the image to identify key feature points of the eye, including the pupil center, eye corners, and iris edges.

[0038] Alternatively, the center of the pupil may be determined by methods based on image grayscale or edge detection, for example, by calculating the grayscale centroid of the pupil region in the image, or by using an edge detection algorithm to extract the edge of the pupil, thereby determining its center.

[0039] Optionally, the corners of the eyes usually have obvious features, such as grayscale changes, curvature changes, etc., and the positions of the corners of the eyes can be located through methods such as feature matching and template matching.

[0040] Optionally, edge detection, Hough transform and other algorithms are used to detect the edge of the iris to determine its position.

[0041] As described in step S60 , before performing optical flow calculations, it is necessary to ensure that a sequence of clear, continuous human eye image frames is captured. These images can be high-quality human eye images obtained through calibration and fusion of multiple infrared cameras on the smart glasses. Furthermore, the time interval Δt between adjacent frames must be known and fixed; this time interval will be used in subsequent velocity calculations.

[0042] In the first frame, select representative and easily trackable feature points. These feature points are typically key areas of the human eye. Methods such as the Shi-Tomasi corner detection algorithm can be used to automatically detect these feature points to ensure their stability and trackability.

[0043] Optical flow is an analysis method based on the movement of brightness patterns in image sequences. It assumes that the brightness of objects remains constant between adjacent frames. The Lucas-Kanade algorithm, a sparse optical flow method, is chosen here because it is suitable for tracking a small number of feature points and has high computational efficiency.

[0044] Optionally, the Lucas-Kanade algorithm is used to track the feature points selected in the first frame and calculate their position changes in subsequent consecutive frames. Based on the position changes of the feature points and the time interval Δt between adjacent frames, the movement speed of each feature point can be calculated. To obtain the overall movement speed of the eye, the speeds of multiple feature points can be averaged. The direction of eye movement can be determined by the displacement vector of the feature point. The direction of the displacement vector (Δx, Δy) represents the movement direction of the feature point. To obtain the overall movement direction of the eye, the movement directions of multiple feature points can be comprehensively analyzed, such as taking the average or using a weighted average method.

[0045] In this way, the optical flow method can be used to accurately calculate the speed and direction of eye movement based on the position changes of feature points in continuous frame images, providing important basic data for subsequent eye movement state recognition.

[0046] As described in step S70, due to the presence of noise and errors in the image acquisition and feature point positioning process, the calculated eye movement speed and direction may fluctuate. In order to improve the stability and accuracy of the eye movement data, it needs to be smoothed.

[0047] Optionally, methods such as sliding average and Kalman filtering can be used to smooth eye movement speed and direction. The sliding average algorithm eliminates noise by calculating the average value within a certain time window; the Kalman filter is a more complex filtering algorithm that can predict and estimate eye movement data based on the system's dynamic model and measurement noise, thereby improving data accuracy.

[0048] After smoothing and adjustment, the processed eye movement velocity and direction data are integrated to generate the final eye movement data. This data can be stored in a suitable format, such as a CSV file or database, for subsequent eye movement analysis and application. Eye movement data typically includes information such as timestamp, x-speed, y-speed, and direction angle.

[0049] As described in step S80, after obtaining the smoothed eye movement data, the eye movement state classification model is used to classify the data.

[0050] The pre-trained eye movement state classification model is pre-trained using a large amount of eye movement data. During training, the model learns the mapping between eye movement data features and different eye movement states (such as fixation, saccade, and blink). This allows it to classify new, unknown eye movement data and determine its corresponding eye movement state.

[0051] The model extracts key features from eye movement data, which may come from information such as the speed, direction, position, and duration of eye movements recorded by an eye tracker. For example, during fixation, the eye position is relatively stable and the speed is low, while during saccades, the eye moves rapidly and at a higher speed. By learning from a large amount of labeled eye movement data, the model identifies characteristic patterns corresponding to different eye movement states. Using these patterns, when encountering new eye movement data, the model can determine the eye movement state it belongs to based on how well the data's characteristics match the learned patterns.

[0052] Among them, a suitable pre-trained eye movement state classification model can be deployed in the smart glasses in advance according to actual conditions. The optional model types include decision trees, support vector machines (SVMs), and neural networks (such as multi-layer perceptrons, convolutional neural networks, etc.).

[0053] Eye movement data is fed into a pre-trained eye state classification model. The model classifies the input eye movement data and outputs the current eye state. This allows accurate identification of eye movement states, providing important support for related applications such as human-computer interaction and eye tracking.

[0054] In this way, when dealing with the problem of ambient light interference, smart glasses can use the ambient light camera to collect data and accurately adjust the exposure parameters and gains of multiple infrared cameras. In this way, whether it is outdoors under strong light or in a low-light indoor environment, clear images of the human eye with complete key features can be collected to avoid overexposure or blur. Faced with the dilemma of limited installation positions of infrared cameras, this method uses infrared cameras located on the left and right sides of the frame and above the bridge of the nose. The images collected by the camera above the bridge of the nose are used to correct and fuse the images on the left and right sides, widening the field of view and fully capturing human eye movements and features, thereby greatly improving the accuracy of eye movement state recognition and providing strong support for the application of smart glasses in multiple fields.

[0055] In one embodiment, ambient light interference is dealt with by adjusting the exposure and gain of the infrared camera based on ambient light data, and the problem of limited installation position is solved by using multi-position cameras to collect and correct fused images, thereby effectively improving the quality of human eye image collection and the viewing angle range, thereby improving the accuracy of smart glasses in recognizing the user's eye movement status.

[0056] In one embodiment, based on the above embodiment, before the step of adjusting the exposure parameters and gains of the multiple infrared cameras of the smart glasses according to the ambient light data, the step further includes: Perform weighted averaging on multiple frames of ambient light data to reduce instantaneous fluctuations in ambient light.

[0057] In this embodiment, ambient light properties such as intensity and color may experience momentary fluctuations in real-world scenarios. Directly using this momentary fluctuation data to adjust the infrared camera's exposure and gain parameters would result in frequent changes in camera parameters, causing the captured human eye images to appear flickering and blurry, impacting the accuracy of subsequent eye movement recognition. Therefore, weighted averaging of multiple consecutive frames of ambient light data is performed to reduce the impact of momentary fluctuations and obtain more stable and reliable ambient light characteristic information, providing a basis for accurate adjustment of camera parameters.

[0058] Optionally, the ambient light camera in the smart glasses continuously collects ambient light data at a certain frame rate, typically storing the collected data as a continuous frame sequence in chronological order. Assuming the frame rate of the ambient light camera is f frames per second, within time t, N = f × t frames of ambient light data will be collected.

[0059] To perform a weighted average of multiple frames of ambient light data, each frame needs to be assigned an appropriate weight. The principle of weight assignment is: the closer the frame is to the current moment, the greater its weight, because it better reflects the current ambient light state; the farther the frame is from the current moment, the smaller its weight.

[0060] After determining the weight of each frame, a weighted average is calculated for multiple consecutive frames of ambient light data. As time passes, the ambient light camera continuously collects new frames of data. Each new frame is added to the data sequence, and the weighted average result is updated.

[0061] After weighted averaging, the resulting ambient light data is smoother, reducing the impact of momentary fluctuations. This stable ambient light data is used to adjust the exposure and gain parameters of the smart glasses' multiple infrared cameras. This allows the camera parameters to be adjusted based on the actual, stable state of ambient light, resulting in higher-quality, clearer images of the human eye, providing a more reliable foundation for subsequent eye movement recognition.

[0062] In one embodiment, based on the above embodiment, before the step of correcting and fusing the human eye images captured by the infrared camera above the nose bridge with the human eye images captured by the infrared cameras on the left and right sides, the step further includes: Adaptive median filtering algorithm is used to reduce noise in human eye images; Enhance the contrast of the denoised human eye image to highlight the characteristic information of the eye.

[0063] In this embodiment, when the infrared camera of the smart glasses captures eye images, the images may be affected by various noises, such as salt and pepper noise and Gaussian noise. Noise can cause the image to have some randomly distributed abnormal pixels, making the image blurry and unclear, seriously affecting the subsequent extraction and analysis of eye feature information, and thus reducing the accuracy of eye movement recognition.

[0064] Adaptive median filtering is a median filtering algorithm that adaptively adjusts the filter window size based on local image characteristics. Compared to traditional fixed-window median filtering, it better handles noise of varying density and distribution while preserving image edges and details. Its basic principle is to determine whether a pixel is a noise point by comparing the values of the current pixel with those of its neighbors. Based on this determination, the appropriate filter window size is selected for filtering.

[0065] Although noise reduction effectively suppresses the image of the human eye, it may still have low contrast, making key features of the eye, such as the pupil, canthus, and iris edge, less distinct and difficult to accurately identify. The purpose of contrast enhancement is to adjust the image's grayscale distribution, making bright areas brighter and dark areas darker, thereby highlighting the eye's features and improving image clarity and legibility.

[0066] Optionally, a histogram equalization method is used to enhance the contrast of the image. The basic principle is to transform the grayscale histogram of the image so that the grayscale distribution of the image is more uniform, thereby increasing the contrast of the image.

[0067] In one embodiment, by performing effective noise reduction and contrast enhancement processing on human eye images, the image quality is significantly improved, providing high-quality image data for subsequent correction and fusion steps, helping to more accurately identify key features of the eyes and improve the accuracy of eye movement state recognition.

[0068] In one embodiment, based on the above embodiment, the step of locating feature points on the fused human eye image and identifying key feature points of the eye includes: A convolutional neural network model is used to locate feature points of the fused human eye image; wherein the convolutional neural network model is pre-trained based on a human eye image dataset and can accurately identify key feature points of the eye.

[0069] In this embodiment, the convolutional neural network model can accurately identify key feature points of the eye after being trained based on a human eye image dataset. This process integrates the synergistic effect of data, model, and training.

[0070] First of all, the human eye image dataset is the foundation, which covers images of different individuals, lighting, expressions and angles, and accurately marks key feature points, providing diverse and accurate samples for model learning.

[0071] Secondly, the unique architecture of convolutional neural networks plays a significant role. Convolutional layers extract local features, pooling layers reduce the amount of data, and fully connected layers integrate features. During training, the input image undergoes forward propagation to obtain a predicted result, which is then compared with the ground truth to calculate the loss. Backpropagation is then used to update the parameters, continuously reducing the loss.

[0072] Furthermore, the model possesses powerful learning capabilities. It can automatically learn hierarchical features from data, from low-level edge textures to high-level shape relationships. It also possesses generalization capabilities, allowing it to process unseen images.

[0073] Ultimately, the trained model demonstrated excellent accuracy and stability. On the test set, it was able to locate feature points with minimal error and consistently and accurately identify key eye features despite varying image variations, providing solid support for related applications.

[0074] After model training is completed, the trained convolutional neural network model is loaded into the smart glasses system. After obtaining the fused human eye image, it is input into the loaded convolutional neural network model.

[0075] The model performs forward propagation calculations on the input eye image and outputs the predicted coordinates of the key eye feature points, thus completing the task of locating the feature points in the fused eye image. This accurately identifies the key feature points in the fused eye image, providing important basic data for subsequent eye movement analysis.

[0076] In one embodiment, based on the above embodiment, the step of smoothing the calculated eye movement speed and direction to generate eye movement data includes: The calculated eye movement speed and direction are smoothed using the Kalman filter algorithm to generate eye movement data.

[0077] In this embodiment, the Kalman filter is an algorithm that uses linear system state equations and observational data from the system's input and output to optimally estimate the system's state. In eye movement data processing scenarios, eye movement velocity and direction are considered system state variables. Due to noise interference in the eye movement measurement process, the originally calculated eye movement velocity and direction data may fluctuate. The Kalman filter algorithm can achieve a more accurate and smooth state estimate at the current moment by weightedly fusing the current measurement value with the previous estimate.

[0078] After Kalman filtering, the state estimate obtained is the smoothed eye movement speed and direction data. This data is sorted in chronological order to generate the final eye movement data.

[0079] In one embodiment, based on the above embodiment, after the step of using a pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state, the method further includes: When the smart glasses are in eye movement control mode, corresponding eye movement interaction control operations are performed on the smart glasses according to the eye movement status.

[0080] In this embodiment, the system within the smart glasses detects whether it is in eye control mode. This can be achieved by checking a mode flag in the system settings or the status of a specific hardware switch. For example, in the settings interface of the smart glasses, the user can manually select to turn eye control mode on or off. The system will record this selection and use it for subsequent judgment.

[0081] Optionally, if the system detects that the smart glasses are in eye control mode, the system will give corresponding feedback, such as flashing an indicator light, emitting a prompt sound, or displaying a specific icon on the display screen, to inform the user that the eye control state has been entered.

[0082] After the pre-trained eye movement state classification model classifies the eye movement data, it will output the current eye movement state results to the control system of the smart glasses.

[0083] In the smart glasses' control system, mappings between different eye movement states and corresponding control actions are predefined. These rules can be customized based on the functional requirements of the smart glasses and user habits. For example, a "long gaze" state can be mapped to an "open an app" action, while a "double-blink" state can be mapped to a "take a photo" action.

[0084] The control system matches the received eye movement state classification results with pre-defined mapping rules to find the corresponding control action. For example, if the current eye movement state is "scan left", the system will search the rule table and find that the corresponding control action is "switch to the previous application".

[0085] Once a matching control operation is found, the control system immediately sends a control instruction to the corresponding functional module of the smart glasses to execute the operation. For example, if you want to open an application, the control system sends a launch instruction to the application management module, which then loads and launches the corresponding application.

[0086] This eliminates traditional manual operation and allows users to control various functions of smart glasses simply by eye movement. For example, when both hands are busy, users no longer need to put down their objects to operate buttons or touch the screen. Just a glance or specific eye movement can perform functions such as taking a photo or switching applications, greatly improving the convenience and efficiency of operation.

[0087] Users can modify and adjust the mapping between eye movement states and control operations based on their usage habits. Different users may have different preferences. For example, some users are accustomed to controlling the number of blinks to take photos, while others may prefer to use a specific eye movement direction to achieve the same function. Personalized customization meets diverse needs and improves user satisfaction.

[0088] In one embodiment, based on the above embodiment, after the step of using a pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state, the method further includes: When it is detected based on the eye movement status that the user closes their eyes for a long time or their eyes leave the display area of the smart glasses, the smart glasses are controlled to automatically enter the screen sleep state; When the user opens his eyes or his gaze returns to the display area based on the eye movement status, the smart glasses screen is awakened and the normal display is restored.

[0089] In this embodiment, a clear time threshold is set for "long-term eye closure", for example, continuous eye closure for more than 5 to 10 seconds. This threshold can be adjusted according to the user's usage habits and actual needs and stored in the system settings of the smart glasses.

[0090] Determining if the gaze has left the display area: A reasonable determination criteria is established based on the physical boundaries of the smart glasses' display area and the visual range. For example, if the eye tracking data shows that the gaze point has exceeded a certain range (e.g., 10-20% of the display area width) from the edge of the display area for more than 5 consecutive seconds, it is considered as leaving the display area.

[0091] The control system of the smart glasses continuously receives the current eye movement state data output by the pre-trained eye movement state classification model. This data includes information such as the open and closed state of the eyes, the position of the eyeballs, and the direction of movement.

[0092] The control system analyzes the eye movement status data received in real time to determine whether the judgment conditions of long-term eye closure or sight leaving the display area are met.

[0093] Optionally, when eye movement data indicates that the user has closed their eyes for a long time or has looked away from the display area of the smart glasses, and pre-set conditions are met, the control system triggers the screen sleep program. At this point, the control system sends a sleep command to the screen driver module of the smart glasses. Upon receiving the command, the screen driver module gradually reduces the screen brightness until the screen is completely off and enters a sleep state. At the same time, the system suspends some unnecessary background programs to save power.

[0094] When the screen is in sleep mode, the eye movement monitoring system of the smart glasses does not stop working, but continues to monitor eye movement data to determine whether the user intends to wake up the screen.

[0095] Optionally, when the control system detects that the user opens his eyes or his gaze returns to the display area based on the eye movement status, and the pre-set wake-up conditions are met (such as the eyes are open for more than 1 second), the screen wake-up program will be triggered.

[0096] The control system sends a wake-up command to the screen driver module, which gradually increases the screen brightness and returns to normal display. At the same time, the system restarts any previously paused background programs, returning the smart glasses to normal operation.

[0097] Smart glasses typically have limited battery capacity, and the screen is a major source of energy consumption. The automatic sleep feature turns off the screen when not in use, reducing unnecessary power consumption and extending device life. For example, if the user closes their eyes for a long time or looks away from the screen, the screen automatically sleeps, effectively conserving power and improving device efficiency.

[0098] In addition, refer to Figure 2 In an embodiment of the present application, a control device Z10 is further provided, comprising: The first acquisition module Z11 is used to collect ambient light data based on the ambient light camera of the smart glasses; An adjustment module Z12 is configured to adjust exposure parameters and gains of multiple infrared cameras of the smart glasses based on ambient light data; wherein the smart glasses include at least three infrared cameras located on the left and right sides of the glasses frame and above the nose bridge; The second acquisition module Z13 is used to continuously acquire human eye images based on the adjusted infrared camera; The correction and fusion module Z14 is used to use the human eye image captured by the infrared camera above the nose bridge to correct and fuse the human eye images captured by the infrared cameras on the left and right sides; Identification module Z15, used to locate feature points of the fused human eye image and identify key feature points of the eye; the key feature points include the pupil center, eye corners and iris edges; A calculation module Z16 is used to calculate the eye movement speed and direction using an optical flow method based on the position changes of feature points in consecutive frame images; A generation module Z17 is used to smooth the calculated eye movement speed and direction to generate eye movement data; The classification module Z18 is used to classify the eye movement data using a pre-trained eye movement state classification model and output the current eye movement state.

[0099] Optionally, the control device Z10 may be a virtual control device (such as a virtual machine) or a physical device (such as a physical device other than smart glasses that can execute the corresponding method).

[0100] In addition, the embodiment of the present application also provides a smart glasses, the internal structure of the smart glasses can be as follows Figure 3 As shown, it includes a processor, a memory, a communication interface, and an input interface connected via a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database is used to store data called by the computer program. The communication interface is used to communicate data with an external terminal. The input interface is used to receive signals input by an external device. When the computer program is executed by the processor, it implements an eye movement state recognition method based on smart glasses as described in the above embodiment.

[0101] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present invention and does not limit the smart glasses to which the present invention is applied. For example, in some optional embodiments, the smart glasses may further include an output interface (not shown in the figure), which is also connected to the system bus and is used to output corresponding signals to peripheral devices.

[0102] In addition, this application also provides a computer-readable storage medium, which includes a computer program. When executed by a processor, the computer program implements the steps of the eye movement state recognition method based on smart glasses as described in the above embodiment. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0103] In summary, the eye movement state recognition method based on smart glasses, the control device, the smart glasses and the computer-readable storage medium provided in the embodiments of the present application cope with ambient light interference by adjusting the exposure and gain of the infrared camera according to the ambient light data, and use multi-position cameras to collect and correct the fused image to solve the problem of limited installation position, thereby effectively improving the quality of human eye image collection and the viewing angle range, thereby improving the accuracy of smart glasses in recognizing the user's eye movement state.

[0104] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM).

[0105] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0106] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for recognizing eye movement status based on smart glasses, characterized in that: include: An ambient light camera based on smart glasses collects ambient light data; Adjusting exposure parameters and gains of multiple infrared cameras of the smart glasses based on ambient light data; wherein the smart glasses include at least three infrared cameras located on the left and right sides of the glasses frame and above the bridge of the nose; Based on the adjusted infrared camera, human eye images are continuously collected; The human eye images captured by the infrared camera above the bridge of the nose are used to correct and fuse the human eye images captured by the infrared cameras on the left and right sides; Positioning the feature points of the fused human eye image to identify the key feature points of the eye; the key feature points include the pupil center, eye corners and iris edges; According to the position changes of feature points in continuous frame images, the optical flow method is used to calculate the speed and direction of eye movement; Smoothing the calculated eye movement speed and direction to generate eye movement data; Use the pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state.

2. The eye movement state recognition method based on smart glasses according to claim 1, characterized in that: Before the step of adjusting the exposure parameters and gains of the multiple infrared cameras of the smart glasses according to the ambient light data, the method further includes: Perform weighted averaging on multiple frames of ambient light data to reduce instantaneous fluctuations in ambient light.

3. The eye movement state recognition method based on smart glasses according to claim 1, characterized in that: Before the step of correcting and fusing the human eye images captured by the infrared camera on the left and right sides using the human eye image captured by the infrared camera above the nose bridge, the method further includes: Adaptive median filtering algorithm is used to reduce noise in human eye images; Enhance the contrast of the denoised human eye image to highlight the characteristic information of the eye.

4. The eye movement state recognition method based on smart glasses according to claim 1, characterized in that: The step of locating feature points of the fused human eye image and identifying key feature points of the eye includes: A convolutional neural network model is used to locate feature points of the fused human eye image; wherein the convolutional neural network model is pre-trained based on a human eye image dataset and can accurately identify key feature points of the eye.

5. The eye movement state recognition method based on smart glasses according to claim 1, characterized in that: The step of smoothing the calculated eye movement speed and direction to generate eye movement data includes: The calculated eye movement speed and direction are smoothed using the Kalman filter algorithm to generate eye movement data.

6. The eye movement state recognition method based on smart glasses according to any one of claims 1 to 5, characterized in that: After the step of using the pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state, the method further includes: When the smart glasses are in eye movement control mode, corresponding eye movement interaction control operations are performed on the smart glasses according to the eye movement status.

7. The eye movement state recognition method based on smart glasses according to any one of claims 1 to 5, characterized in that: After the step of using the pre-trained eye movement state classification model to classify the eye movement data and output the current eye movement state, the method further includes: When it is detected based on the eye movement status that the user closes their eyes for a long time or their eyes leave the display area of the smart glasses, the smart glasses are controlled to automatically enter the screen sleep state; When the user opens his eyes or his gaze returns to the display area based on the eye movement status, the smart glasses screen is awakened and the normal display is restored.

8. A control device, characterized in that: include: The first acquisition module is used to collect ambient light data based on the ambient light camera of the smart glasses; An adjustment module, configured to adjust exposure parameters and gains of multiple infrared cameras of the smart glasses based on ambient light data; wherein the smart glasses include at least three infrared cameras located on the left and right sides of the glasses frame and above the bridge of the nose; The second acquisition module is used to continuously acquire human eye images based on the adjusted infrared camera; The correction and fusion module is used to use the human eye image captured by the infrared camera above the nose bridge to correct and fuse the human eye images captured by the infrared cameras on the left and right sides; A recognition module is used to locate feature points of the fused human eye image and identify key feature points of the eye; the key feature points include the pupil center, eye corners and iris edges; A calculation module is used to calculate the eye movement speed and direction using the optical flow method based on the position changes of feature points in continuous frame images; A generation module is used to smooth the calculated eye movement speed and direction to generate eye movement data; The classification module is used to classify eye movement data using a pre-trained eye movement state classification model and output the current eye movement state.

9. A pair of smart glasses, characterized in that: The smart glasses include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the eye movement state recognition method based on smart glasses as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the eye movement state recognition method based on smart glasses according to any one of claims 1 to 7.

Images