Computer vision-based AI smart glasses and human-computer interaction method

Abstract

The application discloses an AI intelligent glasses based on computer vision and a man-machine interaction method, and relates to the technical field of man-machine interaction. The application comprises the following steps: collecting eye movement data of a wearer, fitting a fixation rhythm baseline model according to the eye movement data, and obtaining interval duration and continuous duration of a fixation window; collecting a turning angle, a zoom ratio, an angular velocity and an angular acceleration parameter in a running process of a camera in real time. The application fits a fixation rhythm baseline model by collecting eye movement data of a wearer, combines real-time motion parameters of a camera to predict a period of a stable fixation window, divides an identification result into two types of static basic information and dynamic change information to perform hierarchical rendering scheduling, keeps the static basic information continuously displayed in a continuous motion process of the camera, and makes a smooth position offset according to the motion parameters, so that the wearer can continuously acquire basic scene information in the whole lens adjustment process without waiting for the motion to stop and then reading the content again.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to an AI smart glasses based on computer vision and a human-computer interaction method. Background Technology

[0002] With the development of computer vision and wearable device technologies, smart glasses equipped with camera capture and information overlay functions are gradually becoming an important carrier for age-friendly assistive scenarios. These devices can recognize surrounding objects, text, and user interfaces in real time, presenting assistive information directly to the wearer's field of vision. They also support remote assistance from other personnel, helping the elderly cope with various daily needs such as medication recognition, operating electronic devices, and outdoor activities. This combination of visual recognition and remote assistance effectively adapts to the physiological characteristics and usage habits of the elderly, possessing broad application potential.

[0003] Chinese patent application CN120993617A discloses an age-friendly smart glasses assistance system and method based on human-computer interaction. The solution simultaneously acquires two video feeds—one for operation and one for the environment—and processes them separately. An image stabilization algorithm optimizes the display of the environmental image. A two-level authentication mechanism is established, assigning corresponding operation permissions according to different assistant roles. Assistants can remotely adjust the angle and focus of the smart glasses' camera, and after observing a specific scene, overlay guidance information onto the image to provide corresponding assistance to the wearer.

[0004] In practical remote assistance applications, when the assistant continuously turns or zooms the camera, the system maintains the same frequency of updating the image recognition results and overlay information. Elderly individuals have slower eye movement speeds and longer stable fixation times. When information updates occur precisely during the wearer's eye movement, the overlay text and annotations can easily create visual afterimages. Frequent and irregular content refreshes also disrupt the wearer's fixation rhythm. This results in the wearer being unable to effectively read the overlay information throughout the entire camera adjustment process, having to wait until the adjustment completely stops before rereading the content. This not only prolongs the time required for each assistance session but also exacerbates eye fatigue due to the continuous image jumps. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides AI smart glasses based on computer vision and a human-computer interaction method, which solves the problems of irregular information updates, visual afterimages, interrupted gaze rhythm, low recognition efficiency, and increased eye fatigue in existing smart glasses when the camera is continuously adjusted.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a human-computer interaction method for AI smart glasses based on computer vision, comprising: The wearer's eye movement data is collected, and a fixation rhythm baseline model is fitted based on the eye movement data to obtain the interval duration and duration of the fixation window; Real-time acquisition of parameters such as turning angle, zoom ratio, angular velocity, and angular acceleration during camera operation; Based on the gaze rhythm baseline model and camera motion parameters, predict the start time and duration of the next stable gaze window; Content recognition is performed on the real-time video stream captured by the camera, and the recognition results are divided into two categories: static basic information and dynamic change information. When the camera is detected to be in continuous motion, the static basic information is kept continuously displayed in the picture, and the position of the static basic information is smoothly shifted according to the real-time motion parameters. When the predicted gaze window start time arrives, an update operation is performed on the dynamically changing information, and the update process uses an transparency transition method; Once the camera stops moving, resume the normal recognition and update frequency.

[0007] Furthermore, the step of fitting the fixation rhythm baseline model further includes: Continuously collect eye movement data from wearers during daily use; Record the time interval from start to end of each gaze, the time interval from start to end of each eye saccade, and the maximum speed of movement during each eye saccade. Eye movement data during blinking, rapid head turning, and when the gaze leaves the field of vision are excluded, and only valid samples from normal visual scenarios are retained. The fixation stability coefficient is calculated based on fixation duration data recorded over multiple consecutive periods, the saccade rhythm coefficient is calculated based on saccade duration data recorded over multiple consecutive periods, and the saccade speed baseline is calculated based on maximum saccade speed data recorded over multiple consecutive periods. The fixation stability coefficient, saccade rhythm coefficient, and saccade velocity baseline are stored as the core parameters of the fixation rhythm baseline model. The gaze rhythm baseline model is continuously revised as usage time increases to adapt to eye movement differences under different usage conditions.

[0008] Furthermore, the step of excluding eye movement data during blinking, rapid head turning, and eye movement out of the field of vision further includes: Continuously monitor blinking features in eye movement signals. When a blinking signal is detected, remove the eye movement samples collected within that blink cycle from the valid dataset. Continuously monitor changes in head posture. When the speed of head posture change exceeds the preset head turning threshold, it is judged as a rapid head turning state, and the eye movement samples during the head turning process are removed from the valid dataset. The pupil center position is continuously monitored. When the pupil center position deviates from the preset visual field range, it is judged as a state of gaze leaving the visual field. The eye movement samples during the gaze-leaning period are removed from the valid dataset.

[0009] Furthermore, the step of predicting the next stable gaze window further includes: Retrieve the gaze window interval duration and window duration stored in the current gaze rhythm baseline model; Get the end time of the most recent gaze state; Add the most recent fixation end time to the fixation window interval to obtain the predicted start time of the next fixation window; The predicted start time value is added to the window duration to obtain the predicted end time value of the next fixation window; Based on the current camera motion angular velocity and image complexity, the time required for recognition calculation is estimated, and a pre-calculation instruction is sent to the target recognition module before the start time of the gaze window. The target recognition module completes a new round of image content recognition calculations during the camera's eye-scratching phase and stores the recognition results in a cache queue for later retrieval.

[0010] Furthermore, the process of issuing and executing the pre-computation instructions further includes: The timing prediction and scheduling module queries a preset performance mapping table based on the current camera motion angular velocity to obtain the estimated time consumption for recognition operations corresponding to different angular velocities. Subtract the estimated processing time from the start time of the gaze window to obtain the latest time when the pre-calculated command is issued; After the pre-computation instruction is issued, the timing prediction scheduling module continuously monitors the completion status of the target recognition module's calculations; If the calculation is detected to be completed before the start time of the gaze window, then the pre-calculated result is confirmed to be usable; If the computation is not completed when the start time of the gaze window arrives, the update operation will be postponed to the next gaze window.

[0011] Furthermore, the step of smoothing the position offset of the static basic information based on real-time motion parameters further includes: Obtain the camera's current turning angle and zoom level, and calculate the current offset position of static basic information in the image; Obtain the offset position of the static basic information at the previous moment, and calculate the position deviation between the current moment and the previous moment; A quadratic smoothing interpolation algorithm is used to smooth the positional deviation and eliminate screen jitter caused by abrupt changes in motion parameters; The processed deviation is added to the offset position of the previous moment to obtain the offset position of the current moment. The upper limit of the offset speed is set based on the baseline of eye saccade speed output by the eye-tracking learning module. When the calculated offset speed exceeds the upper limit, the offset speed is restricted. Static basic information remains displayed throughout the entire camera movement process, only undergoing positional shifts without content switching.

[0012] Furthermore, the step of performing an update operation on the dynamically changing information further includes: The timing prediction and scheduling module monitors the wearer's eye movement in real time. When it detects that the pupil center has entered the stable focusing area and the focusing duration has reached the fixation stability threshold, it is determined that the stable fixation stage has been entered. During the stable gaze phase, the latest recognition result is extracted from the cache queue of the target recognition module; Compare the dynamically changing information in the latest identification results with the currently displayed content to identify the information items that have changed. Update the content of changed information items, and use a transparent transition method to implement the update process; The transition time for transparency is one-tenth of the duration of the gaze window; The transparency gradually transitions from its initial value to full display, completing the content switching process; The entire update process is completed within the predicted duration of the gaze window.

[0013] Furthermore, the step of determining whether a person has entered the stable fixation phase further includes: Continuously collect the coordinate sequence of the wearer's pupil center position; Calculate the variance of the pupil center position over multiple consecutive sampling periods; When the variance value is continuously less than the fixation stability variance threshold, it is determined that the pupil center has entered a stable focusing state; Record the moment when the pupil center enters a stable focusing state; Calculate the duration of continuous focusing from the start of entering the stable focusing state to the current time; When the duration of sustained focus reaches the fixation stability threshold, it is determined that the fixation has entered the stable fixation stage.

[0014] Furthermore, the method also includes remote collaboration steps: The remote collaboration module receives the authentication request sent by the assistant through a remote terminal and completes the assistant's authentication. After identity verification, the remote collaboration module receives camera adjustment instructions sent by the assistant; The camera adjustment commands are synchronously forwarded to the image acquisition module and the timing prediction and scheduling module. Receive guidance information sent by the assistant and add the guidance information to the dynamic information queue; The remote collaboration module provides real-time feedback to the assisting terminal on the wearer's current gaze status, including three states: gaze, saccade, or gaze out of sight. The assistant selects the appropriate adjustment time based on the gaze status prompts. After the adjustment command is issued, the timing prediction scheduling module automatically enters the rhythm matching scheduling mode.

[0015] The present invention also provides AI smart glasses based on computer vision, including: The eye-tracking learning module is used to collect the wearer's eye-tracking data, fit a fixation rhythm baseline model based on the eye-tracking data, obtain the interval duration and duration parameters of the fixation window, and continuously eliminate abnormal data to correct the model parameters. The target recognition module is used to receive real-time video streams captured by the camera, perform content recognition on the video streams, and output the recognition results into two categories: static basic information and dynamic change information. The temporal prediction scheduling module receives the gaze rhythm baseline model data and camera motion parameters output by the eye-tracking learning module, predicts the time period of the next stable gaze window based on the gaze rhythm baseline model, evaluates the recognition calculation time based on the camera motion parameters and issues a pre-calculation instruction, and sends an update instruction to the layered rendering module during the predicted gaze window time period. The layered rendering module receives static basic information and dynamic change information output by the target recognition module. It maintains the display of static basic information while the camera is in continuous motion and performs smooth position shifts based on motion parameters. Upon receiving an update command, it completes the update and switching of dynamic change information within the gaze window, using a transparency transition method during the switching process.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention uses eye movement data collected from the wearer to fit a baseline model of gaze rhythm. Combined with real-time camera motion parameters, it predicts the duration of a stable gaze window. The recognition results are divided into two categories: static basic information and dynamic change information, and layered rendering scheduling is performed. During continuous camera movement, static basic information is continuously displayed, and a smooth positional shift is made based on motion parameters. This allows the wearer to continuously acquire basic scene information throughout the lens adjustment process, without waiting for the movement to stop before reading content. Dynamic change information is updated only within the predicted stable gaze window. The update process uses a transparency transition to avoid refreshing content during saccade periods, preventing irregular content refreshes from interrupting the wearer's gaze rhythm, reducing visual afterimages, and minimizing the abruptness of content switching. This is suitable for the physiological characteristics of the elderly, who have slower eye movement speeds and longer stable gaze durations. Combined with a pre-calculation mechanism for the saccade phase, recognition calculations can be completed during periods of weaker visual perception, shortening the information update response delay. Simultaneously, abnormal data filtering continuously corrects the rhythm model parameters, ensuring prediction accuracy under different usage conditions. In remote collaboration scenarios, the wearer's gaze state can be synchronized with the assisting end, making it easier for the assister to choose the appropriate adjustment time. This allows the dynamic information update to match the wearer's eye movement rhythm, shortening the time required for a single remote assistance session, reducing eye fatigue caused by frequent screen jumps, and steadily improving the readability and smoothness of use during lens adjustment. Attached Figure Description

[0017] Figure 1 This is a flowchart of the main human-computer interaction method of the present invention; Figure 2 This is a flowchart of the gaze rhythm fitting and window prediction process of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figures 1-2This invention provides a human-computer interaction method for AI smart glasses based on computer vision, comprising: collecting the wearer's eye movement data; fitting a gaze rhythm baseline model based on the eye movement data to obtain the interval duration and duration of the gaze window; collecting the turning angle, zoom ratio, angular velocity, and angular acceleration parameters of the camera during operation in real time; predicting the start time and duration of the next stable gaze window based on the gaze rhythm baseline model and camera motion parameters; performing content recognition on the real-time video stream collected by the camera, and classifying the recognition results into two categories: static basic information and dynamic change information; when the camera enters a continuous motion state, maintaining the static basic information continuously displayed in the image, while simultaneously smoothing the position shift of the static basic information according to the real-time motion parameters; when the predicted gaze window start time arrives, performing an update operation on the dynamic change information, with the update process using a transparency transition method; and restoring the normal recognition update frequency after the camera motion stops.

[0020] Specifically, eye-tracking data is output from an infrared eye-tracking acquisition unit inside the glasses, with a sampling frequency of 30 Hz. It outputs raw data such as pupil center coordinates, pupil area, and eye movement velocity. Camera motion parameters are jointly output by a six-axis attitude sensor and a zoom drive unit, with the data update frequency consistent with the video stream frame rate. The stable gaze window corresponds to the period when the wearer's eyes are in a fixed gaze without significant saccades, during which the wearer's visual perception is most stable and suitable for information updates. Static basic information includes content with fixed spatial positions and unchanging attributes in the scene, such as wall markings, fixed object names, and frequently displayed text prompts; dynamic change information includes real-time status data, remote guidance annotations, and temporary prompts that may change at any time. The camera enters continuous motion mode when its angular velocity continuously exceeds a preset threshold for a set duration. The threshold can be set to 15 degrees per second, and the duration threshold to 200 milliseconds, referencing the typical speed of manual lens adjustment in remote assistance scenarios. Static basic information remains displayed during movement, allowing the wearer to continuously acquire basic scene information while the lens adjusts, without waiting for the movement to stop before reading. Smooth positional shifts prevent static information from changing abruptly with lens movement, ensuring visual continuity. Dynamic information is updated only within the predicted gaze window, avoiding periods of eye movement and preventing visual afterimages caused by content refreshes that could interrupt the wearer's gaze rhythm. Transparency transitions minimize the abruptness of content switching, further reducing visual stimulation. Once the camera stops moving, the device resumes its normal recognition and update frequency, ensuring real-time information. This implementation method is suitable for the physiological characteristics of elderly people, who have slower eye movement speeds and longer periods of stable gaze, maintaining information readability during continuous lens adjustments and reducing eye fatigue caused by frequent image jumps.

[0021] In a specific embodiment, the step of fitting the gaze rhythm baseline model further includes: continuously collecting eye movement data of the wearer during daily use; recording the time interval from start to end of each gaze, the time interval from start to end of each saccade, and the maximum movement speed during each eye saccade; excluding eye movement data during blinking, rapid head turning, and when the gaze leaves the field of vision, retaining only valid samples under normal visual scenarios; calculating the gaze stability coefficient based on the gaze duration data recorded over multiple consecutive cycles, calculating the saccade rhythm coefficient based on the saccade duration data recorded over multiple consecutive cycles, and calculating the saccade speed baseline based on the maximum saccade speed data recorded over multiple consecutive cycles; storing the gaze stability coefficient, saccade rhythm coefficient, and saccade speed baseline as key parameters of the gaze rhythm baseline model; and continuously correcting the gaze rhythm baseline model as usage time increases to adapt to eye movement differences under different usage conditions.

[0022] Specifically, eye-tracking data during daily use is collected continuously, with the device automatically recording relevant motion parameters in the background, eliminating the need for manual calibration by the wearer. Fixation and saccade states are distinguished by the speed and displacement of pupil position movement; periods with speeds below 5 degrees per second are marked as fixation, while periods with speeds above 30 degrees per second are marked as saccades, with intermediate transitional periods grouped into their corresponding adjacent phases. Screening for valid samples eliminates interfering data from abnormal visual states, improving the accuracy of model parameters.

[0023] The gaze stability coefficient is calculated using the same analytical approach as data dispersion in statistics. It reflects the stability of the rhythm by the ratio of the mean to the standard deviation. The calculation method is as follows: ; In the formula, For gaze stability coefficient, This represents the average fixation duration over multiple consecutive cycles, in milliseconds. This represents the standard deviation of fixation duration, expressed in milliseconds. This coefficient reflects the degree of fluctuation in the wearer's fixation duration; a higher value indicates a more stable fixation rhythm. For example, if a wearer's average fixation duration over 10 consecutive fixations is 320 milliseconds with a standard deviation of 48 milliseconds, the fixation stability coefficient is approximately 6.67. Initial parameters can be obtained through a three-minute calibration during first use, and are updated every 50 valid samples thereafter.

[0024] The saccade rhythm coefficient is calculated using the same logic; simply substitute the saccade duration data. The baseline for saccade velocity is the average of the maximum velocities from 20 consecutive effective saccades. For example, if the average maximum velocity is 210 degrees per second, this can be stored as the baseline value. The model's continuous correction mechanism adapts to differences in eye movement under different wearer states. For instance, in a fatigued state, fixation duration shortens and saccade velocity decreases; the model can automatically adjust parameters to ensure prediction accuracy.

[0025] In one specific embodiment, the step of excluding eye movement data during blinking, rapid head turning, and out-of-field states further includes: continuously monitoring blinking features in eye movement signals; when a blinking signal is detected, removing eye movement samples collected within that blink cycle from the valid dataset; continuously monitoring head posture changes; when the speed of head posture change exceeds a preset head turning threshold, determining it as a rapid head turning state, and removing eye movement samples during the head turning process from the valid dataset; and continuously monitoring the pupil center position; when the pupil center position deviates from a preset visual field range, determining it as an out-of-field state, and removing eye movement samples during the out-of-field period from the valid dataset.

[0026] Specifically, blink recognition is based on the characteristics of pupil area changes. During normal vision, pupil area fluctuations do not exceed 10%, while during blinking, the area drops to less than 20% of the normal value within a short period. This is used as the basis for blink detection. After detection, samples within a 50-millisecond range before and after the blink are all removed to avoid interference from data due to eyelid partial occlusion. Head posture change data is output by the glasses' built-in six-axis sensor. The head-turning threshold can be set to 30 degrees per second. This value is derived from head movement statistics in everyday visual scenarios. During normal reading and observation, head turning speed is usually lower than this value; exceeding this value indicates a large head turn, in which case eye movement is passively driven by the head and does not reflect the voluntary gaze rhythm, and samples in the corresponding time period are all removed. The preset visual field is based on the central 80% area of ​​the eye-tracking image. When the pupil center deviates from this range, it means the wearer's gaze is not within the displayed image, and data for the corresponding time period is not included in the model calculation. After layer-by-layer filtering of these three types of abnormal data, the remaining samples are all valid data under normal voluntary vision. The model parameters fitted in this way better match the real gaze rhythm, reducing the error of subsequent window prediction.

[0027] In one specific embodiment, the step of predicting the next stable gaze window further includes: retrieving the gaze window interval duration and window duration stored in the current gaze rhythm baseline model; obtaining the end time of the most recent gaze state; adding the end time of the most recent gaze to the gaze window interval duration to obtain the predicted start time of the next gaze window; adding the predicted start time to the window duration duration to obtain the predicted end time of the next gaze window; assessing the time required for recognition calculation based on the current camera motion angular velocity and image complexity, and issuing a pre-calculation instruction to the target recognition module before the gaze window start time arrives; the target recognition module completes a new round of image content recognition calculation during the camera saccade phase and stores the recognition result in a cache queue for later retrieval.

[0028] Specifically, the fixation window interval refers to the time interval between two adjacent fixation states, corresponding to the total duration of saccades and brief adjustments, and is output by the baseline model based on historical data. The prediction logic is based on the periodic characteristics of the rhythm, using the end point of the previous fixation as a baseline, adding the interval duration to obtain the starting point of the next fixation, and then adding the duration to obtain the end point. Image complexity is obtained by using the edge detection operator to count the proportion of edge pixels in the image. A higher proportion indicates more image details and a longer recognition time. The corresponding estimated time can be obtained by looking up a preset mapping table. The design of the pre-computation mechanism references the timing optimization approach of pipeline scheduling, utilizing the period when the wearer's visual perception is weaker during the saccade phase to complete the recognition calculation, avoiding the occupation of computing resources during the stable fixation phase.

[0029] The latest issuance time of the pre-calculated instruction is determined using a timing-backward logic, and the calculation method is as follows: ; In the formula, The latest time the pre-calculated instruction will be issued, in milliseconds. The predicted gaze window start time, in milliseconds. The estimated processing time is measured in milliseconds. For example, if the predicted gaze window start time is 1200 milliseconds and the estimated processing time is 180 milliseconds, the latest possible execution time is 1020 milliseconds. Issuing the command before this time ensures the processing is completed before the gaze window arrives. Once the recognition result is stored in a first-in-first-out (FIFO) cache queue, it doesn't require repeated processing and can be directly retrieved when the update is needed, shortening the response latency for information updates.

[0030] In a specific embodiment, the process of issuing and executing the pre-calculation instruction further includes: the timing prediction scheduling module queries a preset performance mapping table based on the current camera motion angular velocity to obtain the estimated time consumption for recognition operations corresponding to different angular velocities; the estimated time consumption for recognition operations is subtracted from the start time of the gaze window to obtain the latest issuance time of the pre-calculation instruction; after the pre-calculation instruction is issued, the timing prediction scheduling module continuously monitors the completion status of the target recognition module's operation; if the operation is detected to be completed before the start time of the gaze window, the pre-calculation result is confirmed to be usable; if the operation is detected to be incomplete when the start time of the gaze window arrives, the update operation is postponed to the next gaze window for execution.

[0031] Specifically, the performance mapping table is constructed through prior calibration tests. During testing, the target recognition algorithm is run at different angular velocities, and the average computation time for each level is calculated to form a one-to-one mapping relationship. The higher the camera angular velocity, the greater the motion blur in the image, requiring an additional deblurring preprocessing step before recognition, thus increasing the computation time accordingly. The timing prediction scheduling module monitors the computation status by polling the status register, reading the completion status every 10 milliseconds to track the processing progress in real time. When the computation is completed ahead of schedule, the cached result remains in a waiting state and is not updated prematurely to avoid refreshing the content during the saccade phase. When the computation is not completed on time, the delayed update avoids the appearance of a partially completed recognition result within the fixation window and prevents the content refresh process from spanning the saccade and fixation phases, causing visual interference. The delayed update mechanism ensures that each dynamic information update is within a complete and stable fixation window, maintaining a consistent visual experience.

[0032] In a specific embodiment, the step of smoothing the position offset of static basic information based on real-time motion parameters further includes: obtaining the camera's turning angle and zoom ratio at the current moment, and calculating the current offset position of the static basic information in the image; obtaining the offset position of the static basic information at the previous moment, and calculating the position deviation between the current moment and the previous moment; using a quadratic smoothing interpolation algorithm to smooth the position deviation and eliminate image jumps caused by sudden changes in motion parameters; adding the processed deviation to the offset position at the previous moment to obtain the offset position at the current moment; setting an upper limit for the offset speed based on the eye saccade speed baseline output by the eye-tracking learning module, and limiting the offset speed when the calculated offset speed exceeds the upper limit; the static basic information remains displayed throughout the entire camera movement process, only undergoing position offset without content switching.

[0033] Specifically, the initial position of static basic information is bound to the coordinates of the corresponding object in the real scene. When the camera turns or zooms, the new position of the corresponding information in the current frame is calculated through a viewpoint transformation matrix. If the sensor data fluctuates when the original position deviation is directly superimposed, it will cause abrupt changes in static information, affecting visual continuity.

[0034] The smoothing reference exponential smoothing filter method eliminates abrupt changes by weighted fusion of historical and current values. The calculation logic is as follows: ; In the formula, This represents the smoothed positional deviation at the current moment, in pixels. This represents the smoothed deviation from the previous time step, expressed in pixels. This is the raw deviation calculated at the current moment, in pixels. This is the smoothing coefficient. A smoothing coefficient of 0.7 is acceptable, balancing the response speed of position tracking with jitter suppression. A coefficient that is too high will cause position tracking lag, while a coefficient that is too low will result in insufficient smoothing. The upper limit of the offset speed is set to 0.8 times the baseline of the saccade speed. For example, if the baseline saccade speed is 210 degrees per second, then the upper limit of the offset speed is 168 degrees per second. When the calculated offset speed exceeds this value, the movement will proceed at a constant speed according to the upper limit to avoid the information moving faster than the human eye can follow, causing visual fatigue. Static basic information remains displayed throughout the entire camera movement process, only undergoing positional offset without content switching, allowing the wearer to continuously read basic information even when adjusting the lens.

[0035] In a specific embodiment, the step of performing an update operation on dynamically changing information further includes: the temporal prediction scheduling module monitors the wearer's eye movement state in real time, and when it detects that the pupil center has entered the stable focusing area and the focusing duration has reached the fixation stability threshold, it determines that it has entered the stable fixation stage; within the stable fixation stage, the latest recognition result is extracted from the cache queue of the target recognition module; the dynamically changing information content in the latest recognition result is compared with the currently displayed content to identify the changed information items; the changed information items are updated, and the update process is implemented using a transparency transition method; the transition duration of the transparency transition is the first one-tenth of the fixation window duration; the transparency gradually transitions from the initial value to full display, completing the content switching; the entire update process is completed within the predicted fixation window duration.

[0036] Specifically, the determination of the stable gaze phase is used to verify the accuracy of the prediction window, ensuring that the wearer is indeed in a stable gaze state when the update operation is performed. Updating only the changed information items reduces the amount of rendering computation and shortens the transition time. The transparency transition uses a linear interpolation method, with the transparency of new content gradually increasing from 0 to full display, while the transparency of old content decreases synchronously, achieving a seamless switch. The transition duration is set to the first tenth of the gaze window's duration. For example, if the gaze window lasts for 300 milliseconds, the transition duration is 30 milliseconds. This period is in the initial visual adjustment phase of gaze, and the content switch is not easily noticeable, so it will not interfere with subsequent information reading. The entire update process is completed within the gaze window, and the content remains stable in subsequent periods, allowing the wearer to clearly read the complete information. This update mechanism avoids irregular content refreshes from interrupting the gaze rhythm, reduces the probability of visual afterimages, and is adapted to the slower visual reaction speed of the elderly.

[0037] In a specific embodiment, the step of determining whether the wearer has entered the stable fixation stage further includes: continuously acquiring the wearer's pupil center position coordinate sequence; calculating the variance value of the pupil center position within multiple consecutive sampling periods; determining that the pupil center has entered a stable focusing state when the variance value is continuously less than the fixation stability variance threshold; recording the start time of the pupil center entering the stable focusing state; calculating the duration of continuous focusing from the start time of entering the stable focusing state to the current time; and determining that the wearer has entered the stable fixation stage when the duration of continuous focusing reaches the fixation stability threshold.

[0038] Specifically, the pupil center position coordinate sequence is output by the eye-tracking image processing unit, with the sampling frequency consistent with that of the eye-tracking acquisition unit. The position variance is calculated using statistical methods for the dispersion of two-dimensional data, reflecting the eye's stable state through the degree of fluctuation in coordinates over multiple consecutive frames. The calculation method is as follows: ; In the formula, The variance of the pupil center position is expressed in squared pixels. The x and y coordinates of the pupil center within a single sampling period, in pixels. The variance is the average of the coordinates over n consecutive sampling periods, expressed in pixels, where n is the number of sampling periods. A sampling period of 5 frames is chosen to balance sensitivity and interference resistance; too few frames increase susceptibility to micro-jitter interference and lead to misjudgments, while too many frames result in delayed judgment. For example, if the average x-coordinate of the pupil center is 320 pixels and the average y-coordinate is 240 pixels over 5 consecutive frames, the average of the squared deviations of the coordinates in each frame yields a variance of 0.3 pixels squared. The fixation stability variance threshold can be set to 0.5 pixels squared, based on statistical analysis of pupil micro-jitter amplitude during normal fixation. A fixation stability threshold of 80 milliseconds aligns with the physiological characteristics of human eye fixation stability. Timing and judgment are only made after the variance condition is consistently met, avoiding misjudgments caused by brief periods of positional stability and improving the accuracy of stage-based judgments.

[0039] In one specific embodiment, the method further includes a remote collaboration step: the remote collaboration module receives an authentication request sent by the assistant through a remote terminal and completes the assistant's authentication; after successful authentication, the remote collaboration module receives a camera adjustment command sent by the assistant; the camera adjustment command is synchronously forwarded to the image acquisition module and the timing prediction scheduling module; the module receives guidance information sent by the assistant and categorizes the guidance information into a dynamic change information queue; the remote collaboration module provides real-time prompts to the assistant terminal regarding the wearer's current gaze status, including three states: gaze, saccade, or gaze away from the field of vision; the assistant selects an appropriate adjustment time based on the gaze status prompts, and after the adjustment command is issued, the timing prediction scheduling module automatically enters the rhythm matching scheduling mode.

[0040] Specifically, identity verification employs a dual mechanism of account permission verification and temporary verification code. Only after successful verification can the assistant access the device, ensuring the security and privacy of remote assistance. Camera adjustment commands are simultaneously forwarded to two modules: the image acquisition module executes the physical lens adjustment action, while the temporal prediction and scheduling module obtains motion parameters in advance and adjusts its prediction and scheduling strategies. Guidance information is placed in the dynamic change information queue and, following the update rules of dynamic information, is only refreshed and displayed within the stable gaze window, without arbitrarily popping up to interrupt the wearer. Gazing status prompts are synchronized to the assistant in real time. The assistant can choose to issue lens adjustment commands during the saccade phase, when the wearer's visual perception is weaker, minimizing the impact of image changes caused by lens adjustment. In rhythm matching scheduling mode, the scheduling module optimizes the window prediction accuracy by referring to command parameters and gaze rhythm, ensuring that dynamic information updates perfectly match the wearer's eye movement rhythm. Static information follows smoothly throughout the remote lens adjustment process, while dynamic information updates in a timely manner. The wearer can complete lens adjustment without interrupting reading, improving the efficiency of remote assistance and reducing visual burden.

[0041] This invention also provides AI smart glasses based on computer vision, comprising: an eye-tracking learning module for collecting the wearer's eye-tracking data, fitting a gaze rhythm baseline model based on the eye-tracking data, obtaining parameters for the interval duration and duration of the gaze window, and continuously excluding abnormal data to correct the model parameters; a target recognition module for receiving real-time video streams captured by a camera, performing content recognition on the video stream, and outputting the recognition results into two categories: static basic information and dynamic change information; a temporal prediction and scheduling module for receiving the gaze rhythm baseline model data and camera motion parameters output by the eye-tracking learning module, predicting the time period of the next stable gaze window based on the gaze rhythm baseline model, evaluating the recognition computation time based on the camera motion parameters and issuing a pre-calculation instruction, and sending an update instruction to the layered rendering module during the predicted gaze window time period; and a layered rendering module for receiving the static basic information and dynamic change information output by the target recognition module, maintaining the display of static basic information while the camera is in continuous motion and performing smooth position shifts based on the motion parameters, and completing the update and switching of dynamic change information within the gaze window after receiving the update instruction, with the switching process using a transparency transition method.

[0042] Specifically, the eye-tracking learning module continuously acquires raw eye-tracking data through the infrared acquisition unit. It first performs an abnormal data removal process to filter out valid samples from normal visual scenarios. Then, based on these valid samples, it calculates the fixation stability coefficient, saccade rhythm coefficient, and saccade speed baseline, fitting and generating a fixation rhythm baseline model. The parameters are continuously adjusted as data accumulates, and finally, the model parameters are output to the temporal prediction and scheduling module. The target recognition module receives the video stream from the front-facing camera, runs a computer vision recognition algorithm, and outputs scene recognition results. These results are categorized into static basic information and dynamic change information based on content attributes and transmitted to the layered rendering module and the cache unit, respectively. The temporal prediction and scheduling module receives camera motion parameters and rhythm model data. It first predicts the start and end times of the next stable fixation window, then queries the performance mapping table to obtain the estimated time for recognition computation. It then works backward to determine the issuance time of the pre-calculation command and schedules the target recognition module to complete the pre-calculation according to the temporal issuance command. When the fixation window arrives, it sends a dynamic information update command to the layered rendering module. The layered rendering module adopts a two-layer rendering architecture. The static information layer receives camera motion parameters in real time, and continuously updates the display position after smooth interpolation. The dynamic information layer only performs content switching when it receives an update command, and loads transparency transition effects during the switching process. After the camera stops moving, the timing prediction scheduling module closes the rhythm scheduling mode, and the system resumes full information updates at a fixed frequency.

[0043] In summary, this invention uses eye movement data collected from the wearer to fit a baseline model of the gaze rhythm. Combined with real-time camera motion parameters, it predicts the duration of stable gaze windows. The recognition results are categorized into static basic information and dynamic change information, and layered rendering scheduling is implemented. Static basic information is continuously displayed during continuous camera movement, with smooth positional shifts based on motion parameters. This allows the wearer to continuously acquire basic scene information throughout the lens adjustment process, without waiting for the movement to stop before reading content. Dynamic change information is updated only within the predicted stable gaze window. The update process uses a transparency transition to avoid refreshing content during saccade periods, preventing irregular content refreshes from interrupting the wearer's gaze rhythm, reducing visual afterimages, and minimizing the abruptness of content switching. This is suitable for the physiological characteristics of the elderly, who have slower eye movement speeds and longer stable gaze durations. Combined with a pre-calculation mechanism for the saccade phase, recognition calculations can be completed during periods of weaker visual perception, shortening information update response delays. Simultaneously, abnormal data filtering continuously corrects the rhythm model parameters, ensuring prediction accuracy under different usage conditions. In remote collaboration scenarios, the wearer's gaze state can be synchronized with the assisting end, making it easier for the assister to choose the appropriate adjustment time. This allows the dynamic information update to match the wearer's eye movement rhythm, shortening the time required for a single remote assistance session, reducing eye fatigue caused by frequent screen jumps, and steadily improving the readability and smoothness of use during lens adjustment.

[0044] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A human-computer interaction method for AI smart glasses based on computer vision, characterized in that, include: The wearer's eye movement data is collected, and a fixation rhythm baseline model is fitted based on the eye movement data to obtain the interval duration and duration of the fixation window; Real-time acquisition of parameters such as turning angle, zoom ratio, angular velocity, and angular acceleration during camera operation; Based on the gaze rhythm baseline model and camera motion parameters, predict the start time and duration of the next stable gaze window; Content recognition is performed on the real-time video stream captured by the camera, and the recognition results are divided into two categories: static basic information and dynamic change information. When the camera is detected to be in continuous motion, the static basic information is kept continuously displayed in the picture, and the position of the static basic information is smoothly shifted according to the real-time motion parameters. When the predicted gaze window start time arrives, an update operation is performed on the dynamically changing information, and the update process uses an transparency transition method; Once the camera stops moving, resume the normal recognition and update frequency.

2. The human-computer interaction method for AI smart glasses based on computer vision according to claim 1, characterized in that, The step of fitting the gaze rhythm baseline model further includes: Continuously collect eye movement data from wearers during daily use; Record the time interval from start to end of each gaze, the time interval from start to end of each eye saccade, and the maximum speed of movement during each eye saccade. Eye movement data during blinking, rapid head turning, and when the gaze leaves the field of vision are excluded, and only valid samples from normal visual scenarios are retained. The fixation stability coefficient is calculated based on fixation duration data recorded over multiple consecutive periods, the saccade rhythm coefficient is calculated based on saccade duration data recorded over multiple consecutive periods, and the saccade speed baseline is calculated based on maximum saccade speed data recorded over multiple consecutive periods. The fixation stability coefficient, saccade rhythm coefficient, and saccade velocity baseline are stored as the core parameters of the fixation rhythm baseline model. The gaze rhythm baseline model is continuously revised as usage time increases to adapt to eye movement differences under different usage conditions.

3. The human-computer interaction method for AI smart glasses based on computer vision according to claim 2, characterized in that, The step of excluding eye movement data during blinking, rapid head turning, and when the gaze leaves the field of vision further includes: Continuously monitor blinking features in eye movement signals. When a blinking signal is detected, remove the eye movement samples collected within that blink cycle from the valid dataset. Continuously monitor changes in head posture. When the speed of head posture change exceeds the preset head turning threshold, it is judged as a rapid head turning state, and the eye movement samples during the head turning process are removed from the valid dataset. The pupil center position is continuously monitored. When the pupil center position deviates from the preset visual field range, it is judged as a state of gaze leaving the visual field. The eye movement samples during the gaze-leaning period are removed from the valid dataset.

4. The human-computer interaction method for AI smart glasses based on computer vision according to claim 1, characterized in that, The step of predicting the next stable gaze window further includes: Retrieve the gaze window interval duration and window duration stored in the current gaze rhythm baseline model; Get the end time of the most recent gaze state; Add the most recent fixation end time to the fixation window interval to obtain the predicted start time of the next fixation window; The predicted start time value is added to the window duration to obtain the predicted end time value of the next fixation window; Based on the current camera motion angular velocity and image complexity, the time required for recognition calculation is estimated, and a pre-calculation instruction is sent to the target recognition module before the start time of the gaze window. The target recognition module completes a new round of image content recognition calculations during the camera's eye-scratching phase and stores the recognition results in a cache queue for later retrieval.

5. The human-computer interaction method for AI smart glasses based on computer vision according to claim 4, characterized in that, The process of issuing and executing the pre-calculation instructions further includes: The timing prediction and scheduling module queries a preset performance mapping table based on the current camera motion angular velocity to obtain the estimated time consumption for recognition operations corresponding to different angular velocities. Subtract the estimated processing time from the start time of the gaze window to obtain the latest time when the pre-calculated command is issued; After the pre-computation instruction is issued, the timing prediction scheduling module continuously monitors the completion status of the target recognition module's calculations; If the calculation is detected to be completed before the start time of the gaze window, then the pre-calculated result is confirmed to be usable; If the computation is not completed when the start time of the gaze window arrives, the update operation will be postponed to the next gaze window.

6. The human-computer interaction method for AI smart glasses based on computer vision according to claim 1, characterized in that, The step of smoothing the position offset of static basic information based on real-time motion parameters further includes: Obtain the camera's current turning angle and zoom level, and calculate the current offset position of static basic information in the image; Obtain the offset position of the static basic information at the previous moment, and calculate the position deviation between the current moment and the previous moment; A quadratic smoothing interpolation algorithm is used to smooth the positional deviation and eliminate screen jitter caused by abrupt changes in motion parameters; The processed deviation is added to the offset position of the previous moment to obtain the offset position of the current moment. The upper limit of the offset speed is set based on the baseline of eye saccade speed output by the eye-tracking learning module. When the calculated offset speed exceeds the upper limit, the offset speed is restricted. Static basic information remains displayed throughout the entire camera movement process, only undergoing positional shifts without content switching.

7. The human-computer interaction method for AI smart glasses based on computer vision according to claim 1, characterized in that, The step of performing an update operation on dynamically changing information further includes: The timing prediction and scheduling module monitors the wearer's eye movement in real time. When it detects that the pupil center has entered the stable focusing area and the focusing duration has reached the fixation stability threshold, it is determined that the stable fixation stage has been entered. During the stable gaze phase, the latest recognition result is extracted from the cache queue of the target recognition module; Compare the dynamically changing information in the latest identification results with the currently displayed content to identify the information items that have changed. Update the content of changed information items, and use a transparent transition method to implement the update process; The transition time for transparency is one-tenth of the duration of the gaze window; The transparency gradually transitions from its initial value to full display, completing the content switching process; The entire update process is completed within the predicted duration of the gaze window.

8. The human-computer interaction method for AI smart glasses based on computer vision according to claim 7, characterized in that, The step of determining whether a person has entered the stable fixation phase further includes: Continuously collect the coordinate sequence of the wearer's pupil center position; Calculate the variance of the pupil center position over multiple consecutive sampling periods; When the variance value is continuously less than the fixation stability variance threshold, it is determined that the pupil center has entered a stable focusing state; Record the moment when the pupil center enters a stable focusing state; Calculate the duration of continuous focusing from the start of entering the stable focusing state to the current time; When the duration of sustained focus reaches the fixation stability threshold, it is determined that the fixation has entered the stable fixation stage.

9. The human-computer interaction method for AI smart glasses based on computer vision according to claim 1, characterized in that, The method also includes remote collaboration steps: The remote collaboration module receives the authentication request sent by the assistant through a remote terminal and completes the assistant's authentication. After identity verification, the remote collaboration module receives camera adjustment instructions sent by the assistant; The camera adjustment commands are synchronously forwarded to the image acquisition module and the timing prediction and scheduling module. Receive guidance information sent by the assistant and add the guidance information to the dynamic information queue; The remote collaboration module provides real-time feedback to the assisting terminal on the wearer's current gaze status, including three states: gaze, saccade, or gaze out of sight. The assistant selects the appropriate adjustment time based on the gaze status prompts. After the adjustment command is issued, the timing prediction scheduling module automatically enters the rhythm matching scheduling mode.

10. A computer vision-based AI smart glasses, used to implement the computer vision-based AI smart glasses human-computer interaction method according to any one of claims 1 to 9, comprising: The eye-tracking learning module is used to collect the wearer's eye-tracking data, fit a fixation rhythm baseline model based on the eye-tracking data, obtain the interval duration and duration parameters of the fixation window, and continuously eliminate abnormal data to correct the model parameters. The target recognition module is used to receive real-time video streams captured by the camera, perform content recognition on the video streams, and output the recognition results into two categories: static basic information and dynamic change information. The temporal prediction scheduling module receives the gaze rhythm baseline model data and camera motion parameters output by the eye-tracking learning module, predicts the time period of the next stable gaze window based on the gaze rhythm baseline model, evaluates the recognition calculation time based on the camera motion parameters and issues a pre-calculation instruction, and sends an update instruction to the layered rendering module during the predicted gaze window time period. The layered rendering module receives static basic information and dynamic change information output by the target recognition module. It maintains the display of static basic information while the camera is in continuous motion and performs smooth position shifts based on motion parameters. Upon receiving an update command, it completes the update and switching of dynamic change information within the gaze window, using a transparency transition method during the switching process.

Images