Gaze deviation reminding method, smart glasses, system and readable storage medium

Abstract

The application discloses a line-of-sight deviation reminding method, smart glasses, a system and a readable storage medium, relates to the technical field of glasses, and is applied to the smart glasses. The smart glasses comprise a first camera module and a second camera module. The first camera module is an infrared camera module. The method comprises the following steps: acquiring a first eye image collected by the infrared camera module and a second eye image collected by the second camera module; determining a corneal center position based on the first eye image, and determining a pupil center position based on the first eye image and the second eye image; determining a current line-of-sight direction based on the corneal center position and the pupil center position, so as to output line-of-sight deviation prompt information when the included angle between the current line-of-sight direction and a preset baseline line-of-sight direction is greater than a preset threshold value, and remind a user that the line of sight deviates from the optical center of the glasses. The application realizes timely detection and reminding of line-of-sight deviation.

Description

Technical Field

[0001] This application relates to the field of eyewear technology, and in particular to a method for alerting eye deviation, smart glasses, a system, and a readable storage medium. Background Technology

[0002] With the rapid development of augmented reality, virtual reality, and smart wearable devices, smart glasses, as an emerging human-computer interaction terminal, are gradually being applied in daily life, office assistance, and vision health management. For users wearing optical lenses (especially defocused lenses and progressive multifocal lenses), whether their line of sight accurately passes through the optical center of the lens directly affects visual clarity, wearing comfort, and myopia control.

[0003] Currently, the detection and reminders regarding the alignment of the line of sight with the optical center of the lenses when wearing glasses still rely on the traditional static assessment method used in the eyewear industry. Users undergo a one-time static line of sight calibration using professional optical instruments when initially fitting glasses or adjusting frames; this is considered a complete fit thereafter, and no further dynamic monitoring is performed during daily wear.

[0004] However, this static, one-off detection and reminder method has significant shortcomings. During daily wear of smart glasses, factors such as frame displacement and changes in head posture can cause the line of sight to deviate from the optical center of the lenses. Because current methods cannot detect and remind users of such deviations in a timely manner, wearers can only passively perceive the line of sight shift through visual discomfort, which can easily lead to long-term visual burden and fail to effectively guarantee the visual correction and usability of smart glasses. Therefore, how to achieve timely detection and reminders of line of sight deviation has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to provide a method, smart glasses, system, and readable storage medium for alerting eye deviation, aiming to solve the technical problem of how to achieve timely detection and alerting of eye deviation.

[0006] To achieve the above objectives, this application provides a gaze deviation reminder method for use in smart glasses, the gaze deviation reminder method comprising the following steps:

[0007] Acquire a first eye image captured by the infrared camera module and a second eye image captured by the second camera module; The corneal center position is determined based on the first eye image, and the pupil center position is determined based on the first eye image and the second eye image. Based on the corneal center position and the pupil center position, the current gaze direction is determined. When the angle between the current gaze direction and the preset baseline gaze direction is greater than a preset threshold, a gaze deviation prompt message is output to remind the user that their gaze has deviated from the optical center of the lens.

[0008] In one embodiment, the step of determining the corneal center position based on the first eye image includes: Obtain the pre-calibrated radius of curvature of the user's cornea; Identify at least one corneal reflection point based on the first eye image; Based on the corneal reflection point, the corneal radius of curvature, and the spatial position of at least one pre-stored infrared light source, calculate the three-dimensional coordinates of the corneal center in the world coordinate system; The location of the corneal center is determined by the three-dimensional coordinates of the corneal center in the world coordinate system.

[0009] In one embodiment, the step of determining the pupil center position based on the first eye image and the second eye image includes: The pupil center is detected based on the first eye image, and the two-dimensional projection coordinates of the first pupil center are obtained. The pupil center is detected based on the second eye image, and the two-dimensional projection coordinates of the second pupil center are obtained. Based on the pre-calibrated relative pose parameters between the infrared camera module and the second camera module, triangulation is performed on the two-dimensional projection coordinates of the first pupil center and the two-dimensional projection coordinates of the second pupil center to obtain the three-dimensional coordinates of the pupil center in the world coordinate system. The three-dimensional coordinates of the pupil center in the world coordinate system are used to determine the position of the pupil center.

[0010] In one embodiment, the step of determining the current gaze direction based on the corneal center position and the pupil center position includes: Obtain the rotation matrix corresponding to the pre-calibrated Kappa angle; The vector pointing from the center of the cornea to the center of the pupil is defined as the optical axis direction; Multiplying the optical axis direction by the rotation matrix yields the current line-of-sight direction.

[0011] In one embodiment, the gaze alert method further includes: In response to the calibration command, the user is prompted to keep their head still and look at the target object at the preset spatial position through the optical center of the lens; The first gaze direction is calculated based on the first eye image and the second eye image when the user is looking at the target object; The first line of sight is determined as the preset baseline line of sight and stored.

[0012] In one embodiment, the smart glasses further include an inertial measurement unit. After the step of determining the current gaze direction based on the corneal center position and the pupil center position, the method further includes: Acquire the head posture data detected by the inertial measurement unit; When the change in the head posture data is detected to be greater than a preset posture threshold, a temporary gaze monitoring window is triggered. Within the temporary line of sight monitoring window, if the deviation between the calculated line of sight direction and the preset baseline line of sight direction shows an increasing trend over multiple consecutive calculations, it is determined that the frame has physically slipped. Output a frame slippage alert message to remind the user to re-wear the glasses or perform baseline calibration.

[0013] In addition, to achieve the above objectives, this application also provides a smart glasses, which includes a first camera module, a second camera module and a processor, wherein the first camera module is an infrared camera module; The infrared camera module is used to acquire images of the first eye. The second camera module is used to capture images of the second eye; The processor is used to perform the steps of the gaze deviation warning method described above.

[0014] In one embodiment, the second camera module is an RGB camera module.

[0015] In addition, to achieve the above objectives, this application also provides a gaze deviation reminder system, which includes smart glasses and an edge device connected in communication. The smart glasses include a first camera module and a second camera module, wherein the first camera module is an infrared camera module. The smart glasses are used to capture eye images and send the eye images to the edge device; The edge device is used to perform the steps of the gaze deviation warning method described above. In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of the gaze deviation reminder method as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the gaze deviation reminder method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This application embodiment utilizes a dual-camera module working collaboratively to achieve dynamic, real-time detection and timely alerts for deviations in gaze from the optical center of the lens. This effectively solves the problem that existing static, one-time assessment methods cannot handle dynamic factors such as frame displacement and head posture changes during daily wear. Specifically, the first eye image acquired by the infrared camera module determines the corneal center position, which serves as a stable physiological reference point and is not easily affected by changes in lighting conditions. Simultaneously, by combining the first and second eye images acquired by the infrared and second camera modules respectively, the pupil center position can be determined through binocular vision principles or image fusion processing, fully utilizing dual-viewpoint information to improve the accuracy and robustness of pupil positioning. Based on the relative positional relationship between the corneal center and pupil center—two key ocular feature points—the current gaze direction can be calculated and then compared in real-time with a preset baseline gaze direction. When the angle between the two exceeds a preset threshold, a gaze deviation alert is output, allowing users to be aware that their gaze has deviated from the optical center of the lens without relying on passive perception of visual discomfort, thus enabling them to proactively adjust their wearing posture or gaze habits. By using the collaborative acquisition and fusion processing of dual camera modules, the limitation of insufficient positioning accuracy of a single camera in complex wearing scenarios is overcome. This enables a shift from static one-time assessment to dynamic continuous monitoring, ensuring that the direction of gaze can still be accurately tracked and timely reminders can be triggered when the frame shifts or the head posture changes. This effectively reduces long-term visual burden and ensures the visual correction and usage effect of smart glasses. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the line-of-sight deviation warning method of this application; Figure 2 This is a schematic diagram of the visual reminder process involved in one embodiment of the gaze deviation reminder method of this application; Figure 3 This is a schematic diagram of the eyeglass structure of the smart glasses in this application; Figure 4 This is a schematic diagram of the system architecture of the line-of-sight warning system in this application; Figure 5 This is a schematic diagram of the hardware operating environment of the line-of-sight deviation reminder method device in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Currently, the eyewear industry only conducts a one-time static assessment at the initial stage of prescription, failing to monitor dynamic visual shifts caused by frame slippage and changes in head posture during daily wear. Wearing glasses is a dynamic process; frames inevitably experience slight displacements due to gravity, facial muscle movements, and external impacts, leading to changes in the relative position of the lens's optical center and the eye. Existing technology cannot monitor and provide real-time feedback on this dynamic shift. Wearers often only realize the frame shift when they experience significant visual fatigue or discomfort, by which time prolonged visual strain has already occurred. This is particularly true for wearers of progressive multifocal lenses, whose vision requires observing objects at different distances through different areas of the lens, demanding higher precision in aligning their gaze with the optical center. However, current technology cannot provide real-time guidance, making it difficult for wearers to adapt to the lens design.

[0024] To address the aforementioned issues, the main solution of this application is as follows: Smart glasses are equipped with a first camera module and a second camera module. The first camera module is an infrared camera module. The gaze deviation warning method includes the following steps: acquiring a first eye image captured by the infrared camera module and a second eye image captured by the second camera module; determining the corneal center position based on the first eye image; determining the pupil center position based on the first and second eye images; determining the current gaze direction based on the corneal center position and the pupil center position; and outputting gaze deviation warning information when the angle between the current gaze direction and a preset baseline gaze direction is greater than a preset threshold, thus reminding the user that their gaze has deviated from the optical center of the lens.

[0025] This application determines the corneal center position using a first-eye image acquired by an infrared camera module. This position serves as a stable physiological reference point, unaffected by changes in lighting conditions. Simultaneously, by combining the first-eye and second-eye images acquired by the infrared and second camera modules respectively, the pupil center position can be determined through binocular vision principles or image fusion processing, fully utilizing dual-viewpoint information to improve the accuracy and robustness of pupil positioning. Based on the relative positional relationship between the corneal center and pupil center—two key ocular feature points—the current gaze direction can be calculated and compared in real-time with a preset baseline gaze direction. When the angle between the two exceeds a preset threshold, a gaze deviation warning is output, allowing users to promptly recognize that their gaze has deviated from the lens's optical center without relying on passive perception of visual discomfort, thus enabling them to proactively adjust their wearing posture or gaze habits. By using the collaborative acquisition and fusion processing of dual camera modules, the limitation of insufficient positioning accuracy of a single camera in complex wearing scenarios is overcome. This enables a shift from static one-time assessment to dynamic continuous monitoring, ensuring that the direction of gaze can still be accurately tracked and timely reminders can be triggered when the frame shifts or the head posture changes. This effectively reduces long-term visual burden and ensures the visual correction and usage effect of smart glasses.

[0026] It should be noted that the executing entity of the various embodiments of the gaze deviation reminder method of this application can be an edge device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, server, etc., or a smart glasses that can achieve the above functions. The various embodiments of the gaze deviation reminder method of this application do not impose specific limitations on this.

[0027] Based on this, this application proposes a gaze deviation warning method according to a first embodiment. In this embodiment, it is applied to smart glasses, which include a first camera module and a second camera module. The first camera module is an infrared camera module. (Refer to...) Figure 1 As shown, the gaze deviation warning method includes the following steps S10~S30: Step S10: Obtain the first eye image captured by the infrared camera module and the second eye image captured by the second camera module; The first camera module is an infrared camera module, which is equipped with an infrared light source. When acquiring eye images, it can project near-infrared light onto the human eye, thereby forming a clear reflective spot (i.e., the Puerchin spot) on the corneal surface and effectively filtering out the interference of ambient visible light, ensuring that high-quality eye feature images can still be stably acquired under complex lighting conditions (such as backlight, dim or strong light environments).

[0028] The second camera module can be a visible light camera module or an infrared camera module, preferably a visible light camera module, such as an RGB camera module. Using a visible light camera module as the second camera module allows for the acquisition of color images containing rich iris texture, scleral boundaries, and periocular skin details. This provides high-resolution edge and texture information for subsequent precise pupil center localization, complementing the corneal reflection features in the infrared image and significantly improving the accuracy and robustness of eye feature point extraction. Through the fusion processing of the dual-module images, the limitations of a single image source can be effectively overcome, laying a solid foundation for accurate pupil center localization and gaze direction calculation.

[0029] When the smart glasses are in working condition, the infrared camera module and the second camera module can be started synchronously and maintain the same frame rate to collect real-time synchronous images of the wearer's eye area. The infrared grayscale image of the eye collected by the infrared camera module is the first eye image, and the eye image of the same viewpoint collected by the second camera module is the second eye image.

[0030] Step S20: Determine the corneal center position based on the first eye image, and determine the pupil center position based on the first eye image and the second eye image; The first eye image acquired by the first camera module (infrared camera module) is preprocessed and feature recognized. The infrared reflective spot (Pulchin spot) on the corneal surface is extracted by methods such as grayscale threshold segmentation and connected component analysis. Combined with the optical reflection characteristics of the cornea and the pre-calibrated camera module parameters, the three-dimensional spatial coordinates of the corneal center are calculated, that is, the position of the corneal center is obtained.

[0031] The localization of the pupil center relies on the collaborative processing of the first and second eye images: using the principle of binocular stereo vision, the two-dimensional coordinates of the pupil center in the left and right images are obtained by synchronously segmenting and feature matching the pupil region in the two images. Then, based on the parallax information of the binocular camera module and the principle of triangulation, the three-dimensional spatial coordinates of the pupil center are calculated, thus obtaining the position of the pupil center.

[0032] Step S30: Based on the corneal center position and the pupil center position, determine the current gaze direction, and when the angle between the current gaze direction and the preset baseline gaze direction is greater than a preset threshold, output gaze deviation prompt information to remind the user that the gaze has deviated from the optical center of the lens.

[0033] After obtaining the three-dimensional spatial coordinates of the corneal center and the pupil center, the vector formed by connecting these two points can represent the optical axis direction of the eye; through pre-performed personal calibration, the optical axis direction can be converted into the visual axis direction consistent with the actual gaze direction, thereby obtaining the current gaze direction.

[0034] The system compares the current line of sight with a preset baseline line of sight. This preset baseline line of sight is a reference direction that is pre-measured and stored in the same way when the user is wearing the smart glasses, looking straight ahead, and with their gaze passing through the optical center of the lens. The system calculates the spatial angle between the current line of sight and the baseline direction in real time. If this angle exceeds a preset threshold (which can be set according to lens type, individual wearer differences, and comfort requirements, for example, 0.5° to 2°), it is determined that the line of sight has deviated from the optical center of the lens. At this time, a prompting module (such as a vibration motor, speaker, or display unit) outputs a line-of-sight deviation prompt, using sound, light, or tactile signals to promptly remind the user to adjust their head posture or the position of the frames, prompting the line of sight to return to the optical center area. This ensures visual clarity and wearing comfort, effectively utilizing the lens's visual correction or myopia control functions.

[0035] Furthermore, multiple threshold levels can be set according to specific business needs, and graded reminders can be provided to users through vibration intensity changes, voice prompts, or pop-ups in the accompanying application.

[0036] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of determining the corneal center position based on the first eye image includes: Step A10: Obtain the pre-calibrated corneal curvature radius; This parameter is the average radius of curvature of the anterior corneal surface, measured by a professional corneal topographer or corneal topography instrument when the user first wears the smart glasses. It is an individualized physiological characteristic and can be pre-stored in the smart glasses' storage module. This fundamental data ensures that subsequent calculations of the corneal center's three-dimensional coordinates accurately reflect the user's actual eye structure, thus avoiding positioning errors caused by generic curvature parameters.

[0037] Step A20: Identify at least one corneal reflection point based on the first eye image; The first eye image is acquired by an infrared camera module, which integrates at least one infrared LED light source. Near-infrared light emitted by the light source illuminates the anterior surface of the wearer's cornea, causing specular reflection and forming corneal reflection points (i.e., Purkinje spots). In the image processing stage, the first eye image is preprocessed, sequentially performing grayscale enhancement, noise filtering, threshold segmentation, and connected component analysis to effectively filter out background interference areas such as eyelids and eyelashes. The corneal reflection points are located by identifying bright pixels in the image with brightness exceeding a preset threshold.

[0038] Furthermore, to improve the accuracy of subsequent calculations, it is preferable to identify 3 to 6 discretely distributed corneal reflection points to ensure the reliability of collaborative calculation of multiple reflection points.

[0039] Step A30: Based on the corneal reflection point, the corneal radius of curvature, and the spatial position of at least one pre-stored infrared light source, calculate the three-dimensional coordinates of the corneal center in the world coordinate system. It should be noted that the calculation method for the three-dimensional coordinates of the corneal center is not unique and can be flexibly selected according to the actual hardware configuration and accuracy requirements. For example, when configuring a single infrared light source, the geometric relationship of single-point reflection can be used in conjunction with the corneal curvature radius for calculation; when configuring multiple infrared light sources, multiple reflection points can be used to jointly construct an overdetermined system of equations, and the optimal solution can be obtained through optimization methods; alternatively, a spherical fitting algorithm can be used to fit the corneal sphere using multiple reflection points and determine the position of the sphere's center. This embodiment provides a preferred implementation method, as follows: First, the spatial position parameters of the infrared LED light sources, pre-calibrated before the smart glasses left the factory, are retrieved. These parameters, representing the three-dimensional coordinates of each infrared light source in the world coordinate system (usually with the optical center of the lens as the origin), are stored in the processing module. Considering the optical characteristic that the anterior surface of the cornea is approximately spherical, and according to the law of spherical reflection, the normal to the corneal reflection point must pass through the corneal center (i.e., the center of the corneal sphere), and the distance from the corneal reflection point to the corneal center is equal to the corneal radius of curvature. Using each corneal reflection point as a reference, and combining the spatial position of the corresponding infrared light source, a spherical reflection constraint equation is established. The optimal solution is obtained by solving the overdetermined system of multiple constraint equations using the least squares method, effectively eliminating minor errors caused by image recognition and light source calibration. Finally, the three-dimensional coordinates (Xc, Yc, Zc) of the corneal center in the world coordinate system are calculated.

[0040] Step A40: Determine the three-dimensional coordinates of the corneal center in the world coordinate system as the location of the corneal center.

[0041] This location data, serving as a feature point characterizing the spatial orientation of the eyeball, is used together with the subsequently determined pupil center position for calculating the direction of gaze. By employing the aforementioned calculation method based on the geometric relationship between corneal curvature and reflective points, the limitations of missing depth information in monocular images can be effectively reduced, achieving high-precision three-dimensional positioning of the corneal center and providing a reliable reference point for gaze estimation.

[0042] In one possible implementation, the step of determining the pupil center position based on the first eye image and the second eye image includes: Step B10: Detect the pupil center based on the first eye image to obtain the two-dimensional projection coordinates of the first pupil center; Since the first eye image is captured by an infrared camera module, the pupil region typically appears as a dark area with low grayscale values ​​against the iris background under the infrared spectrum, creating a sharp contrast with the surrounding iris. By performing image processing on the first eye image, including grayscale analysis, edge detection, and ellipse fitting, the pupil's contour can be extracted, and then the two-dimensional coordinates of its center point in the infrared camera coordinate system can be calculated, denoted as the two-dimensional projection coordinates of the first pupil center. These coordinates reflect the projected position of the pupil center on the imaging plane of the infrared camera module.

[0043] Step B20: Detect the pupil center based on the second eye image to obtain the two-dimensional projection coordinates of the second pupil center; In this embodiment, the second camera module is an RGB camera module, which captures a second eye image that provides rich iris texture and pupil edge details. For this image, algorithms such as edge detection, threshold segmentation, or deep learning can be used to identify the pupil region and fit the pupil contour to determine its center point, obtaining the two-dimensional projection coordinates of the second pupil center in the RGB camera coordinate system. These coordinates, along with the first pupil center coordinates, originate from different viewing angles of the same eye region at the same time, providing parallax information for subsequent three-dimensional reconstruction.

[0044] Step B30: Based on the pre-calibrated relative pose parameters between the infrared camera module and the second camera module, perform triangulation calculation on the two-dimensional projection coordinates of the first pupil center and the two-dimensional projection coordinates of the second pupil center to obtain the three-dimensional coordinates of the pupil center in the world coordinate system. Before leaving the factory or before use, smart glasses undergo binocular system calibration to obtain the relative pose parameters between the two camera modules, including rotation matrix and translation vector, as well as their respective internal parameters (such as focal length, principal point coordinates, and distortion coefficients).

[0045] Based on the principle of binocular stereo vision, the two obtained two-dimensional projected coordinates are used as input. Combined with the aforementioned calibration parameters, the position of a spatial point in three-dimensional space is calculated using triangulation. Specifically, by utilizing the geometric relationship between the optical centers of the two camera modules and the imaging point, the intersection of two spatial rays is solved, thereby obtaining the precise three-dimensional coordinates of the pupil center in the world coordinate system. This calculation process effectively eliminates the problem of missing depth information in monocular images, achieving high-precision three-dimensional positioning of the pupil center.

[0046] Step B40: Determine the three-dimensional coordinates of the pupil center in the world coordinate system as the pupil center position.

[0047] This location data, serving as another feature point characterizing the spatial orientation of the eyeball, will be used in conjunction with the aforementioned corneal center location for calculating the direction of gaze. By combining binocular stereo vision with image feature recognition, the three-dimensional coordinates of the pupil center can be stably obtained under dynamic wearing conditions, providing reliable data support for subsequent gaze estimation.

[0048] In one possible implementation, the step of determining the current gaze direction based on the corneal center position and the pupil center position includes: Step C10: Obtain the rotation matrix corresponding to the pre-calibrated Kappa angle; There is usually a fixed angle between the optical axis of the human eye (the line connecting the center of the cornea and the center of the pupil) and the visual axis (the actual direction of gaze), known as the Kappa angle. This angle is an individualized physiological parameter that can be calibrated through a specific process when a user first wears smart glasses. For example, the user is guided to gaze at multiple target points in known spatial locations, while simultaneously acquiring eye images and calculating the optical axis direction. By fitting the geometric relationship between the optical axis direction and the gaze direction, the angular components of the Kappa angle in three-dimensional space are calculated.

[0049] Based on this Kappa angle, a corresponding rotation matrix can be constructed. This matrix describes the rotational transformation relationship from the optical axis coordinate system to the visual axis coordinate system. It can be pre-stored in the storage module of the smart glasses and used to correct the optical axis direction to the actual viewing direction later.

[0050] Step C20: Determine the direction of the optical axis as the vector pointing from the center of the cornea to the center of the pupil; Based on the obtained corneal center position Cc(Xc, Yc, Zc) and pupil center position Pp(u, v, w), calculate the spatial vector pointing from the corneal center to the pupil center. Specifically, the optical axis direction... The calculation can be expressed by the formula: .

[0051] Since the center of the cornea is approximately the center of rotation of the eyeball, and the center of the pupil moves with the rotation of the eyeball, this vector directly represents the current optical axis orientation of the eyeball.

[0052] Step C30: Multiply the optical axis direction by the rotation matrix to obtain the current viewing direction.

[0053] The obtained optical axis direction vector is treated as a three-dimensional column vector and multiplied with the acquired rotation matrix. This rotation matrix essentially rotates the optical axis direction around a specific axis by the angle corresponding to the Kappa angle, thereby correcting the geometric optical axis to a direction consistent with the physiological visual axis.

[0054] The new vector obtained after rotation transformation is the user's actual gaze direction at the current moment, i.e., the current line of sight. This line of sight data will be used to compare with the preset baseline line of sight to determine whether it deviates from the optical center of the lens.

[0055] This embodiment introduces a Kappa angle correction mechanism, which can effectively eliminate the influence of individual eye physiological differences on line of sight estimation, and significantly improve the accuracy and personalization of line of sight direction calculation.

[0056] In one possible implementation, the gaze alert method further includes: Step D10: In response to the calibration command, the user is prompted to keep their head still and look at the target object at the preset spatial position through the optical center of the lens. The calibration command can be triggered by the user (e.g., via voice command or accompanying application) or automatically initiated after the smart glasses are first worn or after frame displacement is detected. During the calibration process, the user is first prompted via voice or display interface to keep their head posture stable to avoid introducing additional errors due to head rotation; at the same time, the user is asked to adjust their gaze direction to ensure that they are looking at a target object at a specified location in front of them through the optical center area of ​​the lens (such as a marker point at a specific distance directly in front of the smart glasses or a cursor on the screen).

[0057] This operation aims to simulate the scenario where the line of sight coincides with the optical center of the lens under ideal wearing conditions, laying the foundation for establishing a reference benchmark in the future.

[0058] Step D20: Obtain the first gaze direction calculated based on the first eye image and the second eye image when the user is looking at the target object; After the user completes the gaze action as prompted, the system synchronously triggers the infrared camera module and the second camera module to acquire eye images of the same eye, and performs real-time processing according to the method described in steps S20 to S30 above: first, the corneal center position is determined based on the first eye image, and the pupil center position is determined based on the first eye image and the second eye image; then, the optical axis direction is calculated by combining the positions of the corneal center and the pupil center, and then the optical axis is corrected to the visual axis direction by using a pre-calibrated Kappa angle rotation matrix, and finally the current gaze direction vector is obtained, which is recorded as the first gaze direction.

[0059] Furthermore, to ensure calibration accuracy, multiple frames of images can be continuously acquired and multiple gaze direction vectors can be calculated. The average value is taken as the final result to eliminate interference caused by momentary jitter or blinking.

[0060] Step D30: Determine the first line of sight as the preset baseline line of sight and store it.

[0061] This primary gaze direction is essentially the standard gaze direction when the user's line of sight passes precisely through the optical center of the lens. In three-dimensional space, it is represented by a unit vector pointing from the center of the eyeball to the target object. This vector can be stored as a reference benchmark during subsequent dynamic monitoring.

[0062] Subsequently, the current line of sight calculated in real time during daily wear must be compared with this preset baseline line of sight. The degree of deviation is quantified by calculating the spatial angle between the two, thereby triggering the corresponding deviation reminder.

[0063] The above calibration process can establish a personalized baseline gaze direction for each user, taking into full account individual facial features, wearing habits, and the actual position of the lens optical center, thus providing an accurate and reliable reference for subsequent deviation judgment.

[0064] For example, to aid in understanding the technical concept or principle of the gaze deviation warning method combined with the first embodiment described above, a specific embodiment is now listed. In this specific embodiment, refer to... Figure 2 As shown, the gaze deviation alert process includes: 1. Hardware and Communication Solutions Miniature infrared camera modules and miniature RGB camera modules are embedded on the left and right sides of the frame (near the lens edge or temple connection). The resolution and focal length of the dual cameras should be set so that they can fully capture the eye area (including eyelids, eyelashes, iris and pupil) at a typical eye distance (12-15mm). The glasses integrate a low-power microprocessor, a low-power Bluetooth module, and a certain storage unit. After the image captured by the camera is processed by the processor, the result is transmitted to the edge device through the BLE module; or the image is directly transmitted to the edge device through the BLE module for subsequent calculations.

[0065] 2. Keypoint extraction and matching 2.1 Infrared Corneal Reflection Point Localization Locating the corneal reflection point on the eye illuminated by the infrared LED in the infrared image can be achieved through gray-level thresholding and connected component analysis, or through end-to-end methods such as neural networks. The arrangement of each LED light spot after refraction through the cornea deviates from the original LED light arrangement. By combining the curvature change of the light group and the pre-measured corneal curvature Rc, the coordinates (Xc, Yc, Zc) of the corneal center Cc can be obtained.

[0066] 2.2 Pupil and Iris Boundary Extraction Locating the pupil location on corneal and RGB images can be achieved using an ellipse fitting algorithm based on OpenCV, or end-to-end using methods such as neural networks. Using the principle of binocular stereo vision, the projection points (u) of the same spatial point in the images of two cameras are compared. ir ,v ir ) and (u rgb ,v rgb By performing triangulation and combining the projection matrices of the two cameras in the world coordinate system, the three-dimensional coordinates (u, v, w) of the spatial point in the world coordinate system can be obtained using the least squares method or singular value decomposition. Using this method, the coordinates Pp of the pupil center can be obtained.

[0067] 3. Coordinate transformation to calculate gaze direction After obtaining the three-dimensional coordinates of the corneal center Cc and the pupil center Pp, the vector pointing from the corneal center to the pupil center is the optical axis of the eye:

[0068] Using the Kappa angle, the optical axis is converted into the visual axis:

[0069] 4. Pre-collection parameters When the glasses leave the factory, the relationships between the various components can be obtained based on the product's hardware schematic. In this specific example, the optical center of the lens is used as the origin. The positions of the infrared camera and the RGB camera are converted into three-dimensional coordinates in a coordinate system, and the relative pose, translation, and rotation matrices between the cameras are determined.

[0070] When a user first wears the device, their Kappa angle (the angle between the visual axis and the optical axis) and corneal curvature radius Rc are measured using professional optical instruments. The user remains still and observes an object in front of them through the optical center of the lens. Then, following the series of steps 2-3 above, the user's correct visual baseline is obtained.

[0071] 5. Comparison of gaze direction During the user's daily eye use, the user's eye use is continuously or intermittently observed as needed. During the observation, the user's gaze direction is calculated according to the series of processes in steps 2 and 3 above.

[0072] By comparing the user's gaze direction with the user's line of sight baseline, the degree to which the user deviates from the optical center of the lens can be obtained. Depending on specific business needs, different thresholds can be set to remind the user through vibration or APP pop-up.

[0073] It should be noted that the above examples are only used to help understand this embodiment and do not constitute a limitation on the line-of-sight deviation reminder process of this embodiment. Any simple modifications based on this technical concept are within the protection scope of this application.

[0074] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. In addition, the smart glasses further include an inertial measurement unit. After the step of determining the current gaze direction based on the corneal center position and the pupil center position, the method further includes: Step E10: Obtain the head posture data detected by the inertial measurement unit; An inertial measurement unit (IMU) typically includes a three-axis accelerometer and a three-axis gyroscope, which can output the angular velocity, acceleration, and attitude angles (such as pitch, roll, and yaw) of the smart glasses in three-dimensional space in real time. By fusing and filtering these raw data (such as using a Kalman filter), stable and reliable head attitude changes can be obtained, which can be used to determine the amplitude and direction of the wearer's head rotation.

[0075] Step E20: When the change in the head posture data is detected to be greater than a preset posture threshold, a temporary gaze monitoring window is triggered. During daily wear, the natural turning of the user's head is normal behavior, and the direction of the gaze will change accordingly. However, this does not necessarily mean that the relative position of the optical center of the lens and the line of sight has shifted. However, when the head posture changes drastically or continuously (such as turning the head sharply, looking down, or looking up), it may induce the frame to shift.

[0076] Therefore, to distinguish between brief head movements and frame slippage, a posture change threshold is set (e.g., angular velocity exceeding 30° / s or posture angle change exceeding 15°). Once the amount of head posture change exceeds this threshold within a continuous time period, it is considered that there may be a risk scenario that could lead to frame slippage, and a temporary gaze monitoring window is triggered. This window is usually set to a short time period (e.g., 3-5 seconds), during which intensive monitoring and analysis of the gaze direction are initiated.

[0077] Step E30: If, within the temporary line of sight monitoring window, the deviation between the line of sight calculated multiple times and the preset baseline line of sight shows an increasing trend, it is determined that the frame has physically slipped. Within the monitoring window, the aforementioned gaze direction calculation steps are executed at a higher frequency to obtain the sequence of angles between the current gaze direction and the preset baseline gaze direction. If this angle shows a monotonically increasing trend over time (i.e., the deviation becomes larger and larger), and this trend is consistent with the direction and amplitude of head posture changes, it indicates that the glasses frame has undergone physical displacement relative to the eyes, rather than simply a normal gaze change caused by head rotation. For example, when a user continuously lowers their head, if the glasses frame slides down due to gravity, the position of the gaze through the lenses will gradually shift upwards, causing the deviation from the baseline direction to continuously increase.

[0078] By analyzing this increasing trend, we can effectively identify persistent deviations in vision caused by physical factors such as loose or slipping frames, and eliminate temporary deviations caused by active head rotation.

[0079] Step E40 outputs a frame slippage alert message to remind the user to re-wear the glasses or perform baseline calibration.

[0080] Once it is determined that the frame has physically slipped, specific prompts can be output immediately through the prompting module (such as a vibration motor, speaker, or display unit), such as continuous short vibrations or a voice prompt "The frame has slipped, please adjust your wearing posture," informing the user that the current position of the frame has shifted and that the wearing posture needs to be readjusted to restore the alignment of the line of sight with the optical center.

[0081] Furthermore, users can be advised to re-perform the baseline calibration process to establish a new reference gaze direction, ensuring the accuracy of subsequent deviation detection. By introducing joint analysis of inertial measurement unit and gaze direction, it is possible to intelligently distinguish between head movement and frame slippage, achieving more accurate and targeted wearing status monitoring and reminders, further improving the comfort of using smart glasses and the effectiveness of visual health management.

[0082] Furthermore, embodiments of this application also propose a smart pair of glasses, referring to... Figure 3 As shown, the smart glasses include a first camera module, a second camera module, and a processor, wherein the first camera module is an infrared camera module; The infrared camera module is used to acquire images of the first eye. The second camera module is used to capture images of the second eye; The processor is used to perform the steps of the gaze deviation warning method described above.

[0083] Furthermore, in a preferred embodiment, the second camera module is an RGB camera module.

[0084] Furthermore, this application also proposes a gaze deviation warning system, referring to... Figure 4As shown, the gaze deviation warning system includes smart glasses and an edge device that are connected in communication. The smart glasses include a first camera module and a second camera module, wherein the first camera module is an infrared camera module. The smart glasses are used to capture eye images and send the eye images to the edge device; The edge device is used to perform the steps of the gaze deviation warning method described above.

[0085] refer to Figure 5 It illustrates a structural schematic diagram suitable for implementing the edge device in the embodiments of this application. The edge device in the embodiments of this application may also include, but is not limited to, mobile phones, headphones, etc. Figure 5 The edge device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0086] like Figure 5 As shown, the edge device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the edge device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the edge device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows edge devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0087] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0088] The edge device provided in this application, employing the gaze deviation warning method described in the above embodiments, can solve the technical problem of how to achieve timely detection and warning of gaze deviation. Compared with the prior art, the beneficial effects of the edge device provided in this application are the same as those of the gaze deviation warning method provided in the above embodiments, and other technical features of this edge device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0089] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0091] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the gaze deviation reminder method in the above embodiments.

[0092] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0093] The aforementioned computer-readable storage medium may be included in the edge device; or it may exist independently and not assembled into the edge device.

[0094] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an edge device, cause the edge device to perform the process steps of any embodiment of the aforementioned line-of-sight deviance alert method.

[0095] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0098] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described gaze deviation warning method, thereby solving the technical problem of how to achieve timely detection and warning of gaze deviation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the gaze deviation warning method provided in the above embodiments, and will not be repeated here.

[0099] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the gaze deviation reminder method as described above.

[0100] The specific implementation method of the computer program product in this application is basically the same as the embodiments of the above-described line-of-sight deviation reminder method, and will not be described again here.

[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0102] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause an edge device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0104] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for alerting to gaze deviation, characterized in that, Applied to smart glasses, the smart glasses include a first camera module and a second camera module, wherein the first camera module is an infrared camera module, and the gaze deviation warning method includes the following steps: Acquire a first eye image captured by the infrared camera module and a second eye image captured by the second camera module; The corneal center position is determined based on the first eye image, and the pupil center position is determined based on the first eye image and the second eye image. Based on the corneal center position and the pupil center position, the current gaze direction is determined. When the angle between the current gaze direction and the preset baseline gaze direction is greater than a preset threshold, a gaze deviation prompt message is output to remind the user that their gaze has deviated from the optical center of the lens.

2. The gaze deviation warning method as described in claim 1, characterized in that, The step of determining the corneal center position based on the first eye image includes: Obtain the pre-calibrated radius of curvature of the user's cornea; Identify at least one corneal reflection point based on the first eye image; Based on the corneal reflection point, the corneal radius of curvature, and the spatial position of at least one pre-stored infrared light source, calculate the three-dimensional coordinates of the corneal center in the world coordinate system; The location of the corneal center is determined by the three-dimensional coordinates of the corneal center in the world coordinate system.

3. The gaze deviation warning method as described in claim 1, characterized in that, The step of determining the pupil center position based on the first eye image and the second eye image includes: The pupil center is detected based on the first eye image, and the two-dimensional projection coordinates of the first pupil center are obtained. The pupil center is detected based on the second eye image, and the two-dimensional projection coordinates of the second pupil center are obtained. Based on the pre-calibrated relative pose parameters between the infrared camera module and the second camera module, triangulation is performed on the two-dimensional projection coordinates of the first pupil center and the two-dimensional projection coordinates of the second pupil center to obtain the three-dimensional coordinates of the pupil center in the world coordinate system. The three-dimensional coordinates of the pupil center in the world coordinate system are used to determine the position of the pupil center.

4. The gaze deviation warning method as described in claim 1, characterized in that, The step of determining the current gaze direction based on the corneal center position and the pupil center position includes: Obtain the rotation matrix corresponding to the pre-calibrated Kappa angle; The vector pointing from the center of the cornea to the center of the pupil is defined as the optical axis direction; Multiplying the optical axis direction by the rotation matrix yields the current line-of-sight direction.

5. The gaze deviation warning method as described in claim 1, characterized in that, The gaze alert method also includes: In response to the calibration command, the user is prompted to keep their head still and look at the target object at the preset spatial position through the optical center of the lens; The first gaze direction is calculated based on the first eye image and the second eye image when the user is looking at the target object; The first line of sight is determined as the preset baseline line of sight and stored.

6. The gaze deviation warning method as described in claim 1, characterized in that, The smart glasses also include an inertial measurement unit. After the step of determining the current gaze direction based on the corneal center position and the pupil center position, the method further includes: Acquire the head posture data detected by the inertial measurement unit; When the change in the head posture data is detected to be greater than a preset posture threshold, a temporary gaze monitoring window is triggered. Within the temporary line of sight monitoring window, if the deviation between the calculated line of sight direction and the preset baseline line of sight direction shows an increasing trend over multiple consecutive calculations, it is determined that the frame has physically slipped. Output a frame slippage alert message to remind the user to re-wear the glasses or perform baseline calibration.

7. A type of smart glasses, characterized in that, The smart glasses include a first camera module, a second camera module, and a processor, wherein the first camera module is an infrared camera module; The infrared camera module is used to acquire images of the first eye. The second camera module is used to capture images of the second eye; The processor is configured to perform the steps of the gaze deviation warning method as described in any one of claims 1 to 6.

8. The smart glasses as described in claim 7, characterized in that, The second camera module is an RGB camera module.

9. A gaze deviation warning system, characterized in that, The gaze deviation warning system includes smart glasses and an edge device connected by communication. The smart glasses include a first camera module and a second camera module, wherein the first camera module is an infrared camera module. The smart glasses are used to capture eye images and send the eye images to the edge device; The edge device is used to perform the steps of the gaze deviation warning method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a device control program, which, when executed by a processor, implements the steps of the gaze deviation warning method as described in any one of claims 1 to 6.

Images