Smart glass and its control method comprising discrete eyebox selection structure and software fine distortion compensation

The discrete eyebox selection and master-slave control in smart glasses stabilize the eyebox, addressing image shaking and distortion issues, enhancing user experience by maintaining image stability and reducing system load.

KR103045877B1Active Publication Date: 2026-11-10MEDITHINQ CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020260104159
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2026-06-09
Publication Date
2026-11-10
Estimated Expiration
2046-06-09

AI Technical Summary

Technical Problem

Conventional smart glasses with continuous eyebox tracking suffer from instability due to sensor errors and eye tremors, leading to image shaking and exacerbation of stereoscopic distortion and visual fatigue in 3D content playback.

Method used

A discrete eyebox selection structure with a deterministic master-slave control mechanism that stabilizes the eyebox by designating one eye as the master and the other as the slave, performing fine distortion compensation without AI, and adjusting the display module and optical unit based on predefined discrete intervals.

Benefits of technology

The solution ensures stable image presentation by minimizing dizziness and stereoscopic distortion, reducing system load, and maintaining image stability during gaze changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112026069565250-PAT00001_ABST
    Figure 112026069565250-PAT00001_ABST
Patent Text Reader

Abstract

According to the present invention, a smart glass can be provided comprising a display module that outputs images for the left eye and the right eye, respectively, a sensor unit that measures the user's eye position or gaze, and a control unit that controls an eyebox based on an input value of the sensor unit, wherein the control unit determines an eyebox state by mapping the input value of the sensor unit to one of a plurality of predefined discrete physical intervals.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to smart glasses technology, and more specifically, to smart glasses including a discrete eyebox selection structure capable of stably forming and maintaining left / right eyeboxes in response to changes in a user's gaze or eye position and performing fine distortion compensation through software, and a control method thereof. Background Technology

[0003] In augmented reality (AR) or mixed reality (MR) smart glasses, the eyebox refers to the range within which image information is visibly presented to the user.

[0004] Some conventional technologies have attempted to continuously steer or expand the eyebox based on eye-tracking information. However, this continuous tracking method has a problem in that the image shakes unstably due to minute sensor errors or slight eye tremors, causing dizziness.

[0005] In addition, when playing 3D content that utilizes binocular parallax, if the left / right eyebox alignment is misaligned in real time, stereoscopic distortion and visual fatigue are exacerbated. Therefore, a new control architecture is required that stably fixes the eyebox through deterministic mapping without AI or complex computations and performs limited software compensation. Prior art literature

[0007] (Patent Document 0001) U.S. Published Patent No. 2023-0168513 The problem to be solved

[0008] The present invention aims to provide smart glasses that ensure image stability by discretely determining the eyebox state in response to changes in the user's gaze.

[0009] In addition, the present invention aims to provide a deterministic master-slave structure that, when deviation (exceeding a threshold) of one eye is detected in 3D mode, designates the eye with relatively less distortion or deviation as the master and the opposite eye with more distortion as the slave, thereby controlling the system to perform adjustments within a limit range while dependent on the master. Furthermore, the present invention aims to provide a control method that does not impair hardware stability by limiting the scope of software compensation to eyebox boundary fixing and fine distortion compensation.

[0010] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0012] According to one embodiment of the present invention, a smart glass can be provided comprising: a display module that outputs images for the left eye and the right eye, respectively; a sensor unit that measures the position or gaze of a user's eye; and a control unit that controls an eyebox based on an input value of the sensor unit, wherein the control unit determines an eyebox state by mapping the input value of the sensor unit to one of a plurality of predefined discrete physical intervals.

[0013] In addition, when the control unit detects a situation in which one eye exceeds a set eyebox threshold in 3D content playback mode, it may designate the eye with relatively less distortion among the two eyes as the Master and the opposite eye with more distortion as the Slave, and may dependently control the image data or display module of the Slave side to perform adjustment within a limit range while dependent on the Master side. In addition, the control unit may read the hardware or display change state of the side designated as the Master, and perform dependent fine distortion compensation only on the image of the Slave side to correct the asymmetry of the optical path of both eyes caused by the change of the Master side.

[0014] In addition, the above-mentioned dependent fine distortion compensation can be performed in pixel units to align the relative disparity of the left and right images to a preset value of the 3D content, while within a range that does not change the center and boundary positions of the formed eyebox and the spatial perception position of the entire image.

[0015] In addition, when the control unit determines the compensation value of the slave side in response to a state change of the master side, it can determine the compensation value discretely by referring to a pre-stored dependent mapping table without a separate artificial intelligence (AI) operation.

[0016] In addition, it further includes a tilt angle control unit that adjusts the physical tilt of the display module or optical unit according to the determined eye box state, and when the tilt angle of the side designated as the master is adjusted, the control unit can digitally warp the image data of the slave side to offset stereoscopic distortion of both eyes.

[0017] In addition, the control unit can perform a fail-safe function by forcibly maintaining the state of the eyebox on the slave side or changing it only dependently even if a state change occurs on the master side, thereby ensuring that at least one eyebox is always active without the collapse of the 3D stereoscopic effect.

[0018] In addition, the sensor unit may include an infrared (IR) camera for acquiring the user's pupil position or corneal glint pattern.

[0019] In addition, the control unit software-wise moves the active display area of ​​the display module by referring to a calibration table within a determined eyebox state, and the movement of the active display area can be performed only within a range that does not involve the creation, movement, division, or boundary redefinition of the eyebox, without changing the center position and boundary position of the mechanically formed eyebox.

[0020] According to another embodiment of the present invention, a smart glasses control method may be provided, comprising the steps of: outputting images for the left eye and the right eye, respectively, through a display module; measuring the user's eye position or gaze through a sensor unit; and controlling an eyebox based on an input value of the sensor unit through a control unit, wherein the input value of the sensor unit is mapped to one of a plurality of predefined discrete physical intervals to determine the state of the eyebox. Effects of the invention

[0022] According to the present invention, through a discrete state determination structure, the eyebox remains stable without shaking even when the gaze moves, thereby achieving the effect of significantly improving the user experience.

[0023] In addition, according to the present invention, dizziness can be minimized by immediately correcting stereoscopic distortion of both eyes through master-slave dependent control in 3D mode.

[0024] In addition, according to the present invention, by using a deterministic mapping table instead of a learning-based model, the system load can be reduced and the reliability of the control can be maximized.

[0025] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0027] FIG. 1 is a block diagram showing the overall configuration of smart glasses according to one embodiment of the present invention. FIG. 2 is a block diagram showing the detailed configuration of a control unit according to one embodiment of the present invention. FIG. 3 is a flowchart illustrating a smart glass control method according to one embodiment of the present invention. Specific details for implementing the invention

[0028] Hereinafter, the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement it. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein.

[0029] The terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise.

[0030] As used herein, “comprises” and “comprising” do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0031] In addition, in describing the present invention, if it is determined that a detailed description of related known technology may obscure the essence of the present invention, such detailed description is omitted.

[0032] Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. The structure of the present invention and the resulting effects will be clearly understood through the following detailed description.

[0034] FIG. 1 is a block diagram showing the overall configuration of smart glasses according to one embodiment of the present invention.

[0035] First, referring to FIG. 1, a smart glass system according to one embodiment of the present invention may be configured to include a display module (110), an optical unit (120), a sensor unit (130), a control unit (140), and a tilt angle control unit (150).

[0036] The display module (110) includes a micro display and can independently output images for the left eye and the right eye, respectively, according to a signal from the control unit.

[0037] The optical unit (120) forms an optical path between the display module (110) and the user's eye and can support the output image being accurately projected into the eyebox.

[0038] The sensor unit (130) can measure the user's eye position or gaze in real time and transmit it to the control unit (140), and can detect whether the eyeball has deviated not only through pupil tracking using an infrared camera, but also through capacitive proximity sensing, visible light-based facial landmark recognition, wearable pressure sensing, or ultrasonic ToF method.

[0039] The control unit (140) can determine the optimal i-box state and control the entire system by mapping the continuous input value received from the sensor unit (130) to one of a plurality of predefined discrete physical intervals.

[0040] The tilt angle control unit (150) can mechanically adjust the physical warping or tilt angle of the display module (110) or optical unit (120) according to the determined eyebox state.

[0042] FIG. 2 is a block diagram showing the detailed configuration of a control unit according to one embodiment of the present invention.

[0043] FIG. 2 is a block diagram showing the sub-architecture of a control unit (140), and the control unit of the present invention may include a calibration mapping processing unit (141), a state calculation processing unit (142), a distortion compensation processing unit (143), and a dependent mapping table (144).

[0044] The calibration mapping processing unit (141) can convert sensor input values ​​into a specific eyebox state using a calibration table, a formula, or a discrete correspondence structure equivalent thereto according to a predefined discrete interval, and this can prevent image shaking by adopting a stepped discrete selection structure rather than continuous movement. The discrete eyebox selection structure used in this specification can be defined as a deterministic interval mapping structure that contrasts with a method of varying the image by tracking continuous physical measurement values ​​collected from a user sensor in real time. Here, the mapping result can be directly used as the final eyebox state without additional interpolation, probabilistic computation, or AI inference. Furthermore, specifically, this may refer to a control system that can fundamentally block minute image shaking caused by sensor noise, etc., by mapping the measurement value to one of a plurality of predefined non-overlapping closed intervals and deterministically selecting and fixing a specific eyebox state that is pre-optimized in correspondence with that interval.

[0045] Additionally, the state calculation processing unit (142) can independently calculate the left eye and right eye boxes as different state machines. In particular, when a situation is detected where one eye exceeds a set eye box threshold in 3D content playback mode, the eye with relatively less distortion among the two eyes is designated as the Master and the opposite eye with more distortion is designated as the Slave, thereby driving a deterministic master-slave structure that controls the Slave side dependently to perform adjustments within a limit range while dependent on the Master. Additionally, continuous changes in sensor measurements occurring within the same discrete interval may not be processed as a state transition.

[0046] Additionally, the distortion compensation processing unit (143) can perform software-based fine distortion compensation within a range that does not change the center and boundary positions of the mechanically formed eyebox. For example, the software may be configured so as not to redefine the eyebox boundary, move, create, or split the eyebox, or change the spatial position of the image perceived by the user.

[0047] Additionally, the distortion compensation processing unit (143) may move the active display area within the selected eyebox by referring to the calibration table, and may not perform the creation, movement, division, or boundary redefinition of the eyebox. Additionally, in order to correct binocular parallax distortion caused by tilt adjustment on the master side in 3D mode, dependent fine distortion compensation may be performed only on the slave side image through pixel-unit warping, and in this case, the center and boundary positions of the eyebox and the spatial position of the image perceived by the user are not changed.

[0048] Additionally, the dependent mapping table (144) stores a pre-configured discrete correspondence structure that enables immediate response without AI or complex probabilistic models, and can be used to read the change state of the master side to immediately determine the compensation value of the slave side. At this time, the change state of the master side includes the mechanical driving state of the hardware (tilt angle control unit) or the setting state of the display module.

[0050] FIG. 3 is a flowchart illustrating a smart glass control method according to one embodiment of the present invention.

[0051] The overall control process and detailed embodiment according to the present invention can be described with reference to FIG. 3. First, in the user sensor input reception step (S310), the user's pupil position, facial landmark coordinates, or wearing pressure data, etc., can be collected through the sensor unit (130).

[0052] In the subsequent left / right eye independent state machine driving step (S320), the control unit (140) independently calculates the state of both eyes, and in particular, when a situation is detected where one eye deviates beyond a set eyebox threshold in 3D content playback mode, the degree of eyebox deviation and distortion of both eyes is compared, and the side with relatively less distortion can be designated as the Master. At this time, a specific embodiment for determining the Master side with relatively less deviation among the two eyes may be as follows.

[0053] For example, as a first embodiment, a method based on geometric distance and spatial vector may be applied. Specifically, the 2D or 3D physical distance between the user's pupil center coordinates measured by the sensor unit (130) and the geometric center coordinates of the currently activated eyebox state may be calculated and compared (Method A), and the side with the relatively smaller calculated distance value (the side with less deviation) may be designated as the Master, and the side with the larger distance value may be determined as the Slave. Alternatively, the shortest clearance distance from the pupil position to the closed section boundary of the current eyebox may be calculated, and the side with relatively more clearance space (the side with less distortion and safety) may be determined as the Master, and the side with less clearance space may be determined as the Slave (Method B).

[0054] In a second embodiment, the discrete state mapping structure itself can be utilized. The interior of a single eyebox state can be predefined into multiple sub-discrete physical zones (sub-zones, e.g., center, middle, outer, etc.), and the zone index to which the sensor input value is mapped can be identified (Method C). In this case, by determining the left eye as the Master and the right eye as the Slave—since the index value is relatively smaller and closer to the center (less distortion)—deterministic decisions can be made solely through table mapping without complex distance calculations. Additionally, by measuring the physical speed or amount of change (Delta) at which the sensor measurement value reaches a discrete threshold for transitioning to the next eyebox state during eye movement (Method D), the side with the relatively smaller amount of change (the side with a stable state and less distortion) can be designated as the Master, and the side with the larger amount of change can be designated as the Slave.

[0055] In a third embodiment, a method based on optical and imaging distortion may be applied. It utilizes physical distortion caused by the characteristics of the optical lens as the position of the eye deviates from the center of the eyebox. Specifically, the eccentricity, which is the ratio of the major axis to the minor axis of the pupil in the image acquired by the sensor unit (130), is calculated (Method E). The side with relatively less elliptical distortion of the pupil, which is closer to a circle (less distortion), can be determined as the Master, and the side with greater eccentricity can be determined as the Slave. Additionally, the matching error between the geometric arrangement of the corneal glint pattern formed by the reflection of the infrared LED on the cornea and a predefined calibration template is calculated (Method F). The side with a relatively smaller error (less distortion) can be determined as the Master, and the side with a larger error can be determined as the Slave.

[0056] As a fourth embodiment, it may be based on 3D binocular parallax characteristics. Based on the convergence angle that must be formed according to the depth of an object when viewing a 3D image (Method G), the angle between the actual rotation angle of the eyeball and the center axis of the display are measured respectively, and the side with a smaller asymmetric deviation of the rotation angle relative to the mechanical optical axis (the side with less deviation) can be designated as the Master, and the side with a larger deviation can be designated as the Slave. Meanwhile, the smart glasses according to the present invention may include various embodiments that determine the Master by detecting that mechanical adjustment of the eyebox is required without performing direct eye tracking.

[0057] For example, as a first embodiment, a method based on a multi-zone capacitive proximity sensor can be applied. Micro-capacitive pads can be placed on the inner rim portion of the smart glasses frame, for example, at the top, bottom, left, and right (12 o'clock, 3 o'clock, 6 o'clock, and 9 o'clock directions) of the left and right lens frames, to discretely measure the distance to the facial skin. By utilizing the principle that capacitance increases as a dielectric material, such as skin, approaches the sensor, for example, if the sensor value at the top of the right eye frame decreases and the sensor value at the bottom increases, it can be deterministically determined that the right side of the glasses has sagged downward. At this time, if the difference (Delta) of the sensor values ​​is input into a predefined discrete interval mapping table, the mechanism adjustment unit can immediately correct the tilt angle of the corresponding display corner to realign the eyebox to the eyeball position.

[0058] As a second embodiment, a Passive Visible-light Facial Landmark Tracking method based on facial landmarks can be applied. Instead of tracking the pupil, this method can recognize skeletal and skin feature points of the face that do not change with gaze, such as eyebrows, corners of the eyes, the sides of the nose bridge, or the contours of the cheekbones, through a micro-camera inside the frame. Based on the reference pixel coordinates of feature points relative to the lens center stored during the calibration phase, the amount of coordinate displacement of the feature points that occurs when the glasses are tilted can be calculated. Through this, mechanical adjustment can be performed by referencing the tilt state corresponding to a specific displacement range in a mapping table, and this method enables stable control without infrared glint reflection or complex eye modeling.

[0059] As a third embodiment, an asymmetric wearable pressure or strain sensing method may be applied. Micro-pressure sensors or strain gauges can be embedded in the nose pad and temple parts that contact the face to detect a breakdown in the mechanical balance of the wearer. For example, if the pressure on the right side is measured to be abnormally low compared to the left nose pad, it is determined as physical evidence that the device has tilted to the right or moved away from the right eye; in this case, the left eye side, where the pressure is close to the normal range and the deviation is relatively minimal, can be designated as the Master, and the right eye side, where the pressure is lowered, can be designated as the Slave. Subsequently, while dependent on the Master, the tilt of the Slave side is hardware-driven to restore the eye box for adjustment within the limit range. At the same time, according to the Master-Slave control logic of the present invention, the Slave side performs software micro-distortion compensation by an amount dependent on the driving amount of the Master, thereby maintaining the balance between the two eyes.

[0060] As a fourth embodiment, an Ultrasonic Time-of-Flight Ranging (ToF)-based structural measurement method may be applied. Distance can be calculated by measuring the reflection time to facial bones, such as the brow bone or cheekbone, using ultrasonic transducers positioned at the top and bottom of the frame. Since ultrasound measures only the distance to physical bone structures without being affected by skin color, external lighting conditions, or eye movements, changes in the measured time-of-flight distance—whether shorter or longer than a reference value—can be input into a discrete mapping table to deterministically determine and correct whether the device is in close contact with or lifted from the cheek.

[0061] Subsequently, in the step of determining the i-box state through discrete mapping table reference (S330), the collected physical measurements can be input into the discrete mapping table of the calibration mapping processing unit (141) to map to an immediate discrete state.

[0062] In the step of performing mechanical alignment according to the determined state (S340), the tilt angle control unit (150) can be driven to move the optical module to a standard position in the corresponding discrete section, and at this time, the adjustment can be discretely controlled by referring to the pixel movement amount of the facial landmark or the distance change amount of the capacitive sensor.

[0063] In the subsequent software fine distortion compensation step (S350) within the fixed eyebox boundary, the disparity of the slave-side image can be corrected pixel by pixel to match the 3D content intent while maintaining the mechanically determined eyebox boundary.

[0064] Finally, in the final image output step (S360), the final image can be provided to the user after verifying a fail-safe condition in which at least one eye box remains in an active state. Here, the effective field of view condition refers to a state in which minimum resolution, brightness, and area are secured in a visually perceptible area based on the user's eye position. Additionally, during the fail-safe operation, the state of the eye box on the slave side is not changed independently, but is changed only within a range dependent on the state change of the master side, or the current state is forcibly maintained, thereby ensuring that at least one eye box among both eyes remains in an active state.

[0066] The embodiments disclosed in the specification of the present invention are merely examples and the present invention is not limited thereto. The scope of the present invention should be interpreted by the claims below, and all technologies within the equivalent scope should also be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0068] 110: Display module 120: Optical section 130: Sensor section 140: Control unit 141: Calibration Mapping Processor 142: State calculation processing unit 143: Distortion Compensation Processor 144: Dependency Mapping Table 150: Tilt angle control unit

Claims

Claim 1 Smart glasses comprising: a display module that outputs images for the left eye and the right eye respectively; a sensor unit that measures the user's eye position or gaze; and a control unit that controls the eyebox based on the input value of the sensor unit, wherein the control unit determines the eyebox state by mapping the input value of the sensor unit to one of a plurality of predefined discrete physical intervals, and wherein, when a situation is detected in which one eye exceeds a set eyebox threshold in content playback mode, the control unit designates the eye with relatively less distortion among the two eyes as the Master and the opposite eye with relatively more distortion as the Slave, and is configured to dependently control the image data or display module of the Slave side to perform adjustment within a limit range while dependent on the Master side, and wherein the control unit reads the hardware or display change state of the side designated as the Master and performs dependent fine distortion compensation only on the image of the Slave side to correct the asymmetry of the optical path of both eyes caused by the change of the Master side. Claim 2 delete Claim 3 delete Claim 4 In claim 1, the dependent fine distortion compensation is performed in pixel units to align the relative disparity of the left and right images to a preset value of the 3D content, while being performed within a range that does not change the center and boundary positions of the formed eyebox and the spatial perception position of the entire image. Claim 5 A smart glass according to claim 1, wherein the control unit determines the compensation value of the slave side discretely by referring to a pre-stored dependent mapping table without separate artificial intelligence (AI) computation when determining the compensation value of the slave side in response to a state change of the master side. Claim 6 The smart glasses according to claim 1 further include a tilt angle control unit that adjusts the physical tilt of the display module or optical unit according to a determined eyebox state, and when the tilt angle of the side designated as the master is adjusted, the control unit digitally warps the image data of the slave side to offset stereoscopic distortion of both eyes. Claim 7 Smart glasses according to claim 1, wherein the control unit performs a fail-safe function that ensures at least one eye box is always activated without the collapse of 3D stereoscopic effect by forcibly maintaining or dependently changing the state of the eye box on the slave side even if a state change occurs on the master side. Claim 8 Smart glasses according to claim 1, wherein the sensor unit includes an infrared (IR) camera for acquiring the user's pupil position or corneal glint pattern. Claim 9 In claim 1, the control unit software-wise moves the active display area of ​​the display module by referring to a calibration table within a determined eyebox state, and the movement of the active display area is performed only within a range that does not involve the creation, movement, division, or boundary redefinition of the eyebox without changing the center position and boundary position of the mechanically formed eyebox. Claim 10 A smart glasses control method comprising: a step of outputting images for the left eye and the right eye respectively through a display module; a step of measuring the user's eye position or gaze through a sensor unit; and a step of controlling an eyebox based on the input value of the sensor unit through a control unit, wherein the input value of the sensor unit is mapped to one of a plurality of predefined discrete physical intervals to determine the eyebox state; and further comprising, through the control unit, a step in which, if a situation is detected in which one eye exceeds a set eyebox threshold in content playback mode, the eye with relatively less distortion among the two eyes is designated as the Master and the opposite eye with relatively more distortion is designated as the Slave, and the image data or display module of the Slave side is dependently controlled to perform adjustment within a limit range while dependent on the Master side, and through the control unit, a change state of the hardware or display of the side designated as the Master is read, and dependent fine distortion compensation is performed only on the image of the Slave side to correct the asymmetry of the optical path of both eyes caused by the change of the Master side.

Citation Information

Patent Citations

  • Ar glasses apparatus that operates using eye tracking and glasses tilting and operating method thereof

    KR1020230157085A

  • Method of tracking gaze using lateral pupil images and wearable device using the same

    KR1020260000243A

  • Eye tracking data filtering

    KR1020230004305A

  • Multi-view eye tracking system with a holographic optical element combiner

    WO2023086280A1