Augmented reality apparatus and method for detecting user gaze

By combining a light guide plate, vision-correcting lens, and gaze-tracking sensor in an augmented reality device, the image of the user's eyes is adjusted to compensate for lens characteristic distortion, thereby improving the accuracy of gaze detection and enhancing the user's immersive experience.

CN117730271BActive Publication Date: 2026-11-10SAMSUNG ELECTRONICS CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202280052797.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-11-08
Filing Date
2022-08-05
Publication Date
2026-11-10
Estimated Expiration
2042-08-05

AI Technical Summary

Technical Problem

In augmented reality devices that include vision-correcting lenses, gaze detection accuracy is low, impacting the user's immersive experience.

Method used

It employs a combination of light guide plate, vision correction lens, light reflector and gaze tracking sensor to acquire the user's eye image through light emitter and light receiver, and uses processor to adjust the image and obtain gaze information to compensate for lens characteristic distortion.

Benefits of technology

It improves the accuracy of gaze detection and enhances the augmented reality experience for users wearing vision-correcting lenses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117730271B_ABST
    Figure CN117730271B_ABST
Patent Text Reader

Abstract

A method for detecting a user's gaze in an augmented reality device including a vision-correcting lens is provided. The method includes the following steps: obtaining lens characteristic information about a vision-correcting lens arranged to overlap with a light guide plate in the user's gaze direction; emitting light for gaze tracking via a light emitting unit toward a light reflecting unit, wherein the emitted light is reflected by the light reflecting unit and then guided to the user's eye; receiving the light reflected by the user's eye via a light receiving unit; obtaining an image of the user's eye based on the received light; compensating the user's eye image based on the lens characteristic information about the vision-correcting lens; and obtaining gaze information based on the compensated eye image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to an augmented reality (AR) device and method for detecting a user's gaze. More specifically, this disclosure relates to a gaze tracking sensor and method for detecting a user's gaze in an AR device including a vision-correcting lens. Background Technology

[0002] Augmented reality (AR) is a technology used to synthesize virtual objects or information in a real environment so that the virtual objects or information appear to exist in the real physical environment. Modern computing and display technologies have enabled the development of systems for AR experiences. In AR experiences, digitally reproduced images, or portions thereof, can be perceived as real, or can be presented to the user in a way that is recognized as a real image.

[0003] Furthermore, most AR devices take the form of head-mounted displays (HMDs), which are inconvenient to use when wearing glasses for vision correction. Correcting vision for someone wearing glasses for vision correction can be complex due to myopia, hyperopia, astigmatism, or any combination thereof. When users requiring vision correction use AR devices without wearing their glasses, they may be unable to recognize images to depict the real physical environment, and the level of immersion in AR may be reduced. Therefore, to provide a realistic AR experience also for users who do not wear separate glasses, AR devices incorporating vision-correcting lenses have been developed.

[0004] In various applications using AR or virtual reality (VR) devices, information about the user's gaze direction is often required. Gaze direction information can be used for a variety of operations, such as constructing user interfaces, optimizing the rendering of images provided to the user (e.g., foveated rendering), or determining the distance to the object the user is looking at. Gaze direction information can be generated by the user's eye position tracking sensor (hereinafter also referred to as a gaze tracking sensor (eye-tracking (ET) sensor)).

[0005] Therefore, there is a need to develop a gaze tracking method that can improve the accuracy of detecting user gaze in AR devices that include vision-correcting lenses. Summary of the Invention

[0006] Technical issues

[0007] One aspect of this disclosure is to provide a gaze tracking method and apparatus that can improve the accuracy of gaze detection in augmented reality (AR) devices including vision-correcting lenses.

[0008] Other aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practicing the embodiments presented.

[0009] Technical solution

[0010] According to one aspect of this disclosure, an augmented reality (AR) device is provided. The AR device includes: a light guide plate; a vision-correcting lens configured to overlap the light guide plate in the gaze direction of a user wearing the AR device; a support unit for securing the AR device to the face of a user wearing the AR device; a gaze-tracking sensor including a light receiver and a light emitter mounted on a portion of the support unit; the light reflector for reflecting light for gaze tracking; and at least one processor. The at least one processor may be configured to obtain lens characteristic information about the vision-correcting lens, control the light emitter to emit gaze-tracking light toward the light reflector, obtain an eye image of the user based on light received by the light receiver, adjust the eye image of the user based on the lens characteristic information, and obtain gaze information of the user from the adjusted eye image. Light emitted toward the light reflector may be reflected by the light reflector and directed to the user's eye, and the light received by the light receiver may include light reflected by the user's eye from the light directed to the user's eye. The light reflector may include a pattern. The at least one processor may also be configured to control the light emitter to emit light toward the light reflector for obtaining lens characteristic information about the vision-correcting lens, identify a distortion pattern based on the light received by the light receiver, and obtain lens characteristic information about the vision-correcting lens based on the distortion pattern.

[0011] According to another aspect of this disclosure, a method for detecting a user's gaze is provided, performed by an augmented reality (AR) device including a vision-correcting lens. The method includes: obtaining lens characteristic information about a vision-correcting lens configured to overlap with a light guide plate for displaying an image output from the AR device in the user's gaze direction; emitting gaze-tracking light towards a light reflector via a light emitter mounted at a portion of a support unit of the AR device, wherein the emitted light is reflected by the light reflector and directed to the eye of a user wearing the AR device; receiving the light reflected by the user's eye via a light receiver mounted at the support unit; obtaining an image of the user's eye based on the light received by the light receiver; adjusting the image of the user's eye based on the lens characteristic information about the vision-correcting lens; and obtaining gaze information of the user based on the adjusted eye image.

[0012] According to another aspect of this disclosure, a computer-readable recording medium is provided on which a program for performing at least one of the embodiments of the described methods in a computer can be recorded. Attached Figure Description

[0013] Figure 1 This is a schematic diagram illustrating a method for tracking a user's gaze in an augmented reality (AR) device including a vision-correcting lens, according to an embodiment of the present disclosure.

[0014] Figure 2 This is a flowchart illustrating a method for detecting a user's gaze using an AR device according to an embodiment of the present disclosure.

[0015] Figure 3 This is a block diagram of an AR device according to an embodiment of the present disclosure.

[0016] Figure 4 This is a diagram illustrating the operation of an AR device detecting a user's gaze according to embodiments of the present disclosure.

[0017] Figure 5 This is a diagram illustrating the operation of obtaining lens characteristic information about a vision-correcting lens according to embodiments of the present disclosure.

[0018] Figure 6 This is a diagram illustrating an operation for adjusting an eye image based on lens characteristic information about a vision-correcting lens, according to embodiments of the present disclosure.

[0019] Figure 7 This is a diagram illustrating the operation of obtaining eye gaze information about a user from an eye image adjusted based on lens characteristic information about a vision-correcting lens, according to embodiments of the present disclosure.

[0020] Figure 8 This is a diagram illustrating the operation of obtaining lens characteristic information about a vision-correcting lens according to embodiments of the present disclosure.

[0021] Figure 9 This is a diagram illustrating the operation of obtaining lens characteristic information about a vision-correcting lens according to embodiments of the present disclosure.

[0022] Figure 10 This is a diagram illustrating the control parameters of a vision-correcting lens according to embodiments of the present disclosure.

[0023] Figure 11 This is a diagram illustrating an AR device according to an embodiment of the present disclosure. Detailed Implementation

[0024] The following description, provided with reference to the accompanying drawings, is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these details should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Additionally, descriptions of well-known functions and structures may be omitted for clarity and brevity.

[0025] The terms and words used in the following description and claims are not limited to their literal meaning, but are used by the inventors only to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description providing various embodiments of this disclosure is for illustrative purposes only and is not intended to limit the purpose of this disclosure as defined by the appended claims and their equivalents.

[0026] It should be understood that the singular forms “a,” “an,” and “the” include plural references unless the context explicitly indicates otherwise. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0027] Throughout this disclosure, when something is referred to as "comprising" an element, it may also include one or more other elements unless otherwise specified. Furthermore, as used herein, terms such as "unit" and "module" may refer to a unit that performs at least one function or operation, and a unit may be implemented as hardware or software, or a combination of hardware and software.

[0028] Throughout this specification, when an element is referred to as being “connected” to another element, it may be “directly connected” to the other element, or it may be “electrically connected” to the other element using one or more intermediate elements between the element and the other element. Furthermore, when something is referred to as “comprising” a component, it may also include another component unless otherwise specified.

[0029] The expression “configured as (or set to)” as used herein may be replaced with “suitable,” “capable,” “designed to,” “adapted to,” “manufactured to,” or “able to,” etc., depending on the context. The expression “configured as (or set to)” does not necessarily mean “specifically designed for” at the hardware level. Rather, in some cases, the expression “system configured as…” may mean that the system, together with other devices or components, is “capable of…”. For example, “processor configured as (or set to) perform A, B, and C” can refer to a dedicated processor (e.g., an embedded processor) that performs the corresponding operations, or a general-purpose processor (e.g., a central processing unit or application processor) that performs the corresponding operations by executing one or more software programs stored in memory.

[0030] In this article, "augmented reality (AR) system" can refer to a system that displays virtual images in a real-world physical environment or displays real-world objects together with virtual images.

[0031] Here, "AR device" can be a device capable of displaying "AR", and can include AR glasses in the form of glasses worn on the face, head-mounted displays (HMDs), virtual reality headsets (VRHs), or AR helmets worn on the head, etc.

[0032] In this document, "real scene" can refer to a real-world scene as seen by a user through an AR device, and may include real-world objects. "Virtual image" can be an image generated by an optical engine, and may include both static and dynamic images. Virtual images can be viewed along with the real scene, and can be images representing information about real-world objects in the real scene, information about the operation of the AR device, control menus, etc.

[0033] In embodiments of this disclosure, an AR device may include: an optical engine for generating a virtual image comprising light generated from a light source; and a light guide (waveguide) that guides the virtual image generated by the optical engine to the user's eyes and is formed of a transparent material so that a real-world scene can also be viewed simultaneously. Because the AR device should also be able to view a real-world scene simultaneously, it will essentially require optical elements for altering the path of light with linearity in order to guide the light generated by the optical engine to the user's eyes via the light guide. In this case, the light path can be altered by using reflection such as a mirror, or by diffraction of diffraction elements such as diffractive optical elements (DOE) or holographic optical elements (HOE); however, this disclosure is not limited thereto.

[0034] This disclosure will be described in detail below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram illustrating a method for tracking a user's gaze in an AR device including a vision-correcting lens (prescription lens) according to an embodiment of the present disclosure.

[0036] The AR device according to embodiments of the present disclosure may be a device capable of rendering AR and may display images including virtual objects and real-world physical objects.

[0037] Reference Figure 1 AR devices can be glasses-type display devices and may include glasses-type bodies configured for wear by users.

[0038] The eyeglass-type body may include a support unit for securing an AR device to a user's face. The support unit may include temples 131 and a nose bridge 132. Temples 131 may be used to secure the AR device to the user's head at a side of the eyeglass-type body. The nose bridge 132 may be used to place the AR device on the user's nose and may include, for example, a bridge and a nose; however, this disclosure is not limited thereto.

[0039] In embodiments of this disclosure, the eyeglass-type body may further include a frame. A light guide plate 110, a vision-correcting lens 120, and a light reflector 150 may be disposed at the frame. For example, the frame may be formed around the outer peripheral surfaces of the light guide plate 110 and the vision-correcting lens 120. The light guide plate 110 may be configured to receive projected light in an input region and output at least a portion of the input light in an output region, and the vision-correcting lens 120 may be disposed between the light guide plate 110 and the user's eyes to correct vision when the user recognizes a scene displayed on the light guide plate 110 or recognizes a real scene through the light guide plate 110. In embodiments of this disclosure, the light guide plate 110 may include a left-eye light guide plate and a right-eye light guide plate, and the vision-correcting lens 120 may include a left-eye vision-correcting lens and a right-eye vision-correcting lens.

[0040] A left-eye vision correction lens 120, a left-eye light reflector 150, and a left-eye light guide plate 110 may be arranged at a position corresponding to the user's left eye, and a right-eye vision correction lens, a right-eye light reflector, and a right-eye light guide plate may be arranged at a position corresponding to the user's right eye. For example, the left-eye light reflector 150 may be arranged between the left-eye vision correction lens 120 and the left-eye light guide plate 110, and may be coated on a surface of the left-eye light guide plate 110 or the left-eye vision correction lens 120; however, this disclosure is not limited thereto. Similarly, a right-eye light reflector may be arranged between the right-eye vision correction lens and the right-eye light guide plate, and may be coated on a surface of the right-eye light guide plate or the right-eye vision correction lens; however, this disclosure is not limited thereto.

[0041] In embodiments of this disclosure, the optical engine of the projector used to project display light including an image may include a left-eye optical engine and a right-eye optical engine. The left-eye and right-eye optical engines may be located on two side surfaces of the AR device. Optionally, one optical engine may be included in the central portion around the nose bridge 132 of the AR device. Light output from the optical engine can be displayed via a light guide plate 110.

[0042] In embodiments of this disclosure, the gaze tracking sensor may include a light emitter 141 and a light receiver 143. The light emitter 141 and light receiver 143 may be disposed on the inner surface of the support unit at a location between the AR device's support unit and the user's eye. The light emitter 141 and light receiver 143 may be arranged facing a light reflector 150 on the support unit of the AR device. For example, the light emitter 141 and light receiver 143 may be disposed on the side surface of the AR device's temple 131 at a location spaced from the light guide plate 110 by approximately 2 mm to approximately 25 mm, to emit and receive infrared (IR) light without interference from the user's hair or other objects.

[0043] The AR device may include a vision correction lens 120 for correcting the vision of a user wearing the AR device. In embodiments of this disclosure, the vision correction lens 120 may have preset fixed refractive properties, or it may have variable refractive properties that can be changed as needed.

[0044] Figure 2 This is a flowchart illustrating a method for detecting a user's gaze using an AR device according to an embodiment of the present disclosure.

[0045] Reference Figure 2 In operation S210, the AR device obtains lens characteristic information about the vision-correcting lens. In operation S220, the AR device emits light for gaze tracking towards the light reflector via the light emitter. In operation S230, the AR device receives light reflected from the user's eye via the light receiver. In operation S240, the AR device obtains an image of the user's eye based on the received light. In operation S250, the AR device adjusts the image of the user's eye based on the lens characteristic information about the vision-correcting lens. In operation S260, the AR device obtains the user's gaze information based on the adjusted eye image.

[0046] Figure 3 This is a block diagram of an AR device according to an embodiment of the present disclosure.

[0047] Reference Figure 3 The AR device 300 according to embodiments of the present disclosure may include a display unit 310, a vision correction lens 320, a support unit 330, a gaze tracking sensor 340, a light reflector 350, a processor 360, and a storage unit 370.

[0048] Display unit 310 can display and output information processed by AR device 300. For example, display unit 310 can display information related to services provided based on images captured around AR device 300 and user interfaces captured around AR device 300.

[0049] According to embodiments of this disclosure, the display unit 310 can provide AR images. The display unit 310 according to embodiments of this disclosure may include a light guide plate (e.g., a waveguide) 311 and an optical engine 312. The light guide plate 311 may include a transparent material, through which a portion of the rear surface is visible when the user wears the AR device 300. The light guide plate 311 may include a flat plate of transparent material having a single-layer or multi-layer structure that reflects and transmits light. The light guide plate 311 may face the output surface of the optical engine 312 to receive light from the virtual image projected from the optical engine 312. Here, the transparent material may refer to a light-permeable material, whose transparency may not be 100%, and which may have a specific color. In embodiments of this disclosure, when the light guide plate 311 is formed of a transparent material, the light guide plate 311 may be referred to as a see-through display because the user can see not only virtual objects in the virtual image through the display unit 310 but also the external real scene. The display unit 310 can provide AR images by outputting virtual objects in the virtual image via the light guide plate 311. When the AR device 300 is an eyeglasses-type device, the display unit 310 may include a left display unit and a right display unit.

[0050] The vision-correcting lens 320 can correct the vision of a user wearing the AR device 300. In embodiments of this disclosure, the vision-correcting lens 320 can be disposed between the light guide plate 311 and the user's eye, and can correct the vision of the user recognizing real scenes and virtual images through the light guide plate 311. The vision-correcting lens 320 may have preset fixed refractive properties, or it may have variable refractive properties that can be changed as needed. In embodiments of this disclosure, the lens characteristics of the vision-correcting lens 320 can represent the characteristics of the lens that cause eye image distortion when the light reflector 350 is disposed between the vision-correcting lens 320 and the light guide plate 311, which occurs during the process of obtaining an eye image through the following path: light emitter 341 - vision-correcting lens 320 - light reflector 350 - vision-correcting lens 320 - user's eye - vision-correcting lens 320 - light reflector 350 - vision-correcting lens 320 - light receiver 343. For example, the lens characteristics of the vision correction lens 320 can represent the degree to which the vision correction lens 320 converges or diverges light. The lens characteristics of the vision correction lens 320 can be determined based on the refractive index of the lens and the curvature of the lens's refracting surface. For example, the lens characteristics of the vision correction lens 320 can increase with increasing refractive index or curvature of the refracting surface. The refractive index represents the degree to which light and waves are refracted when passing through an interface (refracting surface) between different media. For example, the refractive index can be determined based on the material of the vision correction lens 320. In embodiments of this disclosure, in addition to the refractive index or curvature of the refracting surface, information regarding the lens characteristics of the vision correction lens 320 may also include various numerical values ​​expressed as representing lens characteristics.

[0051] The light reflector 350 can reflect light emitted from the light emitter 341 described below. The light reflector 350 and the light guide plate 311 can be arranged facing the user's eyes, and the light reflector 350 and the light guide plate 311 can be attached to each other. For example, the light reflector 350 can be coated on at least a portion of the light guide plate 311. Furthermore, the light reflector 350 can be attached to or coated on other components included in the eyeglass-type AR device besides the light guide plate (e.g., a cover glass mounted to protect the vision correction lens 320 or the light guide plate 311). The light reflector 350 can be formed of a material capable of reflecting light (such as IR light emitted from the light emitter 341). The light reflector 350 can include, for example, silver, gold, copper, or a material comprising one or more such metals; however, this disclosure is not limited thereto. Therefore, the light emitted from the light emitter 341 can be reflected by the light reflector 350 and directed toward the user's eye, and the light reflected back from the user's eye can be reflected by the light reflector 350 and directed toward the light receiver 343.

[0052] A light reflector 350 may be coated on a light guide plate 311 to have a specific pattern. The pattern formed on the light reflector 350 may include, for example, dot patterns, line patterns, grid patterns, two-dimensional (2D) markings, etc.; however, this disclosure is not limited thereto. Furthermore, the pattern formed on the light reflector 350 may be formed, for example, on a portion of the light guide plate 311 that does not frequently guide the user's gaze. The pattern formed on the light reflector 350 may be formed, for example, on a portion of the light guide plate 311 that does not interfere with capturing or scanning the user's eye. For example, a specific pattern may refer to the pattern formed by the portion of the light reflector 350 that reflects light emitted from the light emitter 341 and the portion that does not reflect light emitted from the light emitter 341. Because light emitted toward the non-reflective portion of the light reflector 350 is not reflected by the light reflector 350, the light receiver 343 will not be able to receive light emitted toward the non-reflective portion. Therefore, the pattern formed by the reflective and non-reflective portions of the light reflector 350 can be detected from the light received by the light receiver 343.

[0053] In embodiments of this disclosure, when the light emitted from the light emitter 341 is IR light, the specific pattern may be formed by a material used to reflect IR light, and the material used to reflect IR light may include a material invisible to the user's eyes. Because the light or real scene of the real world observed by the user through the AR device 300 mainly includes visible light, the user can observe the light or real scene of the real world without being disturbed by the light reflector 350 in which the specific pattern is formed.

[0054] The gaze tracking sensor 340 may include a light emitter 341 and a light receiver 343. The gaze tracking sensor 340 may include a light emitter 341 that emits light for detecting a user's gaze and a light receiver 343 that receives light reflected from the user's eyes, and may detect data related to the gaze of a user wearing the AR device 300.

[0055] The light emitter 341 of the gaze tracking sensor 340 can emit light toward the light reflector 350, such that the light reflected by the light reflector 350 can be guided toward the user's eyes. The light emitter 341 can emit light toward the light reflector 350, the emitted light can be reflected by the light reflector 350, and the reflected light can be guided toward the user's eyes. The light emitter 341 can be positioned at a location where it can emit light toward the light reflector 350 in the AR device 300. The light emitter 341 can be located on a support unit 330 (such as temples 331 and nose bridge 332) that supports the AR device 300 on the user's face.

[0056] Furthermore, light reflected from the user's eyes can be reflected by the light reflector 350 and received by the light receiver 343 of the gaze tracking sensor 340. Light directed toward the user's eyes can be reflected from the user's eyes, and the light reflected from the user's eyes can be reflected by the light reflector 350, and the light receiver 343 can receive the light reflected by the light reflector 350. The light receiver 343 can be arranged in the AR device 300 at a location where it can receive the light reflected by the light reflector 350. The light receiver 343 can be located on a support unit 330 (such as temples 331 and nose bridge 332) that supports the AR device 300 on the user's face. Furthermore, for example, the nose bridge 332 can include the eyeglass bridge and the eyeglass nose. Furthermore, the eyeglass bridge and the eyeglass nose can be integrally constructed; however, this disclosure is not limited thereto.

[0057] For example, the light emitter 341 may be an infrared light-emitting diode (IR LED) that emits IR light, and the light receiver 343 may be an IR camera that captures the IR light. In this case, the IR camera can capture a user's eye using the IR light reflected by the light reflector 350. When the light emitter 341 is an IR LED and the light receiver 343 is an IR camera, the light emitter 341 may emit at least one type of IR light, either planar light or point light, toward the light reflector 350, and the light receiver 343 may receive reflected light identified when the IR light emitted from the light emitter 341 is sequentially reflected by the light reflector 350, the user's eye, and the light reflector 350. The planar light may be light emitted in a planar form, and the planar light emitted from the light emitter 341 may be directed toward at least a portion of the entire area of ​​the light reflector 350. At least a portion of the entire area of ​​the light reflector 350 may be configured such that the planar light reflected from at least a portion of the entire area of ​​the light reflector 350 may cover the user's eye.

[0058] When the AR device 300 is an eyeglasses-type device, the light emitter 341 and the light receiver 343 can be arranged on the temple 331 of the AR device 300. For example, the light emitter 341 and the light receiver 343 can be arranged on the inner surface of the temple 331 of the AR device 300 at a position between the temple 331 and the user's eye. For example, the light emitter 341 and the light receiver 343 can be arranged on the side surface of the temple 331 of the AR device 300 at a position spaced from the light guide plate 311 by about 2 mm to about 25 mm. The light emitter 341 and the light receiver 343 can be arranged on the temple 331 of the AR device 300 facing the light reflector 350.

[0059] Furthermore, for example, a light emitter 341 and a light receiver 343 may be arranged on the nose bridge 332 of the AR device 300. The light emitter 341 and the light receiver 343 may be arranged on the inner surface of the nose bridge 332 of the AR device 300 at a position between the nose bridge 332 and the user's eye. For example, the light emitter 341 and the light receiver 343 may be arranged on the side surface of the nose bridge 332 of the AR device 300 at a position spaced approximately 2 mm to approximately 25 mm from the light guide plate 311. The light emitter 341 and the light receiver 343 may be arranged on the nose bridge 332 of the AR device 300 as a facing light reflector 350.

[0060] The gaze tracking sensor 340 can provide the processor 360 with data related to the user's eye gaze, and the processor 360 can obtain the user's gaze information based on the data related to the user's eye gaze. The data related to the user's eye gaze can be data obtained by the gaze tracking sensor 340 and may include data about the wavelength, emission area, light characteristics, emission direction, etc., of the light emitted from the light emitter 341, as well as data representing the characteristics of the reflected light received from the light receiver 343. Furthermore, the user's gaze information can be information related to the user's gaze, which can be generated by analyzing the data related to the user's eye gaze, and may include, for example, information about the position of the user's pupil, the position of the pupil's center point, the position of the user's iris, the center of the user's eye, the position of the user's eye blink feature point, the user's gaze point, the user's gaze direction, etc.; however, this disclosure is not limited to this. The user's gaze direction can be, for example, a gaze direction from the center of the user's eye toward the gaze point. For example, the user's gaze direction can be represented by the values ​​of the vectors from the center of the user's left eye toward the gaze point and the values ​​of the vectors from the center of the user's right eye toward the gaze point; however, this disclosure is not limited to this. According to embodiments of this disclosure, gaze tracking sensor 340 can detect data related to the gaze of a user wearing AR device 300 at predetermined time intervals.

[0061] The storage unit 370 can store programs executed by the processor 360 described below, and can also store data input to or output from the AR device 300.

[0062] Storage unit 370 may include at least one of internal memory (not shown) or external memory (not shown). Internal memory may include at least one of, for example, volatile memory (e.g., dynamic random access memory (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.), non-volatile memory (e.g., one-time programmable read-only memory (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, etc.), hard disk drive (HDD), or solid-state drive (SSD). According to embodiments of this disclosure, processor 360 may load commands or data received from at least one of the non-volatile memory or other components into volatile memory and process them. Furthermore, processor 360 may store data received or generated from other components in non-volatile memory. External memory may include at least one of, for example, compact flash (CF), secure digital (SD), micro-secure digital (Micro-SD), mini-secure digital (Mini-SD), ultra-fast digital (xD), or Memory Stick.

[0063] The programs stored in storage unit 370 can be categorized into multiple modules according to their functions, and may include, for example, a lens characteristic acquisition module 371, an eye image acquisition module 372, a distortion compensation module 373, a feature point detection module 374, and a gaze detection module 375. For example, a memory (not shown) may be included in the gaze tracking sensor 340, and in this case, the eye image acquisition module 372 may be stored as firmware in the memory (not shown) included in the gaze tracking sensor 340.

[0064] The processor 360 can control the overall operation of the AR device 300. For example, the processor 360 can control the display unit 310, vision correction lens 320, gaze tracking sensor 340, storage unit 370, etc. by executing a program stored in the storage unit 370.

[0065] The processor 360 can determine the user's gaze point and gaze direction by executing the lens characteristic acquisition module 371, eye image acquisition module 372, distortion compensation module 373, feature point detection module 374 and gaze detection module 375 stored in the storage unit 370.

[0066] According to embodiments of the present disclosure, the AR device 300 may include multiple processors 360, and the lens feature acquisition module 371, eye image acquisition module 372, distortion compensation module 373, feature point detection module 374, and gaze detection module 375 may be executed by the multiple processors 360.

[0067] For example, some of the lens feature acquisition module 371, eye image acquisition module 372, distortion compensation module 373, feature point detection module 374, and gaze detection module 375 may be executed by a first processor (not shown), and the other modules lens feature acquisition module 371, eye image acquisition module 372, distortion compensation module 373, feature point detection module 374, and gaze detection module 375 may be executed by a second processor (not shown); however, this disclosure is not limited thereto.

[0068] For example, gaze tracking sensor 340 may include another processor (not shown) and a memory (not shown). The other processor (not shown) may execute an eye image acquisition module 372 stored in the memory (not shown), and processor 360 may execute a lens characteristic acquisition module 371, a distortion compensation module 373, a feature point detection module 374, and a gaze detection module 375 stored in the storage unit 370.

[0069] The processor 360 can obtain lens characteristic information about the vision correction lens 320 by executing the lens characteristic acquisition module 371 stored in the storage unit 370. In embodiments of this disclosure, the light reflector 350 may include a pattern. By executing the lens characteristic acquisition module 371 stored in the storage unit 370, the processor 360 can emit light for obtaining lens characteristic information towards the light reflector 350 via the light emitter 341, identify the pattern based on the light received by the light receiver 343, and obtain lens characteristic information about the vision correction lens 320 based on the identified pattern.

[0070] In embodiments of this disclosure, a pattern may be formed corresponding to a portion of the light guide plate 311. For example, a specific pattern may be formed on a portion of the light guide plate 311 to reflect light emitted from the light emitter 341 and guide the reflected light toward the light receiver 343. In embodiments of this disclosure, the pattern may be attached to or coated onto a cover glass that is mounted to protect the light guide plate 311 or a vision correction lens 320 for correcting a user's vision.

[0071] In embodiments of this disclosure, an indicator may be included for obtaining lens characteristic information about the vision correction lens 320, and the processor 360 may identify the indicator to obtain the lens characteristic information about the vision correction lens 320 by executing a lens characteristic acquisition module 371 stored in the storage unit 370. For example, the indicator may include a label, such as a barcode or quick response (QR) code, a test, etc.

[0072] In embodiments of this disclosure, the vision-correcting lens 320 may include a coupling unit for coupling to the support unit 330 of the AR device 300, and the processor 360 may obtain electromagnetic signals through the coupling unit and obtain lens characteristic information about the vision-correcting lens 320 based on the obtained electromagnetic signals by executing a lens characteristic acquisition module 371 stored in a storage unit 370. For example, the processor 360 may obtain lens characteristic information about the vision-correcting lens 320 by applying electromagnetic signals to the vision-correcting lens 320 via the coupling unit and obtaining corresponding electromagnetic return signals from the vision-correcting lens 320. In embodiments of this disclosure, the coupling unit may be arranged at the portion of the vision-correcting lens 320 that contacts the support unit 330. For example, the coupling unit may be arranged along the outer surface of the vision-correcting lens 320 where the vision-correcting lens 320 contacts the frame of the support unit 330.

[0073] In embodiments of this disclosure, the vision correction lens 320 may include a zoom lens. The zoom lens may be a lens with a variable focal length and may include, for example, a liquid crystal lens (LC lens), a liquid film lens, an electrowetting lens, or an Alvarez lens. For example, the zoom lens may be implemented as a flexible plastic film covering a transparent fluid. Furthermore, the refractive index (diopter) of the zoom lens may change when the fluid in the zoom lens moves according to an electrical signal applied to the zoom lens. In this case, the processor 360 may identify lens characteristic information about the vision correction lens 320 from the refractive parameters set in the zoom lens by executing the lens characteristic acquisition module 371 stored in the storage unit 370.

[0074] The processor 360 can obtain an eye image from reflected light received by the light receiver 343 by executing the eye image acquisition module 372 stored in the storage unit 370. The reflected light received by the light receiver 343 can represent the light identified when light reflected from the user's eye is reflected back from the light reflector 350. For example, when the light emitter 341 is an IR LED and its corresponding light receiver 343 is an IR camera, the obtained eye image can be an IR image.

[0075] The processor 360 can adjust the user's eye image based on lens characteristic information about the vision correction lens 320 by executing the distortion compensation module 373 stored in the storage unit 370.

[0076] The processor 360 can detect features related to the user's eye gaze by executing the feature point detection module 374 stored in the storage unit 370. For example, the processor 360 can detect the position of the center point of the user's eye and the position of the pupil feature points (pupil feature points) by executing the feature point detection module 374. The pupil feature points may include, for example, the eye's flash feature points. The eye's flash feature points may be portions of the detected eye region that have a brightness greater than or equal to a specific value. The position of the pupil's center point and the position of the feature points can be identified, for example, by coordinate values ​​representing the positions in the coordinate system of the light receiver 343. For example, the coordinate system of the light receiver 343 may be the coordinate system of an IR camera, and the coordinate values ​​in the coordinate system of the light receiver 343 may be 2D coordinate values.

[0077] Processor 360 can detect features related to eye fixation by analyzing the light received by light receiver 343. For example, when light receiver 343 is an IR camera, processor 360 can identify the position of the pupil's center point and the position of feature points in the image obtained by the IR camera. When the position of a feature point is detected, the position of the pupil's center point or the position of the feature point can have a value that is corrected by reflecting information about the lens characteristics of vision-correcting lens 320. (Refer to the description below.) Figure 8 A more detailed description is provided of the position of the center point and the position of the feature point of the pupil, which are corrected by reflecting information about the lens characteristics of the vision-correcting lens 320.

[0078] Furthermore, the processor 360 can obtain coordinate values ​​representing the positions of feature points of the pupil and the center point of the pupil by analyzing the light received by the light receiver 343. For example, when the light receiver 343 includes a camera, the processor 360 can obtain the coordinate values ​​of the center point of the pupil and the feature points in the camera's coordinate system. The camera's coordinate system can be used to represent the positions of the center point of the pupil and the feature points, and, for example, the coordinate values ​​in the camera's coordinate system corresponding to the pixels of the image obtained by the camera can be preset. Furthermore, the coordinate values ​​corresponding to the feature points of the eye can be identified based on the properties of the light received by the camera (e.g., brightness).

[0079] The processor 360 can obtain information about the user's gaze by executing the gaze detection module 375 stored in the storage unit 370. The processor 360 can calculate the position of the center of the user's eye by executing the gaze detection module 375. The center of the user's eye can represent the center of the area corresponding to the iris in the user's eyeball, i.e., the position of the pupil. The user's gaze direction and the position of the user's gaze point can be obtained from the position of the pupil. For example, the position on the light guide plate 311 corresponding to the center point of the user's pupil can be identified as the position of the user's gaze point, and the direction from the user's pupil to the gaze point can be identified as the user's gaze direction. The processor 360 can calculate the position of the user's pupil using lens characteristic information about the lens and feature points included in the image of the user's eye.

[0080] The processor 360 can calculate the position of the user's gaze point by executing the gaze detection module 375. To calculate the position of the user's gaze point, the processor 360 can pre-generate a mapping function for calculating the position of the gaze point from the characteristics of the user's eyes. The mapping function can be a function for calculating the position of the user's gaze point by considering the characteristics of the user's eyes and lens characteristic information about the vision correction lens 320, and can be generated through a pre-executed calibration process. For example, the position of the gaze point can have three-dimensional (3D) coordinate values ​​in a real-space coordinate system; however, this disclosure is not limited thereto. For example, when the center point of the light guide plate 311 is set as the origin and a two-dimensional coordinate system parallel to the plane of the light guide plate 311 is represented as the light guide plate coordinate system, the position of the gaze point can have coordinate values ​​in the light guide plate coordinate system that represent the position on the light guide plate 311 corresponding to the center point of the user's pupil; however, this disclosure is not limited thereto.

[0081] By executing the gaze detection module 375, the processor 360 can correct the gaze-related features obtained from the feature point detection module 374 based on lens characteristic information about the vision correction lens 320. Furthermore, the processor 360 can calculate the position of the user's gaze point by applying the gaze-related features corrected based on lens characteristics to a mapping function. Additionally, the user's gaze direction can be determined based on the positions of the user's gaze point and the center point of the eye calculated by the gaze detection module 375.

[0082] The processor 360 can perform a calibration operation before executing the operation of detecting the user's gaze. In the calibration operation, a mapping function can be calibrated based on lens characteristic information about the vision-correcting lens 320. In the calibration operation, the processor 360 can calibrate the mapping function based on lens characteristics and feature points of the eye to obtain the user's gaze point. For example, in the calibration operation, the degree of distortion of the eye image can be calculated based on the lens characteristic information about the vision-correcting lens 320, and compensation values ​​for the positions of the eye's center point and feature points can be calculated. The processor 360 can calibrate the mapping function so that a target gaze direction value can be output from the mapping function, which is input from the positions of the user's eye's feature points and the lens characteristic information about the vision-correcting lens 320.

[0083] Figure 4 This is a diagram illustrating the operation of an AR device detecting a user's gaze according to embodiments of the present disclosure. (Refer to...) Figure 4 The operation of detecting a user's gaze using an AR device including vision-correcting lenses can include a user calibration process and an eye-tracking process. During user calibration, lens characteristic information about the vision-correcting lenses can be identified, and the degree of distortion in the user's eye image obtained by the AR device including the vision-correcting lenses can be estimated. During eye tracking, the user's gaze while wearing the AR device can be tracked by reflecting the degree of distortion estimated during user calibration.

[0084] Figure 5 This is a diagram illustrating the operation of obtaining lens characteristic information about a vision-correcting lens according to embodiments of the present disclosure.

[0085] Reference Figure 5 The AR device may include a display unit 610, a vision correction lens 620, and a light reflector 650 disposed between the display unit 610 and the vision correction lens 620. The light reflector 650 may include a pattern RP.

[0086] In embodiments of this disclosure, the pattern RP can be formed to correspond to a portion of the display unit 610. For example, a specific pattern RP can be formed at a portion of the display unit 610 through which light emitted from the light emitter can be reflected and directed to the light receiver.

[0087] In embodiments of this disclosure, the gaze tracking sensor 640 emits light via a light emitter toward a light reflector 650 to obtain lens characteristics of the vision correction lens 620. The light emitted from the gaze tracking sensor 640 via the light emitter can be guided from the vision correction lens 620 to the light reflector 650. Thereafter, the light can be reflected by the light reflector 650, pass through the vision correction lens 620, and reach the light receiver of the gaze tracking sensor 640. The AR device can identify a distortion pattern DP formed by the distortion of a pattern RP included in the light reflector 650 based on the light received by the light receiver, and obtain information about the lens characteristics of the vision correction lens 620 based on the identified distortion pattern DP.

[0088] For example, information related to the actual pattern RP included in the light reflector 650 can be pre-stored in the AR device. The AR device can identify the distorted pattern DP from the light received by the light receiver of the gaze tracking sensor 640. Because the gaze tracking sensor 640 captures the actual pattern RP through the vision correction lens 620 and the light reflected from the actual pattern RP is refracted through the vision correction lens 620, the distorted pattern DP can be formed.

[0089] The AR device can obtain lens characteristic information about the vision correction lens 620 by comparing the actual pattern RP included in the light reflector 650 with the distorted pattern DP recognized by light received from the light receiver. For example, the AR device can obtain lens characteristic information about the vision correction lens 620 by comparing the positions of vertices on the real pattern RP with the positions of vertices on the distorted pattern DP and comparing the length, position, orientation, and curvature of the edges on the real pattern RP and the edges on the distorted pattern DP.

[0090] In embodiments of this disclosure, lens characteristic information regarding the vision correction lens 620 may include difference information about each vertex in the pattern or characteristic variation information about each edge.

[0091] The difference information for a specific vertex can represent information about the difference between the position of a specific vertex in the actual pattern RP and the position of the corresponding vertex in the distorted pattern DP. For example, when considering the vector from the position of a specific vertex in the real pattern RP to the position of the corresponding vertex in the distorted pattern DP, information about the magnitude and direction of that vector can be included.

[0092] Information regarding the characteristic variations of a specific edge can represent information about the difference between the characteristics of a specific edge in the actual pattern RP and the characteristics of the corresponding edge in the distorted pattern DP. For example, it can include information about the difference between the length of a specific edge in the actual pattern RP and the length of the corresponding edge in the distorted pattern DP, or information about the difference between the curvature of a specific edge in the actual pattern RP and the curvature of the corresponding edge in the distorted pattern DP.

[0093] In embodiments of this disclosure, when compared with an actual image, the degree of distortion of each point in the image captured by the vision-correcting lens 620 can be obtained from the difference information about each vertex. Furthermore, when compared with an actual image, the degree of distortion of each line in the image captured by the vision-correcting lens 620 can be obtained from the feature change information about each edge. Therefore, based on the lens characteristic information about the vision-correcting lens 620, the extent to which the image captured by the vision-correcting lens 620 has been distorted from the actual image can be detected, and the actual image can be calculated by compensating for the degree of distortion of the obtained (distorted) image.

[0094] The operation by which the AR device obtains information about the lens characteristics of the vision correction lens 620 is not limited to the operations described above, and the lens characteristic information can be obtained through various operations. Then, the obtained lens characteristic information about the vision correction lens 620 can be used to compensate for distortion of the eye image and to adjust the gaze tracking results accordingly.

[0095] Figure 6 This is a diagram illustrating an operation for adjusting an eye image based on lens characteristic information about a vision-correcting lens, according to embodiments of the present disclosure.

[0096] In an AR device including a vision-correcting lens, when a light reflector is positioned between the vision-correcting lens and a light guide plate, the user's eye image obtained from light traveling along the following path may include distortion caused by the vision-correcting lens: light emitter - vision-correcting lens - light reflector - vision-correcting lens - user's eye - vision-correcting lens - light reflector - vision-correcting lens - light receiver. When user gaze information is obtained from this distorted eye image, the obtained gaze information may differ from the actual gaze direction of the user. Therefore, to improve the accuracy of gaze tracking, it may be necessary to compensate for the degree of distortion in the user's eye image.

[0097] The distortion compensation function H(x, y) can be used to compensate for the degree of distortion in an image. Information about the lens characteristics of the vision-correcting lens can be used to generate the distortion compensation function H(x, y).

[0098] In embodiments of this disclosure, when a user wears an AR device, the grid pattern image i can be considered as being located at the position where the user's eyes are to be positioned. In this case, a distorted grid pattern image d can be obtained from light traveling through the following path: light emitter - vision correction lens - light reflector - vision correction lens - grid pattern image - vision correction lens - light reflector - vision correction lens - light receiver.

[0099] Furthermore, in embodiments of this disclosure, the degree of distortion in a user's eye image can be calculated using lens characteristic information about the vision-correcting lens.

[0100] Reference Figure 6 When x represents the index of a specific vertex on the mesh pattern along the x-axis and y represents the index of a specific vertex on the mesh pattern along the y-axis, i(x, y) represents the coordinate value of the vertex (x, y) in the actual mesh pattern image i from the center point (or the preset origin), and d(x, y) represents the coordinate value of the vertex (x, y) in the distorted mesh pattern image d from the center point (or the preset origin). Before compensating for the degree of image distortion, for at least one vertex (x, y), it can be seen that the coordinate values ​​i(x, y) in the actual mesh pattern image and d(x, y) in the distorted mesh pattern image are different from each other.

[0101] The operation of compensating for the distortion of an image can be represented as the operation of obtaining an adjusted mesh pattern image D from a distorted mesh pattern image d, such that for all vertices (x, y), the difference (Δd(x,y)=i(x,y)-D(x,y)) between the coordinate values ​​D(x,y) on the adjusted mesh pattern image D and the coordinate values ​​i(x,y) on the actual mesh pattern image can be less than a preset threshold δ.

[0102] Here, the coordinate values ​​D(x,y) on the adjusted grid pattern image D can be obtained by multiplying d(x,y) by the distortion compensation function H(x,y). That is, the following equation 1 can be satisfied.

[0103] [Equation 1] D(x, y)=H(x, y)*d(x, y)

[0104] In other words, according to embodiments of this disclosure, by using lens characteristic information about the vision-correcting lens, it is possible to determine how much distortion is included when an actual eye image is obtained through a gaze tracking sensor, and a distortion compensation function H(x, y) for adjusting the obtained eye image can be obtained.

[0105] You can refer to the above. Figure 5The described method obtains lens characteristic information about a vision-correcting lens. For example, the lens characteristic information may include information about the direction and distance by which each point in the image captured by the vision-correcting lens has moved compared to the actual image. Furthermore, the lens characteristic information may include information about the degree of distortion to which the length or curvature of each line in the image captured by the vision-correcting lens has been distorted compared to the actual image.

[0106] Therefore, based on information about the lens characteristics of the vision-correcting lens, it is possible to detect the extent to which the image captured by the vision-correcting lens has been distorted from the actual image, and to obtain a distortion compensation function H(x, y) for compensating for the degree of distortion of the acquired (distorted) image.

[0107] Thus, according to embodiments of this disclosure, the accuracy of gaze tracking results can be improved because the distortion of the eye image is compensated by using the obtained distortion compensation function H(x,y), and the user's gaze is detected from the adjusted eye image.

[0108] Figure 7 This is a diagram illustrating the operation of obtaining a user's gaze information from an eye image adjusted based on lens characteristic information about a vision-correcting lens, according to embodiments of the present disclosure.

[0109] Reference Figure 7 The AR device may include a display unit 810, a vision correction lens 820, and a light reflector 850 disposed between the display unit 810 and the vision correction lens 820.

[0110] In embodiments of this disclosure, the gaze tracking sensor 840 emits light via a light emitter toward a light reflector 850 for tracking the gaze of a user wearing an AR device. The light emitted from the gaze tracking sensor 840 via the light emitter is guided to the light reflector 850 via a vision-correcting lens 820. Subsequently, the light can be sequentially reflected again by the light reflector 850, the user's eye, and the light reflector 850, and arrive at the light receiver of the gaze tracking sensor 840. The AR device can obtain an image of the user's eye based on the light received by the light receiver.

[0111] As shown above (refer to the reference) Figure 6In the AR device including the vision-correcting lens 820, when the light reflector 850 is arranged between the vision-correcting lens 820 and the display unit 810, the user's eye image obtained from light traveling through the following path may include distortion caused by the vision-correcting lens 820: light emitter - vision-correcting lens 820 - light reflector 850 - vision-correcting lens 820 - user's eye - vision-correcting lens 820 - light reflector 850 - vision-correcting lens 820 - light receiver. Therefore, to improve the accuracy of gaze tracking, it may be necessary to compensate for the degree of distortion in the user's eye image.

[0112] Reference Figure 7 The AR device can obtain an eye image d(x, y) via a light receiver. The obtained eye image d(x, y) may include distortion caused by the vision correction lens 820. When the user's gaze direction (i.e., the user's pupil position information) is obtained by directly extracting eye feature points from the eye image d(x, y), the obtained pupil position may not correspond to the user's actual gaze direction due to image distortion.

[0113] Therefore, the AR device according to the embodiments of this disclosure can be used via the above reference. Figure 6 The described method uses a distortion compensation function H(x, y) to adjust the eye image d(x, y), and detects the user's pupil position information and gaze information from the adjusted eye image. In other words, according to embodiments of this disclosure, the accuracy of gaze tracking results can be improved because the user's gaze is detected from the eye image adjusted using the distortion compensation function H(x, y), which is obtained using lens characteristic information about the vision correction lens 820.

[0114] Figure 8 This is a diagram illustrating the operation of obtaining lens characteristic information about a vision-correcting lens according to embodiments of the present disclosure, and Figure 9 This is a diagram illustrating the operation of obtaining lens characteristic information about a vision-correcting lens according to embodiments of the present disclosure.

[0115] Reference Figure 8 An indicator 922 for obtaining lens characteristic information about the vision-correcting lens 920 can be displayed on the vision-correcting lens 920. The processor of the AR device can obtain lens characteristic information about the vision-correcting lens 920 by recognizing the indicator 922. In embodiments of this disclosure, the processor can control a light emitter included in the gaze-tracking sensor 940 to emit light toward the indicator 922, and recognize the indicator 922 based on the light received by a light receiver included in the gaze-tracking sensor 940.

[0116] Reference Figure 9 In parts (a) and (b), the vision correction lens 1020 may include a coupling unit 1022 for coupling to a support unit of an AR device. In this case, the processor of the AR device can obtain lens characteristic information about the vision correction lens 1020 by applying an electromagnetic signal to the vision correction lens 1020 via the coupling unit 1022 and obtaining a corresponding electromagnetic return signal from the vision correction lens 1020.

[0117] Figure 10 This is a diagram illustrating the control parameters of a vision-correcting lens according to embodiments of the present disclosure.

[0118] Reference Figure 10 In parts (a) and (b) of this disclosure, the AR device includes a display unit 1110, a light reflector 1150, and a vision correction lens, and the vision correction lens may include a zoom lens 1123. Figure 10 AR devices can correspond to Figure 3 The AR device can control the zoom lens 1123 by using the refractive index and the position of the center point (aperture center) as control parameters. In this case, the processor can set the control parameters for controlling the zoom lens 1123 and obtain lens characteristic information about the vision correction lens based on the set control parameters.

[0119] Figure 11 This is a diagram illustrating an AR device according to an embodiment of the present disclosure.

[0120] Reference Figure 11 An AR device 1200 for detecting a user's gaze is shown. The AR device 1200 may include AR glasses shaped like glasses worn on the user's face, a head-mounted display (HMD), a virtual reality headset (VRH), or an AR helmet worn on the head, etc. In the case of the AR device 1200, by positioning the display in front of the user's eyes, realistic scenes and lifelike virtual images can be provided as the screen moves according to the user's movements.

[0121] In embodiments of this disclosure, a user-wearable AR device 1200 that can display visual extended reality content. The AR device 1200 may include an audio module that can provide audio extended reality content to the user. In embodiments of this disclosure, the AR device 1200 may include one or more cameras that can capture images and video of its surroundings. The AR device 1200 may include a gaze-tracking (eye-tracking) system to determine the user's convergence / disconvergence distance. In embodiments of this disclosure, the AR device 1200 may include a lightweight head-mounted display (HMD) (e.g., goggles, glasses, face mask, etc.). In embodiments of this disclosure, the AR device 1200 may include a non-HMD device, such as laser-projected glasses (e.g., glasses that can project a low-power laser onto the user's retina to project and display images or depth content to the user).

[0122] In embodiments of this disclosure, AR device 1200 can provide AR services whose output appears to be overlaid on at least one virtual object within an area determined to be the user's field of view (FOV). For example, the area determined to be the user's FOV may be an area identified by a user wearing AR device 1200 via AR device 1200, or it may be at least a portion or all of the area of ​​a display including AR device 1200. In embodiments of this disclosure, AR device 1200 may include a plurality of transparent components (e.g., displays 1220 and 1230) respectively corresponding to the user's eyes.

[0123] In embodiments of this disclosure, the AR device 1200 may include a display module 1214, a camera, an audio output unit, and support units 1221 and 1222.

[0124] The camera can acquire images corresponding to the user's field of view (FOV) or measure the distance to objects. In embodiments of this disclosure, the camera can be used for head tracking and spatial recognition. Furthermore, the camera can detect user movement.

[0125] In embodiments of this disclosure, in addition to camera 1213 for detecting an image corresponding to the user's field of view (i.e., object movement) or for spatial recognition, the camera may also include an "eye-tracking (ET) camera 1212." In embodiments of this disclosure, ET camera 1212 may be used to detect and track the user's pupils. ET camera 1212 may be used to adjust the center of the virtual image projected onto AR device 1200 to be positioned according to the gaze direction of the user's pupils wearing AR device 1200. ET camera 1212 may correspond to the above... Figure 3The light receiver 343. For example, a global shutter (GS) camera can be used in the ET camera 1212 to detect the pupil and track rapid pupil movement without delay. The ET camera 1212 may include a left-eye camera 1212-1 and a right-eye camera 1212-2, respectively.

[0126] In embodiments of this disclosure, the display module 1214 may include a first display 1220 and a second display 1230. The display module 1214 may correspond to the above-described... Figure 3 The display unit 310. The virtual objects output by the display module 1214 may include information related to the application running in the AR device 1200 or information related to external objects located in real space corresponding to the area of ​​the field of view (FOV) determined to be the user. For example, the AR device 1200 may identify external objects included in at least a portion of the area corresponding to the area of ​​the FOV determined to be the user from image information related to real space obtained by the camera 1213. The AR device 1200 may output virtual objects related to the external objects identified in at least a portion of the FOV determined to be the user within the display area of ​​the AR device 1200. External objects may include objects existing in real space.

[0127] In embodiments of this disclosure, displays 1220 and 1230 may include a focusing lens, a vision-correcting lens, or a light guide (waveguide) in a transparent member. The light guide may correspond to the above-described... Figure 3 The light guide plate 311. For example, the transparent component may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be completely transparent or translucent. In embodiments of this disclosure, the transparent component may include a first transparent component (e.g., a second display 1230) facing the right eye of the user wearing the AR device 1200 and a second transparent component (e.g., a first display 1220) facing the user's left eye. When the display is transparent, it may be positioned facing the user's eyes to display a screen.

[0128] A light guide plate can transmit light generated by a light source of a display to a user's eyes. For example, the light guide plate may be located at least partially on a portion of a transparent component (e.g., displays 1220 and 1230). According to embodiments of this disclosure, light emitted from the display can enter through one end of the light guide plate, and the input light can be transmitted to the user's eyes via total internal reflection in the light guide plate. The light guide plate may be formed of a transparent material such as glass, plastic, or polymer, and may include nanopatterns, such as polygonal or curved grating structures, formed on the inner or outer surface. In embodiments of this disclosure, the input light can be propagated or reflected in the light guide plate via the nanopatterns and provided to the user's eyes. In embodiments of this disclosure, the light guide plate may include at least one of the following: at least one diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a mirror). In embodiments of this disclosure, the light guide plate can guide display light emitted from a light source to the user's eyes by using at least one diffractive element or reflective element.

[0129] In embodiments of this disclosure, displays 1220 and 1230 may include a display panel or a lens (e.g., glass). For example, the display panel may include a transparent material, such as glass or plastic. In embodiments of this disclosure, the display may be configured as a transparent element, and a user may perceive the real space behind the display through the display. The display may display virtual objects in at least a portion of the transparent element, such that the user sees the virtual objects as if they were added to at least a portion of the real space.

[0130] In embodiments of this disclosure, support units 1221 and 1222 may include printed circuit boards (PCBs) 1231-1 and 1231-2 for transmitting electrical signals to corresponding components of the AR device 1200, speakers 1232-1 and 1232-2 for outputting audio signals, or batteries 1233-1 and 1233-2 for power supply. Support units 1221 and 1222 may correspond to the above-described... Figure 3 Support unit 330. For example, in the glasses-type AR device 1200, support units 1221 and 1222 may be arranged at the temples of the glasses. Support units 1221 and 1222 may include hinge units 1240-1 and 1240-2 for coupling to the body of the AR device 1200. Speakers 1232-1 and 1232-2 may include a first speaker 1232-1 for transmitting audio signals to the user's left ear and a second speaker 1232-2 for transmitting audio signals to the user's right ear.

[0131] Reference Figure 11The AR device 1200 may include a microphone 1241 for receiving the user's voice and ambient sounds. Furthermore, the AR device 1200 may include at least one light-emitting device (illumination LED) 1242 to improve the accuracy of at least one camera (e.g., ET camera 1212, outward-facing camera 1213, or recognition cameras 1211-1 and 1211-2). For example, when the ET camera 1212 is used to photograph the user's pupil, the light-emitting device 1242 may be used as an auxiliary mechanism to improve accuracy, and the light-emitting device 1242 may use an infrared wavelength IR LED instead of a visible light wavelength. In embodiments of this disclosure, the light-emitting device 1242 may correspond to the above-described... Figure 3 The light emitter 341. For example, when the user's gestures are captured by the recognition cameras 1211-1 and 1211-2, the light-emitting device 1242 can be used as an auxiliary mechanism when it is difficult to detect the object due to the dark environment.

[0132] According to embodiments of this disclosure, the display module 1214 may include a first light guide plate (e.g., a first display 1220) corresponding to the left eye and a second light guide plate (e.g., a second display 1230) corresponding to the right eye, and can provide visual information to the user through the first light guide plate (e.g., the first display 1220) and the second light guide plate (e.g., the second display 1230). According to embodiments of this disclosure, the display module 1214 may include a display panel and a lens (e.g., a glass lens, a plastic lens, or an LC lens). The display panel may include a transparent material, such as glass or plastic.

[0133] According to embodiments of this disclosure, the display module 1214 can be configured as a transparent element, and a user can identify the real space behind the display module 1214 and in front of the user through the display module 1214. The display module 1214 can display virtual objects in at least a portion of the transparent element, so that the user can see the virtual objects as if they were added to at least a portion of the real space.

[0134] In embodiments of this disclosure, AR device 1200 can identify external objects included in at least a portion of an area corresponding to a field of view (FOV) determined to be the user's space, from image information related to real space obtained by outward-facing camera 1213. AR device 1200 can output (or display) virtual objects associated with the external objects identified in at least a portion of the FOV determined to be the user's space within its display area. External objects may include objects existing in real space. According to various embodiments of this disclosure, the display area for displaying virtual objects may include a portion of display module 1214 (e.g., at least a portion of display panel). According to embodiments of this disclosure, the display area may be an area corresponding to at least a portion of a first light guide plate (e.g., first display 1220) and a second light guide plate (e.g., second display 1230).

[0135] According to embodiments of this disclosure, AR device 1200 can measure the distance to a physical object located in front of AR device 1200 by using an outward-facing camera 1213. The outward-facing camera 1213 may include a high-resolution (HR) camera, such as a photo-video (PV) camera.

[0136] The AR device 1200 according to embodiments of the present disclosure is not limited to the above configuration and may include various numbers of various components at various locations.

[0137] Various embodiments of this disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed of computer-readable program code and may be included in a computer-readable medium. In this disclosure, the terms "application" and "program" may refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code comprising source code, object code, and executable code. "Computer-readable medium" may include various types of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disc (CD), digital video disc (DVD), or various types of memory.

[0138] Furthermore, machine-readable storage media can be provided in the form of non-transitory storage media. Here, "non-transitory storage media" can be a tangible device and may exclude wired, wireless, optical, or other communication links used for transmitting temporary electrical signals or other signals. Furthermore, "non-transitory storage media" may not distinguish between cases where data is semi-permanently stored in the storage medium and cases where data is temporarily stored in the storage medium. For example, "non-transitory storage media" may include buffers for temporarily storing data. Computer-readable media can be any available medium accessible to a computer and may include volatile or non-volatile media as well as removable or non-removable media. Computer-readable media may include media that can permanently store data and media that can store data and rewrite data later (such as rewritable optical discs or erasable memory devices).

[0139] According to embodiments of this disclosure, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an app store, or directly between two user devices (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily stored or temporarily generated in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

[0140] The foregoing describes embodiments of this disclosure, and those skilled in the art will readily understand that various modifications may be made therein without substantially departing from the spirit or characteristics of this disclosure. Therefore, it will be understood that the above embodiments should be considered in a descriptive sense only and not for limiting purposes. For example, each component described as a single type may also be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a composite manner.

[0141] Although this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. An augmented reality (AR) device, comprising: Light guide plate; The vision-correcting lens overlaps with the light guide plate in the direction of the user's gaze when wearing the AR device; A support is configured to secure the AR device to the face of the user wearing the AR device; A gaze tracking sensor, including a light receiver and a light emitter disposed on a portion of the support; A light reflector is configured to reflect light used for gaze tracking; as well as At least one processor is configured as follows: Obtain information about the lens characteristics of the vision-correcting lens. By controlling the light emitter to emit light toward the light reflector The user's eye image is obtained based on the light received by the light receiver. Adjusting the user's eye image based on the lens characteristic information, and Gazing information about the user is obtained from the adjusted eye image. Light emitted toward the light reflector is reflected by the light reflector and directed to the user's eyes. The light received by the light receiver includes light obtained due to the reflection of light directed to the user's eye by the user's eye. The light reflector includes a pattern, and The at least one processor is further configured to: By controlling the light emitter, light is emitted toward the light reflector to obtain information about the lens characteristics of the vision-correcting lens. Distortion patterns are identified based on the light received by the light receiver, and Information about the lens characteristics of the vision-correcting lens is obtained based on the distortion pattern.

2. The AR device as claimed in claim 1, wherein, The at least one processor is further configured to: Information about the lens characteristics of the vision-correcting lens is obtained by comparing the pattern included in the light reflector with a distortion pattern identified from the light reflected by the light reflector and received by the light receiver.

3. The AR device as claimed in claim 1, wherein, An indicator for obtaining lens characteristic information about the vision-correcting lens is displayed on the vision-correcting lens, and The at least one processor is further configured to: Information about the lens characteristics of the vision-correcting lens is obtained by recognizing the indicator.

4. The AR device as claimed in claim 3, wherein, The at least one processor is further configured to: By controlling the light emitter to emit light toward the indicator, and The indicator is identified based on the light received by the light receiver.

5. The AR device according to claim 1, wherein, The vision-correcting lens includes: a coupler configured to couple with the support of the AR device, and The at least one processor is further configured to: Electromagnetic signals are applied to the vision-correcting lens via the coupler, and By obtaining the corresponding electromagnetic return signal in response to the applied electromagnetic signal, lens characteristic information about the vision-correcting lens is obtained from the vision-correcting lens.

6. The AR device as claimed in claim 1, wherein, The vision-correcting lens includes a zoom lens, and The at least one processor is further configured to: Set control parameters for controlling the zoom lens, and Based on the set control parameters, information about the lens characteristics of the vision correction lens is obtained.

7. The AR device as claimed in claim 1, wherein, The at least one processor is further configured to: Obtain the position of at least one preset feature point and the position of the center point from the adjusted eye image, and The user's gaze information is obtained based on the position of the at least one preset feature point and the position of the center point.

8. A method for detecting a user's gaze performed by an augmented reality (AR) device including a vision-correcting lens, the method comprising: Information about the lens characteristics of the vision-correcting lens that overlaps with the light guide plate, the light guide plate being configured to display an image output from the AR device in the user's gaze direction; A light for gaze tracking is emitted from a light emitter located at a portion of the support of the AR device toward a light reflector, wherein the emitted light is reflected by the light reflector and then directed to the eyes of the user wearing the AR device; The light obtained by the light being guided to the user's eye and reflected by the user's eye is received by a light receiver disposed at the support member; The user's eye image is obtained based on the light received by the light receiver; Adjusting the user's eye image based on lens characteristic information about the vision-correcting lens; and Information about the user's gaze is obtained based on the adjusted eye image.

9. The method of claim 8, wherein, The light reflector includes a pattern, and The step of obtaining lens characteristic information about the vision-correcting lens includes: Light is emitted from the light emitter toward the light reflector to obtain information about the lens characteristics; Distortion patterns are identified based on light received by the light receiver; and Information about the lens characteristics of the vision-correcting lens is obtained based on the distortion pattern.

10. The method of claim 9, wherein, The step of obtaining lens characteristic information about the vision-correcting lens based on the distortion pattern includes: obtaining lens characteristic information about the vision-correcting lens by comparing a pattern included in the light reflector with a distortion pattern identified from light reflected by the light reflector and received by the light receiver.

11. The method of claim 8, wherein, An indicator for obtaining lens characteristic information about the vision-correcting lens is displayed on the vision-correcting lens, and The step of obtaining lens characteristic information about the vision-correcting lens includes: Identify the indicator; and Information about the lens characteristics of the vision-correcting lens is obtained based on the identified indicators.

12. The method of claim 8, wherein, The vision-correcting lens includes: a coupler configured to couple with the support of the AR device, and The step of obtaining lens characteristic information about the vision-correcting lens includes: Electromagnetic signals are applied to the vision-correcting lens via the coupler; Obtain a corresponding electromagnetic return signal in response to the applied electromagnetic signal from the vision-correcting lens; and Information about the lens characteristics of the vision-correcting lens is obtained based on the acquired electromagnetic return signal.

13. The method of claim 8, wherein, The vision-correcting lens includes a zoom lens, and The acquisition of lens characteristic information regarding the vision-correcting lens includes: Set control parameters for controlling the zoom lens; and Information about the lens characteristics of the vision correction lens is obtained based on the set control parameters.

14. The method of claim 8, wherein, The steps for obtaining gaze information about the user based on the adjusted eye image include: Obtain the position of at least one preset feature point and the position of the center point from the adjusted eye image; and The user's gaze information is obtained based on the position of the at least one preset feature point and the position of the center point.

15. A computer-readable recording medium having a program for performing the method as described in any one of claims 8 to 14 by a computer.

Citation Information

Patent Citations

  • Head-up display with eye tracking device determining user spectacles characteristics

    CN106575039A

  • Lens examination equipment and method

    CN109964109A