Electronic device for tracking gaze of user

Abstract

Provided is an electronic device for tracking the gaze of a user. The electronic device comprises: a waveguide; a first optical element and a second optical element configured to allow external light to propagate into the waveguide and be output to the outside of the waveguide; a third optical element located in an area excluding the areas in which the first optical element and the second optical element are located on the waveguide, and configured to diffract infrared light; and an image sensor that receives the light diffracted by the third optical element.

Description

Electronic devices for tracking the user's gaze

[0001] The present disclosure relates to an electronic device for tracking a user's gaze. Specifically, the present disclosure relates to an electronic device for tracking a user's gaze using a miniaturized camera module positioned on a waveguide.

[0002] An AR device is a device that provides augmented reality (AR), such as AR glasses. The primary optical system of an AR device includes a display engine (e.g., a projector) that outputs images and a waveguide that transmits the output images to the eye. The images emitted from the display engine are transmitted to the eye through the waveguide, allowing the user to view the images.

[0003] Meanwhile, a wearable display device is a device that allows a user to view a displayed screen while wearing it. Examples of wearable display devices include wearable glasses and head-mounted displays. For an AR device to be used as a wearable display, it must have an appearance that is comfortable for the average user.

[0004] In order to solve the above-described technical problem, an electronic device according to an embodiment of the present disclosure includes an IR (Infra-red) light source, a waveguide, a first optical element, a second optical element, a third optical element, and an image sensor. The IR light source can output infrared light. The first optical element can be positioned on a first region of the waveguide. The first optical element can be configured to allow external light to be incident into the waveguide. The second optical element can be positioned on a second region of the waveguide. The second optical element can be configured to allow external light incident into the waveguide to be output to the outside of the waveguide. The third optical element can be positioned on a third region of the waveguide. The third region can be different from the first region and the second region. The third optical element can be configured to diffract infrared light. The image sensor can receive the diffracted infrared light.

[0005] A waveguide image combiner according to one embodiment of the present disclosure may include a waveguide, a first optical element, a second optical element, and a third optical element. The first optical element may be positioned on a first region of the waveguide. The first optical element may be configured to allow external light to enter the waveguide. The second optical element may be positioned on a second region of the waveguide. The second optical element may be configured to allow external light to be output to the outside of the waveguide. The third optical element may be positioned on a third region of the waveguide. The third region may be different from the first region and the second region. The third optical element may be configured to diffract infrared light output from a light source.

[0006] An augmented reality device according to an embodiment of the present disclosure may include a display engine and a waveguide image combiner. The display engine may output image light. The waveguide image combiner may include a first optical element positioned on a first region of the waveguide and configured to allow external light to enter the waveguide. The waveguide image combiner may include a second optical element positioned on a second region of the waveguide and configured to allow external light to be output to the outside of the waveguide. The waveguide image combiner may include a third optical element positioned on a third region of the waveguide, different from the first and second regions, and configured to diffract infrared light output from a light source. The waveguide image combiner may be configured to guide the image light to a target region. The target region may include an eye motion box of a user.

[0007] The present disclosure can be readily understood by the combination of the following detailed description and the accompanying drawings, wherein reference numerals refer to structural elements.

[0008] FIG. 1 illustrates an example of a user wearing an electronic device according to an embodiment of the present disclosure.

[0009] FIG. 2 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0010] FIG. 3 is an internal side view of a portion of an electronic device for illustrating the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0011] FIG. 4 is a side view of an electronic device according to an embodiment of the present disclosure.

[0012] FIG. 5 is a conceptual diagram illustrating an operation of tracking a user's gaze according to one embodiment of the present disclosure.

[0013] FIG. 6 is a conceptual diagram illustrating an operation of tracking a user's gaze according to one embodiment of the present disclosure.

[0014] FIG. 7 is a conceptual diagram illustrating an operation of tracking a user's gaze according to one embodiment of the present disclosure.

[0015] FIG. 8 is a side view of an electronic device for explaining the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0016] FIG. 9 is an internal side view of a portion of an electronic device for illustrating the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0017] FIG. 10 is a side view of an electronic device for explaining the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0018] FIG. 11 is an internal side view of a portion of an electronic device for explaining the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0019] FIG. 12 is a conceptual diagram illustrating a third optical element according to an embodiment of the present disclosure.

[0020] FIG. 13 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0021] FIG. 14 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0022] FIG. 15 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0023] FIG. 16 is a block diagram illustrating components of an electronic device according to an embodiment of the present disclosure.

[0024] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0025] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0026] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0027] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0028] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0029] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0030] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0031] In the present disclosure, light may include first light, second light, and third light. The first light may include light of a virtual image output from a display unit. The second light may include light output from a light source and reflected by a user's eyes. The second light may include light in the infrared band and may also be referred to as infrared light. The third light may include light that may be naturally generated, excluding the first light and the second light. The third light may include natural light as noise that is not generated by an electronic device according to an embodiment.

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0033] FIG. 1 illustrates an example of a user wearing an electronic device according to an embodiment of the present disclosure.

[0034] Referring to FIG. 1, an electronic device (100) of one embodiment is a glasses-type display device configured to be worn by a user, and may be, for example, augmented reality glasses.

[0035] The electronic device (100) may include a glasses-shaped body having a frame (110) and temples (120). The frame (110) may have, for example, two rim shapes connected by a bridge. The rim and the bridge of the frame (110) may not be distinguished. The temples (120) may be connected to each end of the frame (110) and may extend in one direction. The ends of the frame (110) and the temples (120) may be connected by, for example, a hinge. As another example, the frame (110) and the temples (120) may be connected integrally.

[0036] A display unit (130) may be mounted on the glasses-shaped body. The display unit (130) is configured to output a first light containing a virtual image. In one embodiment, the display unit (130) may be a projector configured to output the first light.

[0037] In one embodiment, the display unit (130) may be mounted on the temple (120). In one embodiment, the display unit (130) may be mounted on the temple (120) such that the optical axis direction (OA) of the light output from the display unit (130) is the same as the longitudinal direction of the temple (120), but is not limited thereto. As another example, a part of the display unit (130) may be fixed to the temple (120), and another part of the display unit (130) may be fixed to the frame (110). For example, another part of the display (130) may be fixed on the rim shape of the frame (110). When the frame (110) and the temple (120) are formed integrally, the mounting position of the display unit (130) may not be distinguished between the frame (110) and the temple (120). The display unit (130) may be provided on each of the left and right eyes, or may be provided on only one side.

[0038] A waveguide (140) may be mounted on the frame (110). The waveguide (140) is configured to transmit the first light of the virtual image generated by the display unit (130) and the light of the external scene to the user's pupil. The waveguide (140) may have a flat plate shape. The waveguide (140) may be formed as a single-layer or multi-layer structure of a transparent material through which light can be reflected and propagated internally. Here, the transparent material means a material through which light in the visible light band can pass, and the transparency may not be 100% and may have a predetermined color. Since the waveguide (140) is formed of a transparent material, the user can view not only the virtual image but also the real scene through the electronic device (100), and thus the electronic device (100) can implement or provide an augmented reality experience to the user. The wave guide (140) may be provided on each of the left and right eyes in response to the display unit (130), or may be provided on only one side.

[0039] In one embodiment, the eyeglass lenses may be positioned on the frame (110), in which case the waveguide (140) may be attached to the eyeglass lenses or may be fixed to the frame (110) separately from the eyeglass lenses. It should be understood that the eyeglass lenses may be omitted.

[0040] A first optical element (150) may be positioned on one side of a waveguide (140) positioned on the optical axis (OA) of the display unit (130). The first optical element (150) may be positioned on a first surface of the waveguide (140) facing the display unit (130), a second surface opposite to the first surface, or inside the waveguide (140) (for example, between multiple layers in the case of a multi-layer waveguide) to input the first light output from the display unit (130) (for example, the first light (L1) illustrated in FIG. 2) into the waveguide (140). The first optical element (150) may include an input coupler that diffracts the first light (L1) and inputs it into the waveguide (140).

[0041] The first optical element (150) may be a diffractive element that propagates the first light into the waveguide (140). The diffractive element may be implemented with at least one of a diffractive optical element (DOE), a holographic optical element (HOE), a polymer dispersed liquid crystal (PDLC), a metasurface, etc.

[0042] A second optical element (for example, the second optical element (180) illustrated in FIG. 2) may be positioned on the other side of the waveguide (140). The first light of the virtual image propagating inside the waveguide (140) may be output to a target area through the second optical element (180). The target area may be a user's eye motion box (EMB). The second optical element (180) may include an output coupler that diffracts the light propagating inside the waveguide (140) and outputs it to the outside of the waveguide (140).

[0043] In one embodiment, the second optical element (180) may be a diffractive element configured to emit light propagating inside the waveguide (140) to the outside. The diffractive element may be implemented as at least one of a diffractive optical element (DOE), a holographic optical element (HOE), a polymer dispersed liquid crystal (PDLC), a metasurface, etc.

[0044] Information processing and image formation for the display unit (130) may be performed directly in a computer of the electronic device (100) itself, or may be performed in an external electronic device such as a smart phone, tablet, computer, laptop, or any other intelligent (smart) device when the electronic device (100) is connected to the external electronic device. Signal transmission between the electronic device (100) and the external electronic device may be performed via wired communication and / or wireless communication. The electronic device (100) may be powered by at least one of a built-in power source (e.g., a rechargeable battery) and an external device or external power source.

[0045] In one embodiment, the electronic device (100) may include a light source (90), a third optical element (160), and an image sensor (170).

[0046] In one embodiment, the light source (90) can output infrared light. The light source (90) can output infrared light toward the user's eye (E). The light output from the light source (90) can be reflected from the user's eye (E) and then incident toward the third optical element (160).

[0047] In one embodiment, the light source (90) may include an IR (Infra-red) light source that outputs infrared light.

[0048] In one embodiment, the third optical element (160) may be positioned on the waveguide (140). The third optical element (160) may be configured to diffract infrared light. The third optical element (160) may be configured to phase-modulate infrared light reflected from the user's eye (E).

[0049] In one embodiment, the second light reflected from the user's eye (e.g., the second light (L2) illustrated in FIG. 2) may be incident on the third optical element (160). The second light reflected from the user's eye may be diffracted by the third optical element (160). The degree to which the incident light is diffracted may vary depending on the design purpose of the third optical element (160).

[0050] In one embodiment, the third optical element (160) is positioned on the first surface (S1) facing the display portion (130) of the waveguide (140) and can diffract infrared light reflected from the user's eye (E) so that an image is formed on an image sensor (170) positioned on the second surface (S2) of the waveguide (140). The second surface (S2) may be a surface of the waveguide (140) opposite to the first surface (S1).

[0051] In one embodiment, the third optical element (160) may be configured to irregularly phase-modulate infrared light reflected from the user's eye (E). The infrared light may be phase-modulated as it passes through the third optical element (160) and transformed into a first waveform. The first waveform is received by the image sensor (170), and the electronic device may obtain an encoded image based on the received first waveform. The electronic device may track the user's gaze based on the encoded image.

[0052] In one embodiment, the third optical element (160) may be positioned in an area other than an area where the first optical element (150) and the second optical element (180) are positioned on the waveguide (140). The area where the first optical element (150) and the second optical element (180) are positioned to guide the first light of the virtual image output from the display unit (130) to the user's eye (E) and the area where the third optical element (160) is positioned to diffract the infrared light reflected from the user's eye (E) toward the image sensor (170) may be separated so as not to overlap.

[0053] For example, in one embodiment, the first optical element may be disposed within a first region on the waveguide (140). The second optical element may be disposed within a second region on the waveguide (140). The third optical element may be disposed within a third region on the waveguide (140). The third region may be distinct from the first region and the second region. The third region may be separated from the first region and the second region. The electronic device may obtain a clear image of the user's eye by receiving the infrared light entirely through the image sensor so that the first light does not act as noise, and may be advantageous in tracking the user's gaze.

[0054] In one embodiment, the third optical element (160) may be a diffractive element configured to modulate incident light. The diffractive element may be implemented as at least one of a diffractive optical element (DOE), a holographic optical element (HOE), a polymer dispersed liquid crystal (PDLC), a metasurface, etc.

[0055] In one embodiment, the image sensor (170) can receive the second light modulated by the third optical element (160). The electronic device (100) can obtain the user's gaze information by receiving the modulated second light through the image sensor (170). The electronic device (100) can track the user's gaze.

[0056] For example, the electronic device (100) can obtain an encoded image by receiving the modulated second light through the image sensor (170). The electronic device (100) can input the encoded image into an artificial intelligence model to obtain feature points or restore the image. The artificial intelligence model may be a model trained to obtain feature points or restore the image based on an encoded image that has been phase-modulated by a predetermined pattern.

[0057] A feature point may include information about at least one of the position coordinates and shape of an object. For example, a feature point may include at least one of a pupil feature point and a glint feature point of an eye.

[0058] Light is received by an image sensor (170). For example, the image sensor (170) may be a two-dimensional sensor assembled in the form of an array including a plurality of pixels arranged in a matrix, and each of the plurality of pixels may include at least one photoelectric conversion element. The image sensor (170) may detect light using the photoelectric conversion element and output an image signal, which is an electrical signal according to the detected light. Alternatively, the image signal may be an electrical signal corresponding to the detected light. The electronic device (100) may acquire an image by converting light received by the image sensor (170) into an electrical signal.

[0059] In one embodiment, the third optical element (160) may be positioned on the waveguide (140). The image sensor (170) may be positioned on the waveguide (140).

[0060] In one embodiment, the waveguide (140) may include a first surface (S1) and a second surface (S2) onto which second light reflected from the user's eye (E) is incident. The second surface (S2) may be an opposite surface to the first surface (S1). The third optical element (160) may be disposed on the first surface (S1), and the image sensor (170) may be disposed on the second surface (S2). The third optical element (160) and the image sensor (170) may be sequentially arranged according to (or based on) the propagation direction of the infrared light output from the light source (90).

[0061] Next, the operation of the electronic device (100) of the present embodiment will be described with reference to FIG. 2.

[0062] FIG. 2 is a drawing illustrating the operation of an electronic device according to one embodiment. For reference, FIG. 2 is a drawing illustrating the operation of the electronic device by enlarging the configurations of FIG. 1.

[0063] For convenience of explanation, parts that overlap with those described using Figure 1 are simplified or omitted.

[0064] Referring to FIG. 2, the display unit (130) can output a first light (L1) containing a virtual image. The first light (L1) output from the display unit (130) can be incident on one side of the waveguide (140). The first optical element (150) on the path of the first light (L1) can diffract the first light (L1) output from the display unit (130). The first light (L1) output from the display unit (130) can be diffracted by the first optical element (150) and enter the interior of the waveguide (140).

[0065] The first light (L1) entering the waveguide (140) is propagated while being totally reflected inside the waveguide (140) and can be diffracted by the second optical element (180) and emitted outside the waveguide (140). The first light (L1) emitted from the waveguide (140) is directed toward the user's eye (E), and the user can view the virtual image (I) emitted from the display unit (130).

[0066] In one embodiment, the electronic device (100) may further include a fourth optical element. The fourth optical element may be positioned on the waveguide (140). The fourth optical element may be configured to expand external light propagating within the waveguide. The fourth optical element may include a folding element that redirects the input light toward the second optical element (180) or an expanding element that expands the input light.

[0067] The first light (L1) may be propagated while being totally reflected inside the waveguide (140) and may be expanded by being diffracted by the fourth optical element. The first light (L1) may be diffracted by the fourth optical element and expanded in the second direction (y), or may be expanded in the first direction (x) and the second direction (y). However, the function of the fourth optical element may be shared by the second optical element (180), and the technical idea of ​​the present disclosure is not limited thereto. For example, the first light (L1) may be diffracted by the fourth optical element and expanded in the second direction (y), and may be diffracted by the second optical element (180) and expanded in the first direction (x), and then emitted outside the waveguide (140).

[0068] The fourth optical element may be positioned between the first optical element (150) and the second optical element (180), may overlap with the second optical element (180) in some area, or may overlap with the second optical element (180) in the same area.

[0069] In one embodiment, the electronic device (100) may include a light source (90). The light source may output a second light (L2) used to track the user's gaze. In this case, the gaze tracking sensor may utilize an infrared (IR) light source.

[0070] In order to track the gaze, the second light (L2) in the infrared band output from the infrared light source may be reflected by the user's eye (E) and may pass through the third optical element (160). The second light (L2) in the infrared band may be diffracted by the third optical element (160) to form an image on the imaging surface of the image sensor (170) or may be transformed into a first waveform with irregular phase modulation.

[0071] The second light (L2) reflected from the eye (E) and transmitted through the third optical element (160) can be received by the image sensor (170). For example, the second light (L2) transmitted through the third optical element (160) forms an image on the image plane of the image sensor (170), and the electronic device (100) can obtain an image of the user's eye (E) by receiving the second light (L2) transmitted through the third optical element (160) by the image sensor (170).

[0072] As another example, the second light (L2) transmitted through the third optical element (160) may be irregularly phase-modulated by the third optical element (160), thereby generating a first waveform (W1 of FIG. 7). The first waveform (W1) may be received by the image sensor (170). The electronic device (100) may obtain an encoded image of the user's eye (E) by receiving the first waveform (W1) by the image sensor (170). The electronic device (100) may obtain information about the user's gaze based on the encoded image, and may track the user's gaze. For example, the electronic device (100) may obtain a restored image by inputting the encoded image into an artificial intelligence model trained to restore the encoded image, and may track the user's gaze based on the restored image.

[0073] FIG. 3 is an internal side view of a portion of an electronic device for illustrating the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0074] For reference, FIG. 3 is an inner side view illustrating that the third optical element and the image sensor are arranged so as not to overlap with the first optical element and the second optical element, and for convenience of explanation, the explanation will focus on parts that are different from those described using FIGS. 1 and 2.

[0075] Referring to FIG. 3, an electronic device according to one embodiment may include a frame (110), a waveguide (140), a first optical element (150), a second optical element (180), a third optical element (160), and an image sensor (170).

[0076] A waveguide (140) may be mounted on the frame (110). As illustrated in FIG. 3, the frame (110) may support the waveguide (140) along its periphery. Glasses legs (120 in FIG. 1) are respectively connected to both ends of the frame (110), and the glasses legs may extend in a third direction (z).

[0077] In one embodiment, a display unit (130 in FIG. 1) may be mounted on the glasses frame (120). The display unit (130) may output a first light (L1) containing a virtual image in a third direction (z). The display unit (130) may output the first light (L1) toward the end of the waveguide (140).

[0078] For convenience of explanation, in FIG. 3, the first light (L1) is described as being divided into the first light (L11) incident toward the first optical element (150), the first light (L12) propagating into the interior of the waveguide (140), and the first light (L13) emitted from the second optical element (180).

[0079] The first light (L11) can be incident toward the first optical element (150). The first optical element (150) can be located on the first surface facing the display unit (130) of the waveguide (140), the second surface opposite the first surface, or the inside of the waveguide (140) (in the case of a multilayer waveguide, between the two surfaces) to input the first light (L11) output from the display unit (130) into the waveguide (140).

[0080] The first light (L12) entering the waveguide (140) can be propagated while being totally reflected inside the waveguide (140). The first light (L12) of the virtual image (I) propagated inside the waveguide (140) can proceed along the waveguide (140) until it reaches the second optical element (180).

[0081] The first light (L12) can be incident toward the second optical element (180). The first light (L12) can be diffracted by the second optical element (180) and emitted to the outside of the waveguide (140). The second optical element (180) can be positioned on the first surface facing the display unit (130) of the waveguide (140), the second surface opposite the first surface, or the inside of the waveguide (140) (in the case of a multilayer waveguide, between the two surfaces) to cause the first light (L13) output from the display unit (130) to be emitted from the waveguide (140).

[0082] The first light (L13) of the virtual image propagated inside the waveguide (140) can be output through the second optical element (180). The area where the first light (L13) is output can be a target area (TA of FIG. 4). The target area (TA) can be a user's eye motion box (I of FIG. 2). The electronic device (100) provides a virtual image to the user through the first light (L13) of the virtual image emitted across the target area (TA), and at the same time, the user of the electronic device (100) can view a real scene beyond the waveguide (140) formed of a transparent material.

[0083] In one embodiment, the third optical element (160) may be positioned in an area on the waveguide (140) other than an area where the first optical element (150) and the second optical element (180) are positioned. The third optical element (160) may be positioned in an area different from the area where the first optical element (150) and the second optical element (180) are positioned on the waveguide (140). For example, as illustrated in FIG. 3, the third optical element (160) may be positioned at the bottom of the waveguide (140) other than an area where the second optical element (180) is positioned. The third optical element (160) may be positioned at a bottom of the waveguide (140) other than an area where the second optical element (180) is positioned. However, this is only an example and does not limit the technical idea of ​​the present disclosure, and the third optical element (160) may be placed on a side of the waveguide (140) other than the area where the second optical element (180) is positioned. The third optical element (160) may also be placed on a side of the waveguide (140) other than the area where the second optical element (180) is positioned.

[0084] In one embodiment, the third optical element (160) may be positioned in an area excluding the path of light along which the first light (L11, L12, L13) propagates, in an area different from the path of light along which the first light (L11, L12, L13) propagates, or in an area distinct from the path of light along which the first light (L11, L12, L13) propagates. For example, the third optical element (160) may not be positioned on the path of the first light (L12) that is diffracted by the first optical element (150) and propagates into the waveguide (140) toward the second optical element (180). The third optical element (160) may be positioned in an area of ​​the waveguide (140) that does not overlap with the path of the first light (L12) propagating into the waveguide (140).

[0085] In one embodiment, the third optical element (160) and the image sensor (170) may be sequentially arranged according to the direction of propagation of infrared light output from the light source (90). For example, as the infrared light propagates from the inner surface of the waveguide (140) toward the outer surface, the third optical element (160) may be arranged on the inner surface of the waveguide (140), and the image sensor (170) may be arranged on the outer surface of the waveguide (140).

[0086] In one embodiment, the image sensor (170) may be positioned in an area other than the area where the first optical element (150) and the second optical element (180) are positioned on the waveguide (140) or in an area other than the path of light along which the first light (L11, L12, L13) travels.

[0087] FIG. 4 is a side view of an electronic device according to an embodiment of the present disclosure.

[0088] For convenience of explanation, any details overlapping with those described using FIGS. 1 to 3 will be simplified or omitted, and the explanation will focus on the arrangement of some components of the electronic device.

[0089] Referring to FIG. 4, the electronic device (100) includes a waveguide (140), a third optical element (160), and an image sensor (170).

[0090] In one embodiment, the third optical element (160) and the image sensor (170) may be arranged on the waveguide (140). The third optical element (160) and the image sensor (170) may be arranged sequentially according to the propagation direction of the second light incident on the third optical element (160). The second light (L2) incident on the third optical element (160) is light output from the light source (90) and refers to light reflected from the user's eye (E).

[0091] The second light (L2) incident on the third optical element (160) may be light in the infrared band output from a light source (90 in FIG. 2). The light source outputs the second light (L2) in the infrared band toward the user's eye (E), and after being output, the second light (L2) may be reflected from the user's eye (E) and input to the third optical element (160). The output second light (L2) may be diffracted by the third optical element (160) and then input to the image sensor (170).

[0092] In one embodiment, the electronic device (100) can track the user's gaze by receiving the second light (L2) diffracted by the third optical element (160) through the image sensor (170). For example, the electronic device (100) can track the user's gaze by obtaining an image of the user's eye based on the second light (L2) diffracted by the third optical element (160), determining the user's gaze information after obtaining feature points of the user's eye, or obtaining an encoded image of the user's eye and then restoring the image.

[0093] In one embodiment, the waveguide (140) may include a first surface (S1) and a second surface (S2). The first surface (S1) may be a surface onto which the second light (L2) is incident. The second surface (S2) may be a surface opposite to the first surface (S1). The third optical element (160) may be disposed on the first surface (S1). The image sensor (170) may be disposed on the second surface (S2). The second light (L2) incident toward the third optical element (160) may sequentially pass through the third optical element (160) and the waveguide (140) and then be received by the image sensor (170). The second light (L2) diffracted as it passes through the third optical element (160) may be received by the image sensor (170).

[0094] In one embodiment, the element surface (160s) of the third optical element (160) and the sensor surface (170s) of the image sensor (170) may be parallel to each other. As illustrated in FIG. 3, the element surface (160s) of the third optical element (160) and the sensor surface (170s) of the image sensor (170) may be planes parallel to the xy plane.

[0095] In the present disclosure, the element surface (160s) of the third optical element (160) may refer to a plane on which a nanostructure of the third optical element (160) is arranged. The third optical element (160) may be installed on the first surface (S1) of the waveguide (140). For example, the third optical element (160) may be installed on an xy plane, and the element surface (160s) of the third optical element (160) may be a plane parallel to the xy plane.

[0096] In the present disclosure, the sensor surface (170s) of the image sensor (170) may refer to an imaging surface on which an image of incident light is formed. As illustrated in FIG. 3, the image sensor (170) may have an imaging surface configured along the xy plane to receive light incident in a third direction (z) or a diagonal direction in which the third direction and the first direction or the second direction are partially combined. The sensor surface (170s) of the image sensor (170) may be a surface parallel to the xy plane.

[0097] However, for the sake of convenience of explanation, the description is based on the xy plane, and it is obvious that the element surface (160s) and the sensor surface (170s) may be tilted with respect to the xy plane. This does not limit the technical idea of ​​the present disclosure.

[0098] In one embodiment, the target area (TA) may be an area of ​​the waveguide (140) where the first light (L1) is exposed outside the waveguide (140) through the second optical element (180). The electronic device (100) may provide an image to the user through the first light (L1) emitted within the target area (TA). The electronic device (100) may provide a virtual image to the user across the target area (TA), and the wearer of the electronic device (100) may also view a real scene beyond the waveguide (140) formed of a transparent material.

[0099] In one embodiment, the third optical element (160) and the image sensor (170) may be positioned so as not to overlap with the target area (TA). By positioning the third optical element (160) and the image sensor (170) so as not to overlap with the target area (TA) where the first light (L1) is exposed to the user's eye (E), the electronic device (100) can provide a clear image without visual interference of the physical configuration.

[0100] The shape in which the third optical element (160) and the image sensor (170) are positioned so that the first light (L1) does not overlap with the target area (TA) exposed to the user's eye (E) can be easily understood through FIG. 3.

[0101] FIG. 5 is a conceptual diagram illustrating an operation of tracking a user's gaze according to one embodiment of the present disclosure.

[0102] For convenience of explanation, parts that overlap with those described using Figure 4 are simplified or omitted.

[0103] Referring to FIG. 5, the second light (L2) output from the light source may be reflected from the user's eye (E) and incident on the third optical element (160). For convenience of explanation, the second light (L2) reflected from the user's eye (E) in FIGS. 5 to 7 is described as light including a chief ray (CR) and at least one marginal ray (MR).

[0104] In one embodiment, the third optical element (160) is configured to diffract incident light so that an image is formed on the imaging surface of the image sensor (170). A chief ray (CR) passes through the center of the third optical element (160), is diffracted by the third optical element (160), and can be detected by a photoelectric conversion element located at a first position (P1) of the imaging surface of the image sensor (170). A sub-ray (MR) passes through one end of the third optical element (160), is diffracted by the third optical element (160), and can be detected by a photoelectric conversion element located at a first position (P1) of the imaging surface of the image sensor (170). The chief ray (CR) and the sub-ray (MR) can be imaged at the first position (P1) of the imaging surface of the image sensor (170). The third optical element (160) can perform a kind of lens function and diffract the incident second light (L2) so that an image is formed at the first position (P1) of the image plane of the image sensor (170).

[0105] In one embodiment, the image sensor (170) can receive second light (L2) reflected from the user's eye (E). The image sensor (170) can detect light using a photoelectric conversion element disposed at a first position (P1) and output an image signal, which is an electrical signal according to the detected light. The electronic device (100) can acquire an image (10) by converting the second light (L2) received by the image sensor (170) into an electrical signal.

[0106] The electronic device (100) can obtain the user's gaze information from the image (10). The electronic device (100) can track the user's gaze from the image (10). The electronic device (100) can obtain biometric information, such as the user's iris information, from the image (10), and can authenticate the user using the obtained biometric information.

[0107] FIG. 6 is a conceptual diagram illustrating an operation of tracking a user's gaze according to one embodiment of the present disclosure.

[0108] For convenience of explanation, parts that overlap with those described using Figure 5 are simplified or omitted.

[0109] In one embodiment, light passing through the waveguide (140) toward the image sensor (170) may further include first light (L1) and third light (L3) propagated into the waveguide (140) in addition to second light (L2) reflected from the user's eye (E) and transmitted by the third optical element (160).

[0110] The first light (L1) may be light output from the display unit (130 in FIG. 2) and propagated into the waveguide (140). The first light (L1) may be light containing a virtual image. A portion of the first light (L1) may be diffracted by at least one of the first optical element (150), the second optical element (180), and the third optical element (160) and emitted to the outside of the waveguide (140). A portion of the emitted first light (L1) may proceed toward the image sensor (170).

[0111] The third light (L3) may be a naturally occurring light, excluding the first light (L1) and the second light (L2). For example, the third light (L3) may be natural light. The third light (L3) may be directed toward the image sensor (170).

[0112] The image sensor (170) can obtain an image of the user's eye (E) by receiving the second light (L2). The first light (L1) and the third light (L3) incident on the image sensor (170) can act as noise. When the image sensor (170) receives the first to third lights (L1, L2, L3) to implement an image of the user's eye (E), an image of degraded quality can be obtained.

[0113] In one embodiment, the electronic device (100) may further include an optical filter (190).

[0114] In one embodiment, the optical filter (190) may be placed between the third optical element (160) and the image sensor (170). That is, the third optical element (160), the optical filter (190), and the image sensor (170) may be sequentially arranged according to the direction of propagation of light incident on the third optical element (160).

[0115] For example, an optical filter (190) may be placed on a waveguide (140). The optical filter (190) may be placed between the waveguide (140) and the image sensor (170).

[0116] The optical filter (190) can filter light having a wavelength within a predetermined wavelength range. The optical filter (190) can transmit light having a wavelength within a predetermined wavelength range and block light having a wavelength outside the predetermined wavelength range. For example, the optical filter (190) can transmit light in the infrared band and block light outside the infrared band. Accordingly, the image sensor (170) can receive light in a limited manner in the infrared band and acquire an image based on light in the infrared band.

[0117] In one embodiment, the optical filter (190) can filter the second light (L2). The optical filter (190) can transmit the second light (L2) and block the first light (L1) and the third light (L3). The image sensor (170) can restrictively receive the second light (L2) transmitted by the optical filter (190) and capture an image of the user's eye (E) based on the second light (L2). Since the first light (L1) and the third light (L3), which act as noise, are blocked, a higher quality image can be obtained.

[0118] FIG. 7 is a conceptual diagram illustrating an operation of tracking a user's gaze according to one embodiment of the present disclosure.

[0119] For convenience of explanation, parts that overlap with those described using Figures 5 and 6 are simplified or omitted.

[0120] In one embodiment, the third optical element (160) is configured to modulate the phase of the incident second light (L2). The second light (L2), which includes a chief ray (CR) and a sub-ray (MR), can have its phase modulated by the third optical element (160). As the second light (L2) passes through the third optical element (160) having a predetermined pattern, the phase-modulated second light (L2) is transformed into a first waveform (W1), and the first waveform (W1) can reach the imaging surface of the image sensor (170). The pattern of the third optical element (160) can be formed irregularly, and the second light (L2) can be irregularly phase-modulated and transformed into the first waveform (W1). As the second light (L2) passes through the third optical element (160), its phase is modulated, and the first waveform (W1) can propagate beyond the third optical element (160).

[0121] The first waveform (W1) can be transmitted through the optical filter (190). For example, the second light (L2) can be light in the infrared band, and the first waveform (W1) phase-modulated from the second light (L2) can also be light in the infrared band. The first waveform (W1) can be transmitted through the optical filter (190). The first waveform (W1) can be detected by a photoelectric conversion element located on the imaging surface of the image sensor (170).

[0122] In one embodiment, the image sensor (170) can receive a first waveform (W1) that is irregularly phase-modulated by the third optical element (160). The image sensor (170) can detect light using a photoelectric conversion element and output an image signal, which is an electrical signal according to the detected light. The electronic device (100) can obtain an encoded image (20) by converting the first waveform (W1) received by the image sensor (170) into an electrical signal.

[0123] The encoded image (20) may vary depending on at least one of the object from which the light is reflected (E; i.e., the eye), the arrangement shape of the nanostructures included in the third optical element (160), and the optical properties of the light diffracted by the third optical element (160). In general, an object expressed in the encoded image (20) phase-modulated by the third optical element (160) may be difficult to identify with the naked eye.

[0124] In one embodiment, the electronic device (100) can obtain the user's gaze information from the encoded image (20) using a processor. The electronic device (100) can track the user's gaze direction based on the user's gaze information.

[0125] In one embodiment, the electronic device (100) may acquire feature points from an encoded image (20) by a processor. The processor may include an artificial intelligence algorithm (710) or an artificial intelligence network for acquiring feature points. The artificial intelligence algorithm (710) included in the processor may be an algorithm trained to extract feature points from a modified image, i.e., an encoded image (20). The electronic device (100) may track a user's gaze direction based on the feature points extracted using the artificial intelligence algorithm (710).

[0126] For example, the feature point may include at least one of a pupil feature point and a glint feature point. The electronic device (100) may extract at least one of a pupil feature point and a glint feature point from the encoded image (20). The electronic device (100) may track the user's gaze direction based on at least one of the pupil feature point and the glint feature point extracted using an artificial intelligence algorithm (710).

[0127] In one embodiment, the electronic device (100) can restore an image from an encoded image (20) by a processor. An artificial intelligence algorithm (710) or an artificial intelligence network can be used to restore the image. The artificial intelligence algorithm (710) included in or executed by the processor can be an algorithm learned to restore an image from a modified image (e.g., the encoded image (20)). The electronic device (100) can track the user's gaze direction based on the restored image using the artificial intelligence algorithm (710). The electronic device (100) can acquire the user's biometric information based on the restored image using the artificial intelligence algorithm (710) and authenticate the user.

[0128] FIG. 8 is a side view of an electronic device for explaining the arrangement of optical elements and image sensors according to one embodiment of the present disclosure.

[0129] For convenience of explanation, the explanation will focus on differences from those described using Fig. 4.

[0130] Referring to FIG. 8, the third optical element (160) and the image sensor (170) may be placed on the waveguide (140), but may be placed at different heights.

[0131] In one embodiment, the third optical element (160) may be positioned at a first height (H1) on the waveguide (140). The image sensor (170) may be positioned at a second height (H2) on the waveguide (140). The first height (H1) and the second height (H2) may be different positions.

[0132] In the present disclosure, the first height (H1) and the second height (H2) may refer to heights set based on the first direction (x). For example, the first direction (x) may refer to a direction perpendicular to the ground when the user wears the electronic device (100).

[0133] In one embodiment, the third optical element (160) and the image sensor (170) may be arranged at different heights. Since the third optical element (160) and the image sensor (170) are arranged at different heights, the third optical element (160) and the image sensor (170) may not overlap each other when viewed in the third direction (z) that is perpendicular to the first direction (x). Of course, when viewed in the third direction (z), the third optical element (160) and the image sensor (170) may overlap each other in some areas.

[0134] In one embodiment, the third optical element (160) and the image sensor (170) may be positioned at different positions relative to the second direction (y). For example, the third optical element (160) and the image sensor (170) may be positioned misaligned along the horizontal direction.

[0135] FIG. 9 is an internal side view of a portion of an electronic device for explaining the arrangement of optical elements and image sensors according to an embodiment of the present disclosure. FIG. 10 is a side view of an electronic device for explaining the arrangement of optical elements and image sensors according to an embodiment of the present disclosure. An example of the structure of an electronic device according to an embodiment will be described using FIGS. 9 and 10.

[0136] For convenience of explanation, parts that overlap with those described using Figures 1 to 3 are simplified or omitted.

[0137] In one embodiment, the third optical element (160) may be positioned in an area other than the area where the first optical element (150) and the second optical element (180) are positioned on the waveguide (140) or in an area different from the area where the first optical element (150) and the second optical element (180) are positioned.

[0138] In one embodiment, the third optical element (160) and the image sensor (170) may be positioned on the central lower portion of the waveguide (140). The image sensor (170) may be positioned on the waveguide (140) so as to be covered by the frame (110). For a more detailed explanation, FIG. 10 is described, which enlarges some of the configurations and illustrates a side view.

[0139] FIG. 10 is a side view of an electronic device for explaining the arrangement of optical elements and image sensors according to one embodiment of the present disclosure.

[0140] For convenience of explanation, the explanation will focus on differences from those described using FIGS. 4 and 8.

[0141] Referring to FIG. 10, in one embodiment, the frame (110) may surround a third side (S3) and a portion of a first side (S1) of the waveguide (140). The third side (S3) may refer to a lower surface of the waveguide (140). The first side (S1) may refer to a front surface of the waveguide (140) onto which infrared light reflected from the user's eye (E) is incident. The frame (110) may surround a portion of the third side (S3) and the first side (S1) of the waveguide (140) and support the waveguide (140).

[0142] In one embodiment, the third optical element (160) may be disposed on the first surface (S1). The image sensor (170) may be disposed on the second surface (S2). The second surface (S2) may refer to the rear surface of the waveguide (140) opposite to the first surface (S1). The third optical element (160) and the image sensor (170) may be sequentially arranged according to the direction of propagation of light incident on the third optical element (160).

[0143] In one embodiment, the third optical element (160) may be arranged at a first height on the waveguide (140). The image sensor (170) may be arranged at a second height on the waveguide (140). The first height may be different from the second height. The third optical element (160) and the image sensor (170) may each be arranged on the waveguide (140), but may be arranged at different heights with respect to the first direction (x).

[0144] In one embodiment, the image sensor (170) may be disposed on a portion of the second surface (S2) of the waveguide (140) corresponding to a portion of the first surface (S1) of the waveguide (140) surrounded by the frame (110). The image sensor (170) may be covered by the frame (110) surrounding a portion of the first surface (S1) of the waveguide (140) when viewed from a viewpoint along the third direction (z). Alternatively, the image sensor (170) may be covered by the frame (110) surrounding a portion of the first surface (S1) of the waveguide (140).

[0145] The third optical element (160) may be placed on a frame (110) surrounding a portion of the first surface (S1) of the waveguide (140). The third optical element (160) may be placed on a frame (110) surrounding a portion of the first surface (S1) of the waveguide (140).

[0146] By arranging the image sensor (170) so that it is covered by the frame (110), the electronic device (100) according to one embodiment can provide a more natural real scene.

[0147] FIG. 11 is an internal side view of a portion of an electronic device for explaining the arrangement of a third optical element and an image sensor according to one embodiment of the present disclosure.

[0148] For convenience of explanation, the explanation will focus on differences from those described using FIGS. 3 and 9.

[0149] Referring to FIG. 11, in one embodiment, the electronic device (100) may further include a fourth optical element (175).

[0150] The fourth optical element (175) may be positioned on the waveguide (140). The fourth optical element (175) may be configured to expand the first light (L1) propagating within the waveguide (140). In addition, the fourth optical element (175) may be configured to cause the input first light (L1) to propagate toward the second optical element (180). The fourth optical element (175) may include a folding element that changes the direction of the input light toward the second optical element (180) or an expanding element that expands the input light.

[0151] In one embodiment, the fourth optical element (175) may be a diffractive element configured to change the direction of propagation of light or to expand input light. The diffractive element may be implemented as at least one of a diffractive optical element (DOE), a holographic optical element (HOE), a polymer dispersed liquid crystal (PDLC), a metasurface, etc.

[0152] The first light (L1) may be propagated while being totally reflected inside the waveguide (140) and may be diffracted and expanded by the fourth optical element (175). The first light (L1) may be diffracted by the fourth optical element (175) and expanded in the second direction (y), or may be expanded in the first direction (x) and the second direction (y). However, the function of the fourth optical element (175) may be shared by the second optical element (180), and the technical idea of ​​the present disclosure is not limited thereto. For example, the first light (L1) may be diffracted by the fourth optical element (175) and expanded in the second direction (y), and may be diffracted by the second optical element (180) and expanded in the first direction (x), and then emitted outside the waveguide.

[0153] Although FIG. 11 illustrates that the fourth optical element (175) and the second optical element (180) are arranged as separate configurations, the fourth optical element (175) and the second optical element (180) may be formed integrally. For example, the first light (L1) may be incident into the waveguide (140) by the first optical element (150), expanded by the second optical element (180), and emitted outside the waveguide (140). The technical idea of ​​the present disclosure is not limited thereto.

[0154] For convenience of explanation, in FIG. 11, the first light (L1) is divided into the first light (L11) incident toward the first optical element (150), the first light (L12) propagating into the interior of the waveguide (140), the first light (L13) expanded and having its direction of propagation changed inside the waveguide (140), and the first light (L14) emitted from the second optical element (180).

[0155] The first light (L11) can be incident toward the first optical element (150). The first optical element (150) can be located on the first surface facing the display unit (130) of the waveguide (140), the second surface opposite the first surface, or the inside of the waveguide (140) (for example, between the two in the case of a multilayer waveguide) to input the first light (L11) output from the display unit (130) into the waveguide (140).

[0156] The first light (L12) entering the waveguide (140) can be propagated while being totally reflected inside the waveguide (140). The first light (L12) of the virtual image (I) propagated inside the waveguide (140) can proceed along the waveguide (140) until it reaches the fourth optical element (175).

[0157] The first light (L12) may be incident toward the fourth optical element (175). The first light (L12) may be diffracted by the fourth optical element (175) and expanded in the second direction (y), or the direction in which the first light (L12) travels may be changed toward the second optical element (180). As another example, the first light (L12) may be diffracted by the fourth optical element (175) and expanded in the first direction (x) and the second direction (y), but the technical idea of ​​the present disclosure is not limited thereto.

[0158] The first light (L13) can be incident toward the second optical element (180). The first light (L13) can be diffracted by the second optical element (180) and emitted to the outside of the waveguide (140). The second optical element (180) can be positioned on the first surface facing the display unit (130) of the waveguide (140), the second surface opposite the first surface, or the inside of the waveguide (140) (for example, between the first and second surfaces in the case of a multilayer waveguide) so as to cause the first light (L14) output from the display unit (130) to be emitted from the waveguide (140).

[0159] The first light (L14) of the virtual image propagated inside the waveguide (140) can be output through the second optical element (180). The area where the first light (L14) is output can be a target area (TA of FIG. 4). The target area (TA) can be a user's eye motion box (I of FIG. 2). The electronic device (100) provides a virtual image to the user through the first light (L14) of the virtual image emitted across the target area (TA), and at the same time, the user of the electronic device (100) can view a real scene beyond the waveguide (140) formed of a transparent material.

[0160] In one embodiment, the third optical element (160) may be positioned in an area other than the areas where the first optical element (150), the second optical element (180), and the fourth optical element (175) are positioned. The third optical element (160) may be positioned in an area separate from the areas where the first optical element (150), the second optical element (180), and the fourth optical element (175) are positioned. The third optical element (160) may be positioned in an area different from the areas where the first optical element (150), the second optical element (180), and the fourth optical element (175) are positioned.

[0161] In one embodiment, the third optical element (160) may be positioned in an area other than the path of light along which the first light (L11, L12, L13, L14) propagates. The third optical element (160) may be positioned in an area separate from the path of light along which the first light (L11, L12, L13, L14) propagates. The third optical element (160) may be positioned in an area different from the path of light along which the first light (L11, L12, L13, L14) propagates. For example, the third optical element (160) may not be positioned on the path of the first light (L12) that is diffracted by the first optical element (150) and propagates into the waveguide (140) toward the fourth optical element (175). The third optical element (160) may be placed in an area of ​​the waveguide (140) that does not overlap with the path of the first light (L12) propagating inside the waveguide (140).

[0162] FIG. 12 is a conceptual diagram illustrating an optical element according to one embodiment of the present disclosure.

[0163] Referring to FIG. 12, in one embodiment, the electronic device (100) can modulate the phase of incident light using a third optical element (160). The third optical element (160) can be a diffractive element that modulates the phase of incident light. The diffractive element can be implemented as at least one of a diffractive optical element (DOE), a holographic optical element (HOE), a polymer dispersed liquid crystal (PDLC), a metasurface, etc.

[0164] The third optical element (160) may have a pattern formed in a circular shape, for example. The third optical element (160) may have a pattern in which a certain shape is regularly repeated. That is, the third optical element (160) may have a concentric pattern in which circular patterns having different radii are repeated.

[0165] However, the shape of the pattern of the optical element illustrated in FIG. 10 is merely an example, and the technical concept of the present disclosure is not limited thereto. For example, the optical element may have a pattern formed in a square shape. The optical element may have a pattern with repeating square shapes. As another example, the optical element may have a generally circular pattern, but may also include some irregular patterns.

[0166] For convenience of explanation, the third optical element (160) is described by enlarging the second region (R2) of the optical element. In one embodiment, the third optical element (160) may include a substrate (161) and a plurality of nanostructures (162).

[0167] The substrate (161) may have a flat shape. A plurality of nanostructures (162) may be arranged on the substrate (161). The plurality of nanostructures (162) may extend perpendicularly to the upper surface of the substrate (161). As illustrated in FIG. 12, the plurality of nanostructures (162) may include columns having a rectangular parallelepiped shape. The plurality of nanostructures (162) may be arranged to be spaced apart from each other at a predetermined interval.

[0168] The constituent materials of the plurality of nanostructures (162) may be formed of a dielectric material having a high refractive index to maximize interaction with incident light and thereby increase the possibility of complex amplitude control. For example, the plurality of nanostructures (162) may be formed of a-Si, a-Si:H, TiO2, GaN, etc.

[0169] Although the heights of the plurality of nanostructures (162) in FIG. 12 are shown as being irregularly formed, the technical concept of the present invention is not limited thereto. For example, the plurality of nanostructures (162) may be arranged in a form in which the first height and the second height are regularly repeated.

[0170] In one embodiment, the plurality of nanostructures (162) included in the third optical element (160) can guide incident light incident on the third optical element (160) to an arbitrary focus according to their heights and patterns. Here, the 'height and pattern' may refer to the height difference of each of the plurality of nanostructures (162) and the pattern shape on the third optical element (160) due to the height difference. For example, the third optical element (160) may perform the function of a lens.

[0171] In one embodiment, the plurality of nanostructures (162) included in the third optical element (160) can modulate the phase of incident light incident toward the third optical element (160) according to their heights and patterns, and can generate an irregularly deformed first waveform. The electronic device (100) can obtain a coded image by receiving the first waveform through an image sensor. The electronic device (100) can track the user's gaze from the coded image.

[0172] The plurality of nanostructures (162) included in the third optical element (160) can guide incident light incident to the optical system to an arbitrary focus according to their arrangement shape, or can modulate the phase of the incident light to generate an irregularly deformed first waveform. Here, the 'array shape' may mean at least one of the size distribution by position, the shape distribution by position, and the interval distribution by position of the nanostructures (162) with respect to the size, shape, and interval of each of the nanostructures (162), and the area where the meta-lens is located. The detailed arrangement shape of the nanostructures (162) included in the third optical element (160) may vary depending on the optical performance required for the third optical element (160). For example, the arrangement shape of the nanostructures (162) may vary depending on the wavelength band, focal length, etc. of the light to be focused through the third optical element (160).

[0173] In one embodiment, the substrate (161) may be a waveguide (140). A plurality of nanostructures (162) may be installed on the waveguide (140). For example, a dielectric material may be applied on the waveguide (140). An etching process may be performed so that the applied dielectric material may have a certain pattern.

[0174] In one embodiment, a third optical element (160) including a substrate (161) and a plurality of nanostructures (162) may be placed on a waveguide (140). The substrate (161) may be placed on the waveguide (140) separately from the waveguide (140).

[0175] FIG. 13 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0176] Referring to FIG. 13, in step S1310, the electronic device can receive a first waveform generated by modulating the phase of light incident through an optical element through an image sensor. In step S1320, the electronic device can obtain an encoded image based on the first waveform.

[0177] In one embodiment, an electronic device may use an image sensor to receive a first waveform transmitted by an optical element. The first waveform may be a waveform of light that is irregularly phase-modulated from light incident on the optical element. The electronic device may obtain an encoded image by receiving the first waveform. The electronic device may obtain an encoded image corresponding to an object based on a distribution of the first waveform transmitted by the optical element.

[0178] The encoded image may vary depending on at least one of the object from which the light is reflected (e.g., an eye) and the arrangement shape of the nanostructures included in the optical element. Depending on the irregular arrangement shape of the nanostructures in the optical element, the encoded image obtained by receiving the first waveform transmitted by the optical element may be difficult to discern with the naked eye.

[0179] In step S1330, the electronic device can obtain the user's gaze information from the encoded image. The electronic device can obtain the user's gaze information based on the encoded image.

[0180] In one embodiment, an electronic device can acquire feature points by inputting an encoded image into an artificial intelligence model trained to extract feature points from the encoded image. The electronic device can acquire information about the user's gaze based on the extracted feature points. The electronic device can track the user's gaze based on the extracted feature points.

[0181] In one embodiment, the electronic device may obtain a restored image by inputting the encoded image into an artificial intelligence model trained to restore an original image from the encoded image. The electronic device may obtain a restored image by inputting the encoded image into an artificial intelligence model trained to restore an original image based on the encoded image. The electronic device may obtain information about the user's gaze based on the restored image. The electronic device may track the user's gaze based on the restored image.

[0182] FIG. 14 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0183] Referring to FIG. 14, step S1330 of FIG. 13 may include step S1410 and step S1420.

[0184] In step S1410, the electronic device can determine one or more feature points corresponding to the eye from the encoded image.

[0185] In the present disclosure, a feature point may refer to a point representing a key feature or point of interest in an image. For example, a feature point may be a corner where two or more edges intersect, detected based on changes in pixel values, or a point with a maximum or minimum pixel value in an image.

[0186] For example, the feature points related to the eye may include at least one of a pupil feature point and a glint feature point of the eye. The electronic device may extract at least one of the pupil feature point and the glint feature point of the eye from the encoded image.

[0187] In one embodiment, the electronic device can obtain feature points by inputting an encoded image into an artificial intelligence model trained to extract feature points from the encoded image.

[0188] In step S1420, the electronic device can obtain the user's gaze information based on the feature points.

[0189] In one embodiment, the electronic device can track the gaze direction of the user based on at least one of a pupil feature point and an eye glint feature point extracted using an artificial intelligence algorithm.

[0190] In one embodiment, an electronic device can extract feature points from an encoded image. The electronic device may include an artificial intelligence algorithm or an artificial intelligence network for extracting feature points. The artificial intelligence algorithm may be an algorithm trained to extract feature points from a modified image, i.e., an encoded image. The electronic device can track the user's gaze direction based on the extracted feature points using the artificial intelligence algorithm.

[0191] FIG. 15 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0192] Referring to FIG. 15, step S1330 of FIG. 13 may include step S1510 and step S1520.

[0193] In step S1510, the electronic device can obtain a restored image of the eye from the encoded image. The electronic device can obtain a restored image of the eye based on the encoded image.

[0194] In one embodiment, the electronic device can obtain a restored image by inputting the encoded image into an artificial intelligence model trained to restore the original image from the encoded image.

[0195] In step S1520, the electronic device can obtain the user's gaze information based on the restored image.

[0196] In one embodiment, the electronic device can track the user's gaze direction based on a restored image obtained using an artificial intelligence algorithm.

[0197] In one embodiment, an electronic device can obtain a restored image from an encoded image. The electronic device may include an artificial intelligence algorithm or an artificial intelligence network for obtaining the restored image. The artificial intelligence algorithm may be an algorithm trained to obtain a restored image from a modified image, i.e., an encoded image. The electronic device can track the user's gaze direction based on the restored image obtained using the artificial intelligence algorithm.

[0198] FIG. 16 is a block diagram illustrating components of an electronic device according to an embodiment of the present disclosure.

[0199] For convenience of explanation, parts that overlap with those described using Figures 1 to 3 are simplified or omitted.

[0200] The electronic device (1600) may be a device that acquires an image by receiving light transmitted through a third optical element (1613) via an image sensor (1630). The electronic device (1600) may be implemented as various devices, such as, for example, a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, a wearable device, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a camcorder, etc. In one embodiment of the present disclosure, the electronic device (1600) may be an augmented reality device. An augmented reality device is a device that can express 'augmented reality', and can include not only augmented reality glasses that are generally worn on the user's face, but also a head-mounted display apparatus (HMD) or an augmented reality helmet that is worn on the head.

[0201] Referring to FIG. 16, an electronic device (1600) may include an optical element (1610), a waveguide (1620), an image sensor (1630), a processor (1640), and a memory (1650). The image sensor (1630), the processor (1640), and the memory (1650) may each be electrically and / or physically connected to each other.

[0202] The components illustrated in FIG. 16 are merely in accordance with one embodiment of the present disclosure, and the components included in the electronic device (1600) are not limited to those illustrated in FIG. 16. The electronic device (1600) may not include some of the components illustrated in FIG. 16, and may further include components not illustrated in FIG. 16. For example, the electronic device (1600) may further include a power supply (e.g., a battery) that supplies driving power to the optical element (1610), the processor (1640), and the memory (1650).

[0203] In one embodiment, the image sensor (1630) of the present disclosure may be used to track gaze. In this case, the image sensor (1630) typically uses an infrared (IR) light source (1605). To track gaze using the image sensor (1630), infrared band light output from the infrared light source (1605) may be reflected from the eye and transmitted through an optical element (1610). The light reflected from the eye and transmitted through the optical element (1610) may be received by the image sensor (1630).

[0204] In one embodiment, the optical element (1610) may include a first optical element (1611), a second optical element (1612), and a third optical element (1613).

[0205] The first optical element (1611) may be a diffractive element that propagates external light into the waveguide (1620). The second optical element (1612) may be a diffractive element that outputs external light propagated into the waveguide (1620) by the first optical element (1611) to the outside of the waveguide (1620).

[0206] For example, external light containing a virtual image output from a display unit may be incident toward a waveguide (1620). The incident external light may be diffracted by a first optical element (1611) and totally reflected and propagated into the waveguide (1620). The propagated external light may be diffracted by a second optical element (1612) and output to the outside of the waveguide (1620). The external light containing the virtual image may reach the user's eyes as its propagation path is guided by the first optical element (1611) and the second optical element (1612). A user wearing an electronic device (1600) according to an embodiment may recognize the guided external light and view the virtual image. At the same time, since the waveguide (1620) may be made of a transparent material, the user may view a real scene beyond the waveguide (1620) together with the virtual image.

[0207] The third optical element (1613) may be a diffractive element configured to diffract infrared light output from an infrared light source (1605). The third optical element (1613) may be configured to phase-modulate infrared light reflected from the user's eye (E).

[0208] The third optical element (1613) may include a substrate and a plurality of nanostructures extending perpendicularly to an upper surface of the substrate on the substrate. The third optical element (1613) may include a separate substrate. For example, the third optical element (1613) may be disposed on a waveguide (1620), i.e., the substrate may be disposed on the waveguide (1620) and the plurality of nanostructures may be disposed on the substrate.

[0209] In one embodiment, the third optical element (1613) may include a plurality of nanostructures arranged in an area other than an area where the first optical element (1611) and the second optical element (1612) are positioned on the waveguide (1620). The third optical element (1613) may include a plurality of nanostructures arranged in an area separate from an area where the first optical element (1611) and the second optical element (1612) are positioned on the waveguide (1620). The third optical element (1613) may include a plurality of nanostructures arranged in an area different from an area where the first optical element (1611) and the second optical element (1612) are positioned on the waveguide (1620). For example, the third optical element (1613) may be arranged on the waveguide (1620), i.e., the plurality of nanostructures may be arranged in contact with the waveguide (1620).

[0210] In one embodiment, the third optical element (1613) may be a lens in which relatively small-scale nanostructures are arranged in a two-dimensional manner. The third optical element (1613) may include a surface formed of nanostructures arranged on a substrate, and the phase of light transmitted through the third optical element (1613) may be modulated depending on the arrangement and shape of the nanostructures.

[0211] In one embodiment, the third optical element (1613) may be configured to modulate incident light. The third optical element (1613) may be positioned on a waveguide (1620). The light incident on the third optical element may be light reflected from the user's eye.

[0212] In one embodiment, the third optical element (1613) can modulate the phase of the incident light so that the incident light forms an image within the imaging plane of the image sensor (1630). For example, the third optical element (1613) can perform the function of a lens.

[0213] In one embodiment, the third optical element (1613) can generate a first waveform from the incident light by irregularly modulating the phase of the incident light. The third optical element (1613) can generate a first waveform based on the incident light by irregularly modulating the phase of the incident light. The first waveform can be received by the image sensor (1630) after being transmitted by the third optical element (1613).

[0214] In one embodiment, the waveguide (1620) is configured to transmit the first light of the virtual image generated in the display unit and the light of the external scene to the user's pupil. The waveguide (1620) may have a flat plate shape. The waveguide (1620) may be formed of a single-layer or multi-layer structure of a transparent material through which light can be reflected and propagated internally. Here, the transparent material means a material through which light in the visible light band can pass, and the transparency may not be 100% and may have a predetermined color. Since the waveguide (1620) is formed of a transparent material, the user can view the virtual image and the real scene through the electronic device (1600), and thus the electronic device (1600) can implement or provide an augmented reality experience to the user. The waveguide (1620) may be provided respectively for the left and right eyes corresponding to the display unit, or may be provided only on one side.

[0215] In one embodiment, the electronic device (1600) may include a waveguide image combiner configured to receive light of a virtual image and transmit it to the user's eyes. The waveguide image combiner may refer to a configuration including an optical element (1610) and a waveguide (1620).

[0216] In one embodiment, the electronic device may be an augmented reality device comprising a display engine and a waveguide image combiner. The display engine may be configured to output image light. The waveguide image combiner may guide the image light output from the display engine to a target area. The target area may be a user's eye motion box.

[0217] In one embodiment, the augmented reality device may be augmented reality glasses including a left-eye element and a right-eye element corresponding to the user's left and right eyes. A display engine and a waveguide image combiner may be included in one or more of the left-eye elements and one or more of the right-eye elements.

[0218] The image sensor (1630) is a photographing element configured to receive light modulated by the third optical element (1613), convert the brightness or intensity of the received light into an electrical signal, and image the electrical signal, thereby obtaining a coded image. The image sensor (1630) may be implemented as, for example, a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), but is not limited thereto. The image sensor (1630) may be disposed on the waveguide (1620).

[0219] In one embodiment, a third optical element (1613) and an image sensor (1630) may be arranged on a waveguide (1620). The third optical element (1613) and the image sensor (1630) may be arranged sequentially according to the direction of propagation of light incident on the third optical element (1613).

[0220] For example, the waveguide (1620) may be positioned between the third optical element (1613) and the image sensor (1630). The waveguide (1620) may include a first surface and a second surface opposite the first surface, and the third optical element (1613) may be positioned on the first surface and the image sensor (1630) may be positioned on the second surface.

[0221] In one embodiment, the element surface of the third optical element (1613) and the sensor surface of the image sensor (1630) may be parallel to each other.

[0222] In one embodiment, the third optical element (1613) may be positioned at a first height on the waveguide (1620). The image sensor (1630) may be positioned at a second height on the waveguide (1620). The first height and the second height may be different from each other. For example, the third optical element (1613) and the image sensor (1630) may be positioned at different heights on the waveguide (1620).

[0223] In one embodiment, the electronic device may further include a frame surrounding a lower surface and a front portion of the waveguide (1620) or surrounding a lower surface and a front portion of the waveguide (1620). The image sensor (1630) may be positioned on a rear portion of the waveguide (1620) corresponding to the front portion of the waveguide (1620) surrounded by the frame. The image sensor (1630) may be obscured by the frame when viewed from the front of the waveguide (1620).

[0224] In one embodiment, the electronic device can provide a virtual image by outputting a first light containing a virtual image from a waveguide (1620) toward the user's eyes. For example, the first light can be output from a display unit installed on the temple of the glasses, propagated into the waveguide (1620) by a first optical element, and diffracted by a second optical element to be emitted out of the waveguide (1620).

[0225] The waveguide (1620) may include an image combiner region where the first light is exposed outside the waveguide. In one embodiment, the optical element (1610) and the image sensor (1630) may be positioned so as not to overlap the image combiner region.

[0226] The processor (1640) may execute one or more instructions or program codes stored in the memory (1650), and perform functions and / or operations corresponding to the instructions or program codes. The processor (1640) may be configured with hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (1640) may be configured with at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application processor (AP), an application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs), but is not limited thereto.

[0227] Although the processor (1640) is illustrated as a single element in FIG. 16, it is not limited thereto. In one embodiment, the processor (1640) may be configured as one or more than one.

[0228] In one embodiment of the present disclosure, the processor (1640) may be configured as a dedicated hardware chip that performs artificial intelligence (AI) learning.

[0229] The memory (1650) may store instructions and program codes that can be read by the processor (1640). The memory (1650) may include, for example, at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a mask ROM, a flash ROM, a flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0230] The memory (1650) may store instructions or program codes for performing functions or operations of the electronic device (1600). In one embodiment of the present disclosure, the memory (1650) may store at least one of instructions, algorithms, data structures, program codes, and application programs that can be read by the processor (1640). The instructions, algorithms, data structures, and program codes stored in the memory (1650) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0231] The memory (1650) may store instructions, algorithms, data structures, or program codes related to an artificial intelligence model for obtaining user gaze information from an encoded image. For example, the memory (1650) may store program codes for obtaining feature points or restored images from an encoded image and tracking the user's gaze using the feature points or restored images. A 'module' included in the memory (1650) refers to a unit that processes a function or operation performed by the processor (1640), and this may be implemented as software such as instructions, an algorithm, a data structure, or program code.

[0232] The processor (1640) can track the user's gaze from the encoded image. The processor (1640) can track the user's gaze by using an artificial intelligence algorithm trained to obtain the user's gaze information from the encoded image phase-modulated by a predetermined pattern of the optical element (1610).

[0233] In the following embodiments, the processor (1640) may be implemented by executing instructions or program codes stored in the memory (1650).

[0234] In one embodiment, the processor (1640) can obtain a coded image based on a first waveform received by the image sensor (1630). The processor (1640) can obtain the user's gaze information based on the coded image.

[0235] In one embodiment, the processor (1640) can obtain feature points about the eye based on the encoded image. The processor (1640) can obtain information about the user's gaze based on the feature points.

[0236] In one embodiment, the processor (1640) can obtain a restored image of the eye based on the encoded image. The processor (1640) can obtain information about the user's gaze based on the restored image.

[0237] In one embodiment, a processor of an electronic device may utilize an artificial intelligence model trained to obtain user gaze information from an encoded image. In one embodiment of the present disclosure, the artificial intelligence model may be a deep neural network model (DNN) trained by a supervised learning method by applying an encoded image obtained according to a simulated light distribution based on an optical element as input data and applying the user's gaze corresponding to the encoded image as an output ground truth. 'Training' may mean training a neural network to be able to discover or acquire on its own a method of analyzing input data for the neural network, a method of classifying the input data, and / or a method of extracting features necessary for generating result data from the input data. Specifically, through the learning process, the deep neural network model may train learning data (e.g., multiple original images and multiple feature points) to optimize weight values ​​within the neural network. The deep neural network model processes input data through a neural network having optimized weight values, thereby outputting a desired result.

[0238] The type of artificial intelligence model does not limit the technical idea of ​​the present disclosure, and the artificial intelligence model can be implemented as at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network. In addition, the artificial intelligence model can be subdivided. For example, the convolutional neural network model can be subdivided into a deep convolutional neural network (D-CNN) or a Capsnet neural network.

[0239] The processor (1640) can obtain the user's gaze information from the encoded image using a pre-trained artificial intelligence model. In one embodiment of the present disclosure, the processor (1640) inputs the encoded image obtained through the image sensor (1630) into the artificial intelligence model and performs inference using the artificial intelligence model, thereby obtaining the user's gaze information corresponding to the encoded image. The pre-trained artificial intelligence model can obtain, for example, feature points or restored images corresponding to the encoded image, and obtain the user's gaze information based on the feature points or restored images.

[0240] The artificial intelligence model may be stored in the memory (1650) of the electronic device (1600). However, this is merely an example and does not limit the technical concept of the present disclosure. For example, the artificial intelligence model may be stored in an external server. In this case, the electronic device (1600) further includes a communication interface capable of performing data communication with the external server, and may receive the artificial intelligence model or the inference result data (e.g., feature points) by the artificial intelligence model from the external server through the communication interface. In general, the electronic device (1600) may be limited compared to the server in terms of memory storage capacity, computational processing speed, and training data set collection capability. Therefore, operations requiring large-scale data storage and large-scale computational amounts may be performed on the server, and then the data and / or the artificial intelligence model may be transmitted to the electronic device (1600) via a communication network. In this case, the electronic device (1600) may receive and utilize the artificial intelligence model or the inference data by the artificial intelligence model through the server, thereby performing operations quickly and easily, even without a processor having large-scale memory and fast computational capabilities.

[0241] An electronic device according to an embodiment of the present disclosure may include an infrared (IR) light source, a waveguide, a first optical element, a second optical element, a third optical element, and an image sensor. The IR light source may output infrared light. The first optical element may be positioned on a first region of the waveguide. The first optical element may be configured to allow external light to enter the waveguide. The second optical element may be positioned on a second region of the waveguide. The second optical element may be configured to allow external light entering the waveguide to be output to the outside of the waveguide. The third optical element may be positioned on a third region of the waveguide. The third region may be different from the first region and the second region. The third optical element may be configured to diffract infrared light. The image sensor may receive the diffracted infrared light.

[0242] In one embodiment, the third optical element and the image sensor may be sequentially arranged based on the direction of propagation of the infrared light.

[0243] In one embodiment, the third optical element may be configured to diffract infrared light to form an image on the imaging plane of the image sensor.

[0244] In one embodiment, the third optical element may be configured to modulate the phase of the infrared light so that the first waveform reaches the imaging surface of the image sensor. The electronic device may further include at least one processor. The at least one processor may obtain a coded image based on the first waveform received by the image sensor. The at least one processor may obtain information about the user's gaze based on the coded image.

[0245] In one embodiment, at least one processor may determine one or more feature points corresponding to an eye based on an encoded image. At least one processor may obtain gaze information of the user based on the one or more feature points.

[0246] In one embodiment, at least one processor may obtain a reconstructed image of the eye based on the encoded image. At least one processor may obtain gaze information and biometric information of the user based on the reconstructed image.

[0247] In one embodiment, the waveguide may include a first surface through which infrared light is incident and a second surface opposite the first surface. A third optical element may be disposed on the first surface. An image sensor may be disposed on the second surface.

[0248] In one embodiment, the third optical element may be positioned at a first height on the waveguide. The image sensor may be positioned at a second height on the waveguide that is different from the first height.

[0249] In one embodiment, the electronic device may further include a frame surrounding a lower surface and a portion of the first surface of the waveguide. The image sensor may be disposed on a portion of a second surface of the waveguide corresponding to a portion of the first surface of the waveguide.

[0250] In one embodiment, the electronic device may further include a fourth optical element. The fourth optical element may be positioned on a fourth region of the waveguide. The fourth optical element may be configured to expand external light propagating within the waveguide. The fourth region may be different from the third region.

[0251] In one embodiment, the third optical element may include a substrate and a plurality of nanostructures extending perpendicular to an upper surface of the substrate.

[0252] In one embodiment, the first optical element, the second optical element, the third optical element, and the fourth optical element may each include a plurality of nanostructures disposed on a waveguide.

[0253] In one embodiment, the electronic device may further include an optical filter that filters light according to a wavelength range. The third optical element, the optical filter, and the image sensor may be sequentially arranged based on the direction of propagation of the infrared light.

[0254] A waveguide image combiner according to one embodiment of the present disclosure may include a waveguide, a first optical element, a second optical element, and a third optical element. The first optical element may be positioned on a first region of the waveguide. The first optical element may be configured to allow external light to enter the waveguide. The second optical element may be positioned on a second region of the waveguide. The second optical element may be configured to allow external light to be output to the outside of the waveguide. The third optical element may be positioned on a third region of the waveguide. The third region may be different from the first region and the second region. The third optical element may be configured to diffract infrared light output from a light source.

[0255] In one embodiment, the waveguide image combiner may further include a fourth optical element. The fourth optical element may be positioned on a fourth region of the waveguide. The fourth region may be different from the third region. The fourth optical element may be configured to expand external light propagating within the waveguide. The fourth region may be different from the third region.

[0256] An augmented reality device according to one embodiment may include a display engine and a waveguide image combiner. The waveguide image combiner may include a waveguide. The waveguide image combiner may include a first optical element positioned on a first region of the waveguide and configured to allow external light to enter the waveguide. The waveguide image combiner may include a second optical element positioned on a second region of the waveguide and configured to allow external light to be output to the outside of the waveguide. The waveguide image combiner may include a third optical element positioned on a third region of the waveguide, different from the first and second regions, and configured to diffract infrared light output from a light source. The display engine may output image light. The waveguide image combiner may be configured to guide the image light to a target region. The target region may include an eye motion box of a user.

[0257] In one embodiment, the augmented reality device may further include an image sensor that receives light diffracted by the third optical element. The third optical element and the image sensor may be sequentially arranged based on the direction of propagation of the infrared light.

[0258] In one embodiment, the augmented reality device may further include an optical filter that filters light according to a wavelength range. The third optical element, the optical filter, and the image sensor may be sequentially arranged based on the direction of propagation of the infrared light.

[0259] In one embodiment, the augmented reality device may include augmented reality glasses including a left-eye element corresponding to the user's left eye and a right-eye element corresponding to the user's right eye. A display engine and a waveguide image combiner may be included in at least one of the left-eye element and the right-eye element.

[0260] In order to solve the above-described technical problem, one embodiment of the present disclosure provides a computer-readable recording medium having recorded thereon a program for execution on a computer.

[0261] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0262] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. IR (Infra-red) light source (90) that outputs infrared light; Waveguide (140); A first optical element (150) positioned on a first region of the waveguide and configured to allow external light to enter the waveguide; A second optical element (180) positioned on the second region of the waveguide and configured to output external light incident into the waveguide to the outside of the waveguide; A third optical element (160) positioned on a third region different from the first region and the second region of the waveguide and configured to diffract the infrared light; and An image sensor (170) that receives the above diffracted infrared light; An electronic device comprising:

2. In paragraph 1, An electronic device in which the third optical element and the image sensor are sequentially arranged based on the direction in which the infrared light propagates.

3. In either of paragraphs 1 and 2, An electronic device wherein the third optical element is configured to diffract the infrared light so that an image is formed on the imaging surface of the image sensor.

4. In either of paragraphs 1 and 2, The third optical element is configured to modulate the phase of the infrared light so that the first waveform reaches the imaging surface of the image sensor, comprising at least one further processor, At least one processor of the above, Obtaining a coded image based on the first waveform received by the image sensor, An electronic device that obtains user's gaze information based on the encoded image.

5. In paragraph 4, At least one processor of the above, determining one or more feature points corresponding to the eye based on the encoded image; An electronic device that obtains gaze information of the user based on one or more of the above-mentioned features.

6. In paragraph 4, At least one processor of the above, Obtain a restored image of the eye based on the above encoded image, An electronic device that acquires the user's gaze information and biometric information based on the restored image.

7. In any one of paragraphs 1 to 6, The above waveguide includes a first surface onto which the infrared light is incident and a second surface opposite to the first surface, The third optical element is arranged on the first surface, An electronic device wherein the image sensor is disposed on the second surface.

8. In any one of paragraphs 1 to 7, The third optical element is positioned at a first height on the waveguide, An electronic device wherein the image sensor is disposed at a second height different from the first height on the waveguide.

9. In any one of paragraphs 1 to 8, A frame around the lower surface of the waveguide and a portion of the first surface; Including more, An electronic device wherein the image sensor is disposed on a portion of a second surface of the waveguide corresponding to a portion of the first surface of the waveguide.

10. In any one of the clauses 1 to 9, Further comprising a fourth optical element positioned on the fourth region of the waveguide and configured to expand the external light propagating within the waveguide; The fourth area above is an electronic device different from the third area above.

11. In any one of clauses 1 to 10, An electronic device, wherein the third optical element comprises a substrate and a plurality of nanostructures extending perpendicularly to an upper surface of the substrate on the substrate.

12. In any one of paragraphs 1 to 10, An electronic device, wherein the first optical element, the second optical element, the third optical element and the fourth optical element each include a plurality of nanostructures arranged on the waveguide.

13. In clauses 1 to 12, Further comprising an optical filter for filtering light according to wavelength range, An electronic device in which the third optical element, the optical filter, and the image sensor are sequentially arranged based on the direction in which the infrared light propagates.

14. Waveguide; A first optical element positioned on a first region of the waveguide and configured to allow external light to enter the waveguide; A second optical element positioned on a second region of the waveguide and configured to output external light to the outside of the waveguide; A waveguide image combiner comprising a third optical element positioned in a third region different from the first region and the second region of the waveguide and configured to diffract infrared light output from a light source.

15. In paragraph 14, It further includes a fourth optical element (175) positioned on the fourth region of the waveguide and configured to expand the external light propagated within the waveguide, The fourth region above is a waveguide image combiner different from the third region above.

Images