Electronic device for displaying augmented reality

Abstract

An electronic device for displaying AR includes: an optical engine; a first polarizer; a polarization converter configured to maintain or convert a polarization direction of light of the real scene; a waveguide from which light of the virtual image is output and through which light of the real scene is transmitted; a focus adjustable lens; a second polarizer; and one or more processors. The one or more processors are configured to, during a first period, control the polarization converter to convert a polarization direction of first light of the real scene such that at least a portion of the first light of the real scene is blocked by the second polarizer, and during a second period, control the polarization converter to maintain a polarization direction of second light of the real scene such that at least a portion of the second light of the real scene is blocked by the second polarizer. The second light of the real scene is transmitted through the second polarizer.

Description

[0001] This application is a divisional application of the following application: Application number: 202080038897.0; Application date: June 26, 2020; Invention name: "Electronic device and method for displaying augmented reality" Technical Field

[0002] The present disclosure relates to an electronic device for displaying augmented reality (AR). Background Art

[0003] AR devices that allow users to experience AR include, for example, AR glasses. An image optical system of an AR device includes an image generating device for generating an image and a waveguide for transmitting the generated image to the user's eyes. The image output from the image generating device (e.g., a projector) is emitted to the user's eyes through the waveguide, and he / she observes the image.

[0004] AR devices create a three-dimensional (3D) effect by providing images presented to both eyes with binocular disparity. Although the 3D effect using this binocular disparity can be relatively easily created and experienced by using various methods to implement 3D, when users wear the device used to create the 3D effect for a long time, users experience eye fatigue. Eye fatigue may occur because the vergence distance and focal length of the two eyes are not the same. This inconsistency between the vergence distance and focal length is called vergence accommodation conflict (VAC). Summary of the invention

[0005] Technical issues

[0006] An AR device that can reduce eye fatigue is provided.

[0007] An AR device is provided that can respond to an area of ​​interest to a user when providing a plurality of virtual images.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0009] Technical solutions to the problem

[0010] According to one aspect of the present disclosure, an electronic device for displaying AR includes: an optical engine configured to output light of a virtual image; a first polarizer located in the light path of a real scene; a polarization converter configured to maintain or convert the polarization direction of the light; a waveguide, from which the light of the virtual image is output and through which the light of the real scene is transmitted; a focus-adjustable lens, the focal length of which is adjusted for the light of the virtual image output from the waveguide; a second polarizer located between the waveguide and the focus-adjustable lens or between the polarization converter and the waveguide; and one or more processors, which are configured to: during a first time period, control the polarization converter to convert the polarization direction of first light of the real scene that has passed through the first polarizer, so that at least a portion of the first light of the real scene is blocked by the second polarizer; and during a second time period, control the polarization converter to maintain the polarization direction of second light of the real scene that has passed through the first polarizer, so that the second light of the real scene is transmitted through the second polarizer.

[0011] The polarization converter may include a liquid crystal panel.

[0012] The one or more processors may also be configured to: during a first period, control the optical engine to output light of a virtual image, and during a second period, control the optical engine not to output light of a virtual image.

[0013] The one or more processors may also be configured to: during a first time period, control the focus adjustable lens to have a first focal length for light of the virtual image, and during a second time period, control the focus adjustable lens to have a second focal length, and the second focal length may be different from the first focal length.

[0014] The first focal length may be a focal length for moving an image plane of the virtual image to a depth position of the virtual image.

[0015] The second focal length may be infinite.

[0016] The second focal length may be a focal length for correcting the user's vision.

[0017] The first period and the second period may alternate with each other.

[0018] The one or more processors may also be configured to: during a first period, control the optical engine to output light of multiple virtual image layers with different depths by time division, and control the focus adjustable lens according to different first focal lengths corresponding to the depths of the multiple virtual image layers.

[0019] The electronic device may also include an eye tracker configured to track a user's gaze.

[0020] The one or more processors may also be configured to: during the first period, generate the virtual image such that the virtual image has a depth of a gaze point of the user tracked by the eye tracker.

[0021] The one or more processors may also be configured to generate, during a first time period, multiple virtual image layers having different depths, wherein a frequency of virtual image layers having a depth close to a user's gaze point tracked by an eye tracker is greater than a frequency of virtual image layers having a depth far from the user's gaze point.

[0022] The focus-adjustable lens may be a liquid crystal lens, a liquid lens, or a movable lens.

[0023] The electronic device may be a glasses-type device.

[0024] According to one aspect of the present disclosure, an electronic device for displaying AR includes: a waveguide, from which light of a virtual image is output and through which light of a real scene is transmitted; an optical engine located on a portion of a rear surface of the waveguide; an optical shutter located on a front surface of the waveguide and configured to transmit or block incident light of the real scene under electrical control; a focus-adjustable lens located on another portion of the rear surface of the waveguide and having a focal length adjusted by electrical control; and one or more processors configured to: during a first time period, control the optical shutter to block at least a portion of a first light of the real scene, and during a second time period, control the optical shutter to transmit a second light of the real scene.

[0025] The optical shutter may be a liquid crystal optical shutter or an electrochromic optical shutter.

[0026] According to one aspect of the present disclosure, an electronic device for displaying AR includes: a waveguide; an optical engine located on one side of the waveguide; a polarization converter located on the front surface of the waveguide and configured to maintain or convert the polarization direction of incident light under electrical control; a first polarizer located on the front surface of the polarization converter; a focus-adjustable lens located on the rear surface of the waveguide and having a focal length adjusted by electrical control; a second polarizer located between the waveguide and the focus-adjustable lens or between the polarization converter and the waveguide; and one or more processors, wherein the one or more processors are configured to: during a first time period, control the polarization converter to convert the polarization direction of first light of a real scene that has passed through the first polarizer, so that at least a portion of the first light of the real scene input through the first polarizer is blocked by the second polarizer, and during a second time period, control the polarization converter to maintain the polarization direction of second light of the real scene that has passed through the first polarizer, so that the light of the real scene is transmitted through the second polarizer.

[0027] According to one aspect of the present disclosure, a method for displaying AR includes: during a first time period, outputting light of a virtual image from a waveguide to a focus-adjustable lens; during the first time period, controlling the focus-adjustable lens to have a first focal length; during a second time period, not supplying light of the virtual image to the waveguide, and transmitting light of a real scene through the focus-adjustable lens; and during the second time period, controlling the focus-adjustable lens to have a second focal length different from the first focal length.

[0028] Advantageous Effects of the Invention

[0029] According to the present disclosure, an electronic device and method for displaying AR can reduce eye fatigue.

[0030] According to the present disclosure, an electronic device and method for displaying AR can provide a virtual image responsive to an area of ​​interest to a user by tracking the user's gaze. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0032] Figure 1 is a view showing the appearance of an AR device according to an embodiment;

[0033] Figure 2 yes Figure 1 A floor plan of the AR device;

[0034] Figure 3 yes Figure 1 Block diagram of an AR device;

[0035] Figure 4 is a view showing the arrangement of an optical engine and an optical assembly according to an embodiment;

[0036] Figure 5 is a view showing an optical engine according to an embodiment;

[0037] Figure 6 is a view showing a waveguide according to an embodiment;

[0038] Figure 7 is a view showing a first polarizer, a polarization converter, and a second polarizer according to an embodiment;

[0039] Figure 8 is a view showing a focus adjustable lens according to an embodiment;

[0040] Fig. 9 is a view used to describe the divergent operation of a focus adjustable lens;

[0041] Fig.10 is a view used to describe the converging operation of a focus adjustable lens;

[0042] Fig.11 is a flowchart for describing the operation of the AR device according to the embodiment;

[0043] Fig.12 is a view for describing an operation of sequentially generating an AR image by an AR device according to an embodiment;

[0044] Fig.13 is a view for describing an operation of an AR device during a first period according to an embodiment;

[0045] Fig.14 is a view for describing an operation of the AR device during a second period according to an embodiment;

[0046] Fig.15 is a diagram for describing a vergence-accommodation conflict in an AR device of a comparative example;

[0047] Fig.16 is a view for describing an operation of a focus adjustable lens at a depth of a virtual image in an AR device according to an embodiment;

[0048] Fig.17 is a view for describing a situation in which a user sees a real scene through an AR device according to an embodiment;

[0049] Fig.18 is a view for describing a principle based on which the AR device according to the embodiment adjusts the brightness of a virtual image and the brightness of a real scene;

[0050] Fig.19 is a view showing an AR image having multiple depths in an AR device according to an embodiment;

[0051] Fig. 20 is a view for describing an operation of sequentially generating an AR image having multiple depths in an AR device according to an embodiment;

[0052] Fig.21 is a view showing the arrangement of optical components in an AR device according to an embodiment;

[0053] Fig. 22 is a view showing the arrangement of optical components in an AR device according to an embodiment;

[0054] Fig.23 is a view showing a configuration of an optical shutter according to an embodiment;

[0055] Fig.24 is a view showing a configuration of an optical shutter according to an embodiment;

[0056] Fig.25is a view showing the operation of the focus adjustable lens according to the embodiment;

[0057] Fig.26 is a view showing another operation of the focus adjustable lens according to the embodiment;

[0058] Fig. 27 is a block diagram of an AR device according to an embodiment;

[0059] Fig.28 is a view showing an eye tracker according to an embodiment;

[0060] Fig.29 is a view of a 3D eye model showing the direction in which the user is looking;

[0061] Fig.30 It is a view used to describe the relationship between the gaze point and gaze angle of the user's left eye and right eye;

[0062] Fig.31 is a view used to describe the relationship between the gaze point and the gaze angle in the upward gaze direction;

[0063] Fig.32 is a flowchart for describing the operation of the AR device according to the embodiment;

[0064] Fig.33 is a view showing a case where a depth of interest exists in an AR image having three different depths;

[0065] Fig.34 is a flowchart for describing the operation of the AR device according to the embodiment;

[0066] Fig.35 is a diagram for describing a case where an AR image having three different depths is displayed without reflecting a depth of interest according to an embodiment;

[0067] Fig.36 is a diagram for describing a case where an AR image reflecting a depth of interest among three different depths is displayed according to an embodiment;

[0068] Fig.37 is a diagram for describing a case where an AR image reflecting a depth of interest among three different depths is displayed according to an embodiment;

[0069] Fig.38 is a view showing a case where a depth of interest exists in an AR image having six different depths;

[0070] Fig.39 is a diagram for describing a case where an AR image having six different depths is displayed without reflecting a depth of interest according to an embodiment;

[0071] Fig.40 is a diagram for describing a case where an AR image reflecting a depth of interest among six different depths is displayed according to an embodiment;

[0072] Fig.41 is a diagram for describing a case where an AR image reflecting a depth of interest among six different depths is displayed according to an embodiment;

[0073] Fig.42 is a view showing an eye tracker according to an embodiment;

[0074] Fig.43 is a block diagram of an AR device according to an embodiment; and

[0075] Fig.44 is a view showing a case where a virtual image having a depth matching a real-world object is displayed. DETAILED DESCRIPTION

[0076] Embodiments will now be described more fully with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements, and the size of the elements may be exaggerated for clarity. The embodiments of the present disclosure may have different forms and should not be interpreted as being limited to the embodiments set forth herein.

[0077] Although the terms used herein are selected from currently widely used general terms in consideration of their functions in the present disclosure, these terms may change according to the intention of a person of ordinary skill in the art, precedents, or the emergence of new technologies. In addition, in certain cases, these terms are freely selected by the applicant of the present disclosure, and the meanings of these terms will be described in detail in the corresponding parts of the specific embodiments. Therefore, the terms used in the present disclosure are not merely designated terms, but are defined based on the meanings and contents of the terms in the entire disclosure.

[0078] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0079] As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Throughout this application, when a part "includes" an element, it should be understood that, as long as there is no special contrary description, the part also includes other elements, rather than excluding other elements.

[0080] Furthermore, terms such as “unit”, “module” and the like used in the present disclosure indicate a unit that processes at least one function or motion, and the unit may be implemented by hardware or software, or by a combination of hardware and software.

[0081] Depending on the circumstances, the expression "configured to (or set to)" used herein may be replaced by, for example, "suitable for", "capable of", "designed to", "suitable for", "made to", or "capable of". The expression "configured to (or set to)" does not necessarily mean "specially designed to" at the hardware level. On the contrary, in some cases, "a system configured to..." may mean that the system, together with other devices or components, "is capable of...". For example, "a processor configured to (or set to) perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) that can perform the corresponding operations by running one or more software programs stored in a memory.

[0082] The term “augmented reality (AR)” as used herein is a technology that overlays virtual images on a real-world physical environment space or real-world objects as a single image.

[0083] In the present disclosure, examples of “AR devices” capable of creating AR include AR glasses similar to glasses worn on a user, head-mounted display devices, and AR helmets. Such AR devices are very useful in daily life, such as information retrieval, route guidance, and camera shooting. In addition, AR glasses similar to glasses can be worn as a fashion item and can be used for both indoor and outdoor activities.

[0084] The term "real scene" as used herein refers to a scene of the real world seen by an observer or user through an AR device, and may include real-world objects. The term "virtual image" refers to an image generated by an optical engine. Virtual images may include both static images and dynamic images. A virtual image may be an image overlaid on a real scene to show information about real-world objects in the real scene, information about the operation of an AR device, a control menu, and the like.

[0085] The term "focus adjustable lens" as used herein refers to a lens that can change the focal length. A liquid crystal (LC) lens, a liquid lens, a movable lens, or other suitable focus adjustable optical system can be used as the focus adjustable lens. As described below, when a user sees a virtual image, he / she can adjust the focus according to the depth of the virtual image through the focus adjustable lens.

[0086] The term "eye tracker" used herein refers to a device for detecting information about the gaze direction of a user's gaze, and the gaze direction can be calculated by detecting an image of the pupil or detecting the amount of light or the direction of illumination light such as near-infrared rays reflected from the cornea.

[0087] The term "focus" as used herein refers to the point where a ray of light parallel to the optical axis of the lens passes through the lens (or optical system) and then the straight line extending from the ray intersects the optical axis. The distance in air from the principal plane of the lens (or optical system) to the focus is called the focal length.

[0088] The term "depth of a virtual image" as used herein refers to the distance or depth at which a user perceives a virtual image in space when the user sees the virtual image. A 3D image using binocular parallax can generate a left-eye virtual image and a right-eye virtual image with different viewpoints, and in this case, the different viewpoints can be the viewpoint seen by the user's left eye and the viewpoint seen by the user's right eye. Therefore, the viewpoint of the virtual image in a 3D image using binocular parallax can be a distance calculated based on the parallax between the viewpoint seen by the left eye and the viewpoint seen by the right eye (i.e., binocular parallax). A multi-depth virtual image may include virtual image layers with different depths (i.e., different binocular parallaxes).

[0089] The term "gaze direction" used herein refers to the direction in which the user is gazing, and the term "gaze" refers to an imaginary line from the user's pupil along the gaze direction. Mainly, the gaze direction is calculated from information obtained by an eye tracker, and the gaze is estimated.

[0090] The term "gaze point" used herein refers to a point at which a user gazes, and may be calculated as a point at which the gazes of the user's two eyes intersect. Since a user viewing a 3D image using binocular parallax perceives a 3D image due to binocular parallax, the gaze point obtained by the convergence angle of the user's two eyes may be a point at which the user perceives an object (virtual image layer) (i.e., the depth of the virtual image).

[0091] In the real world, the visual convergence distance to the point where the gazes of both eyes intersect and the focusing distance (convergence distance) changed by adjusting the thickness of the eye lens match each other, but in a virtual image (i.e., a 3D image using binocular parallax), the visual convergence distance and the focusing distance of both eyes do not match each other. Therefore, people who watch virtual reality (VR) or AR images using virtual images for a long time or people who are sensitive to VR or AR images may experience dizziness and eye fatigue due to the difference between the visual convergence distance and the focusing distance of the human eye.

[0092] Figure 1 : is a view showing the appearance of the AR device 100 according to the embodiment. Figure 2 yes Figure 1 A plan view of an AR device 100 is shown. Figure 3 yes Figure 1 1 is a block diagram of an AR device 100 . Figure 4 1 is a diagram illustrating an arrangement of an optical engine 120 and an optical assembly 130 (eg, one or more optional elements) according to an embodiment.

[0093] refer to Figure 1 and Figure 2 , the AR device 100 of this embodiment is an AR glasses similar to glasses worn on a user, and includes a glasses body 110 .

[0094] The eyeglass body 110 may include, for example, a frame 111 and temples 119. The frame 111 on which the left lens 101L and the right lens 101R are located may have two rim shapes connected to each other via a bridge 112. The left lens 101L and the right lens 101R are examples and may or may not have a refractive power (ability). Alternatively, the left lens 101L and the right lens 101R may be formed integrally with each other, and in this case, the rims of the frame 111 and the bridge 112 are not separated from each other. The left lens 101L and the right lens 101R may be omitted.

[0095] The temples 119 are respectively connected to both ends 113 of the frame 111 and extend in one direction. The both ends 113 of the frame 111 and the temples 119 may be connected to each other by hinges 115. The hinges 115 are examples, and other suitable members for connecting the both ends 113 of the frame 111 and the temples 119 may be used as the hinges 115. As another example, the both ends 113 of the frame 111 and the temples 119 may be integrally (continuously) connected to each other.

[0096] The optical engine 120 , the plurality of optical components 130 , and the electronic components 190 are located on the eyeglass body 110 .

[0097] The optical engine 120 configured to generate light of a virtual image may be an optical engine of a projector including an image panel, an illumination optical system, and a projection optical system. The optical engine 120 may include a left-eye optical engine 120L and a right-eye optical engine 120R. The left-eye optical engine 120L and the right-eye optical engine 120R may be located in both ends 113 of the frame 111. As another example, the left-eye optical engine 120L and the right-eye optical engine 120R may be located in the left temple 119L and the right temple 119R, respectively.

[0098] Further references Figure 4, the optical assembly 130 configured to transmit the light of the virtual image generated by the optical engine 120 and the light of the real scene to the user's pupil may include a waveguide 140, a polarization converter 150, a first polarizer 158 and a second polarizer 159, and a focus adjustable lens 160. The optical assembly 130 may include a left-eye optical assembly 130L and a right-eye optical assembly 130R. The left-eye optical assembly 130L and the right-eye optical assembly 130R may be attached to the left lens 101L and the right lens 101R, respectively. Alternatively, the left-eye optical assembly 130L and the right-eye optical assembly 130R may be mounted on the frame 111 separately from the left lens 101L and the right lens 101R.

[0099] refer to Figure 4 , the optical engine 120 may be located on the rear surface of a portion of the waveguide 140. The rear surface of the waveguide 140 is the surface facing the user's eyes when the user wears the AR device 100. The optical engine 120 may be positioned to output light of a virtual image along the z-axis direction on a portion of the waveguide 140. The first polarizer 158, the polarization converter 150, the second polarizer 159, and the focus adjustable lens 160 are located on another portion of the waveguide 140. The polarization converter 150 and the first polarizer 158 may be located on the front surface 144 of the waveguide 140, while the second polarizer 159 and the focus adjustable lens 160 may be located on the rear surface 146 of the waveguide 140. That is, the first polarizer 158, the polarization converter 150, the waveguide 140, the second polarizer 159, and the focus adjustable lens 160 may be arranged in the above order. The light of the real scene is incident on another portion of the waveguide 140 along the -z-axis direction.

[0100] Further references Figure 3 The electronic component 190 may include a processor 191, an interface 192, and a memory 193, and may be mounted on the temple 119 of the eyeglass body 110 (refer to Figure 1 ) or Framework 111 (reference Figure 1 ) on; may be distributed on a plurality of parts; or may be mounted on a printed circuit board (PCB), a flexible PCB (FPCB), etc. A polarization converter drive driver circuit for driving the polarization converter 150 and a lens drive driver circuit for driving the focus adjustable lens 160 may be located adjacent to the polarization converter 150 and the focus adjustable lens 160. The polarization converter drive driver circuit and the lens drive driver circuit may be integrally formed with each other. As another example, the polarization converter drive driver circuit and the lens drive driver circuit may be located in whole or in part on a main board mounted on, for example, the temple 119.

[0101] The processor 191 can control the overall operation of the AR device 100 including the optical engine 120, the polarization converter 150, and the focus adjustable lens 160 by driving an operating system or an application, and can perform operations and processing of various data including image data. For example, the processor 191 can process image data including a left-eye virtual image and a right-eye virtual image presented with binocular parallax. The processor 191 may include at least one hardware device, such as but not limited to a central processing unit, a microprocessor, a graphics processing unit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), or a field programmable gate array (FPGA). Although in Figure 3 Only one processor 191 is shown in FIG. 1 , but the embodiment is not limited thereto, and more than one processor may be provided to control the operation of the AR device 100 .

[0102] The interface 192 through which external data or operation commands are input / output may include a user interface such as a touch pad, a controller, or an operation button that can be operated by a user. In an embodiment, the interface 192 may include a wired communication module such as a universal serial bus (USB) module or a wireless communication module such as a Bluetooth module, and may receive data of a virtual image or user's operation information transmitted from an interface included in an external device through the wired communication module or the wireless communication module.

[0103] The memory 193 may include an internal memory such as a volatile memory or a nonvolatile memory. The memory 193 may store various data for driving and controlling the AR device 100 under the control of the processor 191, a program or an application, and signals or data of input / output virtual images.

[0104] The optical engine 120 configured to receive the image data generated by the processor 191 and generate light of a virtual image includes a left-eye optical engine 120L and a right-eye optical engine 120R. Each of the left-eye optical engine 120L and the right-eye optical engine 120R includes a light source that outputs light and an image panel that forms a virtual image by using the light output from the light source, and has the same function as a small projector. The light source may be, for example, a light emitting diode (LED), and the image panel may be, for example, a digital micromirror device (DMD).

[0105] The polarization converter 150 configured to convert the polarization direction of the light of the real scene according to the control signal of the processor 191 includes a left-eye polarization converter 150L and a right-eye polarization converter 150R. Each of the left-eye polarization converter 150L and the right-eye polarization converter 150R may include an optically active material whose optical properties change according to the applied electric field, and may be, for example, a liquid crystal panel. As described below, the polarization converter 150 transmits or blocks the light of the real scene together with the first polarizer 158 and the second polarizer 159.

[0106] The focus adjustable lens 160 whose focal length is changed by a control signal of the processor 191 includes a left eye focus adjustable lens 160L and a right eye focus adjustable lens 160R. Each of the left eye focus adjustable lens 160L and the right eye focus adjustable lens 160R may be configured such that a refractive index or a refractive power varies according to an applied electric field.

[0107] Next, the optical engine 120 and the optical assembly 130 configured to transmit the light of the virtual image generated by the optical engine 120 and the light of the external scene to the user's pupil will be described in more detail. The optical engine 120 and the optical assembly 130 are provided as components for the left eye and the right eye of the user. It will be understood by those skilled in the art that because the components for the left eye and the right eye are symmetrical, although one of the components (e.g., the component for the left eye) is described, the description may be applicable to the other component (e.g., the component for the right eye).

[0108] Figure 5 1 is a diagram showing an optical engine 120 according to an embodiment. The optical engine 120 includes an illumination optical system 121, an image panel 123 that forms a virtual image by light irradiated by the illumination optical system 121, and a projection optical system 124 that projects the light of the virtual image formed on the image panel 123. Figure 5 The optical engine 120 includes a beam splitter 122 that separates light from the illumination optical system 121 and light reflected by the image panel 123. When the optical engine 120 uses the image panel 123 as a transmissive image panel, the beam splitter 122 may be omitted. The optical engine 120 outputs polarized light or non-polarized light according to the method of the illumination optical system 121 or the image panel 123. For example, when the image panel 123 is a liquid crystal on silicon (LCoS) panel or other liquid crystal image panel, when a polarization beam splitter is used to separate / combine light beams, or when a highly polarized light source is used, the optical engine 120 may output linearly polarized light. As another example, when the image panel 123 is a DMD panel, the optical engine 120 may output non-polarized light. Although the optical engine 120 outputs polarized light by using Figure 5The image panel 123 in the optical engine 120 projects light of a virtual image, but the present disclosure is not limited thereto. In an embodiment, the optical engine 120 may form an image by using a scanning method.

[0109] Figure 6 is a view showing a waveguide 140 according to an embodiment.

[0110] refer to Figure 6 , the waveguide 140 has a single-layer or multi-layer structure formed of a transparent material in which light is reflected and propagated. The waveguide 140 may have a flat plate shape or a curved plate shape. The waveguide 140 includes a first area 141 facing the optical engine 120 and receiving the projection light of the virtual image, a second area 142 in which the light of the virtual image incident on the first area 141 propagates, and a third area 143 that outputs the light of the virtual image propagated in the second area 142. The first polarizer 158 and the polarization converter 150 may contact the front surface of the third area 143 of the waveguide 140, or may be separated from the front surface of the third area 143 of the waveguide 140 by a predetermined interval, and the second polarizer 159 and the focus adjustable lens 160 may contact the third area 143 of the rear surface of the waveguide 140, or may be separated from the third area 143 of the rear surface of the waveguide 140 by a predetermined interval. In an embodiment, the first polarizer 158 and the polarization converter 150 may be positioned to completely cover a front surface portion of the third region 143 facing the waveguide 140, and the second polarizer 159 and the focus adjustable lens 160 may be positioned to completely cover the third region 143 of the rear surface of the waveguide 140. In an embodiment, the first polarizer 158 and the polarization converter 150 may be positioned to cover a portion of a front surface portion of the third region 143 facing the waveguide 140, and the second polarizer 159 and the focus adjustable lens 160 may be positioned to cover a portion of the third region 143 of the rear surface of the waveguide 140.

[0111] The waveguide 140 is mounted on the frame 111 (see Figure 1 ), so that when the user wears the AR device 100, the third area 143 is located in front of the user's pupil. Because the waveguide 140 is formed of a transparent material, the user can see the virtual image as well as the real scene through the AR device 100, so the AR device 100 can create AR. The term "transparent material" refers to a material through which visible light can pass, and the transparency of the transparent material may not be 100%, and the transparent material may have a predetermined color.

[0112] In an embodiment, a diffraction grating may be formed to couple light incident on the first region 141 of the rear surface of the waveguide 140 and transmit the coupled light in a first direction (e.g., X direction). The optical engine 120 may be positioned so that the emitted light is perpendicular to the first region 141 or is obliquely incident at a predetermined angle. The arrangement direction of the optical engine 120 may vary according to the pattern of the diffraction grating of the first region 141.

[0113] The second region 142 may be located in a first direction (e.g., Y direction) relative to the first region 141. A diffraction grating is formed on the second region 142 so that at least a portion of the light received from the first region 141 propagates in a second direction (e.g., -X direction). When the waveguide 140 has a single-layer structure, the diffraction grating of the second region 142 may be formed on the same surface as the diffraction grating of the first region 141 (i.e., the rear surface 146 of the waveguide 140) or on a surface opposite to the diffraction grating of the first region 141 (i.e., the front surface 144 of the waveguide 140). Alternatively, when the waveguide 140 has a multi-layer structure, the diffraction grating of the second region 142 may be formed on a layer different from the diffraction grating of the first region 141 on which the diffraction grating is formed. Light incident on the first region 141 is reflected and propagated between the front and rear surfaces of the waveguide 140. Although the second region 142 is a single region in the present embodiment, the second region 142 may include a plurality of regions. When the waveguide 140 has a multi-layer structure, the second region 142 may include a plurality of regions formed on different layers.

[0114] The third region 143 may be located in the second direction (e.g., -X direction) relative to the second region 142. A diffraction grating is formed in the third region 143 so that at least a portion of the light propagating from the second region 142 is output in a direction perpendicular to the waveguide 140 (e.g., -Z direction) or in an inclined direction. When the waveguide 140 has a single-layer structure, the diffraction grating of the third region 143 may be formed on the same surface as the diffraction gratings of the first region 141 and the second region 142. Alternatively, when the waveguide 140 has a multi-layer structure, the diffraction grating of the third region 143 may be formed on a layer different from the diffraction grating of the second region 142 formed thereon, and may be formed on the same layer as the diffraction grating of the first region 141, or may be formed on a layer different from the diffraction grating of the first region 141 formed thereon. Although in the present embodiment, the third region 143 is located in the second direction (e.g., -X direction) relative to the second region 142, the present disclosure is not limited thereto. The third region 143 may be located in a direction (eg, +X direction) opposite to the second direction with respect to the second region 142 .

[0115] The diffraction grating of the first region 141, the diffraction grating of the second region 142, and the diffraction grating of the third region 143 may have different patterns. When the waveguide 140 has a multi-layer structure, the second region 142 may partially overlap the first region 141 and the third region 143.

[0116] Figure 7 1 is a view showing a first polarizer 158 , a polarization converter 150 , and a second polarizer 159 according to an embodiment.

[0117] refer to Figure 7 , the polarization converter 150 may be a liquid crystal panel including a liquid crystal layer 151 in which liquid crystals are encapsulated and electrodes 152 and 153 located on both surfaces of the liquid crystal layer 151. The electrodes 152 and 153 may be formed of a transparent conductive material such as indium tin oxide (ITO). Regarding the liquid crystal panel, various methods such as twisted nematic (TN), vertical alignment (VA), and plane to line switching (PLS) according to the arrangement of the electrodes 152 and 153 and the arrangement method of the liquid crystal are known to those skilled in the art. For example, in the TN method, when no electric field is applied, the liquid crystal molecules are twisted by 90°, and when an electric field is applied, the liquid crystal molecules rotate in the direction of the electric field to untwist. Therefore, when a voltage is not applied to the electrodes 152 and 153 (ie, when the electrodes 152 and 153 are in an off state), the polarization direction of the linearly polarized light incident on the liquid crystal layer 151 using the TN method rotates, so that the S polarization becomes the P polarization and the P polarization becomes the S polarization. P polarization in which the vibration direction of the electric field of light (ie, electromagnetic wave) is parallel to the incident plane is called horizontal polarization, and S polarization in which the vibration direction of the electric field is perpendicular to the incident plane is called vertical polarization. When a voltage is applied to the electrodes 152 and 153 (ie, the electrodes 152 and 153 are in an on state), linearly polarized light incident on the liquid crystal layer 151 using the TN method maintains polarization.

[0118] The first polarizer 158 and the second polarizer 159 are respectively located on the front surface 700 and the rear surface 702 of the polarization converter 150. Each of the first polarizer 158 and the second polarizer 159 allows only a linear polarization component in one direction of the components of the incident light to pass through. When the focus adjustable lens 160 includes a polarizer, the second polarizer 159 may be omitted.

[0119] The arrangement direction of the first polarizer 158 and the second polarizer 159 can be changed according to the method of the polarization converter 150 or the on / off operation mode of the polarization converter 150. In the present embodiment, based on the control of the processor 191, the polarization converter 150 is in the off mode, and the second polarizer 159 transmits the light of the real scene. When the polarization converter 150 is in the on mode based on the control of the processor 191, the second polarizer 159 blocks the light of the real scene. The case where light is transmitted through the second polarizer 159 includes the case where only part of the light is transmitted or only the polarization component of the light is transmitted. In an embodiment, when the polarization converter 150 is a liquid crystal panel using the TN method, the first polarizer 158 and the second polarizer 159 can be positioned so that the polarization directions are perpendicular to each other. For example, the first polarizer 158 can be positioned so that only the light of the P polarization component passes through, and the second polarizer 159 can be positioned so that only the light of the S polarization component passes through. When voltage is not applied to the liquid crystal layer 151 of the polarization converter 150, light that has passed through the first polarizer 158 and has a first polarization direction (e.g., P polarization) has a second polarization direction (e.g., S polarization) perpendicular to the first polarization direction due to the polarization converter 150, and thus passes through the second polarizer 159. When voltage is applied to the liquid crystal layer 151 of the polarization converter 150, light that has passed through the first polarizer 158 and has a first polarization direction (e.g., P polarized light) maintains the first polarization direction due to the polarization converter 150, and thus is blocked by the second polarizer 159. In this way, the polarization converter 150, together with the first polarizer 158 and the second polarizer 159, functions as an optical shutter of light of a real scene.

[0120] In an embodiment, each of the electrodes 152 and 153 of the polarization converter 150 may be provided as a single electrode on the entire portion of the liquid crystal layer 151, corresponding to the first polarizer 158. In an embodiment, because one of the electrodes 152 and 153 of the polarization converter 150 is provided as a pixelated electrode and is independently controllable, only the light of the real scene of the portion corresponding to the virtual object in the real scene is selectively blocked.

[0121] Figure 8 is a view showing a configuration of a focus adjustable lens 160 according to an embodiment.

[0122] refer to Figure 8 The focus-adjustable lens 160 includes an electro-optic layer 161 , a first electrode array 162 located on a first surface 704 of the electro-optic layer 161 , and a second electrode 163 located on a second surface 706 of the electro-optic layer 161 .

[0123] Electro-optic layer 161 may include an electro-optic material whose refractive index varies according to an applied electric field, ie, a material whose local effective refractive index varies as a function of a voltage applied to the entire material. Electro-optic layer 161 may have a single layer structure or a multilayer structure separated by a plurality of insulating layers.

[0124] In an embodiment, liquid crystal in which liquid crystal molecules are rotated by an applied electric field to change the birefringence axis and the effective refractive index may be used as the electro-optic material. In an embodiment, other materials having electro-optic properties similar to the polymer gel may be used for the electro-optic layer 161 .

[0125] The electro-optical material may have a polarization property in which the refractive index varies according to the single-layer / multi-layer structure or orientation method of the liquid crystal molecules, according to the polarization direction, or may have a non-polarization property. Even when the electro-optical material of the electro-optical layer 161 has a polarization property (for example, when the electro-optical material is a liquid crystal material), overlapping focus or blurred images in the focus adjustable lens 160 may be prevented by limiting the polarization direction of the incident light using the second polarizer 159 (refer to Figure 7 ).

[0126] The first electrode array 162 and the second electrode 163 can be formed of a transparent conductive material such as ITO. The first electrode array 162 can be an electrode array operated individually or in groups. The first electrode array 162 is patterned into a series of concentric arrays or pixel arrays and is used to apply a desired voltage so that the effective refractive index is adjusted to a locally desired value. The electro-optical layer 161 can provide a phase modulation profile with a desired focal length by locally adjusting the effective refractive index. The second electrode 163 can be used as a reference electrode relative to the first electrode array 162. If necessary, the second electrode 163 can be implemented as an electrode array. The positions of the first electrode array 162 and the second electrode 163 can be changed with each other.

[0127] In an embodiment, the second polarizer 159 and the focus adjustable lens 160 may be spaced apart from each other or may contact each other. In an embodiment, the second polarizer 159 and the focus adjustable lens 160 may be integrally formed with each other, and the second polarizer 159 may be a component of the focus adjustable lens 160.

[0128] In an embodiment, although the focus adjustable lens 160 may have a refractive power of 0 (i.e., infinite focal length) in the off mode, the present disclosure is not limited thereto, and the focus adjustable lens 160 may have a refractive power even in the off mode according to a single-layer / multi-layer structure or shape of the electro-optical layer 161. The following will describe a case where the refractive power of the focus adjustable lens 160 is 0 in the off mode.

[0129] Fig. 9 is a view used to describe the divergent operation of a focus-adjustable lens. Fig.10 This is a diagram used to describe the converging operation of a focus adjustable lens. Fig. 9 and Fig.10 When a voltage is applied to the electro-optic layer 161 of the focus adjustable lens 160, the effective refractive index can be locally changed, and thus the light path passing through the focus adjustable lens 160 can be changed, thereby changing the visual convergence. Visual convergence is an indicator indicating the degree to which light transmitted through the focus adjustable lens 160 converges or diverges. The visual convergence can be adjusted according to the refractive power of the focus adjustable lens 160. Fig. 9 In the embodiment of the invention, light passing through the portion of the electro-optical layer 161 whose optical properties (for example, in the case of a liquid crystal layer, the arrangement angle of the liquid crystal molecules 165) are changed can form positive visual convergence, so the focus adjustable lens 160 can perform the same function as a convex lens. Once the positive visual convergence is formed, the focal length can be shortened. Fig.10 In the embodiment of the present invention, light passing through the portion of the electro-optical layer 161 whose optical properties (for example, in the case of the liquid crystal layer, the arrangement angle of the liquid crystal molecules 165) are changed can form a negative visual convergence, so the focus adjustable lens 160 can perform the same function as a concave lens. When the negative visual convergence is formed, the focal length can be extended.

[0130] Will refer to Figures 11 to 14 The operation of the AR device 100 of the embodiment will be described.

[0131] Fig.11 is a flowchart for describing the operation of the AR device 100 according to the embodiment. Fig.12 1 is a view for describing an operation of an AR image sequentially generated by the AR device 100 according to an embodiment. Fig.13 is a view for describing an operation of the AR device 100 during a first period according to an embodiment. Fig.14 is a view for describing the operation of the AR device 100 during the second period according to the embodiment.

[0132] refer to Figure 3 and Figures 11 to 14 In operation S210, the processor 191 according to the embodiment generates image data including a virtual image based on the data received through the interface 192 or the data stored in the memory 193. The virtual image includes a left-eye virtual image and a right-eye virtual image with binocular parallax. The depth of the virtual image is provided according to the binocular parallax. The left-eye virtual image and the right-eye virtual image with binocular parallax may be provided from the original data or may be generated by the processor 191 according to the setting of the AR device 100.

[0133] The image data including the virtual image may be image data that is time-divided by temporally alternating a first period for outputting the virtual image and a second period for not outputting the virtual image. Because an image is generally generated and displayed in units of frames, the image data including the virtual image may have a structure in which one or more consecutive first frames ( Fig.12 and one or more consecutive second frames of a second period after the first period ( Fig.12 The first frame (virtual) can contain a virtual image.

[0134] In an embodiment, the second frame (real) is in a blank state like null data, and the processor 191 may control the optical engine 120 not to output light in the second frame (real).

[0135] In an embodiment, the second frame (real) may include a transparent image, and the processor 191 may control the optical engine 120 to output light of the transparent image in the second frame (real).

[0136] In an embodiment, the second frame (real) may include a virtual image, and when the brightness is zero or there is separate light source on / off control information, the processor 191 may control the light source of the optical engine 120 to be turned on / off in the second frame (real), thereby controlling the optical engine 120 not to output light of the virtual image.

[0137] In an embodiment, the processor 191 may maintain the second period without providing the virtual image, or may maintain the first period by continuously providing the virtual image.

[0138] In operation S220, during a first period, the processor 191 controls the optical engine 120 to output the light Lv of the virtual image, controls the polarization converter 150 to turn on so that the light Lo of the real scene does not reach the focus adjustable lens 160, and controls the focus adjustable lens 160 to turn on to have a first focal length.

[0139] like Fig.13As shown, the light Lo of the real scene may be non-polarized light, and only the light incident on the first polarizer 158 and having the first polarization component (e.g., the P polarization component) passes through. The light Lo of the real scene of the first polarization component is incident on the polarization converter 150. When the processor 191 controls the polarization converter 150 to turn on, the liquid crystal may be aligned according to the applied electric field, and the light of the first polarization component passes through the polarization converter 150 while maintaining the polarization direction, and reaches the second polarizer 159 through the waveguide 140. However, because the second polarization direction (e.g., S polarization) of the second polarizer 159 is perpendicular to the first polarization direction of the first polarizer 158, the light of the first polarization component of the real scene is blocked by the second polarizer 159, thereby preventing the user from seeing the real scene.

[0140] When the processor 191 controls the optical engine 120 to output the light Lv of the virtual image, the light Lv of the virtual image output from the optical engine 120 is transmitted to the second polarizer 159 through the waveguide 140. Only the light of the second polarization component (e.g., S polarization component) of the light Lv of the virtual image passes through the second polarizer 159. In an embodiment, when the optical engine 120 outputs unpolarized light, the light of the polarization component (i.e., the second polarization) in the same direction as the second polarization direction (e.g., S polarization) of the second polarizer 159 can reach the pupil of the user, thereby enabling the user to see the virtual image. In an embodiment, when the optical engine 120 outputs light of a specific polarization, the optical engine 120 can make the light Lv have the same polarization direction as the second polarization direction (e.g., S polarization) of the second polarizer 159, thereby eliminating or reducing light loss.

[0141] The processor 191 can control the focus adjustable lens 160 to have a first focal length according to the depth of the virtual image, so that the user can see the virtual image according to the depth. That is, when the user sees the left eye virtual image and the right eye virtual image, due to the binocular parallax θ of the left eye virtual image and the right eye virtual image V The user perceives the depth of the virtual image, and the processor 191 controls the focus adjustable lens 160 according to the first focal length so that the eye lens of the user's eyeball has a focal length f2 corresponding to the depth of the virtual image (see Fig.16 ). In this case, the first focal length refers to the focal length of the focus adjustable lens 160 for adjusting the visual convergence of the focus adjustable lens 160 so that the focal length of the user's eye lens corresponds to the depth of the virtual image.

[0142] The second period is after the first period. In operation S230, during the second period, the processor 191 controls the optical engine 120 not to output the light Lv of the virtual image, controls the polarization converter 150 to be turned off so that the light Lo of the real scene reaches the focus adjustable lens 160, and controls the focus adjustable lens 160 to be turned off to have a second focal length.

[0143] like Fig.14 As shown, the light Lo of the real scene may be non-polarized light, and light incident on the first polarizer 158 and having a first polarization component (e.g., a P polarization component) passes through. The light Lo of the real scene of the first polarization component is incident on the polarization converter 150. When the processor 191 controls the polarization converter 150 to be turned off, the liquid crystal in the polarization converter 150 returns to the original orientation state, and the polarization direction of the light Lo of the real scene rotates 90° to have a second polarization direction (e.g., S polarization). Therefore, the light Lo of the real scene passing through the polarization converter 150 passes through the waveguide 140 in the second polarization state and reaches the second polarizer 159. However, because the second polarization direction of the second polarizer 159 is perpendicular to the first polarization direction of the first polarizer 158, the light Lo of the second polarization component of the real scene passes through the second polarizer 159 and is incident on the focus adjustable lens 160. When the processor 191 controls the focus adjustable lens 160 to be turned off, the focus adjustable lens 160 may not have a refractive power, and the second focal length is infinite. That is, when the focus adjustable lens 160 is controlled to be closed, the visual convergence of the focus adjustable lens 160 will not converge or diverge. Therefore, the light of the second polarization component (i.e., the light of the real scene) incident on the focus adjustable lens 160 controlled to be closed passes through without changing the visual convergence and propagates to the user's pupil. Because the visual convergence is not changed by the light of the real scene, the user can see the real scene itself. For ease of explanation, although the focus adjustable lens 160 is controlled to be closed when the real scene is seen, the present disclosure is not limited to this. For example, when the user suffers from myopia or hyperopia and needs a correction lens, the focus adjustable lens 160 can be controlled to have a correction focal length (second focal length) and can be used as a correction lens.

[0144] The user can see the virtual image through the AR device 100 during the first period, and can see the real scene through the AR device 100 during the second period. The first period and the second period can continue alternately. Although the first period first exists and then the second period exists in the present embodiment, the present disclosure is not limited to this. In an embodiment, the second period may exist first and then the first period may exist. The first period and the second period may be the same time interval, but the present disclosure is not limited to this. The first period and the second period may be different time intervals. In addition, the time interval of the second period and the time interval of the first period may vary according to the situation.

[0145] A virtual image may be an image overlaid on a real scene and showing information about real-world objects in the real scene, information about the operation of an AR device, a control menu, and the like. A virtual image may have depth. When a left-eye virtual image and a right-eye virtual image are generated to have binocular parallax, the depth of the virtual image may be a distance calculated based on the binocular parallax. For example, a virtual image may have a depth corresponding to the distance of a real-world object covered by the virtual image, so that when a user looks at a real-world object, the virtual image (i.e., the virtual object) is located at or near the position of the real-world object. Of course, the depth of the virtual image may be set without regard to real-world objects.

[0146] During the second period, the processor 191 controls the optical engine 120 not to output the light of the virtual image, controls the polarization converter 150 to transmit the light of the real scene, and controls the focus adjustable lens 160 to have a second focal length. The first focal length and the second focal length may be different values. The second focal length may be infinite (i.e., the refractive power of the focus adjustable lens 160 may be 0). When the user suffers from myopia or hyperopia and needs vision correction, the second focal length may be the focal length of the correction lens.

[0147] Will refer to Figures 15 to 17 The focus adjustment function of the focus adjustable lens 160 in the AR device 100 of this embodiment is described.

[0148] Fig.15 is a view for describing the visual convergence-accommodation conflict in the AR device of the comparative example. Fig.16 is a view for describing an operation of a focus adjustable lens at a depth of a virtual image in an AR device according to an embodiment. Fig.17 is a view for describing a situation in which a user sees a real scene through the AR device according to the embodiment.

[0149] refer to Fig.15 In the AR device of the comparative example without the focus adjustable lens 160, when the user sees the left-eye virtual image and the right-eye virtual image, due to the binocular parallax θ of the left-eye virtual image and the right-eye virtual image V , the user perceives the virtual image I V Depth D V That is, the virtual image I V Depth D V When the user sees the virtual image I V When the user perceives the virtual image I in the space V The distance or depth of the virtual image I V Depth D V According to the binocular disparity θ VBecause the left-eye virtual image and the right-eye virtual image are output from the third region 143 of the waveguide 140, the user adjusts the focal length f1 of the eye lens to the virtual image I output from the waveguide 140. V The image plane P V . Virtual Image I V The image plane P V It may be formed on the third region 143 of the waveguide 140 , but the present disclosure is not limited thereto.

[0150] Because the focal length f1 of the user's eye lens and the binocular parallax θ V The perceived virtual image I V Depth D V are different from each other, so the user suffers from vergence-accommodation conflict.

[0151] On the contrary, reference Fig.16 In the AR device 100 of this embodiment, the processor 191 adjusts the visual convergence of the focus adjustable lens 160 so that the AR device 100 makes the focal length of the user's eye lens equal to the virtual image I V Depth D V In other words, because the visual convergence of the focus adjustable lens 160 is adjusted, the user can adjust the focal length f2 to the visual convergence distance (ie, the depth D of the virtual image). V ) are the same, thus preventing visual convergence-accommodation conflict.

[0152] refer to Fig.17 , when the user sees the real scene S R When the focus adjustable lens 160 is in the closed state, the visual convergence of the focus adjustable lens 160 is 0. Therefore, the user can see the real scene S without distortion. R In addition, because the focal length f3 of the eye lens is different from the actual scene S R The binocular parallax θ R The distance (i.e., the real scene S R The depth of the image is the same, so convergence-accommodation conflict is prevented.

[0153] Fig.18 is a view for describing a principle based on which the AR device 100 adjusts the brightness of a virtual image and the brightness of a real scene according to an embodiment.

[0154] When the external environment is a bright daytime outdoor environment, a dark nighttime outdoor environment, or an indoor environment, the amount of light of the real scene entering the user's eyes is greatly different. In an embodiment, the AR device 100 may include an illuminance sensor, and the processor 191 may adjust the first time period βt of the image data by detecting the external brightness sensed by the illuminance sensor and adjusting the first time period βt of the image data based on the external brightness. 1 and the second period βt 2 The time division ratio of βt can be used to adjust the light amount of the real scene and the light amount of the virtual image. When the external environment is a daytime outdoor environment that is too bright, the second time period βt for transmitting the light of the real scene can be reduced. 2 , and the first time period βt for displaying the virtual image may be increased 1 .

[0155] In the AR device 100 according to the embodiment, the processor 191 may adjust the light amount of the virtual image by appropriately controlling the intensity of the light source of the optical engine 120 during the first period.

[0156] In the AR device 100 according to an embodiment, the processor 191 can adjust the amount of light of the real scene by applying a lower voltage to convert only a portion of the first polarization component of light of the real scene incident on the polarization converter 150 into a second polarization component (rather than turning off the polarization converter 150 in the second period).

[0157] In the AR device 100 according to an embodiment, the processor 191 can adjust the amount of light of the real scene and the amount of light of the virtual image by adjusting at least two of the intensity of the light source of the optical engine 120, the intensity of the voltage applied to the polarization converter 150, and the time division ratio between the first time period and the second time period.

[0158] Fig.19 2 is a view illustrating an AR image having multiple depths in the AR device 100 according to an embodiment. Fig. 20 is a view for describing an operation of sequentially generating an AR image having a plurality of depths in the AR device 100 according to an embodiment.

[0159] refer to Fig.19 and Fig. 20 In the AR device 100, the virtual image overlaid on the real scene may include a first virtual image layer Virtual 1 and a second virtual image layer Virtual 2 with different depths. The first virtual image layer Virtual 1 may include a first left-eye virtual image layer and a first right-eye virtual image layer with a first binocular parallax. The second virtual image layer Virtual 2 may include a second left-eye virtual image layer and a second right-eye virtual image layer with a second binocular parallax. The first binocular parallax and the second binocular parallax may be different values.

[0160] In the second period, the processor 191 of the AR device 100 controls the polarization converter 150 and the focus adjustable lens 160 to be turned off, and the optical engine 120 does not output the light of the virtual image. When the polarization converter 150 is controlled to be turned off, the light of the real scene is incident on the user's eyeball through the polarization converter 150 and the focus adjustable lens 160. When the focus adjustable lens 160 is controlled to be turned off, the user can see the real scene without distortion.

[0161] In the first virtual period after the second period, the processor 191 of the AR device 100 controls the polarization converter 150 and the focus adjustable lens 160 to turn on, and controls the optical engine 120 to output the light of the first virtual image layer Virtual 1. In this case, the processor 191 controls the focus adjustable lens 160 with the first voltage V1, and adjusts the visual convergence of the focus adjustable lens 160 so that the user's eye lens has a focal length corresponding to the depth of the first virtual image layer Virtual 1. When the polarization converter 150 is controlled to be turned on, the light of the real scene is blocked by the second polarizer 159, and the light of the first virtual image layer Virtual 1 is incident on the user's eyeball through the waveguide 140 and the focus adjustable lens 160. When the focus adjustable lens 160 is controlled with the first voltage V1, the user can see the first virtual image layer Virtual 1 at the depth of the first virtual image layer Virtual 1.

[0162] In the second virtual period after the first virtual period, the processor 191 of the AR device 100 controls the polarization converter 150 and the focus adjustable lens 160 to turn on, and controls the optical engine 120 to output the light of the second virtual image layer Virtual 2. In this case, the processor 191 controls the focus adjustable lens 160 with the second voltage V2, and adjusts the visual convergence of the focus adjustable lens 160 so that the user's eye lens has a focal length corresponding to the depth of the second virtual image layer Virtual 2. When the polarization converter 150 is controlled to be turned on, the light of the real scene is blocked by the second polarizer 159, and the light of the second virtual image layer Virtual 2 is incident on the user's eyeball through the waveguide 140 and the focus adjustable lens 160. When the focus adjustable lens 160 is controlled with the second voltage V2, the user can see the second virtual image layer Virtual 2 at the depth of the second virtual image layer Virtual 2.

[0163] The processor 191 may generate image data to repeat the second period, the first virtual period, and the second virtual period, and may control the optical engine 120, the polarization converter 150, and the focus adjustable lens 160. Although the order, frequency, and time interval of the second period, the first virtual period, and the second virtual period may be fixed, the present disclosure is not limited thereto. When the user manipulates the interface 192 according to the first virtual image layer, the time interval or frequency of the first virtual period may be increased. Referring to, for example, Fig.36 or Fig.37A method of increasing the frequency of the first virtual period is described below.

[0164] Although the second polarizer 159 is located between the waveguide 140 and the focus adjustable lens 160 in the above-described embodiment of the present disclosure, the present disclosure is not limited thereto. Fig.21 is a view showing the arrangement of optical components in an AR device according to an embodiment. Fig.21 , the second polarizer 159 may be located between the polarization converter 150 and the waveguide 140. In an embodiment, the focus adjustable lens 160 may be a lens having polarization-independent refractive power. In an embodiment, the focus adjustable lens 160 may be a liquid crystal lens having polarization-dependent refractive power, and in this case, the optical engine 120 may be configured to emit light of a specific polarization, or an additional polarizer may be provided between the waveguide 140 and the focus adjustable lens 160. In an embodiment, when the focus adjustable lens 160 has polarization-dependent refractive power, the focus adjustable lens 160 may include a polarizer incorporated therein.

[0165] Although in the above-described embodiments of the present disclosure, light of a real scene is blocked or transmitted by a combination of the polarization converter 150 , the first polarizer 158 , and the second polarizer 159 , the present disclosure is not limited thereto. Fig. 22 1 is a view showing the arrangement of the optical assembly 130 in the AR device according to the embodiment. Fig. 22 The optical assembly 130 in the AR device of this embodiment may include a waveguide 140, an optical shutter 250 in contact with or spaced apart by a predetermined interval from a front surface of a third region 143 of the waveguide 140, and a focus adjustable lens 160 in contact with or spaced apart by a predetermined interval from the third region 143 of the waveguide 140. The processor 191 controls the optical shutter 250 to block or transmit light of a real scene.

[0166] Fig.23 2 is a view showing a configuration of optical shutter 250 according to an embodiment. Fig.23 , the optical shutter 250 may be a liquid crystal optical shutter including the polarization converter 150 and the first and second polarizers 158 and 159, and may be located on the front surface of the waveguide 140. The optical shutter 250 may be implemented by integrally manufacturing the polarization converter 150 and the first and second polarizers 158 and 159.

[0167] Fig.24 2 is a view showing a configuration of optical shutter 250 according to an embodiment. Fig.24The optical shutter 250 is an electrochromic optical shutter including a first transparent electrode 352 , an electrochromic layer 353 , an electrolytic layer 354 , and a second transparent electrode 355 , which are located between two transparent substrates 351 and 356 facing each other.

[0168] The electrochromic layer 353 includes electrochromic particles. Electrochromic particles are particles whose light transmission characteristics change through oxidation and reduction depending on whether power is supplied. In an embodiment, the electrochromic particles may include a support and an electrochromic material surrounding the support. In an embodiment, the support may include a porous material for improving the chromism characteristics of the electrochromic material. In an embodiment, the support may include a transparent conductive oxide such as indium oxide (In2O3) for improving electron mobility. 2 O 3 )、Tin Oxide(SnO 2 ), ITO, indium tin oxide (IZO), antimony tin oxide (ATO), fluorine-doped tin oxide (FTO) or aluminum zinc oxide (AZO). In an embodiment, the carrier may include a non-conductive metal oxide having high transmittance to visible light and a large specific surface area, such as titanium oxide (TiO 2 ), zinc oxide (ZnO) or silicon dioxide (SiO 2 ). In an embodiment, the electrochromic material may include a viologen derivative (such as methylviologen, ethyl viologen, or benzyl viologen), and may be formed by another organic material (such as a phosphonic acid derivative or pyridine) or an inorganic material (such as a transition metal oxide, such as tungsten oxide). The carrier and the electrochromic material are not limited to the above materials. The electrochromic layer 353 may be divided into an oxidized coloring material layer and a reduced coloring material layer, with the electrolytic layer 354 located between them. The electrochromic layer 353 may initially be in a transparent state (i.e., when no power is supplied), and may be in a light-shielding state that absorbs light when power is supplied. An optical shutter 250 that is an electrochromic optical shutter may be used instead of the optical shutter 250 that is a liquid crystal optical shutter. Since optical shutter 250 , which is an electrochromic optical shutter, is independent of polarization, optical shutter 250 can reduce the loss of external light (ie, light of a real scene) compared to a method using polarization.

[0169] Fig.25 and Fig.26 is a view illustrating the operation of the focus adjustable lens 160 according to the embodiment.

[0170] refer to Fig.25 and Fig.26 , the focus adjustable lens 160 may be an electrowetting liquid lens including two transparent substrates 461 and 463, a first medium 465 and a second medium 466 located between the transparent substrates 461 and 463, and a sealing member 462 that seals the first medium 465 and the second medium 466. The first medium 465 and the second medium 466 are materials that do not mix with each other and have different refractive indices. For example, the first medium 465 may be a non-conductive fluid, and the second medium 466 may be a conductive fluid. Electrodes 467 and 468 are arranged in a predetermined pattern inside the sealed by the sealing member 462 and the transparent substrates 461 and 463, and an insulating layer 464 coats at least a portion of the electrodes 467 and 468. When power is supplied to the electrodes 467 and 468, the wetting state of the insulating layer 464 may change, and as Fig.25 and Fig.26 As shown, the shape of the interface between the first medium 465 and the second medium 466 can be changed. Therefore, the curvature of the interface between the first medium 465 and the second medium 466 can be changed by electrical control to control the visual convergence of the focus adjustable lens 160. Because the focus adjustable lens 160 as a liquid lens is independent of polarization, the focus adjustable lens 160 can reduce the loss of light of a virtual image or light of a real scene compared to a method using polarization.

[0171] Fig. 27 is a block diagram of an AR device 100 according to an embodiment. Fig. 27 The AR device 100 of this embodiment includes an optical engine 120, a polarization converter 150, a focus adjustable lens 160, a processor 191, an interface 192, a memory 193, and an eye tracker 570. Figure 3 The AR device 100 is substantially the same as that of FIG. 1 , and the above contents also apply here.

[0172] The eye tracker 570, which is a device for tracking the gaze direction of the user's eyes, can obtain information about the gaze direction indicating the user's gaze direction. The eye tracker 570 may include a left eye tracker 570L and a right eye tracker 570R. The processor 191 can obtain the user's gaze point by using the information about the left eye gaze direction and the information about the right eye gaze direction obtained by the left eye tracker 570L and the right eye tracker 570R, respectively.

[0173] Fig.28 is a diagram showing an eye tracker 570 in an AR device according to an embodiment. Fig.28 , the eye tracker 570 may include an infrared emitter 571 and a plurality of infrared detectors 575a, 575b, 575b, 575d, 575e, and 575f. Fig.28 , the eye tracker 570 includes an infrared emitter 571 and six infrared detectors (ie, infrared detectors 575a to 575f), but the embodiment is not limited thereto.

[0174] The infrared emitter 571 may emit infrared light to the portion of the cornea where the lens of the user's eyeball E is located, and the plurality of infrared detectors 575a to 575f may detect the infrared light reflected by the cornea. In an embodiment, the eye tracker 570 may obtain information about the amount of infrared light detected by each of the plurality of infrared detectors 575a to 575f, and may obtain information about the gaze direction of the user's eyeball E based on the obtained amount of infrared light. The eye tracker 570 may provide the processor 191 with the obtained information about the gaze direction. For example, the information about the gaze direction obtained by the eye tracker 570 may be gaze angle information of the user's left eye in the horizontal and vertical directions, and gaze angle information of the user's right eye in the horizontal and vertical directions.

[0175] Will refer to Figure 29 to Figure 31 A method of tracking gaze based on information about gaze direction measured by eye tracker 570 is described.

[0176] Fig.29 is a view of a 3D eyeball model showing the direction in which the user is looking. Fig.29 According to an embodiment, the gaze direction can be performed based on a 3D eyeball model of gaze. According to the 3D eyeball model, when assuming that the eyeball has a perfect spherical shape and rotates in space ideally according to the gaze, the gaze can be mathematically modeled by using equations 1 and 2.

[0177] [Equation 1]

[0178]

[0179]

[0180] [Equation 2]

[0181]

[0182]

[0183] In Equation 1, d represents the distance between the center Eo of the user's eyeball and the virtual screen S, α represents the rotation angle of the user's eyeball on the x-axis (e.g., horizontal axis) when the user's eyeball is fixed in front of the virtual screen S, and β represents the rotation angle of the user's eyeball on the y-axis (e.g., vertical axis) when the user's eyeball is fixed in front of the virtual screen s. In Equation 2, it is assumed that the user's eyeball is a sphere, and r represents the radius of the sphere.

[0184] The eye tracker 570 according to an embodiment may measure the rotation degrees α and β of the user's eyeball E, and the AR device 100 may calculate the two-dimensional (2D) coordinate values ​​(x, y) of the gaze direction of the user's eyeball E on the virtual screen S by using the rotation degrees α and β. The rotation degrees α and β of the user's eyeball E may be gaze angle information of the eyeball E in the horizontal and vertical directions.

[0185] The actual movement of the eyeball is not the expected 3D rotation, and in particular, in left / right gaze, the relaxation or contraction of the eyeball muscle may be large, so for left / right gaze based on the expected 3D rotation eyeball model, an error may occur during vertical gaze estimation. Due to this error, the AR device 100 may make the user look at a random point, and may compare the gaze direction value estimated by the eye tracker 570 with the actual gaze direction value of the random point, and may statistically process the comparison result to improve accuracy.

[0186] Fig.30 A view used to describe the relationship between the gaze point and gaze angle of the user's left and right eyes. Fig.31 A view used to describe the relationship between the gaze point and the gaze angle in the upward gaze direction.

[0187] refer to Fig.30 and Fig.31 , the focal length can be estimated by using the difference between the gaze directions (or gaze coordinates) of the left eye El and the right eye Er obtained by the eye tracker 570. When the focal length to the gaze point is to be obtained, the gaze axes of the eyes may not cross each other, and in this case, it can be assumed that both eyes are at the same height, and the coordinate of the vertical axis (e.g., y axis) can be calculated as the average of the coordinates of the vertical axes (e.g., y axis) of both eyes. It can be assumed that the distance 'a' between the two eyes is, for example, 7 cm. By the above geometric assumption, the following equation can be obtained using a proportional expression.

[0188] [Equation 3]

[0189]

[0190] The distance 'd' to the virtual screen and the distance 'a' between the eyes are required in Equation 3, and when the user looks forward, the distance 'd' can be obtained by measuring the rotation angle of the eyeball using the gaze image. As a result, the distance 'D' to the gaze point is defined by Equation 4.

[0191] [Equation 4]

[0192]

[0193] x represents the horizontal distance between the gaze coordinates xl and xr of the left eye and the right eye on the virtual screen S, and can be obtained by using the gaze angles of the left eye and the right eye of the user, as shown in Equations 1 and 2.

[0194] Will refer to Fig.32 The operation of the AR device 100 is described.

[0195] Fig.32 1 is a flowchart for describing the operation of the AR device 100 according to the embodiment. Fig. 27 and Fig.32 In operation S610, the processor 191 according to an embodiment generates image data including a virtual image based on data received through the interface 192 or data stored in the memory 193. The image data including the virtual image may be image data that is time-divided by temporally alternating a first period for outputting the virtual image and a second period for not outputting the virtual image.

[0196] In operation S620, during a first period, the processor 191 controls the optical engine 120 to output the light Lv of the virtual image, controls the polarization converter 150 to turn on so that the light Lo of the real scene does not reach the focus adjustable lens 160, and controls the focus adjustable lens 160 to turn on to have a first focal length. The processor 191 may control the focus adjustable lens 160 to have a first focal length according to the depth of the virtual image.

[0197] The second period is after the first period. In operation S630, during the second period, the processor 191 controls the optical engine 120 not to output the light Lv of the virtual image, controls the polarization converter 150 to be turned off so that the light Lo of the real scene reaches the focus adjustable lens 160, and controls the focus adjustable lens 160 to be turned off to have a second focal length. The second focal length may be, for example, an infinite focal length (i.e., when the focus adjustable lens 160 has no refractive power), but the present disclosure is not limited thereto.

[0198] Next, in operation S640, the processor 191 detects the gaze point of the user by controlling the eye tracker 570. Figure 29 to Figure 31 As described above, the left eye tracker 570L can detect information about the horizontal and vertical rotation angles (i.e., horizontal and vertical gaze angles) of the user's left eye, the right eye tracker 570R can detect information about the horizontal and vertical rotation angles (i.e., horizontal and vertical gaze angles) of the user's right eye, and the processor 191 can calculate the user's gaze point based on the detected information.

[0199] In an embodiment, gaze tracking may be performed while operation S620 of displaying a virtual image is performed (ie, a first period).

[0200] In an embodiment, gaze tracking may be performed while operations 620 and 630 are alternately and repeatedly performed. That is, gaze tracking may be performed while the first period and the second period are alternately and repeatedly performed.

[0201] Next, in operation S650, the processor 191 may adjust a first focal length of the focus adjustable lens 160 during a first period based on the detected gaze point.

[0202] In an embodiment, the user may gaze at another point even when the AR device 100 displays a virtual image. In this case, the processor 191 may induce the user to be interested in the virtual image by re-adjusting the depth of the virtual image according to the calculated gaze point and re-adjusting the first focal length of the focus adjustable lens 160.

[0203] In an embodiment, the processor 191 may more accurately control the first focal length of the focus adjustable lens 160 according to the user's personal characteristics based on the detected gaze point. The first focal length of the focus adjustable lens 160 calculated before the user's gaze point is detected may be based on a standard interocular distance or a standard size of an eyeball. However, the first focal length of the focus adjustable lens 160 suitable for the depth of the virtual image (in other words, visual convergence) may vary according to the user's personal characteristics. That is, because the interocular distance or the eyeball size may vary according to the user's personal characteristics, the first focal length of the focus adjustable lens 160 may be more accurately controlled based on the detected gaze point.

[0204] Fig.33 : is a view showing a case where a depth of interest exists in an AR image having three different depths. Fig.34 is a flowchart for describing the operation of the AR device according to the embodiment.

[0205] refer to Fig.33 , the virtual image may include a first virtual image layer 1 and a second virtual image layer 2 having different depths. When the depth of the real scene 0 is included, the AR image includes an image having a total of three depths. In this case, the operation of the AR device of this embodiment will be described assuming that there is a depth of interest to the user in the first virtual image layer 1.

[0206] refer to Fig. 27 and Fig.34In operation S710, the processor 191 according to the embodiment generates image data including a virtual image based on data received through the interface 192 or data stored in the memory 193. The image data including the virtual image may be image data that is temporally divided by temporally alternating a first period for outputting the first virtual image layer 1 and the second virtual image layer 2 and a second period for not outputting the virtual image. The first period includes a first virtual period and a second virtual period. During the first virtual period, the first virtual image layer 1 is output. During the second virtual period, the second virtual image layer 2 is output.

[0207] In operation S720, during the first virtual period, the processor 191 controls the optical engine 120 to output the light Lv of the first virtual image layer 1, controls the polarization converter 150 to turn on so that the light Lo of the real scene does not reach the focus adjustable lens 160, and controls the focus adjustable lens 160 to turn on to have a first focal length. The processor 191 may control the focus adjustable lens 160 to have a first focal length according to the depth of the first virtual image layer.

[0208] In operation S730, during a second virtual period after the first virtual period, the processor 191 controls the optical engine 120 to output the light Lv of the second virtual image layer 2, controls the polarization converter 150 to turn on so that the light Lo of the real scene does not reach the focus adjustable lens 160, and controls the focus adjustable lens 160 to turn on to have a second focal length. The processor 191 may control the focus adjustable lens 160 to have a second focal length according to the depth of the second virtual image layer.

[0209] The second period is after the second virtual period. In operation S740, during the second period, the processor 191 controls the optical engine 120 not to output the light Lv of the virtual image, controls the polarization converter 150 to be turned off so that the light Lo of the real scene reaches the focus adjustable lens 160, and controls the focus adjustable lens 160 to be turned off to have a third focal length. The third focal length may be, for example, an infinite focal length (ie, when the focus adjustable lens 160 has no refractive power), but the present disclosure is not limited thereto.

[0210] Next, in operation S750, the processor 191 detects the gaze point of the user by controlling the eye tracker 570. Figure 29 to Figure 31 As described above, the left eye tracker 570L can detect information about the horizontal and vertical rotation angles of the user's left eye, the right eye tracker 570R can detect information about the horizontal and vertical rotation angles of the user's right eye, and the processor 191 can calculate the user's gaze point based on the detected information.

[0211] In an embodiment, gaze tracking may be performed while operations S720 and S730 of displaying the first virtual image layer and the second virtual image layer are performed (ie, the first virtual period and the second virtual period).

[0212] In an embodiment, gaze tracking may be performed while operations S720 to S740 are alternately and repeatedly performed. That is, gaze tracking may be performed while the first virtual period and the second virtual period and the second period are alternately and repeatedly performed.

[0213] In operation S760, the processor 191 detects or determines the depth of interest based on the detected gaze point. For example, the virtual image of the depth of interest may be the first virtual image layer 1. Thus, when the virtual image of the depth of interest is determined, image data weighted with the depth of interest is generated.

[0214] Fig.35 is a diagram for describing a case where an AR image having three different depths is displayed without reflecting a depth of interest according to an embodiment. Fig.35 In , 0 represents the second time period for showing the real scene, 1 represents the first virtual time period for showing the first virtual image layer, and 2 represents the second virtual time period for showing the second virtual image layer. Fig.35 The AR image shown has the same frequency number for each depth during one period.

[0215] Fig.36 2 is a diagram for describing a case where an AR image reflecting a depth of interest among three different depths is displayed according to an embodiment. Fig.36 , the frequency number of the first virtual period 1 for showing the first virtual image layer may be weighted during one cycle. For example, during one cycle, the first virtual period 1 for showing the image of the depth of interest (i.e., the first virtual image layer) has a frequency number of 2, and each of the second virtual period 2 and the second period 0 for showing the second virtual image layer has a frequency number of 1. Therefore, the AR device 100 can enable the user to see the virtual image of the depth of interest, such as the first virtual layer 1, more clearly.

[0216] Fig.37 2 is a diagram for describing a case where an AR image reflecting a depth of interest among three different depths is displayed according to an embodiment. Fig.37 , for example, when the left eye and the right eye are combined during one cycle, the second period 0 has a frequency number 1, the first virtual period 1 has a frequency number 2, and the second virtual period 2 has a frequency number 1. The depth of the left eye image and the depth of the right eye image may be different from each other, and the order of images in one cycle may be different from the order of images in the next cycle.

[0217] Fig.38: is a view showing a case where a depth of interest exists in an AR image having six different depths. Fig.39 is a diagram for describing a case where an AR image having six different depths is displayed without reflecting a depth of interest according to an embodiment. Fig.40 is a diagram for describing a case of displaying an AR image reflecting a depth of interest among six different depths according to an embodiment.

[0218] refer to Fig.38 , the virtual image may have first to fifth virtual image layers 1, 2, 3, 4, and 5 with different depths. When the depth of the real scene 0 is included, the AR image includes an image with a total of six depths. In this case, the operation of the AR device of this embodiment will be described assuming that there is a depth of interest to the user in the second virtual image layer 2.

[0219] refer to Fig. 27 and Fig.39 , the processor 191 according to the embodiment generates image data based on the data received through the interface 192 or the data stored in the memory 193, wherein the first to fifth virtual image layers 1, 2, 3, 4, and 5 and the real scene 0 (i.e., the period during which the optical engine 120 does not output a virtual image) are sequentially repeated. In addition, the processor 191 may control the optical engine 120 and the optical assembly 130, so that the user may sequentially see the first to fifth virtual image layers 1, 2, 3, 4, and 5 and the real scene 0. In one cycle, the first to fifth virtual image layers 1, 2, 3, 4, and 5 have the same frequency number as the real scene 0.

[0220] The processor 191 detects the gaze point of the user by controlling the eye tracker 570, and then generates image data reflecting the depth of interest based on the detected gaze point.

[0221] refer to Fig.40 , for example, the frequency number of the second virtual period 2 for showing the second virtual image layer may be weighted to be the highest. In addition, the frequency number of each of the first virtual period 1 and the third virtual period 3 for showing the first virtual image layer and the third virtual image layer close to the second virtual image layer may be weighted. For example, during one cycle, the second virtual period 2 for showing an image of the depth of interest (i.e., the second virtual image layer) has a frequency number 3, the first virtual period 1 and the third virtual period 3 for showing the first virtual image layer and the third virtual image layer close to the second virtual image layer have a frequency number 2, and the fourth virtual period 4 and the fifth virtual period 5 for showing the fourth virtual image layer and the fifth virtual image layer and the second period 0 have a frequency number 1. Therefore, the AR device 100 can enable the user to see the virtual image close to the depth of interest more clearly.

[0222] Fig.41 is a diagram for describing a case where an AR image reflecting a depth of interest among six different depths is displayed according to an embodiment. Fig.41 , for example, when the left eye and the right eye are combined during one cycle, the second time period 0 has a frequency number 1, the first virtual time period 1 and the third virtual time period 3 have a frequency number 2, the second virtual time period 2 has a frequency number 3, and the fourth virtual time period 4 and the fifth virtual time period 5 have a frequency number 1. The depth of the left eye image and the depth of the right eye image may be different from each other. In this case, the processor 191 controls the focal length of the left eye focus adjustable lens 160L according to the depth of the left eye image, and controls the focal length of the right eye focus adjustable lens 160R according to the depth of the right eye image. The order of images in one cycle may be different from the order of images in the next cycle. Although in Fig.40 In the embodiment of the present invention, one cycle includes 10 image frames, but in Fig.41 In the embodiment, one cycle may include 5 image frames.

[0223] Fig.42 is a diagram showing an eye tracker 570 according to an embodiment. Fig.42 , the eye tracker 570 may include a plurality of light sources 871a, 871b, and 871c, and one or more image sensors 875. The light sources 871a, 871b, and 871c may be infrared LEDs (IR LEDs) and may be located at different positions. For example, the light sources 871a, 871b, and 871c may be located on the frame 111. When photographing the user's eyes, the light sources 871a, 871b, and 871c may provide light (e.g., infrared light) to the user's eyeball E sequentially in time. Providing light to the user's eyeball E may generate light reflected from the user's eyeball E.

[0224] The image sensor 875 may be an IR image sensor. Because the positions of the light sources 871a, 871b, and 871c are different from each other, the positions of the reflected light in the image of the user's eye captured by the image sensor 875 may be different from each other, so the AR device can track the gaze of the user's eye by using the position information of the reflected light. For example, the eye tracker 570 can obtain the rotation direction information (i.e., gaze angle information) of the user's eye E in the horizontal and vertical directions by detecting the light spot formed on the pupil by the reflected light in the image of the user's eye E, and can obtain the gaze direction information based on the rotation direction information.

[0225] The AR device 100 is used Fig. 27 Eye trackers are not limited to Fig.28 or Fig.42 eye tracker, and an appropriate eye tracker may be used.

[0226] Fig.43 is a block diagram of an AR device 100 according to an embodiment. Fig.43 The AR device 100 of this embodiment includes an optical engine 120, a polarization converter 150, a focus adjustable lens 160, a processor 191, an interface 192, a memory 193, an eye tracker 570, a camera 910, and a depth sensor 920. A 3D camera may be used instead of the camera 910 and the depth sensor 920.

[0227] The AR device of this embodiment is substantially the same as the above-mentioned AR device, and therefore, the above-mentioned descriptions of various embodiments are also applicable here.

[0228] The camera 910 may obtain video and / or still images by capturing the physical environment or space seen by the user. The camera 910 may send the obtained video data and / or still image data to the processor 191. In an embodiment, the camera 910 may visually monitor the user's surrounding space. In an embodiment, the camera 910 may execute one or more controls or operations in an application driven by the processor 191, or may capture gestures or movements performed by the user and real-world objects in the surrounding space to operate features in the application.

[0229] The depth sensor 920 may be a camera that measures the depth value of a real-world object seen by the user. The depth sensor 920 may scan a physical space or environment, may measure the depth value of a real-world object located in the physical space or environment according to the 3D position coordinate value of the real-world object, and may generate a depth map by locating the measured depth value according to each 3D position coordinate value. The depth sensor 920 may store the generated depth map information in the memory 193 or a separate storage device.

[0230] In an embodiment, the depth sensor 920 may obtain a depth image including depth values ​​of real-world objects. The depth image may include a 2D pixel area of ​​the captured scene, and each pixel in the 2D pixel area may indicate a depth value, such as the distance of a real-world object in the scene captured by using the depth sensor 920 in centimeters (cm), millimeters (mm), etc.

[0231] The depth sensor 920 may measure a 3D depth value by using, for example, one of a stereo-type method, time-of-flight (ToF), and a structured pattern. In an embodiment, the depth sensor 920 may include an RGB camera, a stereo camera, an infrared light component, and a 3D camera that can be used to capture a depth image of a real-world object.

[0232] Fig.44is a view showing a case where a virtual object 750 (eg, a virtual image) having a depth matching that of a real-world object 752 is displayed. Fig.43 and Fig.44 , the depth sensor 920 may send a depth image including a depth value of a real-world object to the processor 191, or may store the depth image in the memory 193 or a separate storage device. In an embodiment, the processor 191 may obtain a depth image from the depth sensor 920, or may obtain a depth value of a real-world object by loading a depth map previously stored in the memory 193 or a separate storage device, and may change the depth of a virtual object (e.g., a virtual image) based on the obtained depth value. As in the above-mentioned embodiments of the present disclosure, when the depth of a virtual object (e.g., a virtual image) changes, the focal length of the focus adjustable lens 160 may be readjusted according to the changed depth during the first period, thereby matching the depth of the virtual image including the virtual object with the position of the real-world object.

[0233] Although the AR device 100 is a glasses-type device, a person skilled in the art will understand that the AR device 100 may be specifically embodied as or included in a head-mounted display device or an AR helmet.

[0234] The AR device 100 according to the present disclosure may be implemented as a hardware element, a software element, and / or a combination of hardware elements and software elements. For example, the AR device 100 according to an embodiment of the present disclosure may be implemented by using a processor, an arithmetic logic unit (ALU), an ASIC, a DSP, a DSPD, a PLD, a microcomputer, a microprocessor, or one or more general-purpose computers or special-purpose computers, such as a device capable of executing and responding to instructions.

[0235] The software may include a computer program, code, instructions, or a combination of one or more thereof, and may configure a processing device to operate as desired, or may individually or collectively command a processing device.

[0236] The software may be implemented as a computer program including instructions stored on a computer-readable storage medium. Computer-readable storage media may include magnetic storage media (e.g., read-only memory (ROM), random access memory (RAM), floppy disk, hard disk, etc.) and optical reading media (e.g., compact disk read-only memory (CD-ROM), digital versatile disk (DVD), etc.). Computer-readable recording media may be distributed in computer systems connected via a network, and computer-readable code may be stored and executed in a distributed manner. The medium may be computer-readable, may be stored in a memory, and may be executed by a processor.

[0237] The computer may be a device configured to call an instruction stored in a storage medium and perform operations according to an embodiment of the present disclosure in response to the called instruction, and may include the AR device 100 according to an embodiment of the present disclosure.

[0238] The computer-readable storage medium may be provided as a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether data is stored on the storage medium semi-permanently or temporarily.

[0239] In addition, the AR device 100 or the method of operating the AR device 100 according to the embodiment of the present disclosure may be provided in a computer program product. The computer program product is a product that can be traded between a seller and a buyer as a product.

[0240] The computer program product may include a software program and a computer-readable storage medium in which the software program is stored. For example, the computer program product may include a product (e.g., a downloadable application) distributed electronically by the manufacturer of the AR device 100 or as a software program through an electronic market (e.g., Google Play Store or Apple Store). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a relay server that temporarily stores the software program.

[0241] In a system including a server and a terminal (e.g., an AR device), the computer program product may include a storage medium of the server or a storage medium of the terminal. Alternatively, when there is a third device (e.g., a smartphone) that communicates with the server or the terminal, the computer program product may include the storage medium of the third device. Alternatively, the computer program product may include the software program itself that is sent from the server to the terminal or the third device or from the third device to the terminal.

[0242] In this case, one of the server, the terminal, and the third device can perform the method according to the embodiment of the present disclosure by executing the computer program product. Alternatively, at least two of the server, the terminal, and the third device can perform the method according to the embodiment of the present disclosure in a distributed manner by executing the computer program product.

[0243] For example, a server (eg, a cloud server or an artificial intelligence (AI) server) may execute a computer program product stored in the server and control a terminal communicating with the server to execute a method according to an embodiment of the present disclosure.

[0244] As another example, the third device may execute the computer program product and control a terminal communicating with the third device to perform the method according to the embodiment of the present disclosure.

[0245] When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute a free-loaded computer program product and perform the method according to an embodiment of the present disclosure.

[0246] According to the present disclosure, an electronic device and method for displaying AR can reduce eye fatigue.

[0247] According to the present disclosure, an electronic device and method for displaying AR can provide a virtual image responsive to an area of ​​interest to a user by tracking the user's gaze.

[0248] Although the embodiments of the present disclosure have been specifically shown and described with reference to the accompanying drawings, the embodiments are provided for illustrative purposes, and it will be understood by those skilled in the art that various modifications and other equivalent embodiments may be made according to the present disclosure. Therefore, the true technical scope of the present disclosure is defined by the technical spirit of the appended claims.

Claims

1. An electronic device for displaying augmented reality, the electronic device comprising: an optical engine configured to output light of a virtual image; The first polarizer is located in the light path of the real scene; A polarization converter configured to maintain or convert the polarization direction of light of a real scene; a waveguide, from which light of the virtual image is output, and through which light of the real scene is transmitted; a focus-adjustable lens configured to adjust the focus of light of the virtual image output from the waveguide; A second polarizer located between the waveguide and the focus-adjustable lens or between the polarization converter and the waveguide; and One or more processors configured to: During a first period, controlling the polarization converter to convert the polarization direction of first light of the real scene that has passed through the first polarizer so that at least a portion of the first light of the real scene is blocked by the second polarizer, and During the second period, the polarization converter is controlled to maintain the polarization direction of the second light of the real scene that has passed through the first polarizer, so that the second light of the real scene passes through the second polarizer.

2. The electronic device according to claim 1, wherein: The polarization converter includes a liquid crystal panel.

3. The electronic device according to claim 1, wherein: The one or more processors are further configured to, during the first period, control the optical engine to output light of the virtual image, and during the second period, control the optical engine not to output light of the virtual image.

4. The electronic device according to claim 1, wherein: The one or more processors are also configured to, during the first time period, control the focus adjustable lens to have a first focal length for light of the virtual image, and during the second time period, control the focus adjustable lens to have a second focal length different from the first focal length.

5. The electronic device according to claim 4, wherein: The first focal length is used to move the image plane of the virtual image to a position at the depth of the virtual image.

6. The electronic device according to claim 4, wherein: The second focal length is infinity.

7. The electronic device according to claim 4, wherein: The second focal length is used to correct the user's vision.

8. The electronic device according to claim 1, wherein: The first time period and the second time period alternate.

9. The electronic device according to claim 1, wherein: The one or more processors are also configured to, during the first time period, control the optical engine to output light of multiple virtual image layers with different depths respectively by time division, and control the focus adjustable lens according to different first focal lengths corresponding to the different depths of the multiple virtual image layers.

10. The electronic device of claim 1, further comprising an eye tracker configured to track a user's gaze.

11. The electronic device according to claim 10, wherein: The one or more processors are further configured to generate the virtual image during the first period such that the virtual image has a depth of a gaze point of a user whose gaze is tracked by an eye tracker.

12. The electronic device according to claim 10, wherein: The one or more processors are further configured to, during the first period, respectively generate a plurality of virtual image layers having different depths, and Among the multiple virtual image layers, a frequency of a first virtual image layer having a depth closer to a gaze point of a user whose gaze is tracked by the eye tracker is greater than a frequency of a second virtual image layer having a depth farther from the gaze point of the user.

13. The electronic device according to claim 1, wherein: The focus-adjustable lens includes one of a liquid crystal lens, a liquid lens or a movable lens.

14. The electronic device according to claim 1, wherein: The electronic device includes a glasses-type device.

15. An electronic device for displaying augmented reality, the electronic device comprising: a waveguide, from which light of the virtual image is output, and through which light of the real scene is transmitted; an optical engine located on a portion of the rear surface of the waveguide; an optical shutter located on the front surface of the waveguide and configured to transmit or block light of a real scene; a focus adjustable lens located on another portion of the rear surface of the waveguide; and One or more processors configured to: During a first period, controlling the optical shutter to block at least a portion of the first light of the real scene, and controlling the focus-adjustable lens to have a first focal length, and During a second period, the optical shutter is controlled to project a second light of the real scene, and the focus-adjustable lens is controlled to have a second focal length different from the first focal length.

16. The electronic device according to claim 15, wherein: The optical shutter includes one of a liquid crystal optical shutter or an electrochromic optical shutter.

17. An electronic device for displaying augmented reality, the electronic device comprising: waveguide; an optical engine located on a portion of the rear surface of the waveguide; a polarization converter located on the front surface of the waveguide and configured to maintain or convert the polarization direction of the light; a first polarizer disposed on a front surface of the polarization converter; a focus adjustable lens located on another portion of the rear surface of the waveguide; A second polarizer located between the waveguide and the focus-adjustable lens or between the polarization converter and the waveguide; and One or more processors configured to: During a first period, controlling the polarization converter to convert the polarization direction of first light of a real scene that has passed through the first polarizer so that at least a portion of the first light of the real scene is blocked by the second polarizer, and During the second period, the polarization converter is controlled to maintain the polarization direction of the second light of the real scene that has passed through the first polarizer, so that the second light of the real scene passes through the second polarizer.

Images