Augmented reality device and operation method thereof that adjusts the focus area according to the user's line of sight

Abstract

An augmented reality (AR) device includes a variable focus lens, the focal length of which can be changed by adjusting the refractive power and adjusting the position of a focus area of ​​the variable focus lens according to a user's sight direction. The AR device can use an eye tracker to obtain an eye vector indicating the user's sight direction, adjust the refractive power of a first focus area of ​​a first variable focus lens to change the focal length for displaying a virtual image, and complementarily adjust the refractive power of a second focus lens relative to the adjusted refractive power of the first focus area.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality (AR) device that provides a virtual image to be perceived as displayed on a real-world object, and more particularly, to an AR device that automatically adjusts the position and size of a focus area according to a user's line of sight direction and an operating method thereof. Background Art

[0002] Augmented reality (AR) is a technology that superimposes virtual images on real-world physical environment spaces or real-world objects, and is implemented by AR devices (e.g., smart glasses) that utilize AR technology. AR devices can be used in various situations such as information retrieval, directions, camera shooting, etc. Even smart glasses are worn as fashion items and may be mainly used for outdoor activities.

[0003] While the AR device is worn by the user, the user of the AR device generally sees a scene through a perspective display arranged close to the user's eyes. Here, the scene includes one or more real-world objects in the physical environment or space that the user sees directly through the eyes. The AR device can project a virtual image onto the perspective display or onto the user's eyes through the perspective display, and the user can simultaneously view the real-world objects and the projected virtual images through the perspective display.

[0004] When watching a virtual image through a perspective display while wearing an AR device, although the focal length controlled by the lens of the user's eye is adjusted to the perspective display displaying the virtual image, the vergence distance of the user's eyes that watch the left eye image and the right eye image of the virtual image that provide a stereoscopic effect according to the binocular parallax is formed farther or closer to the perspective display, which leads to inconsistency between the focal length and the vergence distance. Therefore, users who watch the virtual images of the AR device for a long time or are sensitive may suffer from dizziness or motion sickness. This problem is called vergence-accommodation conflict (VAC). The focus position of the virtual image formed thereon can be adjusted by placing an optical lens capable of adjusting the refractive power in front of or behind the perspective display and adjusting the vergence of the optical lens, so that the above problem can be solved. Summary of the invention

[0005] Technical Solution

[0006] Aspects of the embodiments relate to an augmented reality (AR) device and an operating method thereof, the AR device including a variable focus lens capable of changing a focal length by adjusting a refractive power and adjusting a position of a focusing area of ​​the variable focus lens according to a line of sight direction of a user.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other aspects, features and advantages of certain embodiments of the present disclosure will be more apparent from the following description in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a diagram showing an augmented reality (AR) device that displays a virtual image on a real-world object according to an embodiment of the present disclosure;

[0010] Figure 2 is a plan view of an AR device according to an embodiment of the present disclosure;

[0011] Figure 3 is a block diagram illustrating elements of an AR device according to an embodiment of the present disclosure;

[0012] Figure 4 is a flowchart illustrating a method of operating an AR device according to an embodiment of the present disclosure;

[0013] Figure 5A An eye tracker of an AR device according to an embodiment of the present disclosure is shown;

[0014] Figure 5B An eye tracker of an AR device according to an embodiment of the present disclosure is shown;

[0015] Figure 5C is a diagram showing a three-dimensional (3D) eyeball model with respect to a user's line of sight;

[0016] Figure 5D is a reference diagram for describing a method of calibrating an eye tracker according to an embodiment of the present disclosure;

[0017] Fig. 6A is a perspective view of a variable focus lens of an AR device according to an embodiment of the present disclosure;

[0018] Figure 6B is a perspective view for describing a method of adjusting the refractive power of a focus area performed by a variable focus lens of an AR device according to an embodiment of the present disclosure;

[0019] Fig. 7A is a diagram for describing a concept of vergence of a variable focus lens of an AR device according to an embodiment of the present disclosure;

[0020] Figure 7B is a diagram for describing a concept of vergence of a variable focus lens of an AR device according to an embodiment of the present disclosure;

[0021] Figure 8 is a flowchart of a method for adjusting the refractive power of a variable focus lens to change the focal length, performed by an AR device according to an embodiment of the present disclosure;

[0022] Fig. 9 is a perspective view showing a waveguide and a display module of an AR device according to an embodiment of the present disclosure;

[0023] Fig. 10A is a diagram of a method for changing the position of a focus area of ​​a variable focus lens according to a user's sight direction, performed by an AR device according to an embodiment of the present disclosure;

[0024] Fig. 10B is a diagram showing a method of changing the position of a focus area of ​​a variable focus lens according to a user's line of sight direction, performed by an AR device according to an embodiment of the present disclosure;

[0025] Fig.11A is a diagram illustrating a method of changing a position of a focus area of ​​a variable focus lens based on an eye vector, performed by an AR device according to an embodiment of the present disclosure;

[0026] Fig. 11B is a diagram illustrating a method of changing a position of a focus area of ​​a variable focus lens based on an eye vector, performed by an AR device according to an embodiment of the present disclosure;

[0027] Fig. 12A A method for adjusting focal length performed by an AR device according to an embodiment of the present disclosure is shown;

[0028] Fig. 12B A virtual image displayed on a real-world object through an AR device according to an embodiment of the present disclosure is shown.

[0029] Fig.13A is a diagram for describing a method of adjusting focal length performed by an AR device according to an embodiment of the present disclosure;

[0030] Fig. 13B is a diagram of a virtual image displayed on a real-world object by an AR device according to an embodiment of the present disclosure;

[0031] Fig.14 is a flowchart of a method for adjusting the refractive power of a focus area of ​​a variable focus lens based on a depth value of a real-world object, performed by an AR device according to an embodiment of the present disclosure;

[0032] Fig.15A is a diagram illustrating a method of determining a size of a focusing area of ​​a variable-focus lens performed by an AR device according to an embodiment of the present disclosure;

[0033] Fig. 15B is a diagram illustrating a method of determining a size of a focusing area of ​​a variable-focus lens performed by an AR device according to an embodiment of the present disclosure;

[0034] Fig.16A variable focus lens including multiple focus adjustment areas according to an embodiment of the present disclosure is shown; and

[0035] Fig.17 is a block diagram illustrating an AR device according to an embodiment of the present disclosure.

[0036] Best Mode for Carrying Out the Invention

[0037] According to aspects of the present disclosure, an augmented reality (AR) device includes: a plurality of first variable focus lenses and a plurality of second variable focus lenses, wherein each of the first variable focus lenses and each of the second variable focus lenses is an electrically adjustable liquid crystal lens; a plurality of waveguides, wherein each waveguide is disposed between the plurality of first variable focus lenses and the plurality of second variable focus lenses; a plurality of eye trackers configured to obtain a left eye vector by tracking a direction of a left eye of a user of the augmented reality device, and to obtain a right eye vector by tracking a direction of a right eye of the user of the augmented reality device; a display module configured to project light of a virtual image toward the waveguide; and one or more processors configured to: determine a left eye vector based on the left eye vector a first focusing area of ​​a first left-eye variable-focus lens among the multiple first variable-focus lenses, determining a third focusing area of ​​a first right-eye variable-focus lens among the multiple first variable-focus lenses based on a right-eye vector, obtaining a gaze point based on the left-eye vector and the right-eye vector, adjusting the refractive power of the first focusing area and the refractive power of the third focusing area based on the gaze point, and adjusting the refractive power of the second focusing area of ​​the second left-eye variable-focus lens among the multiple second variable-focus lenses and the refractive power of the fourth focusing area of ​​the second right-eye variable-focus lens among the multiple second variable-focus lenses based on the adjusted refractive power of the first focusing area and the adjusted refractive power of the third focusing area.

[0038] For example, the AR device may further include a memory storing a program including one or more instructions, and the one or more processors are further configured to perform operations by executing the one or more instructions of the program stored in the memory.

[0039] For example, when a user wears an AR device, a first left-eye variable-focus lens is set at a position separated from the user's left eye by a first distance, and a second left-eye variable-focus lens is set at a position separated from the user's left eye by a second distance, and the second distance is greater than the first distance.

[0040] For example, the one or more processors may also be configured to control the application of a control voltage to the first left-eye variable-focus lens that produces a phase modulation distribution associated with a position corresponding to the first focusing area, and based on the applied control voltage, adjust the refractive power of the first focusing area by changing the arrangement angle of liquid crystal molecules among the liquid crystal molecules of the first left-eye variable-focus lens arranged at the position of the first focusing area.

[0041] For example, the AR device may also include a depth sensor configured to measure a depth value of a real-world object set at a gaze point, wherein the processor may also be configured to obtain the measured depth value of the real-world object from the depth sensor, and adjust the refractive power of the first focusing area based on the obtained depth value to adjust the focal length of the virtual object, which is a partial area of ​​the virtual image.

[0042] For example, the one or more processors may be further configured to adjust the refractive power of the second focus area to be the same as the refractive power of the first focus area in a direction opposite to the direction of the refractive power of the first focus area.

[0043] For example, the one or more processors may be further configured to determine a position of the second focus area based on the left eye vector such that the first focus area and the second focus area are aligned in the direction of the left eye vector.

[0044] For example, the size of the first focus area may be determined based on the size of a virtual object that is a partial area of ​​a virtual image output through the left-eye waveguide.

[0045] For example, the size of the second focusing area may be determined based on the size of a virtual object which is a partial area of ​​a virtual image output through the left-eye waveguide and a separation distance between the first left-eye variable-focus lens and the second left-eye variable-focus lens.

[0046] For example, a plurality of first focusing areas may be provided on the first left-eye variable-focus lens, wherein the processor may be further configured to adjust the refractive power of the plurality of first focusing areas so that different vergence degrees are formed according to the plurality of first focusing areas.

[0047] According to another aspect of the present disclosure, an AR device includes: a plurality of first variable focus lenses and a plurality of second variable focus lenses, wherein each of the first variable focus lenses and each of the second variable focus lenses is an electrically adjustable liquid crystal lens; a plurality of waveguides, wherein each waveguide is disposed between the plurality of first variable focus lenses and the plurality of second variable focus lenses; a plurality of eye trackers configured to obtain a left eye vector by tracking a direction of a left eye of a user of an augmented reality device, and to obtain a right eye vector by tracking a direction of a right eye of the user of the augmented reality device; a display module configured to project light of a virtual image toward the waveguide; and one or more processors configured to: determine a left eye vector among the plurality of first variable focus lenses based on the left eye vector obtained from the eye tracker The invention relates to a method for determining a first focusing area of ​​a first left-eye variable focus lens and a third focusing area of ​​a first right-eye variable focus lens among the multiple first variable focus lenses based on a right eye vector, obtaining a gaze point based on the left eye vector and the right eye vector, adjusting the refractive power of the first focusing area and the refractive power of the third focusing area based on the gaze point to change the focal length of a real-world object, adjusting the refractive power of the second focusing area of ​​a second left-eye variable focus lens among the multiple second variable focus lenses and the refractive power of the fourth focusing area of ​​a second right-eye variable focus lens among the multiple second variable focus lenses, and independently adjusting the refractive power of the second focusing area and the refractive power of the fourth focusing area regardless of the refractive power of the first focusing area and the refractive power of the third focusing area.

[0048] According to another aspect of the present disclosure, an operating method of an AR device includes: obtaining a left eye vector by tracking the direction of a left eye using a first eye tracker, and obtaining a right eye vector by tracking the direction of a right eye using a second eye tracker; determining a first focusing area of ​​a first left eye variable-focus lens based on the obtained left eye vector, and determining a third focusing area of ​​a first right eye variable-focus lens based on the obtained right eye vector; obtaining a gaze point based on the left eye vector and the right eye vector; adjusting the refractive power of the first focusing area and the refractive power of the third focusing area based on the gaze point; adjusting the refractive power of the second focusing area of ​​the second left eye variable-focus lens and the refractive power of the fourth focusing area of ​​the second right eye variable-focus lens based on the adjusted refractive power of the first focusing area and the adjusted refractive power of the third focusing area; and projecting light of a virtual image toward a waveguide using a display module.

[0049] For example, adjusting the refractive power of the first focusing area and the third focusing area includes: applying a control voltage to the first left-eye variable-focus lens to generate a phase modulation distribution associated with a position corresponding to the first focusing area; and adjusting the refractive power of the first focusing area by changing the arrangement angle of liquid crystal molecules arranged at the position of the first focusing area among the first left-eye variable-focus liquid crystal molecules based on the applied control voltage.

[0050] For example, adjusting the refractive power of the first focusing area and the refractive power of the third focusing area includes: using a depth sensor to measure the depth value of a real-world object set at a gaze point; and adjusting the refractive power of the first focusing area based on the measured depth value to adjust the focal length of a virtual object that is a partial area of ​​a virtual image.

[0051] For example, adjusting the refractive power of the second focusing area and the refractive power of the fourth focusing area includes: adjusting the refractive power of the second focusing area so that the second focusing area forms a vergence in a direction opposite to the direction of the adjusted vergence of the first focusing area; and adjusting the refractive power of the fourth focusing area so that the fourth focusing area forms a vergence in a direction opposite to the direction of the adjusted vergence of the third focusing area.

[0052] For example, adjusting the refractive power of the second focus area and the fourth focus area includes adjusting the refractive power of the second focus area to be the same as the adjusted refractive power of the first focus area in a direction opposite to the adjusted refractive power of the first focus area.

[0053] For example, the operating method may further include: determining the second focus area based on the left eye vector, so that the first focus area and the second focus area are aligned in the direction of the left eye vector.

[0054] For example, the size of the first focusing area may be determined based on the size of a virtual object that is a partial area of ​​a virtual image output through the waveguide.

[0055] For example, the size of the second focusing area may be determined based on the size of the virtual object which is a partial area of ​​the virtual image output through the waveguide and the separation distance between the first left-eye variable-focus lens and the second left-eye variable-focus lens.

[0056] For example, multiple first focusing areas may be provided on the first left-eye variable-focus lens, and adjusting the refractive powers of the first focusing areas and the third focusing areas includes adjusting the refractive powers of the multiple first focusing areas so that different vergence is formed according to the multiple first focusing areas. DETAILED DESCRIPTION

[0057] This application is based on and claims the benefit of U.S. Provisional Patent Application No. 62 / 853,082 filed in the United States Patent and Trademark Office on May 27, 2019, and claims priority under 35 U.S.C. §119 to Korean Patent Application No. 10-2019-0115473 filed in the Korean Intellectual Property Office on September 19, 2019, the disclosures of which are incorporated herein by reference in their entirety.

[0058] Although the terms used in the present disclosure have been described in currently used general terms in consideration of the functions mentioned in the present disclosure, these terms are intended to cover various other terms depending on the intentions of those skilled in the art, precedents, or the emergence of new technologies. In addition, the applicant may select some of the terms used herein. In this case, these terms are defined in detail below. Therefore, the terms used in the present disclosure are not defined based on the meaning of the terms, nor are they defined based on simple term names, but are defined based on the content that runs through the present disclosure.

[0059] An expression used in the singular encompasses an expression in the plural unless an obviously different meaning is provided in the context. Terms including technical terms and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0060] Throughout this disclosure, the expression "at least one of a, b, or c" means: only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0061] Throughout the entire specification of the present disclosure, when a certain part includes a certain element, the term "comprising" means that the corresponding element may also include other elements, unless a specific meaning contrary to the corresponding element is written. The terms used in the embodiments of the present disclosure (such as "unit" or "module") represent units for processing at least one function or operation, and can be implemented in hardware, software, or a combination of hardware and software.

[0062] The expression "configured to" used herein may be used as, for example, an expression "suitable for", "capable of", "designed to", "adapted to", "manufactured to", or "capable of", depending on the context. The term "configured to" is not limited to "specially designed to" in terms of hardware. Rather, the expression "a device configured to" may mean that the device is "capable of" operating with another device or other element. For example, "a processor configured to (or set to) perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that performs the corresponding operations by running one or more software programs stored in a memory device.

[0063] In this disclosure, refractive index refers to the degree to which light flux is reduced in a medium compared to a vacuum.

[0064] In the present disclosure, refractive power refers to the force of changing the direction of light or light path through the curved surface of the lens. Refractive power is the reciprocal of focal length, and the unit of refractive power is m-1 or diopter (D). The sign of refractive power is positive (+) in the case of a convex lens and negative (-) in the case of a concave lens.

[0065] In the present disclosure, vergence is an index indicating the degree of convergence or divergence of light. Vergence can be adjusted according to the refractive power of the lens. In an embodiment of the present disclosure, the variable focus lens can adjust the vergence by adjusting the refractive power of the lens and changing the light or light path.

[0066] In the present disclosure, a virtual image is a projected image projected onto a waveguide by an emission surface of a display module of an augmented reality (AR) device.

[0067] In the present disclosure, a virtual object refers to a partial area of ​​a virtual image output through a waveguide. A virtual object may represent information related to a real-world object. A virtual object may include, for example, at least one of a character, a number, a symbol, an icon, an image, or an animation.

[0068] Hereinafter, the present disclosure will be described in detail by explaining embodiments of the present disclosure with reference to the accompanying drawings. However, the present disclosure may be embodied in many different forms and should not be construed as being limited to the embodiments of the present disclosure set forth herein.

[0069] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings.

[0070] Figure 1 is a diagram illustrating an augmented reality (AR) device 100 that displays a virtual object 20 on a real-world object 10 according to an embodiment of the present disclosure.

[0071] Reference Figure 1 , the user can view the real world object 10 and the virtual object 20 while wearing the AR device 100. The virtual object 20 is displayed by the display module 140 (see Figure 2 and Figure 3 ) and output through the waveguide 130. The virtual object 20 may indicate information related to the real-world object 10. Figure 1 In the illustrated embodiment, the virtual object 20 may be an image including a name, a model name, and price information about a toaster as an example of the real-world object 10 .

[0072] The user may view the real world object 10 and the virtual object 20 through the lens 102. The lens 102 may include a first variable focus lens 110, a second variable focus lens 120, and a waveguide 130.

[0073] When the user wears the AR device 100, the first variable focus lens 110 may be disposed at a position closer to the user's eye 30, and the second variable focus lens 120 may be disposed farther from the user's eye 30 than the first variable focus lens 110 and spaced apart from the user's eye 30. In an embodiment of the present disclosure, the first variable focus lens 110 may be disposed at a position spaced apart from the eye 30 by a first distance, and the second variable focus lens 120 may be disposed at a position spaced apart from the eye 30 by a second distance greater than the first distance. Each of the first variable focus lens 110 and the second variable focus lens 120 may include liquid crystal molecules, and may be configured as an electrically adjustable liquid crystal lens capable of changing a focal length according to an electrical drive signal.

[0074] The waveguide 130 may be disposed between the first variable focus lens 110 and the second variable focus lens 120. The waveguide 130 is an optical element including a transparent material that may be described as a see-through display. The waveguide 130 may be provided between the first variable focus lens 110 and the second variable focus lens 120. Figure 3 ) to project the light of the virtual image. The waveguide 130 may include a plurality of regions in which diffraction gratings are formed. According to the principle of total reflection, the light of the virtual image projected toward the waveguide 130 may be reflected in the waveguide 130. In the light of the virtual image projected onto the waveguide 130, the light path may be changed by the diffraction gratings formed in the plurality of regions so that the virtual object 20 is finally output to the user's eye 30. The waveguide 130 may perform a function similar to that of a light guide plate that changes the light path of the virtual image.

[0075] The first variable focus lens 110 and the second variable focus lens 120 may include a first focus area 112 and a second focus area 122, respectively, and the first focus area 112 and the second focus area 122 are capable of locally adjusting the focus by adjusting the arrangement angle of the liquid crystal molecules arranged in a specific area according to the applied control voltage. In an embodiment of the present disclosure, the first variable focus lens 110 can adjust the refractive power of the first focus area 112 by changing the arrangement angle of the liquid crystal molecules arranged in the first focus area 112 according to the control voltage, thereby adjusting the vergence. Vergence is an index indicating the degree of convergence or divergence of light. Vergence can be adjusted according to the refractive power of the lens.

[0076] The first focusing area 112 can adjust the vergence formed on the first variable focus lens 110 by changing the refractive power like a concave lens, thereby changing the focal length of the virtual object 20 formed therethrough. The first focusing area 112 can change the optical path passing through the first focusing area 112 having the adjusted refractive power, and adjust the focal length of the virtual object 20 projected on the waveguide 130 to be the same as the focal length of the real-world object 10.

[0077] Although the focal length of the virtual object 20 is changed by the adjusted refractive power of the first focusing area 112 of the first variable-focus lens 110, the real-world object 10 is not optimally focused because the focal length relative to the real-world object 10 is also changed, which may cause a problem in which the user feels that the real-world object 10 is blurry. In order to compensate for the focal length distortion of the real-world object 10 caused by the change in vergence due to the adjusted refractive power of the first focusing area 112 of the first variable-focus lens 110, the AR device 100 may complementarily adjust the refractive power of the second focusing area 122 of the second variable-focus lens 120 relative to the first focusing area 112.

[0078] In an embodiment of the present disclosure, the AR device 100 may adjust the refractive power of the second focus area 122 to be the same as the adjusted refractive power of the first focus area 112 in a direction opposite to the direction of the refractive power of the first focus area 112. For example, when the first focus area 112 is adjusted to a refractive power of -1 diopter D, the second focus area 122 may be adjusted to +1 diopter D. In an embodiment of the present disclosure, the second variable focus lens 120 may be an optical element used as a convex lens.

[0079] The AR device 100 may determine the positions of the first focus area 112 and the second focus area 122 according to the direction of the user's line of sight. In an embodiment of the present disclosure, the AR device 100 may obtain an eye vector indicating the direction of the user's line of sight by tracking the position of the user's eye 30 using the eye tracker 150. The AR device 100 may determine the positions of the first focus area 112 and the second focus area 122 according to the eye vector. In the disclosed embodiment, the AR device 100 may determine the positions of the first focus area 112 and the second focus area 122 so that the first focus area 112 and the second focus area 122 are aligned along the direction of the eye vector.

[0080] The AR device 100 may determine the size of the first focus area 112 and the second focus area 122. In an embodiment of the present disclosure, the AR device 100 may determine the size of the first focus area 112 and the second focus area 112 based on the size of the virtual object 20. In another embodiment of the present disclosure, the AR device 100 may determine the size of the first focus area 112 and the second focus area 122 based on the size of the virtual object 20 and the distance between the first variable focus lens 110 and the second variable focus lens 120.

[0081] Conventionally, when the virtual object 20 displayed through the waveguide 130 is viewed while wearing the AR device 100, the focal length formed on the virtual object 20 is fixed to the distance between the waveguide 130 and the eye 30, and the focal length is different from the focal length of the real-world object 10 focused on the gaze point where the user's eyes converge according to binocular parallax. This may cause a problem in which the virtual object 20 is displayed hazy or blurred, and the user may experience dizziness or motion sickness. In order to avoid an undesirable user experience, an optical method of adjusting the focal length of the virtual object 20 by adjusting the refractive power of the optical lens and changing the vergence may be utilized. However, when an optical lens with a fixed focus area is used, the focus may not be adjusted according to the user's line of sight, which leads to a problem of poor optical performance.

[0082] Generally, when the user looks at the center of the lens, the optical performance is excellent. The AR device 100 according to an embodiment of the present disclosure may include a first variable focus lens 110 and a second variable focus lens 120 respectively including a first focusing area 112 and a second focusing area 122, and the first focusing area 112 and the second focusing area 122 are capable of locally adjusting the vergence by changing the refractive power. The AR device 100 can automatically adjust the position of the first focusing area 112 and the second focusing area 122 according to the eye vector obtained using the eye tracker 150, thereby adjusting the focal length of the virtual object 20 to be the same as the focal length of the real-world object 10. Therefore, the AR device 100 of the present disclosure can display the virtual object 20 more clearly and improve the optical performance of the AR device. In addition, the AR device 100 can automatically adjust the focal length of the virtual object 20 according to the user's line of sight, thereby reducing the user's dizziness or motion sickness and providing an improved user experience.

[0083] Figure 2 is a plan view of an AR device 100 according to an embodiment of the present disclosure.

[0084] Reference Figure 2 , the AR device 100 may include a left-eye lens 102 , a right-eye lens 104 , a frame 106 , a nose bridge 108 , a display module 140 , a left-eye tracker 150 - 1 , a right-eye tracker 150 - 2 , a processor 160 , and a camera 170 .

[0085] Figure 2 1 is a diagram showing the structural features of the AR device 100. The structural features are Figure 2 , and the operations and / or functions of the elements included in the AR device 100 are described in Figure 3 Described in detail in.

[0086] The left-eye lens 102 may include a first left-eye variable focus lens 110-1, a second left-eye variable focus lens 120-1, and a left-eye waveguide 130-1. The right-eye lens 104 may include a first right-eye variable focus lens 110-2, a second right-eye variable focus lens 120-2, and a right-eye waveguide 130-2. Each of the first left-eye variable focus lens 110-1, the first right-eye variable focus lens 110-2, the second left-eye variable focus lens 120-1, and the second right-eye variable focus lens 120-2 may include liquid crystal molecules and may be configured as an electrically adjustable liquid crystal lens capable of changing focus according to an electrical drive signal.

[0087] The first left-eye variable focus lens 110-1 may be disposed at a position closer to the user's left eye in the left-eye lens 102, and the second left-eye variable focus lens 120-1 may be disposed farther from the user's left eye than the first left-eye variable focus lens 110-1 in the left-eye lens 102 and spaced apart from the user's left eye. In an embodiment of the present disclosure, when the user wears the AR device 100, the first left-eye variable focus lens 110-1 may be spaced apart from the left eye by a first distance, and the second left-eye variable focus lens 120-1 may be disposed spaced apart from the left eye by a second distance. The second distance may be greater than the first distance.

[0088] The left-eye waveguide 130-1 may be disposed between the first left-eye variable-focus lens 110-1 and the second left-eye variable-focus lens 120-1, and the right-eye waveguide 130-2 may be disposed between the first right-eye variable-focus lens 110-2 and the second right-eye variable-focus lens 120-2. The right-eye waveguide 130-2 is the same as the left-eye waveguide 130-1 except that it is disposed in the right-eye variable-focus lens 110-2, so the left-eye waveguide 130-1 will be described below.

[0089] When the user wears the AR device 100, the left-eye waveguide 130-1 may include a transparent material in which a partial area of ​​the rear side is visible. The rear side of the left-eye waveguide 130-1 refers to a surface that the user's eyes face when the user wears the AR device 100. Conversely, the front side of the left-eye waveguide 130-1 refers to a surface opposite to the rear side (i.e., a side away from the user's eyes).

[0090] The left-eye waveguide 130-1 may be configured as a flat plate of a single-layer or multi-layer structure of a transparent material in which light can be reflected and propagated. The left-eye waveguide 130-1 may include a plurality of regions of an emission surface facing the display module 140 to receive light of a projected virtual image from the display module 140, propagate the light, change the light path, and finally output the light to the user's eyes. A diffraction grating may be formed in the plurality of regions. The left-eye waveguide 130-1 may perform a function similar to that of a light guide plate. Fig. 9 The shape and characteristics of the left-eye waveguide 130 - 1 are described in detail.

[0091] The frame 106 may be a support structure of the AR device 100 mounted on the user's head when the user wears the AR device 100. The frame 106 may be equipped with a display module 140, a processor 160, and a camera 170. The frame 106 may include wires for electrical connection between the display module 140, the processor 160, and the camera 170.

[0092] The nose bridge 108 is a support member connecting the left-eye lens 102 and the right-eye lens 104, and may be supported by the user's nose when the user wears the AR device 100. In an embodiment of the present disclosure, the frame 106 or the nose bridge 108 may have a built-in microphone that records sound and transmits the recorded voice signal to the processor 160.

[0093] The display module 140 may project light of a virtual image toward the left-eye waveguide 130-1. The display module 140 may be disposed on the frame 106, but the arrangement of the display module 140 is not limited thereto. The display module 140 may combine the virtual image generated by the processor 160 with the light to project the combined virtual image onto the left-eye waveguide 130-1 and the right-eye waveguide 130-2 through the emission surface. In an embodiment of the present disclosure, the display module 140 may perform a function similar to that of a projector.

[0094] The display module 140 may include a light source and an image panel. The light source is an optical element that irradiates light, and light can be generated by adjusting RGB colors. The light source may be configured as, for example, a light emitting diode (LED). The image panel may be configured as a reflective image panel that modulates and reflects the light irradiated by the light source as light including a two-dimensional (2D) image. The reflective image panel may be, for example, a digital micromirror device (DMD) panel or a liquid crystal on silicon (LCoS) panel or another known reflective image panel.

[0095] The display module 140 may obtain image data constituting a virtual image from the processor 160, generate a virtual image based on the obtained image data, combine the virtual image with the light output from the light source, and project the light of the combined virtual image toward the left-eye waveguide 130-1 and the right-eye waveguide 130-2 through the emission surface. In an embodiment of the present disclosure, the processor 160 may provide the display module 140 with image data including RGB colors and brightness values ​​of a plurality of pixels constituting the virtual image. The display module 140 may perform image processing and control the light source using the RGB color value and brightness value of each of the plurality of pixels, thereby projecting the light of the virtual image onto the left-eye waveguide 130-1 and the right-eye waveguide 130-2.

[0096] The display module 140 may be configured as a plurality of display modules to project a virtual image according to both eyes. In an embodiment of the present disclosure, the AR device 100 may include a display module 140 that projects a virtual image onto the right-eye waveguide 130-2. However, the display module 140 is not limited thereto and may be configured as one display module. When the AR device 100 includes one display module 140, the AR device 100 may further include a reflective member that reflects light of a virtual image projected from the display module 140 and propagates the reflected light of the virtual image to the left-eye waveguide 130-1 and the right-eye waveguide 130-2. For example, the reflective member may be configured as a mirror.

[0097] Will refer to Fig. 9 The display module 140 is described in detail.

[0098] The left eye tracker 150-1 may be disposed in a left portion of the frame 106 supporting the left eye lens 102. The right eye tracker 150-2 may be disposed in a right portion of the frame 106 supporting the right eye lens 104. The left eye tracker 150-1 may obtain a first eye vector indicating a sight direction of the left eye by tracking the position and direction of the left eye of the user. The right eye tracker 150-2 may obtain a second eye vector indicating a sight direction of the right eye by tracking the position and direction of the right eye of the user.

[0099] In an embodiment of the present disclosure, the left eye tracker 150-1 and the right eye tracker 150-2 may obtain the user's eye vector based on a technique of detecting the sight line direction using corneal reflection of infrared rays. In an embodiment of the present disclosure, the left eye tracker 150-1 may include an infrared radiator 152 and an infrared detector 154, and the right eye tracker 150-2 may include an infrared radiator 156 and an infrared detector 158. The infrared radiators 152 and 156 may irradiate infrared light to the corneal portions of the left eye and the right eye, respectively, and the infrared detectors 154 and 158 may detect the infrared light reflected from the corneas of the left eye and the right eye. The left eye tracker 150-1 and the right eye tracker 150-2 may determine the sight line direction of the user's eyes through the amount of infrared light detected by the infrared detectors 154 and 158, respectively, and obtain an eye vector indicating the respective sight line direction of each eye. The left eye tracker 150-1 may provide the first eye vector to the processor 160. The right eye tracker 150 - 2 may provide the second eye vector to the processor 160 .

[0100] The processor 160 may be located within the frame 106. The processor 160 may be composed of one or more. The processor 160 may obtain a first eye vector and a second eye vector from the left eye tracker 150-1 and the right eye tracker 150-2, and estimate a gaze point G viewed by both eyes based on the first eye vector and the second eye vector. In an embodiment of the present disclosure, the processor 160 may calculate a three-dimensional (3D) position coordinate value of the gaze point G based on the first eye vector and the second eye vector. In an embodiment of the present disclosure, the processor 160 may determine a focus position based on the 3D position coordinate value of the gaze point G.

[0101] The processor 160 may determine the positions of the first focus area 112 and the second focus area 122 so that the first focus area 112 and the second focus area 122 are aligned along the first eye vector. In an embodiment of the present disclosure, the processor 160 may obtain a two-dimensional (2D) position coordinate value of an area reached by the first eye vector in the entire area of ​​the first left-eye variable-focus lens 110-1, and determine the position of the first focus area 112 based on the 2D position coordinate value. Similarly, the processor 160 may obtain a 2D position coordinate value of an area reached by the first eye vector in the entire area of ​​the second left-eye variable-focus lens 120-1, and determine the position of the second focus area 122 based on the 2D position coordinate value.

[0102] The processor 160 may determine the positions of the third focus area 114 and the fourth focus area 124 so that the third focus area 114 and the fourth focus area 124 are aligned along the second eye vector. In an embodiment of the present disclosure, the processor 160 may determine the position of the first focus area 112 by changing the phase of the control voltage applied to the first left-eye variable focus lens 110-1, and may determine the position of the second focus area 112 by changing the phase of the control voltage applied to the second left-eye variable focus lens 120-1. The processor 160 may determine the area corresponding to the first focus area 112 by adjusting the applied control voltage via a plurality of excitation electrodes disposed on the liquid crystal layer of the first left-eye variable focus lens 110-1. Referring to Fig. 6A and Figure 6B A specific method of determining the position of the focus area according to the phase change of the control voltage performed by the processor 160 is described in detail.

[0103] The processor 160 may change the focal length of the virtual object based on the 3D position information value of the gaze point G. In an embodiment of the present disclosure, the processor 160 may calculate the distance at which the eyes of both eyes converge according to the binocular parallax, that is, the convergence distance d between the user's eyes 30 and the gaze point G. con Therefore, the processor 160 may calculate the convergence distance d based on the convergence distance d. con Change the focal length d of the virtual object fIn the disclosed embodiment, the processor 160 may change the vergence of the first left-eye variable-focus lens 110-1 by adjusting the refractive power of the first focusing area 112 of the first left-eye variable-focus lens 110-1, and may change the vergence of the first right-eye variable-focus lens 110-2 by adjusting the refractive power of the third focusing area 114 of the first right-eye variable-focus lens 110-2.

[0104] In an embodiment of the present disclosure, the processor 160 may adjust the refractive power of the first focusing area 112 by changing the arrangement angle of the liquid crystal molecules in the area corresponding to the position of the first focusing area 112. The processor 160 may change the optical path of the virtual object displayed on the left-eye waveguide 130-1 and passing through the eye lens 32 of the left eye by adjusting the refractive power of the first focusing area 112. Because the optical path of the virtual object is changed, the focal length d of the virtual object formed on the retina 34 of the left eye is f Because the processor 160 adjusts the refractive power of the first focusing area 112, the vergence of the first left-eye variable focus lens 110-1 may change, thereby physically forming a focal length d on the left-eye waveguide 130-1. f Can be adjusted to the vergence distance d con The processor 160 may also adjust the refractive power of the third focusing area 114 relative to the first right eye variable focus lens 110 - 2 in the same manner as the first left eye variable focus lens 110 - 1 , so that the virtual object formed on the retina 36 of the right eye may also be changed in the same manner.

[0105] Because the vergence of the first left-eye variable-focus lens 110-1 is changed by the adjustment of the refractive power of the first focus area 112, focus distortion in which the real-world object appears out of focus may occur. In order to compensate for the focus distortion, the processor 160 may adjust the refractive power of the second focus area 122 of the second left-eye variable-focus lens 120-1. In an embodiment of the present disclosure, the processor 160 may adjust the refractive power of the second focus area 122 so that the second focus area 122 forms a complementary vergence for the vergence caused by the adjusted refractive power of the first focus area 112. In an embodiment of the present disclosure, the processor 160 may adjust the refractive power of the second focus area 122 to be the same as the adjusted refractive power of the first focus area 112 in a direction opposite to the direction of the refractive power of the first focus area 112. For example, when the first focus area 112 is adjusted to a refractive power of -1 diopter D, the second focus area 122 may be adjusted to +1 diopter D.

[0106] The camera 170 may be disposed on the frame 106. The camera 170 may obtain video data and still image data by capturing a physical environment or space. The camera 170 may send the obtained video data and still image data to the processor 160. In an embodiment of the present disclosure, the camera 170 may store the video data and still image data in the storage 190 (see Figure 3 )middle.

[0107] Figure 3 is a block diagram showing elements of the AR device 100 according to an embodiment of the present disclosure.

[0108] Reference Figure 3 , the AR device 100 may include a first varifocal lens 110, a second varifocal lens 120, a waveguide 130, a display module 140, an eye tracker 150, a processor 160, a memory 162, a camera 170, a depth sensor 172, a position sensor 180, and a storage 190. The first varifocal lens 110, the second varifocal lens 120, the display module 140, the eye tracker 150, the processor 160, the memory 162, the camera 170, the depth sensor 172, and the storage 190 may be electrically and / or physically connected to each other. Figure 3 The components shown have Figure 2 Elements with the same reference numerals as shown Figure 2 The components shown are the same. Therefore, redundant description will be omitted.

[0109] Each of the first variable-focus lens 110 and the second variable-focus lens 120 may include liquid crystal molecules, and may be configured as an electrically adjustable liquid crystal lens capable of changing focus according to an electrical driving signal.

[0110] Each of the first variable focus lens 110 and the second variable focus lens 120 can change the arrangement angle of the liquid crystal molecules arranged in a specific area according to the applied control voltage, so the position of the focus adjustment area capable of locally changing the refractive power can be moved on the lens. The control voltage can be controlled by the processor 160 and can be applied to each of the first variable focus lens 110 and the second variable focus lens 120 through the voltage control circuit. This will be described in reference to Fig. 6A and Figure 6B Described in detail in the description.

[0111] The first varifocal lens 110 may be configured as a plurality of varifocal lenses. For example, when the user wears the AR device 100, the plurality of first varifocal lenses 110 may include a first left-eye varifocal lens 110-1 disposed in an area corresponding to the left eye and a first right-eye varifocal lens 110-2 disposed in an area corresponding to the right eye. Although the first varifocal lens 110 is illustrated as a single block including the first left-eye varifocal lens 110-1 and the first right-eye varifocal lens 110-2, it is understood that the first left-eye varifocal lens 110-1 is disposed in an area corresponding to the left eye and the first right-eye varifocal lens 110-2 is disposed in an area corresponding to the right eye, as shown in FIG. Figure 2 shown.

[0112] The second varifocal lens 120 may be configured as a plurality of varifocal lenses. For example, when the user wears the AR device 100, the plurality of second varifocal lenses 120 may include a second left-eye varifocal lens 120-1 disposed in an area corresponding to the left eye and a second right-eye varifocal lens 120-2 disposed in an area corresponding to the right eye. Although the second varifocal lens 120 is illustrated as a single block including the second left-eye varifocal lens 120-1 and the second right-eye varifocal lens 120-2, it is understood that the second left-eye varifocal lens 120-1 is disposed in an area corresponding to the left eye and the second right-eye varifocal lens 120-2 is disposed in an area corresponding to the right eye, as shown in FIG. Figure 2 shown.

[0113] The waveguide 130 is an optical element formed of a transparent material. When the user wears the AR device 100, the waveguide 130 may include a transparent material in which a partial area of ​​the rear side is visible. The waveguide 130 may be configured as a flat plate of a single-layer or multi-layer structure of a transparent material in which light can be reflected and propagated through. The waveguide 130 may include a plurality of regions facing an emitting surface of the display module 140 to receive light of a projected virtual image from the display module 140. The light of the virtual image projected onto the waveguide 130 may propagate in the waveguide 130 according to the principle of total reflection. The waveguide 130 may include a plurality of regions for changing the path of light and ultimately outputting the light to the user's eyes. Diffraction gratings may be formed in the plurality of regions. The waveguide 130 may perform a function similar to that of a light guide plate. Reference will be made to Fig. 9 The specific shape and characteristics of the waveguide 130 are described in detail.

[0114] The waveguide 130 may be configured as a plurality of waveguides. The waveguide 130 may include a left-eye waveguide 130-1 and a right-eye waveguide 130-2. Although the waveguide 130 is shown as a single block including the first left-eye waveguide 130-1 and the right-eye waveguide 130-2, it is understood that the left-eye waveguide 130-1 is disposed in a region corresponding to the left eye and the right-eye waveguide 130-2 is disposed in a region corresponding to the right eye, as shown in FIG. Figure 2shown.

[0115] In the embodiment of the present disclosure, the first left-eye variable-focus lens 110-1, the second left-eye variable-focus lens 120-1 and the left-eye waveguide 130-1 may constitute the left-eye lens 102 (see Figure 2 Similarly, in the embodiment of the present disclosure, the first right-eye variable-focus lens 110-2, the second right-eye variable-focus lens 120-2 and the right-eye waveguide 130-2 may constitute the right-eye lens 104 (see Figure 2 ).

[0116] The display module 140 may project light of a virtual image toward the waveguide 130. In an embodiment of the present disclosure, the display module 140 may project light of a virtual image using a transmissive projection technique in which a light source is modulated by an optically active material that emits white light. The display module 140 may project the same virtual image onto the left-eye waveguide 130-1 and the right-eye waveguide 130-2, or may project different virtual images onto the left-eye waveguide 130-1 and the right-eye waveguide 130-2. Here, the virtual image may be generated by the processor 160. Reference will be made to Fig. 9 The display module 140 is described in detail.

[0117] The eye tracker 150 can obtain an eye vector indicating the direction of the user's line of sight by tracking the position and direction of the user's eyes. In an embodiment of the present disclosure, the eye tracker 150 can obtain the user's eye vector by using a technology that detects the line of sight direction using corneal reflection of infrared rays. The eye tracker 150 can provide the obtained eye vector to the processor 160.

[0118] The eye tracker 150 may include a left eye tracker 150-1 that obtains a first eye vector about the left eye and a right eye tracker 150-2 that obtains a second eye vector about the right eye. Although the eye tracker 150 is shown as a single block including the left eye tracker 150-1 and the right eye tracker 150-2, it is understood that the left eye tracker 150-1 is disposed in an area corresponding to the left eye to be tracked and the right eye tracker 150-2 is disposed in an area corresponding to the right eye to be tracked, as shown in FIG. Figure 2 The specific method performed by the eye tracker 150 to obtain the eye vectors related to the binocular sight directions will be described in detail in FIG. 5A to FIG. 5D Described in detail in the description.

[0119] The processor 160 may control the overall functions and / or operations performed by the AR device 100 by executing one or more instructions of a program stored in and read from the memory 162. The processor 160 may include hardware elements that perform arithmetic operations, logical operations, and input / output operations and signal processing.

[0120] Processor 160 may include at least one of hardware such as a central processing unit, a microprocessor, a graphics processing unit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal (DSPD), a digital signal processing device (DSPD), a programmable logic device (PLD) or a field programmable gate array (FPGA), but is not limited thereto.

[0121] The memory 162 may store a program including one or more instructions. The memory 162 may include at least one type of hardware device, such as a flash memory type hardware device, a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read (EPEROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, or an optical disk. The processor 160 may be composed of one or more.

[0122] The processor 160 can change the arrangement angle of the liquid crystal molecules set in the first focusing area by applying a control voltage to the first left-eye variable focus lens 110-1, so that the refractive power of the first focusing area can be adjusted, thereby adjusting the refractive index of the light passing through the first focusing area. The processor 160 can adjust the vergence of the first left-eye variable focus lens 110-1 by adjusting the refractive power of the first focusing area. Vergence is an index indicating the degree of convergence or divergence of light, and can be adjusted according to the refractive power of the lens. In an embodiment of the present disclosure, the processor 160 can adjust the vergence of the first left-eye variable focus lens 110-1 by adjusting the refractive power of the first focusing area in the first direction, and adjust the focal length of the virtual object VO. When the vergence of the first focusing area is adjusted in the divergent direction, the path of the light passing through the first focusing area can be increased, so that the focal length of the virtual object VO formed on the left eye retina can be increased. In an embodiment of the present disclosure, the processor 160 may adjust the vergence of the first focusing area, so that the focal length of the virtual object VO may be adjusted to be the same as the convergence distance of both eyes.

[0123] The processor 160 may change the arrangement angle of the liquid crystal molecules disposed in the third focusing area by applying a control voltage to the first right-eye variable-focus lens 110 - 2 , and thus may adjust the refractive power of the third focusing area.

[0124] In an embodiment of the present disclosure, the processor 160 may calculate the 3D position coordinate value of the gaze point where the sight lines of both eyes converge by using both the first eye vector obtained by the left eye tracker 150-1 and the second eye vector obtained by the right eye tracker 150-2. Therefore, based on the convergence distance, which is the distance between the two eyes and the gaze point, the processor 160 may adjust the refractive power of the first focusing area of ​​the first left-eye variable focus lens 110-1 and the third focusing area of ​​the first right-eye variable focus lens 110-2. The processor 160 may adjust the refractive power of the first focusing area to change the focal length of the virtual object VO displayed through the left-eye waveguide 130-1 to be the same as the convergence distance. Similarly, the processor 160 may adjust the refractive power of the third focusing area to change the focal length of the virtual object VO displayed through the right-eye waveguide 130-2 to be the same as the convergence distance.

[0125] In an embodiment of the present disclosure, the processor 160 may store information about the 3D position coordinate value of the gaze point in the memory 160 and / or the storage 190 .

[0126] In an embodiment of the present disclosure, the first left-eye variable-focus lens 110-1 may include a plurality of first focus areas, and the processor 160 may adjust the refractive power of the plurality of first focus areas to have different focal lengths according to the plurality of first focus areas. In an embodiment of the present disclosure, the first right-eye variable-focus lens 110-2 may include a plurality of third focus areas, and the processor 160 may adjust the refractive power of the plurality of third focus areas to have different focal lengths according to the plurality of third focus areas.

[0127] The processor 160 may change the arrangement angle of the liquid crystal molecules disposed in the second focus area by applying a control voltage to the second left-eye variable-focus lens 120-1, and thus may adjust the refractive index of the light passing through the second focus area, thereby adjusting the refractive power of the second left-eye variable-focus lens 120-1. In an embodiment of the present disclosure, in order to compensate for the focus distortion in which the real-world object is perceived as blurred due to the adjusted refractive power of the first focus area, the processor 160 controlling the second left-eye variable-focus lens 120-1 may complementarily adjust the refractive power of the second focus area of ​​the second left-eye variable-focus lens 120-1 relative to the adjusted refractive power of the first focus area.

[0128] In an embodiment of the present disclosure, the processor 160 may adjust the refractive power of the second focus area to be the same as the adjusted refractive power of the first focus area in a direction opposite to the adjusted refractive power of the first focus area. For example, when the first focus area is adjusted to a refractive power of -1 diopter D, the processor 160 may adjust the second focus area to +1 diopter D.

[0129] The processor 160 can adjust the refractive power of the fourth focusing area to be the same as the adjusted refractive power of the third focusing area in a direction opposite to the direction of the refractive power of the third focusing area, so that the fourth focusing area of ​​the second right-eye variable-focus lens 120-2 forms a convergence complementary to the convergence of the third focusing area of ​​the first right-eye variable-focus lens 110-2.

[0130] In an embodiment of the present disclosure, the processor 160 may independently adjust the refractive power of the second focus area regardless of the refractive power of the first focus area. In an embodiment of the present disclosure, the processor 160 may adjust the refractive power of the second focus area of ​​the second left-eye variable focus lens 120-1 to change the focus of the real-world object viewed through the second left-eye variable focus lens 120-1. In an embodiment of the present disclosure, when the user experiences hyperopia or myopia, the processor 160 may adjust the refractive power of the second focus area of ​​the second left-eye variable focus lens 120-1 and the fourth focus area of ​​the second right-eye variable focus lens 120-2 for the purpose of correcting the user's vision. In this case, the processor 160 may adjust the refractive power of the second focus area and the fourth focus area based on the refractive index input by the external input, or may adjust the refractive power of the second focus area and the fourth focus area to have a refractive index previously stored in the memory 162 and / or the storage 190.

[0131] The processor 160 may determine the position of the first focus area of ​​the first variable focus lens 110 based on the eye vector obtained from the eye tracker 150. The processor 160 may determine the positions of the first focus area and the second focus area so that the first focus area of ​​the first left-eye variable focus lens 110-1 and the second focus area of ​​the second left-eye variable focus lens 120-1 are aligned according to the direction of the first eye vector. In an embodiment of the present disclosure, the processor 160 may obtain a two-dimensional (2D) position coordinate value of an area reached by the first eye vector in the entire area of ​​the first left-eye variable focus lens 110-1, and determine the position of the first focus area 112 based on the 2D position coordinate value. Similarly, the processor 160 may obtain a 2D position coordinate value of an area reached by the first eye vector in the entire area of ​​the second left-eye variable focus lens 120-1, and determine the position of the second focus area 122 based on the 2D position coordinate value. Similarly, the processor 160 may determine positions of the third and fourth focus areas such that the third and fourth focus areas of the first right-eye variable-focus lens 110-2 are aligned according to the direction of the second eye vector.

[0132] The processor 160 may determine the size of the first focus area and the second focus area. In an embodiment of the present disclosure, the processor 160 may determine the size of the first focus area and the second focus area based on the size of the virtual object VO projected onto the user's eye through the left-eye waveguide 130-1. In an embodiment of the present disclosure, the processor 160 may determine the size of the first focus area and the second focus area based on the size of the virtual object VO and the distance between the first left-eye variable focus lens 110-1 and the second left-eye variable focus lens 120-1. In an embodiment of the present disclosure, the processor 160 may determine the size of the third focus area and the fourth focus area based on the size of the virtual object VO projected through the right-eye waveguide 130-2. In an embodiment of the present disclosure, the processor 160 may determine the size of the third focus area and the fourth focus area based on the size of the virtual object VO and the distance between the first right-eye variable focus lens 110-2 and the second right-eye variable focus lens 120-2.

[0133] The camera 170 can obtain video data and still image data by capturing the physical environment or space viewed by the user. The camera 170 can send the video data and still image data to the processor 160. In an embodiment of the present disclosure, the camera 170 can visually monitor the user's surrounding space. In an embodiment of the present disclosure, the camera 170 can perform one or more controls or operations within an application controlled by the processor 160, or can capture gestures or actions performed by the user and real-world objects in the surrounding space to operate for providing input within the application.

[0134] The depth sensor 172 may be a sensor and / or a camera that measures the depth value of a real-world object viewed by the user. The depth sensor 172 may scan a physical space or environment, measure the depth value of a real-world object set in the physical space or environment according to the 3D position coordinate value of the real-world object, and measure the measured depth value in each 3D position coordinate value to generate a depth map. The depth sensor 172 may store the depth map in the memory 162 and / or the storage 190.

[0135] In an embodiment of the present disclosure, the depth sensor 172 may obtain a depth image including depth values ​​of real-world objects. The depth image includes a 2D pixel area of ​​the captured scene, wherein each pixel in the 2D pixel area may represent, for example, a depth h value in units of centimeters (cm), millimeters (mm), etc., such as the distance of the real-world object in the captured scene, by using the depth sensor 172.

[0136] The depth sensor 172 can measure 3D depth values ​​by using, for example, any one of stereo, time of flight (ToF), and structured modes. In an embodiment of the present disclosure, the depth sensor 172 may include an RGB camera, an infrared light element, and a 3D camera that can be used to capture depth images about real-world objects.

[0137] The depth sensor 172 may send a depth image including depth values ​​of real-world objects to the processor 160 or store the depth image in the memory 162 and / or the storage 190. In an embodiment of the present disclosure, the processor 160 may obtain a depth image from the depth sensor 172 or obtain depth values ​​of real-world objects by loading a depth map stored in the memory 162 and / or the storage 190, and may adjust the refractive power of the first focusing area based on the obtained depth value to change the focal length of the virtual object VO. In an embodiment of the present disclosure, the processor 160 may adjust the vergence of the first left-eye variable-focus lens 110-1 by adjusting the refractive power of the first focusing area, and adjust the focal length of the virtual object VO by adjusting the vergence. Reference will be made to Fig.13A and Fig. 13B A specific method of adjusting the focal length of a virtual object VO using a depth value of a real-world object is described in detail.

[0138] The position sensor 180 may obtain position information of the AR device 100. In an embodiment of the present disclosure, the position sensor 180 may obtain the current location or geographic location coordinates of the AR device 100. For example, the position sensor 180 may include a GPS sensor.

[0139] The storage 190 may store at least one of a virtual image generated by the processor 160, an image of a real-world object captured by the camera 170, or a depth image or a depth map about a real-world object captured by the depth sensor 172. In an embodiment of the present disclosure, the storage 190 may store information about the refractive power of the first focusing area of ​​the first left-eye variable-focus lens 110-1 and the second focusing area of ​​the second left-eye variable-focus lens 120-1. In an embodiment of the present disclosure, the storage 190 may store information about the refractive power of the third focusing area of ​​the first right-eye variable-focus lens 110-2 and the fourth focusing area of ​​the second right-eye variable-focus lens 120-2.

[0140] Storage 190 may include at least one type of storage medium, such as flash memory, hard disk memory, multimedia card micro memory, card-type memory (e.g., SD or XD memory, etc.), magnetic memory, disk or optical disk, but storage 190 is not limited to the above examples.

[0141] Figure 4is a flowchart illustrating a method of operating the AR device 100 according to an embodiment of the present disclosure.

[0142] In operation S410, the AR device 100 may use a first eye tracker to obtain a first eye vector about a first sight line direction of the left eye, and use a second eye tracker to obtain a second eye vector about a second sight line direction of the right eye. The first eye tracker and the second eye tracker may obtain a first eye vector and a second eye vector indicating a sight line direction of the user by tracking the respective positions and directions of pupils of both eyes, and provide the first eye vector and the second eye vector to the AR device 100. In an embodiment of the present disclosure, the eye tracker may obtain the eye vector of the user by using a technique for detecting a sight line direction using corneal reflection of infrared rays. In an embodiment of the present disclosure, the eye tracker may use visual technology to obtain an image of the pupil, use the image of the pupil to track changes in the position of the pupil, and obtain an eye vector based on the changes in the position. The eye tracker may provide a signal to the processor 160 (see Figure 3 ) provides data regarding the vector value and direction of each of the obtained first eye vector and second eye vector.

[0143] In operation S420, the AR device 100 may determine the position of the first focus area of ​​the first left-eye variable-focus lens based on the first eye vector, and may determine the position of the third focus area of ​​the first right-eye variable-focus lens based on the second eye vector. The first left-eye variable-focus lens and the first right-eye variable-focus lens may be liquid crystal lenses that can adjust the refractive index of light passing through the first focus area and the third focus area by changing the arrangement angles of liquid crystal molecules respectively arranged in the first focus area and the third focus area according to the phase of the control voltage applied thereto. In an embodiment of the present disclosure, the first left-eye variable-focus lens and the first right-eye focusing lens may be configured as electrically adjustable liquid crystal lenses that can change the focus according to an electrical drive signal.

[0144] In the embodiment of the present disclosure, the processor 160 (see Figure 3 ) may calculate position coordinate values ​​of a first left-eye variable-focus lens and a first right-eye variable-focus lens through which the left eye and the right eye of the user are focused based on data about the vector value and direction of each of the first eye vector and the second eye vector obtained from the eye tracker. The processor 160 may determine each of the first focus area and the third focus area based on the calculated position coordinate values.

[0145] The AR device 100 may adjust the refractive power for a first focus area, which is an area through which the first eye vector passes in the entire area of ​​the first left-eye variable-focus lens. Similarly, the AR device 100 may adjust the refractive power for a third focus area, which is an area through which the second eye vector passes in the entire area of ​​the first right-eye variable-focus lens. Because the direction of the first eye vector changes, the position of the first focus area may change. Similarly, because the direction of the second eye vector changes, the position of the third focus area may change.

[0146] In operation S430, the AR device 100 may obtain a gaze point where the first eye vector and the second eye vector converge according to the binocular parallax. In an embodiment of the present disclosure, the gaze point may be obtained by using triangulation. The AR device 100 may calculate a 3D position coordinate value of the gaze point based on the distance between the two eyes, the first eye vector and the second eye vector. The AR device 100 may calculate a convergence distance, which is the distance between the two eyes and the obtained gaze point.

[0147] In operation S440, the AR device 100 may adjust the refractive power of the first focus area and the third focus area based on the gaze point. In an embodiment of the present disclosure, the AR device 100 may adjust the refractive power of the first focus area and the third focus area based on the vergence distance to adjust the focal length of the virtual object. The vergence formed in the first focus area and the third focus area may be adjusted by adjusting the refractive power of the first focus area and the third focus area.

[0148] In an embodiment of the present disclosure, the AR device 100 may obtain the vergence distance of the real-world object set at the gaze point, and adjust the refractive power of the first focusing area and the third focusing area based on the vergence distance so that the virtual object has the same focal length as the real-world object. The AR device 100 may change the vergence formed in the first focusing area to a divergent direction by adjusting the refractive power of the first focusing area, thereby adjusting the focal length of the virtual object formed on the retina of the left eye. Similarly, the AR device 100 may change the vergence formed in the third focusing area to a divergent direction by adjusting the refractive power of the third focusing area, thereby adjusting the focal length of the virtual object formed on the retina of the right eye.

[0149] In the embodiment of the present disclosure, the processor 160 (see Figure 3) can adjust the refractive power of the first focus area by controlling the amount of control voltage applied to the plurality of excitation electrodes of the first left-eye variable focus lens. The processor 160 can apply the control voltage to the excitation electrode corresponding to the position of the first focus area (which is the target area whose refractive power is to be adjusted) among the plurality of excitation electrodes of the first left-eye variable focus lens, and adjust the refractive power of the first focus area by adjusting the amount of the control voltage. The specific method of determining the position of the focus area of ​​the focus lens and adjusting the refractive power of the focus area performed by the processor 160 will be referred to in Fig. 6A and Figure 6B Detailed description.

[0150] In an embodiment of the present disclosure, the AR device 100 can obtain a depth value of a real-world object included in a depth image captured by using a depth sensor, and adjust the refractive power of the first focusing area and the third focusing area so that the focal length of the virtual object is adjusted based on the obtained depth value of the real-world object.

[0151] In operation S450, the AR device 100 may complementarily adjust the refractive power of each of the second focus area of ​​the second left-eye variable focus lens and the fourth focus area of ​​the second right-eye variable focus lens relative to the refractive power of each of the first focus area and the third focus area. In an embodiment of the present disclosure, the AR device 100 may adjust the refractive power of the second focus area and the fourth focus area so that the second focus area and the fourth focus area complementarily adjust the vergence relative to the vergence of the first focus area and the third focus area, respectively. Although the focal length of the virtual object is changed by the vergence of the first focus area adjusted in operation S440, the focal length relative to the real-world object is also changed by the vergence of the first focus area, which may cause focus distortion in which the real-world object appears blurred. In order to compensate for the focus distortion of the real-world object caused by the adjusted convergence of the first focus area of ​​the first left-eye variable-focus lens and the third focus area of ​​the first right-eye variable-focus lens, the AR device 100 can adjust the refractive power of the second focus area of ​​the second left-eye variable-focus lens and the fourth focus area of ​​the second right-eye variable-focus lens to be the same as the adjusted refractive power of the first focus area and the third focus area in the direction opposite to the direction of the refractive power of the first focus area and the third focus area, respectively. For example, when the first focus area is adjusted to a refractive power of -1 diopter D, the AR device 100 can adjust the refractive power of the second focus area to +1 diopter D. Similarly, when the third focus area is adjusted to a refractive power of -2 diopter D, the AR device 100 can adjust the refractive power of the fourth focus area to +2 diopter D.

[0152] In operation S460, the AR device 100 may project light of a virtual image toward a waveguide. The waveguide may be an optical element formed of a transparent material in which light may be reflected and propagated. The waveguide may be configured as a flat plate of a single layer or a multi-layer structure. The waveguide may perform the function of a light guide plate that changes the direction of light through the principle of diffraction of light and transmits the light to the user's eyes. The waveguide may include a display module 140 (see Figure 3 ) A first area receiving virtual image light, a second area in which the virtual image light incident on the first area propagates, and a third area outputting the virtual image by changing the direction of the light of the virtual image propagating in the second area to the direction of the user's eyes. Diffraction gratings may be formed in the first area, the second area, and the third area. Due to the diffraction gratings formed in the first area, the second area, and the third area of ​​the waveguide, the light of the virtual image projected from the display module 140 can be transmitted to the user's eyes, so that the user can observe the virtual object of the virtual image. The structure and light guiding principle of the waveguide will be referred to Fig. 9 Detailed description.

[0153] Display module 140 (see Figure 3 ) can project light of a virtual image toward the waveguide so that the virtual object is displayed on the display by the processor 160 (see Figure 3 ) is at a position coordinate value determined based on a first eye vector and a second eye vector obtained by an eye tracker.

[0154] The AR device 100 may determine the eye vector as the eye vector through the waveguide 130 (see Figure 3 ) outputted by the processor 160. In an embodiment of the present disclosure, the processor 160 may obtain the 2D position coordinate value of the area reached by the first eye vector in the entire area of ​​the left-eye waveguide 130-1 from the eye tracker 150, and form a virtual image so that the virtual object is displayed at the 2D position coordinate value. The processor 160 may provide the generated image data of the virtual image to the display module 140. The processor 160 may combine the virtual image with light by controlling the light source, and control the display module 140 to project the combined virtual image, so that the virtual object of the virtual image is displayed on the area corresponding to the 2D position coordinate value of the left-eye waveguide 130-1.

[0155] Figure 5A An eye tracker 150 of an AR device according to an embodiment of the present disclosure is shown.

[0156] Reference Figure 5A , the eye tracker 150 may include an infrared radiator 152 and a plurality of infrared detectors 154a, 154b, 154c, 154d, 154e, and 154f. Figure 5ASix infrared detectors 154a to 154f are shown in FIG. 1 , but this is for convenience of description, and the number of the plurality of infrared detectors 154a to 154f and their positioning are not limited to this configuration.

[0157] The infrared radiator 152 may irradiate infrared light to a portion of the cornea corresponding to a position where the lens 32 of the eye 30 is set. A plurality of infrared detectors 154a to 154f may detect infrared light reflected from the cornea. In an embodiment of the present disclosure, the infrared radiator 152 may include a reflector that changes the path of the infrared light to irradiate the infrared light to the direction of the eye 30. In an embodiment of the present disclosure, the eye tracker 150 may obtain information about the amount of infrared light detected by each of the plurality of infrared detectors 154a to 154f, determine the line of sight direction of the user's eye 30 based on the obtained amount of infrared light, and obtain an eye vector indicating the line of sight direction. The eye tracker 150 may provide information to the processor 160 (see Figure 3 ) provides data indicating the vector value and direction of the obtained eye vector.

[0158] Figure 5B An eye tracker of an AR device according to an embodiment of the present disclosure is shown.

[0159] Reference Figure 5B , the eye tracker 153 may track the user's eyes based on the positions of reflected lights 511, 512, 513, 514, and 515 reflected from the user's eyes 30, and obtain an eye vector based on the positions of reflected lights 511, 512, 513, 514, and 515 reflected from the user's eyes 30. The eye tracker 153 may include a light source 153a and a camera 153b.

[0160] The light source 153a may include an infrared light emitting diode (IR LED). Figure 5B In the embodiment shown in FIG. 1 , the light source 153 a may include a plurality of LEDs disposed at different positions. When an image of the eye 30 is captured by the camera 153 b, the light source 153 a may provide light (e.g., infrared light) to the user's eye 30. Because the light is provided to the user's eye 30, reflected light reflected from the user's eye 30 may be generated.

[0161] The camera 153b may be configured to include at least one camera. The camera 153b may be implemented as an infrared camera (IR). The AR device may track the view of the user's eye 30 using images 501, 502, 503, 504, and 505 of the user's eye 30 captured by the camera 153b. For example, the eye tracker 153 may track the user's line of sight by detecting the pupil 500 and reflected light 511 to 515 from the images 501 to 505 of the user's eye 30, thereby obtaining an eye vector. The eye tracker 153 may detect the positions of the pupil 500 and reflected light 511 to 515 from the images 501 to 505 of the user's eye 30, and determine the line of sight direction of the user's eye 30 based on the relationship between the position of the pupil 500 and the position of the reflected light 511 to 515.

[0162] For example, the eye tracker 153 may detect the pupil 500 and the reflected light 511 from the captured first eye image 501, and determine the sight direction 521 of the user's eye 30 based on the relationship between the position of the pupil 500 and the position of the reflected light 511. In the same manner, the eye tracker 153 may detect the pupil 500 and the reflected light 512, 513, 514, and 515 from the second to fifth eye images 502, 503, 504, and 500, respectively, and determine the sight directions 522, 523, 524, and 525 of the user's eye 30 based on the relationship between the position of the pupil 500 and the positions of the reflected light 512, 513, 514, and 515.

[0163] In an embodiment of the present disclosure, the eye tracker 153 may obtain an eye vector based on information about the line of sight direction. The eye tracker 153 may provide the processor 160 (see Figure 3 ) provides data indicating the vector value and direction of the eye vector.

[0164] In another embodiment of the present disclosure, the eye tracker 153 may provide only coordinate values ​​regarding the position of the pupil 500 and the positions of the reflected lights 511 to 515 detected from the plurality of eye images 501 to 505 to the processor 160 (see Figure 3 ), and the processor 160 can calculate the eye vector of the user's eye 30 based on the coordinate value obtained from the eye tracker 153.

[0165] Figure 5C FIG. 3 is a diagram showing a 3D eyeball model regarding the user's line of sight.

[0166] Reference Figure 5C, the AR device can use the eye trackers 150 and 153 to determine the sight direction of the user's eye 30. For example, the AR device can determine the sight direction based on an average eyeball model of a person. The eyeball model can be modeled by assuming that the human eye 30 has a spherical shape and the eyeball 30 rotates ideally according to the sight direction. In addition, the eyeball model can be mathematically expressed as shown in the following equations 1 and 2.

[0167] [Mathematics 1]

[0168] x=d·tan α,

[0169] y=d·secα·tanβ,

[0170] [Mathematics 2]

[0171] β=sin -1 (diff_y / r),

[0172] α=sin -1 (diff_x / rcosβ).

[0173] In Equation 1, d represents the distance between the center 33 of the user's eye 30 and the virtual screen 530, α represents the rotation angle of the user's eye 30 in the x-axis direction based on the following situation: the user's eye 30 is fixed in front of the virtual screen 530, and β represents the rotation angle of the user's eye 30 in the y-axis direction based on the following situation: the user's eye 30 is fixed in front of the virtual screen 530. In addition, in Equation 2, r represents the radius of a sphere assuming that the user's eye 30 is a sphere.

[0174] The eye trackers 150 and 153 according to embodiments of the present disclosure may be used Figure 5A and Figure 5B The method described in the article can measure the rotation degree (e.g., α and β) of the user's eye 30, and the AR device can use the rotation degrees α and β of the user's eye 30 to calculate the 2D coordinate value of the line of sight of the user's eye 30 on the virtual screen 530.

[0175] Figure 5D is a reference diagram for describing a method of calibrating an eye tracker according to an embodiment of the present disclosure.

[0176] Reference Figure 5D When the user uses the AR device for the first time, the AR device may perform a process of calibrating the eye trackers 150 and 153 to accurately measure the sight lines of the left and right eyes. The AR device may output virtual images VI1, VI2, and VI3 of different depths (e.g., d1, d2, and d3) marked with multiple points to guide the user's sight lines and induce the user to gaze at each of the multiple points. Figure 5D, although 9 points are shown in each of the virtual images VI1, VI2, and VI3, this is an example, and the number of points and the positions of the points are not limited.

[0177] When the user gazes at each point included in the virtual images VI1, VI2, and VI3, the AR device may store the data (e.g., eye vectors) output from the eye trackers 150 and 153 in the form of a table in the storage 190 (see Figure 3 )middle.

[0178] like Figure 5A As shown, the method of tracking the line of sight using the amount of infrared light reflected from the cornea can store information about the reflection angle and the amount of light at each point in the form of a table in advance as line of sight information in the storage 190. Figure 5B As shown, the method of capturing the user's eyes using infrared light can store images including the user's eyes captured at each point and reflected light in the storage 190 in advance as line of sight information in a table form.

[0179] The AR device may determine the sight direction of the user's eyes by comparing the pre-stored sight information with the sight information output from the eye trackers 150 and 153. The processor 160 of the AR device (see Figure 3 ) can determine the sight lines of the left eye and the right eye using the sight line information output from the eye trackers 150 and 153. In an embodiment of the present disclosure, the processor 160 may use the sight line information output from the eye trackers 150 and 153 to calculate a first eye vector indicating the sight line direction of the left eye and a second eye vector indicating the sight line direction of the right eye.

[0180] The AR device can estimate the gaze point G by using binocular parallax and line of sight information about the line of sight direction of the left eye and the line of sight direction of the right eye (see Figure 2 ) position coordinate value. For example, processor 160 (see Figure 3 ) can use coordinate mapping to pre-set the coordinates to be mapped to 3D position coordinate values ​​(e.g., x coordinate value, y coordinate value, and z coordinate value) in Figure 2 The point (gaze point G) that the user is gazing at in the entire space described in the figure, or the 3D position coordinate value of the gaze point G can be stored in the storage 190 in the form of a table.

[0181] Fig. 6A is a perspective view of a variable-focus lens of an AR device according to an embodiment of the present disclosure.

[0182] Reference Fig. 6A, the variable focus lens 600 may include a liquid crystal layer 610 , a common electrode 620 , a transparent film 630 , and first and second excitation electrodes 640 and 650 . The variable focus lens 600 may further include a transparent layer formed to contact a lower surface of the common electrode 620 .

[0183] The variable focus lens 600 may be an electrically adjustable liquid crystal lens capable of adjusting the refractive index of light by changing the arrangement angle of the liquid crystal molecules 612 based on a control voltage applied from a power supply VAC via the first excitation electrode 640 and the second excitation electrode 650. In an embodiment of the present disclosure, the variable focus lens 600 may include an electro-optical material having a pixel grid. The pixels may be arranged in a matrix of N rows and M columns. Each N×M pixel may accommodate a set of possible gray levels independent of all other pixels.

[0184] The liquid crystal layer 610 may be an electro-optical layer including a plurality of liquid crystal molecules 612. The liquid crystal layer 610 may be an electro-optical layer whose liquid crystal properties are changed by an applied control voltage. In an embodiment of the present disclosure, the liquid crystal layer 610 may be configured as a polarization-independent liquid crystal layer (e.g., cholesteric liquid crystal). In the liquid crystal layer 610, the arrangement angle of the liquid crystal molecules 612 disposed in a specific region within the active region is changed by a control voltage applied via the first excitation electrode 640 and the second excitation electrode 650, so that the refractive index of the specific region can be locally adjusted.

[0185] The common electrode 620 and the first and second excitation electrodes 640 and 650 may receive a control voltage from a power source VAC and apply the control voltage to the liquid crystal layer 610. The common electrode 620 may contact the first surface 610-1 of the liquid crystal layer 610.

[0186] The first excitation electrode 640 and the second excitation electrode 650 may be disposed in contact with an upper surface of a transparent film 630 on a second surface 610-2 of the liquid crystal layer 610 opposite to the first surface 610-1. Each of the first excitation electrode 640 and the second excitation electrode 650 may be oriented on the upper surface of the transparent film 630 in directions orthogonal to the X-axis and Y-axis directions, respectively. Each of the first array excitation electrode 640 and the second array excitation electrode 650 may include parallel conductive material strips extending above the active area. In an embodiment of the present disclosure, the first array excitation electrode 640 and the second array excitation electrode 650 may include a transparent conductive material, such as indium tin oxide (ITO).

[0187] Pixels can be defined by the areas where strips of first array excitation electrodes 640 overlap with strips of second array excitation electrodes 650. The center-to-center distance between strips of first array excitation electrodes 640 and strips of second array excitation electrodes 650 can define the pitch of the pixel array, and the width of the strips can define the size of the pixel.

[0188] Processor 160 (see Figure 2 and Figure 3 ) can be controlled to apply a control voltage waveform having a phase modulation distribution to each of the first array excitation electrodes 640 and the second array excitation electrodes 650 through the power supply VAC, and modulate the control voltage applied to each of the first array excitation electrodes 640 and the second array excitation electrodes 650. The processor 160 can simultaneously control to modulate the control voltage waveform applied to the first array excitation electrodes 640 and the second array excitation electrodes 650, thereby generating a specific phase modulation distribution in the liquid crystal layer 610.

[0189] Because a control voltage having a waveform modulated by the processor 160 is applied, the refractive power of the variable focus lens 600 can be locally adjusted in a specific area within the active area by the phase modulation distribution of the applied control voltage. The variable focus lens 600 can be used as a lens that adjusts the vergence according to the adjusted refractive power. Here, vergence is an index indicating the degree of convergence or divergence of light, and can be adjusted according to the refractive power of the variable focus lens 600. In an embodiment of the present disclosure, the variable focus lens 600 can adjust the vergence by adjusting the refractive power of the lens to change the light or light path.

[0190] The processor 160 can change the focal length by adjusting the vergence of a specific area (ie, the focus adjustment area) of the variable focus lens 600. Figure 6B Detailed description of determining the specific area 612A of the variable focus lens 600 (see Figure 6B )'s position and the specific method of adjusting the refractive power of the specific area 612A.

[0191] Figure 6B is a perspective view for describing a method of adjusting the refractive power of a focus area performed by a variable focus lens of an AR device according to an embodiment of the present disclosure. Figure 6B In the embodiment, the variable focus lens 600 may include first variable focus lenses 110-1 and 110-2 for the left eye or the right eye (see Figure 2 ) or the second variable focus lens 120-1 and 120-2 (see Figure 2 ).

[0192] Reference Figure 6B, the variable focus lens 600 may include a liquid crystal layer 610, a common electrode 620, a plurality of first array excitation electrodes 640-1, 640-2, 640-3, 640-4 and 640-5, and a plurality of second array excitation electrodes 650-1, 650-2, 650-3, 650-4 and 650-5. Figure 6B In, with Fig. 6A The situation is different, and for the convenience of description, the transparent film 630 is not shown.

[0193] The plurality of first array excitation electrodes 640-1 to 640-5 may be arranged along the X-axis direction, and the plurality of second array excitation electrodes 650-1 to 650-5 may be arranged along the Y-axis direction. The plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5 may be arranged to be orthogonal to each other.

[0194] A plurality of first driver terminals 660-1, 660-2, 660-3, 660-4, and 660-5 that control a control voltage applied from a power supply VAC to the plurality of first array excitation electrodes 640-1 to 640-5 may be connected to each of the plurality of first array excitation electrodes 640-1 to 640-5. A plurality of second driver terminals 670-1, 670-2, 670-3, 670-4, and 670-5 that control a control voltage applied from a power supply VAC to the plurality of second array excitation electrodes 650-1 to 650-5 may be connected to each of the plurality of second array excitation electrodes 650-1 to 650-5.

[0195] The controller 680 for controlling the operation of the variable focus lens 600 may be connected to the plurality of first driver terminals 660 - 1 to 660 - 5 , the plurality of second driver terminals 670 - 1 to 670 - 5 , and a power source VAC.

[0196] The controller 680 can control the control voltage applied to the plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5 by controlling the plurality of first driver terminals 660-1 to 660-5 and the plurality of second driver terminals 670-1 to 670-5. Therefore, the arrangement angle of the liquid crystal molecules arranged in a specific area can be controlled. In an embodiment, the variable focus lens 600 may not include the plurality of first driver terminals 660-1 to 660-5 and the plurality of second driver terminals 670-1 to 670-5, and the controller 680 may be directly connected to the plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5.

[0197] The controller 680 may receive line of sight information including an eye vector value indicating a line of sight direction of the user's eye and direction information of the vector from the processor 160, and determine the position of the area A 612A whose focus is to be adjusted based on the line of sight information received from the processor 160. In an embodiment of the present disclosure, the eye tracker 150 may obtain the eye vector by tracking the line of sight direction of the user's eye, and provide the obtained eye vector to the processor 160. The processor 160 may calculate a position coordinate value of an area reached by the line of sight in the entire area of ​​the variable focus lens 600 based on the vector direction of the eye vector, and provide information about the position coordinate value to the controller 680. The controller 680 may determine the area A 612A as a target area whose focus is to be adjusted based on the position coordinate value obtained from the processor 160.

[0198] In this disclosure Figure 6B In the embodiment shown in , in order to change the arrangement angle of the liquid crystal molecules arranged in the area A 612A among the multiple liquid crystal molecules 612 included in the liquid crystal layer 610, the controller 680 can control the voltage to be applied to the 1st-2nd excitation electrode 640-2, the 1st-3rd excitation electrode 640-3 and the 1st-4th excitation electrode 640-4 among the multiple first array excitation electrodes 640-1 to 640-5 and the 2nd-2nd excitation electrode 650-2, the 2nd-3rd excitation electrode 650-3 and the 2nd-4th excitation electrode 650-4 among the multiple second array excitation electrodes 650-1 to 650-5. In an embodiment of the present disclosure, the controller 680 can control the voltage applied to the 1-2 excitation electrodes 640-2, the 1-3 excitation electrodes 640-3 and the 1-4 excitation electrodes 640-4 and the 2-2 excitation electrodes 650-2, the 2-3 excitation electrodes 650-3 and the 2-4 excitation electrodes 650-4 via the power supply VAC by controlling the 1-2 driver terminals 660-2 to the 1-4 driver terminals 660-4 and the 2-2 driver terminals 670-2 to the 2-4 driver terminals 670-4. In this configuration, the controller 680 can control the voltage not to be applied to the 1-1th excitation electrode 640-1 and the 1-5th excitation electrode 640-5 by controlling the 1-1th driver terminal 660-1 and the 1-5th driver terminal 660-5, and can control the voltage not to be applied to the 2-1th excitation electrode 650-1 and the 2-5th excitation electrode 650-5 by controlling the 2-1th driver terminal 670-1 and the 2-5th driver terminal 670-5.

[0199] The controller 680 can control whether to apply a control voltage from the power supply VAC, and can control the amplitude of the control voltage applied from the power supply VAC. The controller 680 can control the amplitude of the arrangement angle of the liquid crystal molecules by controlling the amplitude of the applied control voltage. For example, when the controller 680 controls the application of a control voltage of a first amplitude to the 1-2 excitation electrode 640-2 by controlling the 1-2 driver terminal 660-2 and controls the application of a control voltage of a second amplitude greater than the first amplitude to the 1-3 excitation electrode 640-3 by controlling the 1-3 driver terminal 660-3, the arrangement angle of the liquid crystal molecules in the area where the 1-3 excitation electrode 640-3 is set in the entire area of ​​the liquid crystal layer 610 can be adjusted to be greater than the arrangement angle of the liquid crystal molecules in the area where the 1-2 excitation electrode 640-2 is set.

[0200] That is, the controller 680 can determine the region in which the arrangement angle of the liquid crystal molecules 612 is changed in the entire region of the liquid crystal layer 610 by modulating the phase distribution of the control voltage applied to the plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5 via the plurality of first driver terminals 660-1 to 660-5 and the plurality of second driver terminals 670-1 to 670-5. Therefore, the controller 680 can determine the region A 612A of the variable focus lens 600 as the focus adjustment region. In addition, the controller 680 can adjust the refractive power of the focus adjustment region of the variable focus lens 600 by modulating the phase distribution of the control voltage and adjusting the arrangement angle of the liquid crystal molecules 612 in the liquid crystal layer 610.

[0201] In the above-described embodiment of the present disclosure, a method of adjusting the refractive power of the area corresponding to the area A 612A of the variable focus lens 600 by changing the arrangement angle of the liquid crystal molecules 612 disposed in the area A 612A is described. When the variable focus lens 600 is the first variable focus lens 110-1 or 110-2 for the left eye or the right eye (see Figure 2 ), the second variable focus lens 120-1 or 120-2 (see Figure 2 ) for adjusting the refractive power. That is, the focus control area can be determined differently according to the line of sight direction. For example, when area A 612A is determined as the focus area in the first variable focus lens, the area aligned with the position of area A 612A of the first variable focus lens along the line of sight direction in the entire area of ​​the second variable focus lens can be determined as the focus area. In an embodiment of the present disclosure, information about the line of sight direction can be obtained from the eye tracker 150. The processor 160 can use the information about the line of sight direction information obtained from the eye tracker 150 to determine the position of the area aligned with area A 612A of the first variable focus lens along the line of sight direction in the entire area of ​​the second variable focus lens.

[0202] In this disclosure Figure 6B In the embodiment shown in , when the variable focus lens 600 is a second variable focus lens, the region aligned with the region A of the region 612A along the line of sight direction may be a region where the 1-3rd to 1-5th excitation electrodes 640-3 to 640-5 and the 2-1st to 650-1 to 2-3rd excitation electrodes 650-3 intersect. In order to change the arrangement angle of the liquid crystal molecules arranged in the region aligned with the region A of the region 612A along the line of sight direction among the plurality of liquid crystal molecules 612 included in the liquid crystal layer 610, the controller 680 may control the voltage to be applied to the 1-3rd excitation electrode 640-3, the 1-4th excitation electrode 640-4, and the 1-5th excitation electrode 640-5 among the plurality of first array excitation electrodes 640-1 to 640-5 and the 2-1st excitation electrode 650-1, the 2-2nd excitation electrode 650-2, and the 2-3rd excitation electrode 650-3 among the plurality of second array excitation electrodes 650-1 to 650-5. In an embodiment of the present disclosure, the controller 680 can control the voltage applied to the 1-3rd excitation electrodes 640-3, the 1-4th excitation electrode 640-4 and the 1-5th excitation electrode 640-5 and the 2-1st excitation electrode 650-1, the 2-2nd excitation electrode 650-2 and the 2-3rd excitation electrode 650-3 via the power supply VAC by controlling the 1-3rd driver terminals 660-3 to the 1-5th driver terminals 660-5 and the 2-1st driver terminals 670-1 to the 2-3rd driver terminals 670-3 based on the information about the line of sight received from the processor 160. In this case, the controller 680 can control the voltage not to be applied to the 1-1 excitation electrode 640-1 and the 1-2 excitation electrode 640-2 by controlling the 1-1 driver terminal 660-1 and the 1-2 driver terminal 660-2, and can control the voltage not to be applied to the 2-4 excitation electrode 650-4 and the 2-5 excitation electrode 650-5 by controlling the 2-4 driver terminal 670-4 and the 2-5 driver terminal 670-5.

[0203] By using the above method, the controller 680 can determine the position of the area A 612A of the first variable focus lens and the focus area of ​​the second variable focus lens, and adjust the refractive power of the determined focus area. Fig.11A and Fig. 11B A specific method for determining the positions of the first focusing area of ​​the first variable-focus lens and the second focusing area of ​​the second variable-focus lens according to the line of sight is described in detail.

[0204] Fig. 7A is a conceptual diagram illustrating formation of vergence of a variable focus lens of an AR device according to an embodiment of the present disclosure.

[0205] Reference Fig. 7A and Figure 7B , because a control voltage modulated to have a specific phase distribution is applied to the liquid crystal layer 610 of the variable focus lens 600, the arrangement angle of the liquid crystal molecules 612 set at a specific position in the active area can be changed. Because the arrangement angle of the liquid crystal molecules 612 set in a specific area of ​​the liquid crystal layer 610 is changed, the refractive index of the light passing through the liquid crystal molecules 612 can be changed. When the refractive index of the light changes, the refractive power of the variable focus lens 600 changes, and the path of the light passing through the variable focus lens 600 changes, so the vergence can be changed. The vergence is an index indicating the degree to which the light passing through the variable focus lens 600 converges or diverges. The vergence can be adjusted according to the refractive power of the variable focus lens 600.

[0206] In this disclosure Fig. 7A In the embodiment shown in FIG. 1 , light passing through a region where the alignment angle of the liquid crystal molecules 612 included in the liquid crystal layer 610 changes can form positive vergence, so the variable focus lens 600 can perform a function similar to that of a convex lens. When positive vergence is formed, the focal length can be reduced.

[0207] Figure 7B is a conceptual diagram illustrating formation of vergence of a variable focus lens of an AR device according to an embodiment of the present disclosure.

[0208] In this disclosure Figure 7B In the embodiment shown in FIG. 1 , light passing through a region where the rotation angle of the liquid crystal molecules 612 included in the liquid crystal layer 610 changes can form negative vergence, so the variable focus lens 600 can perform a function similar to that of a concave lens. When negative vergence is formed, the focal length can be increased.

[0209] Figure 8 is a flowchart illustrating a method of adjusting the refractive power of a variable focus lens and changing the focal length, performed by an AR device according to an embodiment of the present disclosure.

[0210] In operation S810, the AR device may apply a control voltage to the first left-eye variable focus lens, the control voltage generating a phase modulation distribution associated with a position corresponding to the first focus area. Figure 6B ) can be based on the eye tracker 150 (see Figure 6B ) to calculate the position coordinate value of the area reached by the user's eye line of sight in the entire area of ​​the variable focus lens. The processor 160 may provide the calculated position coordinate value to the controller 680 of the variable focus lens (see Figure 6B ), and the controller 680 may determine the position of a first focus area, which is a target area whose focus is to be adjusted based on the position coordinate value.

[0211] The controller 680 can control the power supply VAC (see Fig. 6A and Figure 6B ) applies a control voltage waveform having a phase modulation distribution to a plurality of first array excitation electrodes 640-1 to 640-5 (see Figure 6B ) and a plurality of second array excitation electrodes 650-1 to 650-5 (see Figure 6B ), and modulates the control voltage applied to each of the plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5. The processor 160 may modulate the control voltage through the controller 680 so that, among pixels formed by overlapping the plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5, pixels disposed in an area corresponding to the first focus area have a phase value different from other pixels.

[0212] In operation S820, the AR device may change the arrangement angle of the liquid crystal molecules at the position of the first focusing area among the liquid crystal molecules of the first left-eye variable focus lens based on the control voltage, thereby adjusting the refractive power of the first focusing area. The AR device may apply a control voltage having a phase modulation distribution to the first left-eye variable focus lens, thereby changing the arrangement angle of the liquid crystal molecules arranged in the area corresponding to the first focusing area among all the liquid crystal molecules included in the liquid crystal layer. Because the arrangement angle of the liquid crystal molecules in the area corresponding to the first focusing area changes, the refractive power of the light passing through the first focusing area may change. The AR device may adjust the refractive power of the first focusing area by adjusting the arrangement angle of the liquid crystal molecules in the area corresponding to the first focusing area.

[0213] In operation S830, the AR device may adjust the vergence of the first focusing area by refractive power. The AR device may adjust the path of light by adjusting the refractive power of the first focusing area, thereby adjusting the degree of convergence or divergence of light. In an embodiment of the present disclosure, the AR device may adjust the refractive power of the first focusing area in a positive direction or a negative direction, thereby correspondingly reducing or increasing the focal length, which is the distance of the image formed on the retina by passing through the lens of the eye. When the refractive power is adjusted so that the first focusing area has a positive vergence, the first focusing area may perform a function similar to that of a convex lens. When the refractive power is adjusted so that the first focusing area has a negative vergence, the first focusing area may perform a function similar to that of a concave lens.

[0214] In an embodiment of the present disclosure, the refractive power of the first focusing area may be adjusted in a negative direction to form a negative vergence. In this case, the focal length of the virtual object output to the user's eye through the left-eye waveguide 130-1 may be increased.

[0215] Reference Figure 8 The method for adjusting the vergence of the first focusing area described above can also be applied to the first right-eye variable focus lens. Figure 8 The method shown can adjust the refractive power of the third focusing area of ​​the first right-eye variable-focus lens, thereby adjusting the vergence of the third focusing area.

[0216] Fig. 9 is a perspective view showing a waveguide and a display module of an AR device according to an embodiment of the present disclosure.

[0217] Reference Fig. 9 When the user wears the AR device, the waveguide 130 may include a transparent material in which a portion of the rear side is visible. The rear side of the waveguide 130 refers to a surface that the user's eyes face when the user wears the AR device, and the front side of the waveguide 130 refers to a surface opposite to the rear side (i.e., a side away from the user's eyes).

[0218] The waveguide 130 may be configured as a flat plate of a single-layer or multi-layer structure of a transparent material through which light can be reflected and propagated. The waveguide 130 may include a first region 132 that receives light of a virtual image VI projected through an emitting surface 142 facing the display module 140, a second region 134 in which the light of the virtual image VI incident on the first region 132 propagates, and a third region 136 that outputs the light of the virtual image VI propagated in the second region 134 to the direction of the user's eyes. Here, the transparent material may be a material through which light can pass, and its transparency may not be 100%, and may have a predetermined color or hue.

[0219] In an embodiment of the present disclosure, because the waveguide 130 includes a transparent material, the user can view the virtual object of the virtual image VI through the AR device, and can also view the external scene. Therefore, the waveguide 130 can be called a perspective display. AR can be implemented by outputting the virtual object of the virtual image VI via the waveguide 130.

[0220] Diffraction gratings may be formed in the first region 132, the second region 134, and the third region 136 to change the optical path of the light of the virtual image VI. The waveguide 130 may change the propagation path of the light of the virtual image VI using the diffraction gratings formed in the first region 132, the second region 134, and the third region 136, and may perform the function of a light guide plate so that the light of the virtual image VI reflected by the third region 136 may be output to the user's eyes.

[0221] The diffraction grating may be formed in the first region 132 to couple the light of the virtual image VI incident from the emission surface 142 of the display module 140 and transmit the light in the X-axis direction. The display module 140 may be arranged so that the emitted light is perpendicular to the first region 132 or is inclined at a predetermined angle. The arrangement direction of the display module 140 may vary according to the pattern of the diffraction grating of the first region 132.

[0222] The second region 134 may be spaced apart in the X-axis direction relative to the first region 132. A diffraction grating may be formed in the second region 134 to propagate at least a portion of the light received from the first region 132 downward along the Z-axis direction. When the waveguide 130 is formed as a single-layer structure, the diffraction grating of the second region 134 may be formed on the same plane as the diffraction grating of the first region 132. Alternatively, when the waveguide 130 is formed as a multi-layer structure, the diffraction grating of the second region 134 may be formed on a layer different from the layer on which the diffraction grating of the first region 132 is formed. Light incident on the first region 132 may propagate by being reflected between the front side and the rear side of the waveguide 130.

[0223] The third region 136 may be spaced apart downward in the Z-axis direction relative to the second region 134. A diffraction grating may be formed in the third region 136 so that at least a portion of light propagating from the second region 134 is output in a two-dimensional (2D) plane. When the waveguide 130 is formed as a single-layer structure, the diffraction grating of the third region 136 may be formed on the same plane as the diffraction gratings of the first region 132 and the second region 134. Alternatively, when the waveguide 130 is formed as a multi-layer structure, the diffraction grating of the third region 136 may be formed on a layer different from the layer on which the diffraction grating of the second region 134 is formed, and may be formed on the same layer as the diffraction grating of the first region 132 or on a different layer.

[0224] The diffraction grating of the first region 132 may have a different pattern from the grating of the second region 134 and the diffraction grating of the third region 136 .

[0225] The display module 140 can process the processor 160 (see Figure 2 and Figure 3 ) is combined with the light to project the combined virtual image VI onto the waveguide 130 through the emitting surface 142. The display module 140 of the present disclosure can perform the same function as a projector.

[0226] The display module 140 may further include an illumination optical system, an optical path converter, an image panel, a beam splitter, and a projection optical system.

[0227] The illumination optical system is an optical element that irradiates light, and may include a light source and a lens. The light source is an element that generates light by adjusting RGB colors, and may be configured as, for example, a light emitting diode (LED).

[0228] The image panel may be a reflective image panel that reflects light irradiated by a light source and modulates it into light including a 2D image. The reflective image panel may be, for example, a digital micromirror device (DMD) panel or a liquid crystal on silicon (LCoS) panel or another known reflective image panel. The DMD panel may be operated using a digital light processing (DLP) method of irradiating RGB of light output from a light source with a plurality of mirrors each having a pixel size, turning on and off each of the plurality of mirrors, and mixing the RGB of light to project a virtual image VI. The LCoS panel may be operated using a liquid crystal display (LCD) method of separating light output from a light source into RGB by a mirror that allows only light of a specific wavelength to pass, inputting the light to an image panel, and projecting a virtual image VI generated by mixing the RGB.

[0229] The beam splitter may be disposed between the image panel and the projection optical system. The beam splitter may be configured to reflect light output from the light source and transmit light reflected by the image panel.

[0230] The projection optical system is an element that projects light including an image reflected by the image panel onto the waveguide 130, and may include a single or multiple projection lenses. Fig. 9 In the embodiment shown in , the projection surface of the projection optical system refers to the emission surface 142 of the outermost projection lens among a single or multiple projection lenses.

[0231] The display module 140 can receive the signal from the processor 160 (see Figure 2 and Figure 3 ) obtains image data constituting a virtual image VI, generates a virtual image VI based on the obtained image data, and combines the virtual image VI with the light output from the light source to transmit the combined virtual image VI to the waveguide 130 through the emitting surface 142. In an embodiment of the present disclosure, the processor 160 may provide the display module 140 with image data including RGB colors and brightness values ​​of a plurality of pixels constituting the virtual image VI. The display module 140 may project the light of the virtual image VI onto the waveguide 130 by performing image processing using the RGB color value and brightness value of each of the plurality of pixels and controlling the light source.

[0232] In the embodiment of the present disclosure, the display module 140 may use the data stored in the storage 190 (see Figure 3 ) to generate a virtual image VI, combine the virtual image VI with light by controlling the light source, and project the light of the virtual image VI onto the waveguide 130.

[0233] Fig. 10A is a diagram illustrating a method of changing a position of a focus area of ​​a variable focus lens according to a line of sight direction of a user, performed by an AR device according to an embodiment of the present disclosure.

[0234] Fig. 10B is a diagram illustrating a method of changing a position of a focus area of ​​a variable focus lens according to a line of sight direction of a user, performed by an AR device according to an embodiment of the present disclosure.

[0235] exist Fig. 10A and Fig. 10B , the first varifocal lens 110 may include a first focus area 112, and the second varifocal lens 120 may include a second focus area 122. The waveguide 130 may be disposed between the first varifocal lens 110 and the second varifocal lens 120 and project a virtual object VO of a light or virtual image. The virtual object VO may be a virtual object displayed on a partial area of ​​the virtual image.

[0236] Reference Fig. 10A , the AR device can determine the positions of the first focus area 112, the second focus area 122, and the virtual object VO according to the direction of the user's line of sight. In the embodiment of the present disclosure, the eye tracker 150 of the AR device can obtain an eye vector indicating the direction of the line of sight by tracking the position and direction of the user's eye 30. The eye tracker 150 can provide the eye vector to the processor 160 (see Figure 2 and Figure 3 ), and the processor 160 may determine the positions of the first focus area 112 and the second focus area 122 so that the first focus area 112 and the second focus area 122 are aligned according to the direction of the eye vector. In an embodiment of the present disclosure, the processor 160 may obtain the 2D position coordinate value of the area reached by the eye vector in the entire area of ​​the first varifocal lens 110, and determine the position of the first focus area 112 based on the obtained 2D position coordinate value. Similarly, the processor 160 may obtain the 2D position coordinate value of the area reached by the eye vector in the entire area of ​​the second varifocal lens 120, and determine the position of the second adjustment area 122 based on the obtained 2D position coordinate value. The processor 160 may determine the position at which a virtual object VO providing information related to a real-world object in a virtual image is displayed, so that the virtual object VO is aligned with the first focus area 112 and the second focus area 122 along the direction of the eye vector.

[0237] Reference Fig. 10B, the AR device may change the positions of the first focus area 112 and the second focus area 122 in response to a change in the direction of the user's line of sight. When the user's eye 30 looks in the horizontal direction and looks upward at α° relative to the horizontal direction, the eye tracker 150 may obtain an eye vector indicating the direction of the line of sight by tracking the position and direction changes of the eye 30, and provide the eye vector to the processor 160. The processor 160 may change the positions of the first focus area 112 and the second focus area 122 based on the eye vector. In an embodiment of the present disclosure, the processor 160 may change the positions of the first focus area 112 and the second focus area 122 so that the first focus area 112 and the second focus area 122 are aligned according to the direction of the eye vector.

[0238] In the embodiment of the present disclosure, the AR device can use the processor 160 (see Figure 2 and Figure 3 ) modulates the phase distribution of the control voltage, and applies the control voltage with the modulated phase distribution to the first excitation electrode 640 and the second excitation electrode 650 (see Fig. 6A ), thereby changing the positions of the first focus area 112 and the second focus area 122. The first variable focus lens 110 and the second variable focus lens 120 may include partial areas of the entire area where the focus is changed, namely the first focus area 112 and the second focus area 122. The positions of the first focus area 112 and the second focus area 122 are changed according to the direction of the eye vector, and the first variable focus lens 110 and the second variable focus lens 120 may perform a function similar to that of a moving lens.

[0239] Fig.11A is a diagram illustrating a method of changing positions of first and second focusing areas of a first varifocal lens and a second varifocal lens based on an eye vector, performed by an AR device according to an embodiment of the present disclosure.

[0240] Reference Fig.11A , when the user is looking at the real-world object 10, the AR device may obtain an eye vector indicating the direction of the user's sight line. In an embodiment of the present disclosure, the eye tracker may obtain a first eye vector p(u, v) representing the direction of the sight line. The eye tracker may provide the obtained eye vector p(u, v) to the processor 160 (see Figure 2 and Figure 3 ).

[0241] The AR device may obtain the gaze point G based on the binocular disparity and the direction of the first eye vector p(u, v). The processor 160 of the AR device may calculate the 3D position coordinate value of the gaze point G using the binocular disparity and the first eye vector p(u, v) obtained for the left eye and the second eye vector obtained for the right eye. The processor 160 may identify a virtual line l connecting the gaze point G from the user's eyes 30. The processor 160 may obtain coordinate information about the position where the virtual line l intersects with the first varifocal lens 110 in the entire area of ​​the first varifocal lens 110. The processor 160 may determine the area including the position where the virtual line l intersects on the first varifocal lens 110 as the first focusing area 112. The first focusing area 112 may be formed in a circular shape, but is not limited thereto.

[0242] The AR device may determine the position of the second focus area 122 based on the eye vector, so that the first focus area 112 and the second focus area 122 are aligned in the direction of the first eye vector p(u, v). In an embodiment of the present disclosure, the processor 160 may determine the position of the second focus area 122, so that the first focus area 112 and the second focus area 122 are aligned along a virtual line l connecting the first eye vector p(u, v) and the gaze point G. In an embodiment of the present disclosure, the first focus area 112, the second focus area 122, and the virtual object VO may be aligned along the virtual line l.

[0243] The AR device may determine the size of the first focus area 112 based on the size of the virtual object VO of the virtual image output through the waveguide 130 .

[0244] In an embodiment of the present disclosure, the AR device may determine the size of the second focus area 122 based on the spacing distance between the first variable focus lens 110 and the second variable focus lens 120. This will refer to Fig.15A and Fig. 15B Detailed description.

[0245] Fig. 11B is a diagram illustrating a method of changing positions of first and second focusing areas of a first varifocal lens and a second varifocal lens based on an eye vector, performed by an AR device according to an embodiment of the present disclosure.

[0246] Reference Fig. 11B , when the user looks at the real-world object 10 moving in the X-axis direction, the eye tracker may track the user's sight direction and obtain a changed second eye vector p'(u', v'). The eye tracker may provide the second eye vector p'(u', v') to the processor 160. The processor 160 may change the positions of the first focus area 112 and the second focus area 122 based on the changed second eye vector p'(u', v').

[0247] The AR device may obtain the gaze point G using the second eye vector p′(u′, v′) and the binocular parallax information. In an embodiment of the present disclosure, the processor 160 may identify a virtual line l connecting the user's eye 30 and the gaze point G and coordinate information about a position where the virtual line l intersects with the first varifocal lens 110 in the entire region of the first varifocal lens 110. The processor 160 may determine the region including the position where the virtual line l intersects on the first varifocal lens 110 as the first focus region 112. The processor 160 may determine the position of the second focus region 122 based on the eye vector so that the first focus region 112 and the second focus region 122 are aligned in the direction of the second eye vector p′(u′, v′). In an embodiment of the present disclosure, the processor 160 may determine the position of the second focus region 122 so that the first focus region 112 is aligned with the second focus region 122 along the virtual line l obtained based on the second eye vector p′(u′, v′).

[0248] Reference Fig.11A and Fig. 11B When the real-world object 10 moves, the AR device may change the positions of the first focus area 112 and the second focus area 122 based on the eye vector obtained by the eye tracker. In another embodiment of the present disclosure, when the user looks at another real-world object 10, the sight direction may change. In this case, the AR device may use the eye tracker to track the direction change of the eye vector and change the positions of the first focus area 112 and the second focus area 122 according to the direction change of the eye vector.

[0249] Fig. 12A A method for adjusting the focal length of an AR device according to an embodiment of the present disclosure is shown. Fig. 12B A virtual image is shown displayed on a real-world object through an AR device.

[0250] Reference Fig. 12A , the AR device 100 may project light of a virtual object VO providing information about a real-world object viewed by a user in a virtual image onto the waveguide 130, and adjust the refractive power of the first focusing area 112 of the first variable-focus lens 110, thereby adjusting the vergence of the first variable-focus lens 110. Because the vergence of the first focusing area 112 is adjusted, the focal length of the virtual object VO may be adjusted.

[0251] The AR device 100 can adjust the vergence of the first focusing area 112 by adjusting the refractive power of the first focusing area 112. Vergence is an index indicating the degree to which light passing through the first focusing area 112 converges or diverges. Because the vergence is adjusted, the path of light incident on the lens 32 changes, and thus the position of the focus on the retina 34 can change. Therefore, the focal length can be adjusted.

[0252] In this disclosure Fig. 12A In the embodiment shown in FIG. 1 , the AR device 100 can adjust the vergence of the first focusing area 112, thereby adjusting the focal length of the virtual object VO displayed through the waveguide 130. The AR device 100 can use binocular parallax to obtain the gaze point G where the sight lines of the left eye and the sight lines of the right eye converge, the sight lines of the left eye and the sight lines of the right eye, and use triangulation to calculate the vergence distance d as the distance between the two eyes and the gaze point G. con In the embodiment of the present disclosure, the AR device 100 may be based on the vergence distance d con To change the focal length d f , focal length d f is the physical distance between the waveguide 130 on which the virtual object VO is displayed and the eyes. In an embodiment of the present disclosure, the processor 160 of the AR device 100 may adjust the vergence of the first focusing area 112 to offset the focal length of the virtual object VO through the eyes by Δd so that the focal length is equal to the vergence distance d. con Here, the distance Δd is a distance at which the user does not feel dizzy when observing the virtual object VO while wearing the AR device 100, and can be calculated by the processor 160 and stored in the storage 190 ( Figure 3 )middle.

[0253] In an embodiment of the present disclosure, the first focusing area 112 may perform a function similar to that of a concave lens that reflects an optical path to form a negative vergence, thereby increasing the optical path through the lens 32 and changing the focal position formed on the retina 34. When negative vergence is formed, the focal length of the virtual object VO may be increased. In an embodiment of the present disclosure, in order to compensate for the focus distortion in which a real-world object or a physical space appears hazy or blurred due to the adjusted refractive power of the first focusing area 112, the AR device 100 may adjust the refractive power of the second focusing area 122 so that the vergence of the second focusing area 122 of the second variable-focus lens 120 is formed complementarily to the vergence of the first focusing area 112. Referring to Figure 1 and Figure 3 A method of complementarily adjusting the refractive power of the second focus adjustment area 122 with respect to the refractive power of the first focus adjustment area 112 is described, and thus a redundant description thereof will be omitted.

[0254] Reference Fig. 12B , the AR device 100 can display a virtual object VO on the real-world object 10, which indicates information related to the real-world object 10 viewed through the first varifocal lens 110, the second varifocal lens 120, and the waveguide 130. The virtual object VO refers to a portion of a virtual image output to the user's eyes by diffraction with the aid of the waveguide 130.

[0255] In the embodiment of the present disclosure, the AR device 100 can be connected to the camera 170 (see Fig. 12A ) captures the physical environment or space viewed by the user to obtain an image, and provides the obtained image to the processor 160 (see Fig. 12A ). The processor 160 may recognize the real-world object 10 in the acquired image through image processing, and generate a virtual object VO including information related to the real-world object 10. Fig. 12B In the embodiment shown in , the AR device 100 can obtain an image of a subway entrance by photographing the subway entrance via the camera 170, and the processor 160 can identify the real-world object 10 that describes the subway line and exit information at the subway entrance and generate a virtual object VO including information related to the identified real-world object 10 (i.e., information that “subway service is stopped”).

[0256] In the embodiment of the present disclosure, the AR device 100 may use the position sensor 180 (see Figure 3 ) to obtain the position information of the user wearing the AR device 100, and generate a virtual object VO including information related to the position information. For example, the AR device 100 can generate a virtual object VO including a road guidance UI, a navigation UI, etc.

[0257] The AR device 100 may project the generated virtual image onto the waveguide 130 and adjust the refractive power of the first focusing area 112 to form the vergence of the first focusing area 112 , thereby changing the focal length of the virtual object VO which is an area of ​​the virtual image formed on the retina 34 .

[0258] Fig.13A is a diagram for describing a method for adjusting focal length performed by an AR device according to an embodiment of the present disclosure, Fig. 13B An AR device is a virtual image displayed on a real-world object.

[0259] Reference Fig.13A , the eye tracker can obtain an eye vector by tracking the sight line direction of the user's eyes, and provide the eye vector to the processor 160. The processor 160 can obtain the gaze point G on the real-world object 10 viewed by the user based on the eye vector. According to an embodiment of the present disclosure, the processor 160 can use the eye tracker to obtain a first eye vector indicating the sight line direction of the left eye and a second eye vector indicating the sight line direction of the right eye. The processor 160 can calculate the 3D position coordinate value of the gaze point G where the sight lines of both eyes converge based on the binocular disparity, the first eye vector and the second eye vector. The processor 160 can calculate the convergence distance as the distance between the two eyes and the gaze point G. dcon In the embodiment of the present disclosure, the convergence distance d con This can be determined by triangulation.

[0260] The processor 160 may identify the real-world object 10 viewed by the user based on the gaze point G and obtain depth value information about the real-world object 10. In an embodiment of the present disclosure, the depth sensor 172 may scan the physical space or environment around the user and measure the depth value of the real-world object 10 set in the physical space or environment. The depth sensor 172 may measure the depth value according to the 3D position coordinate value of the real-world object 10 in the physical space or environment, and arrange the measured depth value according to each 3D position coordinate value to generate a depth map. The depth sensor 172 may measure the depth value of the real-world object 10 by using, for example, any one of stereo, ToF, and structured pattern technologies.

[0261] Depth sensor 172 may store the depth map in storage 190 (see Figure 3 The processor 160 may load a depth map from the storage 190 to obtain depth information indicating a distance between the user and a real-world object 10 disposed on the gaze point G.

[0262] In an embodiment of the present disclosure, the depth sensor 172 may obtain the 3D position coordinate value of the gaze point G from the processor 160 , measure the depth value corresponding to the obtained 3D position coordinate value, and provide the measured depth value to the processor 160 .

[0263] The AR device 100 can adjust the vergence of the first focusing area 112 by adjusting the refractive power of the first focusing area 112 of the first variable-focus lens 110. The processor 160 can change the focal position of the virtual object VO, which is a region of the virtual image output by the waveguide 130, by adjusting the vergence of the first focusing area 112. In an embodiment of the present disclosure, the processor 160 can adjust the vergence of the first focusing area 112, thereby changing the light path passing through each lens 32 of both eyes, thereby changing the focal position of the virtual object VO formed on the retina 34, thereby adjusting the focal length of the virtual object VO. In an embodiment of the present disclosure, the processor 160 can obtain the depth value of the real-world object 10 set on the gaze point G by loading the depth map pre-stored in the storage 190, and the focal length d of the virtual object VO output through the waveguide 130. f Adjusted to be the same as the obtained depth value of the real world object 10.

[0264] In the embodiment of the present disclosure, the AR device 100 can adjust the refractive power of the first focusing area 112 so as to adjust the focal length d of the virtual object VO output through the waveguide 130 to f Change to the convergence distance d conIn this case, the first focusing area 112 may be adjusted to have a refractive power by which the focal length focused on the virtual object VO output through the waveguide 130 may be offset by the same distance as the depth value of the real-world object 10 .

[0265] In an embodiment of the present disclosure, the AR device 100 can adjust the refractive power of the first focusing area 112 to place the focus position focused on the virtual object VO within the target position T, so the focus position of the virtual object VO can be adjusted. The target position T represents a position within a predetermined range from the position of the gaze point G, where the sight lines of both eyes converge on the real world object 10. In this case, the focal length d of the virtual object VO is f may not be exactly the same as the depth value of the real-world object 10. In the embodiment of the present disclosure, the focal length d f The depth value may be greater or less than the predetermined distance of the real-world object 10 .

[0266] In an embodiment of the present disclosure, in order to compensate for the focus distortion in which the real-world object or the physical space looks hazy or blurred due to the adjusted refractive power of the first focusing area 112 by adjusting the refractive power of the first focusing area 112, the AR device 100 may adjust the refractive power of the second focusing area 122 so that the vergence of the second focusing area 122 of the second variable-focus lens 120 is formed complementarily with respect to the vergence of the first focusing area 112. Figure 1 and Figure 3 A method of complementarily adjusting the refractive power of the second focus adjustment area 122 with respect to the refractive power of the first focus adjustment area 112 is described, and thus a redundant description thereof will be omitted.

[0267] Reference Fig. 13B , the AR device 100 may use the depth sensor 172 (see Figure 3 ) to capture the physical space or environment around the user wearing the AR device 100, and obtain depth value information of each of the multiple first to third real-world objects 12, 14 and 16 included in the physical space or environment.

[0268] The AR device 100 can use the depth value information of each of the multiple first to third real-world objects 12, 14 and 16 to change the focal length so that the virtual objects VO1, VO2 and VO3 are the same as the depth value of the multiple first to third real-world objects 12, 14 and 16 or are respectively displayed within a predetermined range of the depth value.

[0269] In an embodiment of the present disclosure, the AR device 100 may use the depth sensor 172 to measure depth values ​​according to the 3D position coordinate values ​​of the first to third real-world objects 12, 14, and 16 in the physical space or environment viewed by the user, and arrange the measured depth values ​​according to the respective 3D position coordinate values ​​to generate a depth map. The generated depth map may be stored in the storage 190 (see Figure 3 ). Processor 160 (see Fig.13A ) can load a depth map from the storage 190 to obtain depth information, which is the distance between the first to third real-world objects 12, 14, and 16 and the user. In an embodiment of the present disclosure, the processor 160 can obtain a depth value that is the distance between the first real-world object 12 and the user from the storage 190, and adjust the refractive power of the first focusing area 112 so that the focal length of the first virtual object VO1 is the same as the depth value of the first real-world object 12. The processor 160 can adjust the refractive index of the light passing through the first virtual object VO1 by adjusting the refractive power of the first focusing area 112, thereby changing the vergence of the first variable-focus lens 110, and thus can adjust the focal length of the first virtual object VO1.

[0270] In this disclosure Fig. 13B In the illustrated embodiment, the positions of the first real-world object 12, the second real-world object 14, and the third real-world object 16 are set to be different from each other, so the depth values ​​relative to the first real-world object 12, the second real-world object 12, and the third real-world object 16 are different. In the embodiment of the present disclosure, the AR device 100 can track the direction of the user's line of sight by using an eye tracker to obtain the gaze point where the user's eyes converge among the first real-world object 12, the second real-world object 14, and the third real-world object 16, and use the depth value of the real-world object set at the gaze point to adjust the refractive power of the first focusing area 112. The depth values ​​of the first real-world object 12, the second real-world object 14, and the third real-world object 16 can be obtained using a depth map pre-stored in the storage 190. The AR device 100 can adjust the refractive power of the second focusing area 122 to form a complementary vergence relative to the vergence formed due to the adjusted refractive power of the first focusing area 112.

[0271] The AR device 100 may recognize the plurality of first to third real-world objects 12, 14, and 16, and generate virtual objects VO1, VO2, and VO3, respectively including information about the plurality of first to third real-world objects 12, 14, and 16. For example, the AR device 100 may generate virtual objects VO1, VO2, and VO3, which include at least one of detailed descriptions, price information, discount information, a website address of a purchaser, user reviews, or advertisements about the plurality of first to third real-world objects 12, 14, and 16. Fig. 13B In the embodiment shown in , the AR device 100 can recognize that the first real-world object 12 is a shoe, generate a first virtual object VO1 including a member discount, coupon download, etc. related to the shoe, and display the first virtual object VO1 through the waveguide 130. The AR device 100 can also generate a second virtual object VO2 including price information and user evaluation information about the second real-world object 14, and display the second virtual object VO2 through the waveguide 130. The AR device 100 can recognize that the third real-world object 16 is a shirt, generate a third virtual object VO3 including price information and user evaluation information about the shirt, and display the third virtual object VO3 through the waveguide 130.

[0272] Fig.14 is a flowchart of a method for adjusting a refractive index of a focusing area of ​​a variable-focus lens based on a depth value of a real-world object, performed by an AR device according to an embodiment of the present disclosure.

[0273] In operation S1410, the AR device may use a depth camera to measure the depth value of a real-world object set at a gaze point. In an embodiment of the present disclosure, the depth sensor may scan the physical space or environment around the user wearing the AR device, and measure the depth value of the real-world object set in the physical space or environment. The depth sensor may measure the depth value based on the 3D position coordinate value of the real-world object in the physical space or environment, and arrange the measured depth value according to each 3D position coordinate value to generate a depth map. The depth sensor may measure the depth value of the real-world object by using, for example, any one of stereo, ToF, and structured pattern technologies. The depth sensor may store the generated depth map in the storage 190 (see Figure 3 The AR device may load a depth map from storage 190 to obtain depth information, where the depth information is the distance between the user and a real-world object set at the gaze point.

[0274] In an embodiment of the present disclosure, the depth sensor can be obtained from the processor 160 (see Figure 3 ) obtains the 3D position coordinate value of the gaze point, measures the depth value corresponding to the obtained 3D position coordinate value and provides the measured depth value to the processor 160.

[0275] In operation S1420, the AR device may adjust the refractive power of the first focusing area based on the measured depth value to adjust the focal length of the virtual object. The AR device may adjust the focal length of the virtual object corresponding to an area of ​​the virtual image based on the depth value information of the real-world object obtained by loading the depth value information from the storage 190. The AR device may adjust the refractive power of the first focusing area of ​​the first variable focus lens so that the depth value of the real-world object and the focal length focused on the virtual object are the same. In an embodiment of the present disclosure, the AR device may adjust the refractive power of the first focusing area of ​​the first variable focus lens to adjust the focal length of the virtual object to be a predetermined size larger or smaller than the depth value of the real-world object.

[0276] The AR device can adjust the vergence formed in the first focusing area by adjusting the refractive power of the first focusing area. Vergence is an index indicating the degree of convergence or divergence of light, and can be adjusted according to the refractive power of the first focusing area. The AR device can change the light path passing through the lens through the virtual object by adjusting the vergence of the first focusing area, and thus can change the focus position of the virtual object formed on the retina, thereby adjusting the focal length of the virtual object.

[0277] In operation S1430, the AR device may complementarily adjust the refractive power of the second focus area relative to the refractive power of the first focus area to offset the focus distortion of the real-world object caused by the adjusted refractive power of the first focus area. In order to compensate for the focus distortion in which the real-world object appears hazy or blurred due to the vergence formed by the adjusted refractive power of the first focus area of ​​the first variable focus lens, the AR device may adjust the refractive power of the second focus area of ​​the second variable focus lens to be the same as the adjusted refractive power of the first focus area in a direction opposite to the direction of the refractive power of the first focus area. For example, when the first focus area is adjusted to a refractive power of -1 diopter D, the second focus area may be adjusted to +1 diopter D.

[0278] Fig.15A is a diagram illustrating a method of determining a size of a focusing area of ​​a variable-focus lens performed by an AR device according to an embodiment of the present disclosure.

[0279] Reference Fig.15A, the first focusing area 112 of the first variable-focus lens 110, the second focusing area 122 of the second variable-focus lens 120, and the virtual object VO output through the waveguide 130 can be arranged in a line according to the eye vector. In an embodiment of the present disclosure, the AR device can determine the size of the first focusing area 112 based on the size of the virtual object VO output through the waveguide 130. In an embodiment of the present disclosure, the AR device can determine the size of the first focusing area 112 based on the distance between the user's eye 30 and the first variable-focus lens 110 and the size of the virtual object VO.

[0280] When viewing a real-world object through the eye 30, the viewing angle may gradually increase in a direction from the eye 30 toward the real-world object. The AR device may determine the size of the second focus area 122 based on the size of the virtual object VO and the first distance Δd1 spaced between the first and second variable-focus lenses 110 and 120.

[0281] Fig. 15B is a diagram illustrating a method of determining a size of a focusing area of ​​a variable-focus lens, performed by an AR device according to an embodiment of the present disclosure.

[0282] Reference Fig. 15B , even when the viewing angle of the eye 30 is the same, when the second distance Δd1 spaced between the first variable focus lens 110 and the second variable focus lens 120 is greater than the first distance Δd1 (see Fig.15A ), the size of the second focus area 123 may be determined to be larger than Fig.15A The size of the second focusing area 122 is shown. Fig. 15B In the embodiment of the present invention, the size of the first focusing area 113 may be determined based on the size of the virtual object VO and the distance between the first variable-focus lens 110 and the waveguide 130.

[0283] Fig.16 An embodiment of the present disclosure is shown in which the variable focus lens includes a plurality of first to fourth focusing areas.

[0284] Reference Fig.16 , the first variable focus lens 110 may include a plurality of first focus areas 112 and a third focus area 116, and the second variable focus lens 120 may include a plurality of second focus areas 122 and a fourth focus area 126. Fig.16 In the embodiment, the first variable focus lens 110 may include two first focus areas 112 and a third focus area 116, and the second variable focus lens 120 may include two second focus areas 122 and a fourth focus area 126, but this is for the convenience of description. In the present disclosure, the number of focus areas included in the first variable focus lens 110 and the second variable focus lens 120 is not limited to Fig.16In the embodiment of the present disclosure, the first variable-focus lens 110 may include three or more focus adjustment areas, and the second variable-focus lens 120 may also include three or more focus adjustment areas.

[0285] The second focus area 122 of the second variable focus lens 120 may be disposed at a position aligned with the first focus area 112 along the direction of the first eye vector p. Figure 2 and Figure 3 ) can obtain the 2D position coordinate value of the area reached by the first eye vector p in the entire area of ​​the second variable focus lens 120, and determine a preset area around the 2D position coordinate value as the second focusing area 122.

[0286] The fourth focus area 126 of the second variable focus lens 120 may be set at a position aligned with the third focus area 116 along the direction of the second eye vector p'. In an embodiment of the present disclosure, the processor 160 may obtain a 2D position coordinate value of an area reached by the second eye vector p' in the entire area of ​​the second variable focus lens 120, and determine a preset area around the 2D position coordinate value as the fourth focus area 126.

[0287] The refractive powers of the first focusing area 112 and the third focusing area 116 can be adjusted to have different refractive indices. In an embodiment of the present disclosure, the AR device can change the light path passing through the first focusing area 112 by adjusting the first focusing area 112 to the first refractive power, so that the virtual object viewed through the first focusing area 112 is displayed at the first focal length. Thus, the convergence of the first focusing area 112 can be adjusted, and the focal length can be adjusted. In addition, the AR device can adjust the light path passing through the third focusing area 116 by adjusting the third focusing area 116 to the second refractive index, so that the virtual object viewed through the third focusing area 116 is displayed at the second focal length. Thus, the convergence of the third focusing area 116 can be adjusted, and the focal length can be adjusted.

[0288] In an embodiment of the present disclosure, the AR device may adjust the refractive power of each of the plurality of first focus areas 112 and the third focus area 116 on the first variable focus lens 110 based on the depth value of each of the plurality of real-world objects 12 and 14 set at a position corresponding to the plurality of first focus areas 112 and the third focus area 116 on the user's line of sight. The depth value of each of the plurality of first real-world objects 12 and the second real-world objects 14 may be loaded by pre-stored in the storage 190 (see Figure 3 ) in the depth map. The depth map can be obtained by using the depth sensor 172 (see Figure 3) is generated by scanning multiple first real-world objects 12 and second real-world objects 14 and storing a depth value corresponding to each of the 3D position coordinate values ​​of the multiple first real-world objects 12 and second real-world objects 14.

[0289] For example, the AR device may adjust the refractive power of the first focusing area 112 to obtain a first depth value depth1 of a first real-world object 12 viewed in the line of sight direction of the first eye vector p using the depth sensor, and set the focal length of the virtual object to be the same as the first depth value depth1 of the first real-world object 12. For example, the AR device may adjust the refractive power of the third focusing area 116 to obtain a second depth value depth2 of a second real-world object 14 viewed in the line of sight direction of the second eye vector p', and set the focal length of the virtual object to be the same as the second depth value depth2 of the second real-world object 14.

[0290] In the embodiment of the present disclosure, the AR device can modulate the liquid crystal layer 610 (see FIG. Fig. 6A and Figure 6B ) on a plurality of first array excitation electrodes 640-1 to 640-5 (see Figure 6B ) and a plurality of second array excitation electrodes 650-1 to 650-5 (see Figure 6B ) to adjust the refractive power of the plurality of first focus areas 112 and the third focus area 116. For example, in order to adjust the refractive power of each of the plurality of first focus areas 112 and the third focus area 116, when the control voltage is applied through the redundant excitation electrodes among the plurality of first array excitation electrodes 640-1 to 640-5 and the plurality of second array excitation electrodes 650-1 to 650-5, the AR device can adjust the refractive power of each of the plurality of first focus areas 112 and the third focus area 116 through the plurality of first driver terminals 660-1 to 660-5 (see FIG. 6 ) and the plurality of second driver terminals 670-1 to 670-5 (see FIG. 7 ). Figure 6B ) time-divides the control voltage and controls the control voltage to be applied alternately to the excitation electrodes according to the time division.

[0291] A plurality of first driver terminals 660-1 to 660-5 (see Figure 6B ) and a plurality of second driver terminals 670-1 to 670-5 (see Figure 6B ) can be controlled by the controller 680 (see Figure 6B ) control. Processor (see Figure 6B ) may provide position information about the first focus area 112 and the second focus area 122 to the controller 680, and the controller 680 may modulate the phase distribution by the applied control voltage.

[0292] Based on the position information about the first focus area 112 and the second focus area 122 obtained from the processor 160, the first array excitation electrodes 640-1 to 640-5 and the second array excitation electrodes 650-1 to 650-5 are controlled by controlling the first driver terminals 660-1 to 660-5 and the second driver terminals 670-1 to 670-5.

[0293] In an embodiment of the present disclosure, the AR device may adjust the refractive power of the second focus area 122 to compensate for the adjusted refractive power of the first focus area 112. In order to compensate for the focus distortion in which the first real-world object 12 appears hazy or blurred due to the adjusted refractive power of the first focus area 112, the AR device may complementarily adjust the refractive power of the second focus area 122 of the second variable-focus lens 120 relative to the refractive power of the first focus area 112. For example, the AR device may adjust the refractive power of the second focus area 122 to have the same magnitude as the adjusted refractive power of the first focus area 112 in a direction opposite to the direction of the refractive power of the first focus area 112. For example, when the first focus area 112 is adjusted to a refractive power of -1 diopter D, the second focus area 122 may be adjusted to +1 diopter D.

[0294] Similarly, to compensate for the adjusted refractive power of the third focus area 116 , the AR device may complementarily adjust the refractive power of the fourth focus area 126 relative to the refractive power of the third focus area 116 .

[0295] When multiple focus areas are set on the first variable focus lens 110 and the second variable focus lens 120, the AR device can receive virtual game content or 3D motion image content from a server, etc., and adjust the focus on an object including a character displayed on the received content. In an embodiment of the present disclosure, when the AR device receives game content or 3D motion image content from a server or another device and runs the received content or runs game content or 3D motion image content pre-stored in the AR device, the first focus area 112 and the third focus area 116 can be pre-adjusted to a predetermined refractive power. In this case, information about the refractive power of the first focus area 112 and the refractive power of the third focus area 116 can be stored in the storage 190 (see Figure 3 Similarly, information about the refractive powers of the second focusing area 122 and the fourth focusing area 126 may be pre-stored in the storage 190.

[0296] In the embodiment of the present disclosure, the processor 160 (see Figure 3 ) can use the eye tracker 150 (see Figure 3) to obtain an eye vector indicating the direction of the user's sight line, and obtain a gaze point where the sight lines of both eyes converge based on the eye vectors of the left eye and the right eye. The processor 160 may determine one of the plurality of first focus areas 112 and third focus areas 116 included in the first variable-focus lens 110 based on the detected gaze point. The processor 160 may obtain information about the refractive power of the determined focus area from the storage 190, and adjust the refractive power of the focus area using the obtained information about the refractive power.

[0297] For example, when the eye vector obtained by the eye tracker 150 is the first eye vector p, the processor 160 may determine the refractive power of the first focus area 112 as the focus area to be adjusted, and adjust the refractive power of the first focus area 112 using information about the refractive power of the first focus area 112 pre-stored in the storage 190. In order to compensate for the focus distortion caused by the adjusted refractive power of the first focus area 112, the processor 160 may complementarily adjust the refractive power of the second focus area 122 relative to the refractive power of the first focus area 112.

[0298] Fig.17 is a block diagram illustrating an AR device according to an embodiment of the present disclosure.

[0299] Reference Fig.17 , the AR device 200 may include a first variable-focus lens 110 , a second variable-focus lens 120 , a waveguide 130 , a display module 140 , an eye tracker 150 , a processor 160 , a memory 162 , and a communication module 270 . Fig.17 The AR device 200 shown may include Figure 3 For example, the first variable focus lens 110, the second variable focus lens 120, the waveguide 130, the display module 140, and the eye tracker 150 are respectively the same as those of the AR device 100 shown in FIG. Figure 3 The illustrated first varifocal lens 110, the second varifocal lens 120, the waveguide 130, the display module 140, and the eye tracker 150 are identical. Therefore, redundant descriptions will be omitted.

[0300] The processor 160 may control the overall functions and / or operations performed by the AR device 200 by running computer program codes including instructions stored in the memory 162. The processor 160 may control the operations or functions of the first varifocal lens 110, the second varifocal lens 120, the waveguide 130, the display module 140, the eye tracker 150, and the communication module 270.

[0301] The communication module 270 may perform data communication between the AR device 200 and the mobile phone 1000. The AR device 200 may be wirelessly connected to the mobile phone 1000 through the communication module 270.

[0302] The communication module 270 can perform data communication between the AR device 200 and the mobile phone 100 by using at least one of the data communication methods including wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared communication (IrDA), low-power Bluetooth (BLE), near field communication (NFC), wireless broadband Internet (Wibro), world interoperability for microwave access (WiMAX), shared wireless access protocol (SWAP), Wireless Gigabit Alliance (Wiig) or RF communication.

[0303] The mobile phone 1000 may be operated by a user wearing the AR device 200. The mobile phone 1000 may obtain a real-world object image by photographing a physical space or environment around the user via a camera. The mobile phone 1000 may obtain information about the user's location by using a location sensor such as a GPS sensor. The mobile phone 1000 may generate a virtual image by using information about real-world objects. In an embodiment of the present disclosure, the mobile phone 1000 may include a depth sensor and obtain depth value information of real-world objects.

[0304] The processor 160 may control the communication module 270 and receive at least one of position information of a real-world object, real-world object information, or depth value information from the mobile phone 1000 through the communication module 270. The processor 160 may control the display module 140 to project light of a virtual image onto the waveguide 130 based on the received position information and real-world object information. A virtual object, which is a region of a virtual image, may be diffracted by the diffraction grating of the waveguide 130 and output to the user's eyes.

[0305] In an embodiment of the present disclosure, the AR device 200 may receive a virtual object generated by the mobile phone 1000 from the mobile phone 1000 using the communication module 270. The processor 160 may control the display module 140 to project the received virtual object toward the waveguide 130.

[0306] In an embodiment of the present disclosure, the AR device 200 may receive depth value information of a real-world object from the mobile phone 1000 using the communication module 270. The processor 160 may adjust the refractive power of the focus area of ​​the first variable focus lens 110 based on the depth value received from the mobile phone 1000 to change the focal length of the virtual object. In an embodiment of the present disclosure, the processor 160 may adjust the focal length of the virtual object by adjusting the refractive power of the focus area of ​​the first variable focus lens 110 based on the depth value of the real-world object viewed by the user.

[0307] The AR devices 100 and 200 according to the present disclosure may be implemented as hardware elements, software elements, and / or a combination of hardware elements and software elements. For example, the AR devices 100 and 200 according to the embodiments of the present disclosure may be implemented by using a processor, an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a microcomputer, a microprocessor, or one or more general-purpose computers or special-purpose computers (such as devices capable of running and responding to instructions).

[0308] Software may include computer programs, codes, instructions, or a combination of one or more thereof, and may configure a processing device to operate as desired or to command the processing device, either individually or collectively.

[0309] The software may be implemented in a computer program including instructions stored on a computer-readable storage medium. The computer-readable storage medium may include, for example, magnetic storage media (e.g., ROM, RAM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). The computer-readable recording medium may be distributed in computer systems connected in a network, and may store and run computer-readable code in a distributed manner. The medium may be computer-readable, may be stored in a memory, and may be run by a processor.

[0310] The computer may be a device configured to call instructions stored in a storage medium and perform operations according to an embodiment of the present disclosure in response to the called instructions, and may include the AR devices 100 and 200 according to an embodiment of the present disclosure.

[0311] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not contain a signal and is tangible, but does not distinguish whether data is semi-permanently or temporarily stored on the storage medium.

[0312] In addition, the AR devices 100 and 200 or operation methods thereof according to the embodiments of the present disclosure may be provided in a computer program product. The computer program product is a product that can be purchased between a seller and a buyer.

[0313] A computer program product may include a software program and a computer-readable storage medium in which the software program is stored. For example, a computer program product may include a software program-type product (e.g., a downloadable application) electronically released by a manufacturer of AR devices 100 and 200 or an electronic market (e.g., Google PlayTM store, App Store, etc.). For electronic release, at least a portion of the software program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a broadcast server that temporarily stores the software program.

[0314] The computer program product may include a storage medium of a server or a storage medium of a terminal in a system including a server and a terminal (e.g., an ultrasonic diagnostic device). Alternatively, when there is a third device (e.g., a smart phone) connected to the server or the terminal for communication, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program transmitted from the server to the terminal or the third device or from the third device to the terminal.

[0315] In this case, one of the server, the terminal, and the third device may execute the method according to the embodiment of the present disclosure by running the computer program product. Alternatively, at least two of the server, the terminal, and the third device may execute the method according to the embodiment of the present disclosure in a distributed manner by running the computer program product.

[0316] For example, a server (eg, a cloud server or an AI server) may run a computer program product stored in the server and control a terminal connected to the server for communication to perform a method according to an embodiment of the present disclosure.

[0317] As another example, the third device may run the computer program product and control a terminal connected to the third device for communication to perform a method according to an embodiment of the present disclosure.

[0318] When the third device runs the computer program product, the third device can download the computer program product from the server and run the downloaded computer program product. Alternatively, the third device can run the computer program product provided in a free loading state and perform the method according to the embodiment of the present disclosure.

[0319] In addition, although the embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the above-mentioned specific embodiments of the present disclosure. Without departing from the scope of the present disclosure as claimed in the claims, a person of ordinary skill in the art may make various modified embodiments of the present disclosure, and these modifications should not be understood separately from the technical spirit or prospect of the present disclosure.

[0320] Although the embodiments of the present disclosure have been described through the limited embodiments and drawings of the present disclosure as described above, a person skilled in the art may make various modifications and changes from the above description. For example, the described techniques may be performed in a different order than the described methods, and / or the elements of the described electronic devices, structures, circuits, etc. may be combined or integrated in a different form than the described methods, or may be replaced or substituted by other elements or equivalents to achieve appropriate results.

Claims

1. An augmented reality (AR) device, comprising: A plurality of first variable focus lenses and a plurality of second variable focus lenses, wherein each first variable focus lens and each second variable focus lens is an electrically adjustable liquid crystal lens; a plurality of waveguides, wherein each waveguide of the plurality of waveguides is respectively disposed between each first variable-focus lens of the plurality of first variable-focus lenses and each second variable-focus lens of the plurality of second variable-focus lenses; a plurality of eye trackers configured to obtain a left eye vector by tracking a direction of a left eye of a user of the augmented reality device, and to obtain a right eye vector by tracking a direction of a right eye of the user of the augmented reality device; a plurality of display modules configured to project light of a virtual image toward each waveguide of the plurality of waveguides; as well as One or more processors configured to: obtaining a two-dimensional 2D position coordinate value of a first area reached by a left-eye vector in the entire area of ​​a first left-eye variable-focus lens among the plurality of first variable-focus lenses, determining the position of the first focusing area of ​​the first left-eye variable-focus lens based on the obtained 2D position coordinate value of the first area, obtaining a 2D position coordinate value of a third area reached by the right-eye vector in the entire area of ​​the first right-eye variable-focus lens among the plurality of first variable-focus lenses, determining the position of the third focusing area of ​​the first right-eye variable-focus lens based on the obtained 2D position coordinate value of the third area, Get the gaze point based on the left eye vector and the right eye vector, adjusting the refractive power of the first focusing area and the refractive power of the third focusing area based on the gaze point, and The refractive power of the second focus area of ​​the second left-eye variable-focus lens among the plurality of second variable-focus lenses and the refractive power of the fourth focus area of ​​the second right-eye variable-focus lens among the plurality of second variable-focus lenses are complementarily adjusted relative to the adjusted refractive power of the first focus area and the adjusted refractive power of the third focus area, Wherein, the one or more processors are further configured to: adjusting the refractive power of the second focusing area to be the same as the refractive power of the first focusing area in a direction opposite to the direction of the refractive power of the first focusing area, and The refractive power of the fourth focusing area is adjusted to be the same as the refractive power of the third focusing area in a direction opposite to the direction of the refractive power of the third focusing area, wherein the one or more processors are further configured to: determining a size of the first focusing area based on a distance between a left eye of a user and the first left-eye variable focal length lens and a size of a virtual object that is a partial area of ​​a virtual image output through a left-eye waveguide, and The size of the third focusing area is determined based on the distance between the user's right eye and the first right-eye variable focal length lens and the size of a virtual object which is a partial area of ​​the virtual image output through the right-eye waveguide.

2. The AR device according to claim 1, wherein when the user wears the AR device, the first left-eye variable-focus lens is disposed at a position spaced apart from the user's left eye by a first distance, and the second left-eye variable-focus lens is disposed at a position spaced apart from the user's left eye by a second distance, The second distance is greater than the first distance.

3. The AR device according to claim 1, wherein the one or more processors are further configured to control to apply a control voltage to the first left-eye variable-focus lens to generate a phase modulation distribution associated with a position corresponding to the first focusing area, and based on the applied control voltage, adjust the refractive power of the first focusing area by changing the arrangement angle of liquid crystal molecules arranged at the position of the first focusing area among the liquid crystal molecules of the first left-eye variable-focus lens.

4. The AR device according to claim 1, further comprising a depth sensor configured to measure a depth value of a real-world object disposed at a gaze point, The one or more processors are further configured to obtain a measured depth value of a real-world object from a depth sensor, and adjust the refractive power of the first focusing area based on the obtained depth value to adjust the focus of a virtual object, wherein the virtual object is a partial area of ​​the virtual image. 5 . The AR device of claim 1 , wherein the one or more processors are further configured to determine a second focus area based on a left eye vector such that the first focus area and the second focus area are aligned in a direction of the left eye vector.

6. The AR device of claim 1 , wherein a size of the second focusing area is determined based on a size of a virtual object which is a partial area of ​​a virtual image output through a left-eye waveguide and a spacing distance between the first left-eye variable-focus lens and the second left-eye variable-focus lens.

7. A method for operating an augmented reality (AR) device, the method comprising: obtaining a left eye vector by tracking a direction of a left eye of a user using a first eye tracker, and obtaining a right eye vector by tracking a direction of a right eye of the user using a second eye tracker; Determine a first focusing area of ​​a first left-eye variable-focus lens based on the obtained left-eye vector, and determine a third focusing area of ​​a first right-eye variable-focus lens based on the obtained right-eye vector; Obtaining a gaze point based on a left eye vector and a right eye vector; adjusting the refractive power of the first focusing area and the refractive power of the third focusing area based on the gaze point; Complementarily adjusting the refractive power of the second focus area of ​​the second left-eye variable-focus lens and the refractive power of the fourth focus area of ​​the second right-eye variable-focus lens relative to the adjusted refractive power of the first focus area and the adjusted refractive power of the third focus area; as well as projecting light of a virtual image toward each of the plurality of waveguides using a plurality of display modules, Wherein complementarily adjusting the refractive power of the second focusing area of ​​the second left-eye variable-focus lens and the refractive power of the fourth focusing area of ​​the second right-eye variable-focus lens comprises: adjusting the refractive power of the second focusing area to be the same as the refractive power of the first focusing area in a direction opposite to the direction of the refractive power of the first focusing area, and The refractive power of the fourth focusing area is adjusted to be the same as the refractive power of the third focusing area in a direction opposite to the direction of the refractive power of the third focusing area, Wherein, determining the first focusing area and the third focusing area includes: Obtaining the two-dimensional 2D position coordinate value of the first area reached by the left-eye vector in the entire area of ​​the first left-eye variable-focus lens, determining the position of the first focusing area of ​​the first left-eye variable-focus lens based on the obtained 2D position coordinate value of the first area, Obtaining the 2D position coordinate value of the third area reached by the right eye vector in the entire area of ​​the first right eye variable focus lens, and determining the position of the third focusing area of ​​the first right-eye variable-focus lens based on the obtained 2D position coordinate value of the third area, Among them, the size of the first focusing area is determined based on the distance between the user's left eye and the first left-eye variable focal length lens and the size of the virtual object as a partial area of ​​the virtual image output through the left-eye waveguide, and the size of the third focusing area is determined based on the distance between the user's right eye and the first right-eye variable focal length lens and the size of the virtual object as a partial area of ​​the virtual image output through the right-eye waveguide.

8. The operating method according to claim 7, wherein adjusting the refractive power of the first focusing area and the third focusing area comprises: Applying a control voltage to the first left-eye variable focus that produces a phase modulation distribution associated with a position corresponding to the first focus area; as well as The refractive power of the first focus area is adjusted by changing the arrangement angle of liquid crystal molecules arranged at the position of the first focus area among the first left-eye variable-focus liquid crystal molecules based on the applied control voltage.

9. The operating method according to claim 7, wherein adjusting the refractive power of the first focusing area and the refractive power of the third focusing area comprises: Using a depth sensor to measure the depth value of a real-world object set at a gaze point; as well as The refractive power of the first focusing area is adjusted based on the measured depth value to adjust the focus of the virtual object as a partial area of ​​the virtual image.

10. The operating method according to claim 7, wherein adjusting the refractive power of the second focusing area and the refractive power of the fourth focusing area comprises: adjusting the refractive power of the second focusing area so that the second focusing area forms a vergence in a direction opposite to the direction of the adjusted vergence of the first focusing area; and adjusting the refractive power of the fourth focus area so that the fourth focus area forms a vergence in a direction opposite to the direction of the adjusted vergence of the third focus area. 11 . The operating method according to claim 7 , further comprising determining the second focusing area based on the left eye vector such that the first focusing area and the second focusing area are aligned in a direction of the left eye vector.

12. A non-transitory computer-readable recording medium storing a program for executing the operating method according to claim 7.

Images