Variable focus augmented reality video device and video providing method

Abstract

To provide an XR video technology that allows a user to stably view a stereoscopic image according to a visual recognition principle naturally like an environment in which the user views an actual object.SOLUTION: An augmented reality video device includes: an optical system that forms a plurality of focal planes; a sensor that acquires information on a user's line-of-sight; a processor that selects one focal plane among the plurality of focal planes on the basis of the line-of-sight information, changes the focal point of the optical system so that a focal point is formed on the selected focal plane, and generates a binocular parallax focus video in which the focal point is formed on the selected focal plane; and a display that outputs the binocular parallax focus video through control of the processor. A viewing stable area related to the user can be present within a plurality of focal point allowable ranges formed by the plurality of focus planes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a method for providing binocular parallax focused images using a wearable augmented reality (XR) imaging device. [Background technology]

[0002] XR (extended reality) devices (augmented reality imaging devices) are devices that combine reality and virtuality by utilizing extended reality technology. XR is a comprehensive concept that includes virtual reality (VR), augmented reality (AR), and mixed reality (MR). This type of technology provides users with an immersive experience and is used in a variety of industrial fields.

[0003] The main feature of XR devices is that they can provide users with a 3D environment that goes beyond reality, giving them a high level of immersion. They track the user's movements and the environment to enhance interaction and immersion. They are available in a variety of forms, from lightweight AR glasses to highly immersive VR headsets, and use artificial intelligence technology and machine learning to provide a realistic virtual environment. Summary of the Invention [Problem to be solved by the invention]

[0004] This disclosure aims to provide an XR imaging technology that allows users to stably view 3D images based on the principles of natural visual perception, just like viewing a real object, by implementing a variable-focus augmented reality imaging visualization technology that changes the focus in conjunction with the user's time and reproduces binocular time-shifted focus images to realize a 3D stereoscopic image, in order to solve the convergence focus mismatch problem that occurs in variable-focus augmented reality imaging devices. [Means for solving the problem]

[0005] An augmented reality imaging device according to an embodiment of the present disclosure includes an optical system that forms a plurality of focal planes; a sensor that acquires a user's line of sight; a processor that selects one of the plurality of focal planes based on the line of sight information, changes the focus of the optical system so that a focus is formed on the selected focal plane, and generates a binocular parallax condensed image that is focused on the selected focal plane; and a display that outputs the binocular parallax condensed image under the control of the processor, wherein a region that is characteristic of the viewer exists within a plurality of tolerance ranges defined by the plurality of focus planes.

[0006] The viewing stability area can be set based on the size of the allowable distraction source that is focused on the retina of the human eye.

[0007] The magnitude of the allowable scatter source can be pre-calculated based on physiological studies or optical diffraction relationships.

[0008] The size of the permissible circle of confusion can be pre-calculated based on the average visual acuity of a human eye and the size of the pupil.

[0009] The size of the allowable scattering source may be 10 micrometers or more and 15 micrometers or less.

[0010] The stable viewing region can exist within a plurality of focus tolerances according to the plurality of focal planes.

[0011] The optical system can be set so that the stable viewing area exists within a plurality of focus tolerance ranges formed by the plurality of focal planes.

[0012] The optical system may include a variable depth lens module for varying the focus of the optical system.

[0013] The variable depth lens module may include at least one geometric phase lens that changes the focus of the optical system through polarization control.

[0014] Each of the at least one geometric phase lens is made of a birefringent material, and each of the at least one geometric phase lens can form two focal planes.

[0015] The optical system may further include an image visualization lens module integrally formed with each of the at least one geometric phase lenses, for visualizing the binocular parallax focused image on the selected focal plane.

[0016] The processor may generate the binocular parallax focused image through depth of field rendering.

[0017] The processor may generate the binocular parallax focused image using a pre-trained deep learning model.

[0018] The deep learning model can include a Z-buffer algorithm and a ray tracing algorithm.

[0019] During learning, the deep learning model is trained using a first binocular disparity-focused image generated through dynamic fobitation rendering as input data and a second binocular disparity-focused image generated through depth-of-field rendering based on the ray tracing algorithm as output data, and the dynamic fobitation rendering may include a rendering operation of configuring a center portion of the binocular disparity-focused image with high image quality and a peripheral portion of the binocular disparity-focused image with low image quality.

[0020] In a method for providing images in an augmented reality imaging device according to an embodiment of the present disclosure, the method includes the steps of: an optical system of the augmented reality imaging device forming a plurality of focal planes; a sensor of the augmented reality imaging device acquiring gaze information of a user; a processor of the augmented reality imaging device selecting one of the plurality of focal planes based on the gaze information; the processor changing the focus of the optical system so that a binocular focus is formed on the selected focal plane; the processor generating a binocular parallax image centered on the selected focal plane; and outputting the binocular parallax focus image, which generates an enlarged display device, wherein a stable viewing area for the user may exist within a plurality of focus tolerance ranges defined by the plurality of focal planes.

[0021] The viewing stability area can be set based on the size of the allowable distraction source that is focused on the retina of the human eye.

[0022] The size of the allowable scattering source is pre-calculated based on physiological studies or optical diffraction relationships, and the size of the allowable scattering source can be 10 micrometers or more and 15 micrometers or less.

[0023] The stable viewing area may be within a plurality of focus tolerance ranges defined by the plurality of focal planes, and a step of setting the optical system so that the stable viewing area is within a plurality of focus tolerance ranges defined by the plurality of focal planes may be included.

[0024] The method includes generating the binocular parallax-focused image through depth of field rendering, and generating the binocular parallax-focused image using a pre-trained deep learning model, wherein the deep learning model includes a Z-buffer algorithm and a ray tracing algorithm, and the deep learning model uses a first binocular parallax-focused image generated through dynamic fovitated rendering as input data during training, and a second parallax-focused image generated by depth of field rendering based on the ray tracing algorithm as output data, and the dynamic fovitated image is trained to have a high focus, and the peripheral part of the binocular parallax-focused image may include a rendering operation of configuring the image with low image quality. [Effects of the Invention]

[0025] According to the present disclosure, it is possible to minimize eye fatigue of a user viewing a cross-strait time-shifted focused image through an augmented reality imaging device.

[0026] Furthermore, according to the present disclosure, a viewer of an augmented reality imaging device can stably view binocular time-shifted focus images for a long period of time without experiencing viewing fatigue.

[0027] Furthermore, according to the present disclosure, when a user views a cross-strait time-shifted focused image, the user can enjoy a natural viewing experience, just like viewing the real thing.

[0028] Furthermore, according to the present disclosure, it is possible to easily solve the problem of convergence and focus mismatch that occurs when the convergence distance and the focal distance do not match in an augmented reality imaging device. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a flowchart illustrating an image providing method of an augmented reality image device according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a configuration of an augmented reality video device according to an embodiment of the present disclosure. [Figure 3] This shows the convergence distance and focal length of the user's line of sight when looking at an external object. [Figure 4] 1 illustrates the problem of mismatch between convergence distance and focal length according to the prior art. [Figure 5] 1 illustrates convergence distance and focal length according to an embodiment of the present disclosure. [Figure 6] This shows the depth of focus range according to the user's line of sight. [Figure 7] 10 shows the variation in size of the circle of confusion and the viewing stability area with viewing distance for multiple focal planes generated in an embodiment of the present disclosure. [Figure 8] 1 illustrates the configuration of an imaging optical system of an augmented reality imaging device according to an embodiment of the present disclosure. [Figure 9] 1 illustrates the field of view and the visible range of an augmented reality imaging device according to an embodiment of the present disclosure. [Figure 10] 1 illustrates a process in which an augmented reality imaging device according to an embodiment of the present disclosure generates binocular time-differential focus images using deep learning. [Figure 11] 1 illustrates a process in which an augmented reality imaging device provides binocular differential focus images according to a user's line of sight in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0030] [Terminology used in this specification] All embodiments described below are illustrative to facilitate understanding of the present disclosure, and may be modified differently from the embodiments described herein to implement various embodiments. In addition, when describing the present disclosure, if it is determined that a detailed description of related known functions or known components may unnecessarily obscure the gist of the present disclosure, such detailed description will be omitted.

[0031] The accompanying drawings are not drawn to scale to facilitate understanding of the disclosure, and the dimensions of some components may be exaggerated. When reference numerals are used to refer to components, the same numerals are used as much as possible for the same components even when they appear in different drawings.

[0032] Furthermore, when describing components of the embodiments of the present disclosure, terms such as "first," "second," "A," "B," "(a)," and "(b)" may be used. These terms are merely used to distinguish a component from other components and do not limit the nature, order, or sequence of the components. When a component is described as being "coupled," "coupled," or "connected" to another component, it should be understood that the component may be directly coupled, coupled, or connected to the other component, but that another component may be "coupled," "coupled," or "connected" between the component and the other component.

[0033] Therefore, the embodiments described in this specification and the configurations shown in the drawings are merely the most preferred embodiments of the present disclosure and do not represent all of the technical ideas of the present disclosure, and various modifications to the present disclosure are possible.

[0034] Furthermore, the terms and words used in this specification and claims should not be limited to their ordinary or dictionary meanings, but should be interpreted in a meaning and concept that is consistent with the technical idea of ​​the present disclosure, based on the principle that the inventor can appropriately define the concept of the term in order to best explain his / her disclosure.

[0035] Furthermore, the singular expressions used in this application include the plural expressions unless the context clearly indicates otherwise.

[0036] [XR equipment focus mismatch issue] The present disclosure relates to a focus-variable augmented reality imaging device, and more particularly to a method for varying the focus in conjunction with the user's time and reproducing binocular time-varying focus images in order to solve the convergence focus mismatch problem that occurs in augmented reality imaging devices.

[0037] XR (Extended Reality) devices use the stereoscopic method, i.e., binocular disparity, to create 3D images. The user views a 2D virtual image enlarged through an external optical device (or imaging optical system) with both eyes, and the brain interprets the different perspectives to reconstruct a 3D image. While viewing the 2D display image, the user's focus is focused on the display screen, but the image of the 3D object is formed at a distance from the display screen, resulting in a problem known as vergence-accommodation conflict (VAC), where the vergence distance at which the user's two eyes converge does not match the accommodation distance.

[0038] As a result, unlike when a user views an actual image, the focal length and the convergence distance do not match, and thus physiological stimuli to match the convergence distance and the focal length are continuously generated in both eyes of the user while viewing the 3D image. This phenomenon not only increases visual fatigue in the user's eyes, but can also cause serious errors in the user's perception of the 3D image.

[0039] Therefore, a method for solving the convergence focus mismatch problem will be proposed below in order to reduce the user's viewing fatigue and to stably view a stereoscopic image for a long period of time using an augmented reality imaging device.

[0040] [Figure 1: Flowchart of video provision method] FIG. 1 is a flowchart illustrating an image providing method for an augmented reality image device according to an embodiment of the present disclosure.

[0041] As shown in FIG. 1, the method for providing an image of an augmented reality image device (S100) includes steps S110, S120, S130, S131, S140, and S150, which will be described in detail below.

[0042] First, the augmented reality imaging device acquires the user's line of sight information (S110).

[0043] Here, the augmented reality imaging device detects the movement of the user's pupils through a sensor and acquires gaze information based on the movement of the pupils. Here, the augmented reality imaging device can detect the movement of the user's pupils through an eye tracker. Here, the augmented reality imaging device can acquire the movement of the user's pupils and thereby acquire the gaze convergence distance of the user's eyes.

[0044] Next, the augmented reality imaging device selects one focal plane from a plurality of focal planes generated in advance based on the line-of-sight information (S120).

[0045] Here, the stable viewing area exists within a plurality of focus tolerance ranges defined by a plurality of focal planes (S121). The plurality of focal planes each define a focus tolerance range. Here, the imaging optical system of the augmented reality imaging device can be configured so that the stable viewing area exists within the plurality of focus tolerance ranges defined by the plurality of focal planes.

[0046] At the time of manufacturing the augmented reality imaging device or at a specific point after manufacturing, focal planes may be formed so that a comfort zone exists within a depth of field (DOF) (focus range) based on the average visual acuity of the human eye. For example, the number of focal planes may be three, but is not necessarily limited to this.

[0047] That is, conventional variable-focus augmented reality imaging devices form many focal points (e.g., 64) within a specific range to solve the convergence focus mismatch problem, which increases the complexity of the variable-focus imaging optical system and may result in excessive system resources being required for focus changes.In contrast, the augmented reality imaging device according to the present disclosure forms fewer focal points than conventional technologies, while still ensuring a wide range of stable viewing areas that allow users to feel comfortable viewing.

[0048] Here, the focus tolerance is determined by the size of the circle of confusion (CoC) that is focused on the retina of the human eye. The size of the circle of confusion can be determined by physiological experiments or the optical diffraction limit.

[0049] The focus tolerance is set to a range of ±0.3 diopters for each focal plane, but is not necessarily limited to this and can have a range of values ​​greater than 0.3 diopters.

[0050] The focus tolerance can be interpreted as a vergence-accommodation mismatch, i.e., a stable viewing area exists within the focus tolerance, thereby resolving the vergence-accommodation conflict (VAC) problem.

[0051] Next, the augmented reality imaging device changes the focus of the imaging optical system so that the focus is formed on the selected focal plane (S130).

[0052] Here, the imaging optical system can also be defined as an optical device or an optical system. The imaging optical system can be composed of a depth-variable lens module, a pancake lens module, and a lens control driver. Here, the lens control driver can be replaced with a processor, which will be described in reference to 2 hereafter.

[0053] Here, the variable depth lens module can change the focal point by controlling polarization, and can be formed in a modular form with a geometrical phase (GP) lens manufactured using a birefringent material and a polarization control element. Here, one GP lens module can form two focal planes. That is, when n GP lenses are used, 2n focal planes can be formed.

[0054] Next, the augmented reality imaging device generates a binocular parallax focused image (S140).

[0055] Here, the binocular parallax focused image may refer to two images provided independently to each of the user's eyes.

[0056] Next, the augmented reality imaging device provides a binocular differential focus image on the selected focal plane using an imaging optical system (S150).

[0057] Here, the pancake lens module is integrated with the GP lens module, and plays a role in visualizing XR images without distortion at a viewing distance of 28 cm to 7 meters or more and a viewing angle of 110 degrees or more.

[0058] [Figure 2: Augmented reality imaging device structure] FIG. 2 is a block diagram illustrating a configuration of an augmented reality video device according to an embodiment of the present disclosure.

[0059] As shown in FIG. 2, the augmented reality imaging device (200) may include a sensor (210), a processor (220), a memory (230), a display (240), and imaging optics (250).

[0060] The sensor 210 can detect the user's gaze information 21. For example, the sensor 210 can include an eye tracker 211 for tracking the user's gaze. For example, the eye tracker 211 can include a camera for capturing images of the user's eyes. For example, the eye tracker 211 can transmit the gaze information of the user's eyes captured through the camera to the processor 220.

[0061] The processor 220 can generate binocular parallax-focused images. The processor 220 can include a deep learning engine 221 for generating the binocular parallax-focused images.

[0062] The processor 220 can also acquire information about the user's line of sight.

[0063] The processor 220 can also select one focal plane from a plurality of focal planes that are pre-generated based on the line-of-sight information.

[0064] The processor (220) can also change the focus of the imaging optics (optical device (250)) so that the focus is formed at the selected focal plane.

[0065] The processor (220) can also utilize imaging optics to visualize / provide binocular parallax focused images on a selected focal plane.

[0066] For example, the processor 220 can control the display 240 to output a pre-generated binocular parallax-focused image. For example, the processor 220 can output independent binocular parallax-focused images in front of the user's eyes using the left eye display 241 and the right eye display 242. For example, the processor 220 can control the imaging optical system 250 so that the focal points of the user's eyes are formed on a selected focal plane.

[0067] The memory 230 can store commands for the above operations of the processor 220. The memory 230 can also store binocular parallax focused images generated by the processor 220. The memory 230 can also store user's gaze information acquired by the sensor 210.

[0068] The display 240 can output binocular parallax focused images in front of both eyes of the user. For example, the display 240 can include a left eye display 241 for outputting a left eye parallax image in front of the left eye of the user and a right eye display 242 for outputting a right eye parallax image in front of the right eye of the user.

[0069] The imaging optics 250 may have a variable focus under the control of the processor 220. For example, the imaging optics 250 may include a pancake lens 251 and a GP lens 252 that are capable of varying focus.

[0070] [Figure 3-5: Convergence distance vs. focal length] Figure 3 shows the convergence distance and focal length of a user's line of sight when looking at an external object.

[0071] As shown in Figure 3, when a user looks at a real object, a projection image of the real object is projected onto the 2D retina through the eye lens. The 2D projection image information, which is projected onto the retinas of both eyes at different time intervals, is then transmitted to the visual cortex, allowing the brain to naturally reconstruct and perceive the 3D real object.

[0072] In other words, two-dimensional images are input at different times when they are connected to cells at the microscopic scale of the retina of the human eye, and the human brain naturally reconstructs this into a three-dimensional image.

[0073] When viewing a live image, the convergence distance at which the lines of sight of the user's eyes converge and the focal distance of the user's eyes will coincide at the location of the live image.

[0074] FIG. 4 illustrates the convergence distance and focal length mismatch problem according to the prior art.

[0075] As shown in Figure 4, in the case of a conventional augmented reality imaging device, the focus of the user's eyes is fixed on the virtual display screen generated by the augmented reality imaging device, but the line of sight of the user's eyes converges on a location different from the display screen.

[0076] That is, in the case of a conventional augmented reality imaging device, the focal distance of the user's eyes and the convergence distance of the user's eyes are not the same.

[0077] In addition, conventional technology provides a simple projected image to the retina instead of a focus image of the real object. As a result, when viewing a 3D image with binocular parallax focus images, problems arise in that two physical environments are encountered when viewing a real image, namely, convergence-accommodation conflict (VAC) and the provision of a simple projected image instead of a focus image of the real object.

[0078] FIG. 5 illustrates convergence distance and focal length according to an embodiment of the present disclosure.

[0079] As shown in FIG. 5, according to an embodiment of the present disclosure, the augmented reality imaging device acquires the user's line of sight information, and based on the user's line of sight information, acquires the convergence distance at which the lines of sight of the user's eyes (51, 52) converge, and can change the focus from focal plane 2 (502) to focal plane 1 (501) using an imaging optical system so that the convergence distance matches the focal length.

[0080] As a result, augmented reality imaging devices solve the VAC problem by using variable focus imaging technology that tracks the user's line of sight and changes the focus, while providing an intraretinal focused image, creating an environment where users can see the real thing. This makes it impossible for people to distinguish it from the real thing, even when a 3D image is provided by an external imaging optical system.

[0081] [Figure 6: Depth of field range] FIG. 6 shows the depth of focus range according to the user's line of sight.

[0082] The depth of focus range can refer to the range in which a user can see an object clearly, i.e., the range in which the user's eyes can focus. The depth of focus range (63) can also be defined as the depth of field (DOF).

[0083] As shown in FIG. 6, since an image of an object is focused on the retina through the lens of the human eye, the depth of field can be calculated based on the size of the permissible circle of confusion (CoC) (61) focused on the retina of the human eye.

[0084] Augmented reality imaging devices can obtain the depth of field using the following mathematical formula 1. O1(64) is the distance between the focal plane where the object is located and both eyes, c is the size of the circle of confusion, f is the focal length of the human eye, and A(62) is the size of the pupil.

[0085] The augmented reality imaging device can change O2 (65) to obtain a range of focal depths.

[0086]

number

[0087] The size of the permissible circle of confusion that can be focused on the retina of the human eye is calculated in advance from physiological studies and the optical diffraction limit. Physiologically, if visual acuity with an angular resolution of 1 arcmin is defined as 1.0 and the pupil size A is set to 2-3 mm, the permissible circle of confusion size is 10-15 μm.

[0088] As shown in Figure 6, at close distances of approximately 1 meter or less, the depth of field is narrow, and in this case, the blurring of the image formed by both eyes becomes evident when the image is out of focus. In contrast, at relatively far distances, the depth of field is very wide, and the entire area is in focus like a pinhole camera.

[0089] [Figure 7: Change in the size of the circle of confusion due to multiple focal planes] FIG. 7 shows the variation of the size of the circle of confusion and the viewing stability area with the viewing distance for multiple focal planes generated by an embodiment of the present disclosure.

[0090] As shown in FIG. 7, the augmented reality imaging device can generate multiple focal planes (71, 72, 73, 74, 75, 76, 77) in advance.

[0091] For example, in each of a plurality of focal planes having different focal lengths generated by an augmented reality imaging device, the size of the distraction source formed in both eyes of the user varies depending on the viewing distance of the user.

[0092] When the allowable range of the size of the distraction source is set to 13 μm, the focal plane existing in the viewing stability area (area where the size of the distraction source is within the allowable size) (701) changes as the user's viewing distance (the convergence distance of the user's eyes) changes.

[0093] When the user's line of sight converges within a specific range (e.g., 28 cm to 7 m) (when the viewing distance (convergence distance) exists), multiple focal planes can be pre-formed so that the size of the distraction source is within the allowable distraction source size in all areas within the specific range (so that the focal depth range exists in the viewing stability area). The multiple focal planes can be set at the time of manufacturing the augmented reality imaging device or arbitrarily set after manufacturing the augmented reality imaging device, and are not necessarily limited to this.

[0094] For example, the depth of field range can be set to ±0.3 diopters (D) for each focal plane. That is, a ±0.3 diopter depth of field range ensures that the stable viewing area exists within the focus tolerance range, thereby solving the vergence-accommodation conflict (VAC) problem.

[0095] [Figure 8: Imaging optical system configuration] FIG. 8 illustrates the configuration of an imaging optical system of an augmented reality imaging device according to an embodiment of the present disclosure.

[0096] As shown in Figure 8, the imaging optical system (851, 852) is positioned in front of the user's eyes. The imaging optical system (851, 852) can change the focus of the user's eyes. The imaging optical system can be configured in a modular form of a pancake lens (851) and a depth-variable lens (852). The imaging optical system is provided with the pancake lens (851) and the depth-variable lens (852) overlapping in parallel.

[0097] The imaging optical system (851, 852) can be driven by a processor (processor (220) in Figure 2), or can be driven by a lens control unit (or lens driving unit) (not shown) provided adjacent to the imaging optical system.

[0098] The depth-variable lens 852 can vary its focus through polarization control and can be composed of a geometrical phase (GP) lens made of a birefringent material and a polarization control element. For example, the depth-variable lens can have two focal planes. For example, if an augmented reality imaging device includes n depth-variable lenses, the augmented reality imaging device can form 2n focal planes using the n depth-variable lenses.

[0099] [Figure 9: Image angle and visible range] FIG. 9 shows the field of view and the visible range of an augmented reality imaging device according to an embodiment of the present disclosure.

[0100] As shown in FIG. 9, the imaging optical system (950) of FIG. 9 can be configured so that the pancake lens (851) and the variable depth lens (852) of FIG. 8 are integrated.

[0101] Here, the imaging optical system (950) can form a visible distance of approximately 28 cm to 7 m or more using the microdisplay images of the left and right eyes (941, 942).The imaging optical system (950) can also form a field angle of 110 degrees or more.

[0102] [Figure 10: Binocular parallax focused image rendering] FIG. 10 illustrates a process in which an augmented reality imaging device according to an embodiment of the present disclosure generates a binocular parallax focused image using deep learning.

[0103] The processor of the augmented reality imaging device (processor 220 in FIG. 2) can perform DOF rendering to generate binocular parallax focused images. Examples of DOF rendering can include a z-buffer algorithm and a ray-tracing algorithm.

[0104] For example, the processor (220) can input an RGB-D image (1009) into a deep learning engine (1021) included in the processor, and generate multiple binocular focus images (binocular parallax focus images) (1001, 1002, 1003) with different focuses as its output.

[0105] For example, the processor 220 may acquire user gaze information and generate a central portion of an image with a relatively high image quality and process the peripheral portion of the image with a relatively low image quality. Here, a rendering technique that processes the central portion and peripheral portion of an image with different image quality as described above may be defined as a dynamic foveated rendering technique.

[0106] As shown in Figure 10, the augmented reality imaging device can train a deep learning engine (Deep Focus Deep Learning Engine) (1021) in advance. For example, the augmented reality imaging device can use an RGB-D bilateral image dataset generated by ray tracing-based dynamic fobit rendering as input data. Also, for example, the augmented reality imaging device can use an RGB bilateral image dataset generated by DOF rendering as output data.

[0107] Here, the deep learning engine can use a convolution-based deep learning algorithm such as U-Net, which is a conventional technology, or a modified deep learning algorithm.

[0108] [Figure 11: Binocular time-difference focus image provision process] FIG. 11 illustrates a process in which an augmented reality imaging device provides binocular time-shifted images according to a user's line of sight in an embodiment of the present disclosure.

[0109] As shown in FIG. 11, the augmented reality imaging device 1100 acquires the user's line of sight information through a sensor 1111.

[0110] The augmented reality imaging device (1100) can display a pre-generated binocular parallax focused image (binocular focused image) through a left eye display (1141) and a right eye display (1142).

[0111] The augmented reality imaging device (1100) forms, based on the user's line of sight information, a plurality of focal planes (1101, 1102, 1103) that allow the convergence distance of the user's eyes to be within a stable viewing area.

[0112] The augmented reality imaging device (1100) can currently select a focal plane based on the user's line of sight and control the imaging optical system (variable lens) (1150) so that the focal point of both of the user's eyes is formed on the focal plane.

[0113] Here, the user's gaze can be tracked in real time using a gaze tracker, which can measure the vergence angle relative to the user's main gaze point with an accuracy of 0.6 degrees or better.

[0114] [Analysis method described in this specification] Although the embodiments of the present disclosure have been described in further detail above with reference to the accompanying drawings, the present disclosure is not necessarily limited to these embodiments, and various modifications can be made without departing from the technical concept of the present disclosure.

[0115] Therefore, the embodiments disclosed in this disclosure are intended to illustrate, not limit, the technical idea of ​​the disclosure. These embodiments do not limit the scope of the technical idea of ​​the disclosure. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not limiting. The scope of protection of the present disclosure should be interpreted by the scope of the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of rights of the present disclosure.

Claims

1. In augmented reality imaging devices, an optical system that forms multiple focal planes; A sensor that acquires user gaze information; a processor that selects one of the plurality of focal planes based on the line-of-sight information, changes the focus of the optical system so that a focus is formed on the selected focal plane, and generates a binocular parallax focused image in which the focus is formed on the selected focal plane; a display that outputs the binocular parallax focused image under the control of the processor; Including, a stable viewing area for the user exists within a plurality of focus tolerance ranges defined by the plurality of focal planes; Augmented reality imaging device.

2. The viewing stability area is set based on the size of the permissible circle of confusion formed on the retina of the human eye. The augmented reality imaging device according to claim 1 .

3. the size of the permissible circle of confusion is pre-calculated based on physiological studies or optical diffraction relationships; The augmented reality imaging device according to claim 2 .

4. The size of the permissible circle of confusion is pre-calculated based on the average visual acuity of a human eye and the size of the pupil. The augmented reality imaging device according to claim 3 .

5. The size of the permissible circle of confusion is 10 micrometers or more and 15 micrometers or less. The augmented reality imaging device according to claim 4 .

6. the stable viewing area exists within a plurality of focus tolerance ranges according to the plurality of focal planes; The augmented reality imaging device according to claim 1 .

7. The optical system comprises: The stable viewing area is set to exist within a plurality of focus tolerance ranges formed by the plurality of focal planes. The augmented reality imaging device according to claim 6 .

8. the optical system includes a variable depth lens module for varying the focus of the optical system; The augmented reality imaging device according to claim 1 .

9. The variable depth lens module includes at least one geometric phase lens that changes the focus of the optical system through polarization control. The augmented reality imaging device according to claim 8.

10. each of the at least one geometric phase lens is made of a birefringent material, and each of the at least one geometric phase lens forms two focal planes; The augmented reality imaging device according to claim 9.

11. The optical system comprises: and further comprising an image visualization lens module integrally formed with each of the at least one geometric phase lens, for visualizing the binocular parallax focused image on the selected focal plane. The augmented reality imaging device according to claim 10.

12. The processor: generating the binocular parallax focused image by depth of field rendering; The augmented reality imaging device according to claim 1 .

13. The processor: The binocular parallax focused image is generated using a pre-trained deep learning model. The augmented reality imaging device according to claim 12.

14. The processor: The deep learning model includes a Z-buffer algorithm and a ray tracing algorithm. The augmented reality imaging device according to claim 13.

15. During learning, the deep learning model: The first binocular parallax focus image generated by dynamic fobitation rendering is used as input data. The second binocular parallax focus image generated through depth of field rendering based on the ray tracing algorithm is used as output data for learning; The dynamic fobitation rendering The center of the binocular parallax focus image is configured with high image quality, A rendering operation is included in which a peripheral portion of the binocular parallax focus image is configured with low image quality.

15. The augmented reality imaging device according to claim 14.

16. In a method for providing images using an augmented reality image device, an optical system of the augmented reality imaging device forming a plurality of focal planes; a step of acquiring gaze information of a user by a sensor of the augmented reality imaging device; a processor of the augmented reality imaging device selecting one of the plurality of focal planes based on the line-of-sight information; the processor changing the focus of the optical system so that a focus is formed at the selected focal plane; generating a binocular parallax focused image with a focus formed on the selected focal plane by the processor; and a display of the augmented reality imaging device outputs the binocular parallax focused image; Including, A method wherein a stable viewing area exists for the user within a plurality of focus tolerances according to the plurality of focal planes.

17. The viewing stability area is set based on the size of the permissible circle of confusion formed on the retina of the human eye.

17. The method of claim 16.

18. The size of the permissible circle of confusion is pre-calculated based on physiological studies or optical diffraction relationships; The size of the permissible circle of confusion is 10 micrometers or more and 15 micrometers or less.

18. The method of claim 17.

19. the stable viewing area exists within a plurality of focus tolerance ranges defined by the plurality of focal planes; a step of setting the optical system so that the stable viewing area exists within a plurality of focus tolerance ranges formed by the plurality of focal planes; 17. The method of claim 16.

20. generating the binocular parallax focused image by depth-of-field rendering; and generating the binocular parallax focused image using a pre-trained deep learning model; The deep learning model includes a Z-buffer algorithm and a ray tracing algorithm; During learning, the deep learning model: The first binocular parallax focus image generated by dynamic fobitation rendering is used as input data. The second binocular parallax focus image generated through depth-of-field rendering based on the ray tracing algorithm is used as output data for learning; The dynamic fobitation rendering The center of the binocular parallax focus image is configured with high image quality, A rendering operation is included in which a peripheral portion of the binocular parallax focus image is configured with low image quality.

17. The method of claim 16.

Images