Display device comprising a free-formed surface and method of operating the same

Abstract

A display device is provided, including: an image generator configured to generate a plurality of images in time sequence by modulating light; and an optical system including a freeform surface configured to form a plurality of virtual images corresponding to the plurality of images, respectively, in time sequence at different depths from a user's eye, wherein each of error values between the plurality of images and the plurality of virtual images corresponding to the plurality of images, respectively, on the freeform surface is less than or equal to a profile value of the freeform surface.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2021-0052530, filed on April 22, 2021, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to a display device having a freeform surface and a method of operating the same. Background Technology

[0004] Head-mounted displays used to provide virtual reality (VR) have now reached the commercial stage and are widely used in the entertainment industry. Furthermore, head-mounted displays have evolved into forms suitable for medical, educational, and industrial applications.

[0005] Augmented reality (VR) displays, an advanced form of virtual reality displays, are imaging devices that combine the real world and virtual reality, and feature the ability to induce interaction between the real world and virtual reality. This interaction is based on the ability to provide real-time information about the real-world situation, and can further enhance the real-world effect by overlaying virtual objects or information onto the real-world environment. Summary of the Invention

[0006] One or more example embodiments provide a display device and a method of operation thereof for providing multiple images with different depth information using a free-form surface.

[0007] Additional aspects will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of exemplary embodiments of this disclosure.

[0008] According to one aspect of an example embodiment, a display device is provided, comprising: an image generator configured to generate a plurality of images in a time sequence by modulating light; and an optical system including a freeform surface configured to form, in a time sequence, a plurality of virtual images corresponding to the plurality of images at different depths from a user's eye, wherein each of the error values ​​between the plurality of images on the freeform surface and the plurality of virtual images corresponding to the plurality of images is less than or equal to a contour value of the freeform surface.

[0009] On a freeform surface, the error value can be based on the difference in pixel values ​​between multiple images generated in the image generator and multiple virtual images corresponding to those multiple images.

[0010] The contour value of a free-form surface can correspond to the minimum sum of error values ​​between multiple images and multiple virtual images corresponding to those multiple images.

[0011] The freeform surface can also be configured to form multiple virtual images at different depths based on the optical path length between each of the multiple images and the freeform surface.

[0012] When the optical path length between each of the multiple images generated in the image generator and the freeform surface decreases, the depth of each of the multiple virtual images corresponding to the multiple images can be increased.

[0013] The display device may also include a processor configured to control an image generator based on at least one of depth information included in the image information or information about the depth observed by the user.

[0014] The display device may also include a driver configured to adjust the position of an image generator such that the image generator is configured to generate multiple images at different positions based on processor control.

[0015] The driver may include a shape-variable component configured to adjust the position of the image generator based on changes in the shape of the shape-variable component according to a signal applied to the driver.

[0016] Shape-variable components may include materials that have a shape that changes with heat.

[0017] The actuator may include at least one of shape memory alloy or electroactive polymer.

[0018] The processor can also be configured to generate computer-generated holograms based on image information, and the image generator is further configured to generate multiple images with different representative depths based on the computer-generated holograms received from the processor.

[0019] The display device may also include an eye-tracking sensor configured to track the depth of the user's observation, wherein the processor may also be configured to control an image generator such that each of a plurality of virtual images is formed at the depth of the user's observation.

[0020] The optical system can be a combiner configured to converge multiple virtual images and external light corresponding to the external environment to a single focal point, wherein a freeform surface is integrated with the combiner.

[0021] The combiner may include a transparent waveguide configured to transmit multiple virtual images, and a freeform surface may be disposed on the surface of the transparent waveguide.

[0022] The combiner may also include a transmissive and reflective film disposed on a freeform surface.

[0023] Display devices may include augmented reality devices.

[0024] According to another aspect of the example embodiment, a method for operating a display device including a freeform surface is provided, the method comprising: generating a plurality of images in a time sequence by modulating light using an image generator, and forming a plurality of virtual images corresponding to the plurality of images respectively at different depths in a time sequence using the freeform surface, wherein each error value among error values ​​between the plurality of images on the freeform surface and the plurality of virtual images corresponding to the plurality of images is less than or equal to a contour value of the freeform surface.

[0025] On a freeform surface, the error value can be based on the difference in pixel values ​​between multiple images generated in the image generator and multiple virtual images corresponding to those multiple images.

[0026] The contour value of a free-form surface can correspond to the minimum sum of error values ​​between multiple images and multiple virtual images corresponding to those multiple images.

[0027] In the formation of multiple virtual images, multiple virtual images can be formed at different depths based on the optical path length between each of the multiple images and the freeform surface.

[0028] When the optical path length between the multiple images generated in the image generator and the freeform surface decreases, the depth of each of the multiple virtual images corresponding to the multiple images can be increased.

[0029] The method may also include tracking the depth of user observation, wherein, in the formation of multiple virtual images, each of the multiple virtual images is formed at the depth of user observation.

[0030] An image generator may include a light source, a lens, and a spatial light modulator.

[0031] According to another aspect of the example embodiment, a display device is provided, comprising: an image generator configured to generate a plurality of images in a time sequence by modulating light; and an optical system including a freeform surface configured to form, in a time sequence, a plurality of virtual images corresponding to the plurality of images at different depths from a user's eye; and a driver configured to adjust the distance between the image generator and the optical system, wherein each error value among the error values ​​between the plurality of images on the freeform surface and the plurality of virtual images corresponding to the plurality of images is less than or equal to a contour value of the freeform surface, and wherein the contour value corresponds to the minimum sum of the error values ​​between the plurality of images and the plurality of virtual images corresponding to the plurality of images. Attached Figure Description

[0032] The above and / or other aspects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, wherein:

[0033] Figure 1 This is a schematic diagram illustrating the configuration of a display device according to an example embodiment;

[0034] Figure 2 This is a flowchart of a method for operating a display device according to an example embodiment;

[0035] Figure 3 This is a reference view of a method for designing a free-form surface according to an example embodiment;

[0036] Figure 4A and Figure 4B The results of observing virtual images are shown by using a freeform surface optimized for a single depth;

[0037] Figure 5A and Figure 5B The results of observing virtual images are shown by using freeform surfaces optimized for two depths;

[0038] Figure 6 This is a view of a display device including a freeform surface according to an example embodiment;

[0039] Figure 7 yes Figure 6 A view of an example of a driver;

[0040] Figure 8 This is a view of a holographic display device including a freeform surface according to an example embodiment;

[0041] Figure 9 It is used to generate Figure 8 A view of the image generator for holographic images;

[0042] Figure 10 This is a view of a display device including an eye-tracking sensor according to an example embodiment;

[0043] Figure 11 This is a view of a display device including an optical system with a light transmission plate, according to an example embodiment;

[0044] Figure 12 This is a view of a display device for providing images to each of the two eyes, according to an exemplary embodiment;

[0045] Figure 13 This is a view of an example of applying the display device according to the example embodiment to a vehicle; and

[0046] Figure 14 This is a view of an example of applying a display device according to an example embodiment to augmented reality glasses or virtual reality glasses. Detailed Implementation

[0047] Referring now to the exemplary embodiments shown in the accompanying drawings, wherein the same reference numerals refer to the same elements throughout. In this respect, the exemplary embodiments may take different forms and should not be construed as limited to the description set forth herein. Accordingly, the exemplary embodiments are described below only by reference to the accompanying drawings to explain various aspects. As used herein, the term “and / or” includes any and all combinations of one or more of the related listed items. Expressions such as “at least one of…” preceding a list of elements modify the entire list of elements, not individual elements within the list. For example, the expression “at least one of a, b, and c” should be understood to include: only a, only b, only c, a and b, a and c, b and c, or all of a, b, and c.

[0048] The display device 10, including a freeform surface, is described in detail below with reference to the accompanying drawings. Throughout the drawings, the same reference numerals denote the same elements. For ease of explanation, the dimensions of components in the drawings may be enlarged. While exemplary embodiments have been described above, these are merely examples, and those skilled in the art to which this disclosure pertains can make various modifications and changes based on these descriptions.

[0049] When a component is positioned "above" or "on top of" another component, the component may be directly on the other component or in a non-contact manner above it. The singular form used in the specification also includes its plural form, unless the context clearly indicates otherwise. When a component may "comprise" or "include" a component, it shall not be construed as excluding another component, but rather as including other components, unless otherwise stated.

[0050] In the context of describing this disclosure (especially in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, should be interpreted to cover both the singular and the plural. Furthermore, the steps of all methods described herein may be performed in any suitable order unless otherwise stated herein or clearly contradicted by the context. This disclosure is not limited to the described order of steps.

[0051] Furthermore, terms such as “~part,” “~unit,” “~module,” and “~block” used in the specification may refer to a unit that performs at least one function or operation, and the unit may be implemented by hardware, software, or a combination of hardware and software.

[0052] Furthermore, the connecting lines or connectors shown in the various figures are intended to represent functional relationships and / or physical or logical couplings between various components. It should be noted that many alternative or additional functional relationships, physical connections, or logical connections may exist in actual equipment.

[0053] Furthermore, the steps of all methods described herein can be performed in any suitable order unless otherwise stated herein or clearly contradicted by the context. This disclosure is not limited to the order of the steps described.

[0054] Furthermore, the use of any and all examples or language (e.g., "such as") provided herein is intended only to better illustrate this disclosure and not to limit the scope of this disclosure, unless otherwise stated.

[0055] Figure 1 This is a schematic diagram of the configuration of the display device 10 according to an example embodiment, and Figure 2 This is a flowchart of a method for operating a display device according to an example embodiment. (See reference) Figure 1 and Figure 2 The display device 10 according to the example embodiment may include an image generator 110 and an optical system 120. The image generator 110 is used to generate a plurality of images in chronological order, and the optical system 120 is provided with a freeform surface 121 for forming a plurality of virtual images corresponding to images at different depths in chronological order.

[0056] Image generator 110 can generate multiple images sequentially in time by modulating light (S210). The images generated in image generator 110 can be, for example, stereoscopic images, holographic images, light field images, integral photography (IP) images, etc., provided to each of the observer's left and right eyes, and can include multi-view or super-multi-view images. Moreover, the images generated in image generator 110 are not limited to these, and can also be two-dimensional images.

[0057] Image generator 110 may include, for example, liquid crystal on silicon (LCoS) devices, liquid crystal displays (LCDs), organic light-emitting diode (OLED) devices, digital micromirror devices (DMDs), and next-generation displays such as micro-LEDs and quantum dot (QD) LEDs. When image generator 110 is a self-emissive display such as an OLED display or a micro-LED, image generator 110 may include a display panel. When image generator 110 is a non-emissive display such as an LCoS device or an LCD device, image generator 110 may also include a light source 410 for providing illumination light, an optical system for adjusting the path of the illumination light, etc.

[0058] The freeform surface 121 can sequentially form multiple virtual images (S220) at different depths (e.g., at different depths from the observer's eye) that correspond to multiple images respectively. For example, the image generated in the image generator 110 can be reflected from the freeform surface 121 and transmitted to the observer's eye. Based on the optical path length between the image generated in the image generator 110 and the freeform surface 121, the observer can perceive that the image is formed at a specific depth. The observer perceives an image located at a different position than the image generated by the image generator 110, and the image perceived by the observer can be referred to as a virtual image.

[0059] The freeform surface 121 can be a curved surface, optimally designed to focus off-axis input light at a focal point, or to convert overall light input in a direction tilted relative to the optical axis. The profile values ​​of the freeform surface 121 can be designed through an optimization process that satisfies various conditions, such as the viewing angle of the virtual image perceived by the observer, the thickness of the optical system 120 including the freeform surface 121, the size of the eye box, the position of the image generator 110, etc. Accordingly, even slight changes in the optical conditions used to design the freeform surface 121 and the optical conditions used to utilize the freeform surface 121 can result in the observer perceiving a low-quality virtual image.

[0060] When the freeform surface 121 is optically designed for a single focal point, it can have a high-quality virtual image formed at one depth, while the quality of the virtual image formed at different depths may deteriorate. Because the perceived depth perceived by an observer's eyes varies continuously, when the freeform surface 121 provides a virtual image with depth information, the observer may experience dizziness due to the difference between the depth provided by the virtual image and the depth perceived by the observer.

[0061] According to an example embodiment, the freeform surface 121 may include a curved surface, which is designed to allow multiple images to be clearly formed at different depths. Figure 3 This is a reference view of a method for designing a free-form surface 121 according to an example embodiment. Under the same conditions as the display device 10, the first object image I... o1 Second object image I o2 Located at the position where the virtual image is to be formed, and the first target image I t1 Second target image I t2 Located at the position where the image generated by image generator 110 is to be formed. Accordingly, it is possible to design a method that satisfies the condition for minimizing the first target image I. t1 Second target image I t2 With the first object image I o1 Second object image I o2 The error between the conditions of the curved surface profile. First target image I t1 Second target image I t2 With the first object image I o1 Second object image I o2 The error between them can be based on the first target image I t1 Second target image I t2 With the first object image I o1 Second object image I o2 The difference in pixel values ​​between corresponding pixels. For example, the first target image I t1 Second target image I t2 With the first object image I o1 Second object image I o2 Each of the errors between the two can be the first target image I t1 Second target image I t2 With the first object image I o1 Second object image I o2 The square of the absolute value of the difference between the values ​​of corresponding pixels. For example, the first target image I t1 Second target image I t2 With the first object image I o1 Second object image I o2 Each of the errors can be less than or equal to the profile value of the freeform surface.

[0062] In order to form a clear virtual image at multiple depths, the first object image I o1 It can be located at the first depth D1 where the virtual image is to be formed, and the first target image I t1 It can be located at a first position d1 corresponding to the first depth D1, where the image to be generated by the image generator 110 is to be formed. Second object image Io2 It can be located at the second depth D2 where the virtual image is to be formed, and the second target image I t2 It can be located at a second position d2 corresponding to the second depth D2, where the image to be generated by the image generator 110 is to be formed.

[0063] Then, a design can be devised to minimize the first target image I. t1 With the first object image I o1 The error between ||I t1 -I o1 (f(x, y), d1, D1)|| 2 Second target image I t2 With the second object image I o2 The error between ||I t2 -I o2 (f(x, y), d2, D2)|| 2 The curved surface profile of the sum of conditions.

[0064] Equation 1 below represents the curved surface profile in each region of the freeform surface 121, where a clear virtual image is formed at two different depths D1 and D2.

[0065] [Equation 1]

[0066] min{||I t1 -I o1 (f(x, y), d1, D1)|| 2 +||I t2 -I o2 (f(x, y), d2, D2)|| 2}

[0067] Figure 3 A curved surface profile of a freeform surface 121 is shown for providing a clear virtual image at two depths. However, the curved surface profile of the freeform surface 121 can provide a clear virtual image at three or more depths.

[0068] Equation 2 below represents the curved surface profile of the freeform surface 121 used to provide a clear virtual image at k depths, where k is a natural number equal to or greater than 2.

[0069] [Equation 2]

[0070]

[0071] Here, I tn It is at position d n The target image generated at I on (f(x,y),d n,D n ) is at depth D n The virtual image formed at that location, and ||I tn -I on (f(x, y), dn, Dn)|| 2 isI tn with I on The error between them.

[0072] Figure 4A and Figure 4B The results of observing a virtual image using a freeform surface optimized for a single depth are shown. A freeform surface optimized for forming a virtual image at a first depth D1 was designed. When generating an image after the image generator 110 is at the first position d1, as... Figure 4A As shown, the observer can see a clear virtual image.

[0073] When an image is generated after the image generator 110 is in the second position d2, such as Figure 4B As shown, a virtual image is formed at the second depth D2, and the observer can observe a poor-quality and out-of-focus virtual image.

[0074] Figure 5A and Figure 5B The results of observing a virtual image according to an example embodiment are shown using a freeform surface optimized for two depths. The freeform surface was designed to optimize the virtual image to be formed at a first depth D1 and a second depth D2. When an image is generated after the image generator 110 is at the first position d1, as... Figure 5A As shown, a virtual image is formed at D1, and the observer can see a clear virtual image.

[0075] When an image is generated after the image generator 110 is in the second position d2, such as Figure 5B As shown, a virtual image is formed at D2, and the observer can see a clear virtual image.

[0076] As described above, according to the example embodiment, an observer can observe high-quality virtual images at different depths through a freeform surface 121 optimized for multiple depths.

[0077] Figure 6 This is a view of a display device 20 including a freeform surface 121 according to an example embodiment. (Comparison) Figure 1 and Figure 6 , Figure 6The display device 20 may include an image generator 110, a combiner 120a as an optical system for providing to an observer by mixing virtual images with real light including the external environment, a driver 130 for driving the image generator 110 to change the position of the image generator 110, and a processor 140 for controlling the image generator 110 and the driver 130 according to image information.

[0078] The combiner 120a can transmit not only light L1 containing the image generated in the image generator 110 to the observer's eye, but also light L2 containing the external environment in front of the observer to the observer's eye. For example, the combiner 120a can reflect the light L1 containing the image to the observer's eye and transmit the light L2 containing the external environment to the observer's eye.

[0079] The external light L2 can contain real-world objects present in front of the observer, rather than a separate image generated in the image generator 110. Accordingly, the observer can simultaneously perceive both the artificial image from the image generator 110 and the real foreground. Accordingly, the display device 20 can be used as a see-through type display.

[0080] The combiner 120a may include a waveguide 122 for transmitting an image generated in the image generator 110. The waveguide 122 may include a plurality of surfaces, and at least one of the surfaces may include a freeform surface 121.

[0081] like Figure 6 As shown, waveguide 122 may include a first surface S1 and a second surface S2 (the second surface S2 being a freeform surface 121) arranged facing each other, and a third surface S3 and a fourth surface S4 arranged facing each other between the first surface S1 and the second surface S2. Although the second surface S2 is shown as a freeform surface 121, the embodiment is not limited thereto. For example, the first surface S1 may be a freeform surface. The third surface S3 and the fourth surface S4 may be arranged parallel to each other to have no refractive power.

[0082] In the display device 20 according to the example embodiment, the position of the surface on which the virtual image is arranged (i.e., the virtual plane VP) is not fixed to a single value, but rather the position of the virtual plane VP can be changed by reflecting the depth perception of the image to be displayed. For this purpose, the display device 20 may also include a driver 130 for driving the position of the image generator 110.

[0083] The driver 130 can move the image generator 110 to change its distance from the freeform surface 121. As the distance between the image generator 110 and the freeform surface 121 (i.e., the optical path length between the image generated in the image generator 110 and the freeform surface 121) decreases, the depth of the virtual image corresponding to the image generated in the image generator 110 can increase. However, the embodiments are not limited to this. The driver 130 can tilt the image generator 110 corresponding to a pre-designed depth of the freeform surface 121.

[0084] The actuator 130 may include a shape-variable material to increase the driving range of the image generator 110 while keeping the actuator 130 compact. For example, the actuator 130 may deform in response to an applied signal and may provide driving force to the image generator 110. For this shape change, a material having a shape that can be changed by heat may be used in the actuator 130. The actuator 130 may include a shape memory alloy (SMA) or an electroactive polymer (EAP). Reference Figure 7 The position of the image generator 110 is changed by driving the driver 130.

[0085] The processor 140 can generate optical modulation signals to control the image generator 110 and drive signals to control the driver 130 based on image information. Accordingly, the image generator 110 can generate an image at a specific location under the control of the processor 140.

[0086] The display device 20 may also include a memory. In addition to image information, various data, program code, etc. required to drive the image generator 110 may also be stored in the memory.

[0087] The image information may include color and depth information for each image of the frame unit. The processor 140 may generate a light modulation signal for implementing color values ​​determined with reference to the color information for each pixel for each image of the frame unit included in the image information.

[0088] When the depth information, which includes image information, is the depth information of a frame unit, the processor 140 can generate a drive signal based on the depth information.

[0089] When the depth information is not the depth information of a frame unit, such as the depth information of a sub-image in the frame or for each pixel, the processor 140 can determine the representative depth of the frame unit and generate a driving signal based on the representative depth.

[0090] The processor 140 can determine the representative depth of each image for a frame unit by using the color information and / or depth information included in the image information.

[0091] For example, processor 140 can determine representative depth by extracting a color map from image information and analyzing the content and / or saliency information of the color map. Saliency information analysis can be performed to determine areas that an observer is highly likely to observe, such as areas with high visual concentration. To determine areas with high visual concentration, factors such as brightness, color, shape, and object size can be considered. For example, areas with high brightness or color difference relative to the surrounding environment, areas with strong shape characteristics, or areas with large object sizes can be areas with high visual concentration. The depth value corresponding to the aforementioned areas can be determined as the representative depth. According to another example embodiment, locations with high visual concentration can be determined based on the content included in the image.

[0092] In addition, the processor 140 can determine a representative depth by taking the comfort zone into account in the depth map, or by quantizing the depth information included in the depth map and determining a representative depth based on the quantized depth information.

[0093] Processor 140 can generate a drive signal based on a determined representative depth. The drive signal can be, for example, an electrical signal used to generate heat suitable for the deformation of a shape-variable material adapted to the driver 130. The drive signal can be transmitted with a delay of a specific time compared to the light modulation signal. This specific time can be set to exceed the vergence-accommodation time of the observer's eye. This is the time required for the human eye to perceive a virtual image at varying depths.

[0094] Figure 7 yes Figure 6 A view of an example of a driver 130. The driver 130 may include a pair of deformable portions 310 and fixed portions 320, the deformable portions 310 being deformable in response to a drive signal that adjusts the position of the image generator 110, and the fixed portions 320 supporting the deformable portions 310 and being fixed in a specific position.

[0095] The deformable portion 310 may be arranged between the fixed portion 320 and the image generator 110. Each of the two end portions of the deformable portion 310 may contact the fixed portion 320 and the image generator 110, respectively. Although the deformable portions 310 are shown as a pair, this is merely an example. In some example embodiments, one deformable portion or three or more deformable portions may be provided as the deformable portion 310.

[0096] When an electrical signal is applied to the deformable portion 310 and the temperature of the deformable portion 310 increases, the length of each of the deformable portions 310 can be reduced. In this case, the image generator 110 can be moved closer to the fixed portion 320, that is, the distance between the image generator 110 and the freeform surface 121 can be increased.

[0097] As the temperature of the deformable portion 310 is controlled, the degree of length change of each of the deformable portions 310 can be adjusted, and the distance between the image generator 110 and the freeform surface 121 can be controlled.

[0098] According to another example embodiment, the deformable portion 310 may have a linear shape. The length of each of the deformable portions 310 may vary depending on the temperature of the deformable portion 310 or the electric field formed in the deformable portion 310. For example, the deformable portion 310 may include an SMA, an EAP, or a combination thereof. When the deformable portion 310 includes an SMA, the deformable portion 310 may have a shorter length at higher temperatures and a longer length at lower temperatures. When the deformable portion 310 includes an EAP, and an electric field is applied to the deformable portion 310, the length of each of the deformable portions 310 may increase in a direction perpendicular to the applied electric field. In the following description, examples of the deformable portion 310 deforming according to temperature are described.

[0099] The temperature of the deformable portion 310 can be adjusted by an electrical signal applied to it. The electrical signal is based on a drive signal transmitted from the processor 140, and can be a current signal or a voltage signal. For example, as current is applied to the deformable portion 310, its temperature can increase. When no current is applied to the deformable portion 310, its temperature can decrease.

[0100] Despite Figure 6 and Figure 7 In this embodiment, the depth of the virtual image is adjusted by changing the position of the image generator 110 using the driver 130, but the embodiment is not limited to this. When the image generator 110 generates a holographic image, the depth of the virtual image can be implemented differently by changing the holographic surface of the holographic image.

[0101] Figure 8 This is a view of a holographic display device 30 including a freeform surface 121 according to an example embodiment. Figure 9 It is used to generate Figure 8 A view of the image generator 110a of the holographic image. (Reference) Figure 8The hologram display device 30 may include an image generator 110a, a freeform optical system 120a, and a processor 140a. The image generator 110a is used to generate hologram images, the freeform optical system 120a is used to form multiple virtual images with different depths from the hologram images generated in chronological order, and the processor 140a is used to generate a computer-generated hologram (CGH) based on image information and provide the CGH to the spatial light modulator 420. Figure 9 ).

[0102] refer to Figure 9 The image generator 110a may include a light source 410 for providing coherent light, a spatial light modulator 420 for generating a holographic image by diffraction, and a focusing optical system 430 for forming a holographic image in a specific space.

[0103] Light source 410 may include, for example, a laser diode. However, when light has a certain degree of spatial coherence, it can be made coherent by diffraction and modulation by spatial light modulator 420. Therefore, other light sources capable of emitting light with a certain degree of spatial coherence can be used.

[0104] The spatial light modulator 420 can generate a holographic image by diffracting the input light. The holographic method utilizes the principle that when a reference wave is irradiated onto a hologram of the interference fringes between the object wave and the reference wave, it regenerates the object wave. For example, a CGH can be used to form interference fringes.

[0105] The focusing optical system 430 displays the hologram image based on the depth information included in the image information. The focusing optical system 430 may include a configuration that changes the focal position to allow the hologram image generated in the spatial light modulator 420 to be displayed in a space other than the spatial light modulator 420, i.e., displayed on the hologram plane HP.

[0106] The focusing optical system 430 may include one or more lenses. These lenses may be configured to have variable curvature or to move along the optical axis. Accordingly, the focal position is variable, and the position of the hologram plane HP displaying the hologram image may also be variable.

[0107] Processor 140 can determine the representative depth of the hologram plane HP displaying the hologram image from the three-dimensional image information, and generate a CGH corresponding to the representative depth based on the three-dimensional image information. Processor 140 can determine the representative depth by analyzing the color and depth information included in the three-dimensional image information. Since the method for determining the representative depth has already been described above, a detailed description of it is omitted.

[0108] Under the control of the processor 140, the spatial light modulator 420 can display a holographic image on the holographic plane HP corresponding to the representative depth, and the holographic image is reflected by the freeform surface 121, thereby forming a virtual image corresponding to multiple holographic images with different depths.

[0109] Although a representative depth of the image is extracted from image information, the embodiments are not limited thereto. The display device 10 can track the depth observed by the observer and provide depth information about the virtual image.

[0110] Figure 10 This is a view of a display device 40 including an eye-tracking sensor 150 according to an example embodiment. Figure 10 As shown, the display device 40 may also include an eye-tracking sensor 150 for tracking the depth of the observer's observation.

[0111] Eye-tracking sensor 150 can obtain information about the depth of an observer's gaze by tracking the position and orientation of the observer's eyes. For example, eye-tracking sensor 150 can obtain information about the depth of an observer's gaze by using a technique that detects the gaze direction using corneal reflections of infrared light. However, embodiments are not limited to this. Eye-tracking sensor 150 can obtain an image of the pupil using computer vision techniques and use the obtained image to track changes in the pupil's position. Eye-tracking sensor 150 can obtain information about the depth of an observer's gaze by using changes in the pupil's position. Eye-tracking sensor 150 can provide the obtained depth information to processor 140. Eye-tracking sensor 150 may include an infrared camera, a visible light camera, or various other sensors.

[0112] Processor 140 can determine the representative depth of the image based on information received from eye-tracking sensor 150 regarding the depth observed by the observer. Processor 140 can generate a drive signal for driver 130 based on the aforementioned representative depth, or generate a CGH corresponding to the representative depth. Since the image display corresponding to the representative depth has already been described above, its detailed description is omitted.

[0113] although Figure 10 The waveguide surface is shown to be formed as a freeform surface 121, but the embodiment is not limited thereto. The freeform optical system 120a may include elements other than the waveguide.

[0114] Figure 11 This is a view of a display device including an optical system 120b with a light transmission plate 123, according to an example embodiment. Figure 6 Compared to the optical system 120a, Figure 11 The optical system 120b may also include a light-transmitting plate 123 that contacts the waveguide 122. The light-transmitting plate 123 may include a curved surface having a shape complementary to that of the freeform surface 121, and may share the third surface S3 and the fourth surface S4 of the waveguide 122. Light L2 containing the external environment incident on the fourth surface S4 may sequentially pass through the freeform surface 121 and the third surface S3 to reach the observer's eye.

[0115] To reflect the light L1 containing the image and transmit the light L2 containing the external environment, a transflective film can be arranged on the freeform surface 121. The transflective film simply reflects a portion of the incident light and transmits the remaining portion. Then, the portion of the light L1 containing the image can be reflected by the transflective film at the freeform surface 121 to propagate towards the observer's eye, while the portion of the light L2 containing the external environment can be transmitted through the transflective film at the freeform surface 121 to propagate towards the observer's eye.

[0116] When the light L1 containing the image generated in the image generator 110 exhibits polarization characteristics, the transmissive and reflective film can be configured to reflect light with a specific polarization component and transmit light with other polarization components. For example, when the light L1 containing the virtual image has a first polarization component, the transmissive and reflective film can reflect light with the first polarization component and transmit light with a second polarization component perpendicular to the first polarization component.

[0117] Although the transflective membrane is positioned in contact with the freeform surface 121 in the accompanying drawings, the embodiments are not limited thereto. For example, the transflective membrane and the freeform surface 121 may be arranged separately from each other on the waveguide 122.

[0118] Figure 12 This is a view of a display device 60, according to an example embodiment, for providing images to each of the two eyes. The images provided to the two eyes may be the same, or they may be images with parallax information.

[0119] The display device 60 may include a first image generator 110R for generating a right-eye image, a first image convergence member 120R for converging the right-eye image with the real environment into a region, a first image generator 110L for generating a left-eye image, a second image convergence member 120L for converging the left-eye image with the real environment into a region, and a processor 140b for controlling the first image generator 110R and the second image generator 110L to display images at a representative depth.

[0120] The first image generator 110R and the second image generator 110L can generate right-eye and left-eye images respectively under the control of the processor 140b. Since the first image generator 110R and the second image generator 110L have already been described above, their detailed description is omitted.

[0121] The processor 140b can not only generate light modulation signals for the first image generator 110R and the second image generator 110L to generate images, but also determine representative depth based on image information or information received from the eye-tracking sensor 150.

[0122] The first image convergence member 120R can modify at least one of the first optical path L1 of the right-eye image or the second optical path L2 of the real environment to converge the image and the real environment into a region. Here, a region can be the observer's right eye (RE). The first image convergence member 120R can transmit multiple light rays to the observer's eye along the first optical path L1 and the second optical path L2. The second image convergence member 120L can modify at least one of the third optical path L3 of the left-eye image or the second optical path L2 of the real environment to converge the left-eye image and the real environment into a region. Here, a region can be the observer's left eye (LE).

[0123] Each of the first image converging member 120R and the second image converging member 120L may include a freeform surface as described above. Furthermore, each of the first image converging member 120R and the second image converging member 120L may also include, for example, a waveguide, a light-transmitting plate, a beam splitter, a transflective film, etc.

[0124] The image transmitted by light through the first optical path L1 and the third optical path L3 can be an image provided by the augmented reality device. The real environment transmitted by light through the second optical path L2 can be the environment faced by the observer through the augmented reality device. The real environment may include the foreground facing the observer and may include specific background subjects.

[0125] Figure 13 This is a view of an example of applying a display device according to an exemplary embodiment to a vehicle. The display device can be applied to a head-up display 70 of a vehicle. The head-up display 70 may include an image generator 110c disposed in the vehicle area, and at least one optical system 120c for changing the optical path through which the driver views the image generated in the image generator 110c. The optical system 120c may include a freeform optical system according to an exemplary embodiment.

[0126] Figure 14This is a view illustrating an example of applying a display device according to an exemplary embodiment to augmented reality glasses 80 or virtual reality glasses. Augmented reality glasses 80 may include an image generator 110d for generating images and an optical system 120d for guiding images from the image generator 110d into the observer's eyes. The optical system 120d may include a freeform optical system 120a according to an exemplary embodiment.

[0127] Furthermore, the display devices 10, 20, 30, 40, 50 and 60 according to the above example embodiments can be implemented as, for example, various types of wearable devices, head-mounted displays (HMS), eyeglass displays or goggle displays.

[0128] The aforementioned display devices 10, 20, 30, 40, 50, and 60 can operate by being associated with or connected to other electronic devices such as smartphones. For example, a smartphone may be equipped with a processor for driving an image generator. Furthermore, a smartphone may be equipped with the aforementioned display devices.

[0129] Because the aforementioned display device includes a freeform surface forming multiple depths, it can provide clear virtual images.

[0130] The aforementioned display devices can be more easily applied to wearable devices, such as glasses-type augmented reality display devices.

[0131] It should be understood that the exemplary embodiments described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each exemplary embodiment should generally be considered applicable to other similar features or aspects in other embodiments. Although exemplary embodiments have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope defined by the following claims and their equivalents.

Claims

1. A display device, comprising: An image generator is configured to generate multiple images in chronological order by modulating light; as well as An optical system comprising a freeform surface configured to sequentially form multiple virtual images at different depths from the user's eye, each corresponding to a plurality of images. Wherein, each of the error values ​​between the plurality of images on the freeform surface and the plurality of virtual images corresponding to the plurality of images is less than or equal to the contour value of the freeform surface. Wherein, the error value on the freeform surface is based on the difference in pixel values ​​between the plurality of images generated in the image generator and the plurality of virtual images corresponding to the plurality of images, and The contour value of the freeform surface corresponds to the minimum sum of error values ​​between the plurality of images and the plurality of virtual images corresponding to the plurality of images.

2. The display device according to claim 1, wherein, The freeform surface is also configured to form the plurality of virtual images at different depths based on the optical path length between each of the plurality of images and the freeform surface.

3. The display device according to claim 2, wherein, As the optical path length between each of the plurality of images generated in the image generator and the freeform surface decreases, the depth of each of the plurality of virtual images corresponding to the plurality of images increases.

4. The display device of claim 1, further comprising a processor configured to control the image generator based on at least one of depth information included in the image information or information about the depth observed by the user.

5. The display device of claim 4, further comprising a driver configured to adjust the position of the image generator such that the image generator is configured to generate the plurality of images at different positions based on the control of the processor.

6. The display device according to claim 5, wherein, The driver includes a shape-variable component configured to adjust the position of the image generator based on changes in the shape of the shape-variable component according to signals applied to the driver.

7. The display device according to claim 6, wherein, The shape-variable component comprises a material having a shape that changes with heat.

8. The display device according to claim 5, wherein, The actuator comprises at least one of shape memory alloy or electroactive polymer.

9. The display device according to claim 4, wherein, The processor is further configured to generate a computer-generated hologram based on the image information, and the image generator is further configured to generate the plurality of images with different representative depths based on the computer-generated hologram received from the processor.

10. The display device of claim 4, further comprising an eye-tracking sensor configured to track the depth of the user's gaze. in, The processor is also configured to control the image generator such that each of the plurality of virtual images is formed at the depth of the user's observation.

11. The display device according to claim 1, wherein, The optical system is a combiner configured to converge the plurality of virtual images and external light corresponding to the external environment to a single focal point. The freeform surface is integrated with the combiner.

12. The display device according to claim 11, wherein, The combiner includes a transparent waveguide configured to transmit the plurality of virtual images, and The freeform surface is disposed on the surface of the transparent waveguide.

13. The display device according to claim 11, wherein, The combiner also includes a transmissive and reflective film disposed on the freeform surface.

14. The display device according to claim 1, wherein, The display device includes an augmented reality device.

15. The display device according to claim 1, wherein, The image generator includes: light source; Lenses; and Spatial light modulator.

16. A method of operating a display device including a freeform surface, the method comprising: Multiple images are generated sequentially over time by an image generator that modulates light. as well as Multiple virtual images, each corresponding to a plurality of images, are formed sequentially at different depths from a free-form surface. Wherein, each of the error values ​​between the plurality of images and the plurality of virtual images corresponding to the plurality of images is less than or equal to the contour value of the freeform surface. The error value is based on the difference in pixel values ​​between the plurality of images generated in the image generator and the plurality of virtual images corresponding to the plurality of images, and The contour value of the freeform surface corresponds to the minimum sum of error values ​​between the plurality of images and the plurality of virtual images corresponding to the plurality of images.

17. The method according to claim 16, wherein, In the formation of the plurality of virtual images, the plurality of virtual images are formed at different depths based on the optical path length between each of the plurality of images and the freeform surface.

18. The method according to claim 17, wherein, As the optical path length between the plurality of images generated in the image generator and the freeform surface decreases, the depth of each of the plurality of virtual images corresponding to the plurality of images increases.

19. The method of claim 16, further comprising tracking the depth of the user's observation. in, In the formation of the plurality of virtual images, each of the plurality of virtual images is formed at the depth of the user's observation.

Images