Display device including a free-form curved surface and its operation method

Abstract

To provide a display device including a free-form surface and an operating method thereof.SOLUTION: The present invention relates to a display device and an operating method thereof. The display device comprises an image generator configured to sequentially generate a plurality of images by modulating light, and an optical system including a free-form surface configured to sequentially form a plurality of virtual images corresponding to the plurality of images at different depths from a user's eye, where each of error values between the plurality of images and the plurality of virtual images corresponding to the plurality of images on the free-form surface is less than or equal to a profile value of the free-form surface.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display device having a free-form curved surface and a method for operating the same. [Background technology]

[0002] Head-mounted displays that provide virtual reality (VR) are now in the commercialization stage and are trending towards widespread application in the entertainment industry. Simultaneously, they are evolving into forms that can be applied in the medical, educational, and industrial fields.

[0003] Augmented reality (AR) displays, an advanced form of virtual reality displays, are imaging devices that combine the real world and virtual reality, and are characterized by their ability to induce interaction between reality and virtuality. This interaction between reality and virtual reality is based on the function of providing real-time information about the real situation, and by overlaying virtual objects and information onto the real-world environment, the effect of reality can be further enhanced. [Overview of the project] [Problems that the invention aims to solve]

[0004] The problem that this invention aims to solve is to provide a display device and a method for operating the same that utilize a free-form curved surface to provide multiple images with different depth information. [Means for solving the problem]

[0005] A display device according to one embodiment includes an image generator that modulates light to generate a plurality of images in a time series, and an optical system including a freeform surface that forms a plurality of virtual images corresponding to the plurality of images in a time series at different depths from the user's eye, wherein the error values ​​between the plurality of images and the plurality of corresponding virtual images on the freeform surface are less than or equal to the profile value of the freeform surface.

[0006] Furthermore, the error value is based on the difference in pixel values ​​between the image generated by the image generator and the corresponding virtual image on the free-form curved surface.

[0007] Furthermore, the profile of the free-form surface is also constructed by the minimum value of the sum of the errors between each image and its corresponding virtual image on the free-form surface.

[0008] Furthermore, the free-form curved surface can form the multiple virtual images at different depths based on the optical path lengths between the multiple images and the free-form curved surface.

[0009] Furthermore, the shorter the optical path length between the image generated by the image generator and the free-form curved surface, the deeper the corresponding virtual image can become.

[0010] The system may further include a processor that controls the video generator based on at least one of the depth information contained in the video information and the depth information that the user is focusing on.

[0011] The system may further include a drive unit that adjusts the position of the video generators so that they generate the plurality of videos at different locations, under the control of the processor. Furthermore, the drive unit also includes a shape-changing member whose shape is changed by the applied signal, and which adjusts the position of the image generator.

[0012] The shape-variable member also includes a substance whose shape can be changed by heat. It may also include at least one of a shape memory alloy or an electroactive polymer.

[0013] The processor performs computer generated hologram calculations on the video information, and the video generator can generate the plurality of videos with different representative depths based on the computer generated hologram received from the processor.

[0014] It further includes an eye tracking sensor that tracks the depth the user is gazing at, and the processor can control the video generator so that the virtual video is formed at the depth the user is gazing at.

[0015] The optical system is a combiner that combines the plurality of virtual videos and external light corresponding to the external environment at a single point, and the free-form surface can be integrated with the combiner.

[0016] The combiner also includes a transparent waveguide that transmits the virtual video, and the free-form surface can be arranged on the surface of the transparent waveguide.

[0017] The combiner may further include a semi-transmissive film arranged on the free-form surface.

[0018] The display device is also an augmented reality device.

[0019] On the other hand, according to one embodiment, the operation method of a display device including a free-form curved surface includes the steps of: modulating light to generate a plurality of images in a time series; and forming a plurality of virtual images corresponding to each of the plurality of images in a time series at different depths using the free-form curved surface, wherein the error value between each image and its corresponding virtual image on the free-form curved surface is less than or equal to the profile value of the free-form curved surface.

[0020] Furthermore, the error value is based on the difference in pixel values ​​between the image generated by the image generator and the corresponding virtual image on the free-form curved surface.

[0021] Furthermore, the profile of the free-form surface is also constructed by the minimum value of the sum of the errors between each image and its corresponding virtual image on the free-form surface.

[0022] Furthermore, in the step of forming the virtual images, the multiple virtual images can be formed at different depths based on the respective optical path lengths of the multiple images and the free-form curved surface.

[0023] Furthermore, the shorter the optical path length between the image generated by the image generator and the free-form curved surface, the deeper the virtual image corresponding to the image can become.

[0024] The step of forming the virtual image further includes tracking the depth at which the user is looking, and the virtual image can be formed at the depth at which the user is looking. [Effects of the Invention]

[0025] The display device of the present invention can provide a clear virtual image because it includes a free-form curved surface that can embody multiple depths.

[0026] The aforementioned display devices are easily applicable to wearable devices, such as glasses-type augmented reality display devices. [Brief explanation of the drawing]

[0027] [Figure 1] This figure shows a schematic configuration of a display device according to one embodiment. [Figure 2] This figure illustrates the operation method of a display device according to one embodiment. [Figure 3] This is a reference diagram illustrating a method for designing a free-form curved surface according to one embodiment. [Figure 4A] This figure shows the results of observing a virtual image using a free-form surface optimized for a single depth. [Figure 4B] This figure shows the results of observing a virtual image using a free-form surface optimized for a single depth. [Figure 5A] This figure shows the results of observing a virtual image using a free-form surface optimized for two depths. [Figure 5B] This figure shows the results of observing a virtual image using a free-form surface optimized for two depths. [Figure 6] This is a diagram illustrating a display device including a free-form curved surface according to one embodiment. [Figure 7] This figure illustrates an example of a drive unit applied to Figure 6. [Figure 8] This figure illustrates a hologram display device including a free-form curved surface according to one embodiment. [Figure 9] Figure 8 shows a diagram illustrating the video generator that produces the hologram image. [Figure 10] This figure illustrates a display device including an eye-tracking sensor according to one embodiment. [Figure 11] This diagram illustrates a display device including an optical system with a light-transmitting plate, according to one embodiment. [Figure 12] This diagram illustrates a display device that provides images to each eye according to one embodiment. [Figure 13]This figure illustrates an example of a display device according to an exemplary embodiment being applied to an automobile. [Figure 14] This figure illustrates an example of a display device according to an exemplary embodiment being applied to augmented reality glasses or virtual reality glasses. [Modes for carrying out the invention]

[0028] The display device 10, including a freeform surface, will be described in detail below with reference to the attached drawings. In the following drawings, the same reference numerals refer to the same components, and the size of each component in the drawings is exaggerated for clarity and convenience of explanation. Furthermore, the embodiments described below are merely illustrative, and various modifications are possible from such embodiments.

[0029] In the following, "top" or "above" may include not only things that are directly above and in contact with the object, but also things that are above but not in contact with the object. Singular expressions include plural expressions unless the context clearly indicates otherwise. Also, when a part "includes" a component, it does not mean that other components are excluded, but rather that other components may be included, unless otherwise specified.

[0030] The term "the foregoing," and similar referential terms, may be singular or plural. Unless explicitly stated otherwise, the steps constituting a method may be performed in any order, but are not necessarily limited to the order stated.

[0031] Furthermore, terms such as "...part" and "module" as used in the specification refer to units that process at least one function or operation, which may be embodied by hardware or software, or by a combination of hardware and software.

[0032] The linear connections or connecting members between components shown in the drawings are illustrative examples of functional and / or physical or circuit connections, and in actual devices, they may also be shown as a variety of other functional, physical, or circuit connections that are interchangeable or additional.

[0033] All use of examples or illustrative terms is solely for the purpose of illustrating the technical idea in detail and, unless otherwise specified by the claims, does not limit the scope of the claims.

[0034] Figure 1 is a diagram showing the schematic configuration of a display device 10 according to one embodiment, and Figure 2 is a diagram illustrating the operation method of the display device according to one embodiment. Referring to Figures 1 and 2, the display device 10 according to one embodiment also includes an image generator 110 that generates multiple images in a time series, and an optical system 120 in which freeform surfaces 121 that form multiple virtual images corresponding to the multiple images at different depths in a time series are arranged.

[0035] The image generator 110 can modulate light and generate multiple images in a time series (S210). The images generated by the image generator 110 may be, for example, stereo images provided to the left and right eyes of the observer, or holographic images, light field images, IP (integral photography) images, etc., and may also include images in a multi-view or super multi-view format. Furthermore, the images formed by the image generator 110 are not limited to these, but can also be general two-dimensional images.

[0036] The image generator 110 may include, for example, an LCoS (liquid crystal on silicon) element, an LCD (liquid crystal display) element, an OLED (organic light emitting diode), or a DMD (digital micromirror device), or it may include a next-generation display such as a Micro LED or QD (quantum dot) LED. If the image generator 110 is a self-emissive display such as an OLED display or a Micro LED, the image generator 110 may include only one display panel. However, if the image generator 110 is a non-emissive type such as an LCoS element or an LCD element, the image generator 110 may further include a light source 410 (Figure 9) that provides illumination light, and an optical system for adjusting the path of the illumination light.

[0037] The freeform surface 121 can sequentially form multiple virtual images corresponding to multiple images at different depths, for example, at different depths relative to the observer's eye. Specifically, the image generated by the image generator 110 is reflected from the freeform surface 121 and transmitted to the observer's eye. Based on the optical path length between the image generated by the image generator 110 and the freeform surface 121, the observer can recognize that an image has been formed at a specific depth. The observer recognizes an image that is at a different position from the image generated by the image generator 110, and the image recognized by the observer can be called a virtual image.

[0038] The freeform surface 121 refers to a surface optimally designed to either focus off-axis incident light into a single focal point or to perfectly form light incident in a direction inclined with respect to the optical axis. The profile values ​​of the freeform surface 121 are also designed through an optimization process that satisfies various conditions such as the field of view of the virtual image perceived by the observer, the thickness of the optical system 120 including the freeform surface 121, the size of the eyebox, and the position of the image generator 110. Therefore, even if the optical conditions applied during the design of the freeform surface 121 differ slightly from the optical conditions when the freeform surface 121 is in use, the observer will perceive a virtual image of reduced quality.

[0039] In addition, if the freeform surface 121 is designed to be optimized for a single focus, the quality of the virtual image formed at one depth by the aforementioned freeform surface 121 is good, while the quality of the virtual images formed at different depths deteriorates. Although the perceived depth continuously changes with both eyes of the observer, if the freeform surface 121 provides a virtual image having one depth information, the observer will feel dizziness due to the sense of divergence between the depth provided by the virtual image and the depth perceived by the observer.

[0040] The freeform surface 121 according to one embodiment also includes a surface designed such that a plurality of images are clearly formed at different depths from each other. FIG. 3 is a reference drawing for explaining a method of designing the freeform surface 121 according to one embodiment. Under the same conditions as the display device 10, an object image I o1 ,I o2 is arranged at the position where the virtual image is formed, and a target image I t1 ,I t2 is arranged at the position of the image generated by the image generator 11). In this way, a surface profile can be designed that satisfies the condition of minimizing the error between the target image I t1 ,I t2 and the object image I o1 ,I o2 . The error between the target image I t1 ,I t2 and the object image I o1 ,I o2 is also based on the pixel value difference between the corresponding pixels of the target image I t1 ,I t2 and the object image I o1 ,I o2 . For example, the error between the target image I t1 ,I t2 and the object image I o1 ,I o2 is the error between the target image I t1 ,I t2 and the object image I o1 ,I o2It is also the square of the absolute value related to the difference in pixel values ​​between corresponding pixels.

[0041] In order to form a clear virtual image at multiple depths, a first object image I is formed at a first depth D1 where the virtual image is formed. o1 The first target image I is placed at the first position d1 of the image generated by the image generator 110 corresponding to the first depth D1. t1 It is possible to place the second object image I at the second depth D2 where the virtual image is formed. o2 The second target image I is positioned at the second position d2 of the image generated by the image generator 110 corresponding to the second depth D2. t1 It can be arranged.

[0042] Subsequently, the first target image I t1 and the first object image I o1 Error

number

number

[0043] Equation 1 below represents the surface profile values ​​in each region of the free-form surface 121 that forms a clear virtual image at two different depths D1 and D2.

number

[0044] Figure 3 illustrates the surface profile of a freeform surface 121 that can provide a clear virtual image at two depths. The surface profile of the freeform surface 121 can also provide a clear virtual image at three or more depths.

[0045] Equation 2 below is the surface profile of a free-form surface 121 that can provide a clear virtual image at k depths (where k is a natural number greater than or equal to 2).

number

[0046] Here, I tn This is the target image generated at position dn, and I on (f(x,y),dn,Dn) is a virtual image formed at a depth of Dn,

number

[0047] Figures 4A and 4B show the results of observing a virtual image using a free-form surface optimized for a single depth. A free-form surface optimized so that a virtual image is formed at a first depth D1 was designed. After placing the image generator 110 at the first position d1, an image was generated, and as shown in Figure 4A, the observer was able to observe a clear virtual image.

[0048] Furthermore, the image generator 110 was positioned at the second position d2 so that the virtual image would be formed at the second depth D2, and then the image was generated. As illustrated in Figure 4B, the observer ended up observing a virtual image that was of poor quality and out of focus.

[0049] Figures 5A and 5B show the results of observing a virtual image using a free-form surface optimized for two depths. A free-form surface was designed that was optimized so that the virtual image would be formed at a first depth D1 and a second depth D2. After placing the image generator 110 at the first position D1 and generating an image, the observer was able to observe a clear virtual image, as illustrated in Figure 5A.

[0050] Even after the video generator 110 was placed at the second position d2 and video was generated, the observer was able to observe a clear virtual image, as illustrated in Figure 5B.

[0051] As described above, through multiple free-form surfaces 121 optimized for different depths, the observer can observe high-quality virtual images at varying depths.

[0052] Figure 6 is a diagram illustrating a display device 20 including a free-form curved surface 121 according to one embodiment. Comparing Figure 1 and Figure 6, the display device 20 in Figure 6 also includes an image generator 110, a combiner 120a as an optical system that mixes a virtual image with real light containing an external landscape and provides it to the observer, a drive unit 130 that drives the image generator 110 so that its position can be varied, and a processor 140 that controls the image generator 110 and the drive unit 130 based on image information.

[0053] The combiner 120a not only transmits light L1 containing the image generated by the image generator 110 to the observer's eye, but can also transmit light L2 containing the external scenery in front of the observer to the observer's eye. For example, the combiner 120a can reflect the light L1 containing the image towards the observer's eye and transmit the light L2 containing the external scenery towards the observer's eye.

[0054] The external light L2 contains the actual foreground present in front of the observer, rather than the image generated by the separate image generator 110. Therefore, the observer can simultaneously perceive both the artificial image from the image generator 110 and the actual foreground. This allows the display device 10 to function as a see-through type display.

[0055] The combiner 120a also includes a waveguide 122 for transmitting the image generated by the image generator 110. The waveguide 122 also includes a plurality of surfaces, at least one of which includes a free-form curved surface 121.

[0056] As shown in Figure 6, the waveguide 122 may include a first surface S1 and a second surface S2, which is a free-form curved surface 121, arranged facing each other, as well as a third surface S3 and a fourth surface S4, arranged facing each other between the first surface S1 and the second surface S2. The second surface S2 is shown as a free-form curved surface 121, but is not limited to that. The first surface S1 can also be a free-form curved surface. The third surface S3 and the fourth surface S4 are also arranged parallel to each other so as not to have refractive power.

[0057] In this embodiment, the display device 20 does not fix the position of the surface on which the virtual image is placed, i.e., the position of the virtual surface VP, to a single value, but rather the position of the virtual surface VP can be changed to reflect the sense of depth of the displayed image. For this purpose, the display device 20 may further include a drive unit 130 for driving the position of the image generator 110.

[0058] The drive unit 130 can move the image generator 110 in parallel so that the distance to the free-form curved surface 121 is variable. The shorter the distance between the image generator 110 and the free-form curved surface 121, that is, the shorter the optical path length between the image generated by the image generator 110 and the free-form curved surface 121, the deeper the virtual image corresponding to the image generated by the image generator 110 can become. However, it is not limited to this. The drive unit 130 can also tilt the image generator 110 to correspond to the depth of a pre-designed free-form curved surface 121.

[0059] The drive unit 130 may include a shape-variable material in order to have a small volume and to widen the driving range of the image generator 110. That is, the drive unit 130 can be deformed by the applied signal and provide driving force to the image generator 110. For such shape variation, a material whose shape is changed by heat may be used for the drive unit 130. The drive unit 130 may include a shape memory alloy (SMA) or an electroactive polymer (EAP). A specific explanation of how the position of the image generator 110 is changed by driving the drive unit 130 is shown in Figure 7.

[0060] The processor 140 can generate an optical modulation signal to control the video generator 110 and a drive signal to control the drive unit 130 based on the video information. The video generator 110 can then generate video at a specific location under the control of the processor 140.

[0061] Although not shown in the drawings, the display device 20 may further include memory. This memory may store various data necessary for driving the video display device, such as video information and program code.

[0062] The video information may include color information and depth information related to each frame of video. The processor 140 can generate an optical modulation signal in which a defined color value is realized by referring to the pixel-specific color information related to each frame of video included in the video information.

[0063] If the depth information included in the video information is frame-by-frame depth information, the processor 140 can generate a drive signal based on the aforementioned depth information.

[0064] If the aforementioned depth information is not frame-level depth information, for example, if it is sub-image or pixel-level depth information within a frame, the processor 140 can determine a representative depth on a frame-level basis and generate a drive signal based on that representative depth.

[0065] The processor 140 can use the color information and / or depth information contained in the video information to determine the representative depth for each video frame.

[0066] For example, the processor 140 can extract a color map from the video information, perform content analysis and / or saliency information analysis related to the color map, and determine the representative depth. This saliency information analysis is also performed to determine areas that are likely to be focused on by the observer, in other words, areas with high visual concentration. Brightness, hue, outline, object size, etc., may be considered in determining these areas with high visual concentration. For example, areas with a large difference in brightness or hue compared to the surroundings, areas with strong outline features, and areas with large objects may be areas with high visual concentration. The depth value corresponding to such areas can be determined as the representative depth. Alternatively, the content embedded in the video may determine the location with high visual concentration.

[0067] In addition, processor 140 can also determine a representative depth by considering the zone of comfort from the depth map, and can also determine a representative depth by quantizing the depth information included in the depth map.

[0068] The processor 140 can generate a drive signal based on the determined representative depth. This drive signal is also an electrical signal that generates appropriate heat for, for example, the variable material deformation of the drive unit 130. The drive signal is also transmitted with a predetermined delay from the optical modulation signal. This predetermined delay is set to be longer than the vergence-accommodation time of the observer's eye. This also takes into account the time required for the naked eye to perceive a virtual image at a modified depth.

[0069] Figure 7 is a diagram illustrating an example of a drive unit applied to Figure 6. The drive unit 130 also includes a deformation unit 310 whose shape changes according to a drive signal to adjust the position of the video generator 110, and a fixed unit 320 that supports the deformation unit 310 and is fixed in a specific position.

[0070] The deformation section 310 is also positioned between the fixed section 320 and the video generator 110. Each end of the deformation section 310 may contact the fixed section 320 and the video generator 110, respectively. A pair of deformation sections 310 is shown, but this is illustrative. In other exemplary embodiments, one deformation section or three or more deformation sections may be provided.

[0071] When an electrical signal is applied to the deformable part 310 and its temperature rises, the lengths of each part of the deformable part 310 shorten. In that case, the image generator 110 moves closer to the fixed part 320, meaning that the distance between the image generator 110 and the free-form curved surface 121 may increase.

[0072] The temperature of the deformable portion 310 is controlled, the degree of length change of each portion of the deformable portion 310 is adjusted, and the distance between the image generator 110 and the free-form curved surface 121 can be controlled.

[0073] Alternatively, the deformable portion 310 may have a wire shape. The length of the deformable portion 310 also varies depending on the temperature of the deformable portion 310 or the electric field formed within the deformable portion 310. For example, the deformable portion 310 may include a shape memory alloy (SMA), an electroactive polymer (EAP), or a combination thereof. If the deformable portion 310 includes a shape memory alloy, it may have a shorter length at higher temperatures and a longer length at lower temperatures. If the deformable portion 310 includes an electroactive polymer, when an electric field is applied to the deformable portion 310, its length may increase perpendicular to the applied electric field. Below, examples of deformable portions 310 that are deformed by temperature will be described.

[0074] The temperature of the deformation section 310 is also regulated by an electrical signal applied to the deformation section 310. This electrical signal is based on a drive signal transmitted from the processor 140 and can be either a current signal or a voltage signal. For example, applying current to the deformation section 310 can increase its temperature. If no current is applied to the deformation section 310, its temperature may be lower.

[0075] In Figures 6 and 7, the depth of the virtual image is adjusted by changing the position of the image generator 110 using the drive unit 130, but this is not the only way. When the image generator 110 generates a hologram image, the depth of the virtual image can also be expressed in different ways by changing the hologram surface of the hologram image.

[0076] Figure 8 is a diagram illustrating a hologram display device 30 including a free-form curved surface 121 according to one embodiment, and Figure 9 is a diagram illustrating an image generator 110a that generates the hologram image in Figure 8. Referring to Figure 8, the device may also include an image generator 110a that generates a hologram image, a free-form optical system 120 that shapes multiple virtual images with different depths from the hologram images generated in a time series, and a processor 140a that generates a computer-generated hologram (CGH) from the image information and provides the computer-generated hologram (CGH) to a spatial light modulator 420 (Figure 9).

[0077] The image generator 110a, as shown in Figure 9, also includes a light source 410 that provides coherent light, a spatial light modulator 420 that diffracts the incident light to generate a holographic image, and a focusing optical system 430 that forms the holographic image in a predetermined space.

[0078] The light source 410 may include a laser diode. However, other light sources can also be used as long as they emit light with a certain degree of spatial coherence, since they can be diffracted and modulated by the spatial light modulator 420 and thus become coherent.

[0079] The spatial light modulator 420 diffracts the incident light to generate a holographic image. The holographic method utilizes the principle that the object wave can be reconstructed by irradiating a hologram, which has been recorded with interference fringes between an object wave and a reference wave, with the reference wave. Recently, computer-generated holograms (CGH) have been used to generate such interference fringes.

[0080] The focusing optical system 430 is for displaying a holographic image using depth information contained in the image information. The focusing optical system 430 also includes a configuration that allows the focal position to be varied so that the holographic image generated by the spatial light modulator 420 is displayed in a different space from the spatial light modulator 420, namely on the hologram plane HP.

[0081] The focal optical system 430 may include one or more lenses. The one or more lenses may be configured to have a variable curvature or to move along the optical axis, thereby allowing the focal position to be varied and the position of the hologram plane HP on which the hologram image is displayed to be varied.

[0082] The processor 140 determines the representative depth of the hologram plane HP on which the hologram image is displayed from the three-dimensional image information, and generates a computer-generated hologram (CGH) corresponding to the representative depth based on the three-dimensional image information. The processor 140 analyzes the color information and depth information contained in the three-dimensional image information to determine the representative depth, but as mentioned above, a detailed explanation of the method for determining the representative depth will be omitted.

[0083] Under the control of the processor 140, the spatial light modulator 420 displays a hologram image on a hologram plane HP corresponding to a representative depth, and the hologram image is reflected by the free-form curved surface 121 to form virtual images corresponding to multiple hologram images of different depths.

[0084] While the system extracts representative depth from video information, it is not limited to that. The display device 10 can track the depth at which the observer fixates and provide depth information related to the virtual image.

[0085] Figure 10 is a diagram illustrating a display device 40 including an eye-tracking sensor 150 according to one embodiment. As shown in Figure 10, the display device 40 may further include an eye-tracking sensor 150 that tracks the depth to which the observer fixates.

[0086] The eye-tracking sensor 150 can acquire information related to the depth of the observer's gaze by tracking the position and direction of the observer's eyes. For example, the eye-tracking sensor 150 can acquire information related to the depth of the observer's gaze by utilizing infrared corneal reflection and techniques for detecting the direction of the line of sight. However, it is not limited to this. The eye-tracking sensor 150 can acquire an image of the pupil using computer vision technology and track changes in the pupil's position using the acquired image. The eye-tracking sensor 150 can acquire depth information of the observer's gaze by utilizing changes in the pupil's position. The eye-tracking sensor 150 can provide the acquired depth-related information to the processor 140. The eye-tracking sensor 150 may include an infrared camera, a visible light camera, or a variety of other sensors.

[0087] The processor 140 can determine a representative depth of the image based on the depth information of the observer's gaze received from the eye-tracking sensor 150. Based on the aforementioned representative depth, it can also generate a drive signal for the drive unit 130 or generate a computer-generated hologram (CGH) corresponding to the representative depth. The image display corresponding to the representative depth has been described above, but a detailed explanation will be omitted.

[0088] Figure 6 shows a waveguide in which one surface is formed as a free-form curved surface 121, but it is not limited to this. The free-form optical system 120 may also include other components besides the waveguide.

[0089] Figure 11 is a diagram illustrating a display device including an optical system 120 with a light-transmitting plate 123, according to one embodiment. Comparing the optical system 120a of Figure 6 with the optical system 120b of Figure 11, the optical system 120b of Figure 11 also further includes a light-transmitting plate 123 in contact with the waveguide 122. The light-transmitting plate 123 may include a curved surface having a shape complementary to the free-form curved surface 121, and can share the third surface S3 and the fourth surface S4 of the waveguide 122. Light L2 containing the external scenery enters the fourth surface S4, then sequentially passes through the free-form curved surface 121 and the third surface S3, and can enter the observer's eye.

[0090] A transflective film may be placed on the free-form curved surface 121 to reflect light L1 containing the image and transmit light L2 containing the external scenery. This transflective film simply reflects a portion of the incident light and allows the other portion to pass through. As a result, a portion of the light L1 containing the image is reflected from the free-form curved surface 121 by the transflective film and travels towards the observer's eye, while a portion of the light L2 containing the external scenery passes through the transflective film on the free-form curved surface 121 and travels towards the observer's eye.

[0091] If the light L1 containing the image generated by the image generator 110 has polarization characteristics, the semi-permeable film can also be configured to reflect light having a specific polarization component and transmit light having other polarization components. For example, if the light L1 containing the virtual image has a first polarization component, the semi-permeable film can reflect light having the first polarization component and transmit light having a second polarization component perpendicular to the first polarization component.

[0092] In the above description, the semipermeable film has been described as being arranged on a free-form curved surface 121, but it is not limited to this arrangement. It goes without saying that the semipermeable film and the free-form curved surface 121 can also be arranged separately from each other on the waveguide 122.

[0093] Figure 12 is a diagram illustrating a display device 60 that provides images to each eye according to one embodiment. The images provided to both eyes may be identical or images that contain parallax information.

[0094] The display device 60 may include a first image generator 110R for generating an image for the right eye, a first image synthesis member 120R for combining the image for the right eye and the real environment into one area, a first image generator 110L for generating an image for the left eye, a second image synthesis member 120L for combining the image for the left eye and the real environment into one area, and a processor 140b for controlling the first image generator 110R and the second image generator 110L so that the image is displayed at a representative depth.

[0095] The first video generator 110R and the second video generator 110L can each generate images for the right eye and left eye, respectively, under the control of the processor 140b. The first video generator 110R and the second video generator 110L are as described above, but a detailed explanation will be omitted. The processor 140b not only generates optical modulation signals so that the first video generator 110R and the second video generator 110L can generate images, but can also determine a representative depth from the image information or information received from the eye-tracking sensor.

[0096] The first image synthesis member 120R can modify at least one of the optical paths L1 for the right eye image and L2 for the real environment, thereby synthesizing the image and the real environment into a single region. Here, this single region is also the observer's right eye RE. The image synthesis member 120R can transmit multiple lights from multiple optical paths L1 and L2 to the observer's eye. The second image synthesis member 120L can modify at least one of the optical paths L3 for the left eye image and L2 for the real environment, thereby synthesizing the left eye image and the real environment into a single region. Here, this single region is also the observer's left eye RE.

[0097] The first image synthesis member 120R and the second image synthesis member 120L may include the aforementioned free-form curved surfaces. Furthermore, the first image synthesis member 120R and the second image synthesis member 120L may further include waveguides, optical transmission plates, beam splitters, or semi-transparent films.

[0098] The images transmitted by the light from the first optical path L1 and the third optical path L3 are also images provided within the augmented reality device. The real environment transmitted by the light from the second optical path L2 is also the environment that the observer faces through the augmented reality device. This real environment may include a foreground that the observer faces, or it may include a predetermined background subject. Figure 13 illustrates an example of a display device according to an exemplary embodiment applied to an automobile. This display device is also applicable to an automobile head-up display device 70. The head-up display device 70 also includes an image generator 110c provided in a region of the automobile, and at least one optical system 120d that converts the optical path so that the driver can view the image generated by the image generator 110c. The optical system 120d may include a free-form optical system according to one embodiment.

[0099] Figure 14 illustrates an example in which a display device according to an exemplary embodiment is applied to augmented reality glasses or virtual reality glasses. The augmented reality glasses 80 may include an image generator 110d that generates an image, and an optical system 120d that guides the image from the image generator 110d to enter the observer's eyes. The optical system 120d may include a free-form optical system 120 according to one embodiment.

[0100] In addition to those, the display devices 10, 20, 30, 40, 50, and 60 according to one embodiment can also be realized as various types of wearable devices, head-mounted displays (HMDs), glasses-type displays, or goggle-type displays.

[0101] The aforementioned display devices 10, 20, 30, 40, 50, and 60 can operate in conjunction with or linked to other electronic devices such as smartphones. For example, a processor that drives a video generator is also provided in a smartphone. Moreover, a smartphone may be equipped with the aforementioned display devices.

[0102] To aid in understanding the present invention, exemplary embodiments have been described and illustrated in the accompanying drawings. However, it should be understood that such embodiments are merely illustrative and not limiting. Furthermore, it should be understood that the present invention is not limited to the illustrated and described description, for various other modifications are possible for those skilled in the art. [Explanation of symbols]

[0103] 10 Display device 20 Display devices 30 Display devices 40 Display devices 50 Display devices 60 Display devices 70 display devices 80 Augmented Reality Glasses 110, 110a, 110c, 110d, 110R, 110L Video Generator 120,120a,120b,120c,120d Optical system 121 Free-form curved surface 130 Drive unit 140 processors 150 Eye Tracking Sensors

Claims

1. A video generator that modulates light and generates multiple images in a time series, An optical system including a free-form surface that forms multiple virtual images corresponding to the multiple images mentioned above in a time series at different depths from the user's eye, Includes, On the aforementioned freeform surface, the error values ​​between each of the multiple images and the corresponding multiple virtual images are less than or equal to the profile value of the freeform surface. A display device in which the profile value of the free-form curved surface is determined by the minimum value of the sum of the errors between each image and its corresponding virtual image on the free-form curved surface.

2. The display device according to claim 1, wherein the error value is based on the difference in pixel values ​​between the image generated by the image generator and the corresponding virtual image on the free-form curved surface.

3. The display device according to claim 1, wherein the free-form curved surface forms the plurality of virtual images at different depths based on the optical path lengths between the plurality of images and the free-form curved surface.

4. The display device according to claim 3, wherein the shorter the optical path length between the image generated by the image generator and the free-form curved surface, the deeper the corresponding virtual image becomes.

5. The display device according to claim 1, further comprising a processor that controls the video generator based on at least one of depth information contained in video information and depth information that the user is fixated on.

6. The display device according to claim 5, further comprising a drive unit that adjusts the position of the image generators so that the image generators generate the plurality of images at different positions, controlled by the processor.

7. The display device according to claim 6, wherein the drive unit includes a shape-variable member whose shape is changed by an applied signal and which adjusts the position of the image generator.

8. The display device according to claim 7, wherein the shape-changing member includes a material whose shape changes with heat.

9. The display device according to claim 6, wherein the drive unit comprises at least one of a shape memory alloy or an electroactive polymer.

10. The processor performs computer-generated hologram calculations on the video information. The display device according to claim 5, wherein the image generator generates a plurality of images having different representative depths from each other using computer-generated holograms received from the processor.

11. The system further includes an eye-tracking sensor that tracks the depth at which the user is fixating, The display device according to claim 5, wherein the processor controls the image generator so that the virtual image is formed at the depth at which the user gazes.

12. The optical system is a combiner that combines the multiple virtual images and external light corresponding to the external environment at a single point. The display device according to claim 1, wherein the free-form curved surface is integrated with the combiner.

13. The display device according to claim 12, wherein the combiner includes a transparent waveguide for transmitting the virtual image, and the free-form curved surface is arranged on the surface of the transparent waveguide.

14. The display apparatus according to claim 12, wherein the combiner further includes a semipermeable film arranged on the free-form curved surface.

15. The display device according to claim 1, wherein the display device is an augmented reality device.

16. In a method for operating a display device that includes a free-form curved surface, The process involves modulating light and generating multiple images sequentially using an image generator, The steps include: forming multiple virtual images corresponding to each of the multiple images in a time series at different depths using the aforementioned free-form curved surface; Includes, On the aforementioned freeform surface, the error value between each image and its corresponding virtual image is less than or equal to the profile value of the freeform surface. A method for operating a display device, wherein the profile value of the free-form curved surface is determined by the minimum value of the sum of the errors between each image and its corresponding virtual image on the free-form curved surface.

17. The method of operating the display device according to claim 16, wherein the error value is based on the difference in pixel values ​​between the image generated by the image generator and the corresponding virtual image on the free-form curved surface.

18. The method for operating a display device according to claim 16, wherein the step of forming the virtual images is to form the plurality of virtual images at different depths based on the optical path lengths of the plurality of images and the free-form curved surface.

19. The method for operating a display device according to claim 18, wherein the shorter the optical path length between the image generated by the image generator and the free-form curved surface, the deeper the virtual image corresponding to the image becomes.

20. This further includes a step to track the depth at which users focus their attention. The method of operating a display device according to claim 16, wherein the step of forming the virtual image is to form the virtual image at the depth that the user is looking at.

Images