Image display device and image display method

Abstract

An image display device includes: a processor that sets the position of a virtual image plane on which a virtual image is formed according to depth information included in first image data, and generates second image data obtained by correcting the first image data based on the set position of the virtual image plane; an image forming optical system that includes a display element configured to modulate light according to the second image data to form a display image and a light transmission unit that forms the virtual image on the virtual image plane, the virtual image corresponds to the display image formed by the display element, and the light transmission unit includes a focusing member; and a driving unit that drives the image forming optical system to adjust the position of the virtual image plane.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of Korean Patent Application No. 10-2020-0004945 filed on January 14, 2020 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] Example embodiments of the present disclosure relate to a three-dimensional image display apparatus and method. Background Art

[0004] Three-dimensional image display technology is used in various fields. For example, in recent years, the technology has been applied to image devices for virtual reality (VR) and augmented reality (AR).

[0005] A head mounted display that provides virtual reality (VR) is currently commercialized and widely adopted in the entertainment industry. In addition, the head mounted display has been developed to have a structure that can be adopted in the medical, educational, and industrial fields.

[0006] As an advanced form of virtual reality display, an augmented reality (AR) display is an image device that combines the real world with virtual reality and allows interaction between the real situation and virtual reality. The interaction between the real situation and virtual reality is based on a function for providing information about the real situation in real time, and the effect of the real situation can be further increased by overlaying virtual objects or information on the real world environment.

[0007] Such devices generally use stereoscopy technology to display three-dimensional images. Therefore, at this time, a three-dimensional image display method for multi-depth display is needed to avoid visual fatigue that may be caused by vergence-accommodation conflict. Summary of the invention

[0008] Example embodiments provide a three-dimensional (3D) image display apparatus and method allowing multi-depth display.

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of embodiments of the disclosure.

[0010] According to one aspect of the present disclosure, an image display device includes: a processor, which sets the position of a virtual image plane on which a virtual image is formed according to depth information included in first image data, and generates second image data obtained by correcting the first image data based on the set position of the virtual image plane; an image forming optical system, including a display element configured to modulate light according to the second image data to form a display image and a light transmission unit that forms the virtual image on the virtual image plane, the virtual image corresponds to the display image formed by the display element, and the light transmission unit includes a focusing member; and a driving unit, which drives the image forming optical system to adjust the position of the virtual image plane.

[0011] The first image data may include a plurality of frames, and the processor may set a corresponding position of the virtual image plane and generate corresponding second image data for each of the plurality of frames.

[0012] The respective virtual images corresponding to each of the plurality of frames may have the same respective apparent size.

[0013] The processor may generate a driving signal to control the image forming optical system according to the setting position of the virtual image plane, and may transmit the driving signal to the driving unit.

[0014] The driving unit may move the display element in a direction parallel to the optical axis of the display element according to the driving signal so that the distance between the display element and the focusing member changes.

[0015] The driving unit may include a shape-variable portion that is deformed according to a driving signal, and may provide a driving force to the display element.

[0016] The processor may generate second image data by comparing the set position of the virtual image plane with the position of a previous plane of the virtual image of a previous frame for a frame among the plurality of frames and using a correction parameter that offsets a change in magnification according to a change in the position of the virtual image plane.

[0017] The processor may generate the second image data by using a first function applied to the first image data when the position of the virtual image plane is close to the observer, or a second function applied to the first image data when the position of the virtual image plane is far from the observer.

[0018] The first function and the second function may be preset based on a time-travel curve extracted from the driving characteristics of the driving unit.

[0019] The processor can generate second image data by: obtaining an image of first image data captured by a virtual camera located at a first position; and for a frame in a plurality of frames, moving the virtual camera a distance -Δz to a second position, where Δz represents a change value obtained by comparing the set position of the virtual image plane with a previous position of the virtual image plane of a previous frame.

[0020] The processor can generate second image data by: obtaining an image of first image data captured by a virtual camera having a first field of view; and changing the first field of view of the virtual camera by -Δθ to a second field of view for a frame in a plurality of frames, where Δθ represents a change value obtained by comparing the field of view corresponding to the set position of the virtual image plane with a previous field of view corresponding to a previous position of the virtual image plane of a previous frame.

[0021] The processor may set the position of the virtual image plane by analyzing a depth map and a color map included in the first image data.

[0022] The processor may set the position of the virtual image plane by using a histogram of depth values ​​included in the depth map.

[0023] The processor may set the position of the virtual image plane by analyzing saliency information based on the color map.

[0024] The image display device may further include an eye tracking sensor that senses a depth position viewed by the observer, wherein the processor sets a position of the virtual image plane according to the sensed depth position.

[0025] The light transmission unit may form an image at the position of the observer by combining a display image formed by the display element with light entering the light transmission unit from the front side of the observer, and may transmit the image to the field of view of the observer.

[0026] The display element may include a left eye display element and a right eye display element, the second image data may include second left eye image data and second right eye image data, and the left eye display element may be configured to form a left eye display image based on the second left eye image data, and the right eye display element may be configured to form a right eye display image based on the second right eye image data.

[0027] The image display device may be a wearable device.

[0028] According to one aspect of the present disclosure, an image display method includes: setting the position of a virtual image plane on which a virtual image is formed according to depth information included in first image data; generating second image data obtained by correcting the first image data based on the set position of the virtual image plane; controlling an image forming optical system including a display element so that a virtual image is formed at the set position of the virtual image plane, the virtual image corresponding to a display image formed by the display element; and transmitting a light modulation signal based on the second image data to the display element.

[0029] The first image data may include a plurality of frames, and setting the position of the virtual image plane and generating the second image data may be performed for each of the plurality of frames.

[0030] The respective virtual images corresponding to each of the plurality of frames have the same respective apparent size.

[0031] For a frame in the plurality of frames, generating a second image number may include: comparing a set position of the virtual image plane with a previous position of the virtual image plane of a previous frame; and using a correction parameter that offsets a change in magnification according to a change in the position of the virtual image plane.

[0032] Using the correction parameter may include extracting the correction parameter from a time-travel curve preset based on driving characteristics of a driving unit for driving the display element.

[0033] Generating second image data may include: obtaining an image of first image data captured by a virtual camera located at a first position; and for a frame in a plurality of frames, moving the virtual camera a distance -Δz to a second position, where Δz represents a change value obtained by comparing the set position of the virtual image plane with a previous position of the virtual image plane of a previous frame.

[0034] Generating second image data may include: obtaining an image of first image data captured by a virtual camera having a first field of view; and changing the first field of view of the virtual camera by -Δθ to a second field of view for a frame in a plurality of frames, where Δθ represents a change value obtained by comparing a field of view corresponding to a set position of the virtual image plane with a previous field of view corresponding to a previous position of the virtual image plane of a previous frame.

[0035] According to one aspect of the present disclosure, an image display device includes: a processor configured to receive image data, the image data including first frame data corresponding to a first frame and second frame data corresponding to a second frame, the first frame being before the second frame; determining a difference between a first depth value included in the first frame data and a second depth value included in the second frame data; and generating corrected second frame data based on the determined difference; a display element configured to display an image corresponding to the corrected second frame data; and a driving unit configured to adjust a position of the display element based on the determined difference.

[0036] The image display device may also include an optical transmission unit configured to transmit a first image corresponding to the first frame data and a second image corresponding to the second frame data to the position of an observer, wherein the processor is also configured to determine the first depth value to correspond to the depth of the first virtual image perceived by the observer when the first image is transmitted, and to determine the second depth value to correspond to the depth of the second virtual image perceived by the observer when the second image is transmitted.

[0037] The processor may be further configured to determine a ratio for adjusting the size of the second frame data based on the determined difference, and generate corrected second frame data based on the determined ratio.

[0038] The processor may be further configured to adjust a position of a virtual camera viewing the second frame data based on the determined difference, and generate corrected second frame data based on the adjusted position of the virtual camera.

[0039] The processor may also be configured to adjust a field of view of a virtual camera viewing the second frame data based on the determined difference, and generate a corrected second frame data based on the adjusted field of view of the virtual camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other aspects, features and advantages of certain example embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0041] Figure 1 is a diagram showing a configuration and an optical arrangement of an image display device according to an example embodiment;

[0042] Figure 2 It is schematically shown Figure 1 a block diagram of a configuration of a processor included in an image display device;

[0043] FIG. 3A to FIG. 3D is a diagram conceptually showing that an observer's eyes perceive images having different depths as images having different sizes;

[0044] Figure 4 is a detailed diagram showing the Figure 1 A flowchart of an example operation of a processor in an image display device;

[0045] Figure 5A An example of a function for image correction corresponding to a near to far situation is shown;

[0046] Figure 5B An example of a function for image correction corresponding to a far to near situation is shown;

[0047] Figure 6 Show Figure 1 An example configuration of a driving unit included in an image display device;

[0048] Figure 7 is a flowchart illustrating in detail an example operation of a processor in an image display device according to an example embodiment;

[0049] Fig. 8A and Figure 8B Conceptually, Figure 7 When an image forming optical system included in an image display device displays images at different depth positions, corrected image information is inputted in consideration of a change direction of the depth position;

[0050] Fig. 9 is a flowchart illustrating in detail an example operation of a processor in an image display device according to an example embodiment;

[0051] Fig. 10A and Fig. 10B Conceptually, Fig. 9 When an image forming optical system included in an image display device displays images at different depth positions, corrected image information is inputted in consideration of a change direction of the depth position;

[0052] Fig.11 shows a configuration and an optical arrangement of an image display device according to an example embodiment;

[0053] Fig.12 shows a configuration and an optical arrangement of an image display device according to an example embodiment; and

[0054] Fig.13 is a flowchart schematically illustrating a method for image display according to example embodiments. DETAILED DESCRIPTION

[0055] Example embodiments will now be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein.

[0056] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of..." when following a list of elements modify the entire list of elements, rather than modifying the individual elements in the list. For example, the expression "at least one of a, b, and c" should be understood to include: only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0057] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. In the accompanying drawings, similar reference numerals refer to similar elements, and the size of each element in the drawings may be exaggerated for clarity and convenience of description. The embodiments to be described below are merely examples, and various modifications may be made from these embodiments.

[0058] Hereinafter, what is described as “over” or “on” may include not only being directly above and in contact with, but also being above but not in contact with.

[0059] Unless the context clearly indicates otherwise, a singular expression includes a plural expression. In addition, when a component is described as “comprising” a certain configuration element, unless otherwise mentioned, this means that the component may further include other configuration elements.

[0060] The term "above" and similar terms may be used in the singular as well as in the plural.

[0061] Unless the order of the steps of the configuration method is clearly stated, the steps can be performed in any appropriate order. The present disclosure is not limited to the description order of the steps. The use of all examples or example terms (e.g., etc.) is only used to describe the technical concept in detail, and the scope is not limited by the examples or example terms.

[0062] Figure 1 is a diagram illustrating a configuration and an optical arrangement of an image display device according to an example embodiment. Figure 2 It is schematically shown Figure 1 A block diagram of a configuration of a processor included in an image display device.

[0063] refer to Figure 1, the image display device 1000 includes an image forming optical system, a driving unit 200, and a processor 400, wherein the image forming optical system includes a display element 100 that modulates light to form an image (i.e., displays an image) and a light transmission unit 300 that transmits the image formed by the display element 100 to the eyes of an observer, the driving unit 200 drives the image forming optical system to adjust the position of a virtual image plane formed by the image forming optical system, and the processor 400 controls the display element 100 and the driving unit 200 according to image information. The processor 400 includes an image correction unit 450 that corrects the image in consideration of the position of the variable virtual image plane.

[0064] The image display device 1000 also includes a memory 500 which may store image information, various data, and program codes required to drive the image display device 1000 .

[0065] The image display device 1000 may further include an eye tracking sensor that senses a depth position observed by the observer.

[0066] The display element 100 modulates light according to the image information to form an image to be provided to the observer. The image formed by the display element 100 can be provided to the two eyes of the observer, and for convenience, the figure only shows an optical system facing a single eye. The image formed by the display element 100 can be, for example, a stereoscopic image provided to the left and right eyes of the observer, can be a holographic image, a light field image or an integrated photography (IP) image, and can also include a multi-view image (multi-view image) or a super multi-view image. In addition, the present disclosure is not limited thereto, and can include a general two-dimensional image.

[0067] The display element 100 may include, for example, a liquid crystal on silicon (LCoS) element, a liquid crystal display (LCD) element, an organic light emitting diode (OLED) display element, or a digital micromirror device (DMD), and may also include next generation displays such as micro LEDs and quantum dot (QD) LEDs.

[0068] The display element 100 may include a light source for providing light for forming an image, and in addition, the display element 100 may further include a beam splitter for adjusting an optical path, a relay lens for enlarging or reducing an image, a spatial filter for removing noise, and the like.

[0069] The light transmission unit 300 changes the path of the image formed by the display element 100, forms an image suitable for the viewer's field of view, and transmits the image to the viewer's eyes. The light transmission unit 300 may include a focusing member 310, and may further include a beam splitter 330 as a member for diverting light to change the path.

[0070] The focusing member 310 is an imaging member having a refractive power and can magnify or reduce an image formed by the display element 100. The focusing member 310 is shown as a concave mirror, but is not limited thereto. In addition to or instead of the concave mirror, the focusing member 310 may also have a configuration in which a convex lens, a concave lens, etc. are combined.

[0071] The beam splitter 330 may be a half mirror that transmits half of the incident light and reflects the other half. The beam splitter is not limited thereto and may be a polarization beam splitter that transmits or reflects the incident light according to polarization. When the beam splitter 330 is a polarization beam splitter, an additional optical element for polarization switching may be further provided to the optical transmission unit 300.

[0072] like Figure 1 As shown, the light transmission unit 300 may have an integral structure in which the focusing member 310 and the beam splitter 330 are fixed to the transparent light guide member 350. This is an example, and the present disclosure is not limited thereto.

[0073] The light transmission unit 300 can combine the light including the image formed by the display element 100 with the light including the image of the real environment in front of the observer, and transmit the combined light to the observer's field of view. Therefore, the image display device 1000 can be used as a see-through type display that displays both the image formed by the display element 100 and the real environment in front of the observer.

[0074] The light transmission unit 300 is not limited to the shape and configuration shown. An additional optical element for transmitting the image formed by the display element 100 to the pupil of the observer together with the real environment image in front of the observer may be further provided, and an optical window having various shapes and structures may be adopted.

[0075] The light including the image formed by the display element 100 passes through the beam splitter 330, is reflected by the focusing member 310, and then reaches the observer's eyes through a path reflected again by the beam splitter 330. In this path, the observer perceives a virtual image on a virtual image plane VP formed at a certain position behind the beam splitter 330, and there is a difference in the depth sense felt by the observer depending on the position of the virtual image plane VP.

[0076] In the image display device 1000 according to the exemplary embodiment, the position of the virtual image plane VP is not fixed to one position, and the position of the virtual image plane VP can be changed by considering the depth information of the image to be displayed. To this end, the image display device 1000 includes a driving unit 200 that drives the image forming optical system. Figure 1 As shown, the driving unit 200 may be configured to change the position of the display element 100. The driving unit 200 may move the display element 100 in a direction parallel to the optical axis of the display element 100 so that the distance to the focusing member 310 is variable. When the position of the display element 100 changes in the direction A1, the position of the virtual image plane VP changes in the direction A2. This is an example, and in an example embodiment, the driving unit 200 may drive the focusing member 310 to change the position of the virtual image plane VP.

[0077] The driving unit 200 employed in the image display device 1000 may include various types of actuators. The driving unit 200 may employ a shape-variable material in order to increase the driving range of the display element 100 with as small a volume as possible. The type of the driving unit 200 is not particularly limited.

[0078] The processor 400 may generate a light modulation signal SG1 to be transmitted to the display element 100 and a driving signal SG2 to be transmitted to the driving unit 200 according to depth information related to an image to be perceived by an observer.

[0079] like Figure 2 As shown, the processor 400 may set the position of a virtual image plane on which an image is formed according to depth information included in the first image data, and may generate second image data obtained by correcting the first image data based on the position of the virtual image plane.

[0080] The first image data may include a depth map and a color map. That is, the first image data may include color value data for each pixel of a plurality of pixels of each image of a plurality of frames to be provided to an observer, and may also include depth information linked to the position of a virtual image plane VP where each image is imaged. The first image data may be stored in a frame buffer.

[0081] The depth information extraction unit 410 analyzes the first image data to extract depth information, and sets the depth position of the displayed image, that is, the position of the virtual image plane. Position setting can be performed for each of the images of multiple frames. In order to extract depth information, the depth map and color map included in the first image data can be analyzed. For example, the position of the virtual image plane can be set by using a histogram of the frequency of the depth value included in the depth map. Alternatively, the position of the virtual image plane can be set by analyzing the saliency information from the color map. Saliency map analysis can be performed to select an area that the observer may observe, that is, an area with high visual concentration. Brightness, color, contour, object size, etc. can be considered for selecting an area with high visual concentration. For example, an area with large brightness or large color difference compared to the surrounding area, an area with strong contour features, or an area with a large object size can be an area with high visual concentration. Alternatively, a position with high visual concentration can be selected depending on the content included in the image. The position of the virtual image plane can be set in consideration of visual concentration. In addition to this, in order to set the position of the virtual image plane, analysis of motion information in which human visual perception characteristics are taken into consideration or zone of comfort (ZOC) analysis may be performed.

[0082] When the image display device 1000 includes an eye tracking sensor that senses a depth position observed by the observer, the depth information extraction unit 410 may also set the position of the virtual image plane based on the depth position sensed by the eye tracking sensor.

[0083] The drive signal generating unit 420 may generate a drive signal SG2 that moves the display element 100 so that an image is formed at the position of the set virtual image plane. The drive signal SG2 may be, for example, an electrical signal that generates heat suitable for deformation of the variable material of the drive unit 200. The drive signal is not limited thereto. The drive signal SG2 generated by the drive signal generating unit 420 is transmitted to the drive unit 200.

[0084] The image correction unit 450 generates second image data as corrected first image data according to the depth information extracted by the depth information extraction unit 410. The light modulation signal generation unit 460 generates a light modulation signal SG1 according to the second image data and transmits the light modulation signal to the display element 100. That is, the light modulation signal generation unit 460 may generate an electrical signal having a color value determined by referring to data of each pixel included in the second image data as the light modulation signal SG1.

[0085] The image correction performed by the image correction unit 450 is for correcting image distortion that occurs when the position of the virtual image plane changes according to the position movement of the display element 100 .

[0086] Will refer to Figure 1 and Figures 3A to 3D Image distortion that occurs when the position of the virtual image plane changes is described as follows.

[0087] FIG. 3A to FIG. 3D 2 is a diagram conceptually showing that an observer's eyes perceive images having different depth positions as images having different sizes.

[0088] refer to Figure 3A , when a concave mirror forms images of objects ① and ② with different depth positions (far and near), the formed images ①' and ②' have different sizes, respectively, and are also perceived as spanning different fields of view θ1 and θ2.

[0089] like Figure 3B and Figure 3C As shown, when the position of the virtual image plane is the far plane and the near plane, the viewing angle is FOV far and FOV near , and has FOV far >FOV near In addition, Figure 3D As shown, the screen size of the near plane is perceived as smaller than the screen size of the far plane.

[0090] This can also be explained by the concept of magnification. The magnification m is defined as the image distance / object distance and is Figure 1 In the middle is d i / d o Here, d o is the distance from the image display surface of the display element 100 to the focusing member 310, and d i is the distance along the virtual optical path from the focusing member 310 to the virtual image plane VP, and the distance d shown is i and d i ' respectively indicate the sum of the distance from the center of the focusing member 310 to the center of the beam splitter 330 and the distance from the center of the beam splitter 330 to the virtual image plane VP, and the sum of the distance from the center of the focusing member 310 to the center of the beam splitter 330 and the distance from the center of the beam splitter 330 to the virtual image plane VP'.

[0091] When d is determined according to the position of the display element 100 and the position of the virtual image plane VP o and d i When the magnification at this time is referred to as m, the magnification m' corresponding to the position of the changed virtual image plane VP' is d i ' / d o ', and this value is different from m.

[0092] The image display device 1000 according to the exemplary embodiment generates second image data corrected by considering the position of the virtual image plane to be formed to correct the image distortion caused by the magnification change made in the continuous frame image, and provides the light modulation signal SG1 according to it to the display element 100. The second image data is data corrected to display an image of an apparent size as constant as possible at the position of the virtual image plane that changes frame by frame. In addition, the second image data may be image data obtained by correcting other image distortions (e.g., pincushion distortion, barrel distortion, image misalignment, etc.) generated according to the change in the position of the virtual image plane.

[0093] Figure 4 It is shown in detail Figure 1 FIG. 1 is a flowchart of an example operation of the processor 400 in the image display device 1000 .

[0094] When first image data of an image to be formed in each frame is input ( S110 ), a position of a virtual image plane is set according to depth information included in the first image data ( S120 ).

[0095] When the position of the virtual image plane is set, a driving signal SG2 for driving the display element 100 to the position is generated ( S130 ), and the driving signal is transmitted to the driving unit 200 ( S140 ).

[0096] The position of the set virtual image plane is compared with the position of the virtual image plane of the immediately preceding frame (S150).

[0097] Depending on whether the position of the virtual image plane changes from near to far or from far to near (i.e., whether the virtual image plane becomes farther away or closer), correction parameters of different functions may be applied. When the position of the virtual image plane moves farther, the near-to-far function is applied (S164), and when the position of the virtual image plane becomes closer, the far-to-near function is applied (S162).

[0098] Figure 5A and Figure 5B Example graphs showing functions corresponding to a near-to-far situation and a far-to-near situation, respectively, as functions for image correction.

[0099] The two diagrams are time-stroke curves extracted from the drive characteristics of the drive unit. From the time-stroke data for both the far-to-near situation and the near-to-far situation, a function for determining correction parameters can be preset in each case.

[0100] Figure 5AIncluded are results of measurement data in which the position of a display element changes from far to near according to the operation of a driving unit, and a function graph extracted by performing regression analysis of the measurement results using a polynomial function.

[0101] When the position of the virtual image plane changes from far to near, the image becomes smaller, e.g. Figure 3A and Figure 3B As shown, and to counteract this, a resize function for image enlargement may be provided to adjust the image size.

[0102] Can be obtained from Figure 5A The function y extracted from sets the function (ratio) as follows.

[0103] Ratio = 1 + y*(α)

[0104] Here, α is a value between 0 and 1. α may be, for example, 0.01, but is not limited thereto, and other appropriate values ​​may be selected.

[0105] Figure 5B Included are results of measurement data in which the position of a display element changes from near to far according to the operation of a driving unit, and a function graph extracted by performing regression analysis of the measurement results using a polynomial function.

[0106] When the position of the virtual image plane changes from near to far, the image is enlarged, e.g. Figure 3A and Figure 3B As shown, and to counteract this, a resize function for image reduction may be provided to adjust the image size.

[0107] Can be obtained from Figure 5B The function y extracted from sets the function (ratio) as follows.

[0108] Ratio = 1-(1-y)*(α)

[0109] Here, α is a value between 0 and 1. α may be, for example, 0.01, but is not limited thereto, and other appropriate values ​​may be selected.

[0110] Return to reference Figure 4 , generating second image data by correcting the image according to the set correction parameters (S170).

[0111] A light modulation signal SG1 is generated according to the second image data (S180), and the light modulation signal is transmitted to the display element (S190).

[0112] The image enlargement process and the image reduction process may be continuously performed frame by frame according to the change in the position of the display element 100 .

[0113] The image forming optical system displays an image according to the light modulation signal SG1 and the drive signal SG2, and the observer can perceive the displayed image from the changed position of the virtual image plane VP. The observer perceives the image with a small amount of image distortion caused by the change in the position of the virtual image plane.

[0114] Figure 6 Show Figure 1 An example configuration of a driving unit included in an image display device.

[0115] The driving unit 200 may include a shape variable portion 210, a supporting portion 240, and a driving control unit 230. A plurality of shape variable portions 210 may be provided so that both ends thereof may be in contact with the supporting portion 240 and the display element 100, respectively. Although a pair of shape variable portions 210 is shown, this is an example, and the number of shape variable portions 210 may vary.

[0116] The length of the shape variable portion 210 may change depending on the temperature of the shape variable portion 210 or the electric field formed in the shape variable portion 210. For example, the shape variable portion 210 may include a shape memory alloy (SMA), an electroactive polymer (EAP), or a combination thereof. When the shape variable portion 210 includes a shape memory alloy, the shape variable portion 210 may have a short length at a high temperature and a long length at a low temperature. When the shape variable portion 210 includes an electroactive polymer, and when an electric field is applied to the shape variable portion 210, the length of the shape variable portion 210 may increase in a direction perpendicular to the applied electric field.

[0117] The driving control unit 230 may be electrically connected to the shape variable portion 210 to apply an electrical signal to the shape variable portion 210. For example, the driving control unit 230 may apply current to the shape variable portion 210. When the electrical signal is applied to the shape variable portion 210 and the temperature of the shape variable portion 210 increases, the length of the shape variable portion 210 may decrease. In this case, the display element 100 approaches the support portion 240, that is, the distance to the focusing member 310 increases. When no current is applied to the shape variable portion 210, the temperature of the shape variable portion 210 drops, and its length may increase. In this way, the temperature of the shape variable portion 210 is controlled, and therefore, the degree of change in the length of the shape variable portion 210 can be adjusted, and the distance between the display element 100 and the focusing member 310 can be adjusted. As Figure 6 As shown, as the shape variable portion 210 is deformed in the A1 direction, a driving force in the A1 direction is provided to the display element 100, and the distance between the display element 100 and the focusing member 310 can be adjusted. Figure 1The position of the virtual image plane VP on which the image formed by the display element 100 is formed is changed in the A2 direction shown.

[0118] The illustrated shape of the driving unit 200 is an example, and may be changed to other shapes capable of providing a driving force to the display element 100 in the A1 direction.

[0119] Figure 7 is a flowchart specifically illustrating an example operation of a processor in an image display device according to an example embodiment.

[0120] The image display device according to the present exemplary embodiment is provided with a processor in a detailed method of generating second image data obtained by correcting first image data. Figure 1 The image shown shows a different device and its configuration diagram is different from Figure 1 The configuration diagrams shown are essentially the same.

[0121] In the operation of the processor, when the first image data is input (S110), the position of the virtual image plane is set from it according to the depth information (S120), the drive signal SG2 is generated (S130), and the drive signal is transmitted to the drive unit (S140), which is related to Figure 4 Same as described in .

[0122] Processing and reference for generating second image data based on depth information Figure 4 In this example embodiment, the second image data is generated by obtaining an image of the first image data from a virtual camera. At this time, the position of the virtual camera is changed according to the depth information to generate the second image data. The method may have the same Figure 4 The same effect is achieved by using the resize function to correct the image as described.

[0123] First, for the set position of the virtual image plane, a change value Δz of the position of the virtual image plane of the current frame (i.e., the second frame) from the position of the virtual image plane of the previous frame (i.e., the first frame) (i.e., the first depth value) is extracted (S155). Next, the second image data (i.e., the corrected second frame data) is generated by moving the virtual camera in the direction opposite to the change value, i.e., moving the virtual camera by -Δz to obtain an image of the first image data (S175).

[0124] Fig. 8A and Figure 8B Conceptually, Figure 7 When an image forming optical system included in an image display device displays images at different depth positions in different frames, corrected image information is input in consideration of a change direction of the depth position from a previous frame to a current frame.

[0125] like Fig. 8A As shown, when the position of the virtual image plane is moved far (i.e., from near to far) by +Δz, the apparent size of the image is enlarged. At this time, the size of the image input to the image forming optical system can be reduced so that the apparent image size is maintained even at the changed position of the virtual image plane. That is, the second image data obtained by correcting the image is input to the image forming optical system. Correction of the image includes moving the position of the virtual camera by -Δz to obtain an image of the first image data from the new position of the virtual camera.

[0126] like Figure 8B As shown, when the position of the virtual image plane approaches from far to near, that is, when the position of the virtual image plane changes by -Δz, the apparent size of the image decreases. At this time, the image input to the image forming optical system can be enlarged so that the apparent image size is maintained even at the changed position of the virtual image plane. That is, the second image data obtained by correcting the image is input to the image forming optical system. Correction of the image includes moving the position of the virtual camera by +Δz to obtain an image of the first image data from the new position of the virtual camera.

[0127] In this way, even when the position of the virtual image plane changes, an image with a relatively constant image size can be perceived.

[0128] Fig. 9 is a flowchart illustrating in detail an example operation of a processor in an image display device according to an example embodiment.

[0129] The image display device according to the present exemplary embodiment is provided with a processor in a detailed method of generating second image data obtained by correcting first image data. Figure 1 The image shown shows a different device and its configuration diagram is different from Figure 1 The configuration diagrams shown are essentially the same.

[0130] In the operation of the processor, when the first image data is input (S110), the position of the virtual image plane is set from it according to the depth information (S120), a driving signal (SG2) is generated (S130), and the driving signal is transmitted to the driving unit (S140). In addition, as shown in reference Figure 7 The second image data is generated by obtaining an image of the first image data from a virtual camera.

[0131] In this example embodiment, the second image data is generated by changing the field of view (FOV) of the virtual camera according to the depth information. Figure 4 The same effect is achieved by using the resize function to correct the image as described.

[0132] First, for the set virtual image plane, a change value Δθ between the perspective (i.e., field of view) of the position of the virtual image plane of the current frame and the perspective of the position of the virtual image plane of the previous frame is extracted (S157). Next, the second image data is generated by changing the perspective of the virtual camera by -Δθ to obtain an image of the first image data using the new perspective of the virtual camera (S177).

[0133] Fig. 10A and Fig. 10B Conceptually, Fig. 9 When an image forming optical system included in an image display device displays an image at different depth positions, corrected image information is input in consideration of a changing direction of the depth position.

[0134] like Fig. 10A As shown, when the position of the virtual image plane moves away (i.e., from near to far), the viewing angle changes by Δθ, and the apparent size of the image is enlarged. At this time, the image input to the image forming optical system can be reduced so that the apparent image size is maintained even at the changed position of the virtual image plane. The second image data obtained by correcting the image is input to the image forming optical system. Correction of the image includes changing the viewing angle of the virtual camera by -Δθ.

[0135] like Fig. 10B As shown, when the position of the virtual image plane approaches (i.e., from far to near), the viewing angle changes by -Δθ, and the apparent size of the image decreases. At this time, the image input to the image forming optical system can be enlarged so that the apparent image size is maintained even at the changed position of the virtual image plane. The second image data obtained by correcting the image is input to the image forming optical system. Correction of the image includes enlarging the viewing angle of the virtual camera by +Δθ.

[0136] Even when the position of the virtual image plane changes from frame to frame, an image with a relatively constant image size can be perceived.

[0137] Fig.11 A configuration and an optical arrangement of an image display device according to an example embodiment are shown.

[0138] The image display device 1001 according to the present exemplary embodiment and Figure 1 The image display device 1000 is different from the image display device 1000 in that the device and method for driving to change the position of the virtual image plane are different. The rest of the configuration is the same as Figure 1 The configurations shown are essentially the same.

[0139] Different from moving its position by the driving unit 200 Figure 1In the display element 100 of the embodiment, in the present exemplary embodiment, the position of the focusing member 315 is adjusted by the driving unit 201. When the focusing member 315 is driven in the A1 direction by the driving unit 201, the position of the virtual image plane VP may be adjusted in the A2 direction.

[0140] In addition, the position of the virtual image plane formed by the image forming optical system may be adjusted in various ways. For example, the refractive power of the focusing member 315 may be electrically controlled. By changing the shape of the curved surface of the focusing member 315 or by changing the refractive index of the focusing member 315, the refractive power may be adjusted and thus the position of the virtual image plane may be adjusted.

[0141] Fig.12 A configuration and an optical arrangement of an image display device according to an example embodiment are shown.

[0142] The image display device 2000 according to the present exemplary embodiment may have a configuration of providing images to both eyes. The images provided to both eyes may be the same or may have parallax information.

[0143] The image display device 2000 may include a first display element 160 (ie, a left eye display element), a first driving unit 260 , a first light transmission unit 360 , a second display element 170 (ie, a right eye display element), a second driving unit 270 , and a second light transmission unit 370 .

[0144] The first driving unit 260 and the second driving unit 270 drive the positions of the first display element 160 and the second display element 170, respectively, and the first light transmission unit 360 and the second light transmission unit 370 transmit images formed by the first display element 160 and the second display element 170 (i.e., a left-eye display image and a right-eye display image) to the left eye and the right eye of the observer, respectively, based on the second left-eye image data and the second right-eye image data, respectively included in the second image data.

[0145] The image display device 2000 further includes a processor 800 and a memory 700. The processor 800 controls the first display element 160, the second display element 170, the first drive unit 260, and the second drive unit 270 according to the image stored in the memory 700 and the depth information included in the image. The processor 800 also includes an image correction unit 850 that corrects image distortion according to the change in the position of the virtual image plane. Program code for operating the processor may be stored in the memory 700.

[0146] The image information stored in the memory 700 may include information about a pair of left-eye and right-eye images that can be perceived as a three-dimensional image of a frame. The left-eye image and the right-eye image have certain differences. The processor 800 generates light modulation signals SG1_L and SG1_R so that the left-eye image is formed by the first display element 160 and the right-eye image is formed by the second display element 170, and therefore, the first display element 160 and the second display element 170 are controlled by the processor 800.

[0147] The image information may further include depth information linked to the position of the virtual image plane VP where the left eye image and the right eye image will be imaged, and the processor 800 may generate drive signals SG2_L and SG2_R so that the virtual image plane VP is formed at a set depth position according to the depth information, and control the first drive unit 260 and the second drive unit 270.

[0148] The processor 800 includes an image correction unit that corrects image distortion caused by a change in the position of the virtual image plane. The detailed process of correcting the image by determining whether the position of the virtual image plane is changed and using the image correction unit 850 is the same as that of the above reference. Figure 4 , Figure 7 and Fig. 9 The described processing is substantially the same, and the only difference is that the processing is performed on the binocular images. The degree of correction of each of the binocular images may be the same as or may be different from each other.

[0149] The image display apparatus 2000 according to example embodiments may display a three-dimensional image by combining a binocular parallax method and depth expression.

[0150] Fig.13 is a flowchart schematically illustrating a method for image display according to example embodiments.

[0151] The method for image display may be performed by the above-described image display apparatuses 1000 , 1001 , and 2000 or an image display apparatus having a configuration modified from the image display apparatus.

[0152] First, the position of a virtual image plane is set according to depth information included in first image data (S200).

[0153] Next, second image data is generated by correcting the first image data based on the position of the set virtual image plane (S300).

[0154] The step of setting the position of the virtual image plane (S200) and the step of generating the second image data (S300) are performed for each of the plurality of frames, and generating the second image data is a process of correcting the first image data so that an image of a constant apparent size is displayed at the changed position of the virtual image plane. That is, in order to generate the second image data, the position of the virtual image plane of the current frame is compared with the position of the virtual image plane of the previous frame, and then, the second image data is generated by correcting the first image data based on the comparison result.

[0155] In order to generate the second image data, a method of determining whether the position of the set virtual image plane is near to far or far to near relative to the position of the previous frame and using a resizing correction parameter associated therewith (S310), or a method of moving the virtual camera in a direction opposite to the change in the position of the virtual image plane (S320), or a method of changing the viewing angle of the virtual camera so that the screen size according to the change in the position of the virtual image plane is offset (S330) may be used, as in Figure 4 , Figure 7 and Fig. 9 Although the above description mainly focuses on correcting distortion according to changes in the apparent size of the image, the present disclosure is not limited thereto. Additional correction processing of other image distortions (e.g., pincushion distortion, barrel distortion, image misalignment, etc.) generated according to changes in the position of the virtual image plane may be further performed.

[0156] In addition, an image forming optical system including a display element is controlled so that an image formed by the display element corresponds to a virtual image formed at the position of the set virtual image plane (S400). In order to adjust the position of the virtual image plane formed by the image forming optical system, the position of the display element may be adjusted (S410), or the position or the refractive power of the focusing member may be adjusted (S420), as described above with reference to Figure 1 and Fig.11 described.

[0157] Next, the display element outputs an image according to the light modulation signal based on the second image data (S500). The output image is a corrected image that provides the observer with a sense of depth provided to the set virtual image plane and also has small image distortion, such as changes in image size according to changes in the position of the virtual image plane.

[0158] The above-described method for image display may be performed by the above-described image display devices 1000 , 1001 , and 2000 , and may be applied to modified examples thereof and various optical devices and electronic devices that may utilize the image display devices.

[0159] The above-mentioned image display devices 1000, 1001 and 2000 may be configured in a wearable form. All or part of the configuration elements of the image display device may be configured in a wearable form.

[0160] For example, the image display devices 1000, 1001, and 2000 may be employed in the form of a head mounted display (HMD). In addition, the present disclosure is not limited thereto, and may be applied to a glasses type display or a goggle type display.

[0161] The above-described image display devices 1000 , 1001 , and 2000 may be employed to realize augmented reality (AR) because an image formed by a display element and an image of the real world may be displayed together to an observer.

[0162] Augmented reality (AR) can further improve the effectiveness of real-world images by displaying a combination of virtual objects or information with the real-world environment. For example, additional information about the real-world environment formed by an image forming unit at the observer's position can be provided to the observer. Augmented reality (AR) displays can be applied to ubiquitous environments or Internet of Things (IoT) environments.

[0163] The image of the real world is not limited to the real environment, and may be, for example, an image formed by another image device. Therefore, the above-mentioned image display device may be employed in a multi-image display device that displays two images together.

[0164] The image display devices 1000, 1001, and 2000 described above may be combined with other electronic devices (e.g., smart phones) or may be operated in a state of being connected to other electronic devices. For example, a processor that drives the image display devices 1000, 1001, or 2000 may be provided in a smart phone. In addition, the image display devices 1000, 1001, or 2000 described above may be provided in a smart phone.

[0165] According to the above-described apparatus and method for three-dimensional image display, different depths can be expressed for each of consecutive frame images while maintaining an apparent image size, and a high-quality image with less image distortion can be provided.

[0166] The above-mentioned device for three-dimensional image display can be easily used in wearable devices, such as glasses-type augmented reality display devices.

[0167] It should be understood that the example embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. The description of features or aspects within each example embodiment should generally be considered applicable to other similar features or aspects in other embodiments. Although the example embodiments have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. An image display device, comprising: The processor is configured as: setting, for each of the plurality of frames, a position of a virtual image plane on which a virtual image is formed, according to depth information included in first image data including a plurality of frames; as well as generating, for each of the plurality of frames, second image data obtained by correcting the first image data based on a set position of the virtual image plane by comparing the set position of the virtual image plane with a previous position of a virtual image plane of a previous frame and using a correction parameter that cancels a change in magnification according to a change in the position of the virtual image plane; An image forming optical system comprising: A display element configured to modulate light according to the second image data to form a display image; and a light transmission unit configured to form the virtual image on the virtual image plane, wherein the virtual image corresponds to the display image formed by the display element, and the light transmission unit includes a focusing member; and a driving unit configured to drive the image forming optical system to adjust the position of the virtual image plane, wherein the processor is further configured to generate a driving signal to control the image forming optical system according to the setting position of the virtual image plane, and transmit the driving signal to the driving unit, wherein the driving unit is configured to move the display element in a direction parallel to the optical axis of the display element according to the driving signal so as to change the distance between the display element and the focusing member, wherein the processor is further configured to generate the second image data by using a first function applied to the first image data when the position of the virtual image plane is close to the observer, or a second function applied to the first image data when the position of the virtual image plane is far from the observer, The first function is preset based on a first time-travel curve regarding the position change of the display element from far to near, the second function is preset based on a second time-travel curve regarding the position change of the display element from near to far, and the first time-travel curve and the second time-travel curve are extracted from the driving characteristics of the driving unit.

2. The image display device according to claim 1, wherein: The respective virtual images corresponding to each of the plurality of frames have the same respective apparent size.

3. The image display device according to claim 1, wherein: The driving unit includes a shape-variable portion that is deformed according to the driving signal, and provides a driving force to the display element.

4. The image display device according to claim 1, wherein: The processor is further configured to generate the second image data by: obtaining an image of first image data captured by a virtual camera located at a first position; as well as For a frame in the plurality of frames, the virtual camera is moved a distance -Δz to a second position, wherein Δz represents a change value obtained by comparing the set position of the virtual image plane with a previous position of the virtual image plane of a previous frame.

5. The image display device according to claim 1, wherein: The processor is further configured to generate the second image data by: obtaining an image of first image data captured by a virtual camera having a first field of view; as well as For a frame in the plurality of frames, a first field of view of the virtual camera is changed by -Δθ to a second field of view, wherein Δθ represents a change value obtained by comparing a field of view corresponding to a set position of the virtual image plane with a previous field of view corresponding to a previous position of the virtual image plane of a previous frame.

6. The image display device according to claim 1, wherein: The processor is further configured to set the position of the virtual image plane by analyzing a depth map and a color map included in the first image data.

7. The image display device according to claim 6, wherein: The processor is further configured to set a position of the virtual image plane by using a histogram of depth values ​​included in the depth map.

8. The image display device according to claim 6, wherein: The processor is further configured to set a position of the virtual image plane by analyzing saliency information based on the color map.

9. The image display device according to claim 1, further comprising an eye tracking sensor configured to sense a depth position viewed by an observer, in, The processor is further configured to set the position of the virtual image plane according to the sensed depth position.

10. The image display device according to claim 1, wherein: The light transmission unit is also configured to set an image at the position of the observer by combining the display image formed by the display element with light entering the light transmission unit from the front of the observer, and transmit the image to the observer's field of view.

11. The image display device according to claim 1, wherein: The display element includes a left-eye display element and a right-eye display element, The second image data includes second left-eye image data and second right-eye image data, and The left-eye display element is configured to form a left-eye display image according to the second left-eye image data, and the right-eye display element is configured to form a right-eye display image according to the second right-eye image data.

12. The image display device according to claim 1, wherein: The image display device is a wearable device.

13. An image display method using the image display device according to any one of claims 1 to 12, the image display method comprising: setting, for each of the plurality of frames, a position of a virtual image plane on which a virtual image is formed, according to depth information included in first image data including a plurality of frames; generating, for each of the plurality of frames, second image data obtained by correcting the first image data based on the set position of the virtual image plane by comparing the set position of the virtual image plane with a previous position of the virtual image plane of a previous frame and using a correction parameter that cancels out a change in magnification according to a change in the position of the virtual image plane, wherein using the correction parameter comprises presetting the correction parameter based on a time-travel curve regarding a position change of the display element, the time-travel curve being extracted from a driving characteristic of the driving unit; controlling the image forming optical system so that the virtual image is formed at a setting position of the virtual image plane, wherein the virtual image corresponds to a display image formed by the display element; and A light modulation signal based on the second image data is transmitted to the display element.

14. The image display method according to claim 13, wherein: The respective virtual images corresponding to each of the plurality of frames have the same respective apparent size.

15. The image display method according to claim 13, wherein: Generating the second image data comprises: obtaining an image of the first image data captured by a virtual camera located at a first position; and For a frame in the plurality of frames, the virtual camera is moved a distance -Δz to a second position, wherein Δz represents a change value obtained by comparing the set position of the virtual image plane with a previous position of the virtual image plane of a previous frame.

16. The image display method according to claim 13, wherein: Generating the second image data comprises: obtaining an image of the first image data captured by a virtual camera having a first field of view; and For a frame in the plurality of frames, the first field of view of the virtual camera is changed by -Δθ to a second field of view, wherein Δθ represents a change value obtained by comparing the field of view corresponding to the set position of the virtual image plane with a previous field of view corresponding to a previous position of the virtual image plane of a previous frame.

17. An image display device, comprising: The processor is configured as: receiving image data, the image data comprising first frame data corresponding to a first frame and second frame data corresponding to a second frame, the first frame being before the second frame; Determine a difference between a first depth value included in the first frame of data and a second depth value included in the second frame of data; as well as generating a corrected second frame of data according to the determined difference; A display element configured to display an image corresponding to the corrected second frame data; as well as a drive unit configured to adjust a position of the display element based on the determined difference, wherein the processor is further configured to determine a ratio for adjusting the size of the second frame data based on the determined difference, and to generate the corrected second frame data based on the determined ratio, Therein, the ratio is preset based on a time-travel curve with respect to a position change of the display element, the time-travel curve being extracted from a driving characteristic of the driving unit.

18. The image display device according to claim 17, further comprising a light transmission unit configured to transmit a first image corresponding to the first frame data and a second image corresponding to the second frame data to a position of an observer, in, The processor is further configured to determine the first depth value to correspond to a depth of a first virtual image perceived by an observer when the first image is transmitted, and to determine the second depth value to correspond to a depth of a second virtual image perceived by an observer when the second image is transmitted.

19. The image display device according to claim 17, wherein: The processor is further configured to adjust a position of a virtual camera viewing the second frame data based on the determined difference, and generate the corrected second frame data based on the adjusted position of the virtual camera.

20. The image display device according to claim 17, wherein: The processor is further configured to adjust a field of view of a virtual camera viewing the second frame data based on the determined difference, and generate the corrected second frame data based on the adjusted field of view of the virtual camera.

Images