Light field display device
By combining a light combiner and an eye-tracking sensor, the optical system of the light field display device is adjusted in real time, solving the problem of limited resolution and viewing angle at specific locations and achieving a better 3D image experience.
Patent Information
- Application Number
- CN202111024827.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-09-04
- Filing Date
- 2021-09-02
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-09-02
AI Technical Summary
Existing light field display devices have limited resolution and viewing angle for displaying 3D images at specific locations, resulting in a decrease in 3D image quality when the display extends beyond those locations.
By employing a technology that combines optical couplers and eye-tracking sensors, the position and spacing of multi-view virtual cameras are adjusted in real time through components such as optical waveguides, beam splitters, concave mirrors, and polarizers in the optical coupler, combined with the eye-tracking sensor, thereby optimizing the depth perception and resolution of 3D images.
It improves the depth and resolution of 3D images, expands the field of view, and reduces user visual fatigue and backsight.
Smart Images

Figure CN114137737B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a light field display device and its image processing method. Background Technology
[0002] Three-dimensional display devices that display different images to each eye can be either glasses-based or glasses-free. Glasses-based devices filter the desired image using methods such as polarization, time-division, and wavelength division to change the wavelength of the primary color. Glasses-free devices, on the other hand, utilize parallax barriers or lenticular lenses to allow each image to be viewed only in a specific space.
[0003] In particular, among naked-eye methods, there is the method of reproducing the light field. A light field refers to the distribution of light rays emitted from an object at various positions or in various directions. If such a light field is optically reproduced on any surface, a user located behind that surface can experience the same light distribution as when the actual object exists, and thus see a three-dimensional image of a natural object.
[0004] As mentioned above, the way of reproducing a light field can be described as the way of reproducing the light emitted from a point in a predetermined space in various directions by a display device. Summary of the Invention
[0005] However, the following problem may occur: the 3D image displayed by the light field display device is most clearly seen at a specific location due to the optical system designed for it, and the resolution of the 3D image will decrease when it is outside that location.
[0006] The technical problem to be solved by the present invention is to provide a light field display device that improves the depth perception (or three-dimensional depth) of three-dimensional images.
[0007] Furthermore, the technical problem to be solved by the present invention is to provide an image processing method for a light field display device that improves the resolution and viewing angle of three-dimensional images.
[0008] However, the purpose of this invention is not limited to the above-mentioned purpose, and various extensions can be realized without departing from the spirit and concept of this invention.
[0009] A light field display device according to an embodiment of the present invention for solving the above-mentioned technical problems includes: a display module; a light combiner that emits a third image formed by combining a first image emitted from the display module and a second image emitted from a different path than the first image, and is equipped with a first surface on which the first image is incident, a second surface arranged perpendicular to the first surface on which the second image is incident, and an emission surface facing the second surface and emitting the third image; and a control unit that generates image data based on three-dimensional modeling data and provides the image data to the display module.
[0010] The optical coupler includes: an optical waveguide equipped with a first surface, a second surface, an emission surface, and a third surface facing the first surface; a beam splitter disposed obliquely inside the optical waveguide relative to the emission surface; and a concave mirror disposed adjacent to the third surface of the optical waveguide.
[0011] The control unit moves the concave mirror in the vertical direction based on the third surface, according to the depth information included in the three-dimensional modeling data.
[0012] If the depth value included in the depth information is greater than the reference depth value, the control unit may move the concave mirror downward from the third surface according to the magnitude of the difference between the depth value and the reference depth value.
[0013] When the 3D modeling data has the reference depth value, the first image can be displayed on the second surface. The greater the depth value of the 3D modeling data is than the reference depth value, the more likely the first image is to be displayed in the region closer to the beam splitter.
[0014] The beam splitter can be a half mirror.
[0015] The light combiner may include multiple polarizing plates, so that light incident on the emission surface will not be emitted back to the emission surface via the beam splitter and the concave mirror.
[0016] The plurality of polarizing plates may include: a first polarizing plate disposed on the first surface; and a second polarizing plate disposed on the emission surface, wherein the polarization axis is perpendicular to the first polarizing plate.
[0017] The optical coupler may further include a 1 / 4 wavelength plate arranged between the third surface and the concave mirror.
[0018] The light combiner may further include a third polarizing plate disposed on the second surface, and the polarizing axis being parallel to the second polarizing plate.
[0019] The display module may include: a display panel including multiple pixels; and a microlens array disposed on the display panel, including multiple microlenses.
[0020] The spacing of the plurality of microlenses can be designed such that the viewing angle of the basic viewing angle range is approximately 6°.
[0021] It may also include: eye-tracking sensors that track the position of the user's eyes.
[0022] The control unit may include: a multi-view image generator that converts the 3D modeling data into a first set of multi-view images using a multi-view virtual camera; and a graphics processor that maps the first set of multi-view images onto the plurality of pixels.
[0023] The multi-view virtual camera can be arranged in at least two configurations, corresponding to the user's left and right eyes respectively.
[0024] The left and right sides of the basic view range may include multiple sub-view ranges, each of which has a view range equivalent to half of the basic view range.
[0025] The viewpoint of the extended viewpoint range, which includes the basic viewpoint range and the multiple sub-viewpoint ranges, can be 5 to 10 times the viewpoint of the basic viewpoint range.
[0026] The multi-view image generator can change the position of the multi-view virtual camera in response to changes in the position of the user's eyes detected by the eye-tracking sensor, so as to convert the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images.
[0027] The multi-view image generator can calculate the distance between the user and the eye-tracking sensor, and change the interval between the multi-view virtual cameras accordingly to convert the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images.
[0028] If the calculated distance between the user and the eye-tracking sensor increases, the multi-view image generator can increase the spacing between the multi-view virtual cameras; if the calculated distance between the user and the eye-tracking sensor decreases, the multi-view image generator can decrease the spacing between the multi-view virtual cameras.
[0029] An image processing method for a light field display device according to an embodiment of the present invention for solving the above-mentioned technical problems includes the following steps: receiving three-dimensional modeling data; converting the three-dimensional modeling data into a first group of multi-view images using a multi-view virtual camera; and mapping each of the first group of multi-view images to the plurality of pixels to output the image data, wherein the light field display device includes a display module containing a plurality of pixels and displaying an image based on the image data, a light combiner, and an eye-tracking sensor.
[0030] It also includes the following steps: detecting the position of the user's eyes using the eye-tracking sensor; and changing the position of the multi-view virtual camera in response to the detected changes in the position of the user's eyes, thereby converting the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images.
[0031] The display module may include: a display panel including the plurality of pixels; and a microlens array arranged on the display panel and including a plurality of microlenses, wherein the spacing between the plurality of microlenses is designed such that the viewing angle of the basic viewing angle range is approximately 6°.
[0032] It may also include the following steps: calculating the distance between the user and the eye-tracking sensor; and changing the interval between the multi-view virtual cameras corresponding to the calculated distance to convert the 3D modeling data into a third set of multi-view images different from the second set of multi-view images.
[0033] According to an embodiment of the present invention, the light field display device can change the position of the concavemirror included in the optical system, thereby improving the depth perception (or three-dimensional depth) of the three-dimensional image.
[0034] Furthermore, the image processing method of the light field display device according to an embodiment of the present invention can improve the resolution and viewing angle of a three-dimensional image by utilizing eye-tracking.
[0035] However, the effects of the present invention are not limited to those described above, and various extensions can be achieved without departing from the spirit and concept of the present invention. Attached Figure Description
[0036] Figure 1 This is a schematic perspective view of a light field display device according to an embodiment of the present invention.
[0037] Figure 2 yes Figure 1 The side view of the light field display device shown.
[0038] Figure 3This is a diagram illustrating a method for driving a concave mirror according to an embodiment of the present invention.
[0039] Figure 4a and Figure 4b This is a simplified exploded perspective view of a display module according to an embodiment of the present invention.
[0040] Figure 4c This is an example of using Figure 4a The display module shown is used to display a stereoscopic image.
[0041] Figure 5 This is a block diagram schematically showing the control section of a light field display device according to an embodiment of the present invention.
[0042] Figure 6 and Figure 7 This diagram illustrates how the arrangement of multiple virtual cameras is changed based on the position of the user's eyes detected by an eye-tracking sensor.
[0043] Figure 8 This is a diagram used to illustrate how the coordinates of a user's left and right eyes are calculated using an eye-tracking sensor.
[0044] Figure 9 It is a graph showing the distance between the user and the eye-tracking sensor, corresponding to the coordinate differences between the user's left and right eyes.
[0045] Figure 10 This is a schematic side view of a light field display device according to an embodiment of the present invention.
[0046] Figure 11 This is a flowchart illustrating an image processing method for a light field display device according to an embodiment of the present invention. Detailed Implementation
[0047] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same constituent elements in the drawings, and repeated descriptions of the same constituent elements are omitted.
[0048] Figure 1 This is a schematic perspective view of a light field display device according to an embodiment of the present invention.
[0049] Reference Figure 1 The light field display device 1000 may include a display module 100, a light combiner 200, an eye-tracking sensor 300, and a control unit (not shown).
[0050] According to one embodiment of the present invention, the light field display device 1000 can provide a user UE with a third image that combines a first image OJ (or three-dimensional object) generated by the display module 100 and the light combiner 200 with an external panoramic view (i.e., a real-time image (or a second image)) through the second surface S2 and the emission surface OS of the light combiner 200. In this case, the first image OJ (or three-dimensional object) can be a virtual reality image including additional information about the real-time image. The light field display device 1000 is a see-through type display device where the second surface S2 and the emission surface OS of the light combiner 200 are transparent, and can be an augmented reality display device.
[0051] However, the embodiments of the light field display device 1000 are not limited to this. For example, the second surface S2 of the light combiner 200 may also be non-transmissive and provide only the first image OJ generated by the display module 100 and the light combiner 200 to the user UE through the emission surface OS.
[0052] According to one embodiment of the present invention, the light field display device 1000 can detect the position of the user UE's eyes via an eye-tracking sensor 300. The display device 1000 can change accordingly to the detected eye position, as described below. Figures 5 to 9 The position and spacing of the multi-view virtual camera are adjusted to create more viewpoint images, thereby reducing inversion and enhancing the stereoscopic effect of the first image (OJ).
[0053] The eye-tracking sensor 300 can capture images of the user (UE) in real time and transmit the captured images to the control unit in real time. For example, the eye-tracking sensor 300 can utilize a web camera, a monocular camera, a stereo camera, a multi-camera system, or a camera capable of measuring depth information. The control unit can calculate the positions of the user's left and right eyes from the captured images.
[0054] Control Department (refer to) Figure 5 It may include multiple internal units. The internal units may be directly mounted on the motherboard, or mounted on a separate substrate and electrically connected to the motherboard via connectors or the like.
[0055] The control unit can control the overall operation of the light field display device 1000. For example, the control unit can turn the display module 100 on or off. The control unit can generate image data based on 3D modeling data and provide the generated image data to the display module 100. Here, the 3D modeling data may include the coordinate information and depth information of the 3D object included in the first image OJ. Furthermore, the control unit can control the operation of the actuator (not shown) included in the light combiner 200.
[0056] Figure 2 yes Figure 1 The side view of the light field display device shown. Figure 3 This is a diagram illustrating a method for driving a concave mirror according to an embodiment of the present invention.
[0057] Reference Figure 1 and Figure 2 The display module 100 modulates light according to the image data to form a first light L1 for the first image OJ. The first image OJ can be a two-dimensional image or a three-dimensional image, and the three-dimensional image can be a hologram image, a stereo image, a light field image, an integrated photography image, etc., and can include images in multi-view or super multi-view mode.
[0058] The display module 100 may include, for example, a liquid crystal on silicon (LCoS) element, a liquid crystal display (LCD) element, an organic light-emitting diode (OLED) element, a digital micromirror device (DMD), and may include next-generation display elements such as micro LEDs and quantum dot (QD) light-emitting diodes. The display module 100 may include a display panel 110 and a microlens array 120. The following will describe... Figure 4a , Figure 4b and Figure 4c The display module 100 is described in detail.
[0059] The light combiner 200 combines a first light L1 for a first image OJ and a second light L2 for a second image to emit a third light L3. It may include a first surface S1 into which the first image OJ is incident, a second surface S2 into which the second image is incident, and an emission surface OS into which the third image combining the first image OJ and the second image is emitted.
[0060] The optical coupler 200 may include: an optical waveguide 210 equipped with a first surface S1, a second surface S2, an emission surface OS, and a third surface S3 facing the first surface S1; a beam splitter 220 disposed within the optical waveguide 210 and inclined relative to the emission surface OS; and a concave mirror 230 disposed adjacent to the third surface S3.
[0061] Beam splitter 220 reflects a portion of the incident light and transmits a portion of the incident light; it can be a half mirror. However, it is not limited to this; a polarizing beam splitter that reflects a portion of the polarized light and transmits another portion of the polarized light can also be used.
[0062] The concave mirror 230 is equipped with a recessed reflective surface, so that the first light L1 transmitted through the beam splitter 220 is focused into the user UE's field of view after passing through the concave mirror 230 and the beam splitter 220.
[0063] Reference Figure 3 According to one embodiment, the control unit can move the concave mirror 230 in the vertical direction with the third surface S3 as a reference, in accordance with the depth information included in the three-dimensional modeling data.
[0064] For example, if the depth value included in the depth information is greater than the reference depth value, the control unit may move the concave mirror 230 downward relative to the reference position according to the magnitude of the difference between the depth value included in the depth information and the reference depth value.
[0065] At this time, if the depth value included in the three-dimensional modeling data is equal to the reference depth value, the concave mirror 230' can be arranged adjacent to the third surface S3, and the first image OJ' can be displayed on the second surface S2 of the optical waveguide 210.
[0066] The greater the depth value included in the 3D modeling data compared to the reference depth value, the further the concave mirror 230 can be positioned from the third surface S3. The greater the difference between the depth value included in the 3D modeling data and the reference depth value, the closer the first image OJ can be displayed in the region adjacent to the beam splitter 220.
[0067] Although not shown in the figure, an actuator can be provided at the lower end of the concave mirror 230, and the control unit can drive the actuator based on the depth information and reference depth value included in the 3D modeling data, thereby moving the concave mirror 230' arranged at the position corresponding to the reference depth value to the position corresponding to the difference between the depth value included in the depth information and the reference depth value.
[0068] As described above, the control unit can adjust the placement position of the concave mirror 230 in real time based on the depth values included in the 3D modeling data. For example, when the depth values included in the 3D modeling data are low (i.e., when displaying subtitles, etc.), the control unit can place the concave mirror 230' adjacent to the third surface S3, and when the depth values included in the 3D modeling data are high (i.e., when displaying a 3D image), the control unit can place the concave mirror 230 at a predetermined distance from the third surface S3. As described above, since the display depth of the first image OJ adaptively changes according to the situation, it is expected to reduce eye fatigue when the user UE views the image.
[0069] However, the control unit can also move the concave mirror 230 to a position preset by the user UE when the light field display device 1000 is turned on.
[0070] Figure 4a and Figure 4b This is a simplified exploded perspective view of a display module according to an embodiment of the present invention.
[0071] Reference Figure 4a The display module 100 may include a display panel 110 and a microlens array 120.
[0072] The display panel 110 may include a plurality of pixels PX. Each pixel may include, for example, red pixels, green pixels, and blue pixels. The pixels PX may be arranged in a matrix along a first direction D1 and a second direction D2 substantially perpendicular to the first direction D1. However, the arrangement of the pixels PX is merely exemplary and is not limited thereto. For example, the pixels PX may be a PenTile structure.
[0073] A microlens array 120 is disposed on the display panel 110. The microlens array 120 may include multiple microlenses ML. Light generated from the pixels PX of the display panel 110 can pass through the microlenses ML of the microlens array 120 to form a light field.
[0074] The microlens array 120 can be constructed using active lenses. The microlens array 120 can generate an electric field by applying a voltage to the electrodes constituting the microlens array 120, thereby altering the alignment of liquid crystal molecules. In two-dimensional display mode, the microlens array 120 allows direct transmission of the image displayed on the display panel 110; in three-dimensional display mode, the microlens array 120 can separate the viewing areas of the image on the display panel 110. For example, the microlens array 120 operating in three-dimensional display mode can utilize the diffraction and refraction of light to form images of multi-viewpoint images displayed on the display panel 110 in viewing areas corresponding to each viewpoint image.
[0075] The arrangement of pixels PX in the display panel 110 used to form the light field, the arrangement of microlenses ML constituting the microlens array 120, and the relative positional relationship between pixels PX and microlenses ML are not limited to... Figure 4a The embodiments shown can be implemented in various ways.
[0076] Image quality and field of view have an inverse relationship. That is, as the pixel pitch P of the microlens ML decreases, the image quality increases, but the field of view decreases; conversely, as the pixel pitch P of the microlens ML increases, the field of view increases, but the image quality decreases.
[0077] According to one embodiment of the present invention, in order to increase image resolution, the pixel pitch P of the microlens ML can be designed such that the viewing angle (or the viewing angle of the basic viewing angle range MV) is about 6°.
[0078] Additionally, refer to Figure 4b The display module 100' may include a display panel 110 and a lenticular lens array 120'.
[0079] A lenticular lens array 120' is arranged on the display panel 110. The lenticular lens array 120' may include multiple lenticular lenses RL. Light generated from the pixels PX of the display panel 110 can pass through the lenticular lenses RL of the lenticular lens array 120' to form a light field.
[0080] The lenticular lens array 120' may include a plurality of lenticular lenses RL arranged along one direction. Each lenticular lens RL may extend along one direction (e.g., a second direction D2). The color arrangement of pixel rows corresponding to and adjacent to each lenticular lens RL may be different from each other. That is, the basic color of the first pixel PX of the adjacent pixel row corresponding to each lenticular lens RL may be different from each other. For this purpose, the extending direction of each lenticular lens RL may be inclined at an acute angle to the second direction D2, which is the column direction, or it may be approximately parallel to the second direction D2.
[0081] The arrangement of pixels PX in the display panel 110 used to form the light field, the arrangement of lenticular lenses RL constituting the lenticular lens array 120', and the relative positional relationship between pixels PX and lenticular lenses RL are not limited to... Figure 4b The embodiments shown can be implemented in various ways.
[0082] Image quality and field of view have an inverse relationship. That is, as the pixel pitch P' of the cylindrical lens RL decreases, the image quality increases, but the field of view decreases; conversely, as the pixel pitch P' of the cylindrical lens RL increases, the field of view increases, but the image quality decreases.
[0083] According to one embodiment of the present invention, in order to increase image resolution, the pixel pitch P' of the cylindrical lens RL can be designed such that the viewing angle (or the viewing angle of the basic viewing angle range MV) is about 6°.
[0084] For ease of explanation, the following will be used as... Figure 4a The illustrated display module embodiment is used as a reference for explanation.
[0085] Figure 4c This is an example of using Figure 4a The display module shown is used to display a stereoscopic image.
[0086] Reference Figure 4c A "light field" is a concept that describes the spatial distribution of light rays. Using this concept, light reflected from or produced by an object can be defined as light that travels in a straight line through space and enters the human eye, and three-dimensional space can be composed of countless light fields. To represent each light field mathematically, for example, a five-dimensional plenooptic function I can be used. That is, the light field can be defined by the three-dimensional spatial coordinates (x, y, z) of the point on a specific plane through which the light ray passes and the spatial direction angle of the light ray. The brightness is represented by the light field. The light field can be captured by digitizing the total optical function value of light passing through the specific plane described above. That is, the light field can be represented by angles relative to the coordinates (x, y, z) of a predetermined region. The brightness value is obtained. For example, image data is provided to display panel 110, and the light ray generated by display panel 110 forms a light field in a specific direction (viewpoint) through microlens array 120, and user UE can view objects OJ1 and OJ2 in a three-dimensional image.
[0087] exist Figure 4a and Figure 4c For ease of explanation, the illustration shows a scenario where the user (UE) directly views the 3D image formed by the display module 100. However, if... Figure 1 and Figure 2 As shown, the optical combiner 200 can be used to provide the user UE with an image that combines a 3D image and a real-time image.
[0088] Figure 5 This is a block diagram schematically showing the control section of a light field display device according to an embodiment of the present invention. Figure 6 and Figure 7 This diagram illustrates how the arrangement of multiple virtual cameras is changed based on the position of the user's eyes detected by an eye-tracking sensor.
[0089] According to one embodiment of the present invention, the control unit 400 may include a multi-view image generator 410 and a graphics processor 420.
[0090] The multi-view image generator 410 can convert 3D modeling data into a first multi-view image. For example, the multi-view image generator 410 can be implemented using a multi-view virtual camera. In this case, the 3D modeling data can be provided from a memory (not shown) included in the light field display device 1000. However, it is not limited to this; the 3D image data can also be provided from an external device. According to one embodiment, the multi-view image generator 410 can be an application processor (AP), a central processing unit (CPU), or a microprocessor.
[0091] A multi-view virtual camera can be implemented as an application. A multi-view virtual camera may include multiple virtual cameras, and a multi-view image generator 410 can convert 3D image data into a first multi-view image based on settings such as the spacing and angles between the multiple virtual cameras. At least two multi-view virtual cameras may be arranged to correspond to the left and right eyes of the user UE, respectively.
[0092] The graphics processor 420 can map multi-view images to... Figure 4a The image data of a frame is output by representing multiple pixels (PX). The graphics processor 420 can perform a series of operations as follows: by using a multi-view image mapping algorithm, the image with multi-view information is rearranged in pixels according to the light field viewpoint arrangement rules.
[0093] The graphics processor 420 can provide image data to Figure 4a The driving unit (not shown) of the display panel 110. The display panel 110 (or display module 100) can display three-dimensional images based on received image data.
[0094] Reference Figure 5 and Figure 6 The multi-view image generator 410 can generate first to sixteenth multi-view images by capturing three-dimensional objects included in the first image OJ using first to sixteenth virtual cameras. In this case, the first to eighth multi-view images can be left-view images corresponding to the left eye of the user UE, and the ninth to sixteenth multi-view images can be right-view images corresponding to the right eye of the user UE. However, the number of virtual cameras is not limited to this and can vary according to the number of viewpoints and viewpoint intervals required to display the first image OJ.
[0095] The graphics processor 420 can generate image data such that a first multi-view image to a sixteenth multi-view image are displayed within a basic viewing angle interval MV, and can also generate image data such that the first multi-view images to the sixteenth multi-view images are sequentially and repeatedly displayed in a plurality of sub-view angle intervals SV1 to SV8 arranged to the left and right based on the basic viewing angle interval MV. In this case, the basic viewing angle interval MV can be approximately 6°, and the sub-view angle intervals SV1 to SV8 can be approximately 3°, which is equivalent to half the basic viewing angle interval MV. According to an embodiment of the present invention, the viewing angle of the extended viewing angle interval including the basic viewing angle interval MV and the plurality of sub-view angle intervals SV1 to SV8 can be 5 to 10 times the viewing angle of the basic viewing angle interval MV. That is, the number of the plurality of viewing angle intervals can be 18.
[0096] For example, the graphics processor 420 can generate image data such that a first multi-view image to an eighth multi-view image (or a left view image) is displayed in the first sub-view interval SV1, the third sub-view interval SV3, the fifth sub-view interval SV5, and the seventh sub-view interval SV7, and generate image data such that a ninth multi-view image to a sixteenth multi-view image (or a right view image) is displayed in the second sub-view interval SV2, the fourth sub-view interval SV4, the sixth sub-view interval SV6, and the eighth sub-view interval SV8.
[0097] However, assuming the distance between the user's left and right eyes is 62mm, the viewing angle for viewing 3D images without backsight is approximately 12°. Therefore, as... Figure 6 As shown, when the user UE's left eye position LL1 corresponds to the first sub-viewing angle interval SV1 and the user UE's right eye position RR1 corresponds to the fourth sub-viewing angle interval SV4, the left viewpoint image (or the first multi-viewpoint image to the eighth multi-viewpoint image) is displayed in the first sub-viewing angle interval SV1, and the right viewpoint image (or the ninth multi-viewpoint image to the sixteenth multi-viewpoint image) is displayed in the fourth sub-viewing angle interval SV4. Therefore, the user UE can view a high-resolution three-dimensional image.
[0098] Conversely, if the user UE's left eye position LL2 corresponds to the fourth sub-viewing angle interval SV4, and the user UE's right eye position RR2 corresponds to the fifth sub-viewing angle interval SV5, then the right viewpoint image (or the ninth to sixteenth multi-viewpoint images) is displayed in the fourth sub-viewing angle interval SV4, and the left viewpoint image (or the first to eighth multi-viewpoint images) is displayed in the fifth sub-viewing angle interval SV5. Therefore, the user UE may experience dizziness due to the reverse viewing phenomenon. That is, as... Figure 6In the embodiment shown, when a multi-viewpoint image is fixedly displayed in the basic viewing angle range MV and the first sub-viewpoint range SV1 to the eighth sub-viewpoint range SV8, the user UE can view a high-resolution image, but there is a problem of limited viewing angle.
[0099] The present invention aims to maintain a high level of image resolution, and by utilizing... Figure 1 The eye-tracking sensor 300 shown is used to improve the user's (UE) field of view.
[0100] According to one embodiment of the present invention, the eye-tracking sensor 300 can capture images of a user UE in real time to generate captured images. The multi-view image generator 410 can receive position information of the user UE's left and right eyes from the captured images. The multi-view image generator 410 can adjust the position of the multi-view virtual camera according to the position changes of the user UE's left and right eyes detected by the eye-tracking sensor 300, thereby converting the 3D modeling data into a second set of multi-view images different from the first set of multi-view images.
[0101] Reference Figure 7 The user UE's left eye position LL3 corresponds to the second sub-viewing angle interval SV2, and the user UE's right eye position RR3 corresponds to the basic viewing angle interval MV. However, compared with... Figure 6 Unlike the illustrated embodiment, the multi-view images displayed in each of the basic viewing angle interval MV and the first sub-view angle intervals SV1 to the eighth sub-view angle intervals SV8 are not fixed but variable. That is, the multi-view image generator 410 can change the position of the multi-view virtual camera in response to changes in the position LL3 of the user UE's left eye and the position RR3 of the right eye detected by the eye tracking sensor 300, thereby converting the existing multi-view images into new multi-view images.
[0102] For example, if the eye-tracking sensor 300 detects that the user UE's left eye position LL3 corresponds to the second sub-viewing angle interval SV2, the multi-view image generator 410 can arrange the first to eighth multi-view virtual cameras in the second sub-viewing angle interval SV2, and generate the first group of multi-view images (i.e., the ninth to sixteenth multi-view images, refer to...) Figure 6 ) is converted into the second group of multiview images (i.e., the first to eighth multiview images, see reference). Figure 7 ).
[0103] Subsequently, the graphics processor 420 can generate image data in a manner that the first multi-view image to the sixteenth multi-view image are displayed sequentially and repeatedly based on the second sub-view interval SV2. For example, image data can be generated in a manner that the ninth to sixteenth multi-view images (or right view images) are displayed in the first sub-view interval SV1, the third sub-view interval SV3, the fifth sub-view interval SV5, and the seventh sub-view interval SV7, and image data can be generated in a manner that the first to eighth multi-view images (or left view images) are displayed in the second sub-view interval SV2, the fourth sub-view interval SV4, the sixth sub-view interval SV6, and the eighth sub-view interval SV8.
[0104] Figure 8 This is a diagram used to illustrate how the coordinates of a user's left and right eyes are calculated using an eye-tracking sensor. Figure 9 It is a graph showing the distance between the user and the eye-tracking sensor, corresponding to the coordinate differences between the user's left and right eyes.
[0105] Reference Figure 8 and Figure 9 According to one embodiment of the present invention, the multi-view image generator 410 can calculate the distance Z between the user UE and the eye-tracking sensor 300, and the interval between the multi-view virtual cameras varies corresponding to the calculated distance Z, thereby converting the 3D modeling data into a second multi-view image different from the first multi-view image. In this case, the distance Z between the user UE and the eye-tracking sensor 300 can be defined as the shortest distance from the eye-tracking sensor 300 to the virtual straight line LV connecting the left and right eye positions of the user UE.
[0106] The eye-tracking sensor 300 can generate captured images in real time by capturing images of the user (UE). The multi-view image generator 410 can receive the coordinate difference between the user's two eyes from the captured images. At this time, the coordinate difference between the two eyes can be expressed in pixels (PX).
[0107] Figure 9 The graph shown is the Z-axis calibration function obtained by experimentally determining the distance Z between the user UE and the eye-tracking sensor 300 based on the coordinate difference between the user UE's eyes, with a distance of 62mm between the user's eyes. The Z-axis calibration function can be expressed by the following mathematical formula.
[0108] [Mathematical expression]
[0109] y = 0.0416x 2 -13.478x+1299.6 (Here, x is the difference in eye coordinates between the user UE and y is the distance Z between the user UE and the eye tracking sensor 300).
[0110] For example, in an image captured by the eye-tracking sensor 300, with a coordinate difference of 50 pixels between the user UE's two eyes, the distance Z between the user UE and the eye-tracking sensor 300 is approximately 800 mm according to the Z-axis calibration function.
[0111] Furthermore, although the Z-axis calibration function was calculated exemplarily based on a distance of 62mm between the user UE's eyes, it is not limited to this and can be calculated based on the distance between the user UE's eyes. The multi-view image generator 410 can obtain the distance between the user UE's eyes directly through the eye-tracking sensor 300 when initially driving the light field display device 1000, or obtain the distance between the user UE's eyes through input from the user UE.
[0112] As described above, the multi-view image generator 410 can use the eye-tracking sensor 300 and the calibration function to calculate the distance Z between the user UE and the eye-tracking sensor 300. Then, if the distance Z between the user UE and the eye-tracking sensor 300 increases, the interval between the multi-view virtual cameras is increased; conversely, if the distance Z between the user UE and the eye-tracking sensor 300 decreases, the interval between the multi-view virtual cameras is decreased.
[0113] An increase in the distance Z between the user UE and the eye-tracking sensor 300 indicates an increase in the distance between the 3D object included in the 3D modeling data and the user UE; conversely, a decrease in the distance Z indicates a decrease in the distance between the 3D object included in the 3D modeling data and the user UE. Therefore, when using the same number of virtual cameras to photograph 3D objects, as the distance between the 3D object and the user UE increases, the area that each virtual camera needs to capture increases; conversely, as the distance between the 3D object and the user UE decreases, the area that each virtual camera needs to capture decreases, thus allowing for a more stereoscopic display of the 3D object.
[0114] The following describes another embodiment. In the following embodiments, descriptions of configurations identical to those in the embodiments described above will be omitted or simplified, and the descriptions will focus primarily on the differences.
[0115] Figure 10 This is a schematic side view of a light field display device according to an embodiment of the present invention.
[0116] Reference Figure 10 , Figure 10 The light field display device 1000_1 shown is... Figure 2 The difference between the light field display device 1000 and the light field display device 1000 is that it is also equipped with an optical component to prevent external light incident on the emission surface OS from being emitted back to the emission surface OS via the beam splitter 220 and the concave mirror 230. The other components are substantially the same.
[0117] Specifically, the light field display device 1000_1 may include a display module 100 and a light combiner 200_1, wherein the light combiner 200_1 combines a first light L1 for a first image formed on the display module 100 and a second light L2 for a second image from a different path than the first image.
[0118] According to one embodiment, the optical combiner 200_1 may include a first polarizing plate 241 disposed on a first surface S1, a second polarizing plate 242 disposed on an emission surface OS, and a quarter-wavelength plate 244 disposed between a third surface S3 and a concave mirror 230.
[0119] A first polarizing plate 241 can be arranged on the first surface S1 of the optical waveguide 210, and a second polarizing plate 242 can be arranged on the exit surface OS of the optical waveguide 210. The first polarizing plate 241 and the second polarizing plate 242 can have polarization axes that are perpendicular to each other. For example, the first polarizing plate 241 has a function to polarize the first light... The second polarizing plate 242 has a polarizing axis that allows the second polarized light (⊙) to be transmitted. Furthermore, a quarter-wavelength plate 244 can be arranged between the third surface S3 of the optical waveguide 210 and the concave mirror 230.
[0120] The first light L1, directed at the first image, incident on the first surface S1 through the first polarizing plate 241, the second polarizing plate 242, and the quarter-wave plate 244, can reach the user UE's field of view. That is, the first light L1 passes through the first polarizing plate 241 and becomes the first polarized light. Light, first polarized light A portion of the light is transmitted through the beam splitter 220, which acts as a semi-reflective mirror. It then passes through the quarter-wave plate 244 and becomes circularly polarized light. After being reflected by the concave mirror 230, it becomes circularly polarized light in the opposite direction. Next, after passing through the quarter-wave plate 244 again, it becomes second-polarized light (⊙), which is reflected by the beam splitter 220 and incident on the second polarizing plate 242. Since the second polarizing plate 242 only transmits second-polarized light (⊙), the incident second-polarized light (⊙) can be transmitted and delivered to the user's (UE) field of view.
[0121] Furthermore, external noise light Ln incident on the emission surface OS via the second polarizer 242 and the quarter-wave plate 244 may not reach the user UE's field of view. Specifically, the noise light Ln, after passing through the second polarizer 242 and becoming second-polarized (⊙), is reflected by the beam splitter 220, then transmitted through the quarter-wave plate 244 and becoming circularly polarized in the opposite direction. After being reflected again by the concave mirror 230, it becomes circularly polarized. Then, after passing through the quarter-wave plate 244 again, it becomes first-polarized light. The light is reflected by beam splitter 220 and incident on second polarizer 242. Since second polarizer 242 only transmits the second polarized light (⊙), the incident first polarized light... The noise light Ln is absorbed.
[0122] According to one embodiment, the optical combiner 200_1 may further include a third polarizing plate 243 disposed on the second surface S2. The third polarizing plate 243 may have the same polarization axis as the second polarizing plate 242. That is, it may have a polarization axis that allows the second polarized light (⊙) to be transmitted. Therefore, the second light L2 incident on the second surface S2 of the optical waveguide 210 for the second image can reach the user UE's field of view. That is, the second light L2 passes through the third polarizing plate 243 and becomes second polarized light (⊙), and then is incident on the second polarizing plate 242. Since the second polarizing plate 242 only allows the second polarized light (⊙) to be transmitted, the incident second polarized light (⊙) can be transmitted and delivered to the user UE's field of view.
[0123] Figure 11 This is a flowchart illustrating an image processing method for a light field display device according to an embodiment of the present invention.
[0124] Reference Figures 1 to 11 The image processing method of the light field display device 1000 can receive three-dimensional modeling data through the multi-viewpoint image generator 410 (S10). The light field display device 1000 includes: a display module 100, including multiple pixels PX, which displays a three-dimensional image based on the image data; and an eye tracking sensor 300, which tracks the position of the user UE's left and right eyes.
[0125] At this point, the three-dimensional image data may include the coordinate and depth information of the three-dimensional object included in the first image OJ.
[0126] Three-dimensional image data can be provided from a memory (not shown) included in the light field display device 1000. However, it is not limited to this; three-dimensional image data can also be provided from an external device.
[0127] Next, the 3D modeling data can be converted into the first set of multi-view images using a multi-view virtual camera (S20).
[0128] The multi-view virtual camera can be implemented as an application. The multi-view virtual camera may include multiple virtual cameras, and the multi-view image generator 410 can convert 3D image data into a first set of multi-view images based on settings such as the intervals and angles between the multiple virtual cameras. At least two multi-view virtual cameras may be assigned to the left and right eyes of the user UE, respectively.
[0129] Next, the image data can be output by mapping each of the multi-view images in the first group to multiple pixels PX through the graphics processor 420 (S30).
[0130] The graphics processor 420 can perform a series of operations as follows: by using a multi-viewpoint image mapping algorithm, it rearranges images with multi-viewpoint information in pixels according to the light field viewpoint arrangement rules.
[0131] The graphics processor 420 can provide image data to Figure 4a The driving unit (not shown) of the display panel 110. The display panel 110 (or display module 100) can display three-dimensional images based on the received image data.
[0132] Next, the position of the user UE's eyes can be detected by the eye-tracking sensor 300 (S40).
[0133] The eye-tracking sensor 300 can capture images of the user (UE) in real time and generate captured images. The eye-tracking sensor 300 can provide the captured images to the multi-view image generator 410 of the control unit 400. For example, the eye-tracking sensor 300 can utilize a web camera, a monocular camera, a stereo camera, a multi-camera system, or a camera capable of measuring depth information. The control unit 400 can calculate the positions of the user's left and right eyes from the captured images.
[0134] Next, the position of the multi-view virtual camera can be changed according to the detected changes in the position of the user's eyes, thereby converting the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images (S50).
[0135] The multi-view image generator 410 can receive position information of the user UE's left and right eyes from the captured images. The multi-view image generator 410 can change the position of the multi-view virtual camera according to the position changes of the user UE's left and right eyes detected by the eye tracking sensor 300, thereby converting the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images.
[0136] Reference Figure 7 The user UE's left eye position LL3 corresponds to the second sub-viewing angle interval SV2, and the user UE's right eye position RR3 corresponds to the basic viewing angle interval MV. However, compared with... Figure 6Unlike the illustrated embodiment, the multi-view images displayed in each of the basic viewpoint interval MV and the first sub-viewpoint intervals SV1 to the eighth sub-viewpoint intervals SV8 are not fixed but variable. That is, the multi-view image generator 410 can change the position of the multi-view virtual camera in response to changes in the position LL3 of the user UE's left eye and the position RR3 of the right eye detected by the eye-tracking sensor 300, thereby converting existing multi-view images of the 3D modeling data into new multi-view images.
[0137] For example, if the eye-tracking sensor 300 detects that the position LL3 of the user UE's left eye corresponds to the second sub-viewing angle interval SV2, the multi-view image generator 410 can arrange the first to eighth multi-view virtual cameras in the second sub-viewing angle interval SV2, and generate the first group of multi-view images (i.e., the ninth to sixteenth multi-view images, refer to...) Figure 6 ) is converted into the second group of multiview images (i.e., the first to eighth multiview images, see reference). Figure 7 ).
[0138] Subsequently, the graphics processor 420 can generate image data in a manner that the first multi-view image to the sixteenth multi-view image are displayed sequentially and repeatedly based on the second sub-view interval SV2. For example, image data can be generated in a manner that the ninth to sixteenth multi-view images (or right view images) are displayed in the first sub-view interval SV1, the third sub-view interval SV3, the fifth sub-view interval SV5, and the seventh sub-view interval SV7, and image data can be generated in a manner that the first to eighth multi-view images (or left view images) are displayed in the second sub-view interval SV2, the fourth sub-view interval SV4, the sixth sub-view interval SV6, and the eighth sub-view interval SV8.
[0139] According to one embodiment, the distance between the user UE and the eye-tracking sensor 300 can be calculated, and the interval between the multi-view virtual cameras corresponding to the calculated distance can be converted into a third set of multi-view images that are different from the second set of multi-view images.
[0140] Although the above description has been made with reference to embodiments of the present invention, those skilled in the art should understand that various modifications and alterations can be made to the present invention without departing from the spirit and concept of the invention as set forth in the claims.
Claims
1. A light field display device, comprising: Display module; The light combiner emits a third image formed by combining a first image emitted from the display module and a second image emitted from a different path than the first image, and is equipped with a first surface into which the first image is incident, a second surface arranged perpendicular to the first surface into which the second image is incident, and an emission surface facing the second surface into which the third image is emitted. The control unit generates image data based on 3D modeling data and provides the image data to the display module; as well as Eye-tracking sensors track the position of the user's eyes. The display module includes: a display panel comprising multiple pixels. The optical coupler includes: An optical waveguide is provided with a first surface, a second surface, an emission surface, and a third surface facing the first surface; A beam splitter is arranged obliquely inside the optical waveguide relative to the emission surface; and A concave mirror is arranged adjacent to the third surface of the optical waveguide. The control unit moves the concave mirror vertically along the third surface, based on the depth information included in the 3D modeling data. The control unit includes: A multi-view image generator converts the 3D modeling data into a first set of multi-view images using a multi-view virtual camera; and The graphics processor maps the multi-view images of the first group onto the plurality of pixels. The multi-view image generator changes the position of the multi-view virtual camera in response to changes in the position of the user's eyes detected by the eye-tracking sensor, so as to convert the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images.
2. The light field display device according to claim 1, wherein, If the depth value included in the depth information is greater than the reference depth value, the control unit moves the concave mirror downward from the third surface according to the magnitude of the difference between the depth value and the reference depth value.
3. The light field display device according to claim 2, wherein, When the 3D modeling data has the reference depth value, the first image is displayed on the second surface. The greater the depth value of the 3D modeling data is than the reference depth value, the more likely the first image is to be displayed in the region that is closer to the beam splitter.
4. The light field display device according to claim 1, wherein, The beam splitter is a semi-reflective mirror.
5. The light field display device according to claim 1, wherein, The optical coupler includes: Multiple polarizing plates ensure that light incident on the emission surface will not be re-emitted to the emission surface via the beam splitter and the concave mirror. The plurality of polarizing plates include: A first polarizing plate is disposed on the first surface; and A second polarizing plate is arranged on the emission surface, and the polarization axis is perpendicular to the first polarizing plate.
6. The light field display device according to claim 5, wherein, The optical coupler also includes: A quarter-wavelength plate is arranged between the third surface and the concave mirror.
7. The light field display device according to claim 5, wherein, The optical coupler also includes: A third polarizing plate is disposed on the second surface, and the polarizing axis is parallel to the second polarizing plate.
8. The light field display device according to claim 1, wherein, The display module further includes: A microlens array is arranged on the display panel and includes multiple microlenses.
9. The light field display device according to claim 8, characterized in that, The spacing of the multiple microlenses is designed such that the basic viewing angle range is 6°.
10. The light field display device according to claim 1, characterized in that, At least two multi-view virtual cameras are arranged to correspond to the user's left and right eyes, respectively.
11. The light field display device according to claim 9, wherein, The basic viewing angle range includes multiple sub-viewing angle ranges on its left and right sides. Each of the sub-view intervals has a view equivalent to half of the basic view interval.
12. The light field display device according to claim 11, characterized in that, The viewpoint of the extended viewpoint range, which includes the basic viewpoint range and the multiple sub-viewpoint ranges, is 5 to 10 times the viewpoint of the basic viewpoint range.
13. A light field display device, comprising: Display module; The light combiner emits a third image formed by combining a first image emitted from the display module and a second image emitted from a different path than the first image, and is equipped with a first surface into which the first image is incident, a second surface arranged perpendicular to the first surface into which the second image is incident, and an emission surface facing the second surface into which the third image is emitted. The control unit generates image data based on 3D modeling data and provides the image data to the display module; as well as Eye-tracking sensors track the position of the user's eyes. The display module includes: a display panel comprising multiple pixels. The optical coupler includes: An optical waveguide is provided with a first surface, a second surface, an emission surface, and a third surface facing the first surface; A beam splitter is arranged obliquely inside the optical waveguide relative to the emission surface; and A concave mirror is arranged adjacent to the third surface of the optical waveguide. The control unit moves the concave mirror vertically along the third surface, based on the depth information included in the 3D modeling data. The control unit includes: A multi-view image generator converts the 3D modeling data into a first set of multi-view images using a multi-view virtual camera; and The graphics processor maps the multi-view images of the first group onto the plurality of pixels. The multi-view image generator calculates the distance between the user and the eye-tracking sensor, and changes the spacing between the multi-view virtual cameras accordingly to convert the 3D modeling data into a second set of multi-view images that are different from the first set of multi-view images.
14. The light field display device according to claim 13, characterized in that, If the calculated distance between the user and the eye-tracking sensor increases, the multi-view image generator increases the spacing between the multi-view virtual cameras. If the calculated distance between the user and the eye-tracking sensor decreases, the multi-view image generator reduces the spacing between the multi-view virtual cameras.
Citation Information
Patent Citations
See-through type display apparatus
CN110941088A
