Display device for providing plurality of optical views and control method therefor

Abstract

A display device is disclosed. This display device comprises: a memory storing a projection matrix; a display panel including a plurality of sub-pixels; a lenticular lens disposed on the front surface of the display panel and configured to provide a plurality of optical views while dividing a viewing zone; and one or more processors connected to the display panel and the lenticular lens and configured to control the display device, wherein the processors obtain a plurality of images respectively corresponding to the plurality of optical views, render a final image from the plurality of images on the basis of the projection matrix, and display the final image through the display panel, and the projection matrix may be a matrix obtained through ray tracing on the basis of the arrangement pattern of the plurality of sub-pixels and the shape of the lenticular lens.

Description

Display device for providing multiple optical views and control method thereof

[0001] The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device providing multiple optical views and a control method thereof.

[0002] Advances in electronic technology have led to the development of electronic devices offering a variety of functions. In particular, various types of display devices have recently been developed, enhancing user convenience.

[0003] For example, multi-view display devices are currently being developed. These multi-view display devices can spatially divide the viewing area to provide multiple optical views, allowing viewers to view different images depending on their viewing position.

[0004] These multi-view display devices include lenticular lenses, but aberrations of the lenticular lenses prevent them from evenly splitting the optical view.

[0005] According to one embodiment of the present disclosure for achieving the above object, a display device includes a memory storing a projection matrix, a display panel including a plurality of sub-pixels, a lenticular lens arranged on a front surface of the display panel to separate a field of view and provide a plurality of optical views, and at least one processor connected to the memory, the display panel, and the lenticular lens to control the display device, wherein the processor obtains a plurality of images each corresponding to the plurality of optical views, renders a final image from the plurality of images based on the projection matrix, and displays the final image through the display panel, and the projection matrix may be a matrix obtained through ray tracing based on an arrangement pattern of the plurality of sub-pixels and a shape of the lenticular lens.

[0006] Additionally, the shape of the lenticular lens may include at least one of a thickness, curvature, pitch, focal length, or slanted angle of the lenticular lens.

[0007] And, the processor can render the final image by multiplying the plurality of images and the projection matrix.

[0008] Additionally, the processor can convert the plurality of images into a vector form and render the final image by multiplying the plurality of images converted into the vector form and the projection matrix.

[0009] And, the processor can identify a refractive index at each point of the lenticular lens based on the shape of the lenticular lens, perform ray tracing based on the arrangement pattern of the plurality of sub-pixels and the identified refractive index to obtain the projection matrix, and store the projection matrix in the memory.

[0010] Additionally, the processor can perform the ray tracing by identifying the position when each of the plurality of light rays starting from the plurality of sub-pixels reaches a viewing distance by being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens.

[0011] And, the processor can perform the ray tracing by randomly identifying some sub-pixels from the plurality of sub-pixels, and identifying a position when each of the plurality of light rays starting from the some sub-pixels reaches the viewing distance by being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

[0012] In addition, the processor can view-map the plurality of images using a preset view mapping method to obtain a view-mapped image, and based on the view-mapped image, identify a position when each of the plurality of light rays originating from the plurality of sub-pixels reaches a viewing distance by being refracted by the refractive index of the corresponding point at a corresponding point of the lenticular lens, thereby performing the ray tracing.

[0013] And, the processor can perform the ray tracing by identifying the position when each of the plurality of light rays departing from the plurality of optical views reaches the display panel after being refracted by the refractive index of the point at the corresponding point of the lenticular lens.

[0014] Additionally, the processor can perform the ray tracing by identifying a position when each of the plurality of light rays departing from the plurality of optical views based on the plurality of images reaches the display panel after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

[0015] Meanwhile, according to one embodiment of the present disclosure, a method for controlling a display device may include a step of obtaining a plurality of images each corresponding to a plurality of optical views, a step of rendering a final image from the plurality of images using a projection matrix obtained through ray tracing based on an arrangement pattern of a plurality of sub-pixels included in a display panel and a shape of a lenticular lens disposed on a front side of the display panel to separate a field of view and provide the plurality of optical views, and a step of displaying the final image through the display panel.

[0016] Additionally, the shape of the lenticular lens may include at least one of a thickness, curvature, pitch, focal length, or slanted angle of the lenticular lens.

[0017] And, the rendering step can render the final image by multiplying the plurality of images and the projection matrix.

[0018] In addition, the rendering step may convert the plurality of images into a vector form, and render the final image by multiplying the plurality of images converted into a vector form and the projection matrix.

[0019] And, the method may further include a step of identifying a refractive index at each point of the lenticular lens based on a shape of the lenticular lens, a step of performing ray tracing based on an arrangement pattern of the plurality of sub-pixels and the identified refractive index to obtain the projection matrix, and a step of storing the projection matrix in a memory of the display device.

[0020] In addition, the step of obtaining the projection matrix can perform the ray tracing by identifying the position when each of the plurality of light rays starting from the plurality of sub-pixels reaches the viewing distance by being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens.

[0021] And, the step of obtaining the projection matrix may perform the ray tracing by randomly identifying some sub-pixels from the plurality of sub-pixels, and identifying a position when each of the plurality of light rays starting from the some sub-pixels reaches the viewing distance by being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

[0022] In addition, the step of obtaining the projection matrix may include view-mapping the plurality of images using a preset view-mapping method to obtain a view-mapped image, and, based on the view-mapped image, identifying a position when each of the plurality of light rays departing from the plurality of sub-pixels reaches a viewing distance by being refracted by a refractive index of the corresponding point at a corresponding point of the lenticular lens, thereby performing the ray tracing.

[0023] And, the step of obtaining the projection matrix can perform the ray tracing by identifying the position when each of the plurality of rays departing from the plurality of optical views reaches the display panel after being refracted by the refractive index of the point at the corresponding point of the lenticular lens.

[0024] In addition, the step of obtaining the projection matrix may perform the ray tracing by identifying a position when each of the plurality of rays departing from the plurality of optical views based on the plurality of images reaches the display panel after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

[0025] FIGS. 1A to 1E are drawings for explaining a re-point display device to help understanding the present disclosure.

[0026] FIG. 2 is a block diagram showing the configuration of a display device according to one embodiment of the present disclosure.

[0027] FIG. 3 is a block diagram showing a detailed configuration of a display device according to an embodiment of the present disclosure.

[0028] FIGS. 4 to 7 are drawings for explaining ray tracing based on an optical simulator according to an embodiment of the present disclosure.

[0029] FIG. 8 is a flowchart illustrating rendering using ray tracing according to one embodiment of the present disclosure.

[0030] FIGS. 9 to 11 are drawings illustrating the advantages of ray tracing according to one embodiment of the present disclosure.

[0031] FIG. 12 is a diagram for explaining the rendering operation of a final image according to one embodiment of the present disclosure.

[0032] FIG. 13 is a diagram for explaining a projection matrix according to an embodiment of the present disclosure.

[0033] FIG. 14 is a diagram for explaining ray tracing through random sampling according to one embodiment of the present disclosure.

[0034] FIG. 15 is a drawing for explaining image quality improvement according to one embodiment of the present disclosure.

[0035] FIG. 16 is a flowchart for explaining a method for controlling a display device according to an embodiment of the present disclosure.

[0036] The purpose of the present disclosure is to provide a display device and a control method thereof for providing an evenly divided optical view.

[0037] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0038] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0039] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0040] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0041] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0042] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0043] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0044] Various embodiments of the present disclosure will be described in more detail with reference to the attached drawings below.

[0045] FIG. 1 is a block diagram showing the configuration of a display device (100) according to one embodiment of the present disclosure.

[0046] The display device (100) can automatically change the screen settings. For example, the display device (100) is implemented as a modular display and can change the screen settings based on other display devices connected to the display device (100). However, the present invention is not limited thereto, and the display device (100) may be any device that has a display, can be connected to other display devices, and can change the screen settings. For example, the display device (100) may be a device that has a display, is connected to other display devices, and changes the screen settings based on other display devices, such as a TV, a desktop PC, a laptop, a smartphone, or a tablet PC.

[0047] FIGS. 1A to 1E are drawings for explaining a re-point display device to help understanding the present disclosure.

[0048] Figure 1a is a diagram illustrating a rendering method when providing seven optical views. First, an optical view may be optical information provided to a user at a viewing position, and multiple optical views may be optical information provided to a user at multiple viewing positions. Multiple optical views may also be referred to as multi-views. A multi-view display device may provide multiple optical views and provide different optical information for each of the multiple viewing positions.

[0049] A multi-view display device includes a lenticular lens, and pixels viewable at each of a plurality of viewing positions may be different due to the lenticular lens. Here, the lenticular lens includes a plurality of lens regions, and an image displayed on the multi-view display device may be refracted through the plurality of lens regions. Each lens region is formed to have a size corresponding to at least one pixel, so that light passing through each pixel may be differently dispersed for each of the plurality of viewing positions.

[0050] The multi-view display device can display seven images by dividing them to provide seven optical views, as illustrated in FIG. 1A, and can render the seven images for this purpose. For example, the first pixel in the upper left corner can display the color of a preset position in the first image among the seven images, and the second pixel to the right of the first pixel can display the color of a preset position in the second image among the seven images. In this way, seven consecutive pixels can each display the color of a preset position in the seven images, and for this operation, the seven images can be rendered as a single image.

[0051] However, in order to view only the precisely corresponding pixels in each of the multiple optical views, the lenticular lens must be manufactured very precisely, which is practically difficult. Accordingly, each of the multiple optical views is influenced by adjacent pixels. For example, as shown in Fig. 1b, view v0 views only the first pixel from the left, while view v1 views the first pixel from the left and the second pixel from the left. However, since the pixel that influences view v1 more is the second pixel from the left, the image of the second pixel from the left can be rendered to display the color corresponding to view v1. In addition, view v3 views the third pixel from the left and the fourth pixel from the left. However, since the pixel that influences view v3 more is the third pixel from the left, the image of the third pixel from the left can be rendered to display the color corresponding to view v3. Using this method, the color corresponding to view v2 may not be displayed.

[0052] To address this issue, images can be rendered by blending the colors corresponding to two adjacent views. For example, as shown in Fig. 1c, since the first pixel on the left is viewed from both views v0 and v1, the image can be rendered such that the first pixel on the left outputs a color that is a mixture of the color corresponding to view v0 and the color corresponding to view v1. Here, the blending method can be a weighted sum of colors based on the area ratio occupied by each view. Using this method, colors corresponding to specific views can be prevented from being omitted.

[0053] However, even when using this method, the multiple optical views may not be uniform due to aberrations of the lenticular lens. For example, if there were no aberrations of the lenticular lens, the distances between adjacent optical views among the multiple optical views may be equal, as shown on the left side of Fig. 1d. However, due to aberrations of the lenticular lens, the distances between adjacent optical views among the multiple optical views may be different, as shown on the right side of Fig. 1d. In this case, depth distortion of the image may occur.

[0054] Additionally, if there is no aberration of the lenticular lens, the focus of each of the plurality of optical views is aligned with the corresponding pixel, as shown on the left side of Fig. 1e. However, due to the aberration of the lenticular lens, the focus of at least some of the plurality of optical views may not be aligned with the corresponding pixel, as shown on the right side of Fig. 1e. In this case, crosstalk may occur due to images of adjacent views.

[0055] FIG. 2 is a block diagram showing the configuration of a display device (100) according to one embodiment of the present disclosure.

[0056] The display device (100) may be a device that provides multiple optical views. For example, the display device (100) may be a device equipped with a display, such as a TV, desktop PC, laptop, smartphone, tablet PC, etc., and that provides multiple optical views. However, the present invention is not limited thereto, and the display device (100) may be any device that can provide multiple optical views.

[0057] According to FIG. 1, the display device (100) includes a memory (110), a display panel (120), a lenticular lens (130), and a processor (140).

[0058] Memory (110) may refer to hardware that stores information such as data in an electrical or magnetic form so that a processor (140) or the like can access it. To this end, memory (110) may be implemented as at least one piece of hardware from among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid state drive (SSD), RAM, ROM, etc.

[0059] The memory (110) may store at least one instruction required for the operation of the display device (100) or the processor (140). Here, the instruction is a unit of code that instructs the operation of the display device (100) or the processor (140), and may be written in machine language, which is a language that a computer can understand. Alternatively, the memory (110) may store a plurality of instructions for performing a specific task of the display device (100) or the processor (140) as an instruction set.

[0060] The memory (110) may store data, which is information in bit or byte units that can represent characters, numbers, images, etc. For example, a projection matrix, a ray tracing module, etc. may be stored in the memory (110).

[0061] The memory (110) is accessed by the processor (140), and reading / writing / modifying / deleting / updating instructions, instruction sets, or data can be performed by the processor (140).

[0062] The display panel (120) is a configuration that displays an image and can be implemented as a display of various forms such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display panel (120) may also include a driving circuit, a backlight unit, etc. that can be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display panel (120) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0063] A lenticular lens (130) is arranged on the front of the display panel (120) to provide different viewpoints, i.e., multiple optical views, for each viewing area.

[0064] For example, the lenticular lens (130) may include a plurality of lens regions. Accordingly, the lenticular lens (130) may refract an image displayed on the display panel (120) through the plurality of lens regions. Each lens region may be formed to have a size corresponding to at least one pixel, so that light passing through each pixel may be differently dispersed for each of the plurality of optical views.

[0065] Alternatively, the lenticular lens (130) may be implemented in a form in which a plurality of lenticular lenses are combined. For example, the lenticular lens (130) may be implemented in a form in which a plurality of lenticular lenses are laminated in multiple layers. Alternatively, the lenticular lens (130) may be implemented as a liquid crystal lens. For example, the lenticular lens (130) may be an electronically variable lens using a liquid crystal. Alternatively, the lenticular lens (130) may be implemented as a barrier film. However, the present invention is not limited thereto, and the lenticular lens (130) may have any form as long as it can provide a plurality of optical views.

[0066] The lenticular lens (130) may operate by tilting at a certain angle to improve image quality. The processor (140) may divide a plurality of images, each corresponding to a plurality of optical views, based on the angle at which the lenticular lens (130) is tilted, and combine them to render a final image. Accordingly, the user may view an image displayed with a certain tilt on the subpixels of the display panel (120), rather than viewing an image displayed vertically or horizontally on the subpixels.

[0067] The processor (140) controls the overall operation of the display device (100). Specifically, the processor (140) is connected to each component of the display device (100) and can control the overall operation of the display device (100). For example, the processor (140) is connected to components such as a memory (110), a display panel (120), a lenticular lens (130), and the like and can control the operation of the display device (100).

[0068] The one or more processors (140) may include one or more of a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. The one or more processors (140) may control one or any combination of other components of the display device (100) and perform operations or data processing related to communication. The one or more processors (140) may execute one or more programs or instructions stored in the memory (110). For example, the one or more processors (140) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in the memory (110).

[0069] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0070] One or more processors (140) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (140) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0071] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0072] In embodiments of the present disclosure, one or more processors (140) may refer to a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the display device (100) is described below using the expression processor (140).

[0073] The processor (140) can acquire multiple images corresponding to multiple optical views. For example, the processor (140) can acquire multiple images corresponding to multiple optical views from an original image based on a depth map. Here, the depth map has a resolution corresponding to the original image and can include depth information for each pixel in the original image.

[0074] The processor (140) can render a final image from a plurality of images based on a projection matrix and display the final image through the display panel (120). For example, the processor (140) can render the final image by multiplying the plurality of images and the projection matrix. For example, the processor (140) can convert the plurality of images into vector form and multiply the plurality of images converted into vector form by the projection matrix to render the final image.

[0075] Here, the projection matrix may be a matrix obtained through ray tracing based on the arrangement pattern of a plurality of sub-pixels and the shape of the lenticular lens (130). Ray tracing may be an operation of tracing information on a plurality of arrival locations, brightness, color, etc., corresponding to each of the plurality of arrival locations when a plurality of light rays emitted from a plurality of starting locations reach a plurality of arrival locations through a medium, etc. Here, the shape of the lenticular lens (130) may include at least one of the thickness, curvature, pitch, focal length, or slanted angle of the lenticular lens (130). That is, the projection matrix may be a matrix that takes into account not only the arrangement pattern of a plurality of sub-pixels but also the refraction of light rays by the lenticular lens.

[0076] The projection matrix can be obtained by an external device during the manufacturing process of the display device (100). For example, the external device can obtain the projection matrix through ray tracing simulation based on the specifications of the display device (100), etc., and provide the projection matrix to the display device (100).

[0077] However, this is not limited to this, and the display device (100) may also perform ray tracing to obtain a projection matrix. Below, two methods for the display device (100) to perform ray tracing are described.

[0078] The processor (140) can identify the refractive index at each point of the lenticular lens (130) based on the shape of the lenticular lens (130), perform ray tracing based on the arrangement pattern of a plurality of sub-pixels and the identified refractive index to obtain a projection matrix, and store the projection matrix in the memory (110).

[0079] For example, the processor (140) can perform ray tracing by identifying the position when each of a plurality of light rays originating from a plurality of sub-pixels reaches the viewing distance by being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens (130).

[0080] However, the present invention is not limited thereto, and the processor (140) may perform ray tracing by randomly identifying some sub-pixels from a plurality of sub-pixels, and identifying the position when each of the plurality of light rays starting from some of the sub-pixels reaches the viewing distance after being refracted by the refractive index of the point at the corresponding point of the lenticular lens (130). For example, the processor (140) may perform ray tracing so that the light rays start by shifting by a distance of less than 1 pixel in a random direction from the position corresponding to each of the plurality of sub-pixels. That is, the ray tracing is performed in a planar form rather than a grid, so that the image quality can be improved.

[0081] The processor (140) may obtain a view-mapped image by view-mapping a plurality of images using a preset view-mapping method, and may perform ray tracing by identifying a position when each of a plurality of light rays originating from a plurality of sub-pixels based on the view-mapped image reaches a viewing distance after being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens (130). Here, the view mapping may be a method of rendering a final image from a plurality of images each corresponding to a plurality of optical views, and the preset view mapping method may be a method of rendering a final image using a method such as that shown in FIG. 1b or FIG. 1c. However, the present invention is not limited thereto, and the preset view mapping method may be a conventional view-mapping method, and any method may be used.

[0082] In the above, it was explained that multiple rays originate from multiple sub-pixels, but ray tracing is also possible in the opposite direction.

[0083] For example, the processor (140) may perform ray tracing by identifying the position when each of a plurality of light rays originating from a plurality of optical views reaches the display panel (120) after being refracted by the refractive index of the point at the corresponding point of the lenticular lens (130).

[0084] For example, the processor (140) can perform ray tracing by identifying the position when each of a plurality of light rays originating from a plurality of optical views based on a plurality of images reaches the display panel (120) by being refracted by the refractive index of the point at the corresponding point of the lenticular lens (130).

[0085] For convenience of explanation, the processor (140) is described as performing ray tracing below. However, the entity performing ray tracing may be an external device.

[0086] FIG. 3 is a block diagram showing a detailed configuration of a display device (100) according to one embodiment of the present disclosure.

[0087] The display device (100) may include a memory (110), a display panel (120), a lenticular lens (130), and a processor (140). In addition, according to FIG. 3, the display device (100) may further include a communication interface (150), a user interface (160), a camera (170), a microphone (180), and a speaker (190). Among the components illustrated in FIG. 3, a detailed description of the overlapping parts with the components illustrated in FIG. 2 will be omitted.

[0088] The communication interface (150) is a configuration that performs communication with various types of external devices according to various types of communication methods. For example, the display device (100) can perform communication with a server through the communication interface (150).

[0089] The communication interface (150) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0090] Wi-Fi and Bluetooth modules communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received. Infrared communication modules use infrared data association (IrDA) technology, which wirelessly transmits data over short distances using infrared light, which lies between visible light and millimeter waves.

[0091] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0092] Alternatively, the communication interface (150) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0093] In addition, the communication interface (150) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or an optical fiber cable.

[0094] The user interface (160) may be implemented with buttons, a touch pad, a mouse, a keyboard, etc., or may be implemented with a touch screen capable of performing both display and operation input functions. Here, the buttons may be various types of buttons, such as mechanical buttons, touch pads, wheels, etc. formed on any area of ​​the front, side, or back of the main body of the display device (100).

[0095] The camera (170) is configured to capture still images or moving images. The camera (170) can capture still images at a specific point in time, but can also capture still images continuously.

[0096] The camera (170) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time at which light reflected from a subject enters the camera (170), and the aperture mechanically increases or decreases the size of the opening through which light enters to control the amount of light incident on the lens. When the solid-state image sensor accumulates light reflected from a subject as a photocharge, the image generated by the photocharge is output as an electrical signal. The TG outputs a timing signal for reading out pixel data of the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0097] The microphone (180) is configured to receive sound and convert it into an audio signal. The microphone (180) is electrically connected to the processor (140) and can receive sound under the control of the processor (140).

[0098] For example, the microphone (180) may be formed as an integrated unit integrated into the upper side, front side, side side, etc. of the display device (100). Alternatively, the microphone (180) may be provided in a remote control separate from the display device (100). In this case, the remote control may receive sound through the microphone (180) and provide the received sound to the display device (100).

[0099] The microphone (180) may include various configurations such as a microphone that collects sound in analog form, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0100] Meanwhile, the microphone (180) may be implemented in the form of a sound sensor, and any method may be used as long as it has a configuration capable of collecting sound.

[0101] The speaker (190) is a component that outputs various audio data processed by the processor (140) as well as various notification sounds and voice messages.

[0102] As described above, the display device (100) can render a final image with a projection matrix obtained through ray tracing to provide an image with improved image quality for each of the multiple optical views.

[0103] Hereinafter, the operation of the display device (100) will be described in more detail with reference to FIGS. 4 to 15. For convenience of explanation, individual embodiments are described in FIGS. 4 to 15. However, the individual embodiments of FIGS. 4 to 15 may be implemented in any combination.

[0104] FIGS. 4 to 7 are drawings for explaining ray tracing based on an optical simulator according to an embodiment of the present disclosure.

[0105] The processor (140) can obtain a projection matrix through ray tracing based on an optical simulator. For example, the processor (140) can perform ray tracing as shown in FIG. 4 through a commercial optical simulator such as CodeV, Zemax, Matlab, etc. Here, the processor (140) can perform ray tracing based on the arrangement pattern of a plurality of sub-pixels and the shape of the lenticular lens (130). The shape of the lenticular lens (130) can include at least one of the thickness, curvature, pitch, focal length, or slanted angle of the lenticular lens (130).

[0106] To explain this in more detail, as illustrated in FIG. 5, the processor (140) can optically calculate the path of light rays starting at various angles from the first optical view (Viewpoint) among the plurality of optical views (Viewpoint-1, Viewpoint, Viewpoint+1) until they reach the display panel (120) after being refracted by the lenticular lens (130).

[0107] In this process, the processor (140) can calculate the refraction of each ray based on Snell's law, as shown in FIG. 6.

[0108] The processor (140) performs this operation for all of the plurality of optical views, and as shown in FIG. 7, matches the starting and ending positions of each of the plurality of light rays to obtain ray tracing information, and based on the ray tracing information, obtains a projection matrix by assigning a plurality of images corresponding to each of the plurality of optical views to each subpixel of the display panel (120).

[0109] FIG. 8 is a flowchart illustrating rendering using ray tracing according to one embodiment of the present disclosure.

[0110] First, the processor (140) can project a plurality of light rays based on the pixel pattern of the display panel (120) (S810). Here, the processor (140) can store the projection position, projection angle, etc. of each light ray. In addition, the number of the plurality of light rays may be equal to or greater than the number of the plurality of sub-pixels.

[0111] The processor (140) can identify the location where each ray reaches the lens based on the pitch, attachment angle, pixel spacing, etc. of the lenticular lens (130) (S820). When each ray reaches the lens, the processor (140) can identify the refraction of each ray using the incident angle at each arrival point and Snell's law.

[0112] The processor (140) can calculate the coordinates at which each ray passes through free space and reaches the viewing position (S830) and store the final light field obtained as a pair of starting and arriving coordinates (S840). For example, the processor (140) can obtain a projection matrix based on the pair of starting and arriving coordinates.

[0113] The processor (140) can acquire a viewpoint image using the stored coordinate values ​​and output the image (S850). For example, the processor (140) can acquire a final image based on a plurality of images and a projection matrix, each corresponding to a plurality of optical views, and control the display panel (120) to display the final image.

[0114] Although FIG. 8 illustrates that multiple light rays originate from the display panel (120), this is not limiting. For example, the processor (140) may perform ray tracing in such a way that multiple light rays originate from the viewing position.

[0115] FIGS. 9 to 11 are drawings illustrating the advantages of ray tracing according to one embodiment of the present disclosure.

[0116] In the past, it was assumed that the distance between adjacent optical views among multiple optical views was the same, but in reality, as shown in Fig. 9, the distance between adjacent optical views among multiple optical views may be different.

[0117] These problems can be solved by tracing actual rays using ray tracing. For example, when performing ray tracing, problems like the above can be solved by identifying multiple actual optical views along the ray and performing rendering based on them, as illustrated in Figure 10.

[0118] That is, when using ray tracing, hardware-specific artifacts such as lens aberrations can be resolved, allowing the display device (100) to reproduce the light field with the maximum possible performance.

[0119] Additionally, since the pixel pattern is taken into account in the ray tracing process, the final image can be organically rendered to fit the complex pixel pattern of various panels, as shown in Fig. 11.

[0120] FIG. 12 is a diagram for explaining the rendering operation of a final image according to one embodiment of the present disclosure.

[0121] First, the processor (140) can obtain ray-tracing data by simulating an actual optical system through an optical simulator (1210) and then performing ray tracing.

[0122] The processor (140) can select light information by considering the arrangement pattern of a plurality of sub-pixels included in the display panel (120) and obtain a projection matrix (1220) reflecting the viewpoint.

[0123] The processor (140) can obtain a final image (image data, 1230) based on a plurality of images (Viev images) and a projection matrix corresponding to each of a plurality of optical views, and output the final image through a display panel (120) (1240).

[0124] FIG. 13 is a diagram for explaining a projection matrix according to an embodiment of the present disclosure.

[0125] The processor (140) can obtain a projection matrix from a plurality of images and a final image, each corresponding to a plurality of optical views, based on ray tracing data. Here, the final image may be an image rendered according to a conventional technique from the plurality of images.

[0126] For example, the processor (140) may obtain data to be output by the display panel (120) corresponding to a plurality of images corresponding to a plurality of optical views, respectively, based on ray tracing data, and obtain a projection matrix based on the obtained data and the final image.

[0127] At this time, the processor (140) can perform an operation by converting each data into the dimension shown in FIG. 13. For example, multiple images can be converted into M×1 data. Here, M is x×y×3×n, x and y are coordinate data of each image, 3 represents RGB, and n can be an index for each of the multiple optical views. The final image can be converted into N×1 data. Here, N is x'×y'×3, x' and y' are coordinate data of the final image, and 3 can represent RGB. In this case, the projection matrix can be calculated as N×M.

[0128] The processor (140) can improve rendering efficiency by using a projection matrix.

[0129] For example, in the past, the final image was obtained through calculations in the form of a for loop, as shown below. As the image size increased, the number of for loops increased, which could increase the amount of calculation and delay the calculation time.

[0130]

[0131] Load image_data

[0132]

[0133] For x in range(horizontal):

[0134] For y in range(vertical):

[0135]

[0136] * For c in range(color):

[0137] View_mapping(display_image(x,y,c), image_data)

[0138]

[0139] Return display_image

[0140]

[0141] On the other hand, according to the present disclosure, the final image can be obtained through matrix multiplication as follows, so that high-speed calculation and parallelization can be possible.

[0142]

[0143] Load projection_matrix

[0144] Load image_data

[0145]

[0146] Display image = Multiplication(Projection_matrix, image_data)

[0147]

[0148] Return display_image

[0149]

[0150] FIG. 14 is a diagram for explaining ray tracing through random sampling according to one embodiment of the present disclosure.

[0151] The processor (140) can perform ray tracing using a number of rays equal to or greater than the number of sub-pixels included in the display panel (120). The processor (140) can also obtain ray tracing data by sampling only a portion of them.

[0152] Here, the position of each ray can be sampled in two ways, as shown in Fig. 14.

[0153] First, the processor (140) may sample in a grid form, as in 1420. Alternatively, the processor (140) may randomly shift the sampling position by a distance of less than 1 pixel, as in 1410. Shifting the sampling position may provide an improved image compared to not shifting it.

[0154] FIG. 15 is a drawing for explaining image quality improvement according to one embodiment of the present disclosure.

[0155] In the past, when view mapping was performed out of focus, details could be blurred, as shown on the left side of Fig. 15, but when ray tracing was used, details could be improved, as shown on the right side of Fig. 15.

[0156] In addition, when forming a repeating viewpoint such as black, white, black, even if the intervals between multiple optical views are different, view mapping is performed at equal intervals in the past, but according to the present disclosure, a result in which aberration is taken into account according to ray tracing can be provided.

[0157] FIG. 16 is a flowchart for explaining a method for controlling a display device according to an embodiment of the present disclosure.

[0158] First, a method for controlling a display device acquires a plurality of images, each corresponding to a plurality of optical views (S1610). Then, a final image is rendered from the plurality of images using a projection matrix acquired through ray tracing based on an arrangement pattern of a plurality of sub-pixels included in a display panel and a shape of a lenticular lens arranged on the front of the display panel to separate a field of view and provide a plurality of optical views (S1620). Then, the final image is displayed through the display panel (S1630).

[0159] Additionally, the shape of the lenticular lens may include at least one of a thickness, a curvature, a pitch, a focal length, or a slanted angle of the lenticular lens.

[0160] And, the rendering step (S1620) can render the final image by multiplying multiple images and projection matrices.

[0161] Additionally, the rendering step (S1620) can convert multiple images into vector form and render the final image by multiplying the multiple images converted into vector form and a projection matrix.

[0162] And, the method may further include a step of identifying a refractive index at each point of the lenticular lens based on a shape of the lenticular lens, a step of performing ray tracing based on an arrangement pattern of a plurality of sub-pixels and the identified refractive index to obtain a projection matrix, and a step of storing the projection matrix in a memory of the display device.

[0163] In addition, the step of obtaining the projection matrix can perform ray tracing by identifying the position when each of a plurality of rays originating from a plurality of sub-pixels reaches the viewing distance after being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens.

[0164] And, the step of obtaining the projection matrix can perform ray tracing by randomly identifying some sub-pixels from a plurality of sub-pixels, and identifying the position when each of a plurality of rays starting from some sub-pixels reaches the viewing distance by being refracted by the refractive index of the point at the corresponding point of the lenticular lens.

[0165] In addition, the step of obtaining a projection matrix may include view-mapping a plurality of images using a preset view-mapping method to obtain a view-mapped image, and based on the view-mapped image, ray tracing may be performed by identifying a position when each of a plurality of rays originating from a plurality of sub-pixels reaches a viewing distance by being refracted by a refractive index of a corresponding point at a corresponding point of a lenticular lens.

[0166] And, the step of obtaining the projection matrix can perform ray tracing by identifying the position when each of a plurality of rays originating from a plurality of optical views reaches the display panel after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

[0167] In addition, the step of obtaining the projection matrix can perform ray tracing by identifying the position when each of a plurality of rays originating from a plurality of optical views based on a plurality of images reaches the display panel after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

[0168] According to various embodiments of the present disclosure as described above, the display device can render a final image with a projection matrix obtained through ray tracing to provide an image with improved image quality for each of a plurality of optical views.

[0169] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0170] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0171] Furthermore, according to one embodiment of the present disclosure, the various embodiments described above may be implemented in a computer-readable recording medium or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software. Each software may perform one or more functions and operations described herein.

[0172] Meanwhile, computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0173] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0174] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the display device, Memory where the projection matrix is ​​stored; A display panel comprising a plurality of sub-pixels; A lenticular lens arranged on the front of the display panel to separate the viewing area and provide multiple optical views; and comprising one or more processors connected to the memory, the display panel and the lenticular lens to control the display device; The above processor, Acquire multiple images each corresponding to the multiple optical views, Rendering a final image from the plurality of images based on the projection matrix, The above final image is displayed through the above display panel, The above projection matrix is, A display device, wherein the matrix is ​​obtained through ray tracing based on the arrangement pattern of the plurality of sub-pixels and the shape of the lenticular lens.

2. In paragraph 1, The shape of the above lenticular lens is, A display device comprising at least one of a thickness, curvature, pitch, focal length, or slanted angle of the lenticular lens.

3. In paragraph 1, The above processor, A display device that renders the final image by multiplying the plurality of images and the projection matrix.

4. In paragraph 3, The above processor, Convert the above multiple images into vector format, A display device that renders the final image by multiplying a plurality of images converted into the vector form and the projection matrix.

5. In paragraph 1, The above processor, Identifying the refractive index at each point of the lenticular lens based on the shape of the lenticular lens, The projection matrix is ​​obtained by performing the ray tracing based on the arrangement pattern of the plurality of sub-pixels and the identified refractive index, A display device storing the above projection matrix in the above memory.

6. In paragraph 5, The above processor, A display device that performs ray tracing by identifying a position when each of a plurality of light rays starting from the plurality of sub-pixels reaches a viewing distance after being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens.

7. In paragraph 6, The above processor, Randomly identifying some sub-pixels from the above plurality of sub-pixels, A display device that performs ray tracing by identifying a position when each of a plurality of light rays starting from some of the sub-pixels reaches the viewing distance after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

8. In paragraph 6, The above processor, View-mapping the above multiple images using a preset view mapping method to obtain a view-mapped image, A display device that performs ray tracing by identifying a position when each of a plurality of light rays departing from the plurality of sub-pixels based on the view-mapped image reaches a viewing distance after being refracted by the refractive index of the corresponding point at the corresponding point of the lenticular lens.

9. In paragraph 5, The above processor, A display device that performs ray tracing by identifying a position when each of a plurality of light rays departing from the plurality of optical views reaches the display panel after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

10. In paragraph 9, The above processor, A display device that performs ray tracing by identifying a position when each of a plurality of light rays departing from the plurality of optical views based on the plurality of images reaches the display panel after being refracted by the refractive index of the point at a corresponding point of the lenticular lens.

11. In a method for controlling a display device, A step of acquiring a plurality of images, each corresponding to a plurality of optical views; A step of rendering a final image from the plurality of images using a projection matrix obtained through ray tracing based on an arrangement pattern of a plurality of sub-pixels included in a display panel and a shape of a lenticular lens arranged on a front side of the display panel to separate a field of view and provide the plurality of optical views; and A control method, comprising: a step of displaying the final image through the display panel; 12. In paragraph 11, The shape of the above lenticular lens is, A control method comprising at least one of the thickness, curvature, pitch, focal length or slanted angle of the lenticular lens.

13. In paragraph 11, The above rendering steps are: A control method for rendering the final image by multiplying the plurality of images and the projection matrix.

14. In paragraph 13, The above rendering steps are: Convert the above multiple images into vector format, A control method for rendering the final image by multiplying a plurality of images converted into the vector form and the projection matrix.

15. In paragraph 11, A step of identifying a refractive index at each point of the lenticular lens based on the shape of the lenticular lens; A step of obtaining the projection matrix by performing the ray tracing based on the arrangement pattern of the plurality of sub-pixels and the identified refractive index; and A control method further comprising the step of storing the projection matrix in a memory of the display device.

Images