Camera built-in type display

Abstract

To provide a camera built-in type display.SOLUTION: The present application relates to a device and a vehicle having such an apparatus. The device includes a display device (1) and a camera (3). The display device (1) includes a cover glass (7), an LCD panel (2) and a backlight unit. The LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), and the camera (3) is arranged behind the backlight unit with its field of view facing the backlight unit. The field of view of the camera (3) defines an active area (9) of the camera (3). The backlight unit includes an edge light source (8), a light enhancement layer (4) and a light guide (5) arranged behind the light enhancement layer (4). The camera (3) operates in an infrared wavelength spectrum, the light enhancement layer (4) has a cut-out window (6) in the active area (9) of the camera (3), and the light guide (5) is a light guide (5) having a nanostructure.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a camera-integrated display and a vehicle equipped with such an integrated display.

Background Art

[0002] Considering the current trend in display technology, more and more OEMs (OEM: Original Equipment Manufacturer) in various sectors are trying to integrate a camera behind the display in their products. This type of approach can be seen in consumer products such as smartphones, tablets, and laptops, and currently, there is also increasing interest in automotive parts.

[0003] In addition to this, driver monitoring systems are being made mandatory in order to meet new safety standards, and thus, camera systems are beginning to be integrated into various devices located throughout vehicles such as automobiles, trucks, and buses.

[0004] To address this issue, various methods have been used by OEMs and manufacturers in various industries trying to achieve seamless camera integration in their devices.

[0005] The technology used to integrate a camera behind a display is mainly affected by display type technologies such as LCD (LCD: Liquid Cristal Display) displays and LED (LED: Light-Emitting Diode) displays.

[0006] Smartphone manufacturers are addressing this issue by first forming a notch in the display and installing a camera behind the cover glass of the display within the notch. However, such a camera installed in the notch cannot be regarded as seamlessly integrated or as a camera behind the display.

[0007] To achieve seamless integration of the camera behind the display, several options have been found. For example, (Patent Document 1) and (Patent Document 2) describe using a lower pixel density and / or smaller pixels in the active area of the camera. Thus, such an approach proposes a lower pixel density and / or smaller pixels to create sufficient space for light to enter in the area where the camera captures an image. Different pixel arrangements are also possible with such an approach, because this aspect can also affect the quality of the image. However, such a solution of using a lower pixel density and / or smaller pixels in the active area of the camera results in each platform or application requiring an LCD panel specially designed to meet the product requirements, and furthermore, leads to an increase in design complexity and cost in such a solution.

[0008] (Patent Document 3) describes a camera installed in a through-hole behind an LCD display. The LCD display includes a cover glass, an optically clear adhesive (OCA) layer, a first polarizer, a first glass substrate, a color film (CF) layer, a thin film transistor (TFT) layer, a second glass substrate, a second polarizer, and a backlight module. The through-hole is provided so as to penetrate the first polarizer, the CF layer, the TFT layer, the second polarizer, and the backlight module. The through-hole forms an optical path in the LCD for ambient light to enter from the outside of the electronic device, pass through the cover glass and the optical path, and enter the camera. The camera is disposed entirely or partially within the through-hole. The first glass substrate and the second glass substrate are not cut out in the region corresponding to the optical path. However, such a method requires a great deal of labor to align such through-holes in almost all parts of the LCD. Furthermore, opening such through-holes in almost all parts of the LCD can cause damage to each part of the LCD. Therefore, the parts must be managed with respect to damage and accurate alignment with the through-holes. All of the above ultimately leads to an increase in cost.

[0009] (Patent Document 4) describes a camera device located below the display screen. The display includes a display screen and a camera. The display screen has a cover at the top and a display module under the cover. The display module includes ITO (ITO: Indium Tin Oxide) glass, a liquid crystal layer, and a backlight layer. The display screen is provided with a stop hole area, and the stop hole area is formed above an opening that penetrates the entire backlight layer on the liquid crystal layer. The camera is disposed directly below the stop hole, and the lens of the camera extends into the hole of the backlight layer up to the stop hole area, forming an angle of view passing through the stop hole. The display screen further includes an electric field circuit, and the electric field circuit is provided on the outermost ITO glass layer of the liquid crystal layer. The electrolytic circuit changes the optical path of the light passing through the stop hole area due to the change in the electric field, and drives the deflection of the liquid crystal in the stop hole area of the display screen to achieve focusing of the camera. Such a method is complex to implement and means an increase in cost. Furthermore, the complete hole that penetrates the entire backlight layer affects the light guide in that light scattering is disrupted in the area of the hole, resulting in problems with image capture.

[0010] Furthermore, due to the opacity of the LCD panel and the polarizer, it is difficult to develop a seamless camera-integrated device. The LCD panel is composed of a liquid crystal layer sandwiched between two electrodes together with a polarizer, which is an optical filter that transmits light of a specific polarization. Usually, a general LCD panel has one polarizer on each side. When light passes through the LCD panel and the polarizer, the light is partially absorbed and scattered, so the amount of light reaching the camera sensor behind the display decreases. This can lead to a decrease in image quality and camera performance, so it is necessary to increase the amount of hardware compensation.

[0011] (Patent Document 5) describes an electronic device including a front camera disposed behind a front display. Such a device described in (Patent Document 5) includes a display having a front surface and a back surface, the display including a plurality of pixel regions that emit light from the front surface of the display to display a display image, and a plurality of openings that transmit light from the front surface to the back surface. The camera is disposed on the back side of the display. The camera is configured to capture an image. The device further includes an optical correction system disposed between the display and the camera, the optical correction system being configured to reduce distortion of an image caused by the display. The optical correction system may further include an optical filter (e.g., a phase shift mask), a bandpass filter, a lens including a variable focus lens, and / or different and / or additional components. It is further described that various items of the optical correction system can be included in the camera and thus the optical correction system can be eliminated. Further, a processor is coupled to the display and the camera, and the processor is configured to apply a digital filter to at least a portion of the captured image to further reduce distortion of an image caused by the display. Thus, distortion of an image caused by the display is physically reduced by the optical correction system and digitally reduced by the processor. Thus, (Patent Document 5) describes a complex setup that requires reducing distortion of an image caused by the display both physically and digitally. However, such an approach also leads to a great deal of effort and consequently high costs.

[0012] (Patent Document 6) describes an image capture device that operates in the visible spectrum and is disposed either behind a switchable diffuser and behind a light guide, or in a hole or opening within the light guide behind a display. The switchable diffuser is disposed in front of the light guide. (Patent Document 6) further describes that the light guide typically has scattering elements, also called scattering dots, on the back side of the light guide plate to scatter the light conducted from a backlight bulb into the light guide plate towards the front side of the LCD display assembly. Such scattering elements are also known in the art as dot structures, microdots, or microlenses, are provided on the surface of the light guide, typically on the back side of the light guide, and are used to direct light towards the display panel as described in (Patent Document 6). The amount of light resulting therefrom is defined by the density of the population of dot structures, microdots, or microlenses. The higher the density, the more light exits the light guide in the desired area. However, a single microdot or microlens does not produce a uniform light output, so smaller densities are limited. The dot structure is known as small printed white dots on the back side of the light guide, while the microdots or microlenses are hemispherical imprints or injection molded microdots with a spherical cap provided on a surface on the back side or the front side of the light guide plate. The spherical cap, or more precisely the diameter of the microdot or microlens, has a size of several tens of micrometers.

[0013] (Patent Document 6) further describes the importance of using a switchable diffuser so that the image capture device can capture sufficient light, improve the uniformity of the light passing through the LCD panel, and make the scattering elements less visible from the front side of the LCD.

[0014] Furthermore, in order to be able to obtain a clear image, the switchable diffuser, LCD, and image capture device must be precisely controlled so that the camera can capture the image. Therefore, (Patent Document 6) proposes control logic for the LCD panel, camera, and diffuser so that sufficient light is captured by the camera. Therefore, in the solution known from (Patent Document 6), it is necessary to increase the amount of logic and hardware to execute all the steps required to capture an image. Furthermore, considering the control logic required for switching between the diffuser and the LCD screen, it may be difficult to finally incorporate this type of solution into a low-cost product.

Prior Art Documents

Patent Documents

[0015]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Summary of the Invention

Problems to be Solved by the Invention

[0016] Therefore, an object of the present invention is to provide an improved display having a seamless integrated camera behind the display.

Means for Solving the Problems

[0017] According to the independent claims, the above object is achieved by a camera-integrated display and a vehicle comprising such an integrated display. The dependent claims include further advantageous developments and improvements of this principle as described below.

[0018] According to a first aspect, the apparatus comprises a display device and a camera. The display device comprises a cover glass, an LCD panel, and a backlight unit. The LCD panel is arranged behind the cover glass, the backlight unit is arranged behind the LCD panel, and the camera is arranged behind the backlight unit with its field of view directed towards the backlight unit, more precisely towards the light guide of the backlight unit which will be described in more detail later. The cover glass is the part of the display closest to the user looking at the display. For example, behind the cover glass means that each component behind the cover glass, in this case the LCD panel, is arranged on the side of the display glass opposite to the side of the display glass that the user is looking at when looking at the display. The field of view of the camera defines the active area of the camera. It should be understood that the field of view of the camera, and thus the active area, becomes larger, for example, when a wide-angle lens is used and / or when the camera is arranged further away from the backlight unit. Conversely, it should also be understood that the field of view of the camera, and thus the active area, becomes smaller, for example, when a telephoto lens is used and / or when the camera is moved closer to the backlight unit.

[0019] The backlight unit includes an edge light source, a light enhancement layer, and a light guide. The light guide is disposed behind the light enhancement layer. The camera operates in the infrared wavelength spectrum. The light enhancement layer may include one or more foils. The advantage of using a camera that operates in the IR wavelength spectrum is that the infrared light transmittance is not affected or only weakly affected by the LCD panel and its polarizers. The term "operates in the infrared wavelength spectrum" means that the camera, more precisely the image capture element of the camera, is sensitive at least to the infrared spectrum and captures an image at least in the IR spectrum (IR: infrared). The camera may be operable only in the IR spectrum, i.e., configured as an IR camera, or it should be understood that in addition to operating in the IR spectrum, it may also operate in the visible wavelength spectrum. The light enhancement layer has a cut-out window in the active area of the camera. It should be understood that the cut-out window preferably has the same size as the size of the active area. However, the cut-out window may be smaller than the size of the active area. This has the advantage that the cut-out window has less disturbing visual impact on the user looking at the display (e.g., the driver). Alternatively, the cut-out window may be larger than the size of the active area. This has the advantage that the adverse effects of the light enhancement layer do not appear in the camera image. The term "cut-out window" in the light enhancement layer means that there is an area defined by the active area of the camera in the light enhancement layer, and in that area, the light enhancement layer is modified, for example, there are holes in a part of the light enhancement layer, and through that modified area, the camera can see through without being obstructed.

[0020] Furthermore, the light guide is a light guide having a nanostructure instead of a microstructure. It has been found that a light guide having a nanostructure has no effect on infrared light when compared with a light guide having a microstructure. Therefore, a light guide having a nanostructure is suitable for use with a camera operating in the IR wavelength spectrum without the need for further processing. On the surface of the light guide, nanostructures can be provided on the back or front side of the light guide. However, when applied to the back surface, such structures are less visible. The term "nanostructure" means that a structure, such as a spherical cap on the surface of the light guide, has a size smaller than 1 μm with respect to its diameter, more precisely the maximum extent in one dimension. Preferably, the structure has a size of 1 nm to 1 μm. Particularly preferably, the structure has a size of 100 nm to 990 nm. The form, size, and density of the elements, i.e., the density per a given area, can be individually selected according to the technical requirements of the light guide and the device (e.g., display) used. For example, the form of such nanostructures can be something other than spherical caps, such as pyramids, cones, frustums of cones, and / or cylinders. It should be understood that the present invention is not limited to the forms of the structures described above and that there are many further forms that can be used for those structures.

[0021] The present invention proposes a simple solution for seamlessly integrating a camera (e.g., a driver monitoring module) into a display without affecting the display content. For example, assuming that the camera can capture images only when the diffuser is set to the "clear" state and the LCD panel is set to the "transparent" state, this solution eliminates the need for a switchable diffuser and the increased logic for controlling it. This solution further eliminates the need to drive the liquid crystal alignment in the dead pixel area of the display screen and change the optical path of the light passing through the dead pixel area by the need for through holes in the components of the entire LCD panel and the backlight unit and / or the change in the applied electric field to achieve camera focusing. Furthermore, in this solution, it is not necessary to physically reduce the image distortion caused by the display, for example, by using a correction optical system.

[0022] Furthermore, considering the fact that the proposed solution can use any commercially available LCD screen without additional processing, the camera can be accommodated at any location on the LCD panel. The seamless integration of the camera behind the display proposed improves the user experience while adding functionality to the device.

[0023] According to an embodiment of the present application, the cover glass is optically adhered to the LCD panel. It should be understood that the optical adhesion is clear, the cover glass and the optical adhesive material are transparent, and do not affect the behavior of the captured image. By optically adhering the cover glass to the LCD panel, the gap between the cover glass and the LCD panel is eliminated, thereby improving, for example, display performance, visibility, touch accuracy, and impact resistance.

[0024] According to another embodiment of the present invention, the light enhancement layer is a laminated foil including a louver foil and / or a reflective polarizing foil and / or a diffusion foil, and is formed by the laminated foil in which at least one of the louver foil and / or the reflective polarizing foil and / or the diffusion foil includes a cutout window. It should be understood that depending on the characteristics of each foil, it may be sufficient if only one of the foils of the laminated foil has a cutout window. For example, some of such laminated foils may not affect or only slightly affect IR light. Using a laminated foil made of foils having different characteristics has the advantage that the optical performance of such a device (e.g., a display) can be improved according to the technical requirements of the device.

[0025] According to another embodiment of the present application, the apparatus includes a processor, and the processor is provided to apply a post-processing algorithm, preferably a deconvolution algorithm, to the captured image captured by the camera. The post-processing algorithm enhances the quality of the captured image with respect to the influence exerted by the LCD panel on the captured image. It should be understood that the post-processing algorithm is preferably a deconvolution algorithm. However, there are further post-capture image processing methods for enhancing the image quality of the captured image that can also be advantageously used as such a post-processing algorithm.

[0026] According to another embodiment of the present application, the size of the cutout window is smaller than the active area. The camera detects image information of a first area within the cutout window and a second area outside the cutout window. The detected image information from both areas is provided to the post-processing algorithm. Such an embodiment advantageously provides information for calibrating the post-processing algorithm.

[0027] According to another embodiment of the present application, the LCD panel includes two polarizers, and at least one of the two polarizers does not affect or only weakly affects the IR light passing through the corresponding polarizer. Such an embodiment improves the light collection rate of the IR spectrum by a camera behind the display. It should be understood that such an embodiment may include a special physical arrangement of at least one polarizer that allows at least a portion of the IR light to pass through at least one polarizer. Preferably, a portion or the whole of at least one polarizer on the line of sight of the camera includes or is composed of a material that allows at least a portion of the IR light to pass through at least one polarizer.

[0028] According to another embodiment of the present application, the LCD panel includes two polarizers, and both of the polarizers do not affect or only weakly affect the IR light passing through the polarizers. Such an embodiment further improves the light collection rate of the IR spectrum by a camera behind the display. It should be understood that such an embodiment may include a special physical arrangement of both polarizers that allows at least a portion of the IR light to pass through both polarizers. Preferably, a portion or the whole of each of the two polarizers on the line of sight of the camera includes or is composed of a material that allows at least a portion of the IR light to pass through both polarizers.

[0029] Preferably, the liquid crystal material of the LCD panel does not affect or only weakly affects the IR light passing through the LCD panel. Such an embodiment further improves the light collection rate of the IR spectrum by a camera behind the display, similar to that described above for the polarizer.

[0030] According to a second aspect, a vehicle includes an apparatus according to at least one of the above-described embodiments.

[0031] Although the use of such a display having a camera seamlessly integrated behind the display has been described, furthermore, such a camera seamlessly integrated behind the display can be introduced into products that follow the user and can provide an interactive experience taking into account the user's eye orientation, especially for entertainment activities. For example, a camera can be introduced behind a smart TV display, and through certain compatible content, the user can use their digital twin to experience various games and participate in sports classes in real time. A further example is an apparatus that includes a camera seamlessly integrated behind the display and additionally includes a motion sensor input device that enables functions such as skeleton tracking, hand interaction, and voice recognition. Such a motion sensor can track the movement of a person in front of the device and form a link between the movement and the "digital twin" displayed on the display. For this purpose, such a motion sensor input device includes an RGB camera capable of performing real-time gesture recognition as well as an infrared projector and detector. By placing the camera solution according to the present invention behind a monitor or TV display, an integrated seamless solution can be provided without the need for a second device, enabling a better experience for the user.

[0032] Further features of the present invention will become apparent from the following description in conjunction with the drawings and the appended claims.

Brief Description of the Drawings

[0033]

Figure 1

Figure 2

Figure 3

Mode for Carrying Out the Invention

[0034] To better understand the principles of the present invention, embodiments of the present invention will be described in more detail below with reference to the drawings. In the drawings, like reference numerals are used for the same or equivalent elements and are not necessarily repeatedly described for each drawing. It should be understood that the present invention is not limited to the illustrated embodiments, and the features described may be combined or modified without departing from the scope of protection of the present invention as defined in the appended claims.

[0035] FIG. 1 shows a cross-sectional view of an apparatus according to an embodiment of the present invention. The apparatus includes a display device 1 and a camera 3 operating in the infrared wavelength spectrum. The display device 1 includes a cover glass 7, an LCD panel 2, and a backlight unit. The cover glass 7 is optically adhered to the LCD panel 2, and the LCD panel 2 is disposed behind the cover glass 7. The backlight unit includes a light source 8 disposed at the edge of the display device 1, a light enhancement layer 4, and a light guide 5. The light guide is disposed behind the light enhancement layer 4, and the backlight unit as a whole is disposed behind the LCD panel 2. The light source 8 is disposed at the edge of the light guide 5. The light emitted by the light source 8 is coupled to the light guide at the edge of the light guide 5. The camera 3 is disposed behind the backlight unit with its field of view directed towards the backlight unit, more precisely the light guide 5. The field of view of the camera 3 defines the active area 9 of the camera 3. The field of view of the camera is indicated by a dotted line starting from the camera 3. As shown in the figure, the active area 9 expands as the distance from the camera 3 increases. The light enhancement layer 4, which is formed of only one foil here, has a cutout window 6 in the active area 9 of the camera 3. The light guide 5 has a hemispherical nanostructure 11 (not shown here) having a spherical cap with a diameter of about 100 nm to 900 nm. Further, the LCD panel 2 includes two polarizers (not shown here), and both polarizers do not affect or only weakly affect the infrared light passing through the polarizers. Additionally, the liquid crystal material of the LCD panel 2 does not affect or only weakly affects the infrared light passing through the LCD panel 2.

[0036] Figure 2 shows a perspective view of an apparatus according to another embodiment of the present invention. This apparatus, as in Figure 1, includes a display device 1' and a camera 3 operating in the infrared wavelength spectrum. The cover glass 7 is omitted in this figure for simplicity.

[0037] The difference between the embodiment shown in Figure 2 and the embodiment shown in Figure 1 with respect to the embodiment shown in Figure 1 is that the light enhancement layer in Figure 2 is formed as a laminated foil 4'. In Figure 2, only half of the nanostructured light guide 5 is shown in order to emphasize the cutout window 6 in the light enhancement layer, here the laminated foil 4'. Only the first foil 40 located closest to the light guide 5 has the cutout window 6 in the active area of the camera 3. However, it should be understood that there may be embodiments in which the cutout window is arranged in one or more further foils of the laminated foil or through the entire laminated foil 4'. Additionally, the LCD panel 2' includes two polarizers (not shown here), and one of the polarizers does not affect, or only weakly affects, the infrared light passing through the corresponding polarizer. Further, in the embodiment of Figure 2, the apparatus includes a processor (not shown here), and the processor is installed to apply a deconvolution algorithm to the captured image captured by the camera 3. Thereby, the image quality of the captured image affected by the LCD panel is improved, for example, taking into account the pixel alignment. This processor may be implemented as a separate device, integrated into a processor that also performs other tasks, implemented as other hardware, or implemented as an algorithm executed by a computing unit.

[0038] The light guide 5 includes nanostructures 11, which can be seen in more detail in an exemplary symbolic form in the enlarged view of the active area 9 of the camera 3 shown in Figure 3.

[0039] In FIG. 2, user 10 (here, the driver of the vehicle) is looking at display device 1′, which is seamlessly arranged behind the display device 1′ and is monitored by camera 3 that cannot be seen by the user when looking at the display device 1′. Thus, camera 3 is hidden in a place where user 10 cannot see it.

Explanation of Signs

[0040] 1 Display device 2 LCD panel 3 Camera 4 Light enhancement layer 4′ Laminated foil 5 Light guide 6 Cutout window 7 Cover glass 8 Edge light source 9 Active area

Claims

1. A display device (1); Camera (3) and An apparatus comprising: The display device (1), A cover glass (7), An LCD panel (2); Backlight unit and Equipped with said LCD panel (2) being arranged behind said cover glass (7), said backlight unit being arranged behind said LCD panel (2), said camera (3) being arranged behind said backlight unit with its field of view towards said backlight unit, said field of view of said camera (3) defining an active area (9) of said camera (3), The backlight unit comprises an edge light source (8), a light enhancement layer (4), and a light guide (5) disposed behind the light enhancement layer (4). In the apparatus, said camera (3) operating in the infrared wavelength spectrum; a light enhancing layer (4) with a cut-out window (6) in the active area (9) of the camera (3); The light guide (5) is a light guide (5) having a nanostructure. An apparatus comprising:

2. 2. The device according to claim 1, characterized in that the cover glass (7) is optically bonded to the LCD panel (2).

3. 3. The device according to claim 1 or 2, characterized in that the light enhancing layer (4) is formed by a laminate foil (10) comprising a louvre foil and / or a reflective polarising foil and / or a diffusing foil, at least one of the louvre foils and / or the reflective polarising foil and / or the diffusing foil being provided with the cut-out window (6).

4. Apparatus according to any one of claims 1 to 3, characterised in that the apparatus comprises a processor, the processor being arranged to apply post-processing algorithms to captured images taken by the camera (3).

5. 5. The device according to claim 4, characterized in that the size of the crop window (6) is smaller than the active area (9), the camera (3) detects image information of a first area within the crop window (6) and of a second area outside the crop window (6), and the detected image information from both areas is provided to the post-processing algorithm.

6. Device according to any one of the preceding claims, characterized in that the LCD panel (2) comprises two polarisers, at least one of which does not affect or only weakly affects infrared light passing through the corresponding polariser.

7. Device according to any one of the preceding claims, characterized in that the LCD panel (2) comprises two polarisers, both of which do not affect or only weakly affect the infrared light passing through them.

8. Device according to any one of the preceding claims, characterized in that the liquid crystal material of the LCD panel (2) does not affect, or only weakly affects, infrared light passing through the LCD panel (2).

9. A vehicle comprising a device according to any one of claims 1 to 8.

Images