Display integrated with camera

Abstract

The present application relates to an apparatus and a vehicle comprising such an apparatus. The device comprises a display device (1) and a camera (3). A display device (1) is provided with a cover glass (7), an LCD panel (2), and a backlight unit. The LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), and the camera (3) is arranged behind the backlight unit, the field of view of which is directed towards the backlight unit. The field of view of the camera (3) defines an effective area (9) of the camera (3). The backlight unit comprises an edge light source (8), a light enhancement layer (4) and a light guide (5) arranged behind the light enhancement layer (4). The camera (3) operates in the infrared wavelength spectrum, the light enhancement layer (4) comprises a cut-out window (6) in an active area (9) of the camera (3), and the light guide (5) is a light guide (5) having a nanostructure.

Description

Technical Field

[0001] The present invention relates to a display integrated with a camera and a vehicle integrated with such a display. Background Art

[0002] Given the current trends in display technology, more and more original equipment manufacturers (OEMs) in different industries want to integrate a camera behind the display in their products. This approach can be found in consumer products such as smartphones, tablets, and laptops, and there is also an increasing interest in using it for automotive products currently.

[0003] In addition, in order to meet new safety regulations, driver monitoring systems are becoming mandatory, and thus camera systems are starting to be integrated into different devices on vehicles such as cars, trucks, buses, etc.

[0004] To address this issue, OEMs and manufacturers in different industries have adopted various methods in the hope of achieving seamless integration of the camera in their devices.

[0005] The technology for integrating a camera behind the display is mainly influenced by the type of display technology, such as liquid crystal display (LCD) monitors and light emitting diode (LED) monitors.

[0006] Smartphone manufacturers solve this problem by first creating a notch in the display and mounting the camera in the notch behind the cover glass of the display. However, such a camera mounted in the notch cannot be considered seamlessly integrated and cannot be considered a camera behind the display.

[0007] To achieve seamless integration of the camera behind the display, various options have been identified. For example, EP 4 102810A1 and US11 145 233B1 describe using a lower pixel density and / or smaller pixels for the effective area of the camera. Thus, this approach proposes using a lower pixel density and / or smaller pixels in the area where the camera captures an image to create enough space for light to enter. For this method, different pixel arrangements are also possible since this can also affect the quality of the picture. However, this solution of using a lower pixel density and / or smaller pixels for the effective area of the camera will result in each platform or application requiring a specially designed LCD panel to meet the product requirements, which in turn will lead to an increase in design complexity and the cost of this solution.

[0008] US2020 / 0117034 A1 describes a camera installed in a through-hole behind an LCD monitor. The LCD monitor includes a cover glass, an optically clear adhesive (OCA) layer, a first polarizer, a first glass substrate, a color filter (CF) layer, a thin film transistor (TFT) layer, a second glass substrate, a second polarizer, and a backlight module. The through-hole is provided through the first polarizer, the CF layer, the TFT layer, the second polarizer, and the backlight module. The through-hole forms an optical channel in the LCD, and ambient light from outside the electronic device passes through the cover glass and the optical channel and enters the camera. The camera is disposed wholly or partially within the through-hole. The first glass substrate and the second glass substrate are not hollowed out in the area corresponding to the optical channel. However, this method requires a great deal of effort to align such through-holes in almost all components of the LCD. In addition, drilling such through-holes in almost all components of the LCD may cause damage to each individual component of the LCD. Therefore, it is necessary to control the components from being damaged and to correctly align the through-holes. All of the above-mentioned things further result in higher costs.

[0009] WO 2021 / 258921 A1 describes an in-display camera device. The display includes a display screen and a camera. The display screen includes a cover plate at the top and a display module under the cover plate. The display module includes indium tin oxide (ITO) glass, a liquid crystal layer, and a backlight layer. The display screen is provided with a blind hole area formed on the liquid crystal layer above an opening penetrating the entire backlight layer. The camera is arranged directly below the blind hole, and the lens of the camera extends into the hole on the backlight layer to the blind hole area to form a viewing angle through the blind hole. The display screen further includes an electric field circuit provided on the outermost ITO glass layer of the liquid crystal layer, and the electric field circuit drives the liquid crystal in the blind hole area of the display screen to deflect by changing the electric field to change the optical path of the light passing through the blind hole area to achieve focusing of the camera. The implementation of this method is complex and means higher costs. In addition, the complete hole penetrating the entire backlight layer will affect light guiding because the scattering of light is disturbed in the area of the hole, resulting in image capture problems.

[0010] In addition, due to the opacity of the LCD panel and the polarizer, the development of a device seamlessly integrated with a camera is challenging. The LCD panel consists of a liquid crystal layer sandwiched between two electrodes and a polarizer that is an optical filter for transmitting specific polarized light. Generally, a common LCD panel has two polarizers, one polarizer on each side. When light passes through the LCD panel and the polarizer, the light is partially absorbed and scattered, thus reducing the amount of light reaching the camera sensor behind the display. This may lead to a decrease in image quality and a reduction in camera performance, so an increase in the amount of hardware compensation is required.

[0011] US2022 / 0179452 A1 describes an electronic device including a front camera disposed behind a front display. Such a device described in US2022 / 0179452 A1 includes a display having a front surface and a rear surface, the display including a plurality of pixel regions that emit light from the front surface of the display to display a display image and a plurality of apertures that transmit light from the front surface to the rear surface. The camera is disposed on one side of the rear surface of the display. The camera is configured to capture an image. The device further includes corrective optics disposed between the display and the camera, the corrective optics being configured to reduce image distortion caused by the display. Further described is that the corrective optics may include an optical filter (e.g., a phase shift mask), a bandpass filter, a lens including a zoom lens, and / or different and / or additional components. Further described is that various items of the corrective optics may be included in the camera and thus excluded from the corrective optics. In addition, a processor is coupled to the display and the camera, the processor being configured to apply digital filtering to at least a portion of the captured image to further reduce image distortion caused by the display. Accordingly, image distortion caused by the display is physically reduced by the corrective optics and digitally reduced by the processor. Thus, a complex arrangement is described in US2022 / 0179452 A1 that requires both physical and digital reduction of image distortion caused by the display. However, this approach also requires a great deal of effort, resulting in high cost.

[0012] WO 2009 / 051673 A1 describes an image capture device operating in the visible spectrum, the image capture device being placed behind a switchable diffuser, or within a hole or opening in a light guide disposed behind a light guide or behind a display. The switchable diffuser is arranged in front of the light guide. WO 2009 / 051673 A1 further describes that the light guide typically has scattering elements (also referred to as scattering dots) on the back side of the light guide plate to scatter light that has been conducted from a backlight bulb to the light guide plate towards the front side of the LCD display assembly. Such scattering elements are also known in the prior art as dot structures, microdots, or microlenses and are disposed at the surface (typically the back side of the light guide) of the light guide as described in WO 2009 / 051673 A1 for guiding light towards the display panel. The amount of light generated is defined by the density of the dot structure, microdots, or microlens group. The higher the density, the more light exits from the desired region of the light guide. However, a smaller density is limited because a single microdot or microlens will not produce a uniform light output. While the dot structure is referred to as small printed white dots on the back side of the light guide, the microdots or microlenses are embossed or injection molded microdots that are hemispherical in shape and have a spherical cap disposed on the surface of the back side or the front side of the light guide plate. The diameter of the spherical cap or rather the microdots or microlenses has a size of approximately several tens of micrometers.

[0013] WO 2009 / 051673 A1 further describes the importance of using a switchable diffuser to be able to capture sufficient light for an image capturing device, improve the uniformity of light passing through the LCD panel, and make the scattering element less visible from the front side of the LCD.

[0014] In addition, in order to be able to obtain a clear image, the switchable diffuser, LCD, and image capturing device must be accurately controlled so that the camera can capture an image. Therefore, WO 2009 / 051673 A1 proposes a control logic for the LCD panel, camera, and diffuser to allow sufficient light to be captured by the camera. Thus, the solution known from WO 2009 / 051673 A1 requires an increase in the amount of logic and hardware to perform all the mandatory steps in order to capture an image. Moreover, considering the control logic required to switch the diffuser and LCD screen, it may be difficult to integrate this type of solution in low-end, less costly products. SUMMARY OF THE INVENTION

[0015] Accordingly, it is an object of the present invention to provide an improved display in which a camera is seamlessly integrated behind the display.

[0016] This object is achieved by a display integrated with a camera and a vehicle integrated with such a display according to the independent claims. The dependent claims include advantageous further developments and improvements of the principles of the present invention as described below.

[0017] According to a first aspect, an apparatus includes a display device and a camera. The display device includes a cover glass, an LCD panel, and a backlight unit. The LCD panel is arranged behind the cover glass, the backlight unit is arranged behind the LCD panel, and the camera is arranged behind the backlight unit, and the field of view of the camera is directed towards the backlight unit, more precisely towards the light guide of the backlight unit, which will be further described below. The cover glass is the part of the display closest to the user viewing the display. For example, being behind the cover glass means that the corresponding component behind the cover glass (in this case the LCD panel) is arranged on the side of the display glass opposite to the side of the display glass that the user is viewing when viewing the display. The field of view of the camera defines the effective area of the camera. It should be understood that if, for example, a wide-angle lens is used and / or the camera is placed further away from the backlight unit, the field of view of the camera, and thus the effective area, is larger. It should also be understood that, conversely, if, for example, a telephoto lens is used and / or the camera is moved closer to the backlight unit, the field of view of the camera, and thus the effective area, is smaller.

[0018] The backlight unit includes an edge light source, a light enhancement layer, and a light guide. The light guide is arranged behind the light enhancement layer. The camera operates in the infrared wavelength spectrum. The light enhancement layer may include one or more foils. The advantage of using a camera that operates in the infrared wavelength spectrum is that IR light transmission is not or only weakly affected by the LCD panel and its polarizer. The term "operating in the infrared wavelength spectrum" means that the camera or more precisely the image capture element of the camera is at least sensitive to the infrared spectrum and captures at least images in the infrared (IR) spectrum. It should be understood that the camera may operate only in the IR spectrum (i.e., be arranged as an IR camera), or in addition to operating in the IR spectrum, it may also operate in the visible light wavelength spectrum. The light enhancement layer includes a cutout window in the effective area of the camera. It should be understood that the cutout window preferably has the same size as the effective area. However, the cutout window may be smaller than the size of the effective area. This has the advantage that the cutout window has less visual interference effect on the user viewing the display (e.g., the driver). Alternatively, the cutout window may be larger than the size of the effective area. This has the advantage that the light enhancement layer does not have a negative impact on the camera image. The term "cutout window" in the light enhancement layer means that there is an area defined by the effective area of the camera in the light enhancement layer, and the light enhancement layer is modified in this area (e.g., there are holes in a part of the light enhancement layer), and the camera can view through the modified area in an undisturbed manner.

[0019] In addition, the light guide is a light guide with a nanostructure instead of a microstructure. It has been found that a light guide with a nanostructure does not affect infrared light when compared with a light guide with a microstructure. Therefore, a light guide with a nanostructure is suitable for use with a camera that operates in the IR wavelength spectrum without further processing. The surface of the light guide may be provided with such a nanostructure on the back side or the front side of the light guide. However, when applied to the back side surface, such a structure becomes less visible. The term "nanostructure" means, for example, a structure that is a spherical cap at the surface of the light guide, and in terms of its diameter or more precisely its maximum extent in one dimension, it is less than 1 μm in size. Preferably, the size of the structure is between 1 nm and 1 μm. Particularly preferably, the size of the structure is between 100 nm and 990 nm. The form, size, and density (i.e., the density per given area) of the element can be individually selected according to the technical requirements of the light guide and the entire device (e.g., the display) used. For example, the form of such a nanostructure may be similar to a pyramid, a cone, a frustum, and / or a cylinder instead of a spherical cap. It should be understood that the present invention is not limited to the above structural forms, and there are many other forms that can be used for those structures.

[0020] The present invention proposes a simple solution to seamlessly integrate a camera (e.g., a driver monitoring module) into a display without affecting the displayed content. For example, the solution does not require a switchable diffuser and the additional logic for controlling it, because the camera can capture an image only when the diffuser is set to the "clear" state and the LCD panel is set to the "transparent" state. The solution also does not require components passing through the LCD panel, through-holes in the entire backlight unit, and / or does not require deflecting liquid crystals in the blind hole area of the display screen by changing the applied electric field to change the optical path of the light passing through the blind hole area and achieve focusing of the camera. In addition, with this solution, it is not necessary to physically reduce image distortion caused by the display, for example, by using corrective optics.

[0021] Furthermore, since the proposed solution can use any commercially available LCD screen without additional processing, the solution can accommodate the camera at any position of the LCD panel. The proposed seamless integration of the camera behind the display not only enhances the user experience but also brings additional functions to the device.

[0022] According to an embodiment of the present application, a cover glass is optically bonded to the LCD panel. The optical bonding is clear, and it should be understood that the cover glass and the optical bonding material are transparent and do not affect the behavior of capturing an image. By optically bonding the cover glass to the LCD panel, the air gap between the cover glass and the LCD panel is eliminated, thus improving, for example, display performance, readability, touch accuracy, and impact resistance.

[0023] According to another embodiment of the present invention, a light enhancement layer is formed by a stack of foils, which includes louver foils and / or reflective polarizing foils and / or diffuser foils, and at least one of these foils includes a cutout window. It should be understood that according to the characteristics of the corresponding foil, it is sufficient that only one of the foils in the foil stack includes a cutout window. For example, some of the foils in such a foil stack may have little or no effect on IR light. The advantage of using a foil stack of foils with different characteristics is that the optical performance of such a device can be enhanced according to the technical requirements of the device (e.g., a display).

[0024] According to another embodiment of the present application, the device includes a processor that is configured to apply a post-processing algorithm, preferably a deconvolution algorithm, to the captured image captured by the camera. In terms of the impact of the LCD panel on the captured image, the post-processing algorithm improves the quality of the captured image. It should be understood that the post-processing algorithm is preferably a deconvolution algorithm. However, there are other post-capture image processing methods to improve the image quality of the captured image, and these post-capture image processing methods can also be advantageously used as such a post-processing algorithm.

[0025] According to another embodiment of the present application, the size of the cutout window is smaller than the effective area. The camera detects the image information of the first area within the cutout window and the second area outside the cutout window. The image information detected from these two areas is provided to the post-processing algorithm. Such an embodiment advantageously provides information for calibrating the post-processing algorithm.

[0026] According to another embodiment of the present application, the LCD panel includes two polarizers, and at least one of the two polarizers does not affect or only weakly affects the IR light passing through the corresponding polarizer. In the case where the camera is behind the display, such an embodiment improves the light output in the IR spectrum. It should be understood that such an embodiment may include a special physical arrangement of the at least one polarizer such that at least a portion of the IR light can pass through the at least one polarizer. Preferably, the portion of the at least one polarizer in the line of sight of the camera or the entire at least one polarizer includes or is composed of a material that allows at least a portion of the IR light to pass through the at least one polarizer.

[0027] According to another embodiment of the present application, the LCD panel includes two polarizers, and neither of the two polarizers affects or only weakly affects the IR light passing through these polarizers. In the case where the camera is behind the display, such an embodiment further improves the light output in the IR spectrum. It should be understood that such an embodiment may include a special physical arrangement of the two polarizers such that at least a portion of the IR light can pass through the two polarizers. Preferably, the portion of each of the two polarizers in the line of sight of the camera or the two polarizers as a whole includes or is composed of a material that allows at least a portion of the IR light to pass through the two polarizers.

[0028] Preferably, the liquid crystal material of the LCD panel does not affect or only weakly affects the IR light passing through the LCD panel. Similar to what was described above regarding the polarizer, in the case where the camera is behind the display, such an embodiment further improves the light output in the IR spectrum.

[0029] According to a second aspect, a vehicle includes the device according to at least one of the foregoing embodiments.

[0030] Also described is the use of such a display with a camera seamlessly integrated behind the display. Additionally, such a camera seamlessly integrated behind the display can be incorporated into products that follow the user and provide an interactive experience based on the orientation of the user's eyes, especially for entertainment activities. For example, the camera can be incorporated into the back of a smart TV display, and through specific and compatible content, the user can experience in real time through their digital twin, such as playing different games or participating in exercise classes. Another example is a device with a camera seamlessly integrated behind the display, which additionally includes a motion sensor input device, enabling functions such as skeleton tracking, hand interaction, and speech recognition. Such a motion sensor can track the movement of a person in front of the device and create a link between the movement and the "digital twin" shown on the display. To achieve this, such a motion sensor input device is equipped with an RGB camera, an infrared projector, and a detector that can perform real-time gesture recognition. By placing the camera solution according to the present invention behind the monitor or TV display, a better experience can be provided for the user, thus providing an integrated seamless solution without the need for a second device.

[0031] Further features of the present invention will become apparent from the following description and the appended claims in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a device according to an embodiment of the present invention is shown in a cross-sectional view,

[0033] Figure 2 A schematic diagram of a device according to another embodiment of the present invention is shown in a perspective view, and

[0034] Figure 3 Shown in an enlarged view of an optical waveguide according to Figure 2 is a schematic diagram of the optical waveguide. DETAILED DESCRIPTION

[0035] For a better understanding of the principles of the present invention, embodiments of the present invention will be explained in more detail below with reference to the drawings. In the drawings, like reference numerals are used for the same or equivalent elements, and it is not necessary to describe them again for each drawing. It should be understood that the present invention is not limited to the embodiments shown, and the described features can also be combined or modified without departing from the scope of the present invention defined in the appended claims.

[0036] Figure 1A schematic diagram of a device according to an embodiment of the present invention is shown in a sectional view. The device includes a display device 1 and a camera 3 operating in the infrared wavelength spectrum. The display device 1 includes a cover glass 7, an LCD panel 2, and a backlight unit. The cover glass 7 is optically bonded to the LCD panel 2, which is disposed behind the cover glass 7. The backlight unit includes a light source 8 disposed at the edge of the display device 1, a light enhancement layer 4, and a light guide 5, which is disposed behind the light enhancement layer 4, and the backlight unit is disposed entirely behind the LCD panel 2. The light source 8 is disposed at the edge of the light guide 5. The light emitted by the light source 8 is coupled into the light guide at the edge of the light guide 5. The camera 3 is disposed behind the backlight unit, and the field of view of the camera is directed towards the backlight unit, more precisely towards the light guide 5. The field of view of the camera 3 defines the effective area 9 of the camera 3. The field of view of the camera is represented by a dashed line starting from the camera 3. As can be seen, the effective area 9 expands as the distance from the camera 3 increases. The light enhancement layer 4 (formed here by only one foil) includes a cutout window 6 in the effective area 9 of the camera 3. The light guide 5 includes nanostructures 11 (not shown here), which are hemispherical and have a spherical cap with a diameter of approximately 100 nm to 900 nm. In addition, the LCD panel 2 includes two polarizers (not shown here), neither of which affects or only weakly affects the infrared light passing through the polarizer. Additionally, the liquid crystal material of the LCD panel 2 does not affect or only weakly affects the infrared light passing through the LCD panel 2.

[0037] Figure 2 A schematic diagram of a device according to another embodiment of the present invention is shown in a perspective view. The device includes a display device 1' and a camera 3 operating in the infrared wavelength spectrum as Figure 1 shown. For simplicity, the cover glass 7 is omitted in this figure.

[0038] Figure 2 The embodiment shown relative to Figure 1 the embodiment shown is different in that Figure 2 the light enhancement layer in Figure 2 is formed as a foil stack 4'. Only half of the light guide 5 with nanostructures is shown in Figure 2In an embodiment, the device includes a processor (not shown here), which is installed to apply a deconvolution algorithm to a captured image captured by a camera. Thus, for example, considering pixel alignment, the image quality of the captured image that would be affected by the LCD panel is enhanced. The processor may be implemented as a separate device, may be integrated in a processor that also performs other tasks, may be implemented as other hardware, or may also be implemented as an algorithm executed by a computing unit.

[0039] The light guide 5 including the nanostructure 11 can be seen in more detail in an enlarged view of the active area 9 of the camera 3 shown in Figure 3 exemplary symbols.

[0040] In Figure 2 , the user 10 (here the driver in a vehicle) is watching the display device 1' and is monitored by the camera 3, which is seamlessly arranged behind the display device 1' and is not visible to the user when the user is watching the display device 1'. Thus, the camera 3 is hidden from the view of the user 10.

Claims

1. A device comprising: - a display device (1), and - Camera (3), -The display device (1) comprises: - a cover glass (7), - LCD panel (2), and - backlight unit, - wherein the LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), the camera (3) is arranged behind the backlight unit, the field of view of the camera is directed toward the backlight unit, and the field of view of the camera (3) defines an effective area (9) of the camera (3), and - wherein the backlight unit comprises an edge light source (8), a light enhancement layer (4) and a light guide (5) arranged behind the light enhancement layer (4), It is characterized in that - the camera (3) operates in the infrared wavelength spectrum, - the light enhancement layer (4) comprises a cut-out window (6) in the active area (9) of the camera (3), and The light guide (5) is a light guide (5) with a nanostructure.

2. The device according to claim 1, characterized in that The cover glass (7) is optically bonded to the LCD panel (2).

3. The device according to claim 1 or 2, characterized in that The light enhancement layer (4) is formed by a stack (10) of foils including louver foils and / or reflective polarizing foils and / or diffusing foils, at least one of the foils comprising the cut-out window (6).

4. The device according to any one of the preceding claims, characterized in that The device comprises a processor arranged to apply a post-processing algorithm to a captured image taken by the camera (3).

5. The device according to claim 4, characterized in that The size of the incision window (6) is smaller than the effective area (9), and the camera (3) detects image information of a first area within the incision window (6) and a second area outside the incision window (6), and provides the image information detected from these two areas to the post-processing algorithm.

6. The device according to any one of the preceding claims, characterized in that The LCD panel (2) comprises two polarizing plates, at least one of which does not affect or only weakly affects infrared light passing through the corresponding polarizing plate.

7. The device according to any one of the preceding claims, characterized in that The LCD panel (2) includes two polarizing plates, both of which do not affect or only weakly affect infrared light passing through the polarizing plates.

8. The device according to any one of the preceding claims, characterized in that The liquid crystal material of the LCD panel (2) does not affect or only weakly affects the infrared light passing through the LCD panel (2).

9. A vehicle comprising a device according to any one of the preceding claims.

Images