Display integrated with camera

Abstract

The present application relates to an apparatus and a vehicle comprising such an apparatus. The device comprises a display device (1) and a camera (3). A display device (1) is provided with a cover glass (7), an LCD panel (2), and a backlight unit. The LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), and the camera (3) is arranged behind the backlight unit, the field of view of which is directed towards the backlight unit. The field of view of the camera (3) defines an effective area (9) of the camera (3). The backlight unit comprises an edge light source (8) and a light guide (5). The light guide (5) is a light guide (5) having a microstructure (12). And the camera (3) works in an infrared wavelength spectrum. The light guide (5) comprises a camera window (11) having a nanostructure (12) in an active area (9) of the camera (3).

Description

Technical Field

[0001] The present invention relates to a display integrated with a camera and a vehicle integrated with such a display. Background Art

[0002] In view of the current development trend of display technology, more and more original equipment manufacturers (OEMs) in different industries hope to integrate a camera behind the display in their products. This approach can be found in consumer products such as smartphones, tablets, and laptops, and there is also an increasing interest in using it for automotive products currently.

[0003] In addition, in order to meet new safety regulations, driver monitoring systems are becoming mandatory, so camera systems are starting to be integrated into different devices on vehicles such as cars, trucks, buses, etc.

[0004] To solve this problem, OEMs and manufacturers in different industries have adopted various methods, hoping to achieve seamless integration of the camera in their devices.

[0005] The technology for integrating a camera behind the display is mainly affected by display type technologies such as liquid crystal display (LCD) displays and light emitting diode (LED) displays.

[0006] Smartphone manufacturers solve this problem by first creating a notch in the display and mounting the camera in the notch behind the cover glass of the display. However, such a camera mounted in the notch cannot be considered seamlessly integrated and cannot be considered a camera behind the display.

[0007] To achieve seamless integration of the camera behind the display, various options have been identified. For example, EP 4 102810A1 and US11 145 233B1 describe using a lower pixel density and / or smaller pixels for the effective area of the camera. Therefore, this method proposes using a lower pixel density and / or smaller pixels in the area where the camera captures an image to create enough space for light to enter. For this method, different pixel arrangements are also possible because this can also affect the quality of the picture. However, this solution of using a lower pixel density and / or smaller pixels for the effective area of the camera will result in each platform or application requiring a specially designed LCD panel to meet the product requirements, which will in turn lead to an increase in design complexity and the cost of this solution.

[0008] US2020 / 0117034 A1 describes a camera installed in a through-hole behind an LCD monitor. The LCD monitor includes a cover glass, an optically clear adhesive (OCA) layer, a first polarizer, a first glass substrate, a color filter (CF) layer, a thin film transistor (TFT) layer, a second glass substrate, a second polarizer, and a backlight module. The through-hole is provided through the first polarizer, the CF layer, the TFT layer, the second polarizer, and the backlight module. The through-hole forms an optical channel in the LCD, and ambient light from outside the electronic device passes through the cover glass and the optical channel into the camera. The camera is disposed wholly or partially within the through-hole. The first glass substrate and the second glass substrate are not hollowed out in the area corresponding to the optical channel. However, this method requires a great deal of effort to align such through-holes in almost all components of the LCD. In addition, drilling such through-holes in almost all components of the LCD may cause damage to each individual component of the LCD. Therefore, it is necessary to control the components from being damaged and to correctly align the through-holes. All of the above-mentioned further results in higher costs.

[0009] WO 2021 / 258921 A1 describes an under-display camera device. The display includes a display screen and a camera. The display screen includes a cover plate at the top and a display module under the cover plate. The display module includes indium tin oxide (ITO) glass, a liquid crystal layer, and a backlight layer. The display screen is provided with a blind hole area formed on the liquid crystal layer above an opening penetrating the entire backlight layer. The camera is arranged directly below the blind hole, and the lens of the camera extends into the hole on the backlight layer to the blind hole area to form a viewing angle through the blind hole. The display screen further includes an electric field circuit provided on the outermost ITO glass layer of the liquid crystal layer, and the electric field circuit drives the liquid crystal in the blind hole area of the display screen to deflect by changing the electric field to change the optical path of the light passing through the blind hole area to achieve focusing of the camera. The implementation of this method is complex and means higher costs. In addition, the complete hole penetrating the entire backlight layer will affect light guiding because light scattering is disturbed in the area of the hole, resulting in image capture problems.

[0010] In addition, due to the opacity of the LCD panel and the polarizer, the development of devices seamlessly integrated with a camera is challenging. The LCD panel consists of a liquid crystal layer sandwiched between two electrodes and a polarizer that is an optical filter for transmitting specific polarized light. Generally, a common LCD panel has two polarizers, one polarizer on each side. When light passes through the LCD panel and the polarizer, the light is partially absorbed and scattered, thus reducing the amount of light reaching the camera sensor behind the display. This may lead to a decline in image quality and a reduction in camera performance, so an increase in the amount of hardware compensation is required.

[0011] US2022 / 0179452 A1 describes an electronic device including a front camera disposed behind a front display. Such a device described in US2022 / 0179452 A1 includes a display having a front surface and a rear surface, the display including a plurality of pixel regions that emit light from the front surface of the display to display a display image and a plurality of apertures that transmit light from the front surface to the rear surface. The camera is disposed on one side of the rear surface of the display. The camera is configured to capture an image. The device further includes corrective optics disposed between the display and the camera, the corrective optics being configured to reduce image distortion caused by the display. Further described is that the corrective optics may include an optical filter (e.g., a phase shift mask), a bandpass filter, a lens including a zoom lens, and / or different and / or additional components. Further described is that various items of the corrective optics may be included in the camera and thus excluded from the corrective optics. In addition, a processor is coupled to the display and the camera, the processor being configured to apply digital filtering to at least a portion of the captured image to further reduce image distortion caused by the display. Accordingly, image distortion caused by the display is physically reduced by the corrective optics and digitally reduced by the processor. Thus, a complex arrangement is described in US2022 / 0179452 A1 that requires both physical reduction and digital reduction of image distortion caused by the display. However, this approach also requires a great deal of effort, resulting in high costs.

[0012] WO 2009 / 051673 A1 describes an image capture device operating in the visible spectrum, the image capture device being placed behind a switchable diffuser, or in a hole or opening in a light guide behind a light guide or within a display. The switchable diffuser is arranged in front of the light guide. WO 2009 / 051673 A1 further describes that the light guide typically has scattering elements (also referred to as scattering points) on the back side of the light guide plate to scatter light that has been conducted from a backlight bulb to the light guide plate towards the front side of the LCD display assembly. Such scattering elements are also referred to in the prior art as dot structures, microdots, or microlenses, and are disposed at the surface (typically the back side of the light guide) of the light guide as described in WO 2009 / 051673 A1 for guiding light towards the display panel. The amount of light generated is defined by the density of the dot structure, microdot, or microlens group. The higher the density, the more light exits from the desired region of the light guide. However, a smaller density is limited because a single microdot or microlens will not produce a uniform light output. While the dot structure is referred to as a small printed white dot on the back side of the light guide, the microdot or microlens is an embossed or injection-molded microdot that is hemispherical in shape and has a spherical cap disposed on the surface of the back side or the front side of the light guide plate. The diameter of the spherical cap or rather the microdot or microlens has a size of approximately several tens of micrometers.

[0013] WO 2009 / 051673 A1 further describes the importance of using a switchable diffuser to be able to capture sufficient light for an image capture device, improve the uniformity of light passing through the LCD panel, and make the scattering element less visible from the front side of the LCD.

[0014] In addition, in order to be able to obtain a clear image, the switchable diffuser, the LCD, and the image capture device must be accurately controlled so that the camera can capture an image. Therefore, WO 2009 / 051673 A1 proposes a control logic for the LCD panel, the camera, and the diffuser to allow sufficient light to be captured by the camera. Thus, the solution known from WO 2009 / 051673 A1 requires an increase in the amount of logic and hardware to perform all the mandatory steps in order to capture an image. In addition, considering the control logic required to switch the diffuser and the LCD screen, it may be difficult to integrate this type of solution in low-end, less costly products. SUMMARY OF THE INVENTION

[0015] Accordingly, it is an object of the present invention to provide an improved display in which a camera is seamlessly integrated behind the display.

[0016] This object is achieved by a display integrated with a camera and a vehicle integrated with such a display according to the independent claims. The dependent claims include advantageous further developments and improvements of the principles of the present invention as described below.

[0017] According to a first aspect, a device comprises a display device and a camera. The display device comprises a cover glass, an LCD panel, and a backlight unit. The LCD panel is arranged behind the cover glass, the backlight unit is arranged behind the LCD panel, and the camera is arranged behind the backlight unit, the field of view of the camera being directed towards the backlight unit, more precisely towards the light guide of the backlight unit, which will be described further below. The cover glass is the part of the display closest to the user viewing the display. For example, being behind the cover glass means that the corresponding component behind the cover glass (in this case the LCD panel) is arranged on the side of the display glass opposite to the side of the display glass that the user is viewing when viewing the display. The field of view of the camera defines the effective area of the camera. It should be understood that if, for example, a wide-angle lens is used and / or the camera is placed further away from the backlight unit, the field of view of the camera, and thus the effective area, will be larger. It should also be understood that, conversely, if, for example, a telephoto lens is used and / or the camera is moved closer to the backlight unit, the field of view of the camera, and thus the effective area, will be smaller.

[0018] The backlight unit includes an edge light source and a light guide. The camera operates in the infrared wavelength spectrum. The advantage of using a camera that operates in the infrared wavelength spectrum is that IR light transmission is not affected or only weakly affected by the LCD panel and its polarizer. The term "operating in the infrared wavelength spectrum" means that the camera or more precisely the image capture element of the camera is at least sensitive to the infrared spectrum and captures at least an image in the infrared (IR) spectrum. It should be understood that the camera can operate only in the IR spectrum (i.e., be arranged as an IR camera), or in addition to operating in the IR spectrum, it can also operate in the visible light wavelength spectrum.

[0019] In addition, the light guide is a light guide with a microstructure, which includes a camera window with a nanostructure in the effective area of the camera. The camera window defines an area in which the light guide is composed of a light guide with a nanostructure instead of a microstructure. It has been found that when compared with a light guide with a microstructure, the nanostructure used in the light guide does not affect infrared light. Therefore, a light guide including a camera window with a nanostructure is suitable for use with a camera that operates in the IR wavelength spectrum without further processing. It should be understood that the light guide shares two different areas in the effective area of the camera, namely a large microstructure area and a smaller nanostructure area. The claimed device benefits from the advantages of the nanostructure that transmits IR light and the microstructure that enables good scattering of light from the light source towards the LCD panel. Generally, the surface of the light guide is provided with a microstructure on the back side or the front side of the light guide. However, when applied to the back side surface, this structure becomes less visible. It should be understood that the camera window with a nanostructure is preferably provided on the same side of the light guide as the microstructure. The term "microstructure" refers to a structure that is, for example, a spherical cap at the surface of the light guide, and in terms of its diameter or more precisely its maximum extent in one dimension, has a size between 10 μm and 100 μm. In contrast, the term "nanostructure" refers to a structure that is, for example, a spherical cap in the camera window at the surface of the light guide, and in terms of its diameter or more precisely its maximum extent in one dimension, has a size less than 1 μm. Preferably, the size of the structure is between 1 nm and 1 μm. Particularly preferably, the size of the structure is between 100 nm and 990 nm. The form, size, and density of the elements (i.e., the density per given area) can be selected individually according to the technical requirements of the light guide and the entire device (such as the display) used. For example, the form of such a nanostructure can be similar to a pyramid, a cone, a frustum, and / or a cylinder instead of a spherical cap. It should be understood that the present invention is not limited to the above structural forms, and there are many other forms that can be used for those structures.

[0020] The present invention presents a simple solution to seamlessly integrate a camera (e.g., a driver monitoring module) into a display without affecting the displayed content. For example, the solution does not require a switchable diffuser and the additional logic for controlling it, as the camera can capture an image only when the diffuser is set to the "clear" state and the LCD panel is set to the "transparent" state. The solution also does not require components passing through the LCD panel, through-holes in the entire backlight unit, and / or does not require deflecting liquid crystals in the blind hole area of the display screen driven by a change in the applied electric field to change the optical path of the light passing through the blind hole area and achieve focusing of the camera. In addition, with this solution, it is not necessary to physically reduce image distortion caused by the display, for example, by using corrective optics.

[0021] Furthermore, since the proposed solution can use any commercially available LCD screen without additional processing, the solution can accommodate the camera at any position in the LCD panel. The proposed seamless integration of the camera behind the display not only enhances the user experience but also brings additional functionality to the device.

[0022] According to an embodiment of the present application, a cover glass is optically bonded to the LCD panel. The optical bonding is clear, and it should be understood that the cover glass and the optical bonding material are transparent and do not affect the behavior of capturing an image. By optically bonding the cover glass to the LCD panel, the air gap between the cover glass and the LCD panel is eliminated, thus improving, for example, display performance, readability, touch accuracy, and impact resistance.

[0023] According to a preferred embodiment of the present application, the backlight unit further includes a light enhancement layer. The light enhancement layer includes a cutout window in the effective area of the camera. It should be understood that the light enhancement layer is arranged such that a light guide is disposed behind the light enhancement layer. The light enhancement layer may include one or more foils. It should be further understood that the cutout window preferably has the same size as the effective area. However, the cutout window may be smaller than the size of the effective area. This has the advantage that the cutout window has less visual interference effect on the user (e.g., the driver) viewing the display. Alternatively, the cutout window may be larger than the size of the effective area. This has the advantage that the light enhancement layer does not have a negative impact on the camera image. The term "cutout window" in the light enhancement layer refers to an area in the light enhancement layer defined by the effective area of the camera, where the light enhancement layer is modified (e.g., there are holes in a part of the light enhancement layer), and the camera can view through the modified area in an undisturbed manner.

[0024] According to another embodiment of the present invention, the light enhancement layer is formed by a stack of foils, which include louver foils and / or reflective polarizing foils and / or diffusing foils, and at least one of these foils includes a cutout window. It should be understood that, depending on the characteristics of the corresponding foil, it is sufficient if only one of the foils in the stack of foils includes a cutout window. For example, several foils in such a stack of foils may have no or little effect on IR light. The advantage of using a stack of foils with different characteristics is that the optical performance of a device (such as a display) can be enhanced according to the technical requirements of the device.

[0025] According to another embodiment of the present application, the device includes a processor that is configured to apply a post-processing algorithm, preferably a deconvolution algorithm, to a captured image taken by a camera. In terms of the impact of the LCD panel on the captured image, the post-processing algorithm improves the quality of the captured image. It should be understood that the post-processing algorithm is preferably a deconvolution algorithm. However, there are additional post-capture image processing methods to improve the image quality of the captured image, and these post-capture image processing methods can also be advantageously used as such a post-processing algorithm.

[0026] According to another embodiment of the present application, the size of the cutout window is smaller than the effective area. The camera detects the image information of a first region within the cutout window and a second region outside the cutout window. The image information detected from these two regions is provided to the post-processing algorithm. This embodiment advantageously provides information for calibrating the post-processing algorithm.

[0027] According to another embodiment of the present application, the LCD panel includes two polarizers, and at least one of these two polarizers has no or only a weak effect on the IR light passing through the corresponding polarizer. In the case where the camera is behind the display, such an embodiment improves the light output in the IR spectrum. It should be understood that such an embodiment may include a special physical arrangement of the at least one polarizer such that at least a part of the IR light can pass through the at least one polarizer. Preferably, the part of the at least one polarizer in the line of sight of the camera or the entire at least one polarizer includes or is composed of a material that enables at least a part of the IR light to pass through the at least one polarizer.

[0028] According to another embodiment of the present application, the LCD panel includes two polarizers, neither of which affects or only weakly affects the IR light passing through these polarizers. In the case where the camera is behind the display, such an embodiment further improves the light output in the IR spectrum. It should be understood that such an embodiment may include a special physical arrangement of these two polarizers such that at least a portion of the IR light can pass through both polarizers. Preferably, the portion of each of the two polarizers in the line of sight of the camera or the two polarizers as a whole includes or is composed of a material that enables at least a portion of the IR light to pass through both polarizers.

[0029] Preferably, the liquid crystal material of the LCD panel does not affect or only weakly affects the IR light passing through the LCD panel. Similar to what was described above regarding the polarizers, in the case where the camera is behind the display, such an embodiment further improves the light output in the IR spectrum.

[0030] According to a second aspect, a vehicle includes the device according to at least one of the foregoing embodiments.

[0031] The use of such a display with a seamlessly integrated camera behind the display is also described. In addition, such a seamlessly integrated camera behind the display can be incorporated into a product that follows the user and provides an interactive experience based on the orientation of the user's eyes, especially for entertainment activities. For example, the camera can be incorporated into the back of a smart TV display, and through specific and compatible content, the user can experience it in real time through their digital twin, such as playing different games or participating in fitness classes. Another example is a device with a seamlessly integrated camera behind the display, which further includes a motion sensor input device, enabling functions such as skeleton tracking, hand interaction, and voice recognition. Such a motion sensor can track the movement of a person in front of the device and create a link between the movement and the "digital twin" shown on the display. To achieve this, such a motion sensor input device is equipped with an RGB camera, an infrared projector, and a detector that can perform real-time gesture recognition. By placing the camera solution according to the present invention behind the monitor or TV display, a better experience can be provided for the user, thus providing an integrated seamless solution without the need for a second device.

[0032] Further features of the present invention will become apparent from the following description and the appended claims in conjunction with the drawings. Description of the Drawings

[0033] Figure 1 A schematic diagram of the device according to a first embodiment of the present invention is shown in a cross-sectional view.

[0034] Figure 2A schematic diagram of a device according to a second embodiment of the present invention is shown in a sectional view.

[0035] Figure 3 A schematic diagram of a device according to a third embodiment of the present invention is shown in a perspective view, and

[0036] Figure 4 is shown in an enlarged view of an optical waveguide according to Figure 3 a schematic diagram of the optical waveguide. Detailed Description of the Invention

[0037] For a better understanding of the principles of the present invention, embodiments of the present invention will be explained in more detail below with reference to the accompanying drawings. In the drawings, like reference numerals are used for identical or equivalent elements and need not be described again for each drawing. It should be understood that the present invention is not limited to the embodiments shown and that the features described may also be combined or modified without departing from the scope of the present invention as defined in the appended claims.

[0038] Figure 1 A schematic diagram of a device according to a first embodiment of the present invention is shown in a sectional view. The device includes a display device 1 and a camera 3 operating in the infrared wavelength spectrum. The display device 1 includes a cover glass 7, an LCD panel 2, and a backlight unit. The cover glass 7 is optically bonded to the LCD panel 2, which is arranged behind the cover glass 7. The backlight unit includes a light source 8 provided at the edge of the display device 1 and an optical waveguide 5. The backlight unit, especially the optical waveguide 5, is arranged behind the LCD panel 2. The light source 8 is arranged at the edge of the optical waveguide 5. Light emitted by the light source 8 is coupled into the optical waveguide at the edge of the optical waveguide 5. The camera 3 is arranged behind the backlight unit, and the field of view of the camera points to the backlight unit, more precisely to the optical waveguide 5. The field of view of the camera 3 defines the effective area 9 of the camera 3. The field of view of the camera is represented by a dashed line starting from the camera 3. As can be seen, the effective area 9 expands as the distance from the camera 3 increases. The optical waveguide 5 includes microstructures (not shown here). The optical waveguide 5 further includes a camera window 11 with nanostructures 12 (not shown here) in the effective area 9 of the camera 3. These nanostructures are hemispherical and have a spherical cap with a diameter of approximately 100 nm to 900 nm. In addition, the LCD panel 2 includes two polarizers (not shown here), neither of which affects or only weakly affects the infrared light passing through the polarizer. Additionally, the liquid crystal material of the LCD panel 2 does not affect or only weakly affects the infrared light passing through the LCD panel 2.

[0039] Figure 2 A schematic diagram of a device according to a second embodiment of the present invention is shown in a sectional view. Figure 2 The device shown is the same as Figure 1The device shown is different in that the backlight unit of the display 1' further includes a light enhancement layer 4, which is arranged in front of the light guide 5. The light enhancement layer 4 (formed here by only one foil) includes a cutout window 6 in the effective area 9 of the camera 3.

[0040] Figure 3 A schematic diagram of a device according to a third embodiment of the present invention is shown in a perspective view. The device includes a display device 1'' and a camera 3 operating in the infrared wavelength spectrum as Figure 2 shown. For simplicity, the cover glass 7 is omitted in this figure.

[0041] Figure 3 The embodiment shown is different from Figure 2 the embodiment shown in that Figure 3 the light enhancement layer in Figure 3 is formed as a foil stack 4'. Only half of the light guide 5 and the camera window 11 with nanostructures are shown in Figure 3 to highlight the cutout window 6 in the light enhancement layer (here the foil stack 4'). Only the first foil 40 arranged closest to the light guide 5 includes the cutout window 6 in the effective area of the camera 3. However, it should be understood that there may be embodiments where the cutout window is arranged in one or more additional foils in the foil stack or the cutout window is arranged through the entire foil stack 4'. Additionally, the LCD panel 2' includes two polarizers (not shown here), and one of the polarizers does not affect or only weakly affects the infrared light passing through the corresponding polarizer. Furthermore, in

[0042] the embodiment of Figure 4 the device includes a processor (not shown here), which is installed to apply a deconvolution algorithm to the captured images taken by the camera. Thus, for example, considering pixel alignment, the image quality of the captured images that would be affected by the LCD panel is enhanced. The processor can be implemented as a separate device, can be integrated in a processor that also performs other tasks, can be implemented as other hardware, or can also be implemented as an algorithm executed by a computing unit.

[0042] The light guide 5 with microstructures and the camera window 11 with nanostructures 12 can be seen in more detail in an enlarged view of the effective area 9 of the camera 3 shown in Figure 4 in an exemplary symbolic form.

[0043] In Figure 3 a user 10 (here the driver in a vehicle) is viewing the display device 1'' and is monitored by the camera 3, which is seamlessly arranged behind the display device 1'' and is not visible to the user when the user is viewing the display device 1''. Thus, the camera 3 is hidden from the view of the user 10.

Claims

1. A device comprising: - a display device (1), and - Camera (3), -The display device (1) comprises: - a cover glass (7), - LCD panel (2), and - backlight unit, - wherein the LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), the camera (3) is arranged behind the backlight unit, the field of view of the camera is directed toward the backlight unit, and the field of view of the camera (3) defines an effective area (9) of the camera (3), - wherein the backlight unit comprises an edge light source (8) and a light guide (5), and - wherein the light guide (5) is a light guide (5) with a microstructure, It is characterized in that - the camera (3) operates in the infrared wavelength spectrum, and The light guide (5) comprises a camera window (11) with nanostructures (12) in the active area (9) of the camera (3).

2. The device according to claim 1, characterized in that The cover glass (7) is optically bonded to the LCD panel (2).

3. The device according to claim 1 or 2, characterized in that The backlight unit further comprises a light enhancement layer (4), which comprises a cutout window (6) in an active area (9) of the camera (3).

4. The device according to claim 3, characterized in that The light enhancement layer (4) is formed by a stack (10) of foils including louver foils and / or reflective polarizing foils and / or diffusing foils, at least one of the foils comprising the cut-out window (6).

5. The device according to any one of the preceding claims, characterized in that The device comprises a processor arranged to apply a post-processing algorithm to a captured image taken by the camera (3).

6. Device according to claim 5 when dependent on one of claims 3 or 4, characterized in that The size of the incision window (6) is smaller than the effective area (9), and the camera (3) detects image information of a first area within the incision window (6) and a second area outside the incision window (6), and provides the image information detected from these two areas to the post-processing algorithm.

7. The device according to any one of the preceding claims, characterized in that The LCD panel (2) comprises two polarizing plates, at least one of which does not affect or only weakly affects infrared light passing through the corresponding polarizing plate.

8. The device according to any one of the preceding claims, characterized in that The LCD panel (2) includes two polarizing plates, both of which do not affect or only weakly affect infrared light passing through the polarizing plates.

9. The device according to any one of the preceding claims, characterized in that The liquid crystal material of the LCD panel (2) does not affect or only weakly affects the infrared light passing through the LCD panel (2).

10. A vehicle comprising a device according to any one of the preceding claims.

Images