Display integrated with camera

Abstract

The present application relates to an apparatus and a vehicle comprising such an apparatus. The device comprises a display device (1) and a camera (3). A display device (1) is provided with a cover glass (7), an LCD panel (2), and a backlight unit. The LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), and the camera (3) is arranged behind the backlight unit, the field of view of which is directed towards the backlight unit. The field of view of the camera (3) defines an effective area (9) of the camera (3). The backlight unit comprises an edge light source (8) and a light guide (5). The camera (3) operates in the infrared wavelength spectrum, and the light guide (5) is a light guide (5) having a nanostructure (12).

Description

Technical Field

[0001] The present invention relates to a display integrated with a camera and a vehicle integrated with such a display. Background Art

[0002] In view of the current development trend of display technology, more and more original equipment manufacturers (OEMs) in different industries hope to integrate a camera behind the display in their products. This approach can be found in consumer products such as smartphones, tablets, and laptops, and there is also an increasing interest in using it for automotive products currently.

[0003] In addition, in order to meet new safety regulations, driver monitoring systems are becoming mandatory, so camera systems are starting to be integrated into different devices on vehicles such as cars, trucks, buses, etc.

[0004] To solve this problem, OEMs and manufacturers in different industries have adopted various methods and hope to achieve seamless integration of the camera in their devices.

[0005] The technology for integrating a camera behind the display is mainly affected by display type technologies such as liquid crystal display (LCD) displays and light-emitting diode (LED) displays.

[0006] Smartphone manufacturers solve this problem by first creating a notch in the display and mounting the camera in the notch behind the cover glass of the display. However, such a camera mounted in the notch cannot be considered seamlessly integrated and cannot be considered a camera behind the display.

[0007] To achieve seamless integration of the camera behind the display, various options have been identified. For example, EP 4 102810A1 and US11 145 233B1 describe using a lower pixel density and / or smaller pixels for the effective area of the camera. Therefore, this method proposes using a lower pixel density and / or smaller pixels in the area where the camera captures images to create enough space for light to enter. For this method, different pixel arrangements are also possible because this can also affect the quality of the picture. However, this solution of using a lower pixel density and / or smaller pixels for the effective area of the camera will result in each platform or application requiring a specially designed LCD panel to meet the product requirements, which will in turn lead to an increase in design complexity and the cost of this solution.

[0008] US2020 / 0117034 A1 describes a camera installed in a through-hole behind an LCD monitor. The LCD monitor includes a cover glass, an optically clear adhesive (OCA) layer, a first polarizer, a first glass substrate, a color filter (CF) layer, a thin-film transistor (TFT) layer, a second glass substrate, a second polarizer, and a backlight module. The through-hole is provided through the first polarizer, the CF layer, the TFT layer, the second polarizer, and the backlight module. The through-hole forms an optical channel in the LCD, and ambient light from outside the electronic device passes through the cover glass and the optical channel and enters the camera. The camera is disposed wholly or partially within the through-hole. The first glass substrate and the second glass substrate are not hollowed out in the area corresponding to the optical channel. However, this method requires a great deal of effort to align such through-holes in almost all components of the LCD. In addition, drilling such through-holes in almost all components of the LCD may cause damage to each individual component of the LCD. Therefore, it is necessary to control the components from being damaged and to correctly align the through-holes. All of the above further results in higher costs.

[0009] WO 2021 / 258921 A1 describes an under-display camera device. The display includes a display screen and a camera. The display screen includes a cover plate at the top and a display module under the cover plate. The display module includes indium tin oxide (ITO) glass, a liquid crystal layer, and a backlight layer. The display screen is provided with a blind hole area formed on the liquid crystal layer above an opening penetrating the entire backlight layer. The camera is arranged directly below the blind hole, and the lens of the camera extends into the hole on the backlight layer to the blind hole area, forming a viewing angle through the blind hole. The display screen further includes an electric field circuit provided on the outermost ITO glass layer of the liquid crystal layer, and the electric field circuit drives the liquid crystal in the blind hole area of the display screen to deflect by changing the electric field to change the optical path of the light passing through the blind hole area to achieve focusing of the camera. The implementation of this method is complex and means higher costs. In addition, the complete hole penetrating the entire backlight layer will affect light guiding because light scattering is disturbed in the area of the hole, resulting in image capture problems.

[0010] In addition, due to the opacity of the LCD panel and the polarizer, the development of a device seamlessly integrated with a camera is challenging. The LCD panel consists of a liquid crystal layer sandwiched between two electrodes and a polarizer that is an optical filter for transmitting specific polarized light. Generally, a common LCD panel has two polarizers, one polarizer on each side. When light passes through the LCD panel and the polarizer, the light is partially absorbed and scattered, thereby reducing the amount of light reaching the camera sensor behind the display. This may lead to a decrease in image quality and a reduction in camera performance, so an increase in the amount of hardware compensation is required.

[0011] US2022 / 0179452 A1 describes an electronic device including a front camera disposed behind a front display. Such a device described in US2022 / 0179452 A1 includes a display having a front surface and a rear surface, the display including a plurality of pixel regions that emit light from the front surface of the display to display a display image and a plurality of apertures that transmit light from the front surface to the rear surface. The camera is disposed on one side of the rear surface of the display. The camera is configured to capture an image. The device further includes corrective optics disposed between the display and the camera, the corrective optics being configured to reduce image distortion caused by the display. Further described is that the corrective optics may include an optical filter (e.g., a phase shift mask), a bandpass filter, a lens including a zoom lens, and / or different and / or additional components. Further described is that various items of the corrective optics may be included in the camera and thus excluded from the corrective optics. In addition, a processor is coupled to the display and the camera, the processor being configured to apply digital filtering to at least a portion of the captured image to further reduce image distortion caused by the display. Accordingly, image distortion caused by the display is physically reduced by the corrective optics and digitally reduced by the processor. Thus, a complex arrangement is described in US2022 / 0179452 A1 that requires both physical and digital reduction of image distortion caused by the display. However, this method also requires a great deal of effort, resulting in high costs.

[0012] WO 2009 / 051673 A1 describes an image capture device operating in the visible spectrum, the image capture device being placed behind a switchable diffuser, or placed behind a light guide or in a hole or opening within the light guide behind a display. The switchable diffuser is arranged in front of the light guide. WO 2009 / 051673 A1 further describes that the light guide typically has scattering elements (also referred to as scattering dots) on the back side of the light guide plate to scatter the light that has been conducted from a backlight bulb to the light guide plate towards the front side of the LCD display assembly. Such scattering elements are also referred to in the prior art as dot structures, microdots, or microlenses and are disposed at the surface (typically the back side of the light guide) of the light guide as described in WO 2009 / 051673 A1 for guiding light towards the display panel. The amount of light generated is defined by the density of the dot structure, microdots, or microlens group. The higher the density, the more light exits from the desired region of the light guide. However, a smaller density is limited because a single microdot or microlens will not produce a uniform light output. While the dot structure is referred to as small printed white dots on the back side of the light guide, the microdots or microlenses are embossed or injection-molded microdots that are hemispherical in shape and have a spherical cap disposed on the surface of the back side or the front side of the light guide plate. The diameter of the spherical cap or rather the microdots or microlenses has a size of approximately several tens of micrometers.

[0013] WO 2009 / 051673 A1 further describes the importance of using a switchable diffuser to be able to capture sufficient light for an image capturing device, improve the uniformity of light passing through the LCD panel, and make the scattering element less visible from the front side of the LCD.

[0014] In addition, in order to be able to obtain a clear image, the switchable diffuser, LCD, and image capturing device must be accurately controlled so that the camera can capture an image. Therefore, WO 2009 / 051673 A1 proposes a control logic for the LCD panel, camera, and diffuser to allow sufficient light to be captured by the camera. Thus, the solution known from WO 2009 / 051673 A1 requires an increase in the amount of logic and hardware to perform all the mandatory steps in order to capture an image. In addition, considering the control logic required to switch the diffuser and LCD screen, it may be difficult to integrate this type of solution in low-end, less costly products. SUMMARY OF THE INVENTION

[0015] Accordingly, it is an object of the present invention to provide an improved display in which a camera is seamlessly integrated behind the display.

[0016] This object is achieved by a display integrated with a camera and a vehicle integrated with such a display according to the independent claims. The dependent claims include advantageous further developments and improvements of the principles of the present invention as described below.

[0017] According to a first aspect, a device comprises a display device and a camera. The display device comprises a cover glass, an LCD panel, and a backlight unit. The LCD panel is arranged behind the cover glass, the backlight unit is arranged behind the LCD panel, and the camera is arranged behind the backlight unit, the field of view of the camera being directed towards the backlight unit, more precisely towards the light guide of the backlight unit, which will be described further below. The cover glass is the part of the display closest to the user viewing the display. For example, being behind the cover glass means that the corresponding component behind the cover glass (in this case the LCD panel) is arranged on the side of the display glass opposite to the side of the display glass that the user is viewing when viewing the display. The field of view of the camera defines the effective area of the camera. It should be understood that if, for example, a wide-angle lens is used and / or the camera is placed further away from the backlight unit, the field of view of the camera, and thus the effective area, is larger. It should also be understood that, conversely, if, for example, a telephoto lens is used and / or the camera is moved closer to the backlight unit, the field of view of the camera, and thus the effective area, is smaller.

[0018] The backlight unit includes an edge light source and a light guide. The camera operates in the infrared wavelength spectrum. The advantage of using a camera that operates in the infrared wavelength spectrum is that the transmission of IR light is not affected or only weakly affected by the LCD panel and its polarizers. The term "operating in the infrared wavelength spectrum" means that the camera or more precisely the image capture element of the camera is sensitive at least to the infrared spectrum and captures at least an image in the infrared (IR) spectrum. It should be understood that the camera can operate only in the IR spectrum (i.e., be arranged as an IR camera) or, in addition to operating in the IR spectrum, can also operate in the visible light wavelength spectrum.

[0019] Additionally, the light guide is a light guide having a nanostructure. It has been found that a light guide having a nanostructure does not affect infrared light when compared to a light guide having a microstructure. Thus, a light guide having a nanostructure is suitable for use with a camera that operates in the IR wavelength spectrum without further processing. The surface of the light guide can be provided with such a nanostructure on the back side or the front side of the light guide. However, when applied to the back side surface, such a structure becomes less visible. The term "nanostructure" means, for example, a structure that is a spherical cap at the surface of the light guide, which is less than 1 μm in size in terms of its diameter or more precisely its maximum extent in one dimension. Preferably, the size of the structure is between 1 nm and 1 μm. The term "nanostructure" means, for example, a structure that is a spherical cap in the camera window at the surface of the light guide, which is less than 1 μm in size in terms of its diameter or more precisely its maximum extent in one dimension. Preferably, the size of the structure is between 1 nm and 1 μm. Particularly preferably, the size of the structure is between 100 nm and 990 nm. The form, size, and density (i.e., density per given area) of the elements can be selected individually according to the technical requirements of the light guide and the entire device (such as the display) used. For example, the form of such a nanostructure can resemble a pyramid, a cone, a frustum, and / or a cylinder rather than a spherical cap. It should be understood that the present invention is not limited to the above-described structural forms, and there are many additional forms that can be used for those structures.

[0020] The present invention presents simple solutions for seamlessly integrating a camera (e.g., a driver monitoring module) into a display without affecting the content being displayed. For example, this solution does not require a switchable diffuser and the additional logic for controlling it because the camera can capture an image only when the diffuser is set to the "clear" state and the LCD panel is set to the "transparent" state. This solution also does not require components passing through the LCD panel, through-holes in the entire backlight unit, and / or does not require deflecting liquid crystals in the blind hole area of the display screen driven by a change in the applied electric field to change the optical path of the light passing through the blind hole area and achieve focusing of the camera. Additionally, with this solution, it is not necessary to physically reduce image distortion caused by the display, for example, by using corrective optics.

[0021] Furthermore, since the proposed solution can use any commercially available LCD screen without additional processing, the solution can accommodate the camera at any position of the LCD panel. The seamless integration of the camera behind the display proposed not only enhances the user experience but also brings additional functions to the device.

[0022] According to an embodiment of the present application, the cover glass is optically bonded to the LCD panel. The optical bonding is clear, and it should be understood that the cover glass and the optical bonding material are transparent and do not affect the behavior of capturing an image. By optically bonding the cover glass to the LCD panel, the air gap between the cover glass and the LCD panel is eliminated, thus improving, for example, display performance, readability, touch accuracy, and impact resistance.

[0023] According to a preferred embodiment of the present application, the backlight unit further includes a light enhancement layer. The light enhancement layer includes a cutout window in the effective area of the camera. It should be understood that the light enhancement layer is arranged such that the light guide is arranged behind the light enhancement layer. The light enhancement layer may include one or more foils. It should be further understood that the cutout window preferably has the same size as the effective area. However, the cutout window may be smaller than the size of the effective area. This has the advantage that the cutout window has a smaller visual interference effect on the user (e.g., the driver) viewing the display. Alternatively, the cutout window may be larger than the size of the effective area. This has the advantage that the light enhancement layer does not have a negative impact on the camera image. The term "cutout window" in the light enhancement layer refers to an area in the light enhancement layer where an area defined by the effective area of the camera exists, and the light enhancement layer is modified in this area (e.g., there are holes in a part of the light enhancement layer), and the camera can view through the modified area without interference.

[0024] According to another embodiment of the present invention, the light enhancement layer is formed by a stack of foils, which include louver foil and / or reflective polarizing foil and / or diffusing foil, and at least one of these foils includes a cutout window. It should be understood that according to the characteristics of the corresponding foil, it is sufficient that only one of the foils in the stack of foils includes a cutout window. For example, some of the foils in such a stack of foils may have little or no effect on IR light. The advantage of using a stack of foils with different characteristics is that the optical performance of such a device can be enhanced according to the technical requirements of the device (e.g., the display).

[0025] According to another embodiment of the present application, the device includes a processor configured to apply a post - processing algorithm, preferably a de - convolution algorithm, to the captured image taken by the camera. In terms of the impact of the LCD panel on the captured image, the post - processing algorithm improves the quality of the captured image. It should be understood that the post - processing algorithm is preferably a de - convolution algorithm. However, there are additional post - capture image - processing methods to improve the image quality of the captured image, and these post - capture image - processing methods can also be advantageously used as such a post - processing algorithm.

[0026] According to another embodiment of the present application, the size of the cutout window is smaller than the active area. The camera detects the image information of a first region within the cutout window and a second region outside the cutout window. The image information detected from these two regions is provided to the post - processing algorithm. Such an embodiment advantageously provides information for calibrating the post - processing algorithm.

[0027] According to another embodiment of the present application, the LCD panel includes two polarizers, and at least one of the two polarizers does not affect or only weakly affects the IR light passing through the corresponding polarizer. In the case where the camera is behind the display, such an embodiment improves the light output in the IR spectrum. It should be understood that such an embodiment may include a special physical arrangement of the at least one polarizer such that at least a portion of the IR light can pass through the at least one polarizer. Preferably, the portion of the at least one polarizer in the line of sight of the camera or the entire at least one polarizer includes or is composed of a material that allows at least a portion of the IR light to pass through the at least one polarizer.

[0028] According to another embodiment of the present application, the LCD panel includes two polarizers, and neither of the two polarizers affects or only weakly affects the IR light passing through these polarizers. In the case where the camera is behind the display, such an embodiment further improves the light output in the IR spectrum. It should be understood that such an embodiment may include a special physical arrangement of the two polarizers such that at least a portion of the IR light can pass through the two polarizers. Preferably, the portion of each of the two polarizers in the line of sight of the camera or the two polarizers as a whole includes or is composed of a material that allows at least a portion of the IR light to pass through the two polarizers.

[0029] Preferably, the liquid crystal material of the LCD panel does not affect or only weakly affects the IR light passing through the LCD panel. Similar to what was described above regarding the polarizer, in the case where the camera is behind the display, such an embodiment further improves the light output in the IR spectrum.

[0030] According to a second aspect, a vehicle includes the device according to at least one of the foregoing embodiments.

[0031] The use of such a display with a camera seamlessly integrated behind the display is also described. Additionally, such a camera seamlessly integrated behind the display can be incorporated into products that follow the user and provide an interactive experience based on the orientation of the user's eyes, especially for entertainment activities. For example, the camera can be introduced to the back of a smart TV display, and through specific and compatible content, the user can experience in real time through their digital twin, such as playing different games or participating in fitness classes. Another example is a device with a camera seamlessly integrated behind the display, which further includes a motion sensor input device, enabling functions such as skeleton tracking, hand interaction, and voice recognition. Such a motion sensor can track the movement of a person in front of the device and create a link between the movement and the "digital twin" shown on the display. To achieve this, such a motion sensor input device is equipped with an RGB camera, an infrared projector, and a detector that can perform real-time gesture recognition. By placing the camera solution according to the present invention behind the monitor or TV display, a better experience can be provided for the user, thus providing an integrated seamless solution without the need for a second device.

[0032] Further features of the present invention will become apparent from the following description and the appended claims in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a device according to a first embodiment of the present invention is shown in a cross-sectional view.

[0034] Figure 2 A schematic diagram of a device according to a second embodiment of the present invention is shown in a cross-sectional view.

[0035] Figure 3 A schematic diagram of a device according to a third embodiment of the present invention is shown in a perspective view, and

[0036] Figure 4 is shown in an enlarged view of an optical waveguide according to Figure 3 an optical waveguide schematic diagram. DETAILED DESCRIPTION

[0037] To better understand the principles of the present invention, embodiments of the present invention will be explained in more detail below with reference to the drawings. In the drawings, like reference numerals are used for the same or equivalent elements, and need not be described again for each drawing. It should be understood that the present invention is not limited to the embodiments shown, and the described features can also be combined or modified without departing from the scope of the present invention defined in the appended claims.

[0038] Figure 1A schematic diagram of a device according to a first embodiment of the present invention is shown in a sectional view. The device includes a display device 1 and a camera 3 operating in the infrared wavelength spectrum. The display device 1 includes a cover glass 7, an LCD panel 2, and a backlight unit. The cover glass 7 is optically bonded to the LCD panel 2, which is arranged behind the cover glass 7. The backlight unit includes a light source 8 provided at the edge of the display device 1 and an optical waveguide 5. The backlight unit, particularly the optical waveguide 5, is arranged behind the LCD panel 2. The light source 8 is arranged at the edge of the optical waveguide 5. The light emitted by the light source 8 is coupled into the optical waveguide at the edge of the optical waveguide 5. The camera 3 is arranged behind the backlight unit, and the field of view of the camera is directed towards the backlight unit, more precisely towards the optical waveguide 5. The field of view of the camera 3 defines the effective area 9 of the camera 3. The field of view of the camera is represented by a dashed line starting from the camera 3. As can be seen, the effective area 9 expands as the distance from the camera 3 increases. The optical waveguide 5 includes microstructures (not shown here). The optical waveguide 5 is an optical waveguide having nanostructures 11 (not shown here), which are hemispherical and have a spherical cap with a diameter of approximately 100 nm to 900 nm. In addition, the LCD panel 2 includes two polarizers (not shown here), and neither of these two polarizers affects or only weakly affects the infrared light passing through the polarizer. Additionally, the liquid crystal material of the LCD panel 2 does not affect or only weakly affects the infrared light passing through the LCD panel 2.

[0039] Figure 2 A schematic diagram of a device according to a second embodiment of the present invention is shown in a sectional view. Figure 2 The device shown is different from Figure 1 the device shown in that the backlight unit of the display 1' further includes a light enhancement layer 4, which is arranged in front of the optical waveguide 5. The light enhancement layer 4 (formed here only by a foil) includes a cutout window 6 in the effective area 9 of the camera 3.

[0040] Figure 3 A schematic diagram of a device according to a third embodiment of the present invention is shown in a perspective view. The device includes a display device 1” and a camera 3 operating in the infrared wavelength spectrum as Figure 2 shown. For simplicity, the cover glass 7 is omitted in this figure.

[0041] Figure 3 The embodiment shown is different from Figure 2 the embodiment shown in that Figure 3 the light enhancement layer in Figure 3Only half of the light guide 5 with nanostructures is shown to highlight the cutout window 6 in the light enhancement layer (here the foil stack 4'). Only the first foil 40 arranged closest to the light guide 5 includes the cutout window 6 in the effective area of the camera 3. However, it should be understood that there may be embodiments where the cutout window is arranged in one or more additional foils in the foil stack or the cutout window is arranged throughout the entire foil stack 4'. Additionally, the LCD panel 2' includes two polarizers (not shown here), and one of the polarizers does not affect or only weakly affects the infrared light passing through the corresponding polarizer. Furthermore, in Figure 3 the embodiment, the device includes a processor (not shown here), which is installed to apply a deconvolution algorithm to the captured images taken by the camera. Thereby, for example, considering pixel alignment, the image quality of the captured images that would be affected by the LCD panel is enhanced. The processor can be implemented as a separate device, can be integrated in a processor that also performs other tasks, can be implemented as other hardware, or can also be implemented as an algorithm executed by a computing unit.

[0042] The light guide 5 with nanostructures 11 can be seen in more detail in an enlarged view of the effective area 9 of the camera 3 shown in Figure 4 exemplary symbol form.

[0043] In Figure 3 the user 10 (here the driver in a vehicle) is watching the display device 1" and is monitored by the camera 3, which is seamlessly arranged behind the display device 1" and is not visible to the user when the user is watching the display device 1". Thus, the camera 3 is hidden from the view of the user 10.

Claims

1. A device comprising: - a display device (1), and - Camera (3), The display device (1) comprises: - a cover glass (7), - LCD panel (2), and - backlight unit, - wherein the LCD panel (2) is arranged behind the cover glass (7), the backlight unit is arranged behind the LCD panel (2), the camera (3) is arranged behind the backlight unit, the field of view of the camera is directed toward the backlight unit, and the field of view of the camera (3) defines an effective area (9) of the camera (3), and - wherein the backlight unit comprises an edge light source (8) and a light guide (5), It is characterized in that - the camera (3) operates in the infrared wavelength spectrum, and The light guide (5) is a light guide (5) with a nanostructure.

2. The device according to claim 1, characterized in that The cover glass (7) is optically bonded to the LCD panel (2).

3. The device according to claim 1 or 2, characterized in that The backlight unit further comprises a light enhancement layer (4), which comprises a cutout window (6) in an active area (9) of the camera (3).

4. The device according to claim 3, characterized in that The light enhancement layer (4) is formed by a stack (10) of foils including louver foils and / or reflective polarizing foils and / or diffusing foils, at least one of the foils comprising the cut-out window (6).

5. The device according to any one of the preceding claims, characterized in that The device comprises a processor arranged to apply a post-processing algorithm to a captured image taken by the camera (3).

6. Device according to claim 5 when dependent on one of claims 3 or 4, characterized in that The size of the incision window (6) is smaller than the effective area (9), and the camera (3) detects image information of a first area within the incision window (6) and a second area outside the incision window (6), and provides the image information detected from these two areas to the post-processing algorithm.

7. The device according to any one of the preceding claims, characterized in that The LCD panel (2) comprises two polarizing plates, at least one of which does not affect or only weakly affects infrared light passing through the corresponding polarizing plate.

8. The device according to any one of the preceding claims, characterized in that The LCD panel (2) includes two polarizing plates, both of which do not affect or only weakly affect infrared light passing through the polarizing plates.

9. The device according to any one of the preceding claims, characterized in that The liquid crystal material of the LCD panel (2) does not affect or only weakly affects the infrared light passing through the LCD panel (2).

10. A vehicle comprising a device according to any one of the preceding claims.

Images