Camera module, image capture device and vehicle

Abstract

Camera module (1), comprising an image sensor (2) and a selection filter (3) arranged upstream of the image sensor (2) in the light beam path, wherein the image sensor (2) in turn comprises a pixel field (4) made up of photosensitive elements (5) and a color filter field (6) arranged upstream of the pixel field (4) made up of color filter elements (R, G, B) and infrared filter elements (IR), and wherein individual color filter elements (R, G, B) are assigned to individual photosensitive elements (5) to form color pixels and individual infrared filter elements (IR) are assigned to individual photosensitive elements (5) to form infrared pixels, wherein - the selection filter (3) is designed as a bandpass filter with a transmission spectrum corresponding exclusively to visible light, and - the selection filter (3) has recesses (7) extending through the entire thickness of the selection filter in the direction of the light beam path, which recesses coincide with the respective arrangement positions of the infrared pixels in the plane of extension of the image sensor (2), characterized in that an infrared selection filter element (IR-sel) is embedded in each of the recesses (7) of the selection filter (3), which has a transmissivity for light exclusively in the infrared spectrum.

Description

[0001] The invention relates to a camera module according to the type defined in more detail in the preamble of claim 1, an image capture device with such a camera module and a vehicle with such an image capture device.

[0002] The integration of cameras into vehicles allows for the provision of a wide variety of driver assistance functions. Cameras make it possible to observe the vehicle's external surroundings as well as to record the vehicle's interior. External cameras, for example, make it possible to detect and identify static and dynamic objects in the environment, and even position them relative to the vehicle. Interior cameras, for example, can be used to record the driver and determine their direction of gaze or their blinking frequency, which can be used for attention or fatigue analysis.

[0003] Providing such assistance functions requires computer-based analysis of corresponding camera images. This requires the recognition of characteristic features in camera images. The camera images must have sufficient image quality. In this context, it should be noted that vehicle interiors often experience strongly fluctuating lighting conditions, for example when bright sunlight falls through the vehicle windows into the vehicle interior or at night when electric light sources illuminate different parts of the vehicle interior. Vehicles typically move through traffic, so light rays projected into the vehicle interior also move relative to the vehicle interior. The resulting changes in light and dark make it difficult to recognize these relevant features.

[0004] To improve image quality, especially at night, it is known to illuminate the vehicle interior with an infrared light source while capturing camera images. Accordingly, an interior camera capable of capturing infrared light must be provided. For this purpose, it is possible to install a separate infrared camera in addition to a black-and-white or color camera, or to design a color camera so that it can simultaneously generate color and infrared images.

[0005] Such a camera is known, for example, from US 2013 / 0222603 A1. The camera comprises an image sensor with different sensor pixels for capturing colored light for detecting infrared light. A color filter field, also known as a color filter array, is arranged in front of the sensor pixels. However, the color filter elements encompassed by the color filter field not only exhibit transmissivity for the respective light color, but also for infrared light. Furthermore, corresponding infrared pixels can also be excited by colored light. This reduces the dynamic range of camera images generated using such an image sensor. Brightly illuminated vehicle windows, for example, can then be overexposed and appear completely white. The document proposes a bandpass filter arranged in front of the image sensor in the light beam path. This bandpass filter exhibits transmissivity for both visible light and infrared light.However, the color pixels of the image sensor are still stimulated by infrared light. To improve image quality, it is possible to subtract the sensor signals generated by the infrared pixels from those generated by the color pixels. However, this requires additional effort.

[0006] Furthermore, US 2017 / 0347086 A1 discloses an imaging system with a light source, an image sensor, and a double bandpass filter. The image sensor comprises a plurality of photosensitive elements arranged in a pixel array, each of which is assigned color filter elements of a specific wavelength. The image sensor thus comprises color pixels and infrared pixels. To prevent excitation of the color pixels by infrared light, a bandpass filter is placed in front of the color filter elements in the light beam path. This bandpass filter allows visible light to pass through but blocks infrared light.

[0007] Furthermore, US 2022 / 0003907 A1 discloses a spectral sensor system using optical filters arranged in sub-grids. This describes an image sensor whose photosensitive elements are assigned to several color filter elements arranged one above the other in the light beam path. At least one color filter element is provided that allows light in a specified wavelength range to pass through, and at least one color filter element is provided that blocks light in a different wavelength range.

[0008] Furthermore, JP 2006-123314 A discloses a printing method for producing a flat piece. The flat piece can be part of an image sensor.

[0009] Furthermore, US 2018 / 0076242 A1 discloses an image sensor that can receive both visible light and infrared light. The image sensor has two receiving sections, one for visible light with a white filter and one for infrared light with an infrared photodiode, a second white filter, and an infrared pass filter.

[0010] The present invention is based on the object of providing a camera module that is improved compared to the prior art.

[0011] According to the invention, this object is achieved by a camera module having the features of claim 1. Advantageous embodiments and further developments as well as an image capture device having such a camera module and a vehicle having such an image capture device emerge from the dependent claims.

[0012] A generic camera module comprising an image sensor and a selection filter arranged upstream of the image sensor in the light beam path, wherein the image sensor in turn comprises a pixel field of photosensitive elements and a color filter field arranged upstream of the pixel field of color filter elements and infrared filter elements, and wherein individual color filter elements are assigned to individual photosensitive elements to form color pixels and individual infrared filter elements are assigned to individual photosensitive elements to form infrared pixels, provides that - the selection filter is designed as a bandpass filter with a transmission spectrum corresponding exclusively to visible light; and - the selection filter has recesses extending through the entire thickness of the selection filter in the direction of the light beam path, which recesses coincide with the respective arrangement positions of the infrared pixels in the plane of extension of the image sensor.

[0013] According to the invention, an infrared selection filter element is embedded in each of the recesses of the selection filter, which has a transmissivity for light exclusively in the infrared spectrum

[0014] By placing the bandpass filter in front of the color pixels, infrared light can be reliably prevented from penetrating the color pixels. This also prevents the photosensitive elements associated with the color pixels from being stimulated by infrared light. This improves image quality. In contrast to the prior art solution of a bandpass filter that is transparent to both visible and infrared light, it is no longer necessary to subtract a corresponding infrared signal from the color signal. This reduces the processing effort required to generate camera images.

[0015] Nevertheless, it is possible to generate infrared images with the aid of the camera module according to the invention and provide them simultaneously alongside the color images. This allows the infrared light to pass through the cutouts in the selection filter and reach the infrared filter elements and the photosensitive elements located underneath. Since the infrared filter elements have a significantly lower transmissivity for visible light than the color filter elements for infrared light, in the simplest embodiment, no special measures are required to generate high-quality infrared images with the described setup.

[0016] As already described, an infrared selection filter element is embedded in each of the recesses of the selection filter. This element has a transmissivity for light exclusively in the infrared spectrum, preferably for the near infrared, particularly preferably for a wavelength range from 850 nm to 950 nm, and in particular for a wavelength range from 930 nm to 950 nm. To provide infrared pixels, infrared filter elements are arranged in front of the photosensitive elements of the image sensor. However, these conventional infrared filter elements are also partially permeable to colored light, so that the corresponding infrared pixels are also excited by visible light. By additionally providing the infrared selection filter elements, these color components can be further filtered out, so that only infrared light can penetrate to the infrared pixels. This also improves the image quality for generating infrared images.The infrared selection filter elements are also formed by a bandpass filter with a correspondingly narrow transmission band. Light with a wavelength of 940 nm should be particularly advantageously passable. The transmission band of the infrared selection filter element is selected accordingly. The transmission band is particularly advantageously selected depending on the wavelength or wavelength range emitted by an active infrared illumination source.

[0017] An advantageous development of the camera module provides that the selection filter is transparent to light in a wavelength range from 400 nm to 680 nm. This wavelength range essentially corresponds to visible light. This ensures that the desired color spectrum of color images generated by the camera module can also be reproduced.

[0018] A further advantageous embodiment of the camera module according to the invention further provides that the color filter field comprises color filter elements for three different colors, in particular for red, green and blue. By means of a color filter field comprising three different colors, it is possible to generate color images. Red, green and blue color filter elements have proven particularly useful in this regard. For example, the color filter elements can be arranged in the form of a Bayer filter. Instead of red, green and blue, cyan, yellow and magenta could also be used as three different colors. Other embodiments are also possible, each with different colors or more than three different colors. The arrangement of the different color filter elements follows an ordered pattern to form a mosaic. The arrangement patterns common in the literature can be used.

[0019] Preferably, the selection filter comprises a support structure and color selection filter elements, wherein the support structure forms recesses coinciding with the arrangement position of the photosensitive elements, wherein: - first colour selection filter elements are embedded in the recesses coinciding with the colour pixels of a first colour, said first colour selection filter elements having a transmissivity for light exclusively in the wavelength range of the first colour; - second colour selection filter elements are embedded in the recesses coinciding with the colour pixels of a second colour, said filter elements having a transmissivity for light exclusively in the wavelength range of the second colour; and - third colour selection filter elements are embedded in the recesses coinciding with the colour pixels of a third colour, said filter elements having a transmissivity for light exclusively in the wavelength range of the third colour.

[0020] It is thus not only possible to design the selection filter as a coherent structure with holes, but also to provide a supporting structure, for example in the form of a perforated plate, a grid, or the like, wherein at least the color selection filter elements are embedded in the respective recesses. Holes can either remain at the position of the infrared pixels, or said infrared selection filter elements can be embedded. The individual distribution of the color selection filter elements allows the photosensitive elements assigned to the different colors to be exposed even more precisely with the respective target wavelength range. This further improves the color fidelity of camera images generated by the camera module according to the invention. The color selection filter elements are also corresponding bandpass filters that allow a narrower wavelength range to pass through than said known color filter elements.

[0021] A further advantageous embodiment of the camera module according to the invention further provides for a microlens array to be arranged in the light beam path upstream of the selection filter. Microlenses allow light to be captured in a targeted manner and directed onto the photosensitive elements. Each color pixel and each infrared pixel is thus assigned its own microlens.

[0022] According to a further advantageous embodiment of the camera module, a pre-filter is provided, which is arranged upstream of the selection filter in the light beam path, wherein the pre-filter is formed by a band-stop filter with a blocking spectrum in the wavelength range from 650 nm to 950 nm, preferably from 680 nm to 930 nm. With the help of the band-stop filter, the wavelength ranges of the light that lie between visible light and the infrared light to be imaged can be filtered out. It is particularly desirable for the infrared pixels to capture light with a wavelength in the order of 930 nm to 950 nm, in particular 940 nm. However, a correspondingly designed infrared selection filter can be comparatively expensive due to the narrow transmission wavelength range. If the pre-filter is provided, cheaper infrared selection filters with a broader transmission spectrum can be used.

[0023] The pre-filter can block light up to a wavelength of 950 nm. However, it is preferable to allow light with a wavelength of 940 nm to pass through. This variant represents a compromise, as even infrared light with a wavelength above 950 nm can still allow sufficient illumination of the infrared pixels. However, the blocking spectrum of the pre-filter is preferably designed so that the upper limit is approximately 930 nm up to a maximum of 939 nm. For example, the pre-filter can also block light in the wavelength range from 650 nm to 930 nm.

[0024] Preferably, the prefilter is designed as an optical lens or integrated into such an optical lens. This allows the camera module according to the invention to be designed particularly cost-efficiently and with optimized installation space.

[0025] According to the invention, an image capture device comprises a camera module as described above. The image capture device is accordingly a digital camera capable of recording camera images or videos. The image sensor can be embodied as a CMOS or CCD sensor element. By integrating the camera module according to the invention, the installation space of the image capture device can be made compact. In addition, manufacturing costs can be reduced, since only one camera module needs to be provided for the simultaneous recording of color images and infrared images. Furthermore, thanks to the camera module according to the invention, the image capture device according to the invention is characterized by particularly high image quality when generating corresponding color images. If the camera module comprises corresponding infrared selection filter elements, an increased image quality for infrared images can also be achieved.

[0026] A reduction in resolution caused by the inclusion of color and infrared pixels can be compensated for using proven methods such as interpolation or inpainting. This is preferably performed at the hardware level using so-called demosaicing or color interpolation. Artificial intelligence-based image reconstruction methods can also be used. Generative artificial intelligence is preferred for this purpose.

[0027] According to the invention, a vehicle comprises such an image capture device. The image capture device is preferably embodied as an interior camera. Particularly preferably, the image capture device comprises an infrared light source that allows active illumination of the captured scene using infrared light. The infrared light source is triggered together with the camera module in order to illuminate the scene precisely when a camera image is also being captured. As already mentioned, capturing camera images in the vehicle interior is subject to difficult recording conditions. Due to this, an image capture device according to the invention is particularly suitable for use in the automotive environment.

[0028] Further advantageous embodiments of the camera module according to the invention also emerge from the exemplary embodiments which are described in more detail below with reference to the figures.

[0029] Showing: Fig. 1 is a diagram showing the quantum efficiency of the color pixels and the infrared pixels of a known image sensor; Fig. 2 a schematic sectional view of a camera module according to the invention; Fig. 3 a schematic plan view of the image sensor of a camera module according to the invention; Fig. 4 a schematic plan view of the image sensor according to Fig. 3, with an upstream selection filter according to a first embodiment; Fig. 5 a schematic plan view of the image sensor according to Fig. 3, with an upstream selection filter according to a second embodiment; Fig. 6 a schematic plan view of the image sensor according to Fig. 3, with an upstream selection filter according to a third embodiment; and Fig. 7 a schematic plan view of the image sensor according to Fig. 3, with an upstream selection filter according to a fourth embodiment.

[0030] Fig. Figure 1 shows a quantum efficiency diagram 12 of the color pixels and infrared pixels of a prior art image sensor. The wavelength of the light in nanometers is plotted on the abscissa, and the quantum efficiency in percent is plotted on the ordinate. Curves are shown for the red (R), green (G), blue (B), and infrared (IR) light channels. The diagram shows that corresponding color pixels are also excited by infrared light, and infrared pixels are also excited by components of the colored light. This reduces the dynamic range of corresponding camera images. For example, a region 13 is highlighted, which is assigned to the infrared light spectrum but includes corresponding peaks for the red, blue, and green color channels. Furthermore, a peak of the infrared channel can be seen in a region 14.

[0031] To improve image quality, it is therefore desirable to assign only those wavelength ranges to each color channel of the image sensor that are to be imaged.

[0032] This is possible with the help of a camera module 1 shown in the following figures using a corresponding image sensor 2. Fig. 2 shows a sectional view through a camera module 1 according to the invention. The camera module 1 comprises an image sensor 2. The image sensor 2 in turn comprises a substrate 15 with photosensitive elements 5 arranged thereon. In particular, the photosensitive elements 5 are designed together with the substrate 15 as a so-called system-on-a-chip (SoC). Further electronic components such as processing units or memory elements can be arranged on the substrate 15 (not shown). The photosensitive elements 5 form a pixel field 4. A color filter field 6 is arranged on the pixel field 4 in the direction of the light beam path. The color filter field 6 comprises a plurality of color filter elements R, G, B as well as infrared filter elements IR. The color filter elements R, G, B are transparent for three different colors. For example, these can be red light, green light and blue light.However, other color combinations could also be used, such as magenta, yellow, and cyan. The color filter elements R, G, and B, as well as the infrared filter elements IR, are arranged as a mosaic, for example in the form of a Bayer filter, on pixel field 4.

[0033] According to Guttung, a selection filter 3 is arranged in the light beam path in front of the color filter field 6 in the camera module 1. The selection filter 3 is designed as a bandpass filter and has a transmissivity exclusively for visible light. This reliably prevents infrared light components from penetrating the photosensitive elements 5 below the color filter elements R, G, and B. This accordingly improves the image quality of color images generated using the image sensor 2 or the camera module 1. Fig. 2 shows the transmission spectrum of the selection filter 3 in a highly simplified qualitative representation.

[0034] According to Guttung, the selection filter 3 further comprises recesses 7 in those areas that coincide with the infrared pixels or the infrared filter elements IR. This enables exposure of corresponding photosensitive elements 5 with infrared light.

[0035] The selection filter 3 can be arranged directly on the color filter field 6 or can be spaced from it in the direction of the light beam path (not shown).

[0036] Optionally, a microlens array 9 comprising a plurality of microlenses 16 can be provided. Each color pixel and each infrared pixel or each photosensitive element 5 of the pixel array 4 is assigned exactly one microlens 16. Preferably, the microlens array 9 is arranged in the direction of the light beam path in front of the selection filter 3, as shown in Fig. 2. If corresponding recesses 7 are free, no microlenses 16 are arranged there. As will be mentioned later, however, a respective recess 7 can also be filled, so that a microlens 16 can also be arranged there.

[0037] Optionally, the camera module 1 may also comprise one or more optical lenses 11. Furthermore, the camera module 1 may have an optional pre-filter 10. The pre-filter 10 may, as shown in Fig. 2, particularly preferably form the optical lens 11 or be integrated therein. The prefilter 10 is designed as a dual bandpass filter and has a transmissivity for visible light and infrared light.

[0038] Fig. 3 shows a top view of a section of the color filter field 6. Fig. Figure 3 illustrates the general arrangement pattern of the R, G, and B color filter elements and the IR infrared filter elements. The IR infrared filter elements are referred to as "NIR" here to illustrate that near-infrared light is preferentially captured by the infrared pixels.

[0039] The Fig. 4 to 7 show a top view of the image sensor 2, wherein the selection filter 3 according to the invention is shown on the color filter field 6.

[0040] In the Fig. 4 and Fig. 5, the selection filter 3 is designed as a single piece as a coherent structure. In Fig. 4 shows free recesses 7.

[0041] Thus, the Fig. 4 and Fig. 6 generic image sensors 2 and the Fig. 5 and Fig. 7 image sensors according to the invention 2.

[0042] According to the invention, infrared selection filter elements IR-sel are inserted into the recesses 7. These infrared selection filter elements IR-sel are characterized by a very narrow transmission range in the wavelength ranges of infrared light, in particular near infrared light. Particularly preferably, infrared light with a wavelength in the order of 940 nm, in particular 940 nm ± 10 nm, is transmitted. Particularly preferably, this wavelength range is precisely the wavelengths emitted by an active infrared light source for illuminating a scene. This infrared light source can be part of an image capture device according to the invention that houses the camera module 1. This allows high-quality infrared images of the photographed or filmed scene to be generated. Fig. Figure 5 shows the transmission spectrum of the infrared selection filter elements IR-sel in a highly simplified qualitative representation.

[0043] The Fig. 6 and Fig. 7 illustrate two further alternative embodiments of the camera module 1 or image sensor 2 according to the invention. For this purpose, the selection filter 3 comprises a support structure 8, for example in the form of a perforated plate, a grid or wire mesh, or the like. The support structure 8 forms recesses into which color selection filter elements are inserted. In contrast to the color filter elements R, G, B, the color selection filter elements are characterized by a particularly narrow transmission band. Shown are first color selection filter elements R-sel for the transmission of red light, second color selection filter elements G-sel for the transmission of green light, and third color selection filter elements B-sel for the transmission of blue light. The respective transmission spectra are also shown in Fig. 6 and Fig. 7 is shown in a highly simplified manner. By providing the color selection filter elements R-sel, G-sel, B-sel, the color fidelity of camera images generated by the image sensor 2 can be improved.

[0044] In Fig. 6 shows an embodiment in which the recesses 7 in the selection filter 3 remain free. In Fig. 7, however, the said infrared selection filter elements IR-sel are inserted into the recesses 7.

[0045] Particularly advantageously, an image capture device according to the invention, comprising such a camera module 1 according to the invention, is integrated into a vehicle to form an interior camera. Particularly advantageously, a corresponding infrared light source is provided to illuminate the scene during the recording of camera images. By means of the image capture device according to the invention, it is possible to generate high-quality color and infrared images in the demanding environment of camera image generation in the vehicle interior.

Claims

[1] Camera module (1), comprising an image sensor (2) and a selection filter (3) arranged upstream of the image sensor (2) in the light beam path, wherein the image sensor (2) in turn comprises a pixel field (4) made up of photosensitive elements (5) and a color filter field (6) arranged upstream of the pixel field (4) made up of color filter elements (R, G, B) and infrared filter elements (IR), and wherein individual color filter elements (R, G, B) are assigned to individual photosensitive elements (5) to form color pixels and individual infrared filter elements (IR) are assigned to individual photosensitive elements (5) to form infrared pixels, wherein - the selection filter (3) is designed as a bandpass filter with a transmission spectrum corresponding exclusively to visible light, and - the selection filter (3) has recesses (7) extending through the entire thickness of the selection filter in the direction of the light beam path, which recesses coincide with the respective arrangement positions of the infrared pixels in the plane of extension of the image sensor (2), characterized by that an infrared selection filter element (IR-sel) is embedded in each of the recesses (7) of the selection filter (3), which has a transmissivity for light exclusively in the infrared spectrum. [2] Camera module (1) according to claim 1, characterized by that the selection filter (3) is transparent for light in a wavelength range from 400 nm to 680 nm. [3] Camera module (1) according to claim 1 or 2, characterized bythat the infrared selection filter element (IR-sel) has a transmissivity for light exclusively in the infrared spectrum for near infrared, particularly preferably for a wavelength range from 850 nm to 950 nm and in particular for a wavelength range from 930 nm to 950 nm. [4] Camera module (1) according to one of claims 1 to 3, characterized by that the color filter field (6) comprises color filter elements (R, G, B) for three different colors, in particular for red, green and blue. [5] Camera module (1) according to claim 4, characterized by that the selection filter (3) comprises a support structure (8) and color selection filter elements (R-sel, G-sel, B-sel), wherein the support structure (8) forms recesses coinciding with the arrangement position of the photosensitive elements (5), wherein: - first colour selection filter elements (R-sel) are embedded in the recesses coinciding with the colour pixels of a first colour, said first colour selection filter elements having a transmissivity for light exclusively in the wavelength range of the first colour; - second colour selection filter elements (G-sel) are embedded in the recesses coinciding with the colour pixels of a second colour, said filter elements having a transmissivity for light exclusively in the wavelength range of the second colour; and - third colour selection filter elements (B-sel) are embedded in the recesses coinciding with the colour pixels of a third colour, said filter elements having a transmissivity for light exclusively in the wavelength range of the third colour. [6] Camera module (1) according to one of claims 1 to 5, characterized by a microlens field (9) in front of the selection filter (3) in the light beam path. [7] Camera module (1) according to one of claims 1 to 6, characterized by a pre-filter (10) arranged upstream of the selection filter (3) in the light beam path, wherein the pre-filter (10) is formed by a band-stop filter with a blocking spectrum in the wavelength range from 650 nm to 950 nm, preferably from 680 nm to 930 nm. [8] Camera module (1) according to claim 7, characterized by that the pre-filter is designed as an optical lens (11). [9] Image capture device, characterized by , a camera module (1) according to one of claims 1 to 8, in particular further comprising an infrared light source. [10] Vehicle, characterized by an image capture device according to claim 9.

Images