MULTISPECTRAL IMAGE SENSOR, CAMERA SYSTEM AND METHOD FOR MANUFACTURING A MULTISPECTRAL IMAGE SENSOR

Abstract

A multispectral image sensor (1) comprises a plurality of photosensitive elements (11) configured to detect electromagnetic radiation received from a scene or object, and first and second optical modulators (12, 13) arranged on an incident side of the plurality of photosensitive elements (11), wherein the first and second optical modulators (12, 13) are configured to modulate electromagnetic radiation within a respective first and second wavelength range and to transmit electromagnetic radiation outside the respective first and second wavelength ranges. The first wavelength range is different from the second wavelength range.

Description

[0001] This disclosure relates to a lensless multispectral image sensor, a camera system comprising such an image sensor, and a method of manufacturing a multispectral image sensor.

[0002] A wide and growing variety of electronic components rely on cameras for image capture as well as for applications requiring feature recognition, e.g., for identification purposes. These include applications requiring feature recognition, as well as applications for monitoring an object or scene, e.g., by taking distance measurements, typically in a wavelength range outside the visible spectrum, e.g., in the infrared, for eye safety purposes and to ensure that a user is not disturbed or distracted. To enable both visible imaging and applications requiring feature recognition, multiple image sensors are typically employed, with each image sensor dedicated to a specific application, e.g., wavelength range.However, as the functionality of modern devices becomes increasingly demanding in combination with a limited minimum footprint of an image sensor, the need for multispectral image sensors has arisen.

[0003] However, conventional approaches utilize lens-based solutions with dispersive elements, which, due to their multispectral sensitivity, often require complicated and bulky component arrangements or expensive lens element materials. Furthermore, lens-based systems are characterized by considerable vertical height, which, in addition to the footprint, often represents an additional constraint due to spatial limitations.

[0004] One goal is therefore to provide a multispectral image sensor that overcomes the limitations of existing solutions. Another goal is to provide a camera system comprising such an image sensor and a method for manufacturing such an image sensor.

[0005] These objects are achieved by the subject matter of the independent claims. Further developments and embodiments are described in the dependent claims.

[0006] This disclosure overcomes the above-mentioned limitations of modern devices by providing a lensless image sensor having broadband photosensitive elements and a stack of layers over the photosensitive elements, each layer having an optical modulator interacting with a narrow range of wavelengths, the optical modulators each interacting with a different range of wavelengths and transmitting all other relevant optical wavelengths.

[0007] In one embodiment, a multispectral image sensor comprises a plurality of photosensitive elements, each photosensitive element configured to detect electromagnetic radiation received from a scene or an object and generate a photosignal depending on the detected electromagnetic radiation. The image sensor further comprises a first optical modulator arranged on an incident side of the plurality of photosensitive elements, the first optical modulator configured to modulate electromagnetic radiation within a first wavelength range and to transmit electromagnetic radiation outside the first wavelength range.The image sensor further comprises a second optical modulator arranged on an incident side of the plurality of photosensitive elements, wherein the second optical modulator is configured to modulate electromagnetic radiation within a second wavelength range and to transmit electromagnetic radiation outside the second wavelength range. The first wavelength range is different from the second wavelength range.

[0008] The photosensitive elements are configured to detect the electromagnetic radiation received from the scene or object, e.g. light from a natural source or an illuminating light source that is reflected from the scene or object. The photosensitive elements have a sensitivity that covers at least the first and second wavelength ranges. The photosensitive elements have, for example, broadband sensitivity. The photosensitive elements each generate a photosignal depending on the electromagnetic radiation detected by the respective element. The photosensitive elements form the detecting portion of an image sensor and can be arranged on a substrate body, e.g. a wafer or chip substrate. The detecting portion is formed, for example, from a matrix of pixels, wherein each pixel contains a photosensitive element, e.g. a photodiode.

[0009] The first and second optical modulators are arranged between the object or the scene and an active detection area of ​​the image sensor formed from the photosensitive elements. The first optical modulator is realized, for example, on or in a first layer, e.g. a thin-film substrate, which is arranged on a photosensitive surface of the detecting portion. The first layer is located, for example, on an upper side of the substrate body. The second optical modulator can be realized on or in a second layer, e.g. a thin-film substrate, which is arranged on a side of the first layer facing away from the photosensitive surface. In other words, the optical modulators form a stack arranged on the photosensitive surface of the detecting portion.The first optical modulator modulates light within a first wavelength range that differs from the second modulation range in which the second optical modulator operates. Specifically, the first optical modulator is transparent in the second wavelength range, and the second optical modulator is transparent in the first wavelength range. In other words, an optical modulator is transparent to optical wavelengths other than its modulation wavelength range.

[0010] This means that the image sensor can be configured to capture a first set of photosignals during a first exposure phase in the first wavelength range, e.g., in conjunction with an illuminating light source that illuminates the scene or object with light in the first wavelength range, and a second set of photosignals during a second exposure phase in the second wavelength range, e.g., in conjunction with another illuminating light source that illuminates the scene or object with light in the second wavelength range. Alternatively, the information of both the first and second wavelength ranges can be extracted separately from a set of photosignals captured during a single exposure phase by applying an algorithm to reconstruct the image.In such embodiments, the image sensor generates a set of photosignals during this single exposure in response to light detected in both the first and second wavelength ranges, e.g., in conjunction with an illuminating light source that illuminates the scene or object with light in the first and second wavelength ranges.

[0011] Unlike conventional imaging with optical lenses, in which each pixel of an image sensor captures light from a specific region of an object or scene to be imaged, the use of an optical modulator and thus the realization of lensless imaging results in an encoding of the object or scene information, e.g., through a different system response for each pixel of the image sensor. Each pixel of the image sensor can be understood as capturing an effective weighted sum of system response functions from each point of the object or scene. To reconstruct the image, a computational inverse algorithm can be applied to the photosignals. A first algorithm can reconstruct the image in the first wavelength range, and a second algorithm can reconstruct the image in the second wavelength range from the same set of photosignals.The optical modulators can be illumination modulated, mask modulated or implemented as programmable modulator systems.

[0012] In some embodiments, the first and second wavelength ranges do not overlap. To ensure that the first optical modulator transmits the second wavelength range without performing any modulation, and vice versa, the first and second optical modulators can be configured to each modulate wavelength ranges that do not overlap, i.e., that are spectrally separated. In this way, an algorithm used for image reconstruction can be tailored to each optical modulator without the other optical modulator corrupting the captured photosignals. In some embodiments, the first wavelength range is in the visible, near-infrared (NIR), or short-wave infrared (SWIR) portion of the electromagnetic spectrum. For example, the first wavelength range is around 840 and / or 930 nm. This wavelength range is invisible to the human eye, so optional illumination of a scene or object does not affect its appearance to the user. Furthermore, some applications such as 3D mapping or identification methods are typically performed in this range. Furthermore, established technologies for optional light emitters and photosensitive elements in this wavelength range can be used for efficient and cost-effective manufacturing and operation of the image sensor. For example, the photosensitive elements are silicon photodiodes that exhibit sensitivities in both the visible and NIR ranges up to approximately 1100 nm.In order to limit the sensitivity of the photosensitive elements to the first and second wavelength range, the photosensitive elements can, for example, have optical filter elements for blocking visible light.

[0013] In some embodiments, the second wavelength range is in the visible, near infrared (NIR), or shortwave infrared (SWIR) region of the electromagnetic spectrum. The second wavelength range may be in the same range as the first wavelength range, but cover a different portion of the range. For example, the first wavelength range is a region including 840 nm, while the second wavelength range includes 930 nm. Alternatively, the first or second wavelength range may be from a different region than the other, such as the SWIR, visible, or UV range.

[0014] In some embodiments, the photosensitive elements are silicon-based photodiodes or organic photodetectors (OPDs). The photosensitive elements can be manufactured using various technologies and thus tailored to a specific application, for example, in terms of sensitivity, power consumption, and size. Organic photodiodes are characterized by lower power consumption compared to their semiconductor counterparts. However, the manufacturing of semiconductor photodiodes is well established and can easily be combined with the manufacturing of other elements, such as light emitters and integrated circuit elements, using similar and compatible processes. The photosensitive elements can, for example, be silicon microphotodiodes for targeted sensitivity in the NIR range.

[0015] In some embodiments, the multispectral image sensor further comprises a plurality of display pixels configured to generate a display image in the visible range of the electromagnetic spectrum. The display pixels are formed from visible light emitters configured to emit light in the visible range of the electromagnetic spectrum, i.e., at optical wavelengths in the range of 400 to 750 nm. For example, the display pixels are formed from RGB light-emitting micro-LEDs and are arranged in a two-dimensional matrix to form a screen for displaying content. The light emitters of the pixels can be controlled to emit a specific respective color, or each pixel can be formed from red, green, and blue subpixels, for example. The pixels can be connected to a controller that controls an image to be displayed on the screen formed by the pixels.The display pixels and the photosensitive elements can be arranged on one or more transparent substrates, e.g., a glass or thin-film substrate, or a transparent foil. The term "transparent" refers to the visible portion of the electromagnetic spectrum, i.e., light with wavelengths to which the human eye is sensitive. The first and second optical modulators can be configured to transmit visible light.

[0016] In some embodiments, the plurality of display pixels form an OLED display, a micro-LED display, or a liquid crystal display (LCD). The aforementioned display elements can be formed from existing technologies already used for transparent displays such as OLEDs or LCDs. In particular, the production of micro-LEDs can be easily combined with the manufacturing process of micro-photodiodes. Like the photosensitive elements, the display pixels can also be formed using various technologies depending on the requirements of the intended application.

[0017] Micro-LEDs (light-emitting diodes, LEDs for short) are semiconductor light-emitting diodes with a particularly small size. For example, a growth substrate for epitaxial growth of a semiconductor layer sequence of the micro-LED is located away from the micro-LED. In other words, the micro-LED does not have the growth substrate. For example, the thickness or height of the micro-LED in a growth direction of the semiconductor layer sequence is between 1.5 µm and 10 µm.

[0018] A light exit surface of the micro-LED can, for example, be rectangular or have another shape. In particular, each lateral dimension of the light exit surface, viewed from above onto the layers of the semiconductor layer sequence, is at most 100 µm or at most 70 µm. For example, if the micro-LED has a rectangular shape, an edge length of the micro-LED—in particular viewed from above onto the layers of the semiconductor layer sequence—is at most 70 µm or at most 50 µm.

[0019] Micro-LEDs, for example, are provided on wafers with removable support structures, allowing the micro-LED to be removed from the wafer without damaging it. Micro-LEDs can also be referred to as µLEDs, µ-LEDs, uLEDs, u-LEDs, or micro light-emitting diodes.

[0020] In some embodiments, the multispectral image sensor further comprises a plurality of illumination light emitters configured to illuminate a scene or an object with electromagnetic radiation within the first and / or second wavelength range. The illumination light emitters are configured to emit light having an optical illumination wavelength comprising the first and / or second wavelength range, e.g., light in the NIR range in the range of 750 nm and 1.4 µm and / or in the SWIR range in the range of 1.4 µm and 3 µm. The illumination light emitters can be arranged on the same substrate as the photosensitive elements and / or the optional display pixel. For example, the sensing portion of the image sensor is formed from a matrix of pixels, each pixel including a subpixel with a light emitter in the NIR or SWIR range and a subpixel with a photosensitive element.Alternatively, only some of the pixels may contain a subpixel with an illuminating light emitter, e.g., the pixels along the edge of the sensing portion. Alternatively, the illuminating light emitters may be arranged independently of the photosensitive elements, e.g., forming a frame around the sensing portion or arranged on a separate substrate.

[0021] In some embodiments, the illumination light emitters are OLEDs, micro-LEDs, or vertical-cavity surface-emitting lasers (VCSELs). The optional light emitters for illuminating the scene or object can be selected from various existing technologies, such as organic LEDs or micro-LEDs. For example, the illumination light emitters have the same architecture as the photosensitive elements, e.g., both are semiconductor or organic diode structures, so they can be easily fabricated on the same substrate using the same or a similar manufacturing process. Alternatively, the illumination light emitters can have a different architecture, e.g., the display pixels are formed from OLEDs and the photosensitive elements are implemented as micro-photodiodes, or vice versa.In this way, they can be manufactured separately on different substrates, for example, to tailor the display technology and illumination to a specific application. Furthermore, if coherent light or longer-range illumination is required, the illumination light emitters can be formed from VCSELs, which are also arranged on the same substrate as the photosensitive elements or on a different substrate.

[0022] In some embodiments, the first and / or second optical modulator is implemented by an active matrix based on one of the following: liquid crystals, optical switches, digital light processors, and spatial light processors. In these embodiments, the image sensor may further include a controller for controlling elements of the active matrix. Dynamic and programmable modulators, also referred to as spatial light modulators, may be used as an alternative to fixed masks. For example, the active matrix is ​​based on liquid crystal technology, which can be used for programmable amplitude modulation. For programmable phase modulation, LC-on-silica devices can be used in the active matrix. Further examples are vanadium oxide transistors, which act as optical switches in the NIR range while being transmissive at visible frequencies.Programmable modulators have the advantage that a mask pattern can be changed easily and quickly, allowing multiple images, each with a different optical encoding, to be acquired in succession. Furthermore, in embodiments with a relatively small number of photosensitive elements, programmable modulators can facilitate the acquisition of sufficient measurement values ​​for reconstruction. Even in the case of a sensor array composed of photosensitive elements, the ability to change the modulation pattern between acquisitions offers an additional degree of freedom that can further improve reconstruction performance or the resolution of the reconstructed image. However, an active matrix requires additional components such as the aforementioned regulator and is characterized by an additional power consumption channel.

[0023] In some embodiments, the first and / or second optical modulator is implemented by a passive matrix based on one of the following: an amplitude mask, a phase mask, and a plurality of diffractive elements. Alternatively to an active matrix, one or both of the optical modulators can perform modulation passively using amplitude masks, diffractive masks, random reflective surfaces, and modified microlens arrays. Amplitude modulators comprise transparent and occluding regions arranged in a fixed spatial pattern on a 2D mask. Phase modulators, on the other hand, vary the relative optical path length or the effective phase delay in a 2D pattern.Phase masks can be more light-efficient due to no or negligible attenuation of the incident light, and they can focus light to produce sharper and higher-contrast patterns on the image sensor, thereby also improving image reconstruction performance.

[0024] In some embodiments, the first and / or second optical modulator is realized by a spatially distributed plurality of pinholes. An amplitude mask transmits, blocks, or attenuates incident light. One of the simplest architectures of a binary amplitude mask is a distribution of pinholes. Amplitude masks generally have the advantage of being easier to manufacture over a broad range of wavelengths. Outside the visible range, materials that can block light are easier and cheaper to find than those that can refract light, for example. In particular, materials that are transparent in the visible range and opaque in the NIR or SWIR range can be easily formed into a mask for an optical modulator. For example, dye-based polymers such as Fujifilm SIR850W, SIR850N, or SIR940 can be used to form the amplitude masks.These polymers exhibit a narrow absorption band in the infrared range between 850 and 940 nm, while maintaining high transparency in the visible range. Their thickness ranges from 0.8 to 1.36 µm. Another example of an optical modulator suitable for lensless imaging is a diffuser. In addition to amplitude modulation, the interaction of light with the pinholes can also be based on modulating the phase or polarization of an incident light. For this purpose, phase-shifting materials can be used in the range of the illumination wavelength, as long as they are transparent in the other wavelength range of the other optical modulator.

[0025] In some embodiments, the first and / or second optical modulator forms a coded aperture mask, particularly characterized by a uniformly redundant array (URA) or an optimized random pattern (ORA). For computational imaging purposes, the optical modulators of the lensless system may be a regular pattern, such as uniformly redundant arrays (URA) and random (or pseudorandom) M-sequences. In lensless imaging, coded apertures or coded aperture masks are gratings, grids, or other patterns made of materials that are opaque to different wavelengths of electromagnetic radiation, i.e., the illuminating wavelength range. By blocking radiation in a known pattern, a coded shadow is cast on a plane.The properties of the original radiation sources can then be mathematically reconstructed from this shadow using an algorithm that incorporates knowledge of the mask pattern. Examples include Fresnel zone plates, optimized random patterns, URAs, and hexagonal or modified URAs. The concept of coded aperture imaging itself is a well-known one, particularly in X-ray and gamma-ray imaging systems, where the beams cannot be focused with lenses or mirrors designed for visible light.

[0026] In some embodiments, the multispectral image sensor is transparent in the visible range of the electromagnetic spectrum. Particularly in applications where an unobstructed view of the user's scene is desired, e.g., in heads-up displays or augmented reality wearables, the image sensor may be transparent in the visible range. For example, the photosensitive elements are distributed on a transparent substrate and arranged at a distance from one another so that the image sensor appears transparent to the user. Analogously, the first and second optical modulators may be configured to transmit visible light so that they also appear transparent to the user. For example, 70% of the light intensity in the visible range is transmitted by the image sensor.

[0027] In some embodiments, the plurality of photosensitive elements are arranged on a substrate, and the first and second optical modulators are formed by corresponding mask layers arranged on a front side of the substrate. The mask layers may be transparent thin films or foils on or in which structures are arranged that are opaque in the first or second wavelength range, respectively, but transparent outside this range. The substrate of the sensing portion comprising the photosensitive elements may be a semiconductor substrate, e.g., a silicon chip, or a transparent substrate, such as a glass substrate.

[0028] In some embodiments, the multispectral image sensor further comprises one or more further optical modulators arranged on an incident side of the plurality of photosensitive elements, wherein each further optical modulator is configured to modulate electromagnetic radiation within a respective further wavelength range and to transmit electromagnetic radiation outside the respective further wavelength range. For example, a plurality of optical modulators forms a stack, wherein each optical modulator has a different modulation wavelength range that does not overlap. For example, each optical modulator is characterized by a narrowband modulation range, wherein the combined modulation ranges cover a specific portion of the electromagnetic spectrum, e.g., the NIR or SWIR portion.In this way, a corresponding algorithm for each of the optical modulators can be applied to the same set of photosignals to reconstruct an image in each of the modulation regions and obtain a plurality of reconstructed images.

[0029] Furthermore, a multispectral camera system is provided, comprising a multispectral image sensor according to one of the aforementioned embodiments of the multispectral image sensor and a processing unit coupled to the multispectral image sensor and configured to reconstruct a first image for the first wavelength range and a second image for the second wavelength range by applying a set of algorithms to the photosignals generated by the photosensitive elements. The multispectral camera system may be an integrated system in which the processing unit is embedded in the image sensor. Alternatively, the processing unit may be an external component coupled to active elements of the image sensor.The processing unit can further be used to synchronize illumination of the scene or object by activating and deactivating emission from optional illumination light emitters with active detection, i.e., an integration time, of the photosensitive elements. For reconstruction, the processing unit is configured to apply a set of algorithms to the photo signals, wherein the set of algorithms includes an algorithm corresponding to each of the optical modulators that takes into account a property of the respective optical modulator, e.g., a layout or pattern of a coded aperture mask. Depending on the application, the set of algorithms can be used to reconstruct images of the scene, e.g., via deconvolution or deep neural networks, or to perform a specific recognition task, e.g.,Face identification, or feature extraction from the raw sensor image.

[0030] Furthermore, an electronic component is provided which comprises a multispectral image sensor or a camera system according to any of the aforementioned embodiments. The electronic component may be a mobile computing device such as a smartphone, a tablet, or a laptop, or a wearable device such as smart glasses. Alternatively, a transparent image sensor or a transparent camera system may be used in a transparent object, such as a vehicle windshield, to form a heads-up display with imaging capability that does not rely on projections. A transparent image sensor or camera system is particularly used in applications where an image is to be displayed while maintaining an unobstructed line of sight behind the display from the user's point of view.In such components, the display can, for example, serve as an augmented reality display or to display data that is detected in the field of view of the image sensor of the display and the user, e.g. recognized speed limits of road signs.

[0031] Furthermore, a method for manufacturing a multispectral image sensor is provided. The method comprises providing a plurality of photosensitive elements, each configured to detect electromagnetic radiation received from a scene or an object and to generate a photosignal depending on the detected electromagnetic radiation. The method further comprises providing a first optical modulator arranged on an incident side of the plurality of photosensitive elements, wherein the first optical modulator is configured to modulate electromagnetic radiation within a first wavelength range and to transmit electromagnetic radiation outside the first wavelength range.The method further comprises providing a second optical modulator arranged on an incident side of the plurality of photosensitive elements, wherein the second optical modulator is configured to modulate electromagnetic radiation within a second wavelength range and to transmit electromagnetic radiation outside the second wavelength range. The first wavelength range is different from the second wavelength range.

[0032] Further embodiments of the method will become apparent to the expert reader from the aforementioned embodiments of the multispectral image sensor, the camera system, and the electronic component, and vice versa.

[0033] The following description of the figures may further illustrate and explain aspects of the transparent display, the imaging system, and the method for manufacturing a transparent display. Components and parts of the transparent display that are functionally identical or have an identical effect are identified by identical reference numerals. Identical or virtually identical components and parts may be described only with reference to the figures in which they first appear. Their description is not necessarily repeated in subsequent figures.

[0034] In the figures: Fig. 1 shows an embodiment of a camera system including a first embodiment of a multispectral image sensor according to the improved concept; Fig. 2 shows a second embodiment of a multispectral image sensor; Fig. 3 shows a third embodiment of a multispectral image sensor; Fig. 4 shows embodiments of optical modulators used in an image sensor; Fig. Figure 5 shows examples of reconstructed images obtained with the optical modulators of Fig. 4 were recorded; and Fig. 6 shows an embodiment of an electronic component having a camera system.

[0035] Fig. Figure 1 shows a schematic exploded view of an embodiment of a camera system 100 comprising a multispectral image sensor 1 according to the improved concept. The camera system 100 further comprises a processing unit 2 electrically connected to the image sensor 1, e.g., to an integrated circuit of the image sensor 1. The processing unit 2 is configured to receive the photo signals from the image sensor 1 and to reconstruct images by applying a set of algorithms to the photo signals. The reconstruction is carried out in Fig. 5 and further explained in the relevant section of Revelation.

[0036] The image sensor 1 has a substrate 10, which is, for example, a semiconductor substrate such as a silicon chip. The substrate 10 can alternatively be made of a material that is transparent in the visible range of the electromagnetic spectrum, e.g., glass. The substrate 10 can also be a thin film or foil. The substrate has a plurality of photosensitive elements 11, e.g., photodiodes. For example, the photosensitive elements 11 are silicon-based micro-photodiodes with a sensitive surface arranged on a front or top side of the substrate 10. For example, the photosensitive elements 11 are buried photodiodes. Alternatively, the photosensitive elements 11 can be organic photodiodes. The photosensitive elements 11 can be arranged in a matrix, being spaced apart from one another or being in close proximity to one another.Taken together, the photosensitive elements 11 form the detecting portion of the image sensor 1. The photosensitive elements 11 may have a coating or a filter element on their incident side for blocking visible light.

[0037] Optionally, the image sensor 1 can also have one or more illumination light emitters 15 arranged on the substrate 10. The illumination light emitters 15 are configured to illuminate a scene or an object, such that the photosensitive elements 11 detect the light emitted by the illumination light emitters 15 and reflected by the object or scene. The illumination light emitters 15 are configured to emit light in the first and / or second wavelength range. The illumination light emitters 15 can be, for example, micro-LEDs or VCSELs. The illumination light emitters 12 can be arranged in an outer edge of the photosensitive elements 11. The illumination light emitters 12 are configured, for example, to emit light in the NIR or SWIR range.Alternatively, the illumination light emitters 15 may be arranged on a further substrate arranged on a sensing surface of the substrate 10, wherein the further substrate is transparent at least in the first and second wavelength range.

[0038] The image sensor 1 further comprises a first optical substrate 10a and a second optical substrate 10b. The first optical modulator 12 is arranged on the first optical substrate 10a. The second optical modulator 13 is arranged on the second optical substrate 10b. The first and second optical modulators 12, 13 implement optical encoders for lensless imaging. For example, the optical modulators 12, 13 are passive modulators implemented by an amplitude mask, a phase mask, or a plurality of diffractive elements. Examples include an array of pinholes, a diffuser, a Fresnel zone plate, and coded aperture masks characterized, for example, by a uniformly redundant array or an optimized random pattern. The optical modulators 12, 13 are designed to influence only light in or around a specific wavelength range, e.g., a portion of the NIR or SWIR range.The first optical modulator 12 influences light in a first wavelength range, while the second optical modulator 13 influences light in a second wavelength range that differs from the first wavelength range. The first and second wavelength ranges do not overlap. Thus, the first optical modulator is transparent in the second wavelength range, and vice versa. Furthermore, the optical modulators 12, 13 can be transparent in the visible range, allowing the image sensor to perform, for example, conventional imaging.

[0039] In this embodiment, the image sensor 1 further comprises an additional optical modulator 20, which is arranged, for example, on a further optical substrate. Therein, the additional optical modulator 20 interacts with light in a further wavelength range that is different from and does not overlap with the first and second wavelength ranges. Outside the further wavelength range, the additional optical modulator 20 is transparent. Further embodiments can comprise a larger number of optical modulators that form a modulator stack. For example, each optical modulator 12, 13, 20 is characterized by a narrow optical bandwidth at which the light is modulated, wherein the combined bandwidths of all optical modulators 12, 13, 20 extend over a portion of the electromagnetic spectrum, e.g., the NIR or SWIR range.

[0040] The layers formed from the substrate 10, the optical substrates 10a, 10b, and optional additional substrates, e.g., additional optical modulators 20 and / or a substrate with the optional illumination light emitters 15, are arranged one above the other, optionally with spacers in between, to form the finished multispectral image sensor 1 and the camera system 100 with lensless imaging capability. The image sensor 1 may have additional layers, e.g., a circuit substrate containing active and passive circuits for operating the image sensor 1.

[0041] Fig. 2 shows a second embodiment of a multispectral image sensor 1. Compared to the first embodiment of Fig. 1, this embodiment further comprises a display substrate 10c arranged on the incident side of the substrate 10 with the photosensitive elements 11, the latter not being shown in the figure for illustrative purposes. Alternatively, the display substrate 10c can be the topmost layer of the stack formed from the optical modulators 12, 13, 20. The display substrate 10c has a plurality of display pixels 14. The display pixels 14 can be arranged in a two-dimensional matrix arrangement, so that a display for displaying an image in the visible range can be formed. The display pixels 14 can comprise RGB light emitters such as OLEDs or microLEDs.

[0042] The display substrate 10c is transparent in the wavelength ranges of all modulators 12, 13, 20, so that the captured image is not distorted by the display layer. Furthermore, the display pixels 14 can be spaced apart from one another so that the light of the first and second wavelength ranges can penetrate through gaps between the display pixels 14 to reach the photosensitive elements 11.

[0043] Analogously, the optical modulators 12, 13, 20 and the optical substrates 10a, 10b are transparent in the visible range, so that a displayed image on the display layer can be seen by a user. In addition to or alternatively to the illumination light emitters 15 arranged on the substrate 10, the display substrate 10c can also have one or more illumination light emitters 15 arranged on its surface.

[0044] Fig. 3 shows a third embodiment of a multispectral image sensor 1. In comparison to the embodiments of Fig. 1 and Fig. 2, in this embodiment, the first and second optical modulators 12, 13, or alternatively one of them, are active elements, i.e., they are realized by means of an active matrix. Therefore, the image sensor 1 further comprises a controller 3 electrically connected to the active optical modulators 12, 13 for controlling elements of the active matrix. The active matrix may comprise liquid crystals, optical switches, and / or other types of spatial and digital light processors. The optical modulators 12, 13 are based, for example, on liquid crystal technology, which can be used for programmable amplitude modulation. For programmable phase modulation, LC-on-silicon dioxide components can be used in the active matrix. Further examples are vanadium oxide transistors, which act as optical switches in the NIR range while being transmissive at visible frequencies.

[0045] A programmable optical modulator 12, 13 has the advantage that a mask pattern can be changed easily and quickly, allowing multiple images, each with a different optical encoding, to be acquired in succession. Furthermore, in embodiments with a relatively small number of photosensitive elements 11, a programmable modulator 12, 13 can facilitate obtaining sufficient measurement values ​​for reconstruction. In the case of a sensor array of photosensitive elements 11, the ability to change the modulation pattern between acquisitions offers an additional degree of freedom that can further improve reconstruction performance or the resolution of the reconstructed image. The controller 3 can be connected to the processing unit 2 of an imaging system in order to be synchronized with an emission of the illumination light emitters 15 and a detection of the photosensitive elements 11, i.e., an exposure phase.

[0046] Fig. 4 shows embodiments of optical modulators 12, 13 used in an image sensor 1 according to the improved concept. More specifically, panel (a) shows an amplitude mask as the first optical modulator 12, while panel (b) shows a different amplitude mask as the second optical modulator 13. The first optical modulator 12 responds to a first wavelength range characterized by a first central wavelength λ1 and a bandwidth, while the second optical modulator 13 responds to a second wavelength range characterized by a second central wavelength λ2 and a further bandwidth. For example, the bandwidths of the optical modulators 12, 13 are identical, but the central wavelengths λ1, λ2 differ from one another, so that the first and second wavelength ranges do not overlap.More specifically, the amplitude masks illustrate the case where the respective darker regions block radiation only in the respective wavelength range containing λ1 or λ2 and are transparent everywhere else. The bright regions are transparent to every wavelength.

[0047] Fig. Figure 5 shows exemplary reconstructed images obtained with the optical modulators 12, 13 of Fig. 4 were acquired during a single exposure of an acquisition area containing the first and second wavelength ranges. For this purpose, a reconstruction algorithm is applied to each of the reconstructed images. In this example, a simple reconstruction deconvolution is applied for the first and second wavelength ranges for a two-layer stack of optical modulators. However, this is only for illustrative purposes, and more sophisticated algorithms may lead to better results. Nevertheless, the algorithm requirement can be tailored to the application, so that an optimal algorithm is used to maintain a certain level of computational complexity, e.g., for reasons of energy and time savings.

[0048] In the following, the operating principle of the algorithms is explained for a stack of N coded masks, which are arranged one above the other and are formed from a stack of a plurality of optical modulators 12, 13, 20, which are arranged above the photosensitive elements 11 of the detecting part of the image sensor 1. As already described above for the two-layer stack, a mask of the stack, which is labeled A j referred to, to a specific wavelength λ j The object irradiance O at λ j is only used by mask A j processed, which the sensor signal R j generated: Rj=Aj∗O(λj), where "*" denotes the convolution operation. Since the object or scene captured by image sensor 1 also radiates at other wavelengths, the total sensor signal after a single exposure period or integration time is: ∑i=1NRi=∑i=1NAi∗O(λi).

[0049] If the image reconstruction Õ(λ j ) is required, where “~” denotes an approximation, and the matched filter technique is applied, the following results: O˜j=(∑i=1NRi)∗Gj=Rj∗Gj+∑i=1,i≠jNRi∗Gj=O˜(λj)+∑i=1,i≠jNRi∗Gj.

[0050] In the example of a stack of two optical modulators 12, 13, as shown in the Fig. 4 and Fig. 5, it is worth noting that despite the artifacts caused by the term ∑i=1,i≠jNRi∗Gj be created, the object Õ(λ j ), a star shape, is still recognizable.

[0051] Depending on the specific application, the algorithm used to reconstruct the image can be based, for example, on a deconvolutional or convolutional neural network. Alternatively, a specific machine learning algorithm can be used to detect features at different wavelengths. Third, obtaining specific information at different wavelengths can help increase the accuracy of machine learning models. It is known that a large amount of reference data is required to train neural networks. With the present multispectral image sensor 1, multispectral information can be obtained with a single exposure. For example, if the application requires a specific wavelength range without the detection element being located in this specific band, e.g.the detection of IR from RGB images or vice versa, the present idea can provide the reference data in a single system.

[0052] If the materials of the layer stack are selected to be transparent in the visible range, it is also possible to realize so-called bidirectional displays, in which an image sensor is embedded beneath an OLED (or microLED) layer that forms the emissive display. Due to its transparency in the visible range, the layer stack will not interfere with the displayed information.

[0053] Fig. Figure 6 shows an embodiment of an electronic component 200, e.g., a smartphone or smartwatch, comprising a camera system 100 according to the improved concept. The camera system 100 comprises a multispectral image sensor 1 and a processing unit 2 electrically connected to the image sensor 1.

[0054] The embodiments of the multispectral image sensor 1, the camera system 100, and the method for manufacturing a multispectral image sensor 1 disclosed herein have been discussed to familiarize the reader with new aspects of the concept. Although preferred embodiments have been shown and described, changes, modifications, equivalents, and substitutions of the disclosed concepts may be made by one skilled in the art without unnecessarily departing from the scope of the claims.

[0055] It is clear that the disclosure is not limited to the disclosed embodiments and to what has been specifically shown and described herein. Rather, features listed in individual dependent claims or in the description may be advantageously combined. Furthermore, the scope of the disclosure includes those variations and modifications that would be obvious to one skilled in the art and fall within the scope of the appended claims.

[0056] The term "comprising," as used in the claims or the description, does not exclude other elements or steps of a corresponding feature or method. If the terms "a" or "an" are used in connection with features, they do not exclude a plurality of such features. Furthermore, any reference signs in the claims should not be construed as limiting the scope.

[0057] This patent application claims priority from German patent application DE 10 2022 114 678.2, the disclosure of which is hereby incorporated by reference. Reference symbol 1 image sensor 2 processing unit 3 controls 10, 10a, 10b, 10c Substrat 11 photosensitive element 12, 13, 20 optical modulator 14 display pixels 15 illumination light emitters 100 camera system 200 electronic components QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] FROM 10 2022 114 678.2

[0057]

Claims

[1] A multispectral image sensor (1), comprising: - a plurality of photosensitive elements (11), each arranged to detect electromagnetic radiation received from a scene or an object and to generate a photosignal in dependence on the detected electromagnetic radiation; and - a first optical modulator (12) arranged on an incident side of the plurality of photosensitive elements (11), wherein the first optical modulator (12) is configured to modulate electromagnetic radiation within a first wavelength range and to transmit electromagnetic radiation outside the first wavelength range; and - a second optical modulator (13) arranged on an incident side of the plurality of photosensitive elements (11), wherein the second optical modulator (13) is configured to modulate electromagnetic radiation within a second wavelength range and to transmit electromagnetic radiation outside the second wavelength range; - wherein the first wavelength range is different from the second wavelength range. [2] The multispectral image sensor (1) according to claim 1, wherein the first and second wavelength ranges do not overlap. [3] The multispectral image sensor (1) according to claim 1 or 2, wherein the first wavelength range is in the visible, near infrared, NIR, or shortwave infrared, SWIR, portion of the electromagnetic spectrum. [4] The multispectral image sensor (1) according to any one of claims 1 to 3, wherein the second wavelength range is in the visible, near infrared, NIR, or shortwave infrared, SWIR, range of the electromagnetic spectrum. [5] The multispectral image sensor (1) according to one of claims 1 to 4, wherein the photosensitive elements (11) are silicon-based photodiodes or organic photodetectors, OPDs. [6] The multispectral image sensor (1) according to any one of claims 1 to 5, further comprising a plurality of display pixels (14) configured to generate a display image in the visible range of the electromagnetic spectrum. [7] The multispectral image sensor (1) according to claim 6, wherein the plurality of display pixels (14) form an OLED display, a micro-LED display or a liquid crystal display, LCD. [8] The multispectral image sensor (1) according to any one of claims 1 to 7, further comprising a plurality of illuminating light emitters (15) configured to illuminate the scene or object with electromagnetic radiation within the first and / or second wavelength range. [9] The multispectral image sensor (1) according to claim 8, wherein the illumination light emitters (15) are OLEDs, micro-LEDs or vertical cavity surface emitting lasers, VCSELs. [10] The multispectral image sensor (1) according to any one of claims 1 to 9, wherein the first and / or second optical modulator (12, 13) is realized by an active matrix based on one of the following: liquid crystals, optical switches, digital light processors and spatial light processors. [11] The multispectral image sensor (1) according to one of claims 1 to 10, wherein the first and / or second optical modulator (12, 13) is realized by a passive matrix based on one of the following: an amplitude mask, a phase mask, and a plurality of diffractive elements. [12] The multispectral image sensor (1) according to one of claims 1 to 11, wherein the first and / or second optical modulator (12, 13) is realized by a spatially distributed plurality of pinhole diaphragms. [13] The multispectral image sensor (1) according to one of claims 1 to 12, wherein the first and / or second optical modulator (12, 13) is realized by a dye-based polymer. [14] The multispectral image sensor (1) according to one of claims 1 to 13, wherein the first and / or second optical modulator (12, 13) forms a coded aperture mask, in particular characterized bya uniformly redundant array, URA, or an optimized random array, ORA. [15] The multispectral image sensor (1) according to any one of claims 1 to 14, wherein the multispectral image sensor (1) is transparent in the visible range of the electromagnetic spectrum. [16] The multispectral image sensor (1) according to one of claims 1 to 15, wherein - the plurality of photosensitive elements (11) are arranged on a substrate (10); and - the first and second optical modulators (12, 13) are formed by corresponding mask layers (10a, 10b) arranged on a front side of the substrate (10). [17] The multispectral image sensor (1) according to any one of claims 1 to 16, further comprising one or more further optical modulators (20) arranged on an incident side of the plurality of photosensitive elements (11), each further optical modulator (20) being configured to modulate electromagnetic radiation within a respective further wavelength range and to transmit electromagnetic radiation outside the respective further wavelength range. [18] A camera system (100) comprising: - a multispectral image sensor (1) according to one of claims 1 to 17; and - a processing unit (2) connected to the multispectral image sensor (1) and configured to reconstruct a first image for the first wavelength range and a second image for the second wavelength range by applying a set of algorithms to the photosignals generated by the photosensitive elements (11). [19] An electronic component (200) comprising a multispectral image sensor (1) according to one of claims 1 to 17 or a camera system (100) according to claim 18. [20] A method for manufacturing a multispectral image sensor (1), the method comprising: - providing a plurality of photosensitive elements (11), each adapted to detect electromagnetic radiation received from a scene or an object and to generate a photosignal in dependence on the detected electromagnetic radiation; - Providing a first optical modulator (12) arranged on an incident side of the plurality of photosensitive elements (11), wherein the first optical modulator (12) is configured to modulate electromagnetic radiation within a first wavelength range and to transmit electromagnetic radiation outside the first wavelength range; and - providing a second optical modulator (13) arranged on an incident side of the plurality of photosensitive elements (11), wherein the second optical modulator (13) is configured to modulate electromagnetic radiation within a second wavelength range and to transmit electromagnetic radiation outside the second wavelength range; - wherein the first wavelength range is different from the second wavelength range.

Images