Imaging apparatus and multispectral imaging method

By adopting multi-subpixel type detector array and micromirror array in CMOS image sensor, combining filter array and internal light source, imaging within the visible spectrum and long-wave infrared spectrum is achieved, solving the problem of undetectable in the prior art The problem of long-wave infrared radiation simplifies system design and improves image overlap accuracy.

CN119096123APending Publication Date: 2024-12-06AMS SENSORS USA INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380038342.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-18
Filing Date
2023-05-16
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing CMOS image sensors are insensitive within the long-wave infrared spectrum range and cannot directly detect long-wave infrared radiation, resulting in the inability to achieve imaging in visible light and long-wave infrared spectrum ranges.

Method used

Using detector arrays and micromirror arrays with multiple sub-pixel types, the detection and imaging of radiation in the visible and long-wave infrared spectral range is achieved through filter arrays and internal light sources, and the same detector arrays are used to capture visible and thermal images simultaneously. , reduce system cost and power consumption.

Benefits of technology

Multispectral imaging in the visible spectrum and long-wave infrared spectrum is achieved, simplifying system design, reducing cost, size and power consumption, and improving image overlap accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096123A_ABST
    Figure CN119096123A_ABST
Patent Text Reader

Abstract

An imaging device (1) is specified, comprising:-a detector array (2) of a plurality of pixels (25), the pixels (25) comprising a plurality of sub-pixel types (21),-a micromirror array (3) having a plurality of mirror elements (31), and-an internal light source (4), where-at least one of the sub-pixel types (21) is configured to detect a first radiation (R1); -the mirror element (31) is configured to deflect in response to the second radiation (R2),-the internal light source (4) is configured to illuminate the detector array (2) with a third radiation (R3); at least one of the subpixel types (21) is configured to detect a third radiation (R3) deflected by the micromirror array (3). Furthermore, a method of multispectral imaging is specified.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 343,297, filed May 18, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to an imaging device and a multi-spectral imaging method. Background Art

[0004] CMOS image sensors are widely used to detect radiation in the visible or near-infrared spectral range, with a maximum detection wavelength of up to 1100nm. However, silicon used as the photosensitive material in these sensors is insensitive to radiation in the long-wave infrared spectral range, making these sensors unable to directly detect long-wave infrared radiation for thermal imaging.

[0005] The object to be solved is to provide an imaging device which allows imaging both in the visible spectral range and in the long-wave infrared spectral range. Furthermore, a method allowing multispectral imaging will be specified.

[0006] These and other objects are achieved in particular by an imaging device and a method according to the independent claims. Further configurations and advantages are subject matter of the dependent claims.

[0007] Specifies an imaging device. Summary of the invention

[0008] According to at least one embodiment of the imaging device, the imaging device comprises a detector array having a plurality of pixels. Specifically, the pixels comprise a plurality of sub-pixel types. For example, the sub-pixel types differ from each other in their spectral sensitivity.

[0009] For example, silicon is used as the photosensitive material for the detector array.Different spectral sensitivities of the sub-pixel types can be obtained by a filter array arranged on the detector array.

[0010] For example, three different sub-pixel types are sensitive in the visible spectral range. For example, the detector array includes sub-pixel types that are sensitive in the red, green and blue spectral ranges, respectively, for full-color imaging in the visible spectral range.

[0011] According to at least one embodiment of the imaging device, the imaging device comprises a micromirror array having a plurality of mirror elements. For example, the micromirror is a MEMS (micro-electromechanical system) based mirror. For example, the mirror element is configured to deflect in response to long-wave infrared radiation.

[0012] Long-wave infrared radiation specifically includes radiation with wavelengths between 7 µm and 14 µm. Radiation in this spectral range includes radiation within the so-called third atmospheric window, making this wavelength range particularly suitable for thermal imaging.

[0013] According to at least one embodiment of the imaging device, the imaging device comprises an internal light source. For example, the internal light source is configured to emit radiation detectable by the detector array. For example, the radiation of the internal light source is in the ultraviolet visible or near infrared spectral range.

[0014] In this context, near infrared (NIR) refers in particular to radiation between 700 nm and 1100 nm inclusive.

[0015] For example, the peak emission wavelength of the internal light source is at most 1100 nm or at most 1000 nm. In other words, the peak emission wavelength is less than the cut-off wavelength of silicon.

[0016] According to at least one embodiment of the imaging device, at least one of the sub-pixel types is configured to detect a first radiation. The first radiation comes from a scene detected by the imaging device.

[0017] According to at least one embodiment of the imaging device, the mirror element is configured to deflect in response to the second radiation. In particular, the second radiation comprises a larger wavelength than the first radiation.

[0018] In accordance with at least one embodiment of the imaging device, the internal light source is configured to illuminate the detector array with a third radiation.The third radiation may have a peak emission wavelength in the ultraviolet, visible or near infrared spectral range.

[0019] According to at least one embodiment of the imaging device, at least one of the sub-pixel types, for example exactly one or at least two or all sub-pixel types, is configured to detect a third radiation deflected by the micromirror array. Thus, both the first radiation and the third radiation can be detected by the same detector array, wherein the third radiation allows obtaining an image corresponding to the second radiation.

[0020] In at least one embodiment of the imaging device, the imaging device includes a detector having a plurality of pixels, wherein the pixels include a plurality of sub-pixel types. The imaging device also includes a micromirror array having a plurality of mirror elements and an internal light source. At least one of the sub-pixel types is configured to detect a first radiation. The mirror element is configured to deflect in response to a second radiation. The internal light source is configured to illuminate the micromirror array with a third radiation. At least one of the sub-pixel types is configured to detect a third radiation deflected by the micromirror array.

[0021] Therefore, the imaging device uses the same detector array, such as a commercial CMOS imaging sensor combined with a color filter array, to capture both visible and thermal images simultaneously. For example, a full-color visible image is captured using pixels provided with color filters that transmit red, green, and blue radiation, respectively. The thermal image is captured by encoding the incident thermal image onto the light of the internal light source. The encoded radiation of the internal light source is detected using at least one sub-pixel type of the detector array that is sensitive to a third radiation. In other words, the thermal image is encoded onto the position of the light from the internal light source on the detector array. Therefore, the incident thermal radiation modulates the position of the internal radiation on the detector array.

[0022] For example, the micromirror array comprises an optical lever or cantilever having at least two layers of different materials to form a bi-material, for example a bimetallic heat detector. On the side of the micromirror array facing away from the second radiation, the mirror element can be provided with a coating reflecting the third radiation of the internal light source. For example, the coating comprises gold.

[0023] Therefore, a single detector array is sufficient to obtain visible images and thermal images simultaneously, so that there is no need to provide two different cameras, i.e., one camera for thermal radiation and one camera for visible radiation. Using a two-camera system will translate into higher system costs, a bulky system, and increased power consumption. In addition, a two-camera system will result in an additional software burden for identifying features within the image for overlapping of visible light images and thermal images. In contrast, according to the present application, simultaneous visible and thermal imaging using the same detector array and the same readout circuit simplifies the entire system. This reduces cost, size, and power consumption while allowing simplified readout and highly accurate overlap between visible and thermal images.

[0024] According to at least one embodiment of the imaging device, the first radiation comprises radiation in the visible spectral range. For example, the detector array comprises sub-pixels for the red spectral range, the green spectral range and the blue spectral range. The detector array may also comprise sub-pixels sensitive to ultraviolet or near infrared radiation. These sub-pixels may be sensitive only to ultraviolet or near infrared radiation, or to visible radiation and ultraviolet or near infrared radiation.

[0025] According to at least one embodiment of the imaging device, the second radiation comprises thermal radiation. Specifically, the thermal radiation comprises radiation having a wavelength between 7µm and 14µm (inclusive).

[0026] According to at least one embodiment of the imaging device, at least one sub-pixel type configured to detect the third radiation is also sensitive to at least part of the first radiation. Thus, at least one sub-pixel type can be used to detect part of the first radiation and to detect the third radiation. Two or more sub-pixel types or even all sub-pixel types can be sensitive to the third radiation. This helps to increase the spatial resolution for detecting the third radiation.

[0027] According to at least one embodiment of the imaging device, at least one sub-pixel type configured to detect the third radiation is insensitive to the first radiation. For example, at least one sub-pixel type is provided in addition to the sub-pixel type configured to detect the first radiation.

[0028] According to at least one embodiment of the imaging device, at least one sub-pixel type is configured to detect near-infrared radiation included in the first radiation. Therefore, the sub-pixel type can be used to directly obtain a NIR image. For example, the sub-pixel type is insensitive to the third radiation of the internal light source. For example, the imaging device includes two different sub-pixel types that are sensitive to two different wavelengths in the near-infrared. For example, the difference between the two peak detection wavelengths in the near-infrared is at least 50nm or at least 100nm. For example, one sub-pixel type is configured to directly detect the near-infrared radiation included in the first radiation, and the other sub-pixel type is configured to detect the near-infrared radiation from the internal light source.

[0029] According to at least one embodiment of the imaging device, the imaging device includes a first lens configured to direct the first radiation onto the detector array and / or a second lens configured to direct the second radiation onto the micromirror array. Specifically, the first lens and the second lens are arranged side by side in a top view of the imaging device. In other words, the first lens and the second lens do not overlap on the imaging device in a top view. Specifically, the first lens and the second lens are arranged and configured so that they image the same scene.

[0030] The first lens and / or the second lens may be a single lens or a multi-lens configuration. The first lens and / or the second lens may be configured as a conventional lens or a metal lens of a transmissive bulk material. For example, the metal lens may include a dielectric material (e.g., titanium dioxide, niobium pentoxide, or silicon nitride) or a semiconductor material (e.g., silicon or metal). The metal lens may include a structure similar to a column or a groove or a hole, an H, U, V, a plus sign (+), or a cross-shaped structure. For example, the height of the structure is between 500nm and 700nm. For example, the maximum lateral extent or diameter of the structure is between 40nm and 400nm, including 40nm and 400nm. For example, the period of the structure is between 180nm and 450nm, including 180nm and 450nm.

[0031] In addition, the first lens and / or the second lens may also be a microlens array. The microlenses of the microlens array may also be implemented as metal lenses.

[0032] Specifically, the first lens and the second lens are aligned such that they both image the same scene.For example, the first lens and / or the second lens may be coated with an anti-reflection coating for transmitting radiation through the first lens and / or the second lens.

[0033] For example, the second lens overlaps the mirror array onto the imaging device in a top view.

[0034] However, the second lens and the micromirror array may be arranged with an offset to facilitate better overlap of the images of the reference first radiation and the second radiation.

[0035] For example, the second radiation and the third radiation impinge on the micromirror array from opposite directions.

[0036] In accordance with at least one embodiment of the imaging device, a first beam splitter is arranged between the detector array and the first lens, and a second beam splitter is arranged between the internal light source and the micromirror array.

[0037] For example, the first beam splitter and / or the second beam splitter may be configured as a dichroic beam splitter.For example, the first beam splitter is configured to transmit the first radiation and reflect the third radiation, for example at an incident angle of 45°.

[0038] According to at least one embodiment of the imaging device, the detector array and the internal light source are mounted side by side on a common substrate. This contributes to a compact design of the imaging device.

[0039] According to at least one embodiment of the imaging device, the internal light source is configured to emit the third radiation having a predetermined pattern, for example a dot pattern.

[0040] A comparison between the detected dot pattern on the detector array and the calibrated dot pattern can be used to determine an image belonging to the second radiation.

[0041] According to at least one embodiment, the internal light source includes an emitter and a dot pattern generator arranged downstream of the emitter.

[0042] For example, the dot pattern generator is a diffractive optical element (DOE).

[0043] According to at least one embodiment of the imaging device, the internal light source comprises an array of emitters configured to emit a plurality of individual light beams. For example, the emitter or the array of emitters is configured to radiate incoherently or coherently.

[0044] If an emitter array is used as emitter, an additional dot pattern generator can be omitted.

[0045] According to at least one embodiment of the imaging device, the imaging device is configured to be operable in a low power mode. For example, only a subset of the plurality of sub-pixels is operated in the low power mode. For example, the subset is a random selection of sub-pixels or corresponds to a predefined selection. The subset may include 10% or less, or 5% or less, or 1% or less of the total number of sub-pixels of the imaging device. The reduction in the number of sub-pixels operated allows for a significant reduction in power consumption compared to a conventional operating mode in which all sub-pixels are operated.

[0046] A change in the signal obtained from a selected subset of pixels during the low power mode can trigger a switch to a regular operating mode with increased spatial resolution using all sub-pixels or at least using an increased number of sub-pixels. For example, the low power mode is used in low power mode, such as human presence monitoring or occupancy monitoring at thermal wavelengths. This can be followed by a full power mode or a full color mode and / or a full resolution thermal imaging mode. This helps to implement features for object detection, as color imaging can be captured with better spatial resolution than thermal imaging.

[0047] According to at least one embodiment of the imaging device, only one of the sub-pixel types is operated in low-power mode. For example, only the sub-pixel type configured to detect the third radiation is operated in low-power mode. Thus, a change in the second radiation (e.g., thermal radiation) that causes a change in the deflection of one or more of the mirror elements can trigger a switch to a normal operating mode. However, in other embodiments, two or more sub-pixel types can be operated in low-power mode.

[0048] For example, at least two sub-pixels associated with one mirror element are operated in low power mode. This may apply to all mirror elements or to at most 90%, or to at most 70%, or to at most 50% and / or to at least 0.1%, or to at least 1%, or to at least 5%, or to at least 10% of the mirror elements.

[0049] Furthermore, a method for multispectral imaging is specified. The method can be performed using the above-mentioned imaging device. Therefore, features described in conjunction with the imaging device can also be applied to the method, and vice versa.

[0050] According to at least one embodiment of the method, the method comprises the steps of providing an imaging device comprising a detector array having a plurality of pixels, the pixels comprising a plurality of sub-pixel types. The imaging device further comprises a micromirror array and an internal light source, the micromirror array having a plurality of mirror elements configured to deflect in response to the second radiation.

[0051] According to at least one embodiment of the method, the method comprises the step of obtaining a first image using at least one sub-pixel type responsive to the first radiation.For example, the first image is a full-color image in the visible spectral range.

[0052] According to at least one embodiment of the method, the method comprises the step of illuminating the micromirror array with a third radiation emitted by an internal light source.

[0053] According to at least one embodiment of the method, the method comprises the step of detecting the third radiation reflected by the micromirror array using at least one sub-pixel type responsive to the third radiation.

[0054] According to at least one embodiment of the method, the method comprises the step of obtaining a second image corresponding to the second radiation based on the detected third radiation.

[0055] In at least one embodiment, the method of multispectral imaging comprises the following steps:

[0056] a) providing an imaging device, the imaging device comprising:

[0057] - a detector array having a plurality of pixels comprising a plurality of sub-pixel types,

[0058] a micromirror array having a plurality of mirror elements configured to deflect in response to the second radiation, and

[0059] - internal light source;

[0060] b) obtaining a first image using at least one sub-pixel type responsive to the first radiation;

[0061] c) illuminating the micromirror array with a third radiation emitted by an internal light source;

[0062] d) detecting third radiation reflected by the micromirror array using at least one sub-pixel type responsive to the third radiation; and

[0063] e) obtaining a second image corresponding to the second radiation based on the detected third radiation.

[0064] Using the described method, a first image and a second image can be obtained by a common detector array even if the light-sensitive material of the detector array itself is not capable of detecting the second radiation. For example, the method allows obtaining a first image in the visible spectral range and obtaining a thermal image as a second image using the same detector array.

[0065] According to at least one embodiment of the method, the step of obtaining the second image comprises comparing the detected dot pattern of the third radiation with a calibrated dot pattern. For example, the calibrated dot pattern refers to the situation where the micromirror array is exposed to the second radiation with predetermined spatial characteristics.

[0066] According to at least one embodiment of the method, the step of obtaining the second image comprises determining an intensity of the second radiation for a pixel of the detector array based on deviations between the detected dot pattern and the calibrated dot pattern.

[0067] According to at least one embodiment of the method, the steps of obtaining the first image and illuminating the micromirror array with the third radiation are performed simultaneously using different sub-pixel types for the first radiation and the third radiation. For example, at least one sub-pixel type of the detector array is configured to be sensitive to the third radiation but not to the first radiation.

[0068] According to at least one embodiment of the method, the steps of obtaining the first image and illuminating the micromirror array with the third radiation are performed using time multiplexing based on at least one sub-pixel type of the first radiation and the third radiation.

[0069] For example, the internal light source is operated in a pulsed on / off operation such that the on time can be used to detect the third radiation impinging on the detector array to obtain the second image, and the off time can be used to obtain the first image.

[0070] Further features and conveniences will become apparent from the following description of exemplary embodiments in conjunction with the accompanying drawings.

[0071] In the exemplary embodiments and the accompanying drawings, components with similar or similar functions have the same reference numerals. Generally, only the differences relative to the various exemplary embodiments are described. Unless otherwise specified, the description of a part or aspect in one embodiment also applies to the corresponding part or aspect in another embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In the attached picture:

[0073] FIG. 1 shows an exemplary embodiment of an imaging device;

[0074] FIG. 2 shows a schematic representation of the operation of an imaging device according to an exemplary embodiment;

[0075] FIG3 illustrates an exemplary embodiment of a micromirror array;

[0076] 4A and 4B show an exemplary embodiment of a mirror element in a top view ( FIG. 4A ) and a side view ( FIG. 4B );

[0077] 4C and 4D show an exemplary embodiment of a mirror element in a top view ( FIG. 4C ) and a side view ( FIG. 4D );

[0078] 5A and 5B illustrate exemplary embodiments of a calibrated dot pattern ( FIG. 5A ) and a corresponding detected dot pattern ( FIG. 5B ) during operation;

[0079] FIG. 6 illustrates an exemplary embodiment of an imaging device;

[0080] FIG. 7 illustrates an exemplary embodiment of an imaging device;

[0081] FIG8 shows an exemplary embodiment of an imaging device;

[0082] FIG. 9 illustrates an exemplary embodiment of an imaging device;

[0083] FIG. 10A shows an exemplary embodiment of a filter array with corresponding transmission curves of the filters as a function of wavelength λ in FIG. 10B ;

[0084] FIG. 11A shows an exemplary embodiment of a filter array with corresponding transmission curves of the filters as a function of wavelength λ in FIG. 11B ;

[0085] FIG. 12A shows an exemplary embodiment of a filter array with corresponding transmission curves of the filters as a function of wavelength λ in FIG. 12B ;

[0086] FIG. 13A shows an exemplary embodiment of a filter array with corresponding transmission curves of the filters as a function of wavelength λ in FIG. 13B ;

[0087] FIG. 14 illustrates an exemplary embodiment of a multispectral imaging method. DETAILED DESCRIPTION

[0088] The elements shown in the figures and their size relations to each other are not necessarily true to scale. On the contrary, for better representation and / or for better understanding, the individual elements or layer thicknesses may be represented with exaggerated sizes.

[0089] An exemplary embodiment of an imaging device 1 is schematically shown in FIG1 . The imaging device 1 comprises a detector array 2 having a plurality of pixels 25, wherein the pixels 25 each comprise a plurality of sub-pixel types 21. A filter array 29 is arranged on the detector array in order to obtain different spectral sensitivity distributions of the sub-pixel types 21. For easier representation, pixels and sub-pixels are not explicitly shown in FIG1 . Possible arrangements of sub-pixels 21 within a pixel 25 are described in conjunction with FIGS. 10A to 13B .

[0090] The imaging device 1 further comprises a micromirror array 3 having a plurality of mirror elements 31. The micromirror array 3 may be one-dimensional or two-dimensional.

[0091] The imaging device 1 further comprises an internal light source 4 .

[0092] At least one of the sub-pixel types 21 (e.g., three sub-pixel types) is configured to detect first radiation R1. For example, one sub-pixel type 21 is configured to detect radiation in the blue spectral range, one sub-pixel type 21 is configured to detect radiation in the green spectral range, and one sub-pixel type 21 is configured to detect radiation in the red spectral range.

[0093] The radiation R1 is directed onto the detector array 2 using a first lens 51. The first lens 51 is shown as a single lens in Fig. 1, but it may also comprise two or more lenses.

[0094] The imaging device 1 further comprises a second lens 52 arranged laterally beside the first lens 51. The second lens 52 is configured to direct the second radiation R2, for example radiation in the long-wave infrared range, onto the micromirror array 3.

[0095] The first lens 51 and / or the second lens 52 may be a single lens or a multi-lens configuration. The first lens 51 and / or the second lens 52 may be configured as a conventional lens of a transmissive bulk material or a metal lens. For example, the metal lens may include a dielectric material (e.g., titanium dioxide, niobium pentoxide, or silicon nitride) or a semiconductor material (e.g., silicon or metal). The metal lens may include a structure similar to a column or a groove or a hole, an H, U, V, a plus sign (+), or a cross-shaped structure. For example, the height of the structure is between 500nm and 700nm, including 500nm and 700nm. For example, the maximum lateral extent or diameter of the structure is between 40nm and 400nm, including 40nm and 400nm. For example, the period of the structure is between 180nm and 450nm, including 180nm and 450nm.

[0096] In addition, the second lens 52 may also be a micro lens array. The micro lens array may also be implemented as a metal lens array.

[0097] The mirror element 31 is configured to deflect in response to the second radiation R2. The internal light source 4 is configured to illuminate the micromirror array with a third radiation R3. At least one of the sub-pixel types 21 is configured to detect the third radiation R3 reflected by the micromirror array 3.

[0098] For example, a simultaneous readout of a thermal image belonging to the radiation R2 and an optical image belonging to the radiation R1 may be obtained from the same detector array 2 configured as a CMOS image sensor.

[0099] The thermal image is read out from the differences in the dot positions of the dot pattern compared to a calibrated dot pattern belonging to the calibration positions of the mirror element 31 .

[0100] This is illustrated in Figures 5A and 5B, where a calibrated dot pattern 71 is shown and a corresponding detected dot pattern 72 detected during normal operation is shown in Figure 5B. The thermal energy of the second radiation R2 bends the mirror elements 31 of the micromirror array 3. The differences in the positions of the individual dots of the dot pattern on the detector array 2 allow the calculation of a corresponding thermal image belonging to the second radiation R2.

[0101] As shown in Figure 1, a lens 43 may be placed between the internal light source 4 and the micromirror array 3. This helps to obtain sharp spots on the detector array 2, thereby improving the quality of the thermal image obtained.

[0102] For example, the internal light source 4 emits with a peak emission wavelength in the near infrared range, enabling the detector array 2 with silicon based photosensitive areas to detect the third radiation R3.

[0103] However, the internal light source 4 may also emit in a different spectral range, for example in the UV range, as long as the radiation can be detected by at least one sub-pixel type 21 of the detector array 2 .

[0104] In the exemplary embodiment of FIG. 1 , an optical filter 53 is arranged in the beam path of the second radiation R2 to the micromirror array 3, for example downstream of the second lens 53. For example, the optical filter 3 transmits radiation in the long-wave infrared range and blocks radiation in the visible range. However, such an optical filter 53 is optional. The same applies to all further exemplary embodiments described subsequently.

[0105] The imaging device 1 uses two lenses, namely a first lens 51 and a second lens 52, to produce separate imaging channels for a first radiation R1 (e.g., visible radiation) and a second radiation R2 (e.g., thermal radiation in the long-wave infrared spectral range). The first lens 51 and the second lens 52 are spaced apart from each other and are designed and aligned so that they both image the same scene.

[0106] The first lens 51 and / or the second lens 52 may be optionally coated with an anti-reflection coating to obtain better performance in the corresponding spectral range. The optical filter 53 may be used to block any radiation other than long-wave infrared radiation from impinging on the micromirror array 3.

[0107] For example, the mirror element 31 is a cantilever. The optical filter 53 may help reduce measurement noise in the long wave infrared image.

[0108] A first lens is placed above the detector array 2 to form an imaging path for the first radiation R1 , for example visible radiation in the spectral range between 400 nm and 700 nm. The first radiation R1 to be detected may also include radiation in the near infrared spectral range.

[0109] The first lens 51 can be placed directly above the detector array 2, or with a certain offset to meet the design conditions, so as to align the images belonging to the first radiation R1 and the second radiation R2, respectively. The captured visible image belonging to the first radiation R1 can be used as a reference frame for setting and / or aligning the imaging components used to detect the second radiation R2.

[0110] An exemplary embodiment of a micromirror array 3 is shown in FIG3 . The mirror elements 31 are microcantilevers arranged in a rectangular grid. The mirror elements 31 each extend away from the frame 39 and are free to reflect in response to impinging thermal radiation. Thus, the mirror elements 31 represent micro-optical mechanical (MOM) devices, such as dual-material microcantilevers, such as bimetallic microcantilevers.

[0111] The size and spacing of the mirror elements 31 are determined by the required resolution and sensitivity of the image (eg thermal image) belonging to the second radiation. The mirror elements 31 can also be arranged one-dimensionally in a row. Further details of the mirror elements 31 are described in conjunction with Figures 4A to 4D.

[0112] The light of the internal light source 4, ie the third radiation R3, impinges on the side of the micromirror array 3 facing away from the second lens 52. The third radiation R3 is reflected by the micromirror array 3 onto the detector array 2.

[0113] When the side of the mirror element 31 facing the second lens 52 is illuminated by radiation R2 (e.g., thermal energy), the mirror element 31 (e.g., cantilever) bends, thereby deflecting points of the third radiation R3 to different locations on the detector array 2. This is illustrated in Figures 5A and 5B, which show how the arrangement of points on the detector array 2 changes as a response to incident thermal energy, mediated by the bending of the microcantilever array in response to the incident thermal energy. By tracking the position of each point over time, a thermal image can be reconstructed simultaneously with a visible image from the same detector array 2. The sensitivity of the micromirror array and the total path length of the deflected internal light determine the limits of the movement of the points on the detector array 2.

[0114] FIG2 schematically represents an overview of the operation of the device. The detector array 2 is illuminated by radiation R1 (e.g. visible light) from a scene. A second radiation R2 (e.g. long-wave infrared radiation) from the same scene is focused on the micromirror array (block S1). This causes the mirror element 31 to bend (block S2). Thus, the long-wave infrared radiation bends the mirror element 31 (block S3). The micromirror array 3 is illuminated by radiation R3 from an internal light source 4. The deflected third radiation R3 impinges on the detector array 2, wherein at least one type of subpixel 21 is sensitive to the radiation R3 of the internal light source 4.

[0115] 4A to 4D show exemplary embodiments of possible mirror elements 31 in top view (FIG. 4A and FIG. 4C) and side view (FIG. 4B and FIG. 4D). Mirror element 31 comprises a first material 35 and a second material 36. The frame is also made of the second material 36. For example, the first material is gold and the second material is silicon and / or silicon nitride. Other materials can also be used as the first material and the second material, as long as there is a sufficient difference in the thermal linear expansion coefficients of the materials used. In the exemplary embodiment shown in FIG. 4A, mirror element 31 has essentially a rectangular shape extending away from frame 39. In the exemplary embodiment of FIG. 4C, mirror element 31 has a triangular shape.

[0116] The geometry of the mirror element 31 can be modified within wide ranges, as long as the shock heat energy leads to a deflection of the mirror element 31 .

[0117] Optionally, the imaging device 1 is configured to be operable in a low power mode. For example, only a subset of a plurality of sub-pixels is operated in a low power mode. For example, the subset is a random selection of sub-pixels or corresponds to a predefined selection. The subset may include 10% or less, or 5% or less, or 1% or less of the total number of sub-pixels of the imaging device 1. The reduction in the number of sub-pixels operated allows a significant reduction in power consumption compared to a conventional operating mode in which all sub-pixels are operated. For example, if a 40×40 matrix of pixels 25 is assigned to one mirror element 31, the number of pixels activated during the low power mode may be 10 or less, such as 2, 3, 5, or 8, so that less than 1% of the pixels assigned to the mirror element 31 are operated in a low power mode. Of course, the number of mirror elements 31, the number of pixels 25 assigned to one mirror element 31, and the number of pixels of each mirror element 31 activated during the low power mode may vary over a wide range.

[0118] For example, only the sub-pixel types sensitive to the third radiation R3 are operated in low power mode, so that a change in thermal radiation (e.g. due to an approaching animal or person) can trigger a switch to a regular operating mode, e.g. performing high-resolution imaging in monochrome or full color, so that now high-quality imaging can be achieved for object recognition, identification, etc. The low power mode can be used for human presence monitoring, especially for subsequent imaging, for example.

[0119] However, in addition to or instead of thermal radiation, radiation in the visible spectral range can also be detected in low-power mode and trigger a switching event using the corresponding sub-pixel type.

[0120] The described low power mode may also be applied to further exemplary embodiments.

[0121] The exemplary embodiments shown in Fig. 6 and Fig. 7 differ in the way in which a radiation pattern is formed for the third radiation R3 provided by the internal light source 4. In the exemplary embodiment of Fig. 6, the internal light source 4 comprises an emitter 41. The emitter 41 generates a uniform illumination which illuminates a point pattern generator 42, for example a diffractive optical element.

[0122] In the exemplary embodiment of Fig. 7, the emitter of the internal light source 4 is an emitter array 44 that generates an array of collimated light beams. For example, the emitter array 44 is an array of laser diodes or vertical cavity surface emitting laser (VCSEL) diodes. In this case, the additional dot pattern generator can be omitted.

[0123] The exemplary embodiment of Fig. 8 corresponds substantially to the embodiment described in connection with Fig. 1, Fig. 6 and Fig. 7. Here, the internal light source 4 and the detector array 2 are arranged on a common substrate 9. For example, the internal light source 4 and the detector array 2 are configured as surface mounted devices.

[0124] Unlike the previous exemplary embodiments, the imaging device 1 further comprises a first beam splitter 61 arranged between the detector array 2 and the first lens 51 and a second beam splitter 62 arranged between the internal light source 4 and the second lens 52. This results in a very compact arrangement of the imaging device 1, making it suitable also for handheld devices (e.g. smartphones) or home automation devices that need to be implemented in a compact package.

[0125] The first beam splitter 61 and / or the second beam splitter 62 may be implemented as a dichroic or non-dichroic beam splitter. For example, the first beam splitter 61 is configured such that it allows the first radiation R1 (e.g. visible radiation) to pass through and allows reflection of the third radiation R3, which may have a longer wavelength than the first radiation R1. For the first radiation R1 and for the third radiation R3, the angle of incidence on the first beam splitter 61 is about 45°.

[0126] 8 , the third radiation R3 emitted by the internal light source 4 illuminates the micromirror array 3 through the second beam splitter 61. The radiation reflected by the micromirror array 3 is at least partially reflected by the second beam splitter 62 and then reflected by the first beam splitter 61, so that it impinges on the detector array 2. The first radiation R1 is transmitted through the first beam splitter 61 and also impinges on the detector array 2.

[0127] For example, the first beam splitter 61 and the second beam splitter 62 are arranged at an angle of 45° with respect to the mounting side of the substrate 9 .

[0128] For example, the first beam splitter 61 and / or the second beam splitter 62 are implemented as a thin film coating on a suitable substrate.

[0129] The exemplary embodiment shown in FIG. 9 substantially corresponds to the exemplary embodiment of FIG. 8 .

[0130] In this exemplary embodiment, the detector array 2 is configured to detect the near infrared portion (labeled NIR in FIG. 9 ) of the first radiation R1. Thus, the imaging device 1 can provide a full-color image for visible radiation, a second image for radiation R2 in the long-wave infrared range, and a third image of near infrared radiation NIR1 belonging to the first radiation R1. Examples of suitable filter arrays are described in conjunction with FIGS. 13A and 13B.

[0131] Direct detection of near infrared radiation may also be used in the exemplary embodiments described in conjunction with FIGS. 1 , 6 , and 7 .

[0132] 10A together with FIG. 10B illustrate an exemplary embodiment of a filter array 29 for a detector array 2. In this exemplary embodiment, the detector array 2 comprises three different sub-pixel types 21 for detecting radiation in the blue spectral range (marked B), the green spectral range (marked G) and the red spectral range (marked R). The three sub-pixel types 21 are arranged in a so-called Bayer mosaic pattern (BGGR) having one sub-pixel for blue radiation, one sub-pixel for red radiation and two sub-pixels for green radiation.

[0133] 10B , the color filter that transmits red radiation also transmits third radiation R3 in the near infrared range emitted by the internal light source 4. The blue color filter B and the green color filter G also transmit in the near infrared wavelength range NIR. Therefore, the transmission curves of the blue color filter and the green color filter have two transmission maxima that are spectrally spaced apart from each other.

[0134] Using this filter array 29, near infrared light emitted by the internal light source 4 can be detected by all sub-pixel types 21. This allows for high resolution detection of the radiation pattern of the third radiation R3 impinging on the detector array 2.

[0135] In this configuration, the internal light source 4 can be operated in a pulsed (on / off) operation at a specific frequency. When the internal light source 4 is off, the third radiation R3 of the internal light source 4 is not present on the detector array, so that during this time frame, the detector array 2 will only record an image of the radiation R1, for example, in the visible spectrum range. When the internal light source 4 in the near infrared is turned on, a near infrared point appears on the detector array. During this time frame, the imaging device 1 records the changes in the position of the near infrared point, thereby estimating the thermal image. Therefore, in the on / off cycle, the imaging device 1 is able to quickly and continuously record a first image in the visible range and a second image in the thermal range. This method is particularly suitable for cost and / or power sensitive applications.

[0136] In the exemplary embodiment shown in Figures 11A and 11B, the detector array 2 includes an additional sub-pixel type 21 that is sensitive to the radiation of the internal light source 4 in the near infrared spectral range (labeled NIR). In this embodiment, all color channels are available simultaneously, so that the imaging device 1 performs the capture of both thermal and visible images. Such an implementation allows high frame rate applications.

[0137] The exemplary embodiment shown in FIGS. 12A and 12B substantially corresponds to the exemplary embodiment of FIGS. 11A and 11B .

[0138] In contrast to the previous exemplary embodiment, the internal light source 4 does not emit in the near infrared spectral range, but in the ultraviolet spectral range. Therefore, in addition to the sub-pixel type 21 for visible radiation (R, G, B), the detector array 2 also includes a sub-pixel type for detecting ultraviolet radiation (UV).

[0139] The exemplary embodiment shown in FIGS. 13A and 13B allows simultaneous imaging in the visible spectral range, the near infrared range and the long-wave infrared range. In addition to the sub-pixel type 21 for the visible spectral range (R, G, B), a sub-pixel type NIR1 sensitive in the near infrared and another sub-pixel type NIR2 sensitive in the near infrared with a different NIR wavelength are provided. By selecting two NIR channels, one of which is dedicated to thermal imaging via the internal light source 4 and the other NIR channel is dedicated to imaging at the NIR wavelength provided in the first radiation R1.

[0140] For example, the peak wavelength of the internal light source 4 is emitted at 780nm, and the NIR2 channel is selected for imaging at a peak detection wavelength of 940nm. Of course, as described in conjunction with Figures 12A and 12B, it is also possible to use an internal light source 4 for emitting in the ultraviolet spectral range to obtain simultaneous imaging in the visible, near infrared and long-wave infrared ranges. In this case, only one sub-pixel type 21 sensitive in the NIR is required.

[0141] In the above exemplary embodiments, the detector array and the micromirror array are implemented as two-dimensional arrays. However, the micromirror array 3 and / or the detector array 2 may also be implemented as a one-dimensional line scanning array.

[0142] An exemplary embodiment of a method for multispectral imaging is shown in Figure 14. A detector array 2 having a plurality of pixels 25 is provided, wherein the pixels comprise a plurality of sub-pixel types 21. The imaging device 1 further comprises a micromirror array 3 having a plurality of mirror elements 31, wherein the mirror elements 31 are configured to deflect in response to the second radiation R2. The imaging device further comprises an internal light source 4 (step S11).

[0143] In step S12, a first image is obtained using at least one sub-pixel type responsive to the first radiation R1.

[0144] In step S13 , the micromirror array is illuminated with a third radiation R3 emitted by the internal light source 4 .

[0145] In step S14, the third radiation R3 reflected by the micromirror array 3 is detected using at least one sub-pixel type 21 responsive to the third radiation R3.

[0146] In step S15 , a second image corresponding to the second radiation R2 is obtained based on the detected third radiation R3 .

[0147] As described in connection with Figs. 5A and 5B, the step of obtaining the second image may comprise comparing the detected dot pattern of the third radiation R3 with a calibrated dot pattern.

[0148] Steps S12 and S13 may be performed in the time domain in an alternating manner as described in conjunction with FIGS. 10A and 10B , or simultaneously as described in conjunction with FIGS. 11A to 13B .

[0149] Using this method, a first image and a second image in different spectral ranges (e.g., in the visible spectral range and in the long-wave infrared range) can be obtained using the same detector array. In addition, a third image in the near-infrared spectral range can be provided.

[0150] The invention described herein is not limited to the description given with reference to the exemplary embodiments. On the contrary, the invention covers any novel feature and any combination of features, in particular including any combination of features in the claims, even if this feature or this combination itself is not explicitly pointed out in the claims or the exemplary embodiments.

[0151] Reference numerals list

[0152] 1 Imaging equipment

[0153] 2 Detector array

[0154] 21 Sub-pixel type

[0155] 25 pixels

[0156] 29 Filter Array

[0157] 3 Micromirror array

[0158] 31 Mirror element

[0159] 35 First Material

[0160] 36 Second Material

[0161] 39 Frame

[0162] 4 Internal light source

[0163] 41 Transmitter

[0164] 42 Dot Pattern Generator

[0165] 43 Lens

[0166] 44 Transmitter Array

[0167] 51 First lens

[0168] 52 Second lens

[0169] 53 Optical Filters

[0170] 61 First beam splitter

[0171] 62 Second beam splitter

[0172] 71 Calibrated dot pattern

[0173] 72 detected dot patterns

[0174] 9 substrate

[0175] R1 First Radiation

[0176] R2 Second Radiation

[0177] R3 Third Radiation

[0178] S1, S2, S3, S4 boxes

[0179] Steps S11, S12, S13, S14, S15

Claims

1. An imaging device (1), comprising: - a detector array (2) having a plurality of pixels (25), said pixels comprising a plurality of sub-pixel types (21), - a micromirror array (3) having a plurality of mirror elements (31), and - internal light sources (4), in, - at least one of the sub-pixel types (1) is configured to detect a first radiation (R1); - the mirror element (31) is configured to deflect in response to the second radiation (R2), - the internal light source (4) is configured to illuminate the micromirror array (3) with a third radiation (R3); - at least one of the sub-pixel types (21) is configured to detect the third radiation (R3) deflected by the micromirror array (3).

2. The imaging device according to claim 1, in, The first radiation (R1) comprises radiation in the visible spectral range.

3. The imaging device according to claim 1 or 2, in, The second radiation (R2) comprises thermal radiation.

4. An imaging device according to any one of the preceding claims, in, The imaging device (1) comprises a first lens (51) and a second lens (52), wherein the first lens (51) is configured to direct the first radiation (R1) onto the detector array (2), and the second lens (52) is configured to direct the second radiation (R2) onto the micromirror array (3), and the first lens (51) and the second lens (52) are arranged side by side in a top view of the imaging device (1).

5. The imaging device according to claim 4, in, A first beam splitter (61) is arranged between the detector array (3) and the first lens (51), and The second beam splitter (62) is arranged between the internal light source (4) and the micromirror array (3).

6. An imaging device according to any one of the preceding claims, in, The detector array (2) and the internal light source (4) are mounted side by side on a common substrate (9).

7. An imaging device according to any one of the preceding claims, in, The at least one sub-pixel type (21) configured to detect the third radiation (R3) is also sensitive to at least part of the first radiation (R1).

8. The imaging device according to any one of claims 1 to 6, in, The at least one sub-pixel type (21) configured to detect the third radiation (R3) is insensitive to the first radiation (R1).

9. An imaging device according to any one of the preceding claims, in, At least one sub-pixel type (21) is configured to detect near infrared radiation comprised in the first radiation (R1).

10. An imaging device according to any one of the preceding claims, in, The internal light source is configured to emit the third radiation having a dot pattern.

11. An imaging device according to any one of the preceding claims, in, The internal light source (4) comprises an emitter (41) and a dot pattern generator (42) arranged downstream of the emitter (41).

12. The imaging device according to any one of claims 1 to 10, in, The internal light source (4) comprises an array of emitters (44) configured to emit a plurality of individual light beams.

13. The imaging device according to any one of claims 1 to 12, in, The imaging device (1) is configured to be operable in a low power mode, wherein only a subset of the plurality of sub-pixels is operated in the low power mode.

14. The imaging device according to claim 13, in, Only one of the sub-pixel types operates in the low power mode.

15. A method for multispectral imaging, comprising the following steps: a) Providing an imaging device (1), the imaging device comprising: - a detector array (2) having a plurality of pixels (25), said pixels (25) comprising a plurality of sub-pixel types (21), - a micromirror array (3) having a plurality of mirror elements (31), the mirror elements (31) being configured to deflect in response to the second radiation (R2), and - Internal light sources (4); b) obtaining a first image using at least one sub-pixel type responsive to a first radiation (R1); c) illuminating the micromirror array (3) with a third radiation (R3) emitted by the internal light source (4); d) detecting the third radiation (R3) reflected by the micromirror array (3) using at least one sub-pixel type (21) responsive to the third radiation (R3); and e) obtaining a second image corresponding to said second radiation based on the detected third radiation (R3).

16. The method according to claim 15, in, Step e) comprises comparing the detected dot pattern (72) of the third radiation with a calibrated dot pattern (71).

17. The method according to claim 16, in, Step e) comprises determining the intensity of the second radiation (R2) for the pixel of the detector array (2) based on a deviation between the detected dot pattern (72) and the calibrated dot pattern (71).

18. The method according to any one of claims 15 to 17, in, Steps b) and c) are performed simultaneously for the first radiation (R1) and the third radiation (R3) using different sub-pixel types.

19. The method according to any one of claims 15 to 17, in, Steps b) and c) are performed using time multiplexing based on at least one sub-pixel type of the first radiation (R1) and the third radiation (R3).

20. The method according to any one of claims 15 to 19, in, The method is performed using an imaging device (1) according to any one of claims 1 to 14.