Method and system for manufacturing and using spectral basis filters - Patents.com

Abstract

The optical system includes a focal plane array having a plurality of pixels defined by a first number of pixels arranged in a first direction and a second number of pixels arranged in a second direction. The optical system also includes an optical filter optically coupled to the focal plane array. The optical filter has a plurality of superpixels. Each of the plurality of superpixels includes a predetermined number of subpixels, and each of the predetermined number of subpixels is characterized by one of a plurality of oscillatory transmission profiles as a function of wavelength.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 211,672, filed June 17, 2021, entitled "Spectral Basis Filters," and U.S. Patent Application No. 17 / 454,201, filed November 9, 2021, entitled "Method and System for Making and Using Spectral Basis Filters," the disclosures of which are incorporated by reference in their entireties herein. [Background technology]

[0003] Spectral imaging is a method for obtaining information related to the optical properties of an object as a function of wavelength. The method can be applied to a variety of applications, including remote sensing, product inspection, and medical imaging. Extending beyond the three color bands present in red, green, and blue (RGB) color images, a spectral image can contain hundreds or thousands of spectral bands. Line-scan (i.e., push-broom) spectral imaging, wavelength-scan (i.e., stare-in) spectral imaging, and single-shot (i.e., snapshot) spectral imaging are in use.

[0004]

[0003] Despite the advances that have been made in the area of ​​spectral imaging, there remains a need in the art for improved methods and systems related to spectral imaging. Summary of the Invention

[0005]

[0004] Embodiments of the present invention relate to image processing. More specifically, embodiments of the present invention provide methods and systems for spectral imaging. In certain embodiments, a method is provided for receiving image data and processing the image data to generate an image associated with a particular spectral profile. The particular spectral profile can be determined before or after image acquisition. A system for implementing the methods described herein is also provided by embodiments of the present invention.

[0006]

[0005] According to one embodiment of the present invention, an optical system is provided. The optical system includes a focal plane array having a plurality of pixels defined by a first number of pixels arranged in a first direction and a second number of pixels arranged in a second direction. The optical system also includes an optical filter optically coupled to the focal plane array. The optical filter has a plurality of superpixels. Each of the plurality of superpixels includes a predetermined number of subpixels, and each of the predetermined number of subpixels is characterized by one of a plurality of vibration transmission profiles as a function of wavelength. In some embodiments, the predetermined number of subpixels included in each of the plurality of superpixels is a square number, for example, the predetermined number of subpixels is equal to 9. Each of the predetermined number of subpixels can include one or more layers of a film, for example, a single thin film layer. In some embodiments, each of the predetermined number of subpixels can be characterized by a different thickness of the single thin film layer.

[0007] According to another embodiment of the present invention, there is provided a method for defining a spectral profile of an optical filter. The method includes determining one or more characteristics of the spectral profile. The method also includes defining a transmission profile for each of a plurality of sub-pixels, a predetermined number of the plurality of sub-pixels being characterized by an oscillatory transmission profile as a function of wavelength, and associating a weight from the set of relative weights with each of the plurality of sub-pixels.

[0008]

[0007] According to yet another embodiment of the present invention, a method is provided. The method includes transmitting an optical signal through an optical filter including a plurality of superpixels. Each of the plurality of superpixels includes a plurality of subpixels, and a predetermined number of the plurality of subpixels are characterized by a vibration transmission profile as a function of wavelength. The method also includes receiving intensity values ​​associated with each pixel of the detector, each pixel of the detector being associated with one of the plurality of subpixels. The detector may include a focal plane array. The method further includes, for each of the plurality of superpixels, applying a set of relative weights to corresponding intensity values ​​associated with the subpixels of the plurality of superpixels to determine output data.

[0009]

[0008] The embodiments of the present invention provide many benefits. For example, in some embodiments, an optical system using a spectral basis filter allows continuous control of bandpass edge definition by changing the relative weights of a linear combination of subpixel intensities in a superpixel block, resulting in an output image with a continuously adjustable spectral bandpass. The linear combination of subpixel intensities can be modified in real time or in post-processing. For example, a user of the optical system, or a computer, may determine that the contrast of the output image is not sufficient and modify the set of relative weights applied to the subpixel intensities to improve the contrast of the output image in real time or in post-processing. In this example, changing the set of relative weights also changes the output image.

[0010]

[0009] When the optical system is physically remote from the user (e.g., when the optical system is mounted on a remote machine or vehicle, including an airborne platform), the optical system may transmit intensity values ​​associated with the defined spectral bandpasses. This results in data transmission at a superpixel resolution (e.g., 1024×1024), which may be lower than the resolution of the focal plane array (e.g., 3072×3072), resulting in significant data compression. In other embodiments, image data is transmitted at the resolution of the focal plane array (e.g., 3072×3072) and post-processed to provide image data for any particular spectral bandpass at a lower resolution (e.g., 1024×1024).

[0011]

[0010] Furthermore, embodiments of the present invention provide high sensitivity, for example half the sensitivity of a superpixel with a defined bandpass. In some implementations, the optical system may be completely passive without using any electromechanical elements, reducing the risk of mechanical failure in generating image data. These and other embodiments of the present invention, along with their many advantages and features, will be described in more detail in conjunction with the following description and accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] 1 is an exemplary superpixel configuration of a spectral basis filter according to an embodiment of the present invention. [Diagram 2] 1 is a graph of transmittance as a function of wavelength for multiple subpixels in a superpixel in accordance with one embodiment of the present invention. [Figure 3A] 1 is a graph showing the reconstruction of a square wave function by a linear combination of basis function elements. [Figure 3B] 1 is a graph showing the reconstruction of a square wave function by a linear combination of basis function elements. [Figure 3C] 1 is a graph showing the reconstruction of a square wave function by a linear combination of basis function elements. [Figure 3D]1 is a graph showing the reconstruction of a square wave function by a linear combination of basis function elements. [Figure 4A] 5A-5C are plots showing spectral profiles as a function of wavelength for a longpass filter and a notch filter, respectively, in accordance with an embodiment of the present invention. [Figure 4B] 5A-5C are plots showing spectral profiles as a function of wavelength for a longpass filter and a notch filter, respectively, in accordance with an embodiment of the present invention. [Figure 5A] 1 is a plot illustrating the spectral profile of a bandpass filter as a function of wavelength exhibiting continuous bandedge variability, in accordance with an embodiment of the present invention. [Figure 5B] 1 is a plot illustrating the spectral profile of a bandpass filter as a function of wavelength exhibiting continuous bandedge variability, in accordance with an embodiment of the present invention. [Figure 5C] 1 is a plot illustrating the spectral profile of a bandpass filter as a function of wavelength exhibiting continuous bandedge variability, in accordance with an embodiment of the present invention. [Figure 5D] 1 is a plot illustrating the spectral profile of a bandpass filter as a function of wavelength exhibiting continuous bandedge variability, in accordance with an embodiment of the present invention. [Figure 6] 4 is a table showing coefficients used to generate a longpass filter, a notch filter, and multiple bandpass filters according to one embodiment of the present invention. [Figure 7] 4 is a graph of transmittance as a function of wavelength for each sub-pixel in a superpixel according to one embodiment of the present invention. [Figure 8] FIG. 1 shows a block diagram of an optical system with a spectral basis filter. [Figure 9] FIG. 2 is a cross-sectional view illustrating three sub-pixels of a superpixel according to one embodiment of the present invention. [Figure 10] 4 is a simplified flow chart illustrating a method for defining a spectral profile of an optical filter according to an embodiment of the present invention. [Figure 11] 1 is a simplified flow chart illustrating a method according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013]

[0022] Embodiments of the present invention relate to image processing. More particularly, embodiments of the present invention provide methods and systems for spectral imaging. In certain embodiments, methods are provided for receiving image data and processing the image data to generate an image associated with a particular spectral profile. The particular spectral profile can be determined before or after image acquisition. Systems for implementing the methods described herein are also provided by embodiments of the present invention.

[0014]

[0023] As described herein, embodiments of the present invention provide spectral basis filters that allow arbitrary definition of the spectral bandpass of a focal plane array that can be changed on command during operation. Embodiments of the present invention provide some or all of the functionality of a hyperspectral imager in a more compact form than currently available using conventional techniques. Additionally, embodiments of the present invention significantly reduce the bandwidth required to transmit measurement data. Spectral contrast enhancement can be performed automatically in real time, either locally or for the entire imaging array, similar to the implementation of automatic gain control. In some embodiments, the spectral contrast enhancement provided by embodiments of the present invention is useful for target detection, recognition, and / or identification. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0015]

[0024] Embodiments of the present invention can provide a reconfigurable color imager (RCI) applicable to a variety of spectral imaging systems. As described herein, embodiments of the present invention allow the spectral bandpass profile of a broadband imager to be changed, for example, during operation. By way of example only, the spectral bandpass profile of a mobile spectral imager located on an aircraft can be modified during flight to first transmit light over a first spectral range, for example, within a spectral band of 8.0 μm to 10.5 μm, and subsequently transmit light over a second spectral range, for example, within a spectral band of 8.0 μm to 10.5 μm. Thus, embodiments of the present invention allow the spectral bandpass to be modified in real time during operation. For example, first image in a spectral band having a bandwidth of ~0.8 μm centered at 9.4 μm, and then subsequently image in a spectral band having a bandwidth of ~0.8 μm centered at 9.5 μm. This ability to modify the spectral bandpass allows the system operator to increase the contrast of the image by filtering out spectral bands that are saturating the spectral imager or by emphasizing similar specific spectral bands.

[0016]

[0025] In some embodiments, 16 bands are used to perform multispectral imaging, however, embodiments of the present invention are not limited to this number of bands and may use fewer or additional bands. For simplicity and clarity, the exemplary embodiments described herein use 9 bands, however, this is merely exemplary. In contrast to some hyperspectral or multispectral imagers that use a push broom configuration to obtain two-dimensional image data as a function of wavelength, some embodiments of the present invention collect two-dimensional arrays of multispectral imaging data simultaneously or in parallel to support full array situational awareness in a single wavelength band and / or multiple wavelengths.

[0017]

[0026] FIG. 1 is a diagram of a spectral basis filter according to an embodiment of the present invention. In FIG. 1, a plan view of a spectral basis filter, which may also be referred to as an optical filter, is shown. The spectral basis filter 100 may be an optical filter including one or more superpixels. In the embodiment shown in FIG. 1, the superpixel 105 includes a predetermined number of subpixels arranged in an array configuration. In the embodiment shown in FIG. 1, a first number of subpixels 110, 120, 130, 140, 150, 160 or 170, 180 and 190 are arranged in a first direction and a second number of subpixels 110, 140, 170, 120, 150, 180 or 130, 160, 190 are arranged in a second direction, which may be orthogonal to the first direction. The superpixels may also be arranged in an array configuration (not shown) that defines the resolution of the spectral imager. Other spatial configurations of the subpixels may also be used.

[0018]

[0027] The spectral basis filter 100 shown in FIG. 1 is shown as a single superpixel 105 of size 30 μm×30 μm, thereby forming a spectral imager resolution (also referred to as display resolution) of 1024×1024. The illustrated superpixel 105 includes nine subpixels of size 10 μm×10 μm, shown in FIG. 1 as subpixels 110, 120, 130, 140, 150, 160, 170, 180, and 190, respectively. Thus, the resolution is considered at the subpixel level of the spectral basis filter 100 and is 3072×3072. The particular display and subpixel resolution shown in FIG. 1 are merely exemplary, and other resolutions can be used, for example, based on a 1024×1024 focal plane array (FPA) at 12 μm. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0019]

[0028] Each of the nine subpixels 110, 120, 130, 140, 150, 160, 170, 180, and 190 may be implemented as an optical filter characterized by an oscillating transmission profile as a function of wavelength, as discussed more fully in connection with FIG. 2. For example, the first subpixel 110 may have an oscillating transmission profile that transmits light incident on the first subpixel 110 according to an oscillating function. Thus, in contrast to a conventional optical filter that transmits a predetermined wavelength, e.g., a notch filter that transmits light in the wavelength range of 8.0 μm to 8.2 μm, one or more of the subpixels are characterized by an oscillating profile as a function of wavelength. Thus, as discussed more fully in connection with FIG. 2, one or more of the subpixels will have a transmission profile that oscillates many times between a minimum transmission value (e.g., zero transmission) and a maximum transmission value (e.g., total transmission) over the wavelength range of interest.

[0020]

[0029] Thus, in contrast to conventional optical filters that detect across a single spectral band, the subpixels discussed herein transmit light across the entire spectral band of the spectral imager, e.g., 8.0 μm to 10.5 μm, albeit with transmission values ​​that vary as a function of wavelength. Thus, a subpixel detector element, e.g., a pixel in a long wavelength infrared detector, detects light across the entire spectral band and transmits light to the detector element according to its oscillating spectral transmission profile.

[0021]

[0030] The output of the superpixel 105 may be formed as a linear superposition of the weighted intensities of the array of subpixels in the superpixel. The bandwidth of the superpixel may therefore be defined according to the weights applied to each of the subpixels, which are characterized by an oscillatory transmission profile, as discussed more fully herein. Thus, in contrast to conventional systems in which the bandwidth of the spectral imager is defined by the bandwidth of the spectral filters associated with the detector elements, embodiments of the present invention allow for modification of the bandwidth of the spectral imager as a result of the selection of a particular linear superposition of the weighted intensities of the array of subpixels in the superpixel. Thus, the bandwidth of the spectral imager may be modified after data collection, by a user of the optical system including the spectral basis filter 100, or by a processor of the optical system, at any time after data from the spectral basis filter 100 has been collected. For example, a particular spectral profile, such as one corresponding to a long-pass filter, may be applied to the output of the detector elements of the spectral imager using the spectral basis filter 100. Alternatively, a different particular spectral profile, e.g., a short-pass filter, may be implemented by post-processing of the same output from the detector elements of the spectral imager.

[0022]

[0031] As mentioned above, each of the multiple sub-pixels 110, 120, 130, 140, 150, 160, 170, 180 and 190 of the spectral basis filter 100 may have a unique vibration transmission profile, as will be discussed more fully in connection with Figure 2. Thus, the vibration transmission profile associated with each sub-pixel may then be used to determine a weight to be applied to the intensity value output from the detector element corresponding to each particular sub-pixel.

[0023]

[0032] While some embodiments are implemented in such a way that the detector elements associated with each subpixel detect only the light associated with it, in other embodiments, optical elements are utilized to blur the image formed by the detector elements in the focal plane array, resulting in a blur circle on the order of the size of the superpixel. Thus, when an optical filter is optically coupled to the focal plane array, light passing through the subpixels in a particular superpixel will be imaged approximately uniformly by the detector elements of the focal plane array associated with the superpixel. Thus, some implementations use optical systems that image at the size of the superpixel, rather than the size of the subpixel, thereby improving system performance.

[0024]

[0033] Figure 2 is a graph of transmittance as a function of wavelength for multiple subpixels in a superpixel in accordance with one embodiment of the present invention. As shown in Figure 2, a set of overlaid plots of transmittance as a function of wavelength are shown, each associated with one of the subpixels shown in Figure 1. Unlike many optical filters, such as Bayer filters, each subpixel can be constructed from layers of thin films, resulting in an oscillating transmission profile rather than a single transmission peak corresponding to a particular light band.

[0025]

[0034] Referring to FIG. 2, a graph 200 shows eight vibration transmission profiles 210, 220, 230, 240, 260, 270, 280 and 290 corresponding to the nine sub-pixels 110, 120, 130, 140, 160, 170, 180 and 190, respectively, of the spectral basis filter 100 of FIG. 1. In the embodiment illustrated in FIG. 2, each of the plurality of vibration transmission profiles is distinct from the others of the plurality of vibration transmission profiles. The amplitude of each of the eight vibration transmission profiles may vary from 0 to 1, but are shown offset in FIG. 2 for clarity. Thus, referring to the exemplary vibration transmission profile 210, the amplitude of the transmission ranges between 0 and 1 over the wavelength range of 8.0 μm to 10.5 μm. The amplitude of the transmission is 1 for 8.0 μm and 10.5 μm, and zero for 9.25 μm. The ninth transmission profile 250 is a uniform transmission profile with an amplitude of 1 (i.e., transmits all light equally). Thus, the subpixel 150 is transparent and transmits all wavelengths. Thus, the ninth transmission profile 250 corresponding to the subpixel 150 provides full situational awareness for any optical signal filtered by the spectral basis filter 100. Of course, the subpixel 150 can also have an oscillating profile in other embodiments.

[0026]

[0035] In the embodiment illustrated in FIG. 2, the oscillatory transmission profile 210 oscillates once over the wavelength range of 8.0 μm to 10.5 μm with a phase profile such that the transmission peaks are aligned at 8.0 μm and 10.5 μm. The oscillatory transmission profile 220 also oscillates once over the wavelength range of 8.0 μm to 10.5 μm but has a π radian phase shift relative to the oscillatory transmission profile 210. The oscillatory transmission profile 230 oscillates twice over the wavelength range of 8.0 μm to 10.5 μm and the oscillatory transmission profile 290 oscillates four times over the wavelength range of 8.0 μm to 10.5 μm. The oscillatory behavior of each of the transmission profiles is not limited to this particular set and each subpixel may be modified to provide additional sets of oscillatory behaviors. As shown in FIG. 2, each of the multiple oscillatory transmission profiles is measured in wavelengths measured in microns, cm -1 Each vibration profile is characterized by a unique vibration frequency and phase that can be referenced in terms of wave numbers measured in units of Hz, or frequency measured in Hertz. Thus, the illustrated vibration transmission profiles are exemplary, and embodiments of the invention are not limited to these specific nine vibration profiles. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0027]

[0036] The output intensity value of each sub-pixel may be determined by the oscillatory transmission profile of the sub-pixel, and the total transmittance of the super-pixel may be a linear combination of the weighted intensity values ​​of all the sub-pixels within the super-pixel. Thus, as described more fully herein, the output (i.e., intensity value) from each detector element associated with each sub-pixel is weighted and summed with the outputs from the other detector elements to form a weighted output at the super-pixel resolution.

[0028]

[0037] 3A-3D are graphs showing the reconstruction of a square wave function by a linear combination of basis functions. A basis for a vector space vector V is a subset of vectors v1, ..., v nAs will be appreciated by those skilled in the art, the set of vectors in V (v1, v2, ..., v n ) for every v∈V, the basis field v = a1v1+ a2v2+ ... + a n v n, where a1, ..., a n are elements that form a basis if and only if they can be uniquely described as functions in a space V. This is true not only for vectors in Euclidean space (x, y, z) in classical geometry, but also for any function in a space V. A transform expresses a function in V through a weighted linear combination of basis vectors. Examples of basis functions include polynomials (Taylor series), Bessel functions (Hankel), wavelets (Haar), and complex exponentials (Fourier transform). A spectral bandpass envelope defines a function in a one-dimensional vector space. A linear combination of a small number of terms in a suitable basis can efficiently approximate a spectral bandpass function. The fidelity of the approximation for a given number of terms depends largely on the suitability of the basis used for the approximation.

[0029]

[0038] As shown in Figures 3A-3D, if a set of functions can be used to reconstruct any function in a space, the set of functions can be a set of basis functions in the space. The set of basis functions can be used to calculate weights of certain basis functions to approximate a desired function. The basis functions used in Figure 3 are sine functions (e.g., sine and / or cosine functions of different amplitudes and phases) that are used to calculate a Fourier expansion of the square wave function 300. The first Fourier expansion 310 can be a Fourier expansion with only one basis function, which roughly approximates the square wave function 300 by approximating it with a single sine function. Increasing the order of the Fourier expansion to a second Fourier expansion 320 results in a better approximation of the square wave function 300, since another sine function is added to the first sine function according to a certain weight (e.g., w1*sin(x)+w 2* cos(x)). Higher order expansions, such as the third Fourier expansion 330 and the fourth Fourier expansion 340, may improve the accuracy of the approximation of the square wave function 300 as more sinusoidal terms are added to the approximation.

[0030]

[0039] In general, the Fourier expansion of a function f(x) with a period of 2L can be calculated using the following formula:

number

number

number

[0031]

[0040] Fourier expansion techniques can be applied when operating the spectral basis filters. A linear combination of the sub-pixel intensities can be used to efficiently construct the desired spectral profile. By varying the weights in the linear combination of the sub-pixel vibration transmission profiles, the spectral band edges can be adjusted continuously, in real-time, or in post-processing if the full array is read out.

[0032]

[0041] 4A and 4B are plots showing the spectral profile as a function of wavelength for a long pass filter and a notch filter, respectively, according to an embodiment of the present invention. As described above, the spectral profile of the spectral basis filter may be continuously adjusted by varying the relative weights of the sub-pixel intensity combinations of the spectral basis filter (e.g., as described for the vibration transmission profile of FIG. 2). In some implementations, if the complete array is read out, post-processing can be used to modify the relative weights after image acquisition, allowing the spectral profile as a function of wavelength to be modified after image capture.

[0033]

[0042] With reference to Figures 4A and 4B, subpixel oscillatory transmission profiles may be used in combination to approximate various optical filters. In Figure 4A, a longpass filter 400 is illustrated that is characterized by low transmission for wavelengths less than 9.25 μm and high transmission for wavelengths greater than 9.25 μm. Thus, by varying the relative weights of the basis vectors (i.e., subpixel intensities), a 9.25 μm longpass filter can be implemented. In Figure 4B, a notch filter 410 is illustrated that is characterized by low transmission for wavelengths less than 9.8 μm and high transmission between 9.8 μm and 10.2 μm. Thus, a 10 μm notch filter can be implemented. These filters are merely exemplary, and other filters having different bandwidths, center wavelengths, cutoff wavelengths, etc., can be implemented by embodiments of the present invention.

[0034]

[0043] Additionally, the filters may be adjusted in real-time when the full array is read out, or in post-processing. For example, a long-pass filter 400 may be implemented initially, but in post-processing, a notch filter 410 may be determined to be more suitable for imaging. By adjusting the weights in the linear combination of the sub-pixel vibration transmission profiles, the spectral profile may be fine-tuned to enhance contrast in an image read by an optical system including the spectral basis filter. In this manner, the present technique provides the benefits of hyperspectral (e.g., high spectral resolution) while having a low data load because the data used to adjust the spectral profile is a set of weights assigned to multiple sub-pixels. Furthermore, the sensitivity of the spectral basis filter (e.g., light collection efficiency multiplied by light collection area) is high because each sub-pixel passes half of the incident flux in the spectral bandpass. Thus, referring to FIG. 2, the light passing through each vibration transmission profile averages 50% quantum efficiency across the 8.0 μm to 10.5 μm band. Thus, in the 8.0 μm to 10.5 μm band, half of the incident light passes through the spectral basis filter. Furthermore, the light collection area (i.e., the overall size of the spectral basis filter) is large compared to conventional optical filters and can be comparable to the size of a single subpixel. Each superpixel, on the order of 30 μm×30 μm, transmits incident light with 50% quantum efficiency, thereby achieving high sensitivity by integrating light across the superpixel. Thus, embodiments of the present invention enable enhanced image contrast by providing continuous control over the spectral bandpass.

[0035]

[0044] 5A-5D are plots illustrating spectral profiles of bandpass filters as a function of wavelength exhibiting continuous bandedge variability according to an embodiment of the present invention. These figures show four spectral profiles as a function of wavelength of bandpass filters that provide continuous bandedge variation using the spectral basis filters described herein. In the exemplary spectral profiles shown in FIGs. 5A-5D, the spectral bandedges of the bandpass filters change continuously as the linear combination of the subpixel oscillation transmission profiles is modified.

[0036]

[0045] More specifically, a Fourier expansion can be calculated using the sub-pixel vibration transmission profile. For example, the function f(λ) can be approximated using the following equation:

number

number

number

number

[0037]

[0046] FIG. 6 is a table showing coefficients used to generate a long-pass filter, a notch filter, and a number of band-pass filters according to an embodiment of the present invention. In FIG. 6, the values ​​used for the spectral characteristics and weighting coefficients are illustrated for the long-pass filter 400 and the notch filter 410 shown in FIG. 4A and FIG. 4B, respectively, and the band-pass filters 500-530 shown in FIG. 5A-FIG. 5D, respectively. In this example, the nine coefficients a0-a4 and b1-b4 may be a set of relative weights corresponding to the nine sub-pixels 110, 120, 130, 140, 150, 160, 170, 180, and 190 of the spectral basis filter shown in FIG. 1. The constant coefficient a0 may be calculated, for example, using the sub-pixel 150 shown in FIG. 1. The sub-pixel 150 transmits all light equally, as shown by the ninth transmission profile 250 in FIG. 2, and can be used to calculate the required offset (a0) of the Fourier expansion.

[0038]

[0047] The example shown in Figures 5A-5D shows continuous band edge variability. The band edges are shown to vary by 0.05 μm between each band pass filter (e.g., the band edge corresponding to the first band pass filter 500 is shifted by 0.05 μm in the second band pass filter 510). However, the Fourier expansion may be varied to produce smaller or larger band edge variations. Since the vibration transmission function shown in Figure 2 is continuous, the band edges may be continuously adjusted by modifying the Fourier expansion. The spectral profile contained in the table shown in Figure 6 may be stored as a look-up table in the memory of an optical system with spectral basis filters. The table may be accessed as a look-up table, and one or more characteristics of the spectral profile, such as the band edge λ, may be selected. s , λ l , λ S , λ L One can receive relative weights a0-a4 and b1-b4 by inputting: One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0039]

[0048] FIG. 7 is a graph 700 of the transmittance as a function of wavelength for each subpixel in a superpixel according to an embodiment of the present invention. In FIG. 7, the transmittance as a function of wavelength for a subpixel constructed using a thin film is shown. A single layer of thin film may be used to construct the subpixels in the spectral basis filter. The thickness and refractive index of the thin film on each subpixel determines the vibration spectral transmission profile of the subpixel. The vibration as a function of wavelength can also be expressed in wavenumber or frequency. Thus, a similar calculation can be performed in wavenumber space where the vibration is sinusoidal. The transmission profiles shown in FIG. 7 correspond to film thicknesses of 0 μm, 3.5 μm, 3.8 μm, 8.5 μm, 8.8 μm, 13.5 μm, 13.8 μm, 18.5 μm, and 18.8 μm, respectively, for the nine subpixels. It is noted that when converting the spectral scale from wavenumber to wavelength, the vibration transmission is distorted on the long side of the bandpass, but generally follows the vibration transmission profile described in FIG. 2. Therefore, the technique of approximating different filters using Fourier expansions may be applied when subpixels are constructed using a single layer of thin film, hi other embodiments, multiple layers of thin film may be used to construct subpixels with similar vibration transmission profiles.

[0040]

[0049] In the embodiment shown in FIG. 7, each of the multiple subpixels of the spectral basis filter is constructed using a single layer of thin film material. The thin film may include, for example, one or more layers of material (e.g., high refractive index material that is substantially transparent in the infrared, such as germanium, silicon, zinc sulfide, zinc selenide, etc.) deposited as a uniform layer on a filter substrate (e.g., germanium, silicon, zinc sulfide, zinc selenide, etc.) in an array format corresponding to the superpixel array dimensions. The subpixel array may then be patterned (e.g., through lithography and timed etching) to achieve a desired film thickness for each subpixel in the array. In some embodiments, the thin film(s) may be deposited directly on the focal plane array.

[0041]

[0050] FIG. 7 shows oscillation transmission profiles corresponding to subpixels composed of single-layer thin films of different thicknesses. Thus, the first oscillation transmission profile 710 may correspond to a thin film of thickness 0.0 μm, meaning that the corresponding subpixel transmits all light equally, regardless of wavelength. The second oscillation transmission profile 720 may correspond to a thin film of thickness 3.5 μm. The other seven oscillation transmission profiles 730, 740, 750, 760, 770, 780 and 790 correspond to thin film thicknesses ranging from 3.8 μm, 8.5 μm, 8.8 μm, 13.5 μm, 13.8 μm, 18.5 μm and 18.8 μm, respectively, as shown in FIG. 7.

[0042]

[0051] A single layer thin film may be used in which each subpixel contains the same material, for example germanium deposited on a zinc selenide substrate, but has a different thickness, although embodiments of the invention are not limited to this particular implementation. In other embodiments, each subpixel may contain different materials, multiple layers of different materials may be used, some of the subpixels may contain a single material while other subpixels may contain two or more materials, etc. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0043]

[0052] FIG. 8 shows a block diagram of an optical system with a spectral basis filter. The optical system can be used to image an object 800. The object 800 can emit an optical signal (e.g., infrared light, etc.) that is transmitted through the spectral basis filter 810. One or more superpixels in the spectral basis filter 810 can filter the received optical signal according to an array of subpixels in one or more superpixels. After filtering the optical signal, a detector 820 optically coupled to the spectral basis filter can read out the optical signal to receive an intensity value associated with each pixel of the detector, e.g., a focal plane array. Each pixel in the detector can be a focal plane array, and for each superpixel of the spectral basis filter 810, can be associated with the transmission of one of the subpixels in the array of subpixels.

[0044]

[0053] After receiving the intensity values ​​for all sub-pixels of the spectral basis filter 810, the detector 820 may transmit the intensity values ​​to a computer 830, which may be implemented as a computational module. The computer 830 may determine a first set of relative weights to be applied to each intensity value received from the plurality of sub-pixels. The computer 830 may determine a spectral profile to be applied to the intensity values. For example, the characteristics of the spectral profile may include band edges and a general shape. To apply the desired spectral profile to the intensity values, the computer 830 may store a table similar to the table shown in FIG. 6 and determine that a 9.00-9.80 μm band pass filter is to be applied. The computer 830 may then access the table and retrieve a first set of relative weights a0-a4 and b1-b4 associated with the 9.00-9.80 μm band pass filter and apply the appropriate relative weights to the intensity values ​​of the corresponding sub-pixels. In another example, a Fourier expansion may be performed before or after receiving the intensity values ​​to calculate the first set of relative weights to be applied. For each of one or more superpixels in the spectral basis filter 810, a first set of relative weights may be applied to corresponding subpixels in the one or more superpixels to determine a first output (e.g., a first image). In this manner, the computer / computational module effectively modulates the signal strength transmitted by each of a predetermined number of subpixels in a weighted sum of the subpixel signal strengths for the superpixel.

[0045]

[0054] The computer 830, or a user operating the computer 830, may determine that a second set of relative weights may be calculated to increase the contrast of the first output image. The computer 830 may then access a table or calculate a second Fourier expansion to determine the second set of relative weights. The second set of relative weights may then be applied to corresponding subpixels in one or more superpixels to determine a second output (e.g., a second image). A first contrast score may be calculated for the first image and a second contrast score may be calculated for the second image. The contrast score may be determined, for example, from the level and span of intensity values ​​across the image. After calculating the contrast score, the computer 830 may compare the first contrast score to the second contrast score. If the first contrast score is greater than the second contrast score (e.g., the first image has greater contrast than the second image), the computer 830 may reapply the first set of relative weights to the intensity values. If the first contrast score is less than the second contrast score, the computer 830 may leave the second image as is, or may attempt to determine a third set of relative weights to further improve the second contrast score.

[0046]

[0055] FIG. 9 is a cross-sectional view of three subpixels of a superpixel according to an embodiment of the present invention. As shown in FIG. 9, subpixels 110, 120, and 130 shown in FIG. 1 vary in the thickness of the thin film as discussed in connection with FIG. 7. Referring to FIG. 9, subpixel 110 is 3.5 μm thick and 10 μm wide, subpixel 120 is 3.8 μm thick and 10 μm wide, and subpixel 130 is 8.5 μm thick and 10 μm wide. These thicknesses correspond to the thicknesses of vibration transmission profiles 720, 730, and 740 in FIG. 7. The other six pixels in this exemplary embodiment would have thicknesses associated with the six other vibration transmission profiles shown in FIG. 7, including a thickness of 0.0 μm (i.e., no thin film) for one of the subpixels, e.g., the center subpixel in the superpixel. The arrangement of the subpixels in FIG. 9 is an example, but other configurations may be selected to optimize lithographic processing or optical performance.

[0047]

[0056] As mentioned above, the thin film illustrated in Figure 9 can be fabricated by depositing a layer of uniform thickness on a substrate 905 and then using lithography and etching to define the thickness of each subpixel. The separation between the subpixels can be implemented in a spectral basis filter as shown in Figure 9, although this is not required by the present invention.

[0048]

[0057] FIG. 10 is a simplified flow chart illustrating a method for defining a spectral profile of an optical filter according to an embodiment of the present invention. The method 1000 includes determining one or more characteristics of the spectral profile (1010). The one or more characteristics of the spectral profile can include at least one of a bandpass short edge, a bandpass long edge, a spectral range cut-on, or a spectral range cut-off. The method also includes defining a transmission profile for each of a plurality of sub-pixels (1012). A predetermined number of the plurality of sub-pixels are characterized by an oscillatory transmission profile as a function of wavelength. Defining the transmission profile for each of a plurality of the plurality of sub-pixels can include determining a thickness of a thin film associated with each of the plurality of sub-pixels, as shown in FIG.

[0049]

[0058] The method further includes associating a set of relative overlay weights with each of the plurality of sub-pixels (1014). In some embodiments, the set of relative weights is determined by accessing a lookup table of weights using one or more characteristics of the spectral profile. In one embodiment, the method also includes determining a first contrast score using the output of the optical filter, modifying at least one weight in the set of relative weights to determine a modified set of relative weights, associating a modified weight of the modified set of relative weights with each of the plurality of sub-pixels, determining a second contrast score using the output of the optical filter having the modified set of relative weights, and determining that the second contrast score is greater than the first contrast score.

[0050]

[0059] It should be understood that the specific steps illustrated in FIG. 10 provide a particular method of defining a spectral profile of an optical filter according to an embodiment of the present invention. Other sequences of steps may also be performed in accordance with alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Additionally, the individual steps illustrated in FIG. 10 may include multiple sub-steps that may be performed in various orders, as appropriate for the individual step. Additionally, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0051]

[0060] 11 is a simplified flow chart illustrating a method according to an embodiment of the present invention. The method 1100 includes transmitting an optical signal through an optical filter that includes a plurality of superpixels (1110). Each of the plurality of superpixels includes a plurality of subpixels. A predetermined number of the plurality of subpixels are characterized by an oscillatory transmission profile as a function of wavelength. One of the plurality of subpixels can be characterized by a uniform transmission profile as a function of wavelength, thereby providing situational awareness at the resolution of the superpixel. The subpixel can also provide an offset (i.e., a0) for the Fourier expansion.

[0052]

[0061] The method also includes receiving intensity values ​​associated with each pixel of the detector (1112). Each pixel of the detector may be a focal plane array and is associated with one of the plurality of sub-pixels. The method further includes applying (1114) a set of relative weights to corresponding intensity values ​​associated with sub-pixels of the plurality of sub-pixels for each of the plurality of super-pixels to determine (1116) output data. The set of relative weights may be obtained from a look-up table of relative weights using the spectral profile. Alternatively, the set of relative weights may be calculated using a Fourier expansion of the spectral profile.

[0053]

[0062] In some embodiments, the method further includes modifying the first set of relative weights to generate a second set of relative weights, and for each of the plurality of superpixels, applying weights of the second set of relative weights to corresponding subpixels of the plurality of subpixels to determine second output data. Modifying the first set of relative weights can include accessing a lookup table of weights using the spectral profile. Modifying the first set of relative weights can include determining a first contrast score from the first output data, modifying at least one weight in the first set of relative weights to provide a modified set of relative weights, applying weights of the modified set of relative weights to corresponding subpixels of the plurality of subpixels, determining a second contrast score from the second output data, and comparing the first contrast score to the second contrast score. If the first contrast score is greater than the second contrast score, the first set of weights associated with the first contrast score can be reapplied to the plurality of subpixels.

[0054]

[0063] It should be understood that the particular steps illustrated in FIG. 11 provide a particular method according to one embodiment of the present invention. Other sequences of steps may also be performed by alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Additionally, individual steps illustrated in FIG. 11 may include multiple sub-steps that may be performed in various orders, as appropriate for the individual step. Additionally, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0055]

[0064] An embodiment of the present invention allows the collection of image data using an optical system while reducing the data transmission bandwidth. As an example, the spectral profile can define, for example, a bandpass filter centered at 9.4 μm with a bandwidth of 0.8 μm. A set of weights associated with each subpixel that produces this spectral profile or an approximation to this spectral profile can be calculated and provided to the optical system, which can be mounted on, for example, a drone. During image acquisition, intensity values ​​of pixels of the focal plane array are measured, each pixel corresponding to one of the subpixels. Processing can be performed by a processor of the optical system, which applies appropriate weights to each corresponding subpixel and produces an image with superpixel resolution, as discussed in connection with FIG. 6. In this case, after processing, a superpixel resolution image is produced, for example with a resolution of 1024×1024. Thus, an embodiment of the present invention can reduce communication bandwidth, since the superpixel resolution can be a fraction (e.g., 1 / 9) of the focal plane array resolution.

[0056]

[0065] An embodiment of the present invention also includes the interpretation of the complete subpixel array resolution with a defined spectral profile. Because the superpixels are spatially periodic on the focal plane array, every subpixel is surrounded by the eight complementary subpixels required to generate the superpixel value, except for the subpixels adjacent to it on the focal plane array. Thus, the focal plane array can generate a 3070x3070 resolution image with a selected bandpass by treating each subpixel as the center of a superpixel.

[0057]

[0066] 1, superpixel 105 is described as a collection of subpixels 110, 120, 130, 140, 150, 160, 170, 180, and 190, and is a 3×3 subpixel array centered on subpixel 150. Superpixel 103 can also be defined as a collection of subpixels 182, 192, 173, 120, 130, 111, 150, 160, and 141, and a 3×3 subpixel array centered on subpixel 130. The subpixels of superpixel 103 have a similar set of spectral filters as used in superpixel 105. Explicitly, subpixel 182 has a similar filter as subpixel 180, subpixel 192 has a similar filter as subpixel 190, and similarly for subpixels 173 and 170, subpixels 111 and 110, and subpixels 141 and 140. Thus, a predetermined number of sub-pixels of one or more of the first super-pixels may also be sub-pixels of the second super-pixel.

[0058]

[0067] The same process described herein can be applied to each subpixel in the spectral basis filter 100 to combine the subpixel intensities to generate a superpixel with a defined bandpass. As a result, a 3070x3070 resolution image with full spectral bandpass control can be generated from a pixel array with 3072x3072 resolution. Since the optical system can be designed for a blur circle comparable to the size of the superpixel, this mode of operation can generate a spatially oversampled image, which has obvious advantages to those skilled in the art.

[0059]

[0068] In other embodiments, the intensity values ​​of the pixels of the focal plane array can be sent to a separate system for processing. In this case, the resolution of the image data matches the pixel resolution of the focal plane array, and the data communication bandwidth corresponds to the resolution of the focal plane array, for example, a resolution of 3072×3072. By sending the image data at the resolution of the focal plane array and performing post-processing, a different set of weights can be applied to the image data. By way of example only, a first set of weights can be used to apply appropriate weights to each corresponding sub-pixel to generate an image characterized by a first spectral profile, for example, a bandpass filter centered at 9.4 μm with a bandwidth of 0.8 μm. A second set of weights corresponding to a second spectral profile, for example, a bandpass filter centered at 9.45 μm with a bandwidth of 0.8 μm, can then be applied to the image data, resulting in an image characterized by a second spectral profile. Additional sets of weights can be used to modify the spectral profile, in this example, by sweeping the bandpass filter over a predetermined wavelength range to generate image data corresponding to each of the various spectral profiles.

[0060]

[0069] Thus, in contrast to conventional imaging systems that use spectral filters with a predetermined spectral profile, embodiments of the present invention allow image data to be collected across a range of wavelengths (e.g., 8 μm-10.5 μm) and then post-processed to provide a series of images each associated with a different spectral profile. In certain embodiments, image contrast can be enhanced by selecting an image having a higher contrast score than other contrast scores associated with other images. If the image data is recorded at the resolution of the focal plane array, post-processing can modify the spectral profile, thereby applying a different spectral profile to the recorded image data and generating image data associated with the different spectral profiles during post-processing.

[0061]

[0070] It should be understood that image processing using a set of weights can be performed by a processor of the optical system. Continuing with the example discussed above, the processor of the optical system can apply various sets of weights to the image data, effectively sweeping a bandpass filter across the wavelength range of interest. One or more of the images generated by the optical system can then be transmitted to a separate system, such as a ground station that communicates with the optical system, and can be mounted on a drone, as described above. Furthermore, automatic contrast control can be achieved by performing spectral adjustment of the image data to enhance the contrast of the image in a manner similar to automatic gain control. When acquiring image data, various sets of weights can be applied to the image data to measure the image contrast in the image, thereby generating an image with enhanced contrast and automatically controlling the contrast. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0062]

[0071] The above description is illustrative and not limiting. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. Thus, the scope of the invention should not be determined with reference to the above description, but instead with reference to the appended claims along with their full scope or equivalents.

[0063]

[0072] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention.

[0064]

[0073] As used herein, the use of "a," "an," or "the" is intended to mean "at least one," unless specifically indicated to the contrary.

[0065]

[0074] While the principles of the present disclosure have been described above in connection with specific apparatus and methods, it should be clearly understood that this description is made only by way of example and not as a limitation on the scope of the disclosure.

[0066]

[0075] It is also to be understood that the examples and embodiments described herein are for illustrative purposes only, and various modifications or changes in light thereof will be suggested to those skilled in the art and are to be included within the spirit and scope of this application and the appended claims.

Claims

1. A focal plane array having a plurality of pixels defined by a first number of pixels arranged in a first direction and a second number of pixels arranged in a second direction, An optical filter optically coupled to the focal plane array, the optical filter having a plurality of superpixels, each of the plurality of superpixels including a predetermined number of subpixels, each of the predetermined number of subpixels being characterized by one of a plurality of oscillatory light transmission profiles whose amplitude varies between 0 and 1 as a function of wavelength, an optical filter; An optical system comprising:

2. The optical system according to claim 1, wherein one or more of the predetermined number of subpixels of the first superpixel are also subpixels of the second superpixel.

3. The optical system according to claim 1, wherein the predetermined number of subpixels is defined by a first number of subpixels arranged in the first direction and a second number of subpixels arranged in the second direction.

4. The optical system according to claim 1, wherein each of the plurality of oscillatory light transmission profiles is distinguishable from the others of the plurality of oscillatory light transmission profiles.

5. The optical system according to claim 4, wherein each of the plurality of oscillatory light transmission profiles is characterized by a different combination of oscillation frequency and phase.

6. The optical system according to claim 1, wherein each of the predetermined number of subpixels includes one or more layers of a film that is substantially transparent in the infrared.

7. The optical system according to claim 6, wherein the one or more layers of the film include a single thin film layer, and each of the predetermined number of subpixels is characterized by a different thickness of the single thin film layer.

8. The optical system according to claim 1, further comprising a calculation module configured to assign a predetermined weight to each of the predetermined number of sub-pixels. **Claim 9** Determining one or more characteristics of a spectral profile; Defining a transmission profile for each of a plurality of sub-pixels, wherein the predetermined number of the plurality of sub-pixels is characterized by an oscillatory light transmission profile in which the amplitude varies between 0 and 1 as a function of wavelength; Associating a weight of a set of relative weights with each of the plurality of sub-pixels, a method for defining a spectral profile of an optical filter. **Claim 10** The method according to claim 9, further comprising determining the set of relative weights by accessing a weight look-up table using one or more characteristics of the spectral profile. **Claim 11** Determining a first contrast score using the output of the optical filter; Modifying at least one weight within the set of relative weights to define a modified set of relative weights; Associating the modified set of relative weights with each of the plurality of sub-pixels; Determining a second contrast score using the output of the optical filter having the modified set of relative weights; Selecting the set of relative weights to be used based on a comparison of the second contrast score and the first contrast score; The method according to claim 9, further comprising. **Claim 12** The method according to claim 9, wherein the one or more characteristics of the spectral profile include at least one of a bandpass short edge, a bandpass long edge, a spectral range cut-on, or a spectral range cut-off.

13. The step of defining the transmission profile for each of the plurality of sub-pixels includes determining a thickness of a thin film associated with each of the plurality of sub-pixels, the thin film being substantially transparent in the infrared, the method according to claim 9.

14. A step of transmitting an optical signal through an optical filter comprising a plurality of superpixels, each of the plurality of superpixels including a plurality of sub-pixels, a predetermined number of the plurality of sub-pixels being characterized by an oscillatory light transmission profile in which the amplitude varies between 0 and 1 as a function of wavelength, the step; A step of receiving an intensity value associated with each pixel of a detector, each pixel of the detector being associated with one of the plurality of sub-pixels, the step; For each of the plurality of superpixels, Applying a set of relative weights to the corresponding intensity values associated with the sub-pixels of the plurality of superpixels; A step of determining output data; A method comprising.

15. One of the plurality of sub-pixels is characterized by a uniform light transmission profile as a function of wavelength, the method according to claim 14.

16. The method according to claim 14, further comprising retrieving the set of relative weights from a look-up table of relative weights corresponding to a spectral profile.

17. The method according to claim 14, further comprising calculating the set of relative weights using a Fourier expansion of a spectral profile.

18. A step of modifying the set of relative weights; For each of the plurality of superpixels, applying the modified set of the relative weights to corresponding sub-pixels of the plurality of sub-pixels; determining modified output data; The method according to claim 14, further comprising. **Claim 19** The method according to claim 18, wherein the step of modifying the set of the relative weights includes accessing a look-up table of weights corresponding to a modified spectral profile. **Claim 20** The step of modifying the set of the relative weights includes determining a first contrast score corresponding to the output data; determining a second contrast score corresponding to the modified output data; determining that the first contrast score is greater than the second contrast score; and reapplying the set of the relative weights to the corresponding intensity values associated with the sub-pixels of the plurality of sub-pixels. The method according to claim 18, comprising.

Images