Systems and methods for high-precision multi-aperture spectral imaging

Abstract

In general, one or more aspects of the present application correspond to systems and techniques for spectral imaging using a multi-aperture system with a curved multi-bandpass filter located above each aperture. The present disclosure also relates to techniques for using image information received from such an imaging system to perform spectral unmixing and image registration to generate a spectral data cube. Various aspects of the present disclosure relate to using such a data cube to analyze imaged objects, such as analyzing tissues, performing biometrics, or performing material analysis in a clinical setting.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 780,121, filed on December 14, 2018, entitled “SYSTEM AND METHOD FOR HIGH PRECISION MULTIPLE-APORER SPECTROSCOPIC IMAGING,” and U.S. Provisional Application Serial No. 62 / 818,375, filed on March 14, 2019, entitled “SYSTEM AND METHOD FOR HIGH PRECISION MULTIPORER SPECTROSCOPIC IMAGING,” which are expressly incorporated herein by reference in their entirety for all purposes.

[0003] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

[0004] Certain work recorded in this disclosure was completed with support from the U.S. Government under Contract No. HHSO100201300022C awarded by the Biomedical Advanced Research and Development Authority (BARDA) within the Office of the Assistant Secretary for Preparedness and Response of the U.S. Department of Health and Human Services. Certain work recorded in this disclosure was completed with support from the U.S. Government under Contract No. W81XWH-17-C-0170 and / or W81XWH-18-C-0114 awarded by the Defense Health Agency (DHA). The U.S. Government may have certain rights in this invention. Technical Field

[0005] The systems and methods disclosed herein relate to spectral imaging, and more particularly, to high-precision spectral imaging via a multi-aperture imaging system. Background Art

[0006] The electromagnetic spectrum is the range of wavelengths or frequencies over which electromagnetic radiation (e.g., light) extends. From longer wavelengths to shorter wavelengths, the electromagnetic spectrum includes, in order, radio waves, microwaves, infrared (IR) rays, visible light (i.e., light that can be detected by the structure of the human eye), ultraviolet (UV) rays, x-rays, and gamma rays. Spectral imaging refers to a branch of spectroscopy and photography in which some spectral information or a complete spectrum is collected at a location in the image plane. Multispectral imaging systems can capture multiple spectral bands (on the order of a dozen or less, and typically in discrete spectral regions), for which spectral band measurements are collected at each pixel and can reference bandwidths of about tens of nanometers for each spectral channel. Hyperspectral imaging systems measure a larger number of spectral bands, such as up to 200 or more, some of which provide continuous narrowband sampling (e.g., spectral bandwidths of less than nanometers) along a portion of the electromagnetic spectrum. Summary of the invention

[0007] The multispectral imaging systems and techniques disclosed herein have multiple features, no single one of which is solely responsible for its desirable properties. Without limiting the scope as expressed by the appended claims, certain features of the disclosed spectral imaging will be briefly discussed below. Those skilled in the art will understand how the features of the disclosed spectral imaging provide numerous advantages over conventional systems and methods.

[0008] One aspect relates to a multispectral imaging system comprising: at least one image sensor; a first aperture positioned to allow incident light to pass to a first sensor region of the at least one image sensor; a second aperture positioned to allow incident light to pass to a second sensor region of the at least one image sensor; a first multi-bandpass filter positioned above the first aperture, wherein the first multi-bandpass filter is configured to allow light in at least a common wavelength band and a first unique wavelength band to pass; a second multi-bandpass filter positioned above the second aperture, wherein the second multi-bandpass filter is configured to allow light in at least the common wavelength band and the second unique wavelength band to pass; a memory storing instructions for generating a multispectral data cube; and at least A processor configured by the instructions to at least: receive signals from first and second sensor regions; and process the signals to generate the multispectral data cube, wherein the multispectral data cube includes a first spectral channel corresponding to the common band, a second spectral channel corresponding to the first unique band, and a third spectral channel corresponding to the second unique band; wherein processing the signals at least includes: estimating a disparity between first image data captured by the first sensor region and second image data captured by the second sensor region using information from the first spectral channel corresponding to the common band; and aligning the image data of the first, second, and third spectral channels using the disparity to generate the multispectral data cube.

[0009] In some embodiments, the first multi-bandpass filter is located in front of the lens of the multispectral imaging system, behind the lens of the multispectral imaging system, inside the lens of the multispectral imaging system, or directly on the surface of an element of the lens of the multispectral imaging system.

[0010] In some embodiments, the at least one processor is configured by the instructions to at least perform spectral unmixing to generate image data for each of the first, second, and third spectral channels.

[0011] In some embodiments, the multispectral imaging system further comprises: a color filter array covering the first and second sensor areas, the color filter array comprising a repeating pattern of color filters; wherein the first color filter of the repeating pattern is configured to pass light having a wavelength with a first weight distribution including the common band; wherein the second color filter of the repeating pattern is configured to pass light having a wavelength with a second weight distribution including the first unique band; and wherein the third color filter of the repeating pattern is configured to pass light having a wavelength with a third weight distribution including the second unique band. In some embodiments, the common band includes at least one of violet, violet-blue, blue or cyan light. In some embodiments, the common band includes light having a wavelength shorter than 450nm and longer than 380nm. In some embodiments, the common band includes green light. In some embodiments, the common band includes red light. In some embodiments, the first and second multi-bandpass filters are both configured to pass a second common band including at least one of green, cyan or blue light. In some embodiments, the first and second multi-bandpass filters are each configured to pass a second common band including red light. In some embodiments, the first and second multi-bandpass filters are each configured to pass a second common band that is not filtered by at least one color filter patterned in the color filter array, which also does not filter the first common band. In some embodiments, the at least one processor is configured by the instructions to at least perform spectral unmixing to isolate portions of the captured image data corresponding to the common band and the second common band. In some embodiments, the at least one processor is configured by the instructions to at least analyze the imaged tissue using the multispectral data cube.

[0012] In some embodiments, the multispectral image system further includes: a third aperture positioned to allow incident light to pass to a third sensor region of the at least one image sensor; a fourth aperture positioned to allow incident light to pass to a fourth sensor region of the at least one image sensor; a third multi-bandpass filter located above the third aperture, wherein the third multi-bandpass filter is configured to allow light in at least a common band and a third unique band to pass; and a fourth multi-bandpass filter located above the fourth aperture, wherein the fourth multi-bandpass filter is configured to allow light in at least the common band and the fourth unique band to pass. In some embodiments, the multispectral image system further includes: a fifth aperture positioned to allow incident light to pass to a fifth sensor region of the at least one image sensor; and a fifth multi-bandpass filter located above the fifth aperture, wherein the fifth multi-bandpass filter is configured to allow light in at least one band passed by at least one of the first multi-bandpass filter, the second multi-bandpass filter, the third multi-bandpass filter, or the fourth multi-bandpass filter to pass. In some embodiments, the multispectral imaging system includes: a total of up to 25 apertures, which are positioned to allow incident light to pass to a total of up to 25 sensor areas of the at least one image sensor; and up to 25 multi-bandpass filters, each multi-bandpass filter is located above one of the total 25 apertures, wherein each multi-bandpass filter is configured to allow light to pass in at least a common band and no other multi-bandpass filter is configured to allow light to pass through; wherein the total number of bands allowed to pass is more than 4 and less than 51. In some embodiments, the multispectral imaging system also includes: a third aperture, which is positioned to allow incident light to pass to a third sensor area of ​​the at least one image sensor; and a third multi-bandpass filter located above the third aperture, wherein the third multi-bandpass filter is configured to allow light to pass in at least one band passed by at least one of the first multi-bandpass filter or the second multi-bandpass filter. In some embodiments, the first, second, third, and fourth multi-bandpass filters collectively pass 7, 8, 9, or 10 different bands including the common band and the first, second, third, and fourth unique bands. In some embodiments, the first, second, third, and fourth multi-bandpass filters are each configured to pass the common band including blue light, an additional common band including green light, and two additional bands. In some embodiments, the at least one processor is configured by the instructions to at least generate the multispectral data cube to include 10 spectral channels corresponding to the 7, 8, 9, or 10 different bands.

[0013] In some embodiments, the multi-spectral image system also includes an illumination panel having a plurality of light emitting diodes (LEDs), which are configured to emit light in 7, 8, 9 or 10 different bands, wherein each LED emits light under a specific band in 7, 8, 9 or 10 different bands. In some embodiments, the at least one image sensor includes a color filter array located above the photodiode of the at least one image sensor, and wherein the color filter array has at least two different color filters. In some embodiments, the color filter array has more than 2 different color filters and no more than 25 different color filters. In some embodiments, the 7, 8, 9 or 10 different bands include a narrower range of light than that passed by 3 different color filters. In some embodiments, the multi-spectral image system also includes at least one light source, which is configured to emit light under the common band, the first unique band and the second unique band. In some embodiments, the multi-spectral image system also includes a diffuse optical element located above the at least one light source. In some embodiments, at least the first multi-bandpass filter is a curved filter.

[0014] Another aspect relates to a method of generating a multispectral data cube using any of the multispectral image systems described herein, the method comprising: capturing a first exposure including first, second, and third spectral channels; capturing a second exposure including first, second, and third spectral channels; and generating the multispectral data cube based at least in part on reducing the signal-to-noise ratio of the first exposure using the second exposure. In some embodiments, the method further comprises activating a group of light emitters that emit light together at the common band, the first unique band, and the second unique band during the first and second exposures. In some embodiments, the first exposure and the second exposure each comprise at least a fourth spectral channel and no more than a total of 10 spectral channels. In some embodiments, the method further comprises: capturing a third exposure using ambient light; and subtracting the third exposure from the first and second exposures. In some embodiments, the method further comprises generating the multispectral data cube based on the result of subtracting the third exposure from the first and second exposures. In some embodiments, the method further comprises registering the images captured by the first and second sensor regions to each other based on information in the first spectral channel. In some embodiments, the method further comprises generating the multispectral data cube based on the result of registering the images captured by the first and second sensor regions to each other. In some embodiments, the image represents information captured by the first and second sensor areas in a first spectral channel, and the method further includes: identifying a disparity between the images captured by the first and second sensor areas in the first spectral channel corresponding to the common wavelength band; registering a first additional image to the image using the disparity, the first additional image representing information captured by the first sensor area in a second spectral channel; and registering a second additional image to the image using the disparity, the second additional image representing information captured by the second sensor area in a third spectral channel. In some embodiments, each exposure includes more than three spectral channels, and the method further includes registering at least a third additional image to the image using the disparity, the third additional image representing information captured by the third sensor area in a fourth spectral channel, wherein the total number of additional images registered to the image is one less than the total number of spectral channels. In some embodiments, the multispectral imaging system further comprises a light emitter configured to project one or more dots, stripes, grids, random spots or other spatial patterns, and the method further comprises: projecting light comprising one or more dots, stripes, grids, random spots or other spatial patterns into the field of view of the at least one imaging sensor by the light emitter; and using the light projected by the light emitter to capture additional exposures; wherein the registration is based at least in part on the additional exposures. In some embodiments, the light projected by the light emitter comprises at least one of light having a wavelength in the common band, broadband illumination or cumulative visible illumination. In some embodiments, the additional exposures are used to confirm the accuracy of the registration of the image.In some embodiments, the additional exposure is included in the calculation of the parallax to improve the accuracy of the registration. In some embodiments, the illuminant projects a uniquely constructed plurality of dots, stripes, grids, random spots or other spatial patterns in at least the common band, broadband illumination or cumulative visible illumination to further improve the accuracy of the registration. In some embodiments, the multispectral imaging system further comprises an additional sensor configured to detect the shape of an object in the field of view of the at least one imaging sensor, and the registration is based at least in part on the shape detected by the additional sensor. Such an additional sensor may be a single aperture sensor or a multi-aperture sensor sensitive to light field information, and may also be sensitive to other signals (e.g., ultrasound or pulsed lasers). In some embodiments, the method further comprises performing spectral unmixing to isolate the first, second and third spectral channels. In some embodiments, each exposure comprises more than three spectral channels, and the method further comprises performing spectral unmixing to isolate all spectral channels. In some embodiments, the method further comprises using the parallax to determine at least one of terrain information or depth information. In some embodiments, the parallax is used to determine the depth information of each pixel. In some embodiments, the method further comprises determining the volume of the imaged wound using the depth information. In some embodiments, the method further comprises: capturing additional exposures comprising at least a fourth spectral channel representing invisible light; and generating the multispectral data cube based at least in part on the additional exposures. In some embodiments, the method further comprises activating a third set of illuminants emitting the invisible light during the additional exposures.

[0015] Another aspect relates to a method of automated tissue analysis, the method comprising: using any of the multispectral imaging systems described herein to generate a multispectral data cube representing the imaged tissue; supplying the multispectral data cube as input to a machine learning system trained to predict at least one clinical characteristic of the tissue; and predicting at least one clinical characteristic of the imaged tissue based on an output of the machine learning system.

[0016] In some embodiments, the method further comprises generating a multispectral data cube using the multispectral data cube generation method described herein. In some embodiments, the at least one clinical feature comprises a plurality of trauma states or burn states, and the method further comprises generating a classified image of the imaged tissue using the output of the machine learning system, the classified image showing the imaged tissue being classified into regions of different states in the plurality of trauma states or burn states. In some embodiments, for example, the plurality of burn states comprises at least first degree burns, second degree burns, third degree burns or healthy skin categories, and the method further comprises classifying each of the plurality of pixels showing the imaged tissue as one of the first degree burns, second degree burns, third degree burns or healthy skin categories. In some embodiments, for example, the plurality of trauma states comprises at least hemostasis, inflammation, hyperplasia, remodeling or healthy skin categories, and the method further comprises classifying each of the plurality of pixels showing the imaged tissue as one of the hemostasis, inflammation, hyperplasia, remodeling or healthy skin categories. In some embodiments, the at least one clinical feature includes healing potential, and the method further includes using the output of the machine learning system to generate an image of the imaged tissue, the image visually showing the determined healing potential in different areas of the imaged tissue. In some embodiments, the at least one clinical feature includes the cancerous or non-cancerous state of the imaged tissue, and the method further includes classifying each of the multiple pixels showing the imaged tissue as a cancerous or non-cancerous state. In some embodiments, the at least one clinical feature includes wound depth, and the method further includes outputting information identifying the wound depth in the imaged tissue. In some embodiments, the at least one clinical feature includes a debridement boundary, and the method further includes outputting information identifying the debridement boundary within or around the wound in the imaged tissue. In some embodiments, the at least one clinical feature includes diabetes, non-diabetes, or chronic ulcers. In some embodiments, the at least one clinical feature includes diabetic foot ulcers.

[0017] Another aspect relates to a method of automated biometric identification, the method comprising: using any of the multispectral image systems described herein to generate a multispectral data cube representing a user's biometrics; supplying the multispectral data cube as input to a machine learning system trained to authenticate specific biometrics; and authenticating or rejecting the user based on an output of the machine learning system.

[0018] In some embodiments, the method further comprises generating a multispectral data cube using any of the multispectral data cube generation methods described herein.

[0019] Another aspect relates to a method of automated material analysis, the method comprising: using any of the multispectral imaging systems described herein to generate a multispectral data cube representing an imaged material; supplying the multispectral data cube as input to a machine learning system, the machine learning system being trained to determine one or more material properties; and determining one or more material properties of the imaged material based on an output of the machine learning system.

[0020] In some embodiments, the method further comprises generating a multispectral data cube using any of the multispectral data cube generation methods described herein.

[0021] Another aspect relates to a method comprising: capturing image information using a multi-aperture handheld imaging device comprising at least one image sensor area having a color filter array, wherein the image information comprises a greater number of channels than the number of filter colors in the color filter array; and processing the image information into a multispectral data cube comprising a plurality of channels, wherein each of the plurality of channels represents light in a different wavelength band below approximately 40 nm but not zero.

[0022] In some embodiments, the multi-aperture handheld imaging device comprises any of the multispectral image systems described herein. In some embodiments, the method further comprises generating a multispectral data cube using any of the multispectral data cube generation methods described herein.

[0023] Another aspect relates to a method comprising: generating a multispectral data cube using any of the multispectral image systems described herein; and processing the multispectral data cube using at least one processor of the multispectral image system to determine at least one feature of an object represented by the multispectral data cube.

[0024] In some embodiments, the method further comprises generating a multispectral data cube using any of the multispectral data cube generation methods described herein.

[0025] Another aspect relates to a method, comprising: using any of the multispectral image systems described herein to generate a multispectral data cube; transmitting the multispectral data cube to at least one remote processor via a network; and processing the multispectral data cube using the at least one remote processor to determine at least one feature of an object represented by the multispectral data cube. In some embodiments, the method further comprises generating a multispectral data cube using any of the multispectral data cube generation methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1AExamples of light incident on the filter at different chief ray angles are shown.

[0027] Figure 1B is shown for various chief ray angles by Figure 1A A graph of exemplary transmission efficiency provided by the filters.

[0028] Figure 2A An example of a multispectral image data cube is shown.

[0029] Figure 2B shows how some multispectral imaging techniques can generate Figure 2A An example of a data cube.

[0030] Figure 2C It shows that it is possible to generate Figure 2A An exemplary snapshot imaging system of a data cube.

[0031] Figure 3A A schematic cross-sectional view of the optical design of an exemplary multi-aperture imaging system with curved multi-bandpass filters according to the present disclosure is shown.

[0032] Figure 3B-3D Shown for Figure 3A An exemplary optical design of the optical components of one optical path of a multi-aperture imaging system.

[0033] Figures 4A-4E Shown with reference Figure 3A and Figure 3B An embodiment of the optical design of a multispectral multiaperture imaging system is described.

[0034] Figure 5 Shown with reference Figure 3A and Figure 3B Another embodiment of a multispectral multi-aperture imaging system of the optical design is described.

[0035] Figures 6A-6C Shown with reference Figure 3A and Figure 3B Another embodiment of a multispectral multi-aperture imaging system of the optical design is described.

[0036] Figure 7A-7B Shown with reference Figure 3A and Figure 3B Another embodiment of a multispectral multi-aperture imaging system of the optical design is described.

[0037] Figures 8A-8B shows that the Figure 3A and Figure 3B Another embodiment of a multispectral multi-aperture imaging system of the optical design is described.

[0038] Figures 9A-9C Shown with reference Figure 3A and Figure 3B Another embodiment of a multispectral multi-aperture imaging system of the optical design is described.

[0039] Figures 10A-10B Shown with reference Figure 3A and Figure 3B Another embodiment of a multispectral multi-aperture imaging system of the optical design is described.

[0040] Figures 11A-11B Shows that it can be Figure 3A-10B An exemplary set of wavelength bands passed by filters of a multispectral multi-aperture imaging system.

[0041] Fig.12 Shows that it can be used Figure 3A-10B Schematic block diagram of the imaging system of the multi-spectral multi-aperture imaging system.

[0042] Fig.13 Is used for Figure 3A-10B Flow chart of an exemplary process for capturing image data using a multispectral multi-aperture imaging system.

[0043] Fig.14 A schematic block diagram of a workflow for processing image data is shown, for example using Fig.13 The process and / or use of Figure 3A-10B Image data captured by a multispectral multi-aperture imaging system.

[0044] Fig.15 The disparity and disparity correction for processing image data are graphically shown, for example using Fig.13 The process and / or use of Figure 3A-10B Image data captured by a multispectral multi-aperture imaging system.

[0045] Fig.16 The workflow for performing pixel-by-pixel classification on multispectral image data is graphically illustrated, for example using Fig.13 process, based on Fig.14 and Fig.15 Processing and / or use Figure 3A-10B Image data captured by a multispectral multi-aperture imaging system.

[0046] Fig.17 Shown include Figure 3A-10B Schematic block diagram of an exemplary distributed computing system for a multispectral multi-aperture imaging system.

[0047] Figures 18A-18C An exemplary handheld implementation of a multispectral multi-aperture imaging system is shown.

[0048] Fig.19A and Fig.19B An exemplary handheld implementation of a multispectral multi-aperture imaging system is shown.

[0049] Fig. 20A and Fig. 20B An exemplary multi-spectral multi-aperture imaging system is shown for a small USB 3.0 enclosure in a common camera housing.

[0050] Fig.21 An exemplary multi-spectral multi-aperture imaging system including additional light sources for improved image registration is shown. DETAILED DESCRIPTION

[0051] In general, the present disclosure relates to spectral imaging using a multi-aperture system with a curved multi-bandpass filter located above each aperture (aperture). The present disclosure also relates to a technique for implementing spectral unmixing and image registration using image information received from such an imaging system to generate a spectral data cube. The disclosed technology solves the following numerous challenges that are commonly present in spectral imaging, thereby generating image data representing accurate information about the bands reflected from the imaged object. In some embodiments, the systems and methods described herein acquire images from a wide tissue area (e.g., 5.9×7.9 inches) in a short time (e.g., within 6 seconds or less), and without the need to inject a contrast agent. In some aspects, for example, the multispectral imaging system described herein is configured to acquire images from a wide tissue area (e.g., 5.9×7.9 inches) in less than 6 seconds, and wherein the multispectral imaging system is also configured to provide tissue analysis information without a contrast agent, such as identifying multiple burn states, wound states, healing potential, clinical features (including cancerous or non-cancerous states of the imaged tissue), wound depth, debridement margins, or the presence of diabetes, non-diabetes, or chronic ulcers. Similarly, in some methods described herein, the multispectral imaging system acquires images from a wide tissue area (e.g., 5.9×7.9 inches) in less than 6 seconds and the multispectral imaging system outputs tissue analysis information in the absence of contrast agents, such as identifying multiple burn states, wound states, healing potential, clinical characteristics (including cancerous or non-cancerous status of the imaged tissue), wound depth, debridement margins, or the presence of diabetic, non-diabetic, or chronic ulcers.

[0052] One of such challenges in existing solutions is that the captured image may suffer from color distortion that impairs the quality of the image data. This can be particularly problematic for applications that rely on the precise detection and analysis of certain wavelengths of light using optical filters. Specifically, chromatic aberration is a position-dependent variation in the wavelength of light over the entire area of ​​the image sensor due to the fact that the transmittance of the color filter shifts to shorter wavelengths as the angle of the light incident on the filter increases. Typically, this effect is observed in interference-based filters made by depositing thin layers of different refractive indices onto a transparent substrate. Therefore, due to the larger incident light angle, longer wavelengths (e.g., red light) can be blocked more at the edges of the image sensor, resulting in the same incident wavelength of light being detected as spatially non-uniform colors on the image sensor. If not corrected, chromatic aberration will appear as a color shift near the edges of the captured image.

[0053] Relative to other multi-spectral imaging systems on the market, the technology of the present disclosure provides advantages because it has no restrictions on the composition of lenses and / or image sensors and their respective fields of view or aperture sizes. It is understood that changes to the lenses, image sensors, aperture sizes or other components of the imaging system of the present disclosure may involve other adjustments to the imaging system, as known to those of ordinary skill in the art. The technology of the present disclosure also provides improvements to other multi-spectral imaging systems, in that the components (e.g., optical filters, etc.) that perform the function of resolving wavelengths or enabling the system as a whole to resolve wavelengths can be separated from the components (e.g., image sensors, etc.) that convert light energy into digital outputs. This reduces the cost, complexity and / or development time of reconstructing the imaging system for different multi-spectral wavelengths. The technology of the present disclosure can be more robust than other multi-spectral imaging systems because it can achieve the same imaging characteristics as other multi-spectral imaging systems on the market with a smaller and lighter form factor. The technology of the present disclosure is also advantageous relative to other multi-spectral imaging systems in that it can obtain multi-spectral images at snapshots, video rates or high-speed video rates. The disclosed techniques also provide for more robust implementation of multispectral imaging systems based on multi-aperture techniques, since the ability to multiplex multiple spectral bands into individual apertures reduces the number of apertures required to acquire any particular number of spectral bands in an imaging data set, thereby reducing costs through a reduced number of apertures and improved light collection (e.g., since larger apertures can be used within the fixed size and dimensions of commercially available sensor arrays). Finally, the disclosed techniques can provide all of these advantages without compromising on resolution or image quality.

[0054] Figure 1AAn example of a filter 108 disposed along the path of light toward an image sensor 110 is shown, and light incident on the filter 108 at different ray angles is also shown. Rays 102A, 104A, 106A are represented as lines that, after passing through the filter 108, are refracted onto the sensor 110 by a lens 112, which may also be replaced by any other imaging optics, including but not limited to a mirror and / or an aperture. Figure 1A 106A. It is assumed that the light for each ray is a broadband having a spectral component extending over a large wavelength range, for example, selectively filtered by the filter 108. The three rays 102A, 104A, 106A arrive at the filter 108 at different angles, respectively. For the purpose of illustration, the ray 102A is shown as being incident substantially perpendicular to the filter 108, the ray 104A has a larger angle of incidence than the ray 102A, and the ray 106A has a larger angle of incidence than the ray 104A. Due to the angular dependence of the transmission characteristics of the filter 108 as observed by the sensor 110, the obtained filtered rays 102B, 104B, 106B exhibit a unique spectrum. The effect of this dependence causes the bandpass of the filter 108 to shift to shorter wavelengths as the angle of incidence increases. In addition, this dependence may lead to a reduction in the transmission efficiency of the filter 108 and a change in the spectral shape of the bandpass of the filter 108. These combined effects are referred to as angle-dependent spectral transmission. Figure 1B shows what a hypothetical spectrometer at the location of sensor 110 would observe Figure 1A 104C and 106C illustrate the shift in the spectral bandpass of filter 108 in response to an increase in the angle of incidence. Curves 102C, 104C and 106C indicate a shortening of the central wavelength of the bandpass; therefore, the wavelength of light passed by the optical system in the example shortens. It is also shown that the shape of the bandpass and the peak transmission also change due to the angle of incidence. For certain consumer applications, image processing can be applied to eliminate the visible effects of this angle-dependent spectral transmission. However, these post-processing techniques do not allow for the recovery of precise information about which wavelength of light is actually incident on filter 108. Therefore, the image data obtained may be unusable for certain high-precision applications.

[0055] Another challenge faced by some existing spectral imaging systems is the time required to capture a complete spectral image dataset, such as Figure 2A and Figure 2B Discussed. Spectral imaging sensors sample the spectral irradiance I(x, y, λ) of a scene, collecting a three-dimensional (3D) data set often referred to as a data cube. Figure 2AAn example of a spectral image data cube 120 is shown. As shown, the data cube 120 represents three dimensions of image data: two spatial dimensions (x and y) corresponding to the two-dimensional (2D) surface of the image sensor and a spectral dimension (λ) corresponding to a particular wavelength band. The dimensions of the data cube 120 can be represented by N x N y N λ Given, where N x , N y and N λ are the number of sampled elements along the (x, y) spatial dimension and the spectral axis λ, respectively. Because the data cube has a higher dimensionality than currently available 2D detector arrays (e.g., image sensors), a typical spectral imaging system captures time-sequential 2D slices or planes of the data cube 120 (referred to herein as a "scanning" imaging system) or measures all elements of the data cube simultaneously by partitioning them into multiple 2D elements that can be reassembled into the data cube 120 during processing (referred to herein as a "snapshot" imaging system).

[0056] Figure 2B An example of how certain scanning spectral imaging techniques generate a data cube 120 is shown. Specifically, Figure 2B Portions 132, 134, and 136 of the data cube 120 that may be collected during a single detector integration are shown. For example, a point scanning spectrometer may capture portion 132 that extends over all spectral planes λ at a single (x, y) spatial location. A point scanning spectrometer may be used to construct the data cube 120 by performing an integration corresponding to each (x, y) location in the spatial dimensions. For example, a filter wheel imaging system may capture portion 134 that extends over all spatial dimensions x and y but only over a single spectral plane λ. A wavelength scanning imaging system (e.g., a filter wheel imaging system) may be used to construct the data cube 120 by performing an integration corresponding to the number of spectral planes λ. For example, a line scanning spectrometer may capture portion 136 that extends over all spectral dimensions λ and all of one spatial dimension (x or y) but only along a single point in the other spatial dimension (y or x). A line scanning spectrometer may be used to construct the data cube 120 by performing an integration corresponding to each location in the other spatial dimension (y or x).

[0057] For applications where neither the target object nor the imaging system moves (or remains relatively stationary during the exposure time), such scanning imaging systems provide the benefit of generating a high-resolution data cube 120. For line scan and wavelength scan imaging systems, this can be because each spectral or spatial image is captured using the entire area of ​​the image sensor. However, motion of the imaging system and / or the object between exposures can cause artifacts in the acquired image data. For example, the same (x, y) position in the data cube 120 can actually represent a different physical position on the imaged object across the entire spectral dimension λ. This can cause errors in downstream analysis and / or place additional requirements on performing registration (e.g., aligning the spectral dimension λ so that a particular (x, y) position corresponds to the same physical position on the object).

[0058] In contrast, snapshot imaging system 140 may capture the entire data cube 120 in a single integration period or exposure, thereby avoiding such motion-induced image quality issues. Figure 2CAn exemplary image sensor 142 and an optical filter array such as a filter array (CFA) 144 that can be used to create a snapshot imaging system are shown. The CFA 144 in this example is a repeating pattern of color filter units 146 on the surface of the image sensor 142. This method of acquiring spectral information may also be referred to as a multi-spectral filter array (MSFA) or a spectrally resolved detector array (SRDA). In the example shown, the filter unit 146 includes a 5×5 arrangement of different filters, which will generate 25 spectral channels in the acquired image data. Through these different color filters, the CFA can split the incident light into the filter bands and direct the split light to the dedicated photoreceptors on the image sensor. In this way, for a given color 148, only 1 / 25 of the photoreceptors actually detect a signal representing light of that wavelength. Therefore, although 25 different color channels can be generated in a single exposure using this snapshot imaging system 140, each color channel represents an amount of measurement data that is less than the total output of the sensor 142. In some embodiments, the CFA may include one or more of a filter array (MSFA), a spectrally resolved detector array (SRDA), and / or may include a conventional Bayer filter, a CMYK filter, or any other absorption-based or interference-based filter. One type of interference-based filter is an array of thin film filters arranged in a grid, wherein each element of the grid corresponds to one or more sensor elements. Another interference-based filter is a Fabry-Pérot filter. Nano-etched interference Fabry-Pérot filters exhibiting a typical bandpass full width at half maximum (FWHM) of the order of 20 to 50 nm have advantages because they can be used in some embodiments due to the slow roll-off of the filter's passband observed in the transition from its central wavelength to its stopband. These filters also exhibit low OD in these stopbands, further increasing sensitivity to light outside their passbands. These combined effects make these particular filters sensitive to spectral regions that would be blocked by the rapid roll-off of interference filters with high ODs similar to FWHMs made using many thin film layers in a coating deposition process (e.g., in evaporative deposition or in ion beam sputtering). In embodiments having dye-based CMYK or RGB (Bayer) filter configurations, slow spectral roll-off and large FWHM of each filter passband are preferred, providing unique spectral transmission percentages for each wavelength across the entire observed spectrum.

[0059] Therefore, the data cube 120 generated by the snapshot imaging system will have one of two properties that can be problematic for precision imaging applications. As a first option, the data cube 120 obtained by the snapshot imaging system can have an N dimension that is smaller than the (x, y) dimensions of the detector array. x and Ny size, and therefore has a lower resolution than a data cube 120 that would be generated by a scanning imaging system with the same image sensor. As a second option, the data cube 120 obtained from the snapshot imaging system can have the same N dimensions as the (x, y) dimensions of the detector array due to interpolation of certain (x, y) positions. x and N y However, the interpolation used to generate such a data cube means that some of the values ​​in the data cube are not actual measurements of the wavelength of light incident on the sensor, but rather may be estimates based on surrounding values.

[0060] Another existing option for single-exposure multispectral imaging is the multispectral beamsplitter. In this imaging system, a beamsplitter cube splits the incident light into different color bands, each of which is observed by an independent image sensor. Although the beamsplitter design can be changed to adjust the spectral bands measured, it is not easy to split the incident light into more than four beams without affecting the performance of the system. Therefore, four spectral channels appear to be a practical limit for this approach. A closely related approach is to use thin-film filters instead of bulky beamsplitter cubes / prisms to split the light, but this approach is still limited to about six spectral channels due to space limitations and cumulative transmission losses through consecutive filters.

[0061] In some embodiments, the aforementioned problems are solved by the disclosed multi-aperture spectral imaging system and the associated image data processing techniques, which have a curved multi-bandpass filter to filter the light incident through each aperture. This special configuration can achieve all the design goals of fast imaging speed, high-resolution images, and precise fidelity of the detected wavelengths. Therefore, the disclosed optical design and the associated image data processing techniques can be used in portable spectral imaging systems and / or image moving targets, while still generating data cubes suitable for high-precision applications (e.g., clinical tissue analysis, biometric identification, transient clinical events). These higher-precision applications may include diagnosing melanoma in the previous stage (0-3) before metastasis, classifying the severity of burn wounds on skin tissue, or tissue diagnosis of the severity of diabetic foot ulcers. Therefore, the small form factor and snapshot spectral acquisition described in some embodiments enable the present invention to be used in clinical environments with transient events, including diagnosis of several different retinopathies (e.g., non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, and age-related macular degeneration) and imaging examinations of sports pediatric patients. Therefore, those skilled in the art will recognize that the use of a multi-aperture system with flat or curved multi-bandpass filters, as disclosed herein, represents a significant technological advancement over existing spectral imaging implementations. Specifically, based on the calculated parallax of the viewing angle differences between the apertures, the multi-aperture system can achieve the acquisition of object curvature, depth, volume and / or area or 3D spatial images related thereto. However, based on interference or absorptive filtering, the multi-aperture strategy proposed here is not limited to any particular filter and can include flat and / or thin filters. As disclosed herein, the present invention can be modified to include a flat filter in the image space of the imaging system using a suitable lens or aperture with a small or acceptable range of incident angles. The filter can also be placed at the aperture stop of the imaging lens or at the entrance / exit pupil, as can be seen by those skilled in the art of optical engineering.

[0062] Various aspects of the present disclosure will be described below with reference to certain examples and embodiments, which are intended to illustrate rather than limit the present disclosure. Although for illustrative purposes, the examples and embodiments described herein will focus on specific calculations and algorithms, it will be understood by those skilled in the art that these examples are merely illustrative and not restrictive. For example, although some examples are given in the context of multispectral imaging, in other embodiments, the disclosed multi-aperture imaging system and associated filters can be configured to achieve hyperspectral imaging. Further, although certain examples are provided to achieve the benefits of handheld and / or mobile target applications, it will be understood that the disclosed imaging system design and associated processing techniques can generate high-precision data cubes suitable for fixed imaging systems and / or for analyzing relatively immobile targets.

[0063] Overview of Electromagnetic Range and Image Sensors

[0064] Certain colors or portions of the electromagnetic spectrum are referred to herein and discussed below with reference to wavelengths defined by the ISO 21348 definition of irradiance spectral categories. As further described below, in certain imaging applications, wavelength ranges for specific colors may be grouped together to pass through specific filters.

[0065] Electromagnetic radiation ranging from a wavelength of about 760 nm to a wavelength of about 380 nm is generally considered to be the "visible" spectrum, i.e., the portion of the spectrum discernible by the color receptors of the human eye. Within the visible spectrum, red light is generally considered to have a wavelength of about 700 nanometers (nm), or within a range of about 760 nm to about 610 nm or about 610 nm. Orange light is generally considered to have a wavelength of about 600 nm, or within a range of about 610 nm to about 591 nm or about 591 nm. Yellow light is generally considered to have a wavelength of about 580 nm, or within a range of about 591 nm to about 570 nm or about 570 nm. Green light is generally considered to have a wavelength of about 550 nm, or within a range of about 570 nm to about 500 nm or about 500 nm. Blue light is generally considered to have a wavelength of about 475 nm, or within a range of about 500 nm to about 450 nm or about 450 nm. Violet light is generally considered to have a wavelength of about 400 nm, or within the range of about 450 nm to about 360 nm or 360 nm.

[0066] Turning to the range outside the visible spectrum, infrared (IR) light refers to electromagnetic radiation with wavelengths longer than visible light and is generally invisible to the human eye. IR wavelengths extend from the nominal red edge of the visible spectrum at about 760nm or 760nm to about 1 millimeter (mm) or 1mm. Within this range, near infrared (NIR) refers to the portion of the spectrum adjacent to the red range, wavelengths between about 760nm or 760nm and about 1400nm or 1400nm.

[0067] Ultraviolet (UV) light refers to certain electromagnetic radiation with shorter wavelengths than visible light and is generally invisible to the human eye. UV wavelengths extend from the nominal violet edge of the visible spectrum at about 40nm or 40nm to about 400nm. Within this range, near ultraviolet (NUV) refers to the portion of the spectrum adjacent to the violet range, with wavelengths between about 400nm or 400nm and about 300nm or 300nm, mid-ultraviolet (MUV) is wavelengths between about 300nm or 300nm and about 200nm or 200nm, and far ultraviolet (FUV) is wavelengths between about 200nm or 200nm and about 122nm or 122nm.

[0068] Depending on the specific wavelength range suitable for a particular application, the image sensor described herein can be configured to detect electromagnetic radiation within any of the above ranges. The spectral sensitivity of a typical silicon-based charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensor extends in the visible spectrum, and to a considerable extent extends to the near infrared (IR) spectrum, and sometimes extends to the UV spectrum. Some embodiments can selectively or additionally use a back-illuminated or front-illuminated CCD or CMOS array. For applications requiring high SNR and scientific-grade measurements, certain embodiments can selectively or additionally use a scientific research-grade complementary metal oxide semiconductor (sCMOS) camera or an electron multiplying CCD camera (EMCCD). Based on the intended application, other embodiments can selectively or additionally use sensors known to operate in a specific color range (e.g., short-wave infrared (SWIR), medium-wave infrared (MWIR) or long-wave infrared (LWIR)) and corresponding optical filter arrays. These can selectively or additionally include cameras based on surrounding detector materials (including indium gallium arsenide (InGaAs) or indium antimonide (InSb)) or based on surrounding microbolometer arrays.

[0069] The image sensor used in the multispectral imaging technology of the present disclosure can be used in conjunction with an optical filter array such as a color filter array (CFA). Some CFAs can split incident light in the visible light range into red (R), green (G), and blue (B) categories to direct the separated visible light to dedicated red, green, or blue photodiode receivers on the image sensor. A common example of a CFA is the Bayer pattern, which is a specific pattern for arranging RGB color filters on a rectangular grid of a light sensor. The Bayer pattern is 50% green, 25% red, and 25% blue, with repeated rows of red and green filters alternating with repeated rows of blue and green filters. Some CFAs (e.g., for RGB-NIR sensors) can also separate out NIR light and direct the separated NIR light to dedicated photodiode receivers on the image sensor.

[0070] Thus, the wavelength range of the filter component of the CFA can determine the wavelength range represented by each image channel in the captured image. Thus, the red channel of the image can correspond to the red wavelength region of the filter and can include some yellow and orange light, ranging from about 570nm or 570nm to about 760nm or 760nm in various embodiments. The green channel of the image can correspond to the green wavelength region of the filter and can include some yellow light, ranging from about 570nm or 570nm to about 480nm or 480nm in various embodiments. The blue channel of the image can correspond to the blue wavelength region of the filter and can include some violet light, ranging from about 490nm or 490nm to about 400nm or 400nm in various embodiments. As can be appreciated by those of ordinary skill in the art, the exact starting and ending wavelengths (or portions of the electromagnetic spectrum) that define the colors (e.g., red, green, and blue) of the CFA can vary depending on the implementation of the CFA.

[0071] In addition, a typical visible light CFA is transparent to light outside the visible spectrum. Therefore, in many image sensors, IR sensitivity is limited by a thin film reflective IR filter located on the surface of the sensor, which blocks infrared wavelengths while visible light passes through. However, it can be omitted in some of the disclosed imaging systems to allow IR light to pass through. Therefore, the red, green and / or blue channels can also be used to collect IR bands. In some embodiments, the blue channel can also be used to collect certain NUV bands. The different spectral responses of the red, green and blue channels in terms of unique transmission efficiency at each wavelength in the spectral image stack can provide unique weighted responses for spectral bands to be unmixed using known transmission profiles. For example, this can include known transmission responses in the IR and UV wavelength regions for the red, blue and green channels, allowing them to be used to collect bands from these regions.

[0072] As described in more detail below, additional filters may be placed before the CFA along the path of light toward the image sensor in order to selectively optimize specific bands of light incident on the image sensor. Some of the filters disclosed herein may be a combination of dichroic (thin film) and / or absorptive filters, or may be a single dichroic and / or absorptive filter. Some of the filters disclosed herein may be bandpass filters that pass frequencies within a specific range (within the passband) and reject (attenuate) frequencies outside of that range (within the blocking range). Some of the filters disclosed herein may be multi-bandpass filters that pass multiple discrete wavelength ranges. These "bands" may have a smaller passband range, a larger blocking range attenuation, and a steeper spectral roll-off (which is defined as the steepness of the spectral response as the filter transitions from the passband to the blocking range) than the larger gamut of the CFA filter. For example, these disclosed filters may cover a passband of approximately 20 nm or 20 nm or approximately 40 nm or 40 nm. The specific composition of such filters may determine the actual band of light incident on the sensor, which may improve the accuracy of the disclosed imaging techniques. The filters described herein may be configured to selectively block or pass specific bands of electromagnetic radiation within any of the ranges described above, depending on the specific bands suitable for a particular application.

[0073] As described herein, "pixel" can be used to describe the output generated by an element of a 2D detector array. In contrast, a photodiode (a single photosensitive element in the array) acts as a transducer capable of converting photons into electrons via the photoelectric effect, and then converting them into a usable signal for determining a pixel value. A single element of a data cube can be referred to as a "voxel" (e.g., a volume element). A "spectral vector" refers to a vector that describes spectral data at a specific (x, y) location in a data cube (e.g., a spectrum received from a specific point in object space). A single horizontal plane of a data cube (e.g., an image representing a single spectral dimension) is referred to herein as an "image channel." Certain embodiments described herein can capture spectral video information, and the resulting data dimensions can be in the form of a "hypercube" N. x N y N λ N l , where N t is the number of frames captured during the video sequence.

[0074] Overview of an Exemplary Multi-Aperture Imaging System with Curved Multi-Bandpass Filters

[0075] Figure 3AA schematic diagram of an exemplary multi-aperture imaging system 200 with a curved multi-bandpass filter according to the present disclosure is shown. The view shown includes a first image sensor region 225A (photodiodes PD1-PD3) and a second image sensor region 225B (photodiodes PD4-PD6). For example, in a CMOS image sensor, the photodiodes PD1-PD6 can be, for example, photodiodes formed in a semiconductor substrate. In general, each photodiode PD1-PD6 can be a single unit of any material, semiconductor, sensor element, or other device that converts incident light into an electrical current. It should be understood that for the purpose of illustrating its structure and operation, a small portion of the entire system is shown, and in implementation, the image sensor area can have hundreds or thousands of photodiodes (and corresponding filters). Depending on the implementation, the image sensor regions 225A and 225B can be implemented as separate sensors or as separate regions of the same image sensor. Although Figure 3A Two apertures and corresponding optical paths and sensor areas are shown, but it should be understood that depending on the implementation, Figure 3A The optical design principles shown can be extended to more than three apertures and the corresponding light paths and sensor areas.

[0076] The multi-aperture imaging system 200 includes a first opening 210A that provides a first optical path toward a first sensor region 225A and a second opening 210B that provides a second optical path toward a second sensor region 225B. The apertures may be adjustable to increase or decrease the brightness of light falling on an image, or so that the duration of a particular image exposure may be changed and the brightness of light falling on the image sensor region does not change. The apertures may also be located anywhere along the optical axis of the multi-aperture system as deemed reasonable by one skilled in the art of optical design. The optical axes of optical components located along the first optical path are shown by dashed lines 230A, and the optical axes of optical components located along the second optical path are shown by dashed lines 230B, and it should be understood that these dashed lines do not represent the physical structure of the multi-aperture imaging system 200. The optical axes 230A, 230B are separated by a distance D, which may result in parallax between the images captured by the first sensor region 225A and the second sensor region 225B. Parallax is the distance between two corresponding points in the left and right (or top and bottom) images of a stereo pair, so that the same physical point in object space can appear at different locations in each image. Processing techniques for compensating for and changing this parallax are described in further detail below.

[0077] Each optical axis 230A, 230B passes through the center C of the corresponding aperture, and the optical components may also be centered along these optical axes (e.g., points of rotational symmetry of the optical components may be located along the optical axes). For example, the first curved multi-bandpass filter 205A and the first imaging lens 215A may be centered along the first optical axis 230A, and the second curved multi-bandpass filter 205B and the second imaging lens 215B may be centered along the second optical axis 230B.

[0078] As used herein with respect to the positioning of optical elements, "on" and "above" refer to the location of a structure (e.g., a filter or lens) such that light entering the imaging system 200 from object space propagates through the structure and then reaches (or is incident upon) another structure. To illustrate, along a first optical path, a curved multi-bandpass filter 205A is located above an aperture 210A, which is located above an imaging lens 215A, which is located above a CFA 220A, and which is located above a first image sensor region 225A. Thus, light from object space (e.g., the physical space being imaged) first passes through the curved multi-bandpass filter 205A, then the aperture 210A, then the imaging lens 215A, then the CFA 220A, and finally is incident upon the first image sensor region 225A. The second optical path (e.g., curved multi-bandpass filter 205B, aperture 210B, imaging lens 215B, CF 220B, second image sensor region 225B) follows a similar arrangement. In other embodiments, aperture 210A, 210B and / or imaging lens 215A, 215B may be located above curved multi-bandpass filter 205A, 205B. In addition, other embodiments may not use a physical aperture and may rely on a transparent aperture of an optical device to control the brightness of light imaged onto sensor region 225A, 225B. Therefore, lens 215A, 215B may be placed above aperture 210A, 210B and curved multi-bandpass filter 205A, 205B. In this embodiment, aperture 210A, 210B and lens 215A, 215B may also be placed above or below each other if deemed necessary by a person skilled in the art of optical design.

[0079] The first CFA 220A located above the first sensor region 225A and the second CFA 220B located above the second sensor region 225B can act as wavelength selective pass filters and split the incident light in the visible range into red, green and blue ranges (as indicated by the R, G and B labels). The light is "split" by allowing only certain selected wavelengths to pass through each filter in the first CFA 220A and the second CFA 220B. The split light is received by a dedicated red, green or blue diode on the image sensor. Although red, blue and green filters are typically used, in other embodiments, the filters can be varied according to the color channel requirements of the captured image data, such as including ultraviolet, infrared or near infrared pass filters, such as RGB-IR CFA.

[0080] As shown, each filter of the CFA is located on a single photodiode PD1-PD6. Figure 3A Also shown are exemplary microlenses (represented by ML) that may be formed on or otherwise positioned above each color filter to focus incident light onto the active detector area. Other embodiments may have multiple photodiodes (e.g., clusters of 2, 4, or more adjacent photodiodes) under a single filter. In the example shown, photodiode PD1 and photodiode PD4 are under a red color filter and therefore will output red channel pixel information; photodiode PD2 and photodiode PD5 are under a green color filter and therefore will output green channel pixel information; photodiode PD3 and photodiode PD6 are under a blue color filter and therefore will output blue channel pixel information. In addition, as described in more detail below, based on the activated light emitters and / or the specific bands passed by the multi-bandpass filters 205A, 205B, the specific color channel output by a given photodiode can be further limited to a narrower band so that a given photodiode can output different image channel information during different exposures.

[0081] The imaging lenses 215A, 215B may be configured to focus an image of an object scene onto the sensor regions 225A, 225B. Each imaging lens 215A, 215B may be composed of numerous optical elements and surfaces required for image formation and is not limited to, for example, Figure 3AThe single convex lens shown enables the use of a variety of imaging lenses or lens assemblies, which can be obtained commercially or by custom design. Each element or lens assembly can be stacked or bonded together or maintained in series using an optical machine lens barrel with a fixing ring or a strip frame. In some embodiments, the element or lens assembly may include one or more bonded lens groups, such as two or more optical components that are glued or otherwise bonded together. In various embodiments, any multi-bandpass filter described herein may be located in front of the lens assembly of the multispectral image system, in front of the single lens of the multispectral image system, behind the lens assembly of the multispectral image system, behind the single lens of the multispectral image system, inside the lens assembly of the multispectral image system, inside the bonded lens group of the multispectral image system, directly on the surface of the single lens of the multispectral image system, or directly on the surface of the element of the lens assembly of the multispectral image system. In addition, apertures 210A and 210B can be removed, and lenses 215A, 215B can be various lenses commonly used in shooting with digital single-lens reflex (DSLR) or mirrorless cameras. Additionally, these lenses may be various lenses used in machine vision that are mounted using C-mount or S-mount threads. For example, focus adjustment may be provided by movement of the imaging lenses 215A, 215B relative to the sensor regions 225A, 225B or movement of the sensor regions 225A, 225B relative to the imaging lenses 215A, 215B based on manual focus, contrast-based autofocus, or other suitable autofocus techniques.

[0082] The multi-bandpass filters 205A, 205B can each be configured to selectively pass multiple narrow bands of light, such as bands of 10-50 nm in some embodiments (or wider or narrower bands in other embodiments). Figure 3A As shown, the multi-bandpass filters 205A and 205B can make the band λ c ("common wavelength band") is passed through. In embodiments having more than three optical paths, each multi-bandpass filter can pass the common wavelength band. In this way, each sensor region captures image information in the same wavelength band ("common channel"). As described in more detail below, the image information in the common channel can be used to align the image sets captured by each sensor region. Some embodiments can have one common wavelength band and corresponding common channel, or can have multiple common wavelength bands and corresponding common channels.

[0083] In addition to the common band λ cIn addition, each multi-bandpass filter 205A, 205B can be configured to selectively pass one or more unique wavelength bands. In this way, the imaging system 200 is able to increase the number of different spectral channels captured by the sensor regions 205A, 205B collectively beyond the number that can be captured by a single sensor region. Figure 3A Through the unique band λ u1 The multi-bandpass filter 205A passes a unique band λ u2 The multi-bandpass filter 205B is shown, where λ u1 and λ u2 represents different bands from each other. Although shown as passing two bands, the disclosed multi-band passbands can each pass a set of more than two bands. For example, as shown in FIG. Fig.11A and Fig. 11B As noted, some embodiments may each pass four bands. In various embodiments, a greater number of bands may be passed. For example, some four-phase embodiments may include a multi-bandpass filter configured to pass eight bands. In some embodiments, the number of bands may be, for example, 4, 5, 6, 7, 8, 9, 10, 12, 15, 16, or more bands.

[0084] The multi-bandpass filters 205A, 205B have a curvature selected to reduce the angle-dependent spectral transmission at each sensor region 225A, 225B. As a result, when receiving narrowband illumination from object space, each photodiode at a region of the sensor region 225A, 225B that is sensitive to that wavelength (e.g., the color filter above passes that wavelength) should receive light of substantially the same wavelength, rather than photodiodes near the edge of the sensor experiencing the above referenced wavelengths. Figure 1A This can produce more accurate spectral image data than using a flat filter.

[0085] Figure 3B Shown for Figure 3A An exemplary optical design of optical components of an optical path of a multi-aperture imaging system. Specifically, Figure 3B A custom achromatic doublet lens 240 is shown that can be used to provide a multi-bandpass filter 205A, 205B. The custom achromatic doublet lens 240 passes light through a housing 250 to an image sensor 225. The housing 250 can include the openings 210A, 210B and imaging lenses 215A, 215B described above.

[0086] The achromatic doublet 240 is configured to correct for optical aberrations introduced by the surfaces required to introduce the multi-bandpass filter coatings 205A, 205B. The achromatic doublet 240 shown includes two separate lenses that can be made of glass or other optical materials having different amounts of dispersion and different refractive indices. Other embodiments may use more than three lenses. These achromatic doublets can be designed to introduce the multi-bandpass filter coatings 205A, 205B on the curved front surface 242, while eliminating the introduced optical aberrations (existing by introducing the curved single lens optical surface in the deposited filter coatings 205A, 205B), while still limiting the optical or focusing power provided by the achromatic doublet 240 due to the combined action of the curved front surface 242 and the curved back surface 244, while still limiting the primary elements for focusing light to the lens housed within the housing 250. Therefore, the achromatic doublet 240 can contribute to the high accuracy of the image data captured by the system 200. The individual lenses can be mounted next to each other, such as bonded or cemented together, and shaped so that the aberrations of one of the lenses are canceled by the aberrations of the other. The curved front surface 242 or the curved back surface 244 of the achromatic doublet 240 can be coated with a multi-bandpass filter coating 205A, 205B. The systems described herein can be implemented using other doublet designs.

[0087] Further variations of the optical designs described herein may be implemented. For example, in some embodiments, the optical path may include a single lens or other optical single lens, such as Figure 3A The positive or negative meniscus single lens shown replaces Figure 3B Doublet lens 240 is shown. Figure 3C An exemplary embodiment is shown in which a flattening filter 252 is included between the lens housing 250 and the sensor 225 . Figure 3C The achromatic doublet lens 240 in housing 250 provides correction for the optical aberrations introduced by the inclusion of a flattening filter 252 having a multi-bandpass transmission profile, while not contributing significantly to the optical power provided by the lenses housed in housing 250. Figure 3D An example of another implementation is shown where the multi-band pass coating is achieved by applying a multi-band pass coating 254 to the front surface of a lens assembly housed within housing 250. As such, the multi-band pass coating 254 can be applied to any curved surface of any optical element located within housing 250.

[0088] Figures 4A-4E shows that the Figure 3A and Figure 3B The optical design of the multi-spectral multi-aperture imaging system 300 is described in detail. Figure 4AA perspective view of imaging system 300 is shown, with housing 305 shown semi-transparently to reveal internal components. Housing 305 may be larger or smaller than the housing 305 shown, for example, based on a desired number of embedded computing resources. Figure 4B A front view of imaging system 300 is shown. Figure 4C Shown along Figure 4B A cut-away side view of imaging system 300 is shown cut along line CC. Figure 4D A bottom view of the imaging system 300 is shown, showing the processing board 335. Figures 4A-4D .

[0089] The housing 305 of the imaging system 300 may be enclosed within another housing. For example, a handheld embodiment may enclose the system within the housing, optionally with one or more handles shaped to stably hold the imaging system 300. Figures 18A-18C and Figures 19A-19B An exemplary handheld embodiment is shown in more detail in . The upper surface of the housing 305 includes four openings 320A-320D. A different multi-bandpass filter 325A-325D is located above each opening 320A-320D and is held in place by a filter cover 330A-330B. As described herein, the multi-bandpass filters 325A-325D can be curved and each pass common bands and at least one unique band, thereby achieving high-precision multi-spectral imaging on a greater number of spectral channels than captured by the upper filter array of the image sensor. The above-described image sensor, imaging lens, and filter are located within a camera housing 345A-345D. In some embodiments, for example, as Figures 20A-20B As shown, a single camera housing can enclose the above-mentioned image sensor, imaging lens and filter. In the illustrated embodiment, separate sensors are therefore used (e.g., one sensor in each camera housing 345A-345D), but it will be appreciated that in other embodiments, a single image sensor spanning all areas exposed through openings 320A-320D can be used. In this embodiment, camera housings 345A-345D are secured to system housing 305 using support members 340, and other suitable means may be used for securing in various embodiments.

[0090] The upper surface of the housing 305 supports an optional lighting panel 310 covered by an optical diffuser element 315. Figure 4EThe illumination panel 310 is described in more detail. The diffusion element 315 can be composed of glass, plastic or other optical materials to diffuse the light emitted from the illumination panel 310 so that the object space receives substantially spatially uniform illumination. In certain imaging applications (e.g., clinical analysis of imaged tissue), uniform illumination of the targeted object may also be beneficial because it provides a substantially uniform amount of illumination across the entire object surface in a range of wavelengths. In some embodiments, the imaging system disclosed herein can also utilize ambient light instead of or in addition to the light from the optional illumination panel.

[0091] Due to the heat generated by the lighting board 310 during use, the imaging system 300 includes a heat sink 350 having a plurality of heat dissipation fins 355. The heat dissipation fins 355 can extend into the space between the camera housings 345A-345D, and the upper portion of the heat sink 350 can absorb heat from the lighting board 310 to the fins 355. The heat sink 350 can be made of a suitable thermally conductive material. The heat sink 350 can further help dissipate heat from other components, so that some embodiments of the imaging system can be fanless.

[0092] A plurality of supports 365 in the housing 305 hold a processing board 335 in communication with the cameras 345A-345D. The processing board 335 can control the operation of the imaging system 300. Although not shown, the imaging system 300 can also be configured with one or more memories, for example, to store data generated by using the imaging system and / or a module of computer executable instructions for system control. Depending on the system design goals, the processing board 335 can be configured in a variety of ways. For example, the processing board can be configured to control the activation of specific LEDs of the lighting board 310 (e.g., through a module of computer executable instructions). Certain embodiments can use a highly stable synchronous buck LED driver that can implement software control of analog LED current and detect LED failures. Some embodiments can additionally provide image data analysis functions to the processing board (e.g., through a module of computer executable instructions) 335 or to a separate processing board. Although not shown, the imaging system 300 may include a data interconnect between the sensor and the processing board 335 so that the processing board 335 can receive and process data from the sensor, and may include a data interconnect between the lighting board 310 and the processing board 335 so that the processing board can drive the activation of specific LEDs of the lighting board 310.

[0093] Figure 4EAn exemplary lighting board 310 is shown that can be included in the imaging system 300 in isolation from other components. The lighting board 310 includes four arms extending from a central area, and LEDs are placed in three columns along each arm. The spaces between the LEDs in adjacent columns are laterally offset from each other to create separation between adjacent LEDs. Each column of LEDs includes multiple rows with different colored LEDs. Four green LEDs 371 are located in the central area, with one green LED in each corner of the central area. Starting from the innermost row (e.g., closest to the center), each column includes a row of two crimson LEDs 372 (a total of 8 crimson LEDs). Continuing radially outward, each arm has a row of amber LEDs 374 in the center column, a row of two short blue LEDs 376 in the outermost column (8 short blue LEDs total), a row of amber LEDs 374 in the center column (8 amber LEDs total), a row of one non-PPG NIR LED 373 and one red LED 375 in the outermost column (4 each total), and one PPG NIR LED 377 in the center column (4 PPG NIR LEDs total). A "PPG" LED refers to an LED that is activated during multiple sequential exposures for capturing photoplethysmographic (PPG) information representative of pulsatile blood flow in living tissue. It will be appreciated that a variety of other colors and / or arrangements thereof may be used in other embodiments of the lighting panel.

[0094] Figure 5 Shown with reference Figure 3A and Figure 3B Another embodiment of a multi-spectral multi-aperture imaging system of the optical design described. Similar to the design of imaging system 300, imaging system 400 includes four optical paths, shown here as openings 420A-420D with multi-bandpass filter sets 425A-425D, which are fixed to housing 405 by retaining rings 430A-430D. Imaging system 400 also includes: an illumination board 410, which is fixed to the front of housing 405 between retaining rings 430A-430D; and a diffuser 415, which is located above illumination board 410 to help emit spatially uniform light onto a target object.

[0095] The lighting board 410 of system 400 includes four branches of LEDs in the shape of a cross, each branch including two columns of closely spaced LEDs. Thus, the lighting board 410 is more compact than the lighting board 310 described above, and may be suitable for use with imaging systems having smaller form factor requirements. In this exemplary configuration, each branch includes an outermost row of one green LED and one blue LED, moving inward to include two rows of yellow LEDs, a row of orange LEDs, a row with one red LED and one deep red LED, and a row with one amber LED and one NIR LED. Thus, in this embodiment, the LEDs are arranged such that the LEDs emitting longer wavelengths of light are in the center of the lighting board 410, while the LEDs emitting shorter wavelengths of light are at the edges of the lighting board 410.

[0096] Figures 6A-6C shows that the Figure 3A and Figure 3B Another embodiment of the multi-spectral multi-aperture imaging system 500 of the optical design is described. Specifically, Fig. 6A shows a perspective view of an imaging system 500, Figure 6B shows a front view of an imaging system 500, Figure 6C Shown along Figure 6B A cutaway side view of imaging system 500 cut along line CC is shown. Imaging system 500 includes components similar to those described above with respect to imaging system 300 (e.g., housing 505, illumination panel 510, diffuser panel 515, multi-bandpass filters 525A-525D secured to openings via retaining rings 530A-530D), but shows a shorter form factor (e.g., in embodiments with fewer and / or smaller embedded computing components). System 500 also includes a direct camera-to-frame mount 540 to increase stiffness and robustness of camera alignment.

[0097] Figure 7A-7B Another embodiment of a multispectral multi-aperture imaging system 600 is shown. Figure 7A-7B Another possible arrangement of light sources 610A-610C around the multi-aperture imaging system 600 is shown. Figures 3A-3D The four lens assemblies of the illustrated optical design multi-bandpass filter 625A-625D can be arranged in a rectangular or square configuration to provide light to four cameras 630A-630D (including image sensors). Three rectangular light emitting elements 610A-610C can be arranged parallel to each other outside and between the lens assemblies with the multi-bandpass filter 625A-625D. These can be broad spectrum light emitting panels or LED arrangements that emit discrete bands of light.

[0098] Figures 8A-8BAnother embodiment of a multispectral multi-aperture imaging system 700 is shown. Figures 8A-8B Another possible arrangement of light sources 710A-710D around the multi-aperture imaging system 700 is shown. Figures 3A-3D The four lens assemblies of the illustrated optical design multi-bandpass filters 725A-725D can be configured in a rectangular or square configuration to provide light to four cameras 730A-730D (including image sensors). The four cameras 730A-730D are shown in a closer exemplary configuration, which can minimize the difference in viewing angles between the lenses. Four rectangular light emitting elements 710A-710D can be placed in a square around the lens assembly with the multi-bandpass filters 725A-725D. These can be broad spectrum light emitting panels or LED arrangements that emit discrete bands of light.

[0099] Figures 9A-9C Another embodiment of a multi-spectral multi-aperture imaging system 800 is shown. The imaging system 800 includes a frame 805 connected to a lens assembly frame front portion 830, which includes an opening 820 for a micro video lens 825 and a support structure, which may be provided with a reference Figures 3A-3D The optical design of the multi-bandpass filter is illustrated. The micro video lens 825 provides light to four cameras 845 (including imaging lenses and image sensor areas) mounted on the rear 840 of the lens assembly frame. Four linearly arranged LEDs 811 are configured along the four sides of the front 830 of the lens assembly frame, each with its own diffusion element 815. Fig. 9B and Fig. 9C Example dimensions in inches are shown to illustrate one possible size for multi-aperture imaging system 800 .

[0100] Fig. 10A Shown with reference Figures 3A-3D Another embodiment of a multi-spectral multi-aperture imaging system 900 of an illustrated optical design. The imaging system 900 can be implemented as a set of multi-bandpass filters 905 that can be attached to a multi-aperture camera 915 of a mobile device 910. For example, certain mobile devices 910 such as smart phones can be equipped with a stereoscopic imaging system with two openings leading to two image sensor areas. The disclosed multi-aperture spectral imaging technology can be implemented in these devices by providing a set of suitable multi-bandpass filters 905 to pass multiple narrower bands of light to the sensor area. Optionally, the set of multi-bandpass filters 905 can be equipped with a light source (such as an LED array and a diffuser) that provides these bands of light to the object space.

[0101] The system 900 may also include a mobile application configured to perform processing to generate a multispectral data cube and to process the multispectral data cube (e.g., for clinical tissue classification, biometric identification, material analysis, or other applications). Optionally, the mobile application may be configured to cause the device 910 to transmit the multispectral data cube to a remote processing system over a network and then receive and display the analysis results. Fig. 10B An exemplary user interface 910 for such an application is shown in FIG.

[0102] Figures 11A-11B It shows that it can be Figure 3A-10B A set of exemplary wavebands passed by the filters of a four-filter embodiment of a multispectral multi-aperture imaging system of FIG. 5 to an image sensor having, for example, a Bayer CFA (or other RGB or RGB-IR CFA). The spectral transmission response of the wavebands passed by the multi-bandpass filter is given by Fig.11A The solid line in the graph 1000 is represented by where n is the camera number, ranging from 1 to 4. The dotted line indicates Compared to the green pixels present in a typical Bayer CFA , red pixel or blue pixel The combined spectral response of the spectral transmission of the image sensor is shown in Figure 1. These transmission curves also include the effects of quantum efficiency due to the sensors used in this example. As shown, the set of four cameras collectively captures eight unique channels or bands. Each filter passes two common bands (the two leftmost peaks) as well as two additional bands to its respective camera. In this embodiment, the first and third cameras receive light in the first shared NIR band (the rightmost peak) and the second and fourth cameras receive light in the second shared NIR band (the second rightmost peak). Each camera also receives a unique band ranging from approximately 550nm or 550nm to approximately 800nm ​​or 800nm. Therefore, the cameras can capture eight unique spectral channels using a compact configuration. Fig. 11B The graph 1010 in FIG. Figure 4E The spectral irradiance of the LED panel is shown in Fig.11A Illumination of the four cameras shown in .

[0103] In this embodiment, eight bands have been selected based on spectral channels that produce suitable clinical tissue categories, and can also be optimized for signal-to-noise ratio (SNR) and frame rate while limiting the number of LEDs (introducing heat into the imaging system). The eight bands include a common band of blue light (the leftmost peak in graph 1000) that is passed by all four filters, because tissues (e.g., animal tissues including human tissue) show higher contrast at blue wavelengths than at green or red wavelengths. Specifically, as shown in graph 1000, human tissue exhibits its highest contrast when imaged at a band centered at 420nm. Since channels corresponding to the common bands are used for parallax correction, this higher contrast can generate more accurate corrections. For example, in parallax correction, the image processor can use a local or global approach to find a set of parallaxes, thereby maximizing the quality factor corresponding to the similarity between local image patches or images. Alternatively, the image processor can use a similar approach that minimizes the quality factor corresponding to dissimilarity. These quality factors can be based on entropy, correlation, absolute difference, or based on deep learning methods. Global methods for disparity calculation can be run iteratively and terminate when the quality factor stabilizes. Local methods can be used to calculate disparity point by point, using a fixed patch in one image as input to the quality factor and using many different patches, each determined from the other image by a different value of the measured disparity. All of these methods may impose constraints on the range of disparities considered. For example, these constraints can be based on knowledge of the depth and distance of objects. Constraints can also be imposed based on the expected range of gradients in objects. Constraints on the calculated disparities can also be imposed by projective geometry, such as epipolar constraints. Disparities can be calculated at multiple resolutions, and the output of the disparity calculated at a lower resolution will serve as an initial value or constraint on the disparity calculated at the next resolution level. For example, a disparity calculated at a resolution level of 4 pixels in one calculation can be used to set a constraint of ±4 pixels in the calculation of the disparity at the next higher resolution. All algorithms that work from disparity calculation will benefit from higher contrast, especially if the source of this contrast is correlated for all viewpoints. In general, common wavebands may be selected based on the highest contrast imaging corresponding to the material expected to be imaged for a particular application.

[0104] After the image is captured, the color separation between adjacent channels may not be perfect, so the embodiment also has an additional common band that is passed by all filters, shown in the graph 1000 as a green band adjacent to the blue band. This is because the blue filter pixels are sensitive to the region of the green spectrum due to their wide spectral passband. This is usually manifested as spectral overlap between adjacent RGB pixels, and may also be characterized by intentional crosstalk. This overlap makes the spectral sensitivity of the color camera similar to that of the human retina, making the obtained color space similar in quality to human vision. Therefore, by separating out the portion of the signal due to green light, having a common green channel can enable the separation of the portion of the signal generated by the blue photodiode that truly corresponds to the received blue light. This can be achieved by using a spectral unmixing algorithm, which factors in the transmittance of the multi-bandpass filter (represented by T in the legend with solid black lines), the transmittance of the corresponding CFA color filter (represented by Q in the legend with red, green and blue dashed lines). It will be appreciated that some embodiments may use red light as the common band, and in this case, the second common channel may not be necessary.

[0105] Fig.12 A high-level block diagram of an exemplary compact imaging system 1100 having high-resolution spectral imaging capabilities is shown, the system 1100 having a set of components including a processor 1120 linked to a multi-aperture spectral camera 1160 and an illuminator 1165. Working memory 1105, storage 1110, electronic display 1125, and memory 1130 are also in communication with the processor 1120. As described herein, by using different multi-bandpass filters placed on different openings of the multi-aperture spectral camera 1160, the system 1100 can capture a greater number of image channels than if filters of different colors were present in the CFA of the image sensor.

[0106] System 1100 may be a device such as a cellular phone, a digital camera, a tablet computer, a personal digital assistant, etc. System 1100 may also be a more stationary device such as a desktop personal computer, a video conferencing station, etc., which uses an internal or external camera to capture images. System 1100 may also be a combination of an image capture device and a separate processing device that receives image data from the image capture device. A user may use multiple applications on system 1100. These applications may include traditional photography applications, still image and video capture, dynamic color correction applications, brightness and shading correction applications, etc.

[0107] Image capture system 1100 includes a multi-aperture spectral camera 1160 for capturing images. Multi-aperture spectral camera 1160 may be, for example, Figure 3A-10BThe multi-aperture spectral camera 1160 may be connected to the processor 1120 to transmit images captured in different spectral channels and from different sensor regions to the image processor 1120. The illuminators 1165 may also be controlled by the processor to emit certain wavelengths during certain exposures, as described in more detail below. The image processor 1120 may be configured to perform various operations on the received captured images to output a high-quality, parallax-corrected multi-spectral data cube.

[0108] Processor 1120 can be a general purpose processing unit or a processor designed specifically for imaging applications. As shown, processor 1120 is connected to memory 1130 and working memory 1105. In the illustrated embodiment, memory 1130 stores capture control module 1135, data cube generation module 1140, data cube analysis module 1145 and operating system 1150. These modules include instructions for configuring the processor to perform various image processing and device management tasks. Processor 1120 can use working memory 1105 to store a working set of processor instructions contained in the modules of memory 1130. Optionally, processor 1120 can also use working memory 1105 to store dynamic data created during the operation of device 1100.

[0109] As described above, the processor 1120 is composed of several modules stored in the memory 1130. In some embodiments, the capture control module 1135 includes instructions for configuring the processor 1120 to adjust the focus position of the multi-aperture spectral camera 1160. The capture control module 1135 also includes instructions for configuring the processor 1120 to capture images (e.g., multi-spectral images captured at different spectral channels and PPG images (e.g., NIR) captured at the same spectral channel) using the multi-aperture spectral camera 1160. Non-contact PPG imaging typically uses near-infrared (NIR) wavelengths as illumination to take advantage of increased photon penetration into tissue at this wavelength. Therefore, the processor 1120, together with the capture control module 1135, the multi-aperture spectral camera 1160, and the working memory 1105, represent a means for capturing a set of spectral images and / or image sequences.

[0110] The data cube generation module 1140 includes instructions for configuring the processor 1120 to generate a multispectral data cube based on intensity signals received from photodiodes of different sensor regions. For example, the data cube generation module 1140 can estimate the disparity between the same regions of the imaged object based on the spectral channels corresponding to the common bands passed by all multi-bandpass filters, and can use the disparity to register all spectral images on all capture channels to each other (e.g., so that the same point on the object is represented by approximately the same (x, y) pixel position on all spectral channels). The registered images together form a multispectral data cube, and the disparity information can be used to determine the depth of different imaged objects, such as the depth difference between healthy tissue and the deepest location within a wound site. In some embodiments, the data cube generation module 1140 can also perform spectral unmixing to identify which portions of the photodiode intensity signal correspond to which passed band, for example, based on a spectral unmixing algorithm that factors in the transmittance of the filter and the sensor quantum efficiency.

[0111] Depending on the application, the data cube analysis module 1145 can implement various techniques to analyze the multispectral data cube generated by the data cube generation module 1140. For example, some embodiments of the data cube analysis module 1145 can provide the multispectral data cube (and optionally the depth information) to a machine learning model that is trained to classify each pixel according to a specific state. In the case of tissue imaging, these states can be clinical states, such as burn states (e.g., first degree burns, second degree burns, third degree burns, or healthy tissue categories), wound states (e.g., hemostasis, inflammation, hyperplasia, remodeling, or healthy skin categories), healing potential (e.g., a score reflecting the likelihood that the tissue will heal from a wound state with or without a specific therapy), perfusion state, cancerous state, or other wound-related tissue states. The data cube analysis module 1145 can also analyze the multispectral data cube for biometric identification and / or material analysis.

[0112] The operating system module 1150 configures the processor 1120 to manage the memory and processing resources of the system 1100. For example, the operating system module 1150 may include device drivers to manage hardware resources such as the electronic display 1125, the memory 1110, the multi-aperture spectral camera 1160, or the illuminant 1165. Therefore, in some embodiments, the instructions contained in the above-mentioned image processing module may not interact directly with these hardware resources, but rather interact through standard subroutines or APIs located in the operating system component 1150. The instructions within the operating system 1150 can then interact directly with these hardware components.

[0113] The processor 1120 may further be configured to control a display 1125 to display captured images and / or results of analyzing a multispectral data cube (e.g., a classified image) to a user. The display 1125 may be external to the imaging device including the multi-aperture spectral camera 1160, or may be part of the imaging device. The display 1125 may also be configured to provide a viewfinder for the user prior to capturing an image. The display 1125 may include an LCD or LED screen, and may implement touch-sensitive technology.

[0114] The processor 1120 may write data to the storage module 1110, such as data representing captured images, multispectral data cubes, and data cube analysis results. Although the storage module 1110 is graphically represented as a conventional magnetic disk device, it will be understood by those skilled in the art that the storage module 1110 may be configured as any storage medium device. For example, the storage module 1110 may include a magnetic disk drive such as a floppy disk drive, a hard disk drive, an optical disk drive, or a magneto-optical disk drive, or a solid-state memory such as a flash memory, RAM, ROM, and / or EEPROM. The storage module 1110 may also include a plurality of storage units, and any one of the storage units may be configured to be internal to the image capture device 1100, or may be external to the image capture system 1100. For example, the storage module 1110 may include a ROM memory containing system program instructions stored within the image capture system 1100. The storage module 1110 may also include a memory card or high-speed memory that can be removed from the camera and configured to store captured images.

[0115] although Fig.12 A system including separate components including a processor, an imaging sensor, and a memory is shown, but those skilled in the art will recognize that these separate components can be combined in a variety of ways to achieve specific design goals. For example, in an alternative embodiment, a memory component can be combined with a processor component to save cost and improve performance.

[0116] In addition, although Fig.12Two memory components are shown (a memory component 1130 including several modules and a separate memory 1105 including working memory), but those skilled in the art will recognize several embodiments utilizing different memory architectures. For example, a design can utilize ROM or static RAM memory to store processor instructions that implement the modules contained in memory 1130. Alternatively, processor instructions can be read from a disk storage device integrated into system 1100 or connected via an external device port at system startup. The processor instructions can then be loaded into RAM for processor execution. For example, working memory 1105 can be a RAM memory, wherein instructions are loaded into working memory 1105 before being executed by processor 1120.

[0117] Overview of Exemplary Image Processing Techniques

[0118] Fig.13 Is used for Figure 3A-10B and Fig.12 Flowchart of an exemplary process 1200 for capturing image data with a multispectral multi-aperture imaging system. Fig.13 Four exemplary exposures are shown that can be used to generate the multispectral data cube described herein, visible light exposure 1205, additional visible light exposure 1210, invisible light exposure 1215, and ambient light exposure 1220. It will be appreciated that these exposures can be captured in any order and that certain exposures can be optionally removed or added from a particular workflow as described below. Fig.11A and Fig. 11B The band description process 1200 of FIG. 1200 is shown in FIG. 1201 , however, similar workflows can be implemented using image data generated based on other band sets. In addition, in various embodiments, flat field correction can also be implemented according to various known flat field correction techniques to improve image acquisition and / or parallax correction.

[0119] For visible light exposure 1205, the first five peaks can be turned on by control signals to the lighting board ( Fig.11A 1000 corresponds to the left five peaks of visible light. At a time specific to the particular LED (e.g., 10 ms), it may be necessary to stabilize the output wave of light. The capture control module 1135 may start the exposure of the four cameras after that time, and may, for example, continue for a duration of approximately 30 ms. Thereafter, the capture control module 1135 may stop the exposure and pull the data out of the sensor area (e.g., by transferring the raw photodiode intensity signal to the working memory 1105 and / or the data memory 1110). The data may include common spectral channels for parallax correction as described herein.

[0120] To increase the SNR, some embodiments may capture additional visible light exposures 1210 using the same process described for visible light exposure 1205. Having two identical or nearly identical exposures may improve the SNR to generate a more accurate analysis of the image data. However, in embodiments where the SNR of a single image is acceptable, this may be omitted. In some embodiments, repeated exposures using common spectral channels may also enable more accurate parallax correction.

[0121] Some embodiments may also capture invisible light exposures 1215 corresponding to NIR or IR light. For example, the capture control module 1135 may activate a corresponding Fig.11A Two different NIR LEDs for the two NIR channels shown. At a time specific to the specific LED (e.g., 10ms), it may be necessary to stabilize the output wave of the light. The capture control module 1135 can start the exposure of the four cameras after this time, and can, for example, last for a duration of approximately 30ms. Thereafter, the capture control module 1135 can stop the exposure and pull the data out of the sensor area (e.g., by transferring the raw photodiode intensity signal to the working memory 1105 and / or the data memory 1110). In this exposure, there may be no common bands leading to all sensor areas, because it can be safely assumed that the shape or position of the object relative to the exposure 1205, 1210 has not changed, so the previously calculated disparity values ​​can be used to align the NIR channels.

[0122] In some embodiments, multiple exposures may be captured sequentially to generate PPG data representing shape changes of a tissue site due to pulsating blood flow. In some embodiments, these PPG exposures may be captured at non-visible wavelengths. Although the combination of PPG data with multispectral data may improve the accuracy of certain medical imaging analyses, the capture of PPG data may also introduce additional time in the image capture process. In some embodiments, this additional time may introduce errors due to movement of the handheld imager and / or the object. Therefore, some embodiments may omit the capture of PPG data.

[0123] Some embodiments may additionally capture an ambient light exposure 1220. For this exposure, all LEDs may be turned off to capture images using ambient lighting (e.g., daylight, light from other illuminant sources). The capture control module 1135 may start the exposure of the four cameras after this time, and may keep the exposure going for a desired duration of, for example, approximately 30 ms. Thereafter, the capture control module 1135 may stop the exposure and pull the data out of the sensor area (e.g., by transferring the raw photodiode intensity signal to the working memory 1105 and / or the data memory 1110). The intensity value of the ambient light exposure 1220 may be subtracted from the value of the visible light exposure 1205 (or the visible light exposure 1205 corrected for SNR by the second exposure 1210) and from the invisible light exposure 1215 to remove the effects of ambient light from the multispectral data cube. By isolating the portion of the light emitted by the illuminant and reflected from the object / tissue site that represents the generated signal, the accuracy of downstream analysis may be improved. Some embodiments may omit this step if the accuracy of the analysis using only the exposures of visible light 1205, 1210 and invisible light 1215 is sufficient.

[0124] It should be understood that the specific exposure times listed above are examples of one implementation, and in other implementations, the exposure times may vary depending on the image sensor, light source intensity, and the object being imaged.

[0125] Fig.14 A schematic block diagram of a workflow 1300 for processing image data is shown, for example using Fig.13 1200 and / or use of Figure 3A-10B and Fig.12 The workflow 1300 shows the output of two RGB sensor regions 1301A, 1301B, but the workflow 1300 can be extended to a larger number of sensor regions and sensor regions corresponding to different CFA color channels.

[0126] The RGB sensor outputs from the two sensor regions 1301A, 1301B are stored in 2D sensor output modules 1305A, 1305B, respectively. The values ​​of the two sensor regions are sent to non-linear mapping modules 1310A, 1310B, which can perform parallax correction by identifying the parallax between images captured using a common channel and then applying this determined parallax across all channels to register all spectral images to each other.

[0127] The outputs of the two non-linear mapping modules 1310A, 1310B are then fed to a depth calculation module 1335, which can calculate the depth of a particular area of ​​interest in the image data. For example, the depth can represent the distance between an object and the image sensor. In some embodiments, multiple depth values ​​can be calculated and compared to determine the depth of an object relative to an object outside the image sensor. For example, the maximum depth of a wound bed and the depth (maximum, minimum, or average) of healthy tissue surrounding the wound bed can be determined. By subtracting the depth of the healthy tissue from the depth of the wound bed, the deepest depth of the wound can be determined. This depth comparison can be additionally performed at other points in the wound bed (e.g., all or some predetermined samples) to establish a 3D map of the wound depth at each point (in Fig.14 , where z is the depth value). In some embodiments, greater disparity can improve depth calculations, although greater disparity can also result in more computationally intensive algorithms for such depth calculations.

[0128] The outputs of the two nonlinear mapping modules 1310A, 1310B are also fed to the linear equation module 1320, which can treat the sensed values ​​as a set of linear equations for spectral unmixing. One embodiment can use the Moore-Penrose pseudo-inverse equation as a function of at least the sensor quantum efficiency and the filter transmittance value to calculate the actual spectral value (e.g., the intensity of light of a specific wavelength incident at each (x, y) image point). This can be used in embodiments requiring high accuracy, such as clinical diagnostics and other biological applications. The application of spectral unmixing can also provide estimates of photon flux and SNR.

[0129] Based on the parallax-corrected spectral channel images and spectral unmixing, workflow 1300 can generate a spectral data cube 1325, for example in the F(x,y,λ) format shown, where F represents the intensity of light at a specific (x,y) image location at a specific wavelength or band λ.

[0130] Fig.15 The disparity and disparity correction for processing image data are graphically shown, for example using Fig.13 The process and / or use of Figure 3A-10B and Fig.1214. Image data captured by a multispectral, multi-aperture imaging system of FIG. 14. A first set of images 1410 shows image data of the same physical location on an object captured by four different sensor areas. As shown, the object location is not in the same location on the original image based on the (x, y) coordinate system of the photodiode grid of the image sensor area. A second set of images 1420 shows the same object location after parallax correction, which is now at the same (x, y) location in the coordinate system of the registered image. It should be understood that such registration may involve cropping certain data from edge areas of images that do not completely overlap each other.

[0131] Fig.16 A workflow 1500 is graphically illustrated for performing pixel-by-pixel classification on multispectral image data, for example using Fig.13 process, based on Fig.14 and Fig.15 Processing and / or use Figure 3A-10B and Fig.12 Image data captured by a multispectral multi-aperture imaging system.

[0132] At block 1510, a multispectral multi-aperture imaging system 1513 may capture image data representing a physical point 1512 on an object 1511. In this example, the object 1511 includes tissue of a patient having a wound. The wound may include a burn, a diabetic ulcer (e.g., a diabetic foot ulcer), a non-diabetic ulcer (e.g., a pressure sore or a slow-healing wound), a chronic ulcer, a post-surgical incision, an amputation site (before or after amputation surgery), a cancerous lesion, or damaged tissue. In the case of including PPG information, the disclosed imaging system provides a method for evaluating pathologies involving changes in tissue blood flow and pulse rate, including: tissue perfusion; cardiovascular health; wounds, such as ulcers; peripheral arterial disease, and respiratory health.

[0133] At block 1520, the data captured by the multispectral multi-aperture imaging system 1513 may be processed into a multispectral data cube 1525 having a plurality of different wavelengths 1523 and, optionally, a plurality of different images corresponding to different times at the same wavelength (PPG data 1522). For example, the image processor 1120 may be configured by the data cube generation module 1140 to generate the multispectral data cube 1525 according to the workflow 1300. As described above, some embodiments may also associate a depth value with each point along a spatial dimension.

[0134] At box 1530, the multispectral data cube 1525 can be analyzed as input data 1525 to a machine learning model 1532 to generate a classification map 1535 of the imaged tissue. The classified map can assign each pixel in the image data (which, after registration, represents a specific point on the imaged object 1511) to a certain tissue class or a certain healing potential score. Visually different colors or patterns can be used in the output classified image to represent different classifications and scores. Therefore, even if multiple images of the object 1511 are captured, the output can be a single image of the object (e.g., a typical RGB image) overlaid with a visual representation of the pixel-by-pixel classification.

[0135] In some embodiments, the machine learning model 1532 may be an artificial neural network. An artificial neural network is artificial in the sense that it is a computational entity inspired by biological neural networks, but modified by a computing device for implementation. Artificial neural networks are used to model complex relationships between inputs and outputs or to find patterns in data where dependencies between inputs and outputs cannot be easily determined. A neural network typically includes an input layer, one or more intermediate ("hidden") layers, and an output layer, each of which contains multiple nodes. The number of nodes may vary between layers. When a neural network includes more than two hidden layers, it is considered "deep". The nodes in each layer are connected to some or all of the nodes in the next layer, and the weights of these connections are typically learned from the data during training, such as by backpropagation, where the network parameters are adjusted to generate expected outputs given corresponding inputs in labeled training data. Therefore, an artificial neural network is an adaptive system that is constructed to change its structure (e.g., connection structure and / or weights) based on information flowing through the network during training, and the weights of the hidden layers can be considered as encodings of meaningful patterns in the data.

[0136] A fully connected neural network is a network where every node in the input layer is connected to every node in the subsequent layer (first hidden layer), then every node in the first hidden layer is connected to every node in the subsequent hidden layer, and so on, until every node in the final hidden layer is connected to every node in the output layer.

[0137] CNN is an artificial neural network, similar to the artificial neural network described above, which consists of nodes and has learnable weights. However, the layers of CNN can have nodes arranged in three dimensions: width, height and depth, corresponding to a 2×2 array of pixel values ​​in each video frame (e.g., width and height) and the number of video frames in the sequence (e.g., depth). The nodes of the layer can only be locally connected to a small area of ​​the width and height layer before it, called a receptive field. The hidden layer weights can take the form of a convolution filter applied to the receptive field. In some embodiments, the convolution filter can be two-dimensional, so that convolutions with the same filter can be repeated for each frame in the input volume or for a specified subset of frames. In other embodiments, the convolution filter can be three-dimensional, so it extends through the entire depth of the node of the input volume. The nodes in each convolution layer of CNN can share weights, so that the convolution filter of a given layer can be replicated over the entire width and height of the input volume (e.g., over the entire frame), thereby reducing the total number of trainable weights and increasing the applicability of CNN to data sets other than training data. The values ​​of a layer can be pooled to reduce the number of computations in subsequent layers (e.g., values ​​representing certain pixels can be passed forward while others are discarded), and further along the depth of the CNN pool, a mask can reintroduce any discarded values ​​to return the number of data points to the previous size. Many layers (optionally some of which are fully connected) can be stacked to form a CNN architecture.

[0138] During training, the artificial neural network can be exposed to pairs in the training data, and its parameters can be modified to predict the output of the pairs when the input is provided. For example, the training data can include, for example, a multispectral data cube (input) and a classification map (expected output) that has been labeled by a clinician (who has designated wound areas corresponding to certain clinical states), and / or has a healing (1) or non-healing (0) label at some time after the wound was initially imaged when actual healing is known. Other embodiments of the machine learning model 1532 can be trained to make other types of predictions, such as the likelihood that a wound will heal to a specific percentage of area reduction within a specified time period (e.g., at least 50% area reduction within 30 days) or wound state (e.g., hemostasis, inflammation, hyperplasia, remodeling, or healthy skin category). Some embodiments can also incorporate patient metrics into the input data to further improve classification accuracy, or can segment the training data based on patient metrics to train different instances of the machine learning model 1532 for use with other patients with those same patient metrics. Patient indicators may include textual information or medical history describing patient characteristics or patient health conditions or aspects thereof, such as the area of ​​wounds, lesions or ulcers, the patient's BMI, the patient's diabetic status, the presence of peripheral vascular disease or chronic inflammation in the patient, the number of other wounds the patient has or has had, whether the patient is taking or has recently taken immunosuppressant drugs (e.g., chemotherapy) or other drugs that have a positive or negative impact on wound healing rate, HbA1c, stage IV chronic renal failure, type II vs. type I diabetes, chronic anemia, asthma, medication use, smoking status, diabetic neuropathy, deep vein thrombosis, previous myocardial infarction, transient ischemic attack or sleep apnea, or any combination thereof. These indicators may be converted to vector representations, vectors with binary values ​​indicating whether the patient has a patient indicator (e.g., whether the patient has type I diabetes), or numerical values ​​indicating the extent to which the patient has each patient indicator by appropriate processing (e.g., by word-to-vector embedding).

[0139] At box 1540, the classification map 1535 can be output to the user. In this example, the classification map 1535 uses a first color 1541 to represent pixels classified according to a first state, and uses a second color 1542 to represent pixels classified according to a second state. The classification and the obtained classification map 1535 can exclude background pixels, for example, based on object recognition, background color recognition, and / or depth values. As shown, some embodiments of the multi-spectral multi-aperture imaging system 1513 can project the classification map 1535 back onto the tissue site. This may be particularly beneficial when the classification map includes a visual representation of the recommended resection boundary and / or depth.

[0140] These methods and systems can provide assistance to clinicians and surgeons in the process of skin wound management, such as burn excision, amputation level, lesion removal and wound classification diagnosis. The alternatives described herein can be used to identify and / or classify the severity of decubitus ulcers, congestion, limb deterioration, Raynaud's phenomenon, scleroderma, chronic wounds, abrasions, lacerations, hemorrhages, rupture injuries, punctures, penetrating wounds, skin cancer (basal cell carcinoma, squamous cell carcinoma, melanoma, actinic keratosis) or any type of tissue changes and / or classify them, wherein the nature and quality of the tissue are different from the normal state. The devices described herein can also be used to monitor healthy tissue, promote and improve wound treatment procedures, such as allowing faster and more sophisticated methods to determine debridement boundaries, and assess the progress of recovery from trauma or disease, especially after treatment. In some alternatives described herein, a device that allows identification of healthy tissue adjacent to injured tissue, determination of excision boundaries and / or depth, monitoring of the recovery process after implantation of a prosthesis (e.g., a left ventricular assist device), assessment of the viability of a tissue graft or regenerative cell implant, or monitoring of surgical recovery is provided, especially after a reconstruction procedure. In addition, the alternatives described herein can be used to assess post-traumatic changes or the generation of healthy tissue after trauma, particularly following the introduction of therapeutic agents such as steroids, hepatocyte growth factor, fibroblast growth factor, antibiotics, or regenerative cells (such as isolated or concentrated cell populations that include stem cells, endothelial cells and / or endothelial precursor cells).

[0141] Overview of an Exemplary Distributed Computing Environment

[0142] Fig.17 Shown include can be Figure 3A-10B and Fig.12 Schematic block diagram of an example distributed computing system 1600 of a multispectral multi-aperture imaging system 1605 of any multispectral multi-aperture imaging system of the present invention. The data cube analysis server 1615 may include one or more computers, which may be configured as a server cluster or as a server farm. The memory and processors that make up these computers may be located in a single computer or distributed among many computers (including computers that are remote from each other).

[0143] The multispectral multi-aperture imaging system 1605 may include network hardware (e.g., wireless internet, satellite, Bluetooth, or other transceiver) for communicating with the user device 1620 and the data cube analysis server 1615 via the network 1610. For example, in some embodiments, the processor of the multispectral multi-aperture imaging system 1605 may be configured to control image capture and then send raw data to the data cube analysis server 1615. Other embodiments of the processor of the multispectral multi-aperture imaging system 1605 may be configured to control image capture and perform spectral unmixing and parallax correction to generate a multispectral data cube, which is then sent to the data cube analysis server 1615. Some embodiments may perform complete processing and analysis locally on the multispectral multi-aperture imaging system 1605, and may send the multispectral data cube and the obtained analysis to the data cube analysis server 1615 for overall analysis and / or for training or retraining a machine learning model. In this way, the data cube analysis server 1615 can provide updated machine learning models to the multispectral multi-aperture imaging system 1605. The processing load of generating the final results of analyzing the multispectral data cube may be distributed between the multi-aperture imaging system 1605 and the data cube analysis server 1615 in a variety of ways, depending on the processing capabilities of the multi-aperture imaging system 1605 .

[0144] The network 1610 may include any suitable network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. The user device 1620 may include any network-equipped computing device, such as a desktop computer, a laptop computer, a smart phone, a tablet computer, an e-reader, a game console, etc. For example, the results (e.g., classified images) determined by the multi-aperture imaging system 1605 and the data cube analysis server 1615 may be sent to a user device specified by a patient, a doctor, a hospital information system storing a patient's electronic medical record, and / or a centralized health database (e.g., a database of the Centers for Disease Control) in an organizational classification scenario.

[0145] Exemplary Implementation Results

[0146] Background: Morbidity and mortality from burns are a major problem for injured warfighters and their caregivers. The incidence of burns among combat casualties has historically ranged from 5-20%, with approximately 20% of these casualties requiring complex burn surgery at a U.S. Army Institute of Surgical Research (ISR) Burn Center or equivalent. Burn surgery requires specialized training and is therefore provided by ISR personnel rather than personnel in U.S. military hospitals. The limited number of burn specialists complicates the logistics of providing care to soldiers with burns. Therefore, new targeted methods for preoperative and intraoperative detection of burn depth could enable a larger number of medical personnel, including non-ISR personnel, to be recruited to care for patients who sustain burn trauma in combat. This expanded pool of care providers could then be leveraged to provide more complex burn care to further address the care role of warfighters with burn trauma.

[0147] To begin to address this need, a novel cart-based imaging device has been developed that uses multispectral imaging (MSI) and artificial intelligence (AI) algorithms to assist in the preoperative determination of burn healing potential. The device acquires images from a wide tissue area (e.g., 5.9 x 7.9 square inches) in a short time (e.g., within 6, 5, 4, 3, 2, or 1 second) and does not require injection of contrast media. This civilian population-based study demonstrated that the device's accuracy in determining burn healing potential exceeded the clinical judgment of burn specialists (e.g., 70-80%).

[0148] Methods: Civilian subjects of various burn severities were imaged within 72 hours of burn injury and then at subsequent time points up to 7 days post-burn. A 3-week healing assessment or punch biopsy was used to determine the true burn severity for each image. The accuracy of the device was analyzed on a per-image pixel basis to identify and differentiate between healing and non-healing burn tissue in first-, second-, and third-degree burn injuries.

[0149] Results: Data from 38 civilian subjects with 58 burns and 393 images were collected. The AI ​​algorithm achieved 87.5% sensitivity and 90.7% specificity in predicting non-healing burn tissue.

[0150] Conclusions: The device and its AI algorithm demonstrated accuracy in determining the healing potential of burn wounds exceeding the accuracy of clinical judgment by burn specialists. Future work will focus on redesigning the device for portability and evaluating its use in the intraoperative setting. Design changes for portability included reducing the size of the device to a portable system, increasing the field of view, reducing the time to acquire a single snapshot, and evaluating the device for use in the intraoperative setting using a porcine model. These advances were implemented in a benchtop MSI subsystem that demonstrated equivalence on basic imaging tests.

[0151] Additional illuminants for image registration

[0152] In various embodiments, one or more additional illuminants may be used in conjunction with any of the embodiments disclosed herein to improve the accuracy of image registration. Fig.21 An exemplary embodiment of a multi-aperture spectral imager 2100 including a projector 2105 is shown. In some embodiments, the projector 2105 or other suitable light source can be, for example, the one described above with reference to Fig.12One of the illuminants 1165 described. In embodiments including additional illuminants (e.g., projectors 2105 for registration), the method may further include additional exposure. Additional illuminants such as projectors 2105 may project one or more points, stripes, grids, random spots, or any other suitable spatial patterns in a spectral band, multiple spectral bands, or broadband bands that are individually or cumulatively visible in all cameras of the imager 2100 into the field of view of the imager 2100. For example, the projector 2105 may project light, broadband illumination, or cumulative visible illumination of a shared or common channel, which may be used to confirm the accuracy of the registration of the image calculated based on the above-mentioned common band method. As used herein, "cumulative visible illumination" refers to a plurality of wavelengths selected so that the pattern is converted by each image sensor in the multispectral imaging system. For example, the cumulative visible illumination may include a plurality of wavelengths so that even if any of the plurality of wavelengths is not common to all channels, each channel may convert at least one of the plurality of wavelengths. In some embodiments, the type of pattern projected by the projector 2105 may be selected based on the number of apertures in which the pattern is imaged. For example, if the pattern will be viewed through only one aperture, the pattern may preferably be relatively dense (e.g., may have a relatively narrow autocorrelation, such as on the order of 1-10 pixels, 20 pixels, less than 50 pixels, or less than 100 pixels, etc.), while where multiple apertures are used to image the pattern, a pattern with less dense or less narrow autocorrelations may be useful. In some embodiments, additional exposures captured with the projected spatial pattern are included in the calculation of parallax to improve the accuracy of registration compared to embodiments where the exposures are not captured using the projected spatial pattern. In some embodiments, additional illuminants project fringes into the field of view of the imager in one spectral band, multiple spectral bands, or a broadband illumination that can be used to improve the registration of the images based on the phase of the fringes, as compared to embodiments where the exposures are not captured using the projected spatial pattern. In some embodiments, additional illuminators project multiple unique spatial arrangements of dots, grids, and / or spots in one spectral band, multiple spectral bands, or broadband (such as in a shared or common channel) visible to all cameras individually or cumulatively, or broadband illumination that can be used to improve the registration of the images into the field of view of the imager. In some embodiments, the method also includes an additional sensor with a single aperture or multiple apertures that can detect the shape of one or more objects in the field of view. For example, the sensor can use LIDAR, light field, or ultrasound technology to further improve the accuracy of the registration of the images using the above-mentioned common band method. Such additional sensors can be single aperture sensors or multi-aperture sensors that are sensitive to light field information, and can also be sensitive to other signals (e.g., ultrasound or pulsed lasers).

[0153] the term

[0154] All methods and tasks described herein can be performed by a computer system and fully automated. In some cases, a computer system may include a plurality of different computers or computing devices (e.g., physical servers, workstations, storage arrays, computing resources, etc.), which communicate and interconnect operations through a network to perform the functions described. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-temporary computer-readable storage medium or device (e.g., solid-state storage device, disk drive, etc.). Various functions disclosed herein may be embodied in such program instructions, or may be implemented in a dedicated circuit (e.g., ASIC or FPGA) of a computer system. In the case where a computer system includes a plurality of computing devices, these devices may, but need not, be co-located. The results of the disclosed methods and tasks may be permanently stored by converting physical storage devices such as solid-state storage chips or disks into different states. In some embodiments, a computer system may be a cloud-based computing system whose processing resources are shared by a plurality of different business entities or other users.

[0155] The disclosed process can be started on demand in response to an event when initiated by a user or system administrator, such as according to a predetermined or dynamically determined schedule, or in response to some other event. When the process is started, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard disk drives, flash memory, removable media, etc.) can be loaded into a memory (e.g., RAM) of a server or other computing device. The executable instructions can then be executed by a hardware-based computer processor of the computing device. In some embodiments, the process or parts thereof can be implemented serially or in parallel on multiple computing devices and / or multiple processors.

[0156] Depending on the implementation, certain actions, events, or functions of any process or algorithm described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described operations or events are necessary for the practice of the algorithm). In addition, in certain embodiments, operations or events may be performed simultaneously, for example, through multithreading, interrupt handling, or multiple processors or processor cores or on other parallel architectures, rather than sequentially.

[0157] The various illustrative logic blocks, modules, routines, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware (e.g., ASIC or FPGA devices), computer software running on computer hardware, or a combination of the two. In addition, the various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein may be implemented or executed by machines such as processor devices, digital signal processors ("DSP"), application specific integrated circuits ("ASIC"), field programmable gate arrays ("FPGA"), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processor device may be a microprocessor, but alternatively, the processor device may be a controller, a microcontroller, or a state machine, a combination thereof, or the like. The processor device may include a circuit configured to process computer executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logical operations without processing computer executable instructions. The processor device may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Although this article is primarily described with respect to digital technology, the processor device may also include primarily analog components. For example, some or all of the rendering techniques described herein may be implemented in analog circuitry or mixed analog and digital circuitry. The computing environment may include any type of computer system, including but not limited to computer systems based on microprocessors, mainframes, digital signal processors, portable computing devices, device controllers, or computing engines within devices, to name a few.

[0158] The elements of the methods, processes, routines or algorithms described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in software modules executed by a processor device, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of non-temporary computer-readable storage medium. An exemplary storage medium may be connected to a processor device so that the processor device can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium may be integrated with the processor device. The processor device and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor device and the storage medium may reside in a user terminal as discrete components.

[0159] Unless otherwise specifically stated or understood in the context of use, conditional language used herein, such as "may," "might," "might," and the like, is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements, or steps. Thus, such conditional language is generally not intended to imply that a feature, element, or step is in any way necessary for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether such features, elements, or steps are included or will be performed in any particular embodiment with or without other input or prompting. The terms "include," "comprises," "has," and the like are synonymous, inclusive in an open-ended manner, and do not exclude other elements, features, actions, operations, and the like. Similarly, the term "or" is used in its inclusive sense (and not in its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list.

[0160] Unless specifically stated otherwise or otherwise understood in the context of use to be present an item, term, etc., disjunctive language such as the phrase "at least one of X, Y, or Z" can be X, Y, or Z, or any combination thereof (e.g., X, Y, or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0161] Although the above detailed description has shown, illustrated and pointed out the novel features applied to various embodiments, it is understood that various omissions, substitutions and changes in the form and details of the devices or algorithms shown may be made without departing from the scope of the present disclosure. It is recognized that certain embodiments described herein may be implemented in a form that does not provide all of the features and advantages set forth herein, as some features may be used or practiced separately from other features. All changes that fall within the meaning and range of equivalents of the claims are intended to be included within their scope.

Claims

1. A multispectral imaging system, comprising: at least one image sensor; a first aperture positioned to allow incident light to pass to a first sensor region of the at least one image sensor; a second aperture positioned to allow incident light to pass to a second sensor region of the at least one image sensor; a first multi-bandpass filter positioned over the first aperture, wherein the first multi-bandpass filter is configured to allow light in at least the common wavelength band and the first unique wavelength band to pass therethrough; a second multi-bandpass filter positioned over the second aperture, wherein the second multi-bandpass filter is configured to allow light in at least the common wavelength band and the second unique wavelength band to pass therethrough; a memory storing instructions for generating a multispectral data cube; and at least one processor configured by the instructions to at least: receiving signals from the first and second sensor regions; and processing the signal to generate the multispectral data cube, wherein the multispectral data cube includes a first spectral channel corresponding to the common wavelength band, a second spectral channel corresponding to a first unique wavelength band, and a third spectral channel corresponding to a second unique wavelength band; Wherein processing the signal at least comprises: estimating a disparity between first image data captured by a first sensor region and second image data captured by a second sensor region using information from a first spectral channel corresponding to the common wavelength band, wherein the disparity is estimated using an epipolar constraint; performing at least spectral unmixing to generate image data for each of the first, second, and third spectral channels; and aligning the image data of the first, second and third spectral channels using the parallax to generate the multispectral data cube, Wherein at least the first multi-bandpass filter is a curved filter and has a curvature selected to reduce an angular dependence of spectral transmission over at least the first sensor area resulting in a shift of the filter's bandpass toward shorter wavelengths with increasing angle of incidence.

2. The multispectral imaging system according to claim 1, wherein the first multi-bandpass filter is located in front of the lens assembly of the multispectral imaging system, in front of the single lens of the multispectral imaging system, behind the lens assembly of the multispectral imaging system, behind the single lens of the multispectral imaging system, inside the lens assembly of the multispectral imaging system, inside the cemented lens group of the multispectral imaging system, directly on the surface of the single lens of the multispectral imaging system, or directly on the surface of an element of the lens assembly of the multispectral imaging system.

3. The multispectral imaging system according to any one of claims 1 to 2, further comprising: a color filter array covering the first and second sensor areas, the color filter array comprising a repeating pattern of color filters; wherein the first color filter of the repeated pattern is configured to pass light having a wavelength having a first weight distribution including the common wavelength band; wherein the repeating pattern of the second color filter is configured to pass light having a wavelength having a second weight distribution including the first unique wavelength band; and The repeating pattern of the third color filter is configured to pass light having a wavelength having a third weight distribution including a second unique wavelength band.

4. The multispectral image system according to any one of claims 1 to 3, wherein the common wavelength band includes at least one of violet, violet-blue, blue and cyan light.

5. The multispectral imaging system according to any one of claims 1 to 3, wherein the common wavelength band includes light with a wavelength shorter than 450 nm and longer than 380 nm.

6. The multispectral imaging system according to any one of claims 1 to 3, wherein the common wavelength band includes green light.

7. The multispectral imaging system according to any one of claims 1 to 3, wherein the common wavelength band includes red light.

8. The multispectral image system according to any one of claims 4 to 7, wherein the first and second multi-bandpass filters are each configured to pass a second common wavelength band including at least one of green, cyan, and blue light.

9. The multi-spectral image system according to any one of claims 4 to 7, wherein the first and second multi-bandpass filters are each configured to pass a second common wavelength band including red light.

10. The multispectral imaging system of claim 3, wherein the first and second multi-bandpass filters are each configured to pass a second common wavelength band that is not filtered by at least one color filter patterned in the color filter array, and the at least one color filter also does not filter the common wavelength band.

11. The multispectral imaging system of any one of claims 8 and 9, wherein the at least one processor is configured by the instructions to at least perform spectral unmixing to isolate portions of the captured image data corresponding to the common band and a second common band.

12. The multispectral imaging system of any one of claims 1-11, wherein the at least one processor is configured by the instructions to use the multispectral data cube to at least analyze the imaged tissue.

13. The multispectral imaging system according to any one of claims 1 to 12, further comprising: a third aperture positioned to allow incident light to pass to a third sensor region of the at least one image sensor; a fourth aperture positioned to allow incident light to pass to a fourth sensor region of the at least one image sensor; a third multi-bandpass filter positioned over the third aperture, wherein the third multi-bandpass filter is configured to allow passage of light in at least the common wavelength band and a third unique wavelength band; and A fourth multi-bandpass filter is positioned over the fourth aperture, wherein the fourth multi-bandpass filter is configured to allow light in at least the common wavelength band and a fourth unique wavelength band to pass therethrough.

14. The multispectral imaging system according to claim 13, further comprising: a fifth aperture positioned to allow incident light to pass to a fifth sensor region of the at least one image sensor; and A fifth multi-bandpass filter is positioned above the fifth aperture, wherein the fifth multi-bandpass filter is configured to allow light in at least one wavelength band passed by at least one of the first multi-bandpass filter, the second multi-bandpass filter, the third multi-bandpass filter, and the fourth multi-bandpass filter to pass through.

15. The multispectral imaging system according to any one of claims 1 to 13, wherein the multispectral imaging system comprises: up to a total of 25 apertures positioned to allow incident light to pass to a total of up to 25 sensor areas of the at least one image sensor; and up to 25 multi-bandpass filters, each multi-bandpass filter positioned over one of the 25 total apertures, wherein each multi-bandpass filter is configured to allow passage of light in at least the common wavelength band and in a unique wavelength band that no other multi-bandpass filter is configured to allow passage; The total number of bands allowed to pass is more than 4 and less than 51.

16. The multispectral imaging system according to any one of claims 1 to 12, further comprising: a third aperture positioned to allow incident light to pass to a third sensor region of the at least one image sensor; and A third multi-bandpass filter is positioned over the third aperture, wherein the third multi-bandpass filter is configured to allow passage of light in at least one wavelength band passed by at least one of the first multi-bandpass filter and the second multi-bandpass filter.

17. The multispectral imaging system of any one of claims 13-14, wherein the first, second, third and fourth multi-bandpass filters collectively pass 7, 8, 9 or 10 different bands including the common band and the first, second, third and fourth unique bands.

18. The multispectral imaging system of claim 17, wherein the first, second, third, and fourth multi-bandpass filters are each configured to pass the common wavelength band including blue light, an additional common wavelength band including green light, and two additional wavelength bands.

19. The multispectral image system according to any one of claims 17 and 18, wherein the at least one processor is configured by the instructions to at least generate the multispectral data cube to include 10 spectral channels corresponding to the 7, 8, 9 or 10 different bands.

20. The multispectral imaging system of claim 19, further comprising a lighting panel having a plurality of LEDs configured to emit light in 7, 8, 9 or 10 different wavelength bands, wherein each LED emits light in a specific wavelength band of the 7, 8, 9 or 10 different wavelength bands.

21. The multispectral imaging system of any one of claims 17-20, wherein the at least one image sensor comprises a color filter array located above a photodiode of the at least one image sensor, and wherein the color filter array has at least two different color filters.

22. The multispectral imaging system of claim 21, wherein the color filter array has more than 2 different color filters and no more than 25 different color filters.

23. A multispectral imaging system according to any one of claims 21 and 22, wherein the 7, 8, 9 or 10 different wavelength bands comprise a narrower range of light than is passed by 3 different color filters.

24. The multispectral image system according to any one of claims 1-23, further comprising at least one light source configured to emit light in the common wavelength band, the first unique wavelength band, and the second unique wavelength band.

25. The multi-spectral imaging system of claim 24, further comprising a diffusing optical element positioned above the at least one light source.

26. A multispectral imaging system according to any one of claims 1-25, wherein the multispectral imaging system is configured to acquire images from a tissue area as wide as 5.9×7.9 inches in less than 6 seconds, and wherein the multispectral imaging system is configured to provide tissue analysis information in the absence of a contrast agent.

27. The multispectral imaging system according to claim 26, wherein: The tissue analysis information includes identifying multiple burn states, wound states, healing potential, clinical characteristics including cancerous or non-cancerous status of the imaged tissue, wound depth, debridement margins, or the presence of diabetic, non-diabetic, or chronic ulcers.

28. A method for generating a multispectral data cube using the multispectral image system according to any one of claims 1 to 27, comprising: capturing a first exposure comprising first, second, and third spectral channels; capturing a second exposure comprising the first, second, and third spectral channels; and The multispectral data cube is generated based at least in part on reducing a signal-to-noise ratio of the first exposure using the second exposure.

29. The method of claim 28, further comprising activating a group of light emitters that emit light in common at the common wavelength band, the first unique wavelength band, and the second unique wavelength band during the first and second exposures.

30. The method of any one of claims 28 and 29, wherein the first exposure and the second exposure each include at least a fourth spectral channel and no more than a total of 10 spectral channels.

31. The method according to any one of claims 28 to 30, further comprising: capturing a third exposure using ambient light; and The third exposure is subtracted from the first exposure and the second exposure.

32. The method of claim 31 , further comprising generating the multispectral data cube based on a result of subtracting a third exposure from the first and second exposures.

33. The method of any of claims 28-32, further comprising registering images captured by the first and second sensor regions with each other based on information in the first spectral channel.

34. The method of claim 33, further comprising generating the multispectral data cube based on a result of registering images captured by first and second sensor regions with each other.

35. The method of any one of claims 33-34, wherein the image represents information captured by the first and second sensor areas in a first spectral channel, the method further comprising: identifying a disparity between images captured by first and second sensor regions in a first spectral channel corresponding to the common wavelength band; registering a first additional image to the image using the disparity, the first additional image representing information captured by the first sensor area in the second spectral channel; and A second additional image is registered to the image using the disparity, the second additional image representing information captured by the second sensor area in a third spectral channel.

36. A method according to claim 35, wherein each exposure includes more than three spectral channels, the method further comprising using the parallax to register at least a third additional image to the image, the third additional image representing information captured by a third sensor area in a fourth spectral channel, wherein the total number of additional images registered to the image is one less than the total number of spectral channels.

37. The method of any one of claims 33-36, wherein the multispectral imaging system further comprises a light source configured to project one or more dots, stripes, grids, random spots or other spatial patterns, the method further comprising: projecting light comprising one or more dots, stripes, grids, random spots, or other spatial patterns into the field of view of at least the first aperture through the illuminator; and capturing additional exposures using light projected by the light emitter; Wherein the registering is based at least in part on the additional exposure.

38. The method of claim 37, wherein the light projected by the light emitter includes at least one of light having wavelengths in the common wavelength band, broadband illumination, or cumulative visible illumination.

39. The method of claim 37, wherein the additional exposures are used to confirm the accuracy of the registration of the images.

40. The method of claim 37, wherein the additional exposures are included in the calculation of the disparity to improve the accuracy of the registration.

41. The method of claim 37, wherein the illuminant projects a uniquely configured plurality of dots, stripes, grids, spots, or other spatial patterns in at least the common wavelength band, broadband illumination, or cumulative visible illumination.

42. A method according to any one of claims 33-41, wherein the multispectral imaging system further comprises an additional sensor configured to detect a shape of an object in a field of view of the at least one image sensor, and wherein the registration is based at least in part on the shape detected by the additional sensor.

43. The method of any one of claims 37-41, wherein the light emitter projects the light into fields of view of multiple apertures of the multi-spectral imaging system.

44. The method of any one of claims 28-43, further comprising performing spectral unmixing to isolate the first, second, and third spectral channels.

45. The method of claim 37, wherein each exposure comprises more than three spectral channels, the method further comprising performing spectral unmixing to isolate all spectral channels.

46. ​​The method of any of claims 35-45, further comprising using the disparity to determine at least one of terrain information and depth information.

47. The method of claim 46, wherein the disparity is used to determine depth information for each pixel.

48. The method of claim 47, further comprising using the depth information to determine the volume of the imaged wound.

49. The method according to any one of claims 44 and 45, further comprising: capturing an additional exposure including at least a fourth spectral channel representing non-visible light; and The multispectral data cube is generated based at least in part on the additional exposures including the fourth spectral channel.

50. The method of claim 49, further comprising activating a third group of light emitters that emit the invisible light during the additional exposure.

51. A method for automated tissue analysis, comprising: Using the multispectral imaging system according to any one of claims 1 to 27 to generate a multispectral data cube representing the imaged tissue; supplying the multispectral data cube as input to a machine learning system trained to predict at least one clinical characteristic of a tissue; and At least one clinical characteristic of the imaged tissue is predicted based on the output of the machine learning system.

52. The method of claim 51, wherein: The at least one clinical feature comprises trauma or burns.

53. The method of claim 51 , further comprising generating a multispectral data cube using the method of any one of claims 28-50.

54. A method according to claim 51, wherein the at least one clinical feature includes multiple trauma states or burn states, and the method further includes using the output of the machine learning system to generate a classification image of the imaged tissue, the classification image showing areas of the imaged tissue classified into different states among the multiple trauma states or burn states.

55. The method of claim 54, wherein the plurality of burn states include at least first degree burn, second degree burn, third degree burn, or healthy skin categories, or wherein the plurality of trauma states include at least hemostasis, inflammation, hyperplasia, remodeling, or healthy skin categories, the method further comprising classifying each of the plurality of pixels showing the imaged tissue as one of first degree burn, second degree burn, third degree burn, and healthy skin categories, or the method further comprising classifying each of the plurality of pixels showing the imaged tissue as one of hemostasis, inflammation, hyperplasia, remodeling, and healthy skin categories.

56. A method according to claim 55, wherein the at least one clinical characteristic includes healing potential, the method further comprising using the output of the machine learning system to generate an image of the imaged tissue, the image visually illustrating the determined healing potential in different regions of the imaged tissue.

57. The method of any one of claims 51-56, wherein the at least one clinical feature comprises a cancerous or non-cancerous state of the imaged tissue, the method further comprising classifying each of a plurality of pixels showing the imaged tissue as a cancerous or non-cancerous state.

58. The method of any one of claims 51-57, wherein the at least one clinical characteristic comprises lesion depth or volume, the method further comprising outputting information identifying the lesion depth or volume in the imaged tissue.

59. The method of any one of claims 51-58, wherein the at least one clinical feature comprises a debridement boundary, the method further comprising outputting information identifying the debridement boundary within or around a lesion in the imaged tissue.

60. The method of any one of claims 51-59, wherein the at least one clinical feature comprises diabetes, non-diabetes, or chronic ulcers.

61. The method of claim 60, wherein the at least one clinical feature comprises a diabetic foot ulcer.

62. The method of any one of claims 43-61, wherein the multispectral imaging system acquires images from a 5.9 x 7.9 inch wide tissue region in less than 6 seconds, and wherein the multispectral imaging system outputs tissue analysis information in the absence of a contrast agent.

63. The method of claim 62, wherein: The tissue analysis information includes identifying multiple burn states, wound states, healing potential, clinical characteristics including cancerous or non-cancerous status of the imaged tissue, wound depth, wound volume, debridement margins, or the presence of diabetic, non-diabetic or chronic ulcers.

64. A method for automated biometric identification, comprising: Using the multispectral imaging system according to any one of claims 1 to 27 to generate a multispectral data cube representing a biometric feature of a user; feeding the multispectral data cube as input to a machine learning system trained to authenticate specific biometric features; and A user is authenticated or rejected based on the output of the machine learning system.

65. The method of claim 64, further comprising generating a multispectral data cube using the method of any one of claims 28-50.

66. A method for automated material analysis, comprising: Using the multispectral imaging system according to any one of claims 1 to 27 to generate a multispectral data cube representing the imaged material; supplying the multispectral data cube as input to a machine learning system, the machine learning system being trained to determine one or more material properties; and One or more material properties of the imaged material are determined based on the output of the machine learning system.

67. The method of claim 66, further comprising generating a multispectral data cube using the method of any one of claims 28-50.

68. A method for processing image information, comprising: capturing the image information using a multi-aperture handheld imaging device including at least one image sensor region having a color filter array, wherein the image information includes a greater number of channels than the number of filter colors in the color filter array; and processing the image information into a multispectral data cube comprising a plurality of channels, wherein each of the plurality of channels represents light in a different wavelength band below 40 nm but not zero, and wherein processing the image information comprises using epipolar constraints to estimate disparity between images captured via different apertures of the multi-aperture handheld imaging device, wherein at least spectral unmixing is performed to generate image data for each of the plurality of channels, Wherein the multi-aperture handheld imaging device comprises a multispectral imaging system according to any one of claims 1-27.

69. The method of claim 68, further comprising generating a multispectral data cube using the method of any one of claims 28-50.

70. A method for processing a multispectral data cube, comprising: Using the multispectral image system according to any one of claims 1-27 to generate a multispectral data cube; and The multispectral data cube is processed using at least one processor of the multispectral imaging system to determine at least one characteristic of an object represented by the multispectral data cube.

71. The method of claim 70, further comprising generating a multispectral data cube using the method of any one of claims 28-50.

72. A method for processing a multispectral data cube, comprising: Using the multispectral image system according to any one of claims 1-27 to generate a multispectral data cube; transmitting the multispectral data cube to at least one remote processor via a network; and The multispectral data cube is processed using the at least one remote processor to determine at least one characteristic of an object represented by the multispectral data cube.

73. The method of claim 72, further comprising generating a multispectral data cube using the method of any one of claims 28-50.

Images