A snapshot-type high-precision multispectral imaging system and method utilizing multiplexed illumination.

Abstract

To solve the problem of the cumbersome parallax correction between multiple apertures in a multi-aperture spectral imaging system. [Solution] The present invention generally includes a snapshot-type single-aperture multispectral imaging apparatus comprising at least a color camera, a custom-designed 8-band optical filter, a motorized zoom lens, and an 8-wavelength multicolor LED illumination system. Furthermore, the present invention includes methods relating to the design of time-series LED illumination and combinations of LED illumination, as well as unmixing algorithms, for obtaining eight multispectral imaging images. The apparatus and methods of the present invention make it possible to achieve faster multispectral image acquisition speeds (less than 100 milliseconds), faster post-processing speeds without parallax correction calculations by employing a single-aperture system, an adjustable field of view, reduced form factor, reduced costs, and higher accuracy of spectral information.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims the interests of U.S. Provisional Patent Application No. 63 / 168,151, filed on 30 March 2021, entitled “Snapshot-type high-precision multispectral imaging system and method using multiplexed illumination,” which is expressly incorporated herein by reference in its entirety for any purpose.

[0002] Statement on research and development funded by the federal government Some of the inventions described herein were made with U.S. government support granted by the Biomedical Advanced Research and Development Authority (BARDA) within the Assistant Secretary for Preparedness and Response, U.S. Department of Health and Human Services, under contract number HHSO100201300022C. Other parts of the inventions described herein were made with U.S. government support granted by the Defense Health Agency (DHA) under contract numbers W81XWH-17-C-0170 and / or W81XWH-18-C-0114. The U.S. government may retain certain rights with respect to these inventions.

[0003] The systems and methods disclosed herein relate to spectral imaging, and more specifically, to multispectral imaging systems and methods utilizing multiplexed illumination. [Background technology]

[0004] The electromagnetic spectrum is the band of wavelengths or frequencies of electromagnetic radiation (e.g., light). The electromagnetic spectrum includes, in order from longest to shortest wavelength, radio waves, microwaves, infrared light (IR), visible light (i.e., light detectable by the structure of the human eye), ultraviolet light (UV), X-rays, and gamma rays. Spectral imaging refers to a branch of spectroscopy and photography that collects partial or complete spectral information at various locations in the image plane. Multispectral imaging systems can capture multiple spectral bands (usually a dozen or fewer, each in a different spectral region) and collect measurements in those spectral bands for each pixel, with a bandwidth of tens of nanometers per spectral channel.

[0005] The spectral bands may be separated by optical filters and / or a multi-channel image sensor (e.g., a color camera). While single-aperture multispectral imaging systems using filter wheels offer high spectral accuracy, they suffer from low temporal resolution (tens of seconds) due to the slow mechanical rotation of the wheel during filter switching. Multi-aperture spectral imaging systems overcome this limitation by employing multi-color cameras with multi-bandpass optical filters, and can obtain multispectral imaging (MSI) images with a certain degree of temporal resolution (less than 100 milliseconds, so-called snapshots) using unmixing algorithms. However, such multi-aperture spectral imaging systems still suffer from the problem of cumbersome parallax correction between multiple apertures. [Overview of the Initiative] [Means for solving the problem]

[0006] The multispectral imaging systems and techniques disclosed herein have several features, but none of these features alone can achieve the desired characteristics. The following description briefly outlines some of the specific features of spectral imaging disclosed herein, but this description is not intended to limit the scope of the invention as defined in the claims below. Those skilled in the art will be able to fully understand the advantages of the spectral imaging features disclosed herein compared to conventional systems and methods.

[0007] In a first aspect of the technology of the present invention, a multispectral imaging system is A light source configured to selectively emit light containing one or more frequency bands from a set of four or more predetermined frequency bands to illuminate an object; An image sensor configured to receive the portion of the emitted light that has been reflected by the object; An aperture arranged to allow the portion of the emitted light reflected by the object to pass through and be incident on the image sensor; A multibandpass filter, positioned on the aperture and configured to allow light of four or more predetermined frequency bands to pass through; Memory containing instructions for generating a multispectral image and performing unmixing; and at least one processor Includes. The at least one processor, in accordance with the instruction, The light source emits a first subset of light consisting of two or more frequency bands from the predetermined frequency bands; First image data generated based on the reflected light of a first subset of light is received from the image sensor; The light source emits a second subset of light consisting of two or more frequency bands from the predetermined frequency bands; A second image data generated based on the reflected light of a second subset of light is received from the image sensor; Process the first image data and the second image data to generate at least a first multispectral image and a second multispectral image; Perform spectral unmixing to generate multiple images of the object, each corresponding to a single frequency band among the four or more predetermined frequency bands. It is structured in this way.

[0008] In some embodiments, the processor is further configured to cause the light source to emit a third subset or more of light consisting of two or more frequency bands from the predetermined frequency bands, and to receive a third image data or more of image data generated based on the reflected light of the third subset or more of light from the image sensor.

[0009] In some embodiments, the light source includes a plurality of light-emitting diodes (LEDs), each LED configured to emit light in one of four or more predetermined frequency bands. In some embodiments, the at least one processor is further configured to select the frequency bands to be emitted simultaneously by the light source by controlling the activation of the individual LEDs of the light source. In some embodiments, the processor controls the activation of the individual LEDs by controlling an electronic switching shutter that selectively passes through and blocks the light emitted by each LED.

[0010] In some embodiments, the multibandpass filter comprises multiple frequency bands through which light passes, each frequency band corresponding to one of the four or more predetermined frequency bands. In some embodiments, the LEDs and the multibandpass filter comprise the same number of frequency bands, and each frequency band of the LEDs is aligned to coincide with the corresponding frequency band of the multibandpass filter. In some embodiments, the multispectral imaging system further includes bandpass filters placed on each of the multiple LEDs to confine the light emitted by each LED to a precise spectrum. In some embodiments, the set of four or more predetermined frequency bands comprises eight predetermined frequency bands, and the light source comprises eight LEDs, each LED configured to emit light in one of the predetermined frequency bands. In some embodiments, the multibandpass filter comprises an eight-band filter configured to pass each of the eight predetermined frequency bands. In some embodiments, the at least one processor is further configured to cause the light source to emit a third subset of light in the predetermined frequency band according to the instructions, and to receive third image data generated based on the reflected light of the third subset of light from the image sensor. In some embodiments, each of the first subset, second subset, and third subset includes three frequency bands from the predetermined frequency band. In some embodiments, the predetermined frequency band is defined by central wavelengths ranging from ultraviolet (UV) to short-wavelength infrared. In some embodiments, the central wavelengths of the predetermined frequency band are 420 nm, 525 nm, 581 nm, 620 nm, 660 nm, 726 nm, 820 nm, and 855 nm, respectively.

[0011] In some embodiments, the multispectral image system further includes an electric parfocal zoom lens or an electric variable focal length zoom lens disposed on the aperture, and the one or more processors are further configured to control the electric zoom lens to adjust the field of view (FOV) of the multispectral image system. In some embodiments, the parfocal zoom lens or the variable focal length zoom lens is configured to be able to automatically or manually adjust the focus by changing the focal length (FL) of the lens. In some embodiments, even when the field of view of the multispectral image system is adjusted, the shooting distance between the object and the image sensor does not change. In some embodiments, the one or more processors are further configured to adjust the field of view of the multispectral image system by changing the shooting distance between the object and the image sensor. In some embodiments, the processor is further configured to compensate for the reduction in contrast caused by the zoom lens. In some embodiments, the field of view can be adjusted in a range of at least 8 cm to 23 cm, for example, 8 cm, 9 cm, 10 cm, 11 cm, 12 cm, 13 cm, 14 cm, 15 cm, 16 cm, 17 cm, 18 cm, 19 cm, 20 cm, 21 cm, 22 cm or 23 cm, or a length within a range having any two of these lengths as the upper and lower limits. In some embodiments, the multispectral image system maintains a high spatial resolution without reducing the number of samples in the field of view, different from the digital trimming method by post-processing at the expense of resolution.

[0012] In some embodiments, the one or more processors are further configured to visualize the object in a natural pseudo-color based on the multispectral image.

[0013] In some embodiments, the one or more processors are configured to perform spectral unmixing by determining the reflection coefficient of the object using a matrix expression. In some embodiments, the reflection coefficient is determined based at least in part on the value of the incident illuminance, the value of the transmission coefficient of the multi-band pass filter, and the value of the quantum coefficient of the image sensor for each channel. In some embodiments, the multispectral image system can capture three or more multispectral images in less than 100 milliseconds.

[0014] In some embodiments, the object is a tissue region. In some embodiments, the tissue includes a wound. In some embodiments, the wound includes a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a postoperative wound, a cut site, a burn, a cancerous lesion, or damaged tissue. In some embodiments, the multispectral image system is for use in identifying tissue classification or a tissue healing score, where the tissue classification is, for example, living tissue or healthy tissue, dead tissue or necrotic tissue, perfused tissue or non-perfused tissue, or ischemic tissue or non-ischemic tissue, and the tissue healing score is, for example, a tendency for at least 50% of the wound to heal after 30 days of standard wound care treatment, or a tendency for at least 50% of the wound not to heal after 30 days of standard wound care treatment. In some embodiments, the multispectral image system is for use in imaging a wound, cancer, ulcer, or burn, where the wound, cancer, ulcer, or burn is, for example, a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a postoperative wound, a cut site, a burn, a cancerous lesion, or damaged tissue.

Brief Description of the Drawings

[0015] [Figure 1A] An example of light incident on the filter at various principal ray angles of incidence is shown.

[0016] [Figure 1B]This graph shows an example of the transmission efficiency obtained from the filter in Figure 1A, which transmits light at various principal ray incidence angles.

[0017] [Figure 2A] An example of a multispectral image data cube is shown.

[0018] [Figure 2B] This example illustrates how a specific multispectral imaging technique generates the data cube shown in Figure 2A.

[0019] [Figure 2C] An example of a snapshot imaging system capable of generating the data cube shown in Figure 2A is presented.

[0020] [Figure 3A] This disclosure shows a schematic cross-sectional diagram illustrating the optical design of an example of a multi-aperture imaging system equipped with a curved multi-bandpass filter.

[0021] [Figure 3B-3D] Figure 3A shows an example of the optical design of an optical component that constitutes one of the optical paths in the multi-aperture imaging system.

[0022] [Figure 4] A schematic diagram of an example of a single-aperture multispectral imaging system using the technology of the present invention, and a plot showing the performance of each component of this multispectral imaging system are shown.

[0023] [Figure 5] An example of the time series of LED illumination when acquiring a multispectral image using the technology of the present invention, and the corresponding detection coefficients for each channel are shown.

[0024] [Figure 6-7]An example of an embodiment of a single-aperture multispectral imaging system based on the technology of the present invention is shown.

[0025] [Figure 8-10] The results of a test using a single aperture system based on the technology of the present invention are shown. [Modes for carrying out the invention]

[0026] As outlined herein, this disclosure relates to two-dimensional multispectral imaging (MSI) using a single-aperture, single-lens, single-camera system characterized by a fast image acquisition rate (snapshot, less than 100 milliseconds), high-precision spectrum, and adjustable field of view. Furthermore, this disclosure relates to a technique for implementing spectral unmixing to improve the image acquisition rate from such a camera system. As will be discussed later, the technique of this disclosure overcomes current problems in spectral imaging.

[0027] Multispectral imaging using multiplexed illumination allows for the separation of spectral bands, resulting in fast switching, robustness, and cost-effectiveness. Currently, such solutions employ multiplexed illumination using overlapping spectra and a limited number of channels. This multiplexed illumination allows for the acquisition of spectral reflectance (s(λ)) that changes smoothly and can be well approximated in the imaging situation. However, in biomedical applications, the spectral reflectance (s(λ)) of biological tissues is somewhat complex, and highly accurate spectral information is desired. To overcome this constraint, it is necessary to significantly reduce crosstalk between spectra.

[0028] The snapshot-type high-precision multispectral imaging system and method utilizing multiplexed illumination disclosed herein has several features. In some embodiments, the multispectral imaging system of the present invention employs a single-aperture, single-lens, single-RGB color camera for two-dimensional multispectral sensing. In some embodiments, the multispectral imaging system of the present invention employs a multiband optical filter for multispectral filtering. The multiband optical filter in this design may be, for example, an 8-band filter for 8-band MSI. However, the systems and methods disclosed herein are not limited to the number specified herein and can be equally used for MSI using more or fewer wavelength bands than specified herein. In some embodiments, the multispectral imaging system disclosed herein employs eight wavelengths for illumination using multiple light-emitting diodes (LEDs), where each wavelength spectrum overlaps with each MSI band. In some embodiments, a bandpass optical filter with an MSI band matching each LED is placed in front of each LED. In some embodiments, the multispectral imaging system disclosed herein employs a motorized zoom lens. To accelerate the imaging speed, a spectral unmixing method utilizing adjustment of LED illumination may be used. Multiple wavelengths of multiple LEDs can be turned on or off in specific combinations and time series, which will be discussed in more detail later. The unmixing coefficient may be calculated based on the illuminance of the LEDs (I0), the transmittance coefficient of the filter (T%), and / or the sensing quantum efficiency (Q%) of the RGB channels of the color camera.

[0029] Various embodiments of the multispectral imaging systems and methods of this disclosure can provide multispectral imaging with improved features, including, but not limited to, one or more of the following: (a) Because a single aperture system is employed, there is no need to use complex algorithms for parallax correction, and no parallax errors occur. (b) Compared to multi-aperture systems, it has a smaller form factor (e.g., is smaller and lighter) and lower cost. (c) Due to its fast image acquisition speed (less than 100 milliseconds) and fast post-processing speed, it produces fewer motion artifacts and can be used for real-time imaging. (d) High-precision spectral information can be obtained by confining the LEDs with a narrow-band LED illumination, a bandpass optical filter placed in front of the LEDs, and a multiband optical filter. (e) Because it uses an electric zoom lens, the field of view can be adjusted according to various applications. (f) It is possible to visualize the MSI image using natural pseudo-color. (g) It enables multi-functional biomedical imaging.

[0030] The multispectral imaging system of the present invention achieves a faster image acquisition speed compared to a single-aperture multispectral imaging system using a filter wheel equipped with a mechanically rotating filter wheel, because it uses an 8-band filter for multibandpass optical filtering. Simultaneously, in some embodiments, spectral information for each MSI band (e.g., channel) is acquired by employing multiple multicolor LEDs having spectra that approximate the desired MSI band or spectra that precisely overlap the desired MSI band. By utilizing accurate spectral illumination and spectral confinement by the LED filter and the 8-band filter, light leakage outside the spectral band and crosstalk between channels can be avoided, and spectral accuracy can be significantly improved. Since each LED can be quickly and repeatedly switched on and off, monochromatic illumination can be obtained using a time-divided system.

[0031] In some embodiments, the multispectral imaging system of the present invention can be operated in ambient or indoor lighting environments by acquiring a background image (e.g., without LED lighting), and then the background image can be subtracted from the obtained MSI image (e.g., exposed to indoor lighting and LED lighting). Furthermore, in some embodiments, the MSI acquisition speed may be further improved by employing a multichannel image sensor (e.g., an RGB camera), simultaneously illuminating three channels of LEDs, and separating the spectral information obtained from these three channels using an unmixing algorithm. Thus, the snapshot imaging speed of the technology of this disclosure enables very fast imaging and significantly reduces motion artifacts of the target, offering advantages in various biomedical applications.

[0032] In some embodiments, the multispectral imaging system and method of the present invention can employ a configuration consisting of a single aperture, a single lens, and a single camera, compared to a multi-aperture MSI imaging system. Therefore, parallax correction calculations, which may require complex algorithms that can significantly slow down image post-processing speed, are not required. Furthermore, because a configuration consisting of a single aperture, a single lens, and a single camera is implemented, misinterpretation of spectral information based on pixels can be avoided. For example, in existing multi-aperture MSI imaging systems, spectral bands can be separated using the separation of the image sensor (e.g., a configuration consisting of multiple cameras) and an unmixing algorithm. On the other hand, the MSI system and method of this disclosure can separate spectral bands using multiplexed illumination (e.g., multiple individually controllable multi-color LEDs) and an unmixing algorithm. In this disclosure, the operation of LED illumination can be controlled in a time series using an electronically switching shutter that is much faster than a mechanical rotating wheel equipped with multiple single-band bandpass filters incorporated into a filter wheel system. Therefore, instantaneous band switching and imaging acquisition speed are close to that of a multi-aperture MSI imaging system, parallax correction calculations are not required, and post-processing speed is fast.

[0033] The MSI system and method of this disclosure can be used with a bandpass optical filter for LED illumination and an 8-band optical filter for the detector, thereby enabling good confinement of the detection spectrum to the desired band, avoiding crosstalk between illuminations, eliminating light leakage outside the spectral band, and improving spectral accuracy. This disclosure offers the improved advantages of existing systems, including multiplexed illumination, in a simple design.

[0034] Overview of the spectral imaging system Figure 1A shows an example of a filter 108 positioned along the optical path toward the image sensor 110, and light incident on the filter 108 at various angles of incidence. Rays 102A, 104A, and 106A are shown as lines that pass through the filter 108, are refracted by the lens 112, and then incident on the sensor 110. The lens 112 may be replaced with another image forming optical system, such as (but not limited to) a mirror and / or aperture. In Figure 1A, the light from each ray is considered broadband, for example, light with a spectral composition over a wide wavelength range, and only light of a specific wavelength is selectively transmitted by the filter 108. The three rays 102A, 104A, and 106A are incident on the filter 108 at different angles of incidence. For illustrative purposes, ray 102A is shown as incident substantially perpendicular to filter 108, ray 104A has a larger incident angle than ray 102A, and ray 106A has a larger incident angle than ray 104A. Each ray 102B, 104B, and 106B that passes through the filter exhibits a unique spectrum due to the transmission characteristics of filter 108, which depend on the incident angle, and these unique spectra are detected by sensor 110. This incident angle dependence effect causes a shift in the bandpass characteristics of filter 108, where the wavelength becomes shorter as the incident angle increases. Furthermore, the transmission efficiency of filter 108 may decrease due to the incident angle dependence, and the spectral shape exhibiting the bandpass characteristics of filter 108 may change. Such a combined effect is called incident angle-dependent spectral transmission. Figure 1B shows the spectrum of each ray detected in Figure 1A by a spectrometer assumed to be located at the position of sensor 110, illustrating that the spectrum exhibiting the bandpass characteristics of filter 108 shifts as the angle of incidence increases. Curves 102C, 104C, and 106C show that the center wavelength of the bandpass characteristics is shortening, and therefore, in this example, it is shown that the wavelength of the light emitted from the optical system is shortening. Also, as shown in this figure, the spectral shape and transmittance peak of the bandpass characteristics change with the angle of incidence. For certain consumer applications, image processing can be performed to remove the visible effects of such angle-dependent spectral transmission.However, such post-processing techniques cannot recover precise information about which wavelengths of light actually entered the filter 108. Therefore, the resulting image data may not be suitable for certain high-precision applications.

[0035] Another challenge faced by certain existing spectral imaging systems is the time required to acquire a complete spectral image dataset. This will be discussed in relation to Figures 2A and 2B. A spectral image sensor acquires the spectral irradiance I(x,y,λ) of a specific scene and collects a three-dimensional (3D) dataset, commonly called a data cube. Figure 2A shows an example of a spectral image data cube 120. As shown in this figure, the data cube 120 represents three-dimensional image data, of which two spatial dimensions (x and y) correspond to the two-dimensional (2D) surface of the image sensor, and the spectral dimension (λ) corresponds to a specific wavelength range. The size of the data cube 120 is N x N y N λ It can be calculated as N x and N y This is the number of sample points in each spatial dimension (x,y) direction, and N λ is the number of sample points in the spectral axis λ direction. Because the data cube has a higher dimension than currently available 2D detector arrays (e.g., image sensors), a typical spectral imaging system either takes a time-series 2D slice (i.e., a plane) of the data cube 120 (referred to herein as a “scanning” imaging system), or measures all sample points of the data cube 120 simultaneously by dividing the data cube 120 during the computation process to obtain multiple 2D data that can be reintegrated into the data cube 120 (referred herein as a “snapshot” imaging system).

[0036] Figure 2B shows an example illustrating how specific scanning spectral imaging techniques generate data cubes 120. Specifically, Figure 2B shows portions 132, 134, and 136 of data cube 120 that can be acquired in a single detector integration time. A point-scan spectrometer can, for example, capture a portion 132 that extends across the entire spectral plane λ at a single spatial position (x,y). A point-scan spectrometer can construct data cube 120 by performing multiple integrations for each (x,y) position in the spatial dimension. A filter-wheel imaging system can, for example, capture a portion 134 that extends across the entire spatial dimensions x and y but is located only in a single spectral plane λ. Wavelength scanning imaging systems, such as filter-wheel imaging systems, can construct data cube 120 by performing multiple integrations across the spectral plane λ. A line-scan spectrometer can, for example, capture a portion 136 that extends across the entire spectral dimension λ and the entirety of one spatial dimension (x or y) but is located only at a single point in the other spatial dimension (y or x). A line scan spectrometer can construct a data cube 120 by performing multiple integrations with respect to one of the spatial dimensions (y or x) positions.

[0037] In applications where both the target object and the imaging system are stationary (or relatively stationary during exposure), the aforementioned scanning imaging systems have the advantage of obtaining high-resolution data cubes 120. Line scan imaging systems and wavelength scanning imaging systems may achieve this advantage by using the entire surface area of ​​the image sensor to capture each spectral or spatial image. However, if there is movement in the imaging system and / or the object between the first and second exposures, artifacts may occur in the resulting image data. For example, the same (x,y) position in the data cube 120 may actually represent a different physical location on the imaged object in spectral dimension λ. Such artifacts can lead to errors in subsequent analysis and / or necessitate alignment (e.g., the need to adjust the position in spectral dimension λ so that a specific (x,y) position corresponds to the same physical location on the object).

[0038] In contrast, the snapshot imaging system 140 can capture the entire data cube 120 in one integration time or one exposure, thus avoiding the impact of the aforementioned motion on image quality. Figure 2C shows an example of an image sensor 142 and optical filter array 144 (such as a color filter array (CFA)) that can be used to fabricate a snapshot imaging system. In this example, the color filter array (CFA) 144 has a repeating pattern of color filter units 146 arranged on the surface of the image sensor 142. This method for obtaining spectral information can also be called a multispectral filter array (MSFA) or spectral resolution detector array (SRDA). In the example shown in this figure, the color filter unit 146 includes various color filters arranged in a 5x5 pattern, so that 25 spectral channels are generated in the resulting image data. The CFA can divide incident light into the bands of each color filter by using various color filters and guide the divided light to each color photoreceptor on the image sensor. In this way, for a given color 148, only one of the 25 photoreceptors detects a signal indicating light of that wavelength. Therefore, when using such a snapshot imaging system 140, 25 color channels can be generated in a single exposure, but the amount of measurement data for each color channel is smaller than the total output of the image sensor 142. In some embodiments, the CFA may include one or both of a filter array (MSFA) and a spectrally resolved detector array (SRDA), and / or may include a conventional Bayer filter, CMYK filter, or other absorption or interference filter. One type of interference filter is a thin-film filter array arranged in a grid, where the elements in each grid correspond to one or more sensor elements. Another type of interference filter is the Fabry-Perot filter.Nano-etched Fabry-Perot interference filters typically exhibit a bandpass characteristic with a full width at half maximum (FWHM) of approximately 20–50 nm (for example, a FWHM defined by a range of 20 nm, 21 nm, 22 nm, 23 nm, 24 nm, 25 nm, 26 nm, 27 nm, 28 nm, 29 nm, 30 nm, 31 nm, 32 nm, 33 nm, 34 nm, 35 nm, 36 nm, 37 nm, 38 nm, 39 nm, 40 nm, 41 nm, 52 nm, 53 nm, 44 nm, 45 nm, 46 nm, 47 nm, 48 nm, 49 nm, or 50 nm, or a range defined by any two of these wavelengths as upper and lower limits). However, they are advantageous in that they can be used in several embodiments because they exhibit a small roll-off in the transition from the center wavelength of the passband to the stopband. Furthermore, such filters can improve sensitivity to light outside the passband by exhibiting a low optical density (OD) in the stopband. Due to these combined effects, these particular filters are sensitive to spectral regions that would be blocked by the high roll-off of similar high optical density interference filters with FWHM composed of numerous thin film layers by coating deposition methods such as vapor deposition or ion beam sputtering. In embodiments using dye-based CMYK filters or RGB-based (Bayer) filters, it is preferable that the spectral roll-off is small and the FWHM of the passband of each filter is large, thereby making the spectral transmittance for each wavelength unique across the entire observed spectrum.

[0039] Therefore, the data cube 120 obtained by the snapshot imaging system has one of the following two characteristics that can be problematic in high-precision imaging applications. The first problem is that the data cube 120 obtained by the snapshot imaging system is N larger than the size of the detector array (x,y). x and N ySince its size can be reduced, its resolution is lower than that of the data cube 120 obtained by a scanning imaging system having the same image sensor. As a second problem, in the data cube 120 obtained by a snapshot imaging system, by numerical interpolation for a specific (x,y) position, N x and N y may be the same size as the (x,y) size of the detector array. However, when interpolation is performed in generating the data cube, it means that a specific value of the data cube is an estimated value based on surrounding values rather than an actual measured value of the wavelength of the incident light on the sensor.

[0040] As another existing component used in multi-spectral imaging that performs only one exposure, there is a beam splitter for multi-spectral imaging. In such an imaging system, a beam splitter cube splits incident light into different color bands, and each band is observed by a plurality of independent image sensors. Although the design of the beam splitter can be changed to adjust the spectral band to be measured, it is not easy to split the incident light into more than four beams without degrading the performance of the imaging system. Therefore, four spectral channels are considered to be the practical limit for this approach. In a closely related method, instead of a bulky beam splitter cube / prism, thin film filters are used to split light, but this approach is also limited to about six spectral channels due to cumulative transmittance attenuation and spatial limitations by a plurality of consecutive filters.

[0041] In some embodiments, the aforementioned challenges can be addressed by spectral imaging systems and associated image data processing techniques disclosed herein, utilizing multiplexed illumination, a multibandpass filter that selectively transmits illumination rays, and the color channels of an RGB camera. This particular configuration allows for the achievement of all design goals: fast imaging speed, high-resolution images, and high fidelity of detected wavelengths. Thus, the optical designs and associated image data processing techniques disclosed herein can be used in portable spectral imaging systems and / or for imaging moving objects, and can yield data cubes suitable for high-precision applications (e.g., clinical tissue analysis, biometrics, transient clinical signs). Such high-precision applications, which can be achieved using one or more embodiments described herein, include the diagnosis of basal cell carcinoma, squamous cell carcinoma, and malignant melanoma at pre-metastatic stages (0-3); classification of the severity of burns or wounds in skin tissue; identification of necrotic or ischemic tissue and its margins, for example, by comparison with healthy or normal skin; or histological diagnosis or severity determination of peripheral vascular disease or diabetic foot ulcers. Therefore, the acquisition of snapshot spectra and the small form factor described in some embodiments allow the present invention to be used in clinical settings dealing with transient events, such as the diagnosis of various types of retinopathy (e.g., non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, and age-related macular degeneration), as well as imaging of pediatric patients with high mobility. Accordingly, those skilled in the art will understand that the use of the system disclosed herein represents a significant technological advancement compared to conventional spectral imaging implementations.

[0042] Various aspects of this disclosure are described below in relation to specific examples and embodiments. These examples and embodiments are for illustrative purposes only and do not limit the disclosure. The examples and embodiments described herein focus on specific calculations and algorithms for illustrative purposes, but those skilled in the art will understand that these examples are for illustrative purposes only and do not limit the invention. For example, several examples are given with respect to multispectral imaging, but the single-aperture imaging systems and associated filters and multiplexing illuminations disclosed herein can be configured to perform hyperspectral imaging in other implementations. Furthermore, while certain examples are presented to achieve advantages in handheld applications and / or applications for moving objects, it will be understood that the imaging system designs and associated processing techniques disclosed herein can yield high-precision data cubes suitable for fixed imaging systems and / or analysis of relatively stationary objects.

[0043] Electromagnetic spectral range and image sensor overview In this specification, specific colors or parts of the electromagnetic spectrum are referred to, and these wavelengths are described below according to the definitions in ISO 21348, “Definition of Types of Irradiance Spectra.” As detailed below, in certain imaging applications, a wavelength range of a specific color can be passed through a specific filter collectively.

[0044] Electromagnetic radiation in the wavelength range of 760nm to 380nm or approximately 760nm to approximately 380nm is generally considered to be in the "visible" spectrum, that is, the part of the spectrum that can be perceived by the color receptors of the human eye. In the visible spectrum, red light is generally considered to have a wavelength of 700nm or approximately 700nm, or in the range of 760nm to 610nm or approximately 760nm to approximately 610nm. Orange light is generally considered to have a wavelength of 600nm or approximately 600nm, or in the range of 610nm to 591nm or approximately 610nm to approximately 591nm. Yellow light is generally considered to have a wavelength of 580nm or approximately 580nm, or in the range of 591nm to 570nm or approximately 591nm to approximately 570nm. Green light is generally thought to have a wavelength of 550 nm or approximately 550 nm, or in the range of 570 nm to 500 nm. Blue light is generally thought to have a wavelength of 475 nm or approximately 475 nm, or in the range of 500 nm to 450 nm, or approximately 500 nm to 450 nm. Violet (red-violet) light is generally thought to have a wavelength of 400 nm or approximately 400 nm, or in the range of 450 nm to 360 nm, or approximately 450 nm to 360 nm.

[0045] Looking beyond the visible spectrum, infrared (IR) refers to electromagnetic radiation with wavelengths longer than visible light and is usually invisible to the human eye. IR has wavelengths ranging from approximately 760 nm or 760 nm, the nominal lower limit of the visible spectrum, to approximately 1 mm or 1 mm. Within this range, near-infrared (NIR) refers to the portion of the spectrum adjacent to the red range, with wavelengths ranging from approximately 760 nm to approximately 1400 nm or 760 nm to 1400 nm.

[0046] Ultraviolet (UV) light refers to a portion of electromagnetic radiation with wavelengths shorter than visible light and is usually invisible to the human eye. UV has wavelengths ranging from approximately 40 nm or the nominal lower limit of the visible spectrum, which is 40 nm, to approximately 400 nm. Within this range, near-ultraviolet (NUV) refers to the portion of the spectrum adjacent to the violet range, with wavelengths of approximately 400 nm to 300 nm or 400 nm to 300 nm; intermediate-ultraviolet (MUV) is the wavelength range of approximately 300 nm to 200 nm or 300 nm to 200 nm; and far-ultraviolet (FUV) is the wavelength range of approximately 200 nm to 122 nm or 200 nm to 122 nm.

[0047] The image sensors described herein can be configured to detect any electromagnetic radiation in the aforementioned regions, depending on the specific wavelength range suitable for a particular application. The spectral sensitivity of typical silicon charge-coupled (CCD) or complementary metal-oxide-semiconductor (CMOS) sensors extends across the entire visible spectrum, but also covers most of the near-infrared (IR) spectrum and part of the UV spectrum. In some implementations, back-illuminated or front-illuminated CCD or CMOS arrays can be used additionally or alternatively. For applications requiring high signal-to-noise ratio and scientific-grade measurements, some implementations can use scientific complementary metal-oxide-semiconductor (sCMOS) cameras or electron-amplified CCD cameras (EMCCDs) additionally or alternatively. In other implementations, depending on the intended application, sensors known to operate in a specific color range (e.g., short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), or long-wavelength infrared (LWIR)) and corresponding optical filter arrays can be used additionally or alternatively. These sensors and optical filter arrays may additionally or alternatively include cameras depending on the detector material, such as indium gallium arsenide (InGaAs) or indium antimony (InSb), or cameras depending on the microbolometer array.

[0048] The image sensors used in the multispectral imaging techniques disclosed herein may be used in conjunction with optical filter arrays such as color filter arrays (CFAs). Several types of CFAs can split incident light into red (R), green (G), and blue (B) visible regions and direct the split visible light to dedicated red, green, or blue photodiodes on the image sensor. A common example of a CFA is a Bayer pattern, which is a color filter array in which RGB color filters are arranged in a specific pattern on a rectangular grid of a photodetector. A Bayer pattern consists of 50% green, 25% red, and 25% blue, with alternating rows of red and green color filters and rows of blue and green color filters. Several types of CFAs (e.g., CFAs for RGB-NIR sensors) can also split NIR light and direct the split NIR light to dedicated photodiodes on the image sensor.

[0049] Therefore, the wavelength range of the CFA's filter components can determine the wavelength range represented by each image channel in the captured image. Thus, in various embodiments, the red channel of the image may correspond to the red wavelength range of the color filter, and may include some yellow and orange light, in the range of approximately 570 nm to approximately 760 nm or 570 nm to 760 nm. In various embodiments, the green channel of the image may correspond to the green wavelength range of the color filter, and may include some yellow light, in the range of approximately 570 nm to approximately 480 nm or 570 nm to 480 nm. In various embodiments, the blue channel of the image may correspond to the blue wavelength range of the color filter, and may include some violet light, in the range of approximately 490 nm to approximately 400 nm or 490 nm to 400 nm. Those skilled in the art will understand that the exact start and end wavelengths (or parts of the electromagnetic spectrum) that define the colors (e.g., red, green, and blue) of the CFA may vary considerably depending on the implementation of the CFA.

[0050] Furthermore, conventional visible light CFAs transmit light outside the visible spectrum. Therefore, many image sensors limit their sensitivity to infrared by placing a thin-film infrared reflective filter in front of the sensor that blocks infrared wavelengths but allows visible light to pass through. However, some imaging systems disclosed herein may omit the thin-film infrared reflective filter and allow infrared light to pass through. Thus, infrared wavelength bands may be collected using a red channel, a green channel, and / or a blue channel. In some implementations, a blue channel may be used to collect a specific NUV wavelength band. Since the red, green, and blue channels exhibit different spectral responses at each wavelength in the stacked spectral image, each channel has a unique transmission efficiency, known transmission profiles may be used to provide a specific weighted response to each spectral band before unmixing. For example, this weighted response may include known transmission responses of the red, blue, and green channels in the infrared and ultraviolet wavelength regions, thereby enabling the use of each channel for collecting bands from these wavelength regions.

[0051] As will be further detailed below, by placing additional color filters before the CFA along the optical path toward the image sensor, specific bands of light incident on the image sensor can be selectively extracted. The color filters disclosed herein are combinations of (thin film) dichroic filters and / or absorption filters, or a single dichroic filter and / or a single absorption filter. Some of the color filters disclosed herein may be bandpass filters that pass frequencies within a specific region (of the passband) but block (attenuate) frequencies outside that region (in the stop region). Some of the color filters disclosed herein may be multibandpass filters that pass through multiple discontinuous wavelength regions. These “frequency bands” may have a narrower passband, greater attenuation in the stop region, and a steeper spectral roll-off compared to the wide color range of the CFA filter. Roll-off is defined as the steepness of the spectral response at the transition from the passband to the stop region of the filter. For example, the color filters disclosed herein can cover a passband of approximately 20 nm to approximately 40 nm or 20 nm to 40 nm. Such a specific configuration of the color filter may determine the wavelength band that actually enters the sensor, which may improve the accuracy of the imaging techniques disclosed herein. The color filters described herein can be configured to selectively block or pass through a specific band of electromagnetic radiation within the above-mentioned region, depending on the specific wavelength band suitable for a particular application.

[0052] In this specification, the term “pixel” is used to describe the output generated by an element of a two-dimensional detector array. In contrast, a photodiode of a two-dimensional detector array, i.e., a single photosensitive element, behaves as a transducer that can convert photons into electrons via the photoelectric effect, and then convert these electrons into a signal usable for determining pixel values. An element of a data cube can be called a “voxel” (e.g., an element with volume). A “spectral vector” refers to a vector representing spectral data at a specific (x,y) position of the data cube (e.g., the spectrum of light received from a specific point in object space). In this specification, a horizontal plane of a data cube (e.g., an image representing one spectral dimension) is called an “image channel.” In certain embodiments described herein, video information of the spectrum may be captured, and the resulting data dimensions may be N x N y N λ N t It can be assumed that this is the form of a "hypercube" shown by (where N t (This is the number of frames captured during the video sequence.)

[0053] Overview of an example imaging system Figure 3A shows a schematic diagram of an example of a multi-aperture imaging system 200 comprising a curved multibandpass filter according to the present disclosure. The schematic diagram shown herein includes a first image sensor region 225A (photodiodes PD1-PD3) and a second image sensor region 225B (photodiodes PD4-PD6). Photodiodes PD1-PD6 may be, for example, photodiodes formed on a semiconductor substrate (e.g., a CMOS image sensor). Typically, each photodiode PD1-PD6 may be a single unit consisting of some material, semiconductor, sensor element, or other device capable of converting incident light into electric current. This figure shows only a small part of the entire multi-aperture imaging system for the purpose of illustrating the structure and operation of the multi-aperture imaging system, and it should be fully understood that in implementation, the image sensor region can comprise hundreds or thousands of photodiodes (and corresponding color filters). The first image sensor region 225A and the second image sensor region 225B may be implemented as separate sensors or as separate regions on the same image sensor, depending on the implementation. Figure 3A shows two apertures and their corresponding optical paths and sensor regions. However, it should be clear that the optical design principle shown in Figure 3A can be expanded to include three or more apertures and their corresponding optical paths and sensor regions, depending on the implementation.

[0054] The multi-aperture imaging system 200 includes a first aperture 210A that provides a first optical path toward a first sensor area 225A, and a second aperture 210B that provides a first optical path toward a second sensor area 225B. These apertures may be adjustable to increase or decrease the brightness of the light reflected in the image, or by adjusting these apertures, the exposure time of a particular image may be changed so that the brightness of the light incident on the image sensor area does not change. These apertures may be positioned at any location along the optical axis of this multi-aperture system, as can be determined by a person skilled in the field of optical design. The optical axis of optical components positioned along the first optical path is shown by a dashed line 230A, and the optical axis of optical components positioned along the second optical path is shown by a dashed line 230B, but it should be understood that these dashed lines do not represent the physical structure of the multi-aperture imaging system 200. Optical axes 230A and 230B are separated by a distance D, which can be the parallax between the image captured by the first sensor area 225A and the image captured by the second sensor area 225B. "Parallax" refers to the distance between two corresponding points on the left and right (or top and bottom) sides of a stereoscopic image, where the same physical point in object space appears at different positions in each image. Processing techniques that correct and utilize this parallax are described in more detail below.

[0055] Optical axes 230A and 230B pass through the center C of their respective apertures. Other optical components can also be positioned around these optical axes (for example, the rotational symmetry points of optical components can be positioned along these optical axes). For example, a first curved multibandpass filter 205A and a first imaging lens 215A can be positioned around the first optical axis 230A, and a second curved multibandpass filter 205B and a second imaging lens 215B can be positioned around the second optical axis 230B.

[0056] In this specification, the terms “above” and “above” used when describing the arrangement of optical elements refer to the location of a specific structure (e.g., a color filter or lens) that light entering the imaging system 200 from object space passes through before reaching (or being incident on) another structure. To illustrate this, along the first optical path, the curved multibandpass filter 205A is located above the aperture 210A, the aperture 210A is located above the imaging lens 215A, the imaging lens 215A is located above the CFA 220A, and the CFA 220A is located above the first image sensor area 225A. Thus, light entering from object space (e.g., the physical space to be imaged) first passes through the first curved multibandpass filter 205A, then through the aperture 210A, then through the imaging lens 215A, then through the CFA 220A, and finally into the first image sensor area 225A. The second optical path (for example, the optical path including the curved multibandpass filter 205B, aperture 210B, imaging lens 215B, CFA 220B, and second image sensor area 225B) has a similar arrangement. In another implementation, the apertures 210A, 210B and / or imaging lenses 215A, 215B can be placed above the curved multibandpass filters 205A, 205B. Furthermore, in another implementation, a physical aperture may not be used, and the brightness of the light imaged on sensor area 225A and sensor area 225B may be controlled by relying on the clear aperture of the optical components. Thus, the lenses 215A, 215B may be placed above the apertures 210A, 210B and the curved multibandpass filters 205A, 205B. In this implementation, apertures 210A and 210B may be positioned above lenses 215A and 215B, or below lenses 215A and 215B, if deemed necessary by a person skilled in the field of optical design.

[0057] The first CFA 220A, positioned above the first sensor area 225A, and the second CFA 220B, positioned above the second sensor area 225B, can function as filters that selectively pass specific wavelengths, thereby splitting the incident visible light into red, green, and blue regions (indicated by the symbols R, G, and B, respectively). The light is "split" by allowing only the selected specific wavelengths to pass through the respective color filters of the first CFA 220A and the second CFA 220B. The split light is received by dedicated red, green, or blue diodes on the image sensor. While red, blue, and green filters are commonly used, in another embodiment, various color filters can be used depending on the color channels required for the captured image data. For example, filters that selectively pass ultraviolet, infrared, or near-infrared light can be used, similar to an RGB-IR CFA.

[0058] As shown in Figure 3A, each filter of the CFA is positioned above a single photodiode PD1-PD6. Figure 3A also shows an example of microlenses (indicated as ML), which can be formed on each color filter or otherwise positioned on each color filter to focus incident light onto the active detector area. In another implementation, there may be multiple photodiodes below a single filter (for example, a collection of two, four or more adjacent photodiodes). In the example shown in this figure, photodiodes PD1 and PD4 are located below the red filter and therefore output pixel information for the red channel; photodiodes PD2 and PD5 are located below the green filter and therefore output pixel information for the green channel; and photodiodes PD3 and PD6 are located below the blue filter and therefore output pixel information for the blue channel. Furthermore, as detailed below, the channel output of a specific color from a given photodiode can be narrowed to an even narrower frequency band based on a specific frequency band that has passed through an active light source and / or multibandpass filters 205A, 205B, thereby enabling the output of various image channel information from a given photodiode under various exposure conditions.

[0059] The imaging lenses 215A and 215B can be shaped to focus an image of an object scene on the sensor areas 225A and 225B. Each imaging lens 215A and 215B is not limited to a single convex lens as shown in Figure 3A, but may consist of the number of optical elements and surfaces required for image formation, and various types of commercially available or custom-designed imaging lenses or lens assemblies can be used. Each optical element or lens assembly may be formed to house or stack in series or be coupled to one another using an opticomechanical lens barrel with retaining rings or bezels. In some embodiments, the optical elements or lens assembly may include one or more coupled lens groups, which may be two or more optical components joined together or otherwise coupled together. In various embodiments, the multibandpass filters described herein may be positioned in front of the lens assembly of a multispectral imaging system, in front of a single lens of a multispectral imaging system, behind the lens assembly of a multispectral imaging system, behind a single lens of a multispectral imaging system, inside the lens assembly of a multispectral imaging system, inside the combined lens group of a multispectral imaging system, directly on the surface of a single lens of a multispectral imaging system, or directly on the surface of an element of the lens assembly of a multispectral imaging system. Furthermore, apertures 210A and 210B may be removed, and lenses of the type commonly used in photography with digital single-lens reflex (DSLR) or mirrorless cameras may be used for lenses 215A and 215B. Furthermore, lenses 215A and 215B may be lenses of the type used in machine vision using C-mount or S-mount threads for mounting.Focusing can be performed, for example, by moving the imaging lenses 215A and 215B relative to the sensor areas 225A and 225B, or by moving the sensor areas 225A and 225B relative to the imaging lenses 215A and 215B, based on manual focus, contrast-based autofocus, or other suitable autofocus technology.

[0060] Each multibandpass filter 205A, 205B can be configured to selectively pass multiple lights in a narrow frequency band, for example, in some embodiments, they can be configured to selectively pass multiple lights in the 10-50 nm frequency band (in other embodiments, multiple lights in a wider or narrower frequency band). As shown in Figure 3A, both multibandpass filters 205A, 205B can pass through the frequency band λc ("common frequency band"). In implementations with three or more optical paths, each multibandpass filter can pass through this common frequency band. In this way, each sensor region captures image information in the same frequency band ("common channel"). As will be described in more detail below, this image information obtained in this common channel can be used to align a set of images captured by each sensor region. In some implementations, there may be one common frequency band and a corresponding common channel, or there may be multiple common frequency bands and their corresponding common channels.

[0061] Each multibandpass filter 205A, 205B can be configured to selectively pass one or more specific frequency bands in addition to the common frequency band λc. In this way, the imaging system 200 can capture more different spectral channels collectively by the sensor regions 205A, 205B than the number of spectral channels that can be captured by a single sensor region. This embodiment allows for the selective passage of one or more specific frequency bands λc. u1 The multibandpass filter 205A allows the signal to pass through, and the specific frequency band λ u2The multibandpass filter 205B that allows the signal to pass through is shown in Figure 3A, where λ u1 and λ u2 These represent different frequency bands. In this figure, two frequency bands are shown passing through, but each multibandpass filter of the present disclosure can pass through two or more frequency bands at once. For example, as will be discussed later in relation to Figures 11A and 11B, in some implementations each multibandpass filter can pass through four frequency bands. In various embodiments, more frequency bands may be passed through. For example, an implementation using four cameras may have a multibandpass filter configured to pass through eight frequency bands. In some embodiments, the number of frequency bands may be, for example, four, five, six, seven, eight, nine, ten, twelve, fifteen, sixteen or more.

[0062] The multibandpass filters 205A and 205B have curvatures selected to reduce the incident angle-dependent spectral transmission in their respective sensor regions 225A and 225B. As a result, when receiving narrowband illumination from object space, each photodiode (e.g., a color filter covering the sensor region and allowing that wavelength to pass through) placed across the entire surface of sensor regions 225A and 225B, which are sensitive to that wavelength, is thought to receive light of substantially the same wavelength without causing the wavelength shift seen in photodiodes placed near the edges of the sensor, as described above in relation to Figure 1A. This configuration allows for the generation of more accurate spectral image data than when using flat filters.

[0063] Figure 3B shows an example of the optical design of optical components that constitute one optical path of the multi-aperture imaging system shown in Figure 3A. More specifically, Figure 3B shows a custom achromatic doublet 240 that can be used as multibandpass filters 205A, 205B. This custom achromatic doublet 240 allows light to pass through the housing 250 and send it to the image sensor 225. The housing 250 may include apertures 210A, 210B and imaging lenses 215A, 215B as described above.

[0064] The achromatic doublet 240 is configured to correct optical aberrations by incorporating the surfaces required for the multibandpass filter coatings 205A and 205B. The achromatic doublet 240 shown in this figure includes two lenses that can be made from glass or other optical materials with different dispersion and refractive indices. In other implementations, three or more lenses may be used. The design of these achromatic doublet lenses incorporates the multibandpass filter coatings 205A and 205B on a curved front surface 242, canceling out optical aberrations by incorporating the optical surface of a curved single lens with the filter coatings 205A and 205B deposited on it. The combined effect of the curved front surface 242 and curved back surface 244 of the achromatic doublet 240 limits the refractive power or light-gathering power, and these lenses alone, housed in the housing 250, can constitute the primary element for light gathering. Therefore, the achromatic doublet 240 can contribute to improving the accuracy of image data captured by the imaging system 200. These individual lenses can be mounted adjacent to each other, and for example, by combining or joining the individual lenses, they can be shaped so that the aberration of one lens is canceled out by the aberration of the other lens. The curved front 242 or curved back 244 of the achromatic doublet 240 can be coated with multibandpass filter coatings 205A, 205B. Other doublet designs may be implemented in the system described herein.

[0065] Further modifications of the optical designs described herein may be implemented. For example, in some embodiments, instead of the doublet 240 shown in Figure 3B, a single lens or other optical single lens, such as the convex (positive) or concave (negative) lens shown in Figure 3A, may be included in the optical path. Figure 3C shows an example implementation in which a flat filter 252 is placed between the lens housing 250 and the sensor 225. The achromatic doublet 240 introduced in Figure 3C, incorporating the flat filter 252 with multiband transmission characteristics, corrects optical aberrations without significantly contributing to the refractive power of each lens housed in the housing 250. Figure 3D shows another example implementation in which a multibandpass coating is implemented by applying a multibandpass coating 254 to the front surface of the lens assembly housed in the housing 250. Thus, this multibandpass coating 254 may be applied to any curved surface of any optical element housed in the housing 250.

[0066] Single-aperture multispectral imaging system and method Figure 4(a) shows a system diagram of an example of a multispectral imaging system according to the present disclosure. In this design, the color camera is an image sensor, but various other types of image sensors may be used according to the art of the present invention. This multispectral imaging system may be equipped with a motorized zoom lens so that the field of view (FOV) can be adjusted to capture images. A motorized zoom lens can be advantageous in that the field of view can be adjusted according to various imaging conditions. For example, in biomedical applications in wound imaging, a wide field of view may be desirable for imaging large areas of burns, while a narrow field of view may be desirable for imaging small area features such as diabetic foot ulcers (DFUs). In one exemplary embodiment, the diameter of the field of view may be in the range of 8 cm to 23 cm, but is not limited thereto. An 8-band optical filter is placed between the motorized zoom lens and the color camera, and these 8 frequency bands are designed as multispectral imaging bands. In this particular example, the center wavelengths of the eight frequency bands may be 420 nm, 525 nm, 581 nm, 620 nm, 660 nm, 726 nm, 820 nm, and 855 nm, respectively. Eight LED light wavelengths with the same spectral bandwidth can be provided for illumination. In some embodiments, LEDs in each frequency band can be selectively switched on and off independently of the other LEDs. A bandpass optical filter with a spectral bandwidth matching each LED is placed in front of each LED and used to confine the spectrum of the illumination light from each LED.

[0067] Figure 4(b) shows the quantum efficiency (Q%) of the RGB sensor of the color camera. The blue, green, and red lines represent the blue, green, and red channels of the color camera, respectively. Figure 4(c) shows the transmission coefficient of the 8-band filter, which has a total of eight bandpass bands in this spectrum. These eight bandpass bands are desirable as multispectral imaging bands. Similarly, Figure 4(d) shows the illuminance (I0) of eight independent LEDs on the spectrum. The illumination bands of these LEDs can be superimposed on the 8-band filter bands.

[0068] Figure 5 shows an example of the time series of LED illumination during multispectral imaging (MSI) acquisition, and the detection coefficients for each channel corresponding to this time series. This LED illumination time series may be implemented in combination with the multispectral imaging system shown in Figure 4. The time series shown in Figure 5 is an example of illumination and spectral unmixing that may be used within the scope of the present invention. The multispectral imaging system shown in Figure 4 may be used in combination with various time series of LEDs without departing from the gist or scope of the present invention, and / or the time series shown in Figure 5 may be used in combination with various multispectral imaging systems.

[0069] In the first stage, the LED lights of channels 1, 3, and 5 (420nm, 581nm, and 660nm) in Figure 5(I) are turned on. During this time window, one image is acquired by the color camera. Three wavelengths pass through the optical filter and are captured by the RGB channels of the color camera, and these three wavelengths have different filter transmission coefficients (T%) and RGB detection coefficients (Q%) as shown in (I). Next, in the second stage, as shown in (II), the LED lights of channels 1, 3, and 5 are turned off, and the LED lights of channels 2, 4, and 6 (525nm, 620nm, and 726nm) are turned on. Similarly, by acquiring one image by the color camera during this time window, three wavelengths with specific coefficients are captured by the three camera channels. In the third stage, as shown in (III), the LED lights of channels 2 and 4 are turned off (while channel 6 at 726 nm remains lit), and the LED lights of channels 7 and 8 (820 nm and 855 nm) are turned on, and a third image of the wavelengths of channels 6, 7 and 8 with specific coefficients is acquired by the color camera.

[0070] In some embodiments, the initial calibration can be performed. First, three images of a standard plate (e.g., a white Zenith target) are acquired with a color camera in the following order: (1) when the LED illumination is off; (2) when the LED lights for channels 1, 3, and 5 (420nm, 581nm, and 660nm) are on (other LED channels are off); (3) when the LED lights for channels 2, 4, and 6 (525nm, 620nm, and 726nm) are on (other LED channels are off); and (4) when the LED lights for channels 6, 7, and 8 (726nm, 820nm, and 855nm) are on (other LED channels are off). In some embodiments, the standard plate may be a white Zenith target that reflects approximately 95% of the photons in the entire spectrum, and this standard plate can be used to perform flat-field correction of the illumination.

[0071] Next, acquire three images of the imaging target in the same order using a color camera (for example, with LED lighting off; then with channels 1, 3, and 5 lit; then with channels 2, 4, and 6 lit; and finally with channels 6, 7, and 8 lit). By performing background subtraction and flat-field correction, the following formula can be used.

number

number

number

[0072] Similarly, when the LED lighting is switched on with a specific combination of wavelengths, the reflection coefficients of the other channels, namely channels 2, 620 nm, 726 nm, 820 nm, and 855 nm, can also be determined using the corresponding unmixing matrices. Therefore, eight MSI images are generated based on the images for each channel obtained from the three unmixing matrices.

[0073] Referring to Figure 5, the combination of image acquisition time series and LED illumination described above is one example of a method for acquiring eight MSI images. In practical use, the LED illumination and image acquisition time series can be designed in various combinations and / or timings, and the combination of LED illumination and image acquisition time series may be selected and used based on hardware performance, empirical rules, etc. Alternatively, if each LED wavelength is designed to be lit individually, a monochrome camera can be used, and therefore an unmixing matrix is ​​not required. In this case, the performance of the MSI can be evaluated by photographing the target and comparing the quality of the MSI image with the true value of the spectrum (i.e., color calibration).

[0074] An example of the implementation and results of a single-aperture multispectral imaging system. Figures 6 and 7 show an example of an embodiment of a single-aperture, single-camera, multispectral image acquisition apparatus that may be implemented using the MSI system and method disclosed herein. Figure 6 is a perspective view of a handheld apparatus equipped with an LED array and a camera for the aforementioned multiplexed illumination. The handheld apparatus shown in Figure 6 may include one or more processors and memory for storing acquired image data. Further image data processing, such as spectral unmixing and / or generation of individual spectral images as described herein, may be performed within the handheld apparatus shown in Figure 6, or without departing from the scope of this disclosure, with the full or partial control of one or more remote computing devices, or with the handheld apparatus shown in Figure 6 in combination with the full or partial control of one or more remote computing devices.

[0075] Figure 7 shows an example of an LED configuration containing a total of 24 LEDs. In this LED configuration, three LEDs of eight different wavelength bands are arranged in a ring around the single aperture of the image acquisition device. To reduce unmixing errors caused by complexity, the number of LED illumination colors has been increased to eight wavelengths. Each LED covers a predetermined MSI spectral band, and its center wavelengths may be approximately 420 nm, 525 nm, 581 nm, 620 nm, 660 nm, 726 nm, 820 nm, and 880 nm, respectively. Because this image sensor has a single aperture design, a new spatial arrangement of LEDs was developed to reduce the size of the illumination substrate and the front of the image sensor. Due to the ring design, the photon collection efficiency from the LED illumination to the central camera aperture is desirable. To obtain uniform illumination at a desired working distance (e.g., 40 cm in some embodiments), each LED color on the illumination substrate may consist of up to three or more LEDs, and the beam angle of each LED may be 120°. A diffuser can be placed on the illumination substrate. Spacers are placed in place to ensure sufficient distance between the LEDs and the diffuser to prevent thermal damage to the diffuser. To enable testing of various color combinations to optimize the design of the image acquisition time series, one or more processors built into the imaging head (e.g., the handheld device shown in Figure 6) can be used to control the individual LEDs. Furthermore, such a configuration allows for adjustment of the illumination calibration time of individual light sources to obtain a high signal-to-noise ratio.

[0076] Figures 8-10 show the results of tests using a single-aperture system according to the technology of the present invention. Figure 8 shows RGB images of a Macbeth target and a Zenith target, captured using a single-lens, single-aperture, single-camera system under conditions of illumination with seven types of color LEDs, and differentially subtracted from the ambient room lighting as the background. In this test example, seven types of color LEDs (blue, green, PC amber, crimson, far-infrared, NIR I, and NIR II) were placed on the illumination substrate and individually turned on and off in a time series. A total of 7 × 8 = 56 images were acquired on the RGB camera using eight types of MSI bandpass optical filters (center wavelength / full width half-power (FWHM): 420 / 10nm, 520 / 10nm, 580 / 10nm, 620 / 10nm, 660 / 10nm, 730 / 10nm, 810 / 10nm, and 850 / 10nm).

[0077] Using a linear post-processing method, adding eight images acquired by illuminating with LEDs of the same color and passing them through eight different filters yielded the same value as seven RGB images acquired under seven different illumination conditions and passed through eight band filters. A Macbeth target was imaged for spectroscopic analysis, and a Zenith target with 95% reflectivity was imaged for flat-field correction. Next, eight MSI images were generated using an unmixing algorithm. The principle of unmixing is outlined herein. The detected signal (S) is the integral of the product of illuminance (I), target reflectivity (R), transmission coefficient of the optical system including the lens and bandpass filter (T), and quantum efficiency of the sensor in the spectral band (λ) (Q), and is given by the following equation.

number

[0078] After generating MSI images, the spectral accuracy was analyzed by comparing the MSI spectra of each patch of the Macbeth target with the true values ​​measured by a calibrated spectrometer. The comparison results are shown in Figure 10. The comparison results showed that the single-aperture MSI system design improved by the proposed invention resulted in higher spectral measurement accuracy (average correlation coefficient = 0.97).

[0079] term All methods and tasks described herein may be performed by a computer system and may be fully automated. In some cases, this computer system may include several separate computers or computer devices (e.g., physical servers, workstations, storage arrays, cloud computing resources, etc.) that communicate and interoperate over a network to perform the functions described herein. Each such computer device typically includes a processor (or more processors) that executes program instructions or program modules stored in memory or other non-temporary computer-readable storage media or non-temporary computer-readable storage devices (e.g., solid-state storage devices, disk drives, etc.). The various functions disclosed herein may be embodied in such program instructions or implemented in the form of application-specific circuits (e.g., ASICs or FPGAs) for the computer system. If the computer system includes several computer devices, these devices may be located in the same location or in different locations. The results of the methods and tasks disclosed herein may be permanently stored by converting them into various forms of physical storage devices such as solid-state memory chips or magnetic disks. In some embodiments, the computer system may be a cloud-based computing system in which processing resources are shared by several separate enterprises or other users.

[0080] The processes disclosed herein may be initiated when requested by a user or system administrator, in response to an event such as a predetermined schedule or a dynamically determined schedule, or in response to several other events. Once the processes are initiated, executable program instructions stored in one or more non-temporary computer-readable media (e.g., hard drives, flash memory, removable media) may be loaded into the memory (e.g., RAM) of a server or other computer device. The executable instructions may then be executed by a hardware-based computer processor of the computer device. In some embodiments, the processes or parts thereof disclosed herein may be implemented in series or in parallel on multiple computer devices and / or multiple processors.

[0081] Depending on the embodiment, specific actions, events, or functions performed by any of the processes or algorithms described herein may be performed in different orders, added to one another, combined with one another, or omitted entirely (for example, not all of the actions and events described herein are necessary for the execution of the algorithms described herein). Furthermore, in certain embodiments, multiple actions or events may be performed simultaneously rather than sequentially, for example, using multithreading, interrupt handling, or multiple processors or processor cores, or on other parallel structures.

[0082] The various exemplary logic blocks, modules, routines, and algorithmic processes described in relation to the embodiments disclosed herein can be implemented as electronic devices (e.g., ASICs or FPGA devices), computer software running on computer hardware, or a combination thereof. Furthermore, the various exemplary logic blocks and modules described in relation to the embodiments disclosed herein can be implemented by devices such as processor devices, digital signal processors ("DSPs"), application-specific integrated circuits ("ASICs"), field-programmable gate arrays ("FPGAs") or other programmable logic devices, discrete gate logic or transfer logic, discrete hardware components, or combinations of these components designed to perform the functions described herein. The processor device may be a microprocessor, but in another embodiment, the processor device may be a control device, a microcontroller or a state machine, or a combination thereof. The processor device may include electrical circuits configured to process computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable devices that perform logical operations without processing computer-executable instructions. Furthermore, the processor unit can be implemented as a combination of computer units, for example, as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of a DSP core and one or more microprocessors, or other such configurations. Although this specification has mainly described digital technologies, the processor unit may mainly include analog components. For example, all or some of the rendering techniques described herein may be implemented using analog circuits, or using a combination of analog and digital circuits.A computing environment can include, but is not limited to, any type of computer system, including, some examples, microprocessor-based computer systems, mainframe computers, digital signal processors, portable computer devices, device controllers, and computer engines in electrical appliances.

[0083] Components of methods, processes, routines, or algorithms described in relation to embodiments disclosed herein can be embodied directly in hardware, directly in software modules executed by a processor device, or directly in a combination of these hardware and software modules. Software modules may be contained in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or other forms of non-temporary computer-readable storage media. Typical storage media can be connected to the processor device so that the processor device can read information from or write information to such storage media. In this embodiment, the storage media may be essential to the processor device. This processor device and storage media may be contained in an ASIC. This ASIC may be contained in a user terminal. In this embodiment, this processor device and storage media may exist as discrete components of the user terminal.

[0084] As used herein, conditional terms such as “may,” “may,” “possibly,” “may,” and “for example” generally mean, unless otherwise stated or interpreted differently in the context in which they are used, that a particular embodiment includes a particular feature, component, or process, and that another embodiment does not include these features, components, or processes. Therefore, such conditional terms generally do not imply that, regardless of whether a particular embodiment includes or implements a particular feature, component, or process, one or more embodiments require these features, components, or processes in any way, or that one or more embodiments necessarily include a decision logic, in the presence or absence of other inputs or prompting. Furthermore, terms such as “include,” “equip,” and “have” are synonymous and are used in an open-ended, comprehensive sense, not excluding additional components, features, actions, operations, etc. Additionally, the term “or” is used in a comprehensive sense (not exclusively), and for example, when the term “or” is used to list components, it means one, some, or all of the listed components.

[0085] Unless otherwise stated, disjunctive terms such as "at least one of X, Y, or Z" are generally understood in this specification to indicate that a particular item or term may be X, Y, or Z, or any combination thereof (e.g., X, Y, or Z). Therefore, such disjunctive terms are not, and should not, be intended to mean that the presence of at least one X, at least one Y, and at least one Z is required in any particular embodiment.

[0086] While the detailed description above has shown, explained, and pointed out novel features applicable to various embodiments, it will be understood that the forms and details of the aforementioned apparatus or algorithm can be omitted, substituted, and modified in various ways without departing from the scope of this disclosure. Since some of the features described herein can be used or implemented separately from others, it will be readily understood that certain embodiments described herein may be embodied in forms that do not provide all of the features and benefits described herein. Any changes that are equivalent in meaning and scope to the claims are included within the scope of these claims.

[0087] This invention includes the following inventions. [1] A multispectral imaging system comprising: a light source configured to selectively emit light including one or more frequency bands from a set of four or more predetermined frequency bands to illuminate an object; an image sensor configured to receive the portion of the emitted light reflected by the object; an aperture arranged to allow the portion of the emitted light reflected by the object to pass through and be incident on the image sensor; a multibandpass filter placed on the aperture and configured to allow the light of the four or more predetermined frequency bands to pass through; a memory storing instructions for generating a multispectral image and performing unmixing; and at least one processor, wherein the at least one processor performs at least the predetermined A multispectral imaging system configured to emit a first subset of light consisting of two or more frequency bands from a predetermined frequency band from a light source; receive first image data generated based on the reflected light of the first subset of light from an image sensor; emit a second subset of light consisting of two or more frequency bands from a predetermined frequency band from a light source; receive second image data generated based on the reflected light of the second subset of light from an image sensor; process the first image data and the second image data to generate at least a first multispectral image and a second multispectral image; and perform spectral unmixing to generate multiple images of the object, each corresponding to a single frequency band among any one of the four or more predetermined frequency bands. [2] The multispectral imaging system according to [1], wherein the processor is further configured to emit a third subset or more subsets of light consisting of two or more frequency bands from the predetermined frequency bands from the light source, and to receive a third image data or more image data generated based on the reflected light of the third subset or more subsets of light from the image sensor. [3] The multispectral imaging system according to [1] or [2], wherein the light source includes a plurality of light-emitting diodes (LEDs), and each LED is configured to emit light in any one of the four or more predetermined frequency bands. [4] The multispectral imaging system according to [3], wherein at least one processor is further configured to select a frequency band to be simultaneously emitted by the light source by controlling the activation of the individual LEDs of the light source. [5] The multispectral imaging system according to [4], wherein the processor controls the activation of individual LEDs by controlling an electronic switching shutter that selectively passes through and blocks the light emitted by each LED. [6] The multispectral imaging system according to any one of [1] to [5], wherein the multibandpass filter has multiple frequency bands through which light passes, and each frequency band corresponds to one of the four or more predetermined frequency bands. [7] The multispectral imaging system according to any one of [3] to [6], wherein the LED and the multibandpass filter have the same number of frequency bands, and each frequency band of the LED is arranged to coincide with the corresponding frequency band of the multibandpass filter. [8] A multispectral imaging system according to any one of [3] to [6], further comprising a bandpass filter placed on each of the plurality of LEDs, wherein the light emitted by each LED is confined to a precise spectrum. [9] The multispectral imaging system according to [3], wherein the set of four or more predetermined frequency bands includes eight predetermined frequency bands, and the light source includes eight LEDs, each LED configured to emit light of any one of the predetermined frequency bands.

[10] The multispectral imaging system according to [9], wherein the multibandpass filter includes an 8-band filter configured to pass each of the eight predetermined frequency bands.

[11] The multispectral imaging system according to

[10] , wherein at least one processor is further configured to cause a third subset of light in a predetermined frequency band to emit from the light source in accordance with the instructions, and to receive a third image data generated based on the reflected light of the third subset of light from the image sensor.

[12] The multispectral imaging system according to

[11] , wherein each of the first subset, the second subset, and the third subset includes three frequency bands from the predetermined frequency bands.

[13] The multispectral imaging system according to [9], wherein the predetermined frequency band is defined by the central wavelength in the range from ultraviolet (UV) to short-wavelength infrared.

[14] The multispectral imaging system according to

[13] , wherein the center wavelengths of the predetermined frequency bands are 420 nm, 525 nm, 581 nm, 620 nm, 660 nm, 726 nm, 820 nm, and 855 nm, respectively.

[15] A multispectral imaging system according to any one of [1] to

[14] , further comprising a motorized parfocal zoom lens or a motorized variable-focus zoom lens positioned on the aperture, wherein one or more processors are further configured to control the motorized zoom lens to adjust the field of view (FOV) of the multispectral imaging system.

[16] The multispectral imaging system according to

[15] , wherein the cofocus zoom lens or the variable focus zoom lens is configured to automatically or manually adjust the focus by changing the focal length (FL) of the lens.

[17] The multispectral imaging system according to

[15] or

[16] , wherein adjusting the field of view of the multispectral imaging system does not change the shooting distance between the object and the image sensor.

[18] The multispectral image sensor according to any one of

[15] to

[17] , further configured to adjust the field of view of the multispectral image system by changing the shooting distance between the object and the image sensor.

[19] The multispectral image sensor according to

[18] , wherein the processor is further configured to compensate for the reduction in contrast caused by the zoom lens.

[20] A multispectral imaging system according to any one of

[15] to

[19] , wherein the field of view is adjustable in a range of at least 8 cm to 23 cm.

[21] The multispectral imaging system according to

[20] , which maintains high spatial resolution without reducing the number of samples in the field of view, unlike digital trimming methods which involve post-processing that sacrifices resolution.

[22] The multispectral imaging system according to any one of [1] to

[21] , wherein one or more processors are further configured to visualize the object in natural false colors based on the multispectral image.

[23] The multispectral imaging system according to any one of [1] to

[22] , wherein one or more processors are configured to perform spectral unmixing by determining the reflection coefficient of the object using a matrix operation formula.

[24] The multispectral imaging system according to

[23] , wherein the reflection coefficient is determined at least in part on the value of the incident illuminance, the value of the transmission coefficient of the multibandpass filter, and the value of the quantum coefficient of the image sensor for each channel.

[25] A multispectral imaging system described in any one of [1] to

[24] that can capture three or more multispectral images in less than 100 milliseconds.

[26] A multispectral imaging system according to any one of [1] to

[25] , wherein the object is a tissue region.

[27] The multispectral imaging system according to

[26] , wherein the tissue includes a wound, cancer, ulcer, or burn.

[28] The multispectral imaging system according to

[27] , wherein the wound includes a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a postoperative wound, an amputation site, a burn, a cancerous lesion, or damaged tissue.

[29] A multispectral imaging system according to

[26] for use in identifying a tissue classification or identifying a tissue healing score, wherein the tissue classification is, for example, living or healthy tissue, dead or necrotic tissue, perfused or unperfused tissue, or ischemic or non-ischemic tissue, and the tissue healing score is, for example, a tendency for at least 50% of the wound to heal after 30 days of standard wound care treatment, or a tendency for at least 50% of the wound not to heal after 30 days of standard wound care treatment.

[30] A multispectral imaging system according to

[26] for use in imaging wounds, cancers, ulcers or burns, wherein the wound, cancer, ulcer or burn is, for example, a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a postoperative wound, an amputation site, a burn, a cancerous lesion or damaged tissue.

Claims

[Claim 1] A multispectral imaging system, A light source configured to selectively emit light containing one or more frequency bands from a set of four or more predetermined frequency bands to illuminate an object; An image sensor configured to receive the portion of the emitted light that has been reflected by the object; An aperture arranged to allow the portion of the emitted light reflected by the object to pass through and be incident on the image sensor; A multibandpass filter, positioned on the aperture and configured to allow light of four or more predetermined frequency bands to pass through; Memory containing instructions for generating a multispectral image and performing unmixing; and at least one processor Includes, The at least one processor, in accordance with the instruction, The light source emits a first subset of light consisting of two or more frequency bands from the predetermined frequency bands; First image data generated based on the reflected light of a first subset of light is received from the image sensor; The light source emits a second subset of light consisting of two or more frequency bands from the predetermined frequency bands; A second image data generated based on the reflected light of a second subset of light is received from the image sensor; Process the first image data and the second image data to generate at least a first multispectral image and a second multispectral image; Perform spectral unmixing to generate multiple images of the object, each corresponding to a single frequency band among the four or more predetermined frequency bands. It is structured in such a way. Multispectral imaging system.

Images