Multi-sensor spectral imaging

Abstract

Described herein is a filter carousel comprising: a planar body having a rotation point; a plurality of filter pairs disposed in the planar body, the plurality of filter pairs comprising a first selected filter positioned at a first radial direction from the rotation point and a second selected filter positioned at a second radial direction from the rotation point; each of the plurality of filter pairs positioned for simultaneous operation; the first selected filter having a first filter center at a first radial distance from the rotation point; the second selected filter having a second filter center at a second radial distance from the rotation point; the first radial distance being lesser than the second radial distance; the first radial direction and the second radial direction are not parallel. Also described is a computer-implemented method for registration of spectral stereo images, the method comprising: selecting a first reference image from a first plurality of images, the first reference image having a better focus quality than a majority of the first plurality of images; applying a first registration transformation to register the first plurality of images with the first reference image to generate a first plurality of registered images; selecting a second reference image from a second plurality of images, the second reference image having a better focus quality than a majority of the second plurality of images; applying a second registration transformation to register the second plurality of images with the second reference image to generate a second plurality of registered images; selecting a third reference image from the first plurality of registered images, and selecting a fourth reference image from the second plurality of registered images, both the third reference image and the fourth reference image acquired in an at least one common wavelength band; determining a third registration transformation based on comparing the third reference image with the fourth reference image.

Description

MULTI-SENSOR SPECTRAL IMAGINGBACKGROUND OF THE INVENTIONField of the InventionThe present invention relates to spectral imaging, and more particularly to multi-sensor spectral imaging.Description of the Related ArtSpectral imaging uses multiple bands in the electromagnetic spectrum, typically having at least one band being outside the visible range. For example, spectral imaging may use microwave wavelengths, infrared wavelengths, visible wavelengths, ultraviolet wavelengths, x-ray wavelengths, and the like, and may include simultaneous acquisition of image data in multiple wavelength bands. Spectral imaging nomenclature may be subcategorized depending on numbers of bands or width of bands, including for example multispectral imaging typically acquiring / capturing image data with a relatively lower number of bands such as less than 40 bands, or for example hyperspectral imaging typically acquiring / capturing image data with a relatively higher number of bands such as greater than 100 bands.Multiple bands used in spectral imaging are often discrete (ie., non-overlapping), however overlapping bands can also be accommodated. Bandwidth can be defined by the difference between minimum and maximum values of the range of wavelengths or frequencies included in an individual band, while the individual band may be characterized by a central or median wavelength or frequency. Bandwidth can vary widely as may be suited to a particular implementation, and in certain implementations bands may be configured to collapse to a range of wavelength / frequency where the central or median value approaches the maximum and minimum limits.Multiple techniques are available to generate and define multiple bands used in spectral imaging. A common technique is to use multiple filters, each of which limits electromagnetic radiation transmitted or emitted to an associated imaging sensor to a desired band of wavelengths / frequencies. Filters are often bandpass filters that permit passage of a limited range of wavelengths and block wavelengths outside of the permitted limited range. Bandpass filters are distinguished from band-stop or band-reject filters that block passage of a limited range of wavelengths and permit passage of wavelengths outside the blocked limited range. Bandpass filter nomenclature may be subcategorized as narrow-band or wide-band depending on relative width of aband (ie., range of continuous wavelengths or frequencies limited by the band). Furthermore, a narrow-bandpass filter may be collapsed to a very limited number of wavelengths, for example less than a Inm range, or corresponding frequencies.Application of spectral imaging is widespread. Greater numbers of observations, each observation defined by an associated band, used in spectral imaging can benefit imaging quality. However, greater numbers of bands can present challenges in configuring a corresponding greater number of filters, and also challenges in digital processing or digital data enhancement of a corresponding greater number of images to extract useful data from a set of acquired images.Accordingly, there is a continuing need for alternative manifold mechanisms for indexing a plurality of filters. Also, there is a continuing need for alternative digital techniques for processing corresponding acquire images. Such alternative manifold mechanisms and alternative digital processing techniques could be independently beneficial or in combination.SUMMARY OF THE INVENTIONIn an aspect there is provided, a filter carousel comprising: a planar body having a rotation point; a plurality of filter pairs disposed in the planar body, the plurality of filter pairs comprising a first selected filter of a first plurality of filters positioned at a first radial direction from the rotation point and a second selected filter of a second plurality of filters positioned at a second radial direction from the rotation point; each of the plurality of filter pairs positioned for simultaneous operation; the first plurality of filters disposed in the planar body, each of the first plurality of filters having a first filter center positioned at a common first radial distance from the rotation point; the second plurality of filters disposed in the planar body, each of the second plurality of filters having a second filter center positioned at a common second radial distance from the rotation point; the first radial distance being lesser than the second radial distance; the first radial direction and the second radial direction are not parallel.In another aspect there is provided, a computer-implemented method for registration of spectral stereo images, the method comprising: obtaining a first plurality of images spanning a first spectral range acquired by a first image sensor, each of the first plurality of images acquired in a different wavelength band;obtaining a second plurality of images spanning a second spectral range acquired by a second image sensor, each of the second plurality of images acquired in a different wavelength band; the first spectral range different than the second spectral range with at least one common wavelength band; selecting a first reference image from the first plurality of images, the first reference image having a better focus quality than a majority of the first plurality of images; applying a first registration transformation to register the first plurality of images with the first reference image to generate a first plurality of registered images; selecting a second reference image from the second plurality of images, the second reference image having a better focus quality than a majority of the second plurality of images; applying a second registration transformation to register the second plurality of images with the second reference image to generate a second plurality of registered images; selecting a third reference image from the first plurality of registered images, and selecting a fourth reference image from the second plurality of registered images, both the third reference image and the fourth reference image acquired in the at least one common wavelength band; determining a third registration transformation based on comparing the third reference image with the fourth reference image; applying the third registration transformation to the first plurality of registered images.BRIEF DESCRIPTION OF THE DRAWINGSFigure 1 is an exploded perspective view of a dual-sensor multispectral imager showing a generalized spatial alignment of a pair of lenses with a filter carousel and a pair of sensors.Figure 2 is a plan view of a filter carousel shown in Fig. 1.Figure 3 is another exploded perspective view of the dual-sensor multispectral imager shown in Fig. 1 with structural support components and drive components added.Figure 4 is an assembled perspective view corresponding to the exploded perspective view shown in Fig. 3.Figure 5 shows a block diagram depicting an example of steps for generating a registered set of spectral stereo images for imaging data acquired by the dual-sensor multispectral imager.Figure 6 shows a block diagram depicting an example of steps for executing the intra-band registration block shown in Fig. 5.Figure 7 shows a block diagram depicting an example of steps for executing the selection of a region of interest block shown in Fig. 5.Figure 8 shows a block diagram depicting an example of steps for executing the extra-band registration block shown in Fig. 5.Figure 9 shows a plan view of (Fig. 9A) a variant filter carousel providing multiple quadruple clusters of filters and (Fig. 9B) alignment of a single quadruple cluster of filters with a corresponding quadruple group of sensors.Figure 10 shows a plan view of (Fig. 10A) a filter carousel as shown in Fig. 1 providing multiple pairs of filters and (Fig. 10B) alignment of a single pair of filters with a corresponding pair of sensors.Figure 11 shows a side view and a front view of an outer casing for mounting and support of the imager components shown in Fig. 1.Figure 12 shows geometrical lines and parameters relating to filters positioned in two rows in a filter carousel presented in Experimental Example 2.Figure 13 shows filter configurations for compact filter carousel design for various values of wavebands / filters per imager n, first and second filter sets distinguished by horizontal striping compared to vertical striping.Figure 14 shows filter configurations when the closeness of the imagers is the optimization objective for various number of wavebands / filters per imager n, first and second filter sets distinguished by horizontal striping compared to vertical striping.Figure 15 shows plots of imager separation versus number of wavebands / filters per imager when the optimization is minimizing imager separation (-o) and when minimizing carousel diameter (--□).Figure 16 shows plots of carousel diameter versus number of wavebands / filters per imager when the optimization is minimizing imager separation (-o) and when minimizing carousel diameter (— □).Figure 17 shows plots of construction angle q versus number of wavebands / filters per imager when the optimization is minimizing imager separation (-o) and when minimizing carousel diameter (— □).Figure 18 shows a schematic overview of a method of registration of image data acquired using dual heterogenous stereo bands as presented in Experimental Example 3.Figure 19 shows a plot of transmission spectrum of Makrolon shade 5 based on ANSIZ87.1 test result.Figure 20 shows a plot of depth resolution for a dual sensor multispectral imager.Figure 21 shows a schematic diagram depicting homography of stereo images.Figure 22 shows a plot of the focusing psf function obtained using SWIR image at lOOOnm waveband using corresponding 1250nm waveband as reference.Figure 23 shows a result of deblurring and denoising a lOOOnm SWIR image of an imaging target, left side image is original, while right side image is deblurred and denoised.Figure 24 shows a 3D scene reconstruction using VIS-NIR and SWIR image at lOOOnm waveband based on the point cloud data.Figure 25 shows rectified and intra-band registered images of lOOOnm waveband from both VIS-NIR and SWIR sensor.Figure 26 shows images of a hole in the image pattern of the SWIR sensor at lOOOnm waveband, left side image is original, while right side image is after registration based on a disparity map.Figure 27 shows the lOOOnm waveband registered image of the imaging target, based on the imaging target.Figure 28 shows the registration details of the lOOOnm images of VIS-NIR and SWIR images, based on the imaging target.Figure 29 shows the registration details of images of 400nm and 1500nm wavebands.Figure 30 shows rectified and intra-band registered images of lOOOnm waveband from both VIS-NIR and SWIR sensor showing the analog samples in the measuring stand.Figure 31 shows the 3D Scene reconstruction of the analog sample in the measuring stand based on point cloud data.Figure 32 shows the registered 400nm and 1550 nm wavebands of the analog samples on the measuring stand.Figure 33 shows the ideal benchmark image used in a simulation in Experimental Example 4.Figure 34 shows the normalized filter response at various selected overlaps.Figure 35 shows the practical benchmark image in Experimental Example 4.Figure 36 shows a general piecewise linear spectral response.Figure 37 shows plots of spectral response conditions and simulated images for Table 12.Figure 38 shows a multi-sensor filter carousel geometry in Experimental Example 5.Figure 39 shows a typical filter pairing for a quad-sensor filter carousel.Figure 40 shows a quad-sensor filter carousel, image sensors and lenses configuration.Figure 41 shows a triple-sensor filter carousel with triple filter rows in Experimental Example 5. Fig. 41A shows circular filter shapes. Fig. 41B shows rectangular filter shapes.Figure 42 shows a typical filter pairing for triple-sensor filter carousel.Figure 43 shows a triple-sensor filter carousel, image sensors and lenses configuration.Figure 44 shows a filter pair central angle shift when number of filters in each row is even.Figure 45 shows a filter pair central angle shift when number of filters in each row is odd.Figure 46 shows kAth row frames arranged sequentially from lower band of -m_k to higher band of n_k in a registration process in Experimental Example 7.Figure 47 shows the kAth row extra-band registration process.Figure 48 shows a high-level registration process of multiple row filter wheel images.Figure 49 shows a schematic of an example of stereo imaging in Experimental Example 8.Figure 50 shows a schematic of an example of a general intermingled lattice filter wheel in Experimental Example 9.DETAILED DESCRIPTION OF PREFERRED EMBODIMENTSWith reference to the drawings , a multi-sensor multi spectral imager 10 is described. The multispectral imager 10 comprises a first lens 20 and a second lens 22, a first imager 30 and a second imager 32, and a filter carousel 40. The first and second imagers (30, 32) are first and second sensors or detectors of any suitable electromagnetic radiation transmitted by the first and second lenses, respectively, and therefore the terms imager, sensor and detector are used interchangeably. The first lens 20 is aligned with the first imager 30 providing a communicative first beam path of electromagnetic radiation directed by the first lens 20 toward the first imager 30. The second lens 22 is aligned with the second imager 32 providing a communicative second beam path of electromagnetic radiation directed by the second lens 22 toward the second imager 32.The filter carousel 40 is configured to simultaneously position a first selected filter 50 in the first beam path and a second selected filter 52 in the second beam path.The filter carousel 40 is formed as a disc shaped planar body 42 bound by a circular perimeter 44. The planar body 42 has a first side and a second side, the first side of the planar body 42 is configured to face back flanges or back planes of the first lens 20 and the second lens 22, and the second side of the planar body is configured to face image sensor planes of the first imager 30and the second imager 32. Each of the first and second lenses has a front plane and a back plane. The front plane is exposed to an interrogated scene when the first and second lenses are respectively extended through corresponding first and second openings formed in a wall of an outer container or outer casing 70 of a multispectral imager such as illustrated in Fig. 11. It is the most visible (and vulnerable) element. The front plane is the first to receive light entering the first and second lenses from the interrogated scene. The back plane of the first and second lenses is closest to the filter carousel and to the first and second imagers.A plurality of slots 46 or apertures are formed through the planar body 42, and each slot 46 is sized and shaped to mount a filter. Any suitable mounting component such as a grommet, gasket, sleeve, bushing, flange, groove, and the like may be incorporated within slots 46 or perimeters forming slots 46 to bolster mounting of filters. Furthermore, the slots 46 and filters mounted therein, are positioned within the planar body for simultaneous operation, more specifically simultaneous insertion within the first and second beam paths. The first selected filter 50 and the second selected filter 52 together form a first pair of associated filters among the plurality of filters mounted in the filter carousel 40 that are sized, shaped and positioned for simultaneous insertion as each selected filter pair within the first beam path and the second beam path. A rotational center 48 formed in the filter carousel 40 rotationally mounted to a shaft, rod, cylinder, and the like supports rotational motion of the filter carousel relative to the first and second lenses which are stationary and the first and second imagers which are also stationary. The rotational motion of the filter carousel provides rotational motion from a first rotary position having a first pair of filters simultaneously inserted within first and second beam paths to a second rotary position having a second pair of filters inserted within the first and second beam paths. In this manner, rotary motion of the filter carousel provides an indexing function to coordinate simultaneous positioning of filter pairs in alignment with first and second beam paths, respectively.In a typical operation, the first side of the planar body is configured to face the first lens 20 and the second lens 22, the second side of the planar body is configured to face the first imager 30 and the second imager 32, the first lens aligned with the first imager, the second lens aligned with the second imager, and in the first rotary position of the filter carousel 40 the first selected filter 50 and the second selected filter 52 are positioned for simultaneous alignment with the first lens and the second lens, respectively, and a first observation mode is configured by simultaneous occurrence of a first beam path passing through the first selected filter and a second beam path passing through the second selected filter. As the filter carousel is rotated (counter-clockwise in the example shown inFig. 2) from the first rotary position to the second rotary position, a third selected filter 60 of the first plurality of filters and a fourth selected filter 62 of the second plurality of filters are positioned for simultaneous alignment with the first lens and the second lens, respectively, a second observation mode configured by simultaneous occurrence of the first beam path passing through the third selected filter and the second beam path passing through the fourth selected filter. The third and fourth selected filters (60, 62) are so labeled for ease of illustration and distinction from the first and second selected filters. But, stated equivalently, the third and fourth selected filters could be considered as first and second selected filters of the second pair of filters.With each new rotational position of the filter carousel a further associated pair of selected filters is positioned within first and second beam paths, respectively. Fig. 2 shows each associated pair of selected filters labeled with the same letter for ease of illustration, with the second selected filter indicated by the primed counterpart, while the first selected filter is indicated by the corresponding letter without a prime.Comparing the counterpart letter labels with and without prime, makes evident that a plurality of filter pairs are mounted in the planar body, each of the plurality of filter pairs comprising the first selected filter of a first plurality of filters positioned at a first radial direction from the rotational center 48 and the second selected filter of a second plurality of filters positioned at a second radial direction from the rotational center, and the first radial direction is not parallel to the second radial direction.Furthermore, each of the first plurality of filters has a first filter center positioned at a common first radial distance from the rotational center; or in other words, each of the first plurality of filters is the same radial distance from the rotational center and positioned on a first ring or first circle 54 within the filter carousel. Similarly, the second plurality of filters are disposed in the planar body, each of the second plurality of filters having a second filter center positioned at a common second radial distance from the rotational center; or in other words, each of the second plurality of filters is the same radial distance from the rotational center and positioned on a second ring or second circle 56 within the filter carousel. The common first radial distance is lesser than the common second radial distance, so that the first and second circles are not coincident, but are rather concentric and co-axial having a common rotational center. All of the first plurality of filters positioned on the first circle are matched to the first lens and the first imager, while all of the second plurality of filters positioned on the second circle are matched to the second lens and the second imager.As shown in Figs. 1 and 2 the first circle 54 has a shorter radius, than the second circle 56, and typically the first and second plurality of filters are not constrained to any one of the first and second circle, provided that all of the first plurality of filters are on one circle, while all of the second plurality of filters are on another circle. However, in certain examples filters of larger surface area or diameter are mounted on a circle having a greater radius so as to match the relatively larger filters with the relatively larger surface area of the larger circle.Fig. 3 shows an exploded perspective view of the multi-sensor multispectral imager shown in Fig. 1 with structural support components and drive components added. The structural support component is a carousel frame formed of a first side or a lens side 70 and a second side or imager side 72. The first side 70 provides openings for mounting first and second lenses (20, 22), while the second side 72 provides openings for mounting first and second imagers (30, 32); and the openings are positioned for predetermined alignment of the first lens with the first imager and alignment of the second lens with the second imager. The second side 72 provides a shaft or cylinder extending interiorly and rotationally engaging the filter carousel 40. More specifically, a ball bearing is disposed in the rotational center of the filter carousel, the ball bearing defining a rotational aperture for receiving a shaft or cylinder of the carousel. The filter carousel and the ball bearing are fixed to each other so that they do not rotate relative to each other, while both ball bearing and filter carousel rotate in relation to the shaft or cylinder. The position of the shaft and therefore the positioning of the rotational center of the filter carousel in the carousel frame is offset from a center point between the two imagers or offset from a center point between the two lenses - i.e., the rotational center of the filter carousel is offset from a linear line intersecting and extending between the central axes of the first imager and the second imager, or offset from a linear line intersecting and extending between the central axes of the first lens and the second lens. Stated analogously in terms of three dimensional space, the rotational center of the filter carousel is offset from a flat plane intersecting and parallel to the central axes of the first imager and the second imager, or offset from a flat plane intersecting and parallel to the central axes of the first lens and the second lens.The drive component shown in Fig. 3 is a motor 80 mounted in an opening 82 in the second side 72 of the carousel frame and positioned circumferentially with respect to the filter carousel 40. The motor 80 rotates a spindle with an attached drum 84 which frictionally engages perimeter 44 of filter carousel 40 and drives rotation of filter carousel 40 from a first rotary position to a second rotary position. The motor 80 may be biased against the perimeter 40 by one or more resilient members to enhance a frictional engagement to transfer torque of the motor to the filter carousel.Electronic circuitry for supplying power and control signals to the motor 80 may also be mounted in or to the carousel frame. The first side 70 and the second side 72 are attached to maintain orientation and positioning of all the supported components such as the first and second lenses, the first and second imagers, the filter carousel and the motor.Figs. 5-8 show block diagrams depicting an example of a computer-implemented method for registration of spectral stereo images. In some examples, the registration method may be used to process image data acquired by an imaging system incorporating the filter carousel 40, but in other examples implementation with other imaging systems, independent of the filter carousel, is contemplated. As shown in Fig.5 the method 100 can be subdivided into a sequence of steps comprising initial processing 110 or pre-processing in advance of initiating a registration step, followed by intra-band registration 120, followed by a selection of a region of interest 140, followed by processing of image data in the selected region of interest to determine an extra-band registration transformation and then applying the extra-band registration 160. The essential blocks of the method 100 are the first-level intra-band registration 120, selecting region of interest 140, and the second- level extra-band registration 160 and examples of these are depicted in block diagrams shown in Fig. 6, Fig. 7, and Fig. 8, respectively.Typically, the intra-band registration block 120 is initiated by obtaining a first plurality of images and a second plurality of images 122; the first plurality of images 122 spanning a first spectral range acquired by a first image sensor, each of the first plurality of images acquired in a different wavelength band; the second plurality of images 122 spanning a second spectral range acquired by a second image sensor, each of the second plurality of images acquired in a different wavelength band; the first spectral range different than the second spectral range with at least one common wavelength band; and the first image sensor and the second image sensor are positioned relative to each other in a configuration for stereo image acquisition. Prior to applying an intra-band registration transformation, first and second reference images with a desired property are selected 124 in any convenient order (i.e., the numerical labels of first and second do not imply a chronological order and the first reference image may be selected before, simultaneously or after the second reference image) may occur, and include, for example: selecting the first reference image from the first plurality of images, the first reference image having a better focus quality than a majority of the first plurality of images; and selecting a second reference image from the second plurality of images, the second reference image having a better focus quality than a majority of the second plurality of images. As first and second reference images are selected 124, first and secondregistration transformations (126A, 126B) are applied in any convenient order independently of each other, and include, for example: applying a first registration transformation 126A to register the first plurality of images with the first reference image to generate a first plurality of registered images 128; and applying a second registration transformation 126B to register the second plurality of images with the second reference image to generate a second plurality of registered images 128. In some examples, the first registration transformation is a similarity transformation, and the second registration transformation is a similarity transformation.The selection of first and second reference images can accommodate various approaches, but will typically involve selecting images that have a desired quality of focus. For example, the first reference image may be a best focused image within the first plurality of images, and the second reference image may be a best focused image within the second plurality of images. In another example, the first reference image is acquired with a wavelength band associated with a best focus metric among a first plurality of wavelength bands within the first spectral range, and the second reference image acquired with a wavelength band associated with a best focus metric among a second plurality of wavelength bands within the second spectral range. In another example, the first reference image is acquired with a wavelength band closest to a median wavelength of the first spectral range, and the second reference image is acquired with a wavelength band closest to a median wavelength of the second spectral range.The purpose of selecting a region of interest 140, is to identify a region of interest that commonly occurs throughout the image data and with the region of interest corresponding to image data that captures a relatively flat or planar surface among imaged surfaces of an imaged object, so that the image data relating to the relatively flat or planar surface can be compared in two or more images to determine / calculate an extra-band registration transformation (for example, a third registration transformation shown in Fig. 8). The region of interest may be selected by comparing any one or more of the first plurality of registered images with any one or more of the second plurality of registered images. For example, the region of interest may be selected by comparing one of the first plurality of registered images with one of the second plurality of registered images that were both acquired using filters that have a common wavelength band (for example, same or similar wavelength bands). In another example, the region of interest may be selected by comparing one of the first plurality of registered images with one of the second plurality of registered images that are both characterized by higher quality of focus compared to their respective first and seconds pluralities of images.In an example, the selecting a region of interest 140 is initiated by obtaining first and second sets / pluralities of registered images 142 that were generated in step 128. Third and fourth reference images are selected 144 in any convenient order, including for example: selecting a third reference image from the first plurality of registered images, and selecting a fourth reference image from the second plurality of registered images, both the third reference image and the fourth reference image acquired in the common wavelength band. Image rectification 146 can be employed to facilitate comparison of the third and fourth reference images, including for example: rectifying the third reference image with the fourth reference image to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal, and using the rectified image data to generate a 3D scene reconstruction 148. Analysis of the 3D scene reconstruction allows for identification of a suitable region of interest, including for example: analyzing the 3D scene reconstruction to select a common region of interest in the third reference image and the fourth reference image for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object. In other examples, other images (i.e., other than the third or fourth reference images, such as for example the first and second reference images or even fifth or sixth reference images) may be selected from the first and second sets of registered images, including for example: rectifying a fifth reference image selected from the first plurality of registered images with a sixth reference image selected from the second plurality of registered images to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal, and using the rectified image data to generate a 3D scene reconstruction; and analyzing the 3D scene reconstruction to select a common region of interest in the fifth reference image and the sixth reference image for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object.The extra-band registration 160 uses the region of interest image data in images selected from the first and second sets / pluralities of registered images to determine the extra-band registration transformation. In one example, the extra-band registration is initiated by selecting third and fourth reference images 162, including for example: selecting a third reference image from the first plurality of registered images, and selecting a fourth reference image from the second plurality of registered images, both the third reference image and the fourth reference image acquired in the at least one common wavelength band. The region of interest is identified 164 in the third and fourth reference images and the image data captured in the respective region of interest in the third andfourth reference images is compared, including for example: determining 166 a third registration transformation (i.e., an extra-band registration transformation) based on comparing the third reference image with the fourth reference image. Following determination / calculation of the third registration transformation, it is applied to the first set of intra-band registered mages or the second set of intra-band registered images as desired, to register the first set of intra-band registered images against the second set, or vice versa as desired, including for example: applying the third registration transformation to the first plurality of registered images. In some examples, the third registration transformation comprises correction of image center shift, scaling to correct different lens magnifications, and / or imager rotation correction. In further examples, the third registration transformation comprises a rigid transformation.The common wavelength band can be at least one common wavelength band. In some examples, the at least one common wavelength band is multiple wavelength bands. In other examples, the at least one common wavelength band is a single wavelength band.In various examples, initial processing 110 or pre-processing of acquired image data can employ a variety of approaches so as to facilitate comparison of the first plurality of images with the second plurality of image. One example includes identifying a common field of view in the first plurality of images and the second plurality of images and cropping the first plurality of images and the second plurality of images to the common field of view; the cropping may be symmetric and result in a similar or identical common field of view for the first plurality of images and the second plurality of images. In another example, the first plurality of images and the second plurality of images are corrected for chromatic aberrations; in a related example chromatic aberrations can be corrected by deblurring, and the first plurality of images are corrected by deblurring based on the first reference image, and the second plurality of images are corrected by deblurring based on the second reference image. In another example, the first plurality of images and the second plurality of images are corrected for geometrical aberrations.Initial processing 110 or pre-processing facilitates use of image data acquired from first and second imagers that differ in image resolution. For example, the first plurality of images may have a greater pixel size than the second plurality of images. In another example, the first plurality of images may have a lesser pixel number per unit area than the second plurality of images. In certain examples, the first image sensor is a InGaAs-SWIR sensor and the second image sensor is a Silicon- VIS-NIR sensor.In an example, algorithms illustrated in Figs. 5-8 can be used to register multiple images from two heterogeneous image sets from two different points of view. In this example, a summary of the approach and preliminary assumptions are:1. There is one VIS-NIR imager and one SWIR imager.2. The VIS-NIR sensor is sensitive from about 400nm to lOOOnm. It is mounted at position A, has mi x ni pixels of size «i. It can provide images at Zi number of distinct optical bands using appropriate optical filter in the proposed filter wheel. The central waveband for this sensor is the waveband nearest to the center wavelength of the sensitivity at (1000+ 400) / 2 = lOOnmT. . , > , , ,r, . . . , v7 / . It uses a lens with a focal length of j\ which is brought in sharp focus at the central waveband.3. The SWIR sensor is sensitive from about 900nm to 1700nm. It is mounted at position B, has m2* n2pixels of size a2. It can provide images at l2number of distinct optical bands using appropriate optical filter in the proposed filter wheel. The central waveband for this sensor is the waveband nearest to the center wavelength of the sensitivity at (900+ 1700) / 2 = 1300 / ? / n . vz / . It uses a lens with a focal length of j2which has brough in sharp focus at the central waveband.4. There is a common waveband between two sets of wavebands for both sensors, say at lOOOnm from both sensors.5. The position of the system and the target during the image capturing process is fixed and not changing for all bands (h+h).In this example, the following special calibration procedure has been done for both imagers at all wavebands (Zi+Z2):1. Image correction and stereo calibration parameters obtained maintaining the criteria 5 of the previous paragraph.2. Images are magnified and cropped for ease processing and registration. A down sampling will be done at the end. The magnification should be done matching the pixel on the ground of two imagers.In a further example illustrating Figs. 5-8, pre-processing and intra-band registration for each set of bands corresponding to each imager can follow these steps:1. Deblurring the images of each band using the central waveband image (where the image is in the sharp focus) to mitigate the lens focal length variation with wavelength (chromatic aberration).2. Geometrical image correction for any radial and tangential distortion due to lens and filter.3. Using the central waveband image as the reference image and using scaling and translation (similarity type registration) to register images of other bands to the reference waveband image.The outcome of this example is two sets of intra-band registered images.In a still further example, the 3D scene reconstruction using two images, one from each set of intra-band registered images, is done. The selected images can be any of the set, preferably the images corresponding to central waveband, or the common waveband can be selected. Using the 3D reconstructed image, a flat region is selected for extra-band registration process. The flatness is gauged using the depth resolution of the imagers.In yet a further example, a registration of the two sets of intra-band registered images is done by extra-band registration of the common waveband in the selected regions that possess the property of being in same planes. The following steps are followed:1. Image center shift correction2. Scaling (due to the different lens magnification) correction.3. Camera rotation with respect to each other correction, and finally4. Rigid transformation.The currently disclosed multi -sensor spectral imaging device, system and method have been validated by experimental testing. Experimental testing results demonstrate the ability of the currently disclosed device, system and method to provide compact size for multi-sensor spectral imaging and reduce complexity in processing image data. The following experimental examples are for illustration purposes only and are not intended to be a limiting description.Experimental Exemplification: Experimental Example 1 (dual-sensor multispectral imager for a lunar rover).Geological characterization and mapping of the moon surface is getting more and more attention as lunar exploration programs accelerate in recent years. Plans to establish a permanent lunar base for human long-term stay are in development by many nations. These plans will require autonomous robotic navigation, surface mapping and characterization to support these lunaroutposts. A multispectral imager (MSI) provides information for fulfilling these tasks. Many extraterrestrial rover explorers have one or two multispectral imagers in the form of a filter carousel, a lens and a camera on board. The multispectral images provide sampled reflectance spectrum at the desired wavebands. The spectral converge of the imagers are usually from 400nm to lOOOnm that covers Visual to Near Infrared (VIS-NIR) range of spectrum.Two general approaches of multispectral imaging were considered. The first approach is a multiple wavebands multispectral imaging in VIS-NIR range of spectrum, as previously developed in unmanned aerial vehicle (UAV) mostly for agricultural applications. The instrument has five silicon sensors and five small lenses. While the instrument fulfils its objectives, the provided resolutions will not be sufficient for rover applications. Selecting better quality lenses causes the instrument mass to be prohibitive. Furthermore, accommodating multiple Short Wave Infra Red (SWIR) image sensors (typically the cooled InGaAs image sensors) was another issue of weight and power consumption. Therefore, a second approach of indexing multiple waveband filters for an imager was selected. The second approach in which single imager, single lens with a computer- controlled filter carousel is used has superior performance and much better weight and power budget ratio. This smaller ratio provides the possibility of using better imager and higher quality lenses. However, as the lens is brought to optimal focus at one waveband, the lens chromatic aberration should be corrected. The filter carousel MSI cannot capture multiple images simultaneously, which means the target and the instrument should be kept in place during image acquisition. The typical application for this type of MSI is on the rover, belt conveyor, satellites and anywhere else where the target and instrument can be kept stationary during image acquisition.For the current development of a multispectral imager (MSI), a range of 400nm-1700nm was desired. Selecting the suitable image sensor(s) for covering 400-1700nm range of spectrum was the first task of the design. The range spectrum covering VIS-NIR to SWIR is wide in nature. The idea of having a single InGaAs to cover full required spectrum has been considered, but it was discarded based on the facts that the technology is relatively new, and implementing it needs wideband lens and filters with extended rejection bands. Alternatively, the desired spectrum range can be divided into VIS-NIR (from 400- lOOOnm), and SWIR (from 1000-1700nm) with a separate imager for each band. The silicon CMOS sensor covering 400-1000nm was chosen due to its performance and its maturity. For covering 1000-1700nm of spectrum a conventional InGaAs sensor was selected. Due to the technological limitation of InGaAs fabrication technology and other inherent limitations, the pixel size and noise are larger than silicon CMOS image sensors. Higher noise of InGaAs sensor canbe mitigated by cooling down the imager. However, the lower InGaAs image resolution will be determinant in the overall cube image data quality (as the images from the silicon image sensor should be down sampled to match it).Multiple alternatives for aligning silicon CMOS and InGaAs sensors and their corresponding filters were considered as shown for example in Fig. 9 and Fig. 10. The alternative shown in Fig.10, which is consistent with the alternative shown in Fig. 1, was selected for further development. Fig. 11 shows a side view and a front view of an outer casing 70 for mounting and support of the imager components shown in Figs. 1-4 and associated electronic and computer controller components.The lens that has been selected for this version of the instrument is the prime lens with various aperture. The choice of the zoom lens was considered too, but was not selected for the sake of simplicity and mass consideration. The selected lens is the conventional C-mount fixed lens with the possibility of f# adjustment.The concept of a single filter carousel and dual-imagers has been illustrated in Figs. 1-4. As shown in Fig. 1, the optical filters are arranged in two different rows with dedicated sensor and lenses for each row. The image sensors and the lenses can be of any type based on the requirement. In our case, one of the sensors and the corresponding lenses are covering VIS NIR range of spectrum and the other sensor and lens covering SWIR range of spectrum. The selected sensors are identified in Table 1. The selected lenses are identified in Table 2. A comparison of the imager characteristics as relates to each of the two sensors is provided in Table 3. A technical summary of the multispectral imager is provided in Table 4.The physical size of the carousel is determined by many factors. For example, one factor is the number of wavelength bands needed. In our case, the carousel should accommodate at least 5 bands in VIS-NIR and 5 bands in SWIR range of spectrum. We have based our design on 7 bands in each of the VIS-NIR and SWIR ranges. The second factor is the size of the optical filters, which also relates to the size of the imagers.Table 1 : Selected sensor types.Table 2: Selected lens types.Table 3: Comparison of multi spectral imager features related to each sensor type.The dual-sensor MSI is a unique equipment in a sense that it makes the operation of two multispectral imagers in two different range of spectrum in a single platform. This was possible by combining a single carousel filter in alignment with two types of image sensors in a single MSI platform. This combination while compact provides stereo imaging too which can be used to generate 3D point cloud of the scene. While a high resolution and sensitive CMOS sensor provide ample capability in 400-1000nm range of spectrum, a low noise and sensitive InGaAs sensor provide modest resolution in 900-1700nm range of spectrum. Registering the images of these two sensors provides joint FOV of 115.2cm x 85.5cm and spatial resolution of better than 3mm at 3m distance. The mass of combined combined compoenets of the MSI is 1 ,7kg. Including the onboard computer, it consumes 12-17w. Table 4 summarizes the high-level dual-imager specifications.Table 4: Multispectral imager (MSI) technical features.Experimental Exemplification: Experimental Example 2 (filter carousel design for a lunar rover).An objective of the Multi -Spectral Imager (MSI) is the recognition of selected target materials from their signature reflectance. This task is usually done using a minimum of two wavebands and constructing pseudo monochromatic image based of the ratio of images amplitudes. Assigning three colors of red, green, and blue to three different ratios for three corresponding target materials, one pseudo RGB image is enough for recognizing three target materials. As the number of target materials increases, we need more and more wavebands. Table 5 summarizes representative MSI waveband numbers for mounting on a rover which has been developed for Mars exploration.Table 5: Selected summary of multispectral imagers for extraterritorial rover application [1],As seen from Table 5, the number of wavebands is five on the very early Viking Lander in 1976, but it is increased since after. Also, the imager type in all these instruments only covers VIS- NIR range of spectrum. Here we propose, dual imager MSI in which we have one conventional imager for VIS-NIR range of spectrum and the other one for SWIR range of spectrum. This task has been done on a single platform. In the following we present the geometrical design of the filter carousel for simultaneous alignment of filter pairs with corresponding dual imagers. Two optimizations with two different figures of merits of the overall compactness, and closeness of the imagers to each other are considered. The formulation can be used for any number of wavebands per imager with different geometrical parameters. Introducing the construction angle for the second-row, single row design would a special case of the general double row design. The design has been demonstrated through some realistic parameters.Fig. 12 shows the configuration of the optical filters in the carousel with two imagers, where we assumed the corresponding filters for each imager are arranged in a circle. We have two imagers and two circles of radius R1 and R2 and we assume R2 > R1 . For the sake of compactness, the filters corresponding to the second imager (which are arranged on the larger circle R2) are located between filters corresponding to the first imager (which are arranged on the smaller circle Rl) (seeFig. 12). Fig. 12 also shows a ball bearing and a corresponding mounting at the centre of the carousel.If we design the carousel for n number of filters per imager, then the angle difference between the filters in each row, will be:If the distance between the filters in the first row is / , it can be related to the small circle as: Ipl '1Fll $ i^’l I JFsin I "(^2)And the large circle radius Ri can also be expressed as:Where q is the second-row construction angle (see Fig. 12). When R2 = Rl, the double row configuration will be reduced to a single-row configuration. This special case can be achieved if we set q = cp / 4 = 90 / n. To ensure the second-row filters is not coming closer to the carousel centre than the first row’s, i.e., R2 > Rl it is enough to set q > <p / 4.The central angle between the matched filter pairs of the first row to the second row can be expressed as:Where m indicates which filter from the second row has been chosen for pairing with the selected first row’s (see Fig. 12, A(pi and A(p2 correspond to m = 1 and 2 respectively). It can be shown easily that due to the symmetry of the filter positions in the carousel, we only need to consider limited numbers of values for m as:The practical minimum of A(p will be at:However, the maximum will occur at:The imagers centre to centre separation can be expressed as:There are five restrictions that should be satisfied too as: a. Filter diameter and the housing margin should be smaller than 1, i.e.,Where Df is the filter diameter and gf is the required mounting gap between filters. The filter diameter should be decided considering the sensor sizes and the acceptable carousel dynamic rotational tolerances during operation. b. Filter diameter and the housing margin should be smaller than the filter distance from two rows, i.e.:c. The imager centre to centre distance h should be larger than the radiuses of the two optical lenses combined considering some margins, i.e., h > Dn / 2 + DI2 / 2 + 6(1 nWhere Dq and D[2 are lenses diameters and 8 is the gap between the lenses (margin). d. The imager centre to centre distance h should be larger than the combined distance of the imagers and the corresponding electronics and possible enclosures, i.e., h > W 5J1 / l + W 5 Z. / / 2 + <SWhere Wsiand Ws2 is the imagers electronics / enclosure sizes and 8 is a margin. e. As the carousel is pivoted to the center via a ball bearing, the imager near to the center (and the corresponding optical filter) should be kept away by a margin.Where R^j, is the ball bearing radius, g]qis the ball bearing housing width and 8 is a margin.Summarizing the restrictions (9) through (13), we have:Or using Eqs. (2)-(8), and expressing all restriction versus Rl, we have:The base carousel diameter can be expressed as:Two different optimizations with two different objectives have been performed. If the overall filter carousel compactness is the main objective, Eq. (16) can be minimized versus q and 1, for various values of m. The smallest value of m that minimize Eq. (16) is the desired answer. The minimization is constraint to Eqs. (15). The second objective is the minimization of the separation between imagers. In this case Eq. (8) should be minimized with the same constraints of Eqs. (15). Table 6 presents realistic dimensions that we can use to demonstrate these optimizations.Table 6: structural dimensions for optimization examples presented in Tables 7 and 8.These values are based on the available off the shelf components that we have used in our prototype.For compactness of the filter carousel, the minimization of Eq. (16) with the constraints of Eq. (15) has been done using Matlab fmincon. Our experience shows that the results are sensitive to the initial values for R1. To avoid local minimization, various initial values have been suggested and minimum of the results has been selected. Figure 13 shows the filters configuration in the filter carousel versus number of filters per imager n, when the compactness is the objective; first versus second sets of filters are distinguished by horizontal versus vertical stripes.Table 7 shows the minimum carousel diameter, the second-row construction angle q and imager to imager distance versus the number of wavebands / filters per imager n. As you can see from Table 7 for n = 3,4,5 , the single-row configuration provides the smallest filter carousel, however for n > 5 , double row configuration provides the smallest filter carousel.Table 7: the minimum carousel diameter, second row construction angle q, and the imager- to-imager separation versus n.The filter carousel with minimum imager separation is desirable when image registration is a concern. Image registration becomes a concern when the target is near to the imagers and the focal lengths of the imager lenses are small. In this case, the smallest value of m = 1 should be selected and the minimization goal of Eq.(8) with constraints of (15) will do the desired optimization.Figure 14 shows the filters configuration versus number of wavebands / filters per imager when the closeness of the imagers is the optimization goal; first versus second sets of filters are distinguished by horizontal versus vertical stripes.Table 8 shows the carousel diameter, the second-row construction angle q and imager to imager distance versus the number of wavebands / filters per imager n. As can be seen from Table 8, single row configuration is the optimum result only for n = 3, while for n > 3 double-row configuration is the answer.Table 8: the carousel diameter, second row construction angle q, and the minimum imager- to-imager separation versus n.Fig. 15 shows the imager separation versus n (number of wavebands / filters per imager) for both optimization cases. As can be seen from Fig. 15, the imager separation when it is minimized approaches the minimum amount that is physically attainable by the imager sizes (in this case 55mm, see Table 6), however when compact carousel is the optimization goal, the minimum separation is always higher than the other case (except for n = 3,4 where they are equal).Fig. 16 shows the carousel diameter versus n (number of wavebands / filters per imager) for both optimization cases. The carousel diameter when minimized is always lower than the other case where minimum imager separation is the optimization goal (except for n = 3 where they are equal).Fig. 17 shows the construction angle variation versus n (number of wavebands / filters per imager) for both optimization cases. As the number of filters increases, where the other constraints of (Eq. 15) will become inactive except the imager separations, the construction angle q approaches 60° when the carousel compactness is the goal. In this way, the filters separation in each row and between rows remains at its minimum which keeps the overall carousel size to its minimum (see Fig. 16). This will increase the imager separation larger than its minimum (see Fig. 15). However, when the optimization goal is the imager separation, this task has been done by increasing the first-row radius R1 (which increases the gap between filters beyond its minimum) and brings the second row nearer by decreasing the construction angle q below 60°. This procedure keeps the imagers separation at its minimum (see Fig. 15) but sacrifices the total carousel radius (see Fig. 16). This conclusion is also clear by comparing Fig. 13 and Fig. 14 for larger n.While this determination is calculated for the dimensions mentioned in Table 6, Figs. 13-17 provide a basis for a consistent advantage of the double row configuration over the single row configuration as the number of wavebands / filters per imager increases, particularly in relation to minimization of carousel size or imager separation.A systematic approach for designing a double row filter carousel has been presented, which can be used when multispectral imaging with dual imagers is desirable. A general formulation and an optimization approach provide either minimum carousel diameter or the minimum closeness of the imagers. The results based on some realistic parameters indicates that single row filter configuration can be advantageous when the number of wavebands / filters per imagers are low (less than 4). The double row filter carousel configuration is advantageous for higher count (greater than equal to 4) of wavebands / filters numbers per imager.Experimental Exemplification: Example 2 (Reference List).[1] M. Gunn and C. R. Cousins, "Mars surface context cameras past, present, and future," Earth and Space Science, vol. 3, no. 4, pp. 144-162, 2016.[2] F. O. Huck, E. E. Burcher, D. J. Jobson, and S. D. Wall, "Prediction of Viking lander camera image quality," 1976.[3] R. B. Tucker, "Viking lander imaging investigation-Picture catalog of primary mission. Experiment data record," NASA Reference Publication, vol. 1007, 1978.[4] P. H. Smith et al., "The imager for Mars Pathfinder experiment," Journal of Geophysical Research: Planets, vol. 102, no. E2, pp. 4003-4025, 1997.[5] J. F. Bell IH et al., "Mars exploration rover Athena panoramic camera (Pancam) investigation," Journal of Geophysical Research: Planets, vol. 108, no. E12, 2003.[6] J. F. Bell IE et al., "The Mars Science Laboratory Curiosity rover Mastcam instruments: Preflight and in-flight calibration, validation, and data archiving," Earth and Space Science, vol. 4, no. 7, pp. 396-452, 2017.[7] J. Bell et al., "The Mars 2020 perseverance rover mast camera zoom (Mastcam-Z) multispectral, stereoscopic imaging investigation," Space science reviews, vol. 217, no. 1, pp. 1-40, 2021.[8] J.-F. Yang et al., "Design and Ground Verification for Multispectral Camera on the Mars Tianwen-1 Rover," Space Science Reviews, vol. 218, no. 3, pp. 1-27, 2022.[9] A. Coates et al., "The PanCam instrument for the ExoMars rover," Astrobiology, vol. 17, no. 6-7, pp. 511-541, 2017.

[0010] A. Coates et al., "PanCam on the ExoMars 2018 Rover: A stereo, multispectral and highresolution camera system to investigate the surface of Mars," 2015.

[0011] A. Jones et al., "The Emirates Exploration Imager (EXI) instrument on the Emirates Mars mission (EMM) hope mission," Space Science Reviews, vol. 217, pp. 1-56, 2021.Experimental Exemplification: Experimental Example 3 (development of registration procedure for heterogenous dual sensor multispectral imager).Experimental Example 3 presents an example of a registration procedure that can be used to process imaging data acquired by the heterogenous Dual Sensor Multi -Spectral Imager (DS-MSI) covering the spectral range from 400nm to 1700nm, as described in Experimental Examples 1 and 2. It contains dual image sensors of different categories, one silicon and another InGaAs image sensor. The DS-MSI is designed for mounting on a mini rover with the tilt / span capabilities. The viewing range is from 2.5m to infinity with combined resolution of about 2.5mm at the target distance of 2.5m.The selected spectral range was divided into two spectral ranges. The VIS-NIR band ranges from 400 to lOOOnm (using a conventional silicon image sensor) and the SWIR band ranges from 900-1700nm (using a conventional TEC cooled InGaAs image sensor). The basic specification of DS-MSI is depicted in Table 3 presented in Experimental Example 1.The selected wavebands focusing on the characterization of the compositional diversity of the lunar surface are summarized in Table 9.Table 9: selected waveband specifications for two spectral ranges in the DS-MSI.The end product of the imaging process is 11 -waveband image data cube (not 12 wavebands because one channel is common between two sensors).An image registration task can be described as finding a spatial transformation that provides pixel to pixel correspondence between two images. This transformation in its nutshell provides acorresponding map between the fixed image (with a regular mesh) and the moving image (with irregular mesh). This mapping can be expressed as the displacement field (x and y disparity matrix). The transformations that generate such a displacement filed can be categorized as the linear / global and the nonlinear / local registration processes. The linear / global registration techniques seek the transformation based on maximizing a similarity function over the whole area of the images. On the other hand, the nonlinear / local transformation locally maximizes the similarity function. The most important class of transformations is the linear / global transformation, and it is well known and is a well-behaved transformation, but it is strictly valid in limited cases. The image registration can be feature-based matching or intensity-based matching. The featured-based image registration is usually very stable and fast but requires reliable feature extraction and correspondence to overcome missing features. The intensity-based image registration uses full image information, and it does registration by maximizing a similarity function. If the images come from imagers with a same image sensor with a same setting (e.g., same lens and optical filter), monomodality image registration is a proper technique, however, when the images come from different image sensor or even the same sensor with different optical filter or lens, multimodality image registration should be employed

[0017] . In our case, the multimodality image registration has been adapted.When the object is evolving between the first and second imaging times (many medical imaging techniques rely on this type of imaging), or the object / camera positioning changes, we need to apply nonlinear / local transformation. The case of stereo image registration can be categorized as a moving object. For evolving images, diffeomorphic image registration is a suitable technique. This is a class of transformation which provides a smooth and invertible map. Recently it has been applied to the stereo registration too

[0018] . While this type of transformation has many advantages in medical imaging [19, 20], it may fail in stereo image registration where there is a sudden fracture in object topology (which need discontinuous displacement field map). The local optimization of similarity function can also lead to unwanted image distortion (image wrapping) in parts of the image which are relatively featureless.Image magnification and cropping. Images are magnified and cropped for easing the processing and registration. A down sampling will be done at the end. Different magnification should be applied to match the pixel size on the ground of two imagers. For the imager number one, we have:Where d is the distance between imager and the target and mi is an arbitrary magnification factor, ai and fi are the pixel size and lens focal length of the imager number one. Please note that mi<l corresponds to down sampling, similarly:So having the same pixel sizes on the ground, we should have:As an example, consider the case in which: cii = 5.86pm , a2= 15pm and fi =f2= 25mm (see Table 3), if we assume mj= 1 , and m2according to Eq. (19) will be in. = 1 ;7.1 25.86 (20)This oversampling of lower resolution imager will be used for the registration process only.The final results will be available by down sampling both sets of images by mr= m2= 1 / 2.56The image cropping picks the common Field Of View (FOV) of two imagers regardless of the resolution. The cropping will happen symmetrically around the image centers. The FOV of imagers can be express as:The cropped image size normalized versus d will be:And image sizes in pixels will be:Consider an example of Table 3 in which:Using Eq. (22), the normalized cropped image size will be:H = (640 xl5pm) 125mm = 384 and V = (1216 x 5.86pm) / 25mm = 285.0304, so the cropped image size in pixels will be:Where the number of pixels is rounded to the nearest integer.Registration scheme overview. As the imagers in the DS-MSI have a different point of view, the imaging registration process is stereo in nature. Achieving pixel to pixel registration accuracy is important for geological characterization tasks. Such an accuracy should ideally be achieved by automatic registration process; however, any registration process should consider the fact that the equipment will be mounted on a rover on a tilt and span stage which generally is not ideal for pixel level stabilization during the imaging time. Such an uncertainty should be considered when we design our registration process.The adapted registration technique in this paper is depicted in Fig. 18. The registration task is divided into two major tasks. The first task is registering images between images from a same sensor (intra-band registration) and then registering images between different imagers (extra-band registration). The intra-band registration leads to sub data cube for each imager in the dedicated wavebands. The images will be corrected for radial and tangential distortion and deblurred versus the mid- waveband (where the corresponding lens has been fully focused). The registration process may be adapted to take care of optical filter misalignment in the filter wheel and any small equipment movement (for example, small movement of the rover) during the imaging time.Stereo calibration will be performed on the mid-wavebands between images from different sensors as indicated in Fig. 18. The stereo extrinsic parameters will be used to rectify images and 3Dpoint cloud of the scene will be generated. Finally using the common wavebands of the imagers, extra-band registration transformation will be obtained and will be applied to the other wavebands.Intra-band registration. The image registration among images that belongs to each sensor is from the same point of view (intra-band registration) and has been done in following steps: a. The images distortion will be corrected for each waveband individually. This includes, the radial and the tangential distortion. It is important to repeat the calibration process for each band individually to mitigate the chromatic distortion of the lens and taking the optical filters misalignment in the filter wheel into account. b. The images will be deblurred. The blur is due to fact that the prime lens has been used which is been fully focused only at one waveband. The image of the other wavebands will suffer from defocusing as the focal length of the lens varies by wavelength. We are using the fully focused image at the central waveband as a reference to estimate the deblurring kernel. c. The similarity multimodal registration technique will be employed to register all the images from all wavebands versus the image from the central waveband.The output of this process is the sub image data cube for each sensor type.Geometrical distortion correction. Analyzing the multiple images of a regular grid with known geometry is a common technique for radial and tangential distortion correction. Conventional checkerboard imaging target is only suitable for visual imaging, but it loses its contrast at the NIR and SWIR range of spectrum. The imaging target that we have used is made of Makrolon® IR shade 5

[0021] , As you can see from Fig. 19, the transmittance of this sheet in the entire range of our interest is less than about 2%.Our imaging target is made of 40.6 x 40.6 cm2of Makrolon IR Sade 5 with a thickness of 1.12mm with a hexagonal lattice of holes with diameter of 2cm and pitch of 5cm has been machined into it.Image deblurring. As prime lenses have been selected in DS-MSI, the chromatic aberration of the lenes that causes the focal length variation with wavelength causes issues as the lens cannot be fully focused on all wavebands. As we bring the lenses to full focus at the center wavebands of each sensor, the side wavebands (the wavebands further way from the central wavebands) are less focused. Any other lens response degradation with wavelength will also affect the image quality at the other wavebands. Here we explain how these deficiencies can be mitigated using Wiener deconvolution technique. Our approach has two steps, first we try to find the best deblurring PointSpread Function (PSF) based on the focused image at the mid wavebands. Then we try to deblur and remove some noise from images using the best PSF of previous step.Using general images of the focused mid waveband and use it for the focusing PSF estimation of other wavebands can be a challenging task as the reflectance of objects in the images can varies dramatically with wavebands. To overcome this issue, we use the images of the imaging target of previous case (which have a relatively flat reflectance at the desired wavebands, see Fig. 19). For the PSF estimation, we also cropped the images to areas of the imaging target.Consider that the i0is the recorded part of the image containing the imaging target which suffered from blurriness and noise. Also consider that if is the focused image of the same target with some noise too, then we can assume that following equation holds:Where An is the added noise to the observed image, psf(x) is the PSF of the lens that at the observed waveband and encircled X is convolution operator.is the image shift due to misalignment of the optical filter between wavebands. Including noise in Eq. (24) is important factor for robustness of the Weiner optimization process. In Fourier space, we have:Then the best PSF based on the minimizing the mean square error can be obtained according to Weiner theory as

[0022] :Considering the added noise having white noise spectra, the above equation can be simplifiedIn our case, we adjust the value of K to obtain the best robust result. The psf can be obtained by transferring back PSF to the spatial domain. As expected, the obtained psf is shifted by xoand yo,however, as this shift is not important at this stage, the obtained / ^ / function will be centered around origin to be best matched to the following asymmetric 2D Gaussian PSF:where X is the covariance matrix. The matching is based on the least square approximation.Using the best match PSF of Eq. (28), the observed image can be corrected for the blurriness with some noise be removed too using Wiener deconvolution method

[0023] .Intra-band image registration. The images from each waveband after deblurring should be ready for the registration process. However, we should decide what type of registration suits our application. As the images come from different wavebands, they will have spectral information of the target which leads to different contrast of the same object. This characteristic leads to multimodal or phase based registration techniques

[0024] . Furthermore, as images have been taken by inserting optical filter between lens and the sensor, some geometrical distortion and image misalignment may arise. Characterizing the distortion and misalignment type will affect the image registration procedure that we need in this stage. The registration procedure should also take into consideration the uncertainty of equipment position during the imaging procedure.In order to determine which type of registration should be adapted, let us, as it is commonly accepted, adapt a simple pinhole camera model and then add lens and filter distortion to it [25-28],A 3D pointin the word coordinate system can be related to the camera coordinate system with origin at the pinhole location by extrinsic parameters of the camera, i.e., translation and rotation vectors T and R as:Projecting the 3D camera point onto the normalized coordinate system x„ by:Including distortion elements to the normalized coordinates we have:We have included three different types of distortion in our model as:Where radial lend distortioncan be expressed as [26, 28]:where rnis the distance from undistorted image point to the image center, i.e.,The tangential component of the lens distortion ressed as [26, 28]:The distortion due to the filtershas two components. In the paraxial regime, the presence of optical filter causes a focus shift by:Where t is the thickness of the filter and «2 is the effective refractive index of the filter. This is usually will be considered when the camera is brought to focus on the presence of the filter. However, this can be done perfectly for one filter only as the effective refractive index of the filters that are present in the carousel frame are different. This difference is the first source of the filter of distortion. The second source is due to unavoidable misalignment of the filters in the carousel frame. This distortion can be expressed as a global translation. In total this distortion can be expressed as[29, 30]:Then the actual pixel coordinate can be expressed as:Where f is the lens focal length, 8 is the pixel size and image center. Eqs. (38) and (39) can be expressed differently as:As can be seen from Eq. (41), the effect of optical filter distortion is the modifying the focal length and changing of the image center. While the scaling factor changes can be due to the dispersion of the lens system too, the image center variation is unique to the filter distortion. As we use the images at the pixel coordinate system, there is no meaningful way to extract the filter distortion parameters. However, this distortion explains why calibrated images from different filters needs extra registration that includes scaling and translation (similarity type registration)

[0031] .Extra-band registration. The image registration from different viewpoints (extra-band registration) are indeed stereo image registration. As the sensors are not identical and the pixel sizes and pixel counts are not the same, the registration can be considered to be heterogenous stereo imaging registration. Heterogenous stereo image processing can be found in thermal and VIS camera too

[0032] ,The extra-bands registration process basically should find the correspondence between stereo image points by locating the same pixel in different images. 2D search for such a correspondence can be reduced to ID line search by rectification process. The rectification process transforms the calibrated stereo images to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal. The conventional planar rectification approach which uses multiple stereo images of targets with well-known regular pattern such as a checkerboard or grid patterns in different positions and poses to estimate the required projection matrix that has been used. A proper set of images that have been selected for direct stereo registration are from the images of the common wavebands between imagers (lOOOnm waveband) The same setup which has been used for geometrical distortion correction has been used and more than 20 images has been gathered for both imagers. As the imagers have different point of view and field of view, the target position should be selected carefully to capture full view of the target in both images.Finding an estimate of the projection matrix, the images to be registered can be rectified. The horizontal disparity map can be obtained by registering rectified images line by line. The end product is estimated depth derived from the disparity map and scene reconstruction in cloud map format.The resolution of such a depth reading is limited by the disparity resolution. To estimate the depth resolution of the system let us use a simple stereo triangulation and estimate depth by: z(d} = bf]d(44)Where f is the focal length, b is the sensors separation (center to center), and d is the disparity in image plane. Then,f — ^777777 LUsing the following parameterJofapjxe[sjze asthe minimum resolvable disparity in image plane

[0033] , i.e.,Ad = pixel size / 5 = 15 / 5 = 3xlQ~3mm ,Fig 20 shows how depthresolution varies with the target to the MSI imagers. As you can see the depth resolution is at 1.7cm at 3m and reduces to 18.5cm at 10m.While it seems straight-forward to use the disparity map for registration purposes, the result was not satisfactory. The issue is that the rectification process is never perfect. The main idea of stereo registration is after rectification, shred the image to strips of horizontal rows and search the best horizontal disparity in that row. The registered image can be put together after applying the best disparity on each row. However, due to the imperfection in the rectification process there are a small to disparity in y direction. This disparity shifts the corresponding points to different lines which are not included in the registration process. This will lead to unacceptable image distortion after registration. The registration process that we suggest is localized homography based on the scene topography. Using the scene reconstruction of this section, the desired part of the scene that satisfies the planar shape (up the resolution that the cameras provide) can be selected and a unique homography can be sought that leads to acceptable image registration.Stereo image homography of a plane. Without losing the generality, let us locate the world coordinate system on the subject plane at Z = 0, then the points on this plane can be expressed as:M = [A' r 0 ifAnd the image points on the sensors of the VISNIR and SWIR cameracan be related to M using a conventional pinhole model, i.e.,Where X is a constant and P is the projection matrix of VISNIR and SWIR cameras expressed by:and KSIVIR is the VIS-NIR and SWIR camera intrinsic matrix, where can be expressed in a simplified form as:Where CxyisNIRand Cy, VISNIRarethe coordinates of the principal point, fxVISNIRfy, VISNIR are the focal length of the VIS-NIR camera in pixel unit and image skew has been ignored.Similarly Cx>swiRar*d CyswiR are the coordinate of the principal point, fx>swiRar|d fy,swiRarethe focal length of the SWIR camera in pixel unit and image skew has been ignored too.Defini .ng M = \ ILX Y Of J as tuhe world points on a modeled plane (see Fig. 2 >1), then Eq. (49) can be simplified to:The inversion of the projection of VISNIR camera can be obtained as:Where CVISNIR is a singular vector of ^VISNIR^VISNIR- Reprojection of this point on theSWIR camera leads to:Where RSWJR-HSNIR Mid SWIR-VISNIR are the rotation and translation of SWIR camera with respect to VIS-NIR camera respectively. Eq. (56) indicates the homography exists on the image of a plane in stereo images. The homography includes a multiplicative term which is independent of the scene (or the chosen plane)vector which is scenerelated. Please note that the chosen world coordinate system is on the plane with Z=0, and this makes RSWIR and CJ^SNIR scene related.For the registration process, we can applyto the SWIR image which will take care of image centre shifts, scaling (due to different lens amplifications) and camera rotations with respect to each other. Then rigid transformation finalizes the registration process.Results. We demonstrate the results of spatial calibration, deblurring, intra-band, stereo, and extra-band registration process on the imaging targets. To demonstrate the full capacity of the instrument and the adapted techniques, we selected two typical lunar analog samples and present the registration results on them.Results - Spatial calibration. Using the gantry system and the manual tilt / span stage, we captured more than 20 images of different poses of the imaging target at a fixed position.Considering the x-y movement range of the gantry system and the field of view of our images, the azimuthal angle change of +25 degrees and polar angle variation of ±15 degrees were achieved, when the imaging target was about 3m away from the DS-MSI. The light source was a couple of Tungsten Halogen lamps of 1000W each. Before processing the VIS-NIR and SWIR image sizes were modified according to Table 10 to make them the same size considering their different field of view. This step which includes image magnification and cropping is necessary considering the following registration process. Using a various sensor categories and technology, the pixel size and pixel count are not matched. The conventional down sampling of the higher resolution camera is not recommended for registration process, as good quality data of the high-resolution sensor will be thrown away

[0018] . Here, a compromise has been made that involves finding a common ground for the resolution with the option of down-sampling at the end if necessary. As can be seen from Table 10, the SWIR sensor has been magnified by a factor of 2, while the VIS-NIR sensor down sampled by a factor of 0.78. The bicubic interpolation scheme has been used for image magnification. The images can be down sampled by 0.5 factor at the end to reach to the natural lower resolution of SWIR sensor.Table 10: image resizing and cropping results.The radial and tangential correction parameters along intrinsic camera parameters have been extracted using the captured images. The reprojection error was below 0.3 pixel in all cases. Matlab image processing toolbox has been used. The stereo parameters were also extracted for two set of lOOOnm wavebands of VIS-NIR and SWIR sensors. The mean reprojection error for stereo images was less than 0.4 pixels.Results - Image deblurring. We have used the imaging target at the middle of the scene in upright position as the focusing target and use it for verifying registration process too. The images for focusing psf estimation was first geometrically corrected using the parameters obtained in previous step and then procedure described on section IIB has been applied. The noise to signal ratio for focusing psf estimation was selected as 0.1%. The noise to signal ration for denoising process was selected based on S / N of 21dB and 45dB for SWIR and VISNIR sensors respectively. The S / Nmeasurement was done separately in a dark room test setup. Fig. 22 shows the raw psf function for the SWIR imager at lOOOnm waveband when it uses SWIR image at 1250nm waveband as reference.Table 11 shows the best generalized 2D gaussian PSF function parameters for all wavebands which has been fitted to the raw focusing PSF using least square approximation. As can be seen from Table 11 the correlation coefficient in this case is small and can be ignored. For the sake to denoising purposes, the PSF of the reference images are assumed Kronecker delta function.Table 11: the focusing psf estimation.Fig. 23 compares the original SWIR image of the imaging target at lOOOnm waveband at the top with the deblurred one at the bottom where the focusing gaussian PSF with parameters of Table 11 has been used. As can be seen from Fig. 23, the image quality has been improved, but some minor ringing effect at the sharp contrast regions are observed.Results - Intra-band and Stereo Registration. The images of the imaging target after geometrically corrected and deblurring process are used for verifying the registration process. The first step of this registration as depicted in Fig. 18 is intra-band registration of all VIS-NIR waveband images with 750nm waveband image and all of SWIR waveband images with 1250nm waveband image. The second step is 3D scene reconstruction using lOOOnm waveband images of VIS-NIR and SWIR sensors. Fig. 24 shows such an image. The relative position of the imaging target and background objects are clear.Ignoring the nulls in the disparity map, an estimate of the registered images at lOOOnm wavebands can be obtained (see Fig. 25). The green (examples of green color regions marked withleft-to-right up diagonal hatching) and magenta (examples of magenta color regions marked with left-right down diagonal hatching) colors have been used for different images. The predissertation process as expected has registered images at different depths uniformly but as we explained before the registered image suffers from distortion originated from residual y disparity. To demonstrate this fact, Fig. 26 shows the detail of a typical hole in the imaging target of the SWIR image before and after registration.Results - Extra-band registration. Selecting the imaging target as the plane of our interest, it is possible to perform two level registration. The result of such registration is shown in Fig. 27. As you can see the image registration quality at different depth deteriorates as we deviate from the plane of registration (in this case the imaging target). However perfect registration was achieved at the desired imaging target plane. The details of the registration on a typical hole in the imaging target is shown in Fig. 28.As the last step of registration, we apply the same geometrical transformation on all SWIR images. To show how successful the process is, let’s look in details of registration of two extreme wavebands, i.e., 400nm with 1500nm wavebands. Fig. 29 shows details of registration on two extreme wavebands. As you can see sub pixel registration is achieved. Please note that a final resizing of 0.5 should be done to bring the images to the native SWIR image resolution (see Table 10).Results - Registration of Analog Lunar Samples. To fully demonstrate the full capacity of the DS-MSI, we selected two lunar analog samples and repeat the registration process. Comparing the registration process of two extreme wavebands, we calibrate the wavebands with the standard reflectance panels and calculated the average sample reflectance and compare the results with what calibrated spectrometer measured.The selected analog lunar samples are as follows:1. High reflectivity: Mistastin MM 10 011 A Anorthosite - shatter cones2. Low reflectivity: Craters of the Moon COTM 17002 Basalt - rubbly pahoehoe lava The imaging has happened under a similar condition as we used for calibrating the DS-MSI. Fig. 30 shows the rectified and registered images of lOOOnm waveband and Fig. 31 shows the 3D scene reconstruction based on the point cloud data. The approximate analog sample sizes and relative position of the stand and the background wall can be read easily.As the depth resolution at 3m according to Fig. 20 is about 1 ,7cm at 3m, the analog targets can be assumed laying on a plane. Selecting them as the plane of registration, we can apply a twolevel registration process on this plane. To demonstrate the quality of the registration process, Fig. 32 shows registration results of the most extreme wavebands. As you can see registration process is desirable.Experimental Exemplification: Example 3 (Reference List).[1] M. Smith et al., "The artemis program: An overview of nasa's activities to return humans to the moon," in 2020 IEEE Aerospace Conference, 2020: IEEE, pp. 1-10.[2] I. A. Crawford, "Lunar resources: A review," Progress in Physical Geography, vol. 39, no. 2, pp. 137-167, 2015.[3] I. Manolis et al., "The MetOp second generation 3MI mission," in International Conference on Space Optics — ICSO 2012, 2017, vol. 10564: International Society for Optics and Photonics, p. 105640A.[4] H. Fujisada, "Design and performance of ASTER instrument," in Advanced and next-generation satellites, 1995, vol. 2583: SPIE, pp. 16-25.[5] M. Main-Knom, B. Pflug, J. Louis, V. Debaecker, U. Muller-Wilm, and F. Gascon, "Sen2Cor for sentinel-2," in Image and Signal Processing for Remote Sensing XXIII, 2017, vol. 10427: International Society for Optics and Photonics, p. 1042704.[6] N. Bernini, M. Bertozzi, P. Cerri, and R. I. Fedriga, "SWIR Cameras for the Automotive Field: Two Test Cases," International Scholarly Research Notices, vol. 2014, 2014.[7] A. Jenal, G. Bareth, A. Bolten, C. Kneer, I. Weber, and J. Bongartz, "Development of a VNIR / SWIR multispectral imaging system for vegetation monitoring with unmanned aerial vehicles," Sensors, vol. 19, no. 24, p. 5507, 2019.[8] P. Moghadam, D. Ward, E. Goan, S. Jayawardena, P. Sikka, and E. Hernandez, "Plant disease detection using hyperspectral imaging," in 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), 2017: IEEE, pp. 1-8.[9] R. H. Wilson, K. P. Nadeau, F. B. Jaworski, B. J. Tromberg, and A. J. Durkin, "Review of short-wave infrared spectroscopy and imaging methods for biological tissue characterization," Journal of biomedical optics, vol. 20, no. 3, p. 030901, 2015.

[0010] H. Steiner, A. Kolb, and N. Jung, "Reliable face anti-spoofing using multispectral swir imaging," in 2016 international conference on biometrics (ICB), 2016: IEEE, pp. 1-8.

[0011] Y. Liu, Z. Wei, X. Zhu, L. Li, L. Ma, and J. Zhou, "Multi-channel NIR / SWIR Lanthanide Nanoprobe for Forensic Gender Identification of Fingerprints," 2021, doi: 10.21203 / rs.3.rs-439647 / vl.

[0012] J. Bell et al., "The Mars 2020 perseverance rover mast camera zoom (Mastcam-Z) multispectral, stereoscopic imaging investigation," Space science reviews, vol. 217, no. 1, pp. 1-40,2021.

[0013] J.-F. Yang et al., "Design and Ground Verification for Multispectral Camera on the Mars Tianwen-1 Rover," Space Science Reviews, vol. 218, no. 3, pp. 1-27, 2022.

[0014] A. Coates et al., "The PanCam instrument for the ExoMars rover," Astrobiology, vol. 17, no. 6-7, pp. 511-541, 2017.

[0015] A. Coates et al., "PanCam on the ExoMars 2018 Rover: A stereo, multispectral and high-resolution camera system to investigate the surface of Mars," 2015.

[0016] G. Osinski et al., "An Adaptable Integrated Vision System for Lunar Exploration," presented at the 73rd International Astronautical Congress, Paris, France, 2022-09-22,2022, Oral, 73012.

[0017] M. P. S. Tondewad and M. M. P. Dale, "Remote sensing image registration methodology: Review and discussion," Procedia Computer Science, vol. 171, pp. 2390-2399, 2020.

[0018] B. Ayhan, M. Dao, C. Kwan, H.-M. Chen, J. F. Bell, and R. Kidd, "A novel utilization of image registration techniques to process mastcam images in mars rover with applications to image fusion, pixel clustering, and anomaly detection," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 10, no. 10, pp. 4553-4564, 2017.

[0019] F. P. Oliveira and J. M. R. Tavares, "Medical image registration: a review," Computer methods in biomechanics and biomedical engineering, vol. 17, no. 2, pp. 73-93, 2014.

[0020] A. Sotiras, C. Davatzikos, and N. Paragios, "Deformable medical image registration: A survey," IEEE transactions on medical imaging, vol. 32, no. 7, pp. 1153-1190, 2013.

[0021] Covestro. "Makrolon® IR sheet." https: / / www.eplastics.com / pdf / MakrolonIRDataSheet.pdf (accessed 2022.

[0022] D. C. Popescu and A. D. Hellicar, "Point spread function estimation for a terahertz imaging system," EURASIP Journal on Advances in Signal Processing, vol. 2010, pp. 1-8, 2010.

[0023] J. A. a. S. Bar a, "Optics and Deconvolution: Wavefront Sensing," in Optical and digital image processing: fundamentals and applications, G. Cristobal, P. Schelkens, and H. Thienpont Eds.: John Wiley & Sons, 2013, ch. 25.

[0024] J. M. Fitzpatrick, D. L. Hill, and C. R. Maurer, "Image registration," in Handbook of medical imaging, vol. 2, 2000, ch. 8, pp. 447-513.

[0025] R. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses," IEEE Journal on Robotics and Automation, vol. 3, no. 4, pp. 323-344, 1987.

[0026] J. Heikkila and O. Silven, "A four-step camera calibration procedure with implicit image correction," in Proceedings of IEEE computer society conference on computer vision and pattern recognition, 1997: IEEE, pp. 1106-1112.

[0027] Z. Zhang, "A flexible new technique for camera calibration," IEEE Transactions on pattern analysis and machine intelligence, vol. 22, no. 11, pp. 1330-1334, 2000.

[0028] J. Weng, P. Cohen, and M. Hemiou, "Camera calibration with distortion models and accuracy evaluation," IEEE Transactions on pattern analysis and machine intelligence, vol. 14, no. 10, pp. 965-980, 1992.

[0029] J. Brauers and T. Aach, "Geometric calibration of lens and filter distortions for multispectral filter-wheel cameras," IEEE Transactions on Image Processing, vol. 20, no. 2, pp. 496- 505, 2010.

[0030] C. Zhang, R. Celestre, M. Rosenberger, and G. Notni, "Exact correction of image distortion in a filter wheel multispectral camera with focus adjustment," in Journal of Physics: Conference Series, 2018, vol. 1044, no. 1 : IOP Publishing, p. 012031.

[0031] J. Brauers, N. Schulte, and T. Aach, "Multispectral filter-wheel cameras: Geometric distortion model and compensation algorithms," IEEE transactions on image processing, vol. 17, no. 12, pp. 2368-2380, 2008.

[0032] H. Zhou, C. Li, G. Sun, J. Yin, and F. Ren, "Calibration and location analysis of a heterogeneous binocular stereo vision system," Applied Optics, vol. 60, no. 24, pp. 7214-7222, 2021.

[0033] D. Zheng and Y.-q. Wang, "Depth measurement based on bifocus imaging," in 2009 2nd International Congress on Image and Signal Processing, 2009: IEEE, pp. 1-5.Experimental Exemplification: Experimental Example 4 (common filter overlap and its effect on registration process).Experimental Example 4 presents an example of examining behaviour of the optical filters that are used for the extra band registration process. To provide examples of filter configurations and how much overlap would be enough for registration, we made some simplifying assumption: a. The resolution of both imagers assumed identical.b. Imagers have a same amount of image noise. c. As the bandwidth of such filters are usually small, we assume piecewise linear reflection spectrum of the target and the background. d. The target and background are assumed homogenous. e. For the sake of this simulation, the readings of the imagers for target and background are assumed constant for 100% filter overlap. This can be achieved through imagers parameters settings.As the registration process mainly depends on the image contrast between target and the background, we focus our attention to contrast and how the filter overlap affects it.We have used Matlab [MATLAB (mathworks.com)] as a programming language to perform our simulations, the following steps have been taken: a. A ideal multi-bar benchmark image with a constant contrast has been used in this simulation. This image can be assumed as the image of multi -bars of target and background material taken by imager number 1 with the reference filter and the target is reflecting less than the background. The target and the background reflectance are adjusted to read a constant contrast (20% contrast is assumed). Fig. 33 shows the benchmark image that has been used in this simulation. b. The filter response is normalized to one, with the bandwidth of one and the wavelength is shifted to zero. We only considered one sided overlap, and we span the overlap value from 100% (identical filters) to 0% (adjacent to each other). Fig. 34 shows the filter responses. c. A constant Gaussian noise (with zero mean and 5% standard deviation) has been added to the images of both imagers. The practical bench mark image (when 5% noise has been added) is shown in Fig. 35. d. We consider various target and background spectral responses too. As the bandwidth of optical filters of our concerns are usually small, we only assume piecewise linear spectral responses. A general piecewise linear reflectance in a normalized unit is depicted in Fig. 36. The normalized filter responses bandwidth is assumed one and it is shifted to zero wavelength too. e. The Contrast to Noise Ratio CNR as defined in Equation 58 was examined for imager number 2 with various target and background spectral responses and different filter overlaps.where S(target) and S(background) are image intensities of target and the background respectively, while N(targef) and N (background) is the corresponding image noise. For machine vision tasks, the minimum recognizable CNR is one [CNR(Contrast-to-Noise Ratio), eye versus machine. Hamamatsu Photonics], In our simulation with the noise of 5%, the minimum acceptable contrast will be 10%.In Table 12 some senarios are illustrated which demonstrate the overlap effect on the contrast of images from imager #2. This clearly relates to registration capability of these images with the practical benchmark image of imager #1. The first column is the spectral response of the target and the background in the region of interest and the right column shows the simulated images after passing through various overlapping filters. A 100% overlap will generally yield consistent best results, but a reduced overlap to various extents can be accommodated, as illustrated in six scenarios presented in Table 12.Table 12. Filter overlap effect on image contrast.Experimental Exemplification: Experimental Example 5 (generalized multi-row filter wheel geometry).Experimental Example 5 presents an example of implementing the multi-row filter wheel geometry for simultaneous alignment with more than two sensors. Fig. 38shows the filter wheel slice and the optical filter positions providing a filter arrangement that simultaneously align with all sensors and permit simultaneous image acquisition with all sensors. The filters can be paired that fixes the position of the image sensors and the lenses. Fig. 39 shows a typical pairing (quartet groupings of filters providing multiple pairs) for a quad-sensor filter wheel. The filter pairs central angular shift is depicted asthe first and the second pair,for the second and third pair and finally2^^*34for the third and the fourth pair. Fig. 40 shows a model with filter wheel, lenses and image sensors. The scalability of such a configuration allows multiple sensors and multiple spectral bands in a single platform.The filter diameters in each row are constant (or for each row / ring of filters the radius measured from the center of each filter to the center of rotation is held constant). Preferably, larger filters are placed near the periphery of the filter wheel for the sake of filter wheel size reduction. Fig. 41 shows a triple filter row with identical filter size. Generally, the filter shape can be of any shape, the most common ones are circular as shown in Fig. 41A or rectangular as shown in Fig. 41B or a combination of them.Fig. 42 shows a typical filter pairing (triplet grouping of filters providing multiple pairs) and central angular shift for triple row filter wheel. Each row can have up to 7 filters. Fig. 43 shows the corresponding model with filter wheel, lenses and image sensors.Table 13. Central angular shifts in Fig. 39 and Fig. 42.Experimental Exemplification: Experimental Example 6 (filter pair central angular shift practical limits).Experimental Example 6 presents an example of calculations establishing practical minimum and maximum of filter pair central angle shift which is an angle between a filter pair with reference to a center of rotation of the filter wheel.The minimum practical central angle shift between two succeeding rows can be expressed as:Where n is the number of filters in each row. The minimum practical central angle shift has taken into account the physical constraints of the sensors, electronics and lenses, to ensure that they can be accommodated appropriately.The maximum practical filter pair central angle shift can be expressed as:Here, the maximum practical angle ensures that the sensors are not separated too far from each other and thus preserve the necessary common field of view of the instrument. Fig. 44 shows the practical minimum and the maximum of the central angle shift when number of filters in each row is an even number. Similarly, Fig. 45 shows the practical minimum and maximum of the central angle shift when the number of filters in each row is an odd number.Experimental Exemplification: Experimental Example 7 (multi-row registration process).Experimental Example 7 presents an example of a generalized multi-row registration process that compliments the generalized multi-row filter wheel described in Experimental Example 5.The registration process that has been presented for dual band can be extended for greater numbers of multiple bands in a systematic way based on the core of the dual band registration. Starting with a general kAth row registration process as shown in Fig. 46, each row registration process is very similar to the dual row filter wheel registration process; however, we arrange it differently to manage the multi-row filter wheel registration. The bands are arranged from lowerband (-m_k) to the higher band (n_k). Please note that the total number of frames in each row is constant, i.e., nk+ mk+ 1 = const = Number of filters in each row of the filter wheel (61)The band number zero is the central band where the lens is in full focus, and all other bands’ defocus are corrected based on this channel. Furthermore, it is the reference frame for all other frames in this row for intra-band registration. The result of this registration is sub-data cube of this row and the corresponding imager. Also, it is notable that this reference frame is used for stereo calibration with a similar one in the (k+l)Ath row. Two other special frames can be distinguished too. The frame corresponds to band -i_k which is common to the band j_(k-l) of the previous row and frame from band j_k which is common to the band i_(k+l) of the next row. These two frame pairs are used in extra-band registration processes.Fig. 47 shows the kAth row extra-registration process. Please note that the rows arrangement is not physical (i.e., one near to the center of the filter wheel comes first), but they are arranged based on the wavelength. The row of shortest wavelengths is the first row, and the longest wavelength will be the last row. The extra-band registration of the kAth row uses the common band frame from the previous row (one frame from band -j k of the current row to the i (k-1) of the previous row) to register the sub-data cube of this row to the sub-data cube from row 1 to row k-1. For this task the stereo calibration of the band 0 of this row and the previous row will be used. The output of this stage is sub data cube from row 1 to row k. This step also passes the frame corresponding to band i_k for the registration of the next row.Fig. 48 shows the high-level registration scheme for multi-row filter wheel. Clearly the kAth row, the previous row (the (k-l)Ath row) and the next row (the (k+l)Ath row) frames are shown. The bands that have been chosen for the stereo calibration, and extra-band registrations are marked. The sub-data cubes start from the first row and grow by including each row sub-data cubes (after finalizing extra-band registration) till the last row where full data cube will be generated.Experimental Exemplification: Experimental Example 8 (common FOV requirement imposes practical limits on filter pair central angular shift).Experimental Example 8 presents an example of a filter pair central angular shift being constrained or limited by a desired common Field of View (FOV) ratio.A common FOV is important for registration of multispectral image data. The registration process of multispectral imagers can be reduced to dual sensor imaging with the common optical filters. Fig. 49 shows the schematic of the dual sensor separated by a distance y with two identical field of views of 0. The common FOV and the full FOV are defined in Fig. 49, the FOV ratio r is defined as:For typical rover applications, the imaging distance of A=lm and the full FOV of 10° has been used for generating Table 14. Table 14 shows the maximum and minimum central shift angle versus number of filters in each row considering versus threshold for r value. In these applications, it is typically important to maximize the common FOV ratio. A minimum of 55% is required and often 90% or higher is desired. It can be seen from Table 14 that common FOV of 55% and above, the maximum central angle shift for this case does not exceed 135 degrees typically and the minimum central angle shift is greater than 9 degrees.Table 14. Maximum and minimum central shift angle versus number of filters per imager for various useful FOV ratio threshold.Experimental Exemplification: Experimental Example 9 (intermingling of filters of adjacent rings / rows).Experimental Example 9 presents an example of intermingling filters of adjacent rings / rows for a compact design.A key consideration for portable instrumentation for remote sensing applications such as in space based planetary rover camera systems, earth based remote operations, or even handheld instrumentations is SWaP (Size, Weight and Power) requirements. In a multi-imager multi-filter systems, it is desirable and important to have a system that is compact and robust while providing the multispectral functionality required. Here we present an innovative single filter wheel design for multi-imager multi-filter application where the filters are arranged in a unique way with each row of filters (each row references a ring or arc of filters that correspond to a single imager) intermingled with the adjacent rows in a lattice structure to achieve a compact design. This is a unique structure as prior attempts to configure two rows of filters (whether simultaneously or not simultaneously aligning with two imagers) have had each row occupy a distinct radial region with a non-optical or opaque border continuously positioned between and separating the two rows. The filter wheel design configuration can be extended from n rows with m filters with variable size of filters or similar filter sizes as needed in each row. This arrangement allows synchronous image acquisition of each filter corresponding to different imagers (filter grouping) while achieving a small form factor and maximizing common FOV by having the sensors as close as desired.Fig. 50 shows the general intermingled lattice filter wheel. The intermingled lattice filterR + Df wheel design allows merging the boundaries of each row defined ask~J' / 2 to the neighboring row boundary. The intermingling condition for circular filters in adjacent rows can be express as:Equation 63 provides the upper limit for Rk i so the filters in row k+\ intermingle with filters in row k. As it is shown in Fig. 50, the 5th row and the 4th row are intermingled as the upper boundary of the 4th row and the lower boundary of the 5th row are overlapped.Once the number of filters per imagers is selected, the filter wheel configuration can be determined if the closeness of filters in neighboring row (the closest gap between filters perimeters in adjacent rows; the closest gap is the shortest distance between a first filter perimeter in a first row and a second filter perimeter in a second row, where the first row and second row are adjacent rows) and size of the filters are known. This closest gap is a function of external constraints, and it isusually determined through design optimization process. However, it is always small enough to allow the filters in neighboring rows to intermingle.The intermingling condition can be extended to the rectangular filters with size yvf x If (width(w) x length(l)) as follows:The row angle as shown in Tables 7 and 8 being less than 90 degrees is related to the intermingling design in that the row angle is another means of characterizing the closest gap; more specifically, specifically with row angle know and the diameter of filters known the closest gap may be readily calculated, and in Tables 7 and 8, for example, the 60 degree max of the row angle maintains the intermingling of adjacent rows for filters with a diameter greater than a predetermined threshold while not providing intermingling for filter diameters lesser than the predetermined threshold.An illustrative version and several variants of a spectral imaging device, system and method have been described above without any intended loss of generality. Further examples of modifications and variation are now provided. Still further variants, modifications and combinations thereof are contemplated and will be apparent to the person of skill in the art. It is to be understood that illustrative variants or modifications are provided for the purpose of enhancing the understanding of the person of skill in the art and are not intended as limiting statements.An example of variation is that the described device, system or method may accommodate the first plurality of filters having a non-overlapping or overlapping with the second plurality of filters. Typically, some overlap will be present, but the extent of the overlap may vary widely. In certain examples, overlap of wavelength ranges is prominent, but the majority of the wavelength range remains non-overlapping. For example, expressed using set theory operations, the first plurality of filters corresponds to a first set of wavelengths, the second plurality of filters corresponds to a second set of wavelengths, the first set of wavelengths having an intersection (Fl) with the second set of wavelengths, the first set of wavelengths having a symmetric difference (A) with the second set of wavelengths, the symmetric difference having a greater range of wavelengths than the intersection. In other examples, a specific filter band may overlap within the first and second plurality of filters, such as at least one of the plurality of filter pairs comprising the firstselected filter and the second selected filter having an overlapping wavelength band. In another example, the first selected filter and the second selected filter have an identical wavelength band. Furthermore bandwidth of selected filters may vary widely depending on a particular implementation, for example the first selected filter and the second selected filter may be independently configured to have any bandwidth greater than 0.1 nanometer and less than 100 nanometer.In another example of variation, the choice of first and second imagers may determine an extent of overlap of wavelength bands in first and second plurality of filters. Wavelength bands of the first plurality of sensors can be matched to wavelength sensitivity of the first imager, and likewise wavelength bands of the second plurality of sensors can be matched to wavelength sensitivity of the second imager. In certain examples, the first and second imagers may respond to a similar or same wavelength range. In other example, the first and second imagers may be sensitive to overlapping but different wavelength ranges. For example, as described in Experimental Examples 1 and 3, the first plurality of filters corresponds to a VIS-NIR wavelength range and the second plurality of filters corresponds to a SWIR wavelength range. In related examples, the VIS-NIR wavelength range is from about 200nm to about lOOOnm, and the SWIR wavelength range is from about 900nm to about 2500nm.In another example of variation, a deviation of the first radial direction from a parallel orientation with the second radial direction may accommodate significant angular variation depending on requirements of a specific implementation. The first radial direction is a line passing through the rotation point of the filter carousel and the filter center of the first selected filter, and the second radial direction is a line passing through the rotation point of the filter carousel and the filter center of the second selected filter. The rotation point of the filter carousel is the intersection of a configured rotational axis of the filter carousel with the plane of the planar body of the filter carousel, and therefore a rotation point of the planar body may also be referred to as an axis of rotation of the planar body. The rotation point may be centered or offset from center of the planar body as may be suited to a specific implementation. In certain examples, the angular deviation from parallel may be expressed as the first radial direction and the second radial direction forming an interior angle greater than 10 degrees and less than 170 degrees. In another example, the angular deviation may be expressed as the first radial direction and the second radial direction forming an interior angle which is greater than or equal to 540 / n degrees, where n is the number of pairs in the plurality of filter pairs for four or more filter pairs. In a further example, angular deviation may beexpressed as the first radial direction and the second radial direction forming an interior angle less than or equal to 180 -180 / n degrees when n is even and 180 degrees when n is odd, where n is the number of pairs in the plurality of filter pairs. In yet another example, the first radial direction and the second radial direction forms an interior angle greater than or equal to 30 degrees, and is less than 170 degrees, or is greater than 20 degrees and less than 150 degrees. In other examples, the above expressions are not mutually exclusive and may be combined to define angular deviation from parallel. In considering angular deviation from parallel, an interior angle is the smaller angle of an intersection of two lines, where the sum of the smaller angle (interior angle) and the larger angle (exterior angle) equals 360 degrees. In some examples, the interior angle is less than 170 degrees, 160 degrees, 150 degrees, 140 degrees, 130 degrees, 120 degrees, 100 degrees or less than any angle therebetween and the interior angle is greater than 10 degrees, 20 degrees, 30 degrees, 40 degrees or greater than any angle therebetween.In another example of variation, the number of filters mounted in the filter carousel may vary widely. As shown in Experimental Example 2, when minimizing imager separation, first and second ring configurations of four filters each outperforms a single ring of 8 filters, and furthermore greater numbers are even better suited to first and second ring configurations. In certain examples, the first plurality of filters is at least four filters and the second plurality of filters is at least four filters. In other examples, numbers of filters need not be identical between first and second rings, and implementations with the first plurality of filters having a greater number of filters than the second plurality of filters may be accommodated.In another example of variation, the filter carousel need not be circular, and could be any convenient shape such as triangle, square, oval, semi-circle, hexagon, and the like. In certain shape variants the rotation point may not be located in the center (ie., offset from center) of the carousel shape, however the rotation point may still be referred to as a rotational center as the rotation point will be co-centered to both a spatial arrangement of the first plurality of filters and the second plurality of filters. Therefore, the rotational center does not need to be central to the selected shape, but does need to be central to filter arrangement so that the first plurality of filters are defined by a common first radial distance from the rotational center, and the second plurality of filters are defined by a common second radial distance from the rotational center. The term filter carousel may be used interchangeably with its selected shape, for example when the filter carousel shape is a full disc or wheel the term filter carousel may be used interchangeably with filter disc or filter wheel.In another example of variation, the filter shape need not be circular, and could be any convenient shape such as triangle, square, rectangle, oval, semi-circle, hexagon, and the like.In another example of variation, deviation from a dual ring configuration of filters may be accommodated, more specifically certain examples can deviate from the first plurality of filters is arranged in a first ring around the rotational center, the first ring defined by the first radial distance, and the second plurality of filters is arranged in a second ring around the rotational center, the second ring defined by the second radial distance. In certain examples, a full ring pattern may not be necessary, and filters may be arranged in an arc pattern. In other examples, a further ring or arc of filters may be included, for example a third plurality of filters arranged in a third ring or arc around the rotational center defined by a third radial distance different than the first or second radial distances, or for example a fourth plurality of filters arranged in a fourth ring or arc around the rotational center defined by a fourth radial distance different than the first, second or third radial distances. In such examples, three rings of filters correspond to three imagers that simultaneously align with one of a plurality of triplet filter groupings, each triplet filter grouping containing a single filter from each of the three rings, or four rings of filters correspond to four imagers that simultaneously align with one of a plurality of quad filter groupings, each quad filter grouping containing a single filter from each of the four rings.The term filter pairings or filter pairs may be generalized to the term filter groupings to more generally accommodate groups of three or more filters selected from three or more rings of filters for simultaneous alignment with three or more imagers. When two, three or more rings of filters are presents, any grouping of filters in the carousel will include at least a filter pairing - i.e., at least a simultaneous alignment of two filters with two sensors. Filter groupings is a generalized term that is directed to a group of filters selected that are associated by their geometrical positioning or spacing to simultaneously align with a plurality of image sensors, each one of the filter grouping aligned with one image sensor. In addition, analysis of each filter grouping of three or more filters can be simplified into components of multiple pairs. For example, a triplet filter grouping can be considered as composed of a first pairing of filters selected from the first and second ring, a second pairing of filters selected from the second and third ring, and a third pairing of filters selected from the first and third ring.In another example of variation, to achieve compactness of filter groupings, filters of adjacent rings may intermingle in a lattice pattern or structure. For example, filters from a first ring may intermingle with filters from a second ring, and filters from a second ring may intermingle withfilters from a third ring and so forth for additional rings. Intermingling does not mean that filters are contiguous or are connected, but rather mean that a filter from a first ring extends a portion of its filter area into a space between two adjacent filters in a second ring. For example, in an intermingling implementation, the first selected filter perimeter is bound in a radial direction relative to the rotation point by a first proximal perimeter portion and a first distal perimeter portion and the second selected filter perimeter is bound in a radial direction relative to the rotation point by a second proximal perimeter portion and a second distal perimeter portion; and the first distal perimeter portion is at a greater distance from the rotation point than the second proximal perimeter portion. In other words, a portion of the perimeter of the first selected filter is at a greater radial distance relative to the rotation point than a portion of the perimeter of the second selected filter. This intermingling relates to the row angle in Tables 7 and 8 always having a maximum value less than 90 degrees. The perimeter portions may be a straight line, may be a curve, may be a maximum point on a curve or a line.In another example of variation, the staggering of filters from a first ring to a second ring may be centered, and the staggering of filters from a second ring to a third ring may be centered, and so forth for greater numbers of rings. For example, geometrically if a first line connecting the respective centers of two adjacent filters in the first ring form a first side of a triangle, and second and third lines respectively connecting each of the centers of the two adjacent filters with a third center of the second row filter that is in a radial direction that perpendicularly intersects the line forming the first side of the triangle, then connecting the three centers with the first, second and third lines will always form either an isosceles triangle or an equilateral triangle if the staggering is centered.In another example of variation, the offset of the rotational center of the filter carousel from a center point between imagers can be varied according to needs of a specific implementation. In contest, of a dual-lens-dual sensor configuration described in the Experimental Examples, the rotational center of the filter carousel is offset from a linear line intersecting and extending between the central axes of the first imager and the second imager, or offset from a linear line intersecting and extending between the central axes of the first lens and the second lens. Stated analogously in terms of three dimensional space, the rotational center of the filter carousel is offset from a flat plane intersecting and parallel to the central axes of the first imager and the second imager, or offset from a flat plane intersecting and parallel to the central axes of the first lens and the second lens. The magnitude of the offset can vary depending on dimensions and geometries of filter arrangements, butregardless of variation of magnitude of offset, the offset will always be characterized by having unequal distances from the rotational center to the respective central axes. More specifically, since the first radial distance characterizing the first plurality of filters is different than the second radial distance characterizing the second plurality of filters, the rotational center must be mounted at different distances from the central axes of first and second lenses so as to position and align each pair of first and second selected filters for simultaneous operation with first and second lenses.In another example of variation, a motor driver of the filter carousel may be implemented in any suitable manner. For example, direct drive and belt drive configurations are feasible. Also, instead of the motor rotating a drum that frictionally engages a perimeter 44 of the filter carousel, the motor 80 rotates a motor gear attached to a motor spindle, and the motor gear engages a ring gear or ring pinion that circumferentially surrounds perimeter 44 of the filter carousel. Alternatively, the ring pinion may be integrally formed in perimeter 44.In another example of variation, methods for selecting a region of interest corresponding to a relatively flat or planar surface in an imaged object is not limited to the 3D scene reconstruction approach detailed in Experimental Example 3. In brief, the 3D scene reconstruction in Experimental Example 3 uses images of a scene taken by two imagers located at two slightly different locations which provide two points of view (stereo imaging). Alternatively, 3D reconstruction can also be achieved by using images of a scene from two or more points of view taken with a single imager and changing its position. This is the basis of the well-known photogrammetry. Another method involves using a LIDAR where distance of a particular point is determined by measuring the time delay between the emission of a laser pulse and the reflected light back from that point. A point cloud formed by combining an array of these distances along with a conventional 2D image of the same scene, produces a realistic 3D image. If the imaging environment is not air, (e.g., under water or inside the human body), a 3D sonar equipment is used to produce a 3D model similar to the lidar principle. Another example of an alternative technique is structured light imaging, involving analysis of deformation of regular patterns of light in an image. The equipment projects structured patterns of light onto the scene (such as parallel grids or common geometric patterns) and any deformation in the reflected light carries the depth information. Still further alternative approaches are contemplated.In another example of variation, the registration method is not limited to applying the third registration transformation (i.e., extra-band registration) to a specific set of images and may be applied to any of the first or second plurality of registered (registered by intra-band registration)images. However, in certain examples, the third registration transformation is applied to the lower resolution set of images when comparing the first plurality of registered images to the second plurality of registered images. Lower resolution correlates with greater pixel size and lower resolution correlates with lower pixel number per unit area. In the Experimental Examples provided herein, the InGaAs SWIR is relatively lower resolution than Silicon-VIS-NIR.In another example of variation, the term a common region of interest may be used interchangeably with the term plane of interest. The common region of interest is selected for purpose identifying a region within image data that corresponds to a flat planar surface of an imaged object, so that the selected region of interest facilitates determination of the third registration transformation. As a significant purpose of selecting a common region of interest is to identify a planar surface, the term may in this context be semantically equivalent to plane of interest.In another example of variation, image selections for intra-band registration need not be limited to central wavelength bands and other choices may be accommodated. For example, images may be selected on the basis of focus quality.In another example of variation, image selections for image rectification need not be limited to common wavelength bands and other choices may be accommodated. For example, images may be selected on the basis of focus quality. Alternatively, images may be selected on basis of being acquired with a central waveband.In another example of variation, image selections for developing / determining an extra-band registration transformation need not be limited to identical common wavelength bands - deviation from the same or identical wavelength bands may be tolerated and common wavelength band may include overlapping wavelength bands in some examples, or even contiguously adjacent or proximal wavelength bands in other examples.In another example of variation, the imaging devices, systems and methods described herein are not limited to multi-spectral implementations, and alternative spectral imaging implementations may be accommodated.Embodiments disclosed herein, or portions thereof, can be implemented by programming one or more computer systems or devices with computer-executable instructions embodied in a non- transitory computer-readable medium. When executed by a processor, these instructions operate to cause these computer systems and devices to perform one or more functions particular to embodiments disclosed herein. Programming techniques, computer languages, devices, and computer-readable media necessary to accomplish this are known in the art.In an example, a non-transitory computer readable medium embodying a computer program for registration of spectral stereo images may comprise: computer program code for obtaining a first plurality of images spanning a first spectral range acquired by a first image sensor, each of the first plurality of images acquired in a different wavelength band; computer program code for obtaining a second plurality of images spanning a second spectral range acquired by a second image sensor, each of the second plurality of images acquired in a different wavelength band, the first spectral range different than the second spectral range with at least one common wavelength band; computer program code for selecting a first reference image from the first plurality of images, the first reference image having a better focus quality than a majority of the first plurality of images; computer program code for applying a first registration transformation to register the first plurality of images with the first reference image to generate a first plurality of registered images; computer program code for selecting a second reference image from the second plurality of images, the second reference image having a better focus quality than a majority of the second plurality of images; computer program code for applying a second registration transformation to register the second plurality of images with the second reference image to generate a second plurality of registered images; computer program code for selecting a third reference image from the first plurality of registered images, and selecting a fourth reference image from the second plurality of registered images, both the third reference image and the fourth reference image acquired in the at least one common wavelength band; computer program code for determining a third registration transformation based on comparing the third reference image with the fourth reference image; and computer program code for applying the third registration transformation to the first plurality of registered images; and / or optionally the first reference image is a best focused image within the first plurality of images, and the second reference image is a best focused image within the second plurality of images; and / or optionally the first reference image is acquired with a wavelength band associated with a best focus metric among a first plurality of wavelength bands within the first spectral range, and the second reference image acquired with a wavelength band associated with a best focus metric among a second plurality of wavelength bands within the second spectral range; and / or optionally the first reference image is acquired with a wavelength band closest to a median wavelength of the first spectral range, and the second reference image is acquired with a wavelength band closest to a median wavelength of the second spectral range; and / or optionally the first registration transformation is a similarity transformation, and the second registration transformation is a similarity transformation; and / or optionally the third registration transformation comprisescorrection of image center shift, scaling to correct different lens magnifications, and / or imager rotation correction and optionally the third registration transformation further comprises a rigid transformation; and / or optionally the first plurality of images has a greater pixel size than the second plurality of images; and / or optionally the first plurality of images has a lesser pixel number per unit area than the second plurality of images; and / or optionally the first image sensor is a InGaAs-SWIR sensor and the second image sensor is a Silicon-VIS-NIR sensor; and optionally the at least one common wavelength band is multiple wavelength bands or a single wavelength band.In another related example, the computer readable medium further comprises computer program code for selecting a common region of interest for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object, and for example the common region of interest may be selected by comparing any one of the first plurality of registered images with any one of the second plurality of registered images, by comparing the third reference image with the fourth reference image, or by comparing the first reference image with the second reference image. In another related example, the computer readable medium further comprises computer program code for rectifying the third reference image with the fourth reference image to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal, and using the rectified image data to generate a 3D scene reconstruction, and may optionally further comprise computer program code for analyzing the 3D scene reconstruction to select a common region of interest in the third reference image and the fourth reference image for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object. In another related example, the computer readable medium further comprises computer program code for rectifying a fifth reference image selected from the first plurality of registered images with a sixth reference image selected from the second plurality of registered images to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal, and using the rectified image data to generate a 3D scene reconstruction, and optionally further comprises computer program code for analyzing the 3D scene reconstruction to select a common region of interest in the fifth reference image and the sixth reference image for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object.In another related example, the computer readable medium further comprises computer program code for identifying a common field of view in the first plurality of images and the secondplurality of images and cropping the first plurality of images and the second plurality of images to the common field of view, and optionally the cropping is symmetric and results in a similar or identical common field of view for the first plurality of images and the second plurality of images.In another related example, the computer readable medium further comprises computer program code for correcting chromatic aberrations in the first plurality of images and the second plurality of images, and optionally the first plurality of images are corrected by deblurring based on the first reference image, and the second plurality of images are corrected by deblurring based on the second reference image.In another related example, the computer readable medium further comprises computer program code for correcting geometrical aberrations in the first plurality of images and the second plurality of images.In another related example, the computer readable medium further comprises computer program code for controlling the rotational position of the filter carousel relative to the two or more image sensors. For example, the filter carousel and systems incorporating the same as described herein and each variant, modification or combination thereof may be controlled by a suitable computer-implemented method. A method of controlling the filter carousel rotational position may include detecting rotational position data with a position sensor coupled to the filter carousel or other rotational component engaging the filter carousel; receiving the rotational position data with a controller; and generating and communicating a control signal to a rotational actuator, such as a drive motor, to minimize a difference between the real-time position data and a preset desired position for simultaneous alignment of a filter grouping set with the two or more image sensors. For example, control algorithms such as PID algorithms can be used to calculate an error between the measured position data and the desired position for any filter grouping to simultaneously align with the two or more sensors; inputting the error into the controller, where an appropriate control signal is calculated; and communicating the control signal to the actuator of the motor with the motor response providing filter carousel rotation and subsequent measurement and communication of a rotational position value to reinitiate and repeat the loop. Optionally, the computer-implemented method may include an algorithm for calibration of the position sensor. In a related example, a controller can be configured to execute the method to provide a drive signal to an actuator or motor to control indexed rotation of the filter carousel for sequencing a plurality of simultaneous alignments of filter groupings in first and second beam paths defined by first and second lenses, respectively.The filter carousel and its controller may accommodate various controller types and controller algorithms to control rotational movement and a stopped rotational position for simultaneous alignment of a filter grouping with two or more image sensors. For example, proportional-integrative-derivative (PID), proportional -integrative (PI) or proportional (P) algorithms may be used to control the filter carousel movement and stopped position depending on parameters of a specific implementation.The computer readable medium is a data storage device that can store data, which can thereafter, be read by a computer system. Examples of a computer readable medium include readonly memory, random-access memory, CD-ROMs, magnetic tape, optical data storage devices and the like. The computer readable medium may be geographically localized or may be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.Computer-implementation of the system or method typically comprises a memory, an interface and a processor. The types and arrangements of memory, interface and processor may be varied according to implementations. For example, the interface may include a software interface that communicates with an end-user computing device through an Internet connection. The interface may also include a physical electronic device configured to receive requests or queries from a device sending digital and / or analog information. In other examples, the interface can include a physical electronic device configured to receive signals and / or data relating to the imaging method and system, for example from an imaging sensor or a sensor measuring rotational position data of the filter carousel.Any suitable processor type may be used depending on a specific implementation, including for example, a microprocessor, a programmable logic controller or a field programmable logic array. Moreover, any conventional computer architecture may be used for computer-implementation of the system or method including for example a memory, a mass storage device, a processor (CPU), a graphical processing unit (GPU), a Read-Only Memory (ROM), and a Random-Access Memory (RAM) generally connected to a system bus of data-processing apparatus. Memory can be implemented as a ROM, RAM, a combination thereof, or simply a general memory unit. Software modules in the form of routines and / or subroutines for carrying out features of the system or method can be stored within memory and then retrieved and processed via processor to perform a particular task or function. Similarly, one or more method steps may be encoded as a program component, stored as executable instructions within memory and then retrieved and processed via a processor. Auser input device, such as a keyboard, mouse, or another pointing device, can be connected to PCI (Peripheral Component Interconnect) bus. If desired, the software may provide an environment that represents programs, files, options, and so forth by means of graphically displayed icons, menus, and dialog boxes on a computer monitor screen. For example, any number of images and filter carousel rotational data may be displayed, including for example, a selected first reference image, a selected second reference image, a selected third reference image, a selected fourth reference image, wavelength values relating to a first spectral range, wavelength values characterizing a second spectral range, the common region of interest, or the first plurality of registered images before and / or after applying the third registration transformation.Computer-implementation of the system or method may accommodate any type of end-user computing device including computing devices communicating over a networked connection. The computing device may display graphical interface elements for performing the various functions of the system or method, including for example display of a blood flow characteristic determined for a cardiovasculature of interest. For example, the computing device may be a server, desktop, laptop, notebook, tablet, personal digital assistant (PDA), PDA phone or smartphone, and the like. The computing device may be implemented using any appropriate combination of hardware and / or software configured for wired and / or wireless communication. Communication can occur over a network, for example, where remote control of the system is desired.If a networked connection is desired the system or method may accommodate any type of network. The network may be a single network or a combination of multiple networks. For example, the network may include the internet and / or one or more intranets, landline networks, wireless networks, and / or other appropriate types of communication networks. In another example, the network may comprise a wireless telecommunications network (e.g., cellular phone network) adapted to communicate with other communication networks, such as the Internet. For example, the network may comprise a computer network that makes use of a TCP / IP protocol (including protocols based on TCP / IP protocol, such as HTTP, HTTPS or FTP).Embodiments described herein are intended for illustrative purposes without any intended loss of generality. Still further variants, modifications and combinations thereof are contemplated and will be recognized by the person of skill in the art. Accordingly, the foregoing detailed description is not intended to limit scope, applicability, or configuration of claimed subject matter.

Claims

WHAT IS CLAIMED IS:

1. A filter carousel comprising: a planar body having a rotation point; a plurality of filter groupings disposed in the planar body, the plurality of filter groupings comprising a first selected filter of a first plurality of filters positioned at a first radial direction from the rotation point and a second selected filter of a second plurality of filters positioned at a second radial direction from the rotation point; each of the plurality of filter groupings positioned for simultaneous operation; the first plurality of filters disposed in the planar body, each of the first plurality of filters having a first filter center positioned at a common first radial distance from the rotation point; the second plurality of filters disposed in the planar body, each of the second plurality of filters having a second filter center positioned at a common second radial distance from the rotation point; the first radial distance being lesser than the second radial distance; the first radial direction and the second radial direction at the rotation point forms a first interior angle greater than 0 degrees and less than 180 degrees.

2. The filter carousel of claim 1, wherein at least one of the plurality of filter groupings comprises the first selected filter and the second selected filter having an overlapping wavelength band.

3. The filter carousel of claim 2, wherein the first selected filter and the second selected filter have an identical wavelength band.

4. The filter carousel of claim 2, wherein the first selected filter and the second selected filter are independently configured to have a bandwidth greater than 0.1 nanometer and less than 100 nanometer.

5. The filter carousel of claim 1, wherein the first plurality of filters corresponds to a first set of wavelengths, the second plurality of filters corresponds to a second set of wavelengths, the first set of wavelengths having an intersection (A) with the second set of wavelengths, the first set of wavelengths having a symmetric difference (A) with the second set of wavelengths, the symmetric difference having a greater range of wavelengths than the intersection.

6. The filter carousel of claim 1, wherein the first plurality of filters is at least four filters and the second plurality of filters is at least four filters.

7. The filter carousel of claim 1, wherein the first plurality of filters has a greater number of filters than the second plurality of filters.

8. The filter carousel of claim 1, wherein the first radial direction and the second radial direction at the rotation point forms the first interior angle greater than 30 degrees and less than 170 degrees.

9. The filter carousel of claim 1, wherein the first plurality of filters is arranged in a first ring around the rotation point, the first ring defined by the first radial distance, and the second plurality of filters is arranged in a second ring around the rotation point, the second ring defined by the second radial distance, a distal perimeter portion of the first selected filter distal to the rotation point is at a greater radial distance relative to the rotation point than a proximal perimeter portion of the second selected filter that is proximal to the rotation point.

10. The filter carousel of claim 1, wherein a first side of the planar body is configured to face a first lens and a second lens, a second side of the planar body is configured to face a first image sensor and a second image sensor, the first lens aligned with the first image sensor, the second lens aligned with the second image sensor, the first selected filter and the second selected filter positioned for simultaneous alignment with the first lens and the second lens, respectively, a first observation mode configured by simultaneous occurrence of a first beam path passing through the first selected filter and a second beam path passing through the second selected filter.

11. The filter carousel of claim 10, wherein a third selected filter of the first plurality of filters positioned at a first radial direction from the rotation point and a fourth selected filter of the second plurality of filters positioned for simultaneous alignment with the first lens and the second lens, respectively, a second observation mode configured by simultaneous occurrence of the first beam path passing through the third selected filter and the second beam path passing through the fourth selected filter, the third selected filter positioned at a third radial direction from the rotation point, the fourth selected filter positioned at a fourth radial direction from the rotation point, the third radial direction and the fourth radial direction at the rotation point forms a second interior angle greater than 0 degrees and less than 180 degrees, the second interior angle equals the first interior angle.

12. The filter carousel of claim 1, wherein the first plurality of filters corresponds to a VIS-NIR wavelength range and the second plurality of filters corresponds to a SWIR wavelength range.

13. The filter carousel of claim 12, wherein the VIS-NIR wavelength range is from about 200nm to about lOOOnm, and the SWIR wavelength range is from about 900nm to about 2500nm.

14. The filter carousel of claim 1, wherein a ball bearing is disposed in the planar body, a housing of the ball bearing defining a rotational aperture for rotational mounting on a shaft, and an axis of rotation of the ball bearing on the shaft defining the rotation point.

15. The filter carousel of claim 1, wherein the rotation point is configured to be offset from a linear line intersecting central axes of a first image sensor and a second image sensor, or offset from a linear line intersecting central axes of a first lens and a second lens.

16. The filter carousel of claim 1, wherein the planar body is a disc shape bound by a circular circumference, and the rotation point is a rotational center of the planar body.

17. An imaging system, comprising: the filter carousel of any one of claims 1-16 rotationally mounted in a frame; a first lens and a second lens; a first image sensor and a second image sensor.

18. The imaging system of claim 17, wherein the frame is bound by opposing first and second sides and a perimeter sidewall connecting the first and second sides, the first and second lenses are mounted to the first side of the frame in alignment with the first and second image sensors mounted to the second side of the frame, the first lens aligned with the first image sensor, the second lens aligned with the second image sensor.

19. The imaging system of claim 17 or 18, further comprising a motor mounted to the frame, the motor communicative with the filter carousel and the motor providing a driving force for rotating the filter carousel.

20. The imaging system of claim 19, further comprising a controller providing a drive signal to the motor and controlling indexed rotation of the filter carousel for sequencing a plurality of simultaneous alignments of filter groupings in first and second beam paths defined by first and second lenses, respectively.

21. A computer-implemented method for registration of spectral stereo images, the method comprising: obtaining a first plurality of images spanning a first spectral range acquired by a first image sensor, each of the first plurality of images acquired in a different wavelength band; obtaining a second plurality of images spanning a second spectral range acquired by a second image sensor, each of the second plurality of images acquired in a different wavelength band; the first spectral range different than the second spectral range with at least one common wavelength band; selecting a first reference image from the first plurality of images, the first reference image having a better focus quality than a majority of the first plurality of images;applying a first registration transformation to register the first plurality of images with the first reference image to generate a first plurality of registered images; selecting a second reference image from the second plurality of images, the second reference image having a better focus quality than a majority of the second plurality of images; applying a second registration transformation to register the second plurality of images with the second reference image to generate a second plurality of registered images; selecting a third reference image from the first plurality of registered images, and selecting a fourth reference image from the second plurality of registered images, both the third reference image and the fourth reference image acquired in the at least one common wavelength band; determining a third registration transformation based on comparing the third reference image with the fourth reference image; applying the third registration transformation to the first plurality of registered images.

22. The method of claim 21, further comprising selecting a common region of interest for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object.

23. The method of claim 22, wherein the common region of interest is selected by comparing any one of the first plurality of registered images with any one of the second plurality of registered images.

24. The method of claim 24, wherein the common region of interest is selected by comparing the third reference image with the fourth reference image.

25. The method of claim 24, wherein the common region of interest is selected by comparing the first reference image with the second reference image.

26. The method of claim 21, further comprising rectifying the third reference image with the fourth reference image to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal, and using the rectified image data to generate a 3D scene reconstruction.

27. The method of claim 26, further comprising analyzing the 3D scene reconstruction to select a common region of interest in the third reference image and the fourth reference image for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object.

28. The method of claim 21, further comprising rectifying a fifth reference image selected from the first plurality of registered images with a sixth reference image selected from the second plurality ofregistered images to a single imaging plane in which imaging coordinate systems are aligned and epipolar lines are horizontal, and using the rectified image data to generate a 3D scene reconstruction.

29. The method of claim 28, further comprising analyzing the 3D scene reconstruction to select a common region of interest in the fifth reference image and the sixth reference image for determination of the third registration transformation, the common region of interest corresponding to a relatively planar surface among imaged surfaces of an imaged object.

30. The method of claim 21, further comprising identifying a common field of view in the first plurality of images and the second plurality of images and cropping the first plurality of images and the second plurality of images to the common field of view.

31. The method of claim 30, wherein the cropping is symmetric and results in a similar or identical common field of view for the first plurality of images and the second plurality of images.

32. The method of claim 21, wherein the first plurality of images and the second plurality of images are corrected for chromatic aberrations.

33. The method of claim 32, wherein the first plurality of images are corrected by deblurring based on the first reference image, and the second plurality of images are corrected by deblurring based on the second reference image.

34. The method of claim 21, wherein the first plurality of images and the second plurality of images are corrected for geometrical aberrations.

35. The method of claim 21, wherein the first reference image is a best focused image within the first plurality of images, and the second reference image is a best focused image within the second plurality of images.

36. The method of claim 21, wherein the first reference image is acquired with a wavelength band associated with a best focus metric among a first plurality of wavelength bands within the first spectral range, and the second reference image acquired with a wavelength band associated with a best focus metric among a second plurality of wavelength bands within the second spectral range.

37. The method of claim 21, wherein the first reference image is acquired with a wavelength band closest to a median wavelength of the first spectral range, and the second reference image is acquired with a wavelength band closest to a median wavelength of the second spectral range.

38. The method of claim 21, wherein the first registration transformation is a similarity transformation, and the second registration transformation is a similarity transformation.

39. The method of claim 21, wherein the third registration transformation comprises correction of image center shift, scaling to correct different lens magnifications, and / or imager rotation correction.

40. The method of claim 39, wherein the third registration transformation further comprises a rigid transformation.

41. The method of claim 21, wherein the at least one common wavelength band is multiple wavelength bands.

42. The method of claim 21, wherein the at least one common wavelength band is a single wavelength band.

43. The method of claim 21, wherein the first plurality of images has a greater pixel size than the second plurality of images.

44. The method of claim 21, wherein the first plurality of images has a lesser pixel number per unit area than the second plurality of images.

45. The method of claim 21, wherein the first image sensor is a InGaAs-SWIR sensor and the second image sensor is a Silicon-VIS-NIR sensor.

46. The system of any one of claims 17-20, further comprising a processor configured to execute the method according to any one of claims 21-45.

47. A computer readable medium embodying computer readable code for executing the method according to any one of claims 21-45.

48. A system for registration of spectral stereo images comprising: a memory for storing image data; and a processor configured to execute the method according to any one of claims 21-45.

Images