Optical methods

Abstract

An artificial neural network-based computer-implemented method (600) for determining a data cube is also disclosed.

Description

[Technical Field]

[0001] This disclosure relates to a method for acquiring hyperspectral images. [Background technology]

[0002] Hyperspectral imaging involves acquiring a three-dimensional data cube of a scene, collecting intensity across one spectral domain and two spatial domains. The data cube is in format I(x, y, λ). Each slice of the data cube contains an xy image corresponding to a specific wavelength of light. Conventional hyperspectral imaging devices rely on two main acquisition methods: using a scanning 2D sensor or spatially multiplexing spectral information extracted after post-processing to acquire the data cube. Scanning imaging devices are oriented-limited and require scanning (hence the name), which is a time-consuming process and introduces motion artifacts. Multiplexing imaging devices largely avoid the errors introduced by scanning, but require a significant sacrifice of spatial information or complex sensors to achieve hyperspectral imaging. Multiplexing imaging devices have limited resolution and / or are difficult to manufacture.

[0003] Algorithmic imaging techniques are being explored to achieve single-shot hyperspectral capture while avoiding the resolution sacrifices of multiplexed spectrometers. The most notable example is the Coated Aperture Snapshot Spectrum Imaging System (CASSI) and its variants. CASSI relies on compressed sensing, i.e., a signal processing framework that reconstructs an underdetermined linear system. By manipulating the incoming signal into a format viable for compressed sensing reconstruction, CASSI can reconstruct a hyperspectral data cube from signals acquired from a single exposure of a conventional two-dimensional detector. However, while CASSI achieves single-shot imaging without significant spatial resolution sacrifice or complex multiplexing, conventional compressed sensing reconstruction at 256*256*25 is already too long for practical use. Therefore, CASSI can only capture a data cube at a resolution limited by reconstruction time, often accompanied by occasional artifacts.

[0004] Several attempts have been made to use machine learning for image reconstruction, particularly using generative adversarial networks (GANs).

[0005] For example, a paper by Miao, Xin et al., titled "LambdaNet: Reconstructing Hyperspectral Images from Snapshot Measurements," presented at the 2019 IEEE / CVF International Conference on Computer Vision (ICCV), described a method for reconstructing CASSI hyperspectral signals using a trained generative adversarial network. However, this method was merely intended as an overview of Cassi and did not include atlas convolution or correction for differences in viewpoint.

[0006] Furthermore, incorporating a self-attention mechanism into a GAN architecture can enable the model to leverage both local and global spatial information to generate image features, as demonstrated by Zhang, Han et al. in "Self-Attentional Generative Adversarial Networks" at the International Conference on Machine Learning, PMLR, 2019, in the section on conventional computer vision for non-hyperspectral systems. In addition, the paper shows that applying Lipschitz constraints (in the form of spectral normalization) to both the generator and the discriminator improves model performance and stability.

[0007] In addition, Isola, Phillip et al.'s "Image-to-Image Transformation Using Conditional Adversarial Networks," Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, described using conditional gain in combination with a constraint l² on loss to generate high-resolution, high-fidelity images in non-hyperspectral computer vision settings.

[0008] The purpose of this disclosure is to overcome or at least improve upon the shortcomings associated with known coded aperture hyperspectral imaging methods. [Overview of the project]

[0009] According to a first aspect, a computer implementation method is provided for correcting disparity between a spectrally unsheared first image and a spectrally sheared second image. The first image may or may not be encoded, and the second image may or may not be encoded. The method includes using a machine learning algorithm to determine a transformation configured to map the first and second images to each other. The method also includes correcting disparity by transforming one of the first and second images using the determined transformation.

[0010] A spectrally sheared or unsheared field of view represents the field of view captured by any imaging sensor, including but not limited to monochromatic, RGB, RGB-I, RGB-IR, or any Debayer sensor.

[0011] Directly determining the parameters of a transformation (e.g., perspective transformation) to correct disparity between images is particularly difficult when one of the images is spectrally sheared. Conventional parallax correction techniques (e.g., computer vision parallax correction techniques) cannot be applied. Using machine learning techniques to determine a transformation (e.g., perspective transformation) to correct disparity (e.g., parallax or misalignment) between different fields of view (e.g., multiple fields of view detected by a hyperspectral imaging system) may allow for efficient iterative determination of the transformation.

[0012] In the context of hyperspectral imaging systems, the use of machine learning techniques can also provide improved alignment robustness. Compressed sensing reconstruction methods require a complete prior understanding of the position of encoder features in the detector, and misalignment between the mask and the sensor can make accurate hyperspectral data cube reconstruction impossible. Using machine learning techniques to determine a transformation (e.g., perspective transformation) and provide a mapping to a common field-of-view base for different fields of view to compensate for arbitrary parallax (e.g., parallax or misalignment) means that a prior understanding of the difference is not required, and the parallax is taken into account during the training process to determine the transformation. This can make it possible to accurately reconstruct the hyperspectral data cube even when, for example, the encoder is relayed to the sensor with an error of a few pixels. To accurately reconstruct the hyperspectral data cube, the typically expensive and redundant alignment process required in hyperspectral imaging devices may not be necessary. The alignment robustness may also be advantageous in dealing with thermal expansion and / or deformation, vibration-induced noise, and overall robustness (for example, bumping or dropping a hyperspectral imaging device can typically impair alignment).

[0013] Parallax can include parallax errors, rotation errors, alignment errors, and magnification errors.

[0014] Machine learning algorithms can incorporate neural networks.

[0015] The method may include concatenating a first image and a second image to form concatenated image data. The method may also include providing the concatenated image data to a neural network. The method may further include performing one or more convolution operations using the neural network. The method may include using the neural network to determine a transformation from the output of one or more convolution operations. The method may also include transforming one of the first image and the second image using the determined transformation.

[0016] Performing one or more convolution operations using a neural network may further involve downsampling.

[0017] The method may further include using a neural network to reconstruct the output of one or more convolution operations before determining the transformation.

[0018] Reconstructing the output of one or more convolution operations may include reconstructing the output from a two-dimensional dataset into a one-dimensional dataset.

[0019] A neural network may comprise one or more tightly coupled layers configured to determine transformations.

[0020] A perspective transformation may comprise a transformation matrix. The method may include using a neural network to determine one or more parameters of the transformation matrix. The transformation matrix may comprise a transformation matrix that determines at least one of rotation, translation, shear, and perspective shift, and scaling.

[0021] A neural network may include a plurality of parallel paths for performing one or more convolution operations. This can provide improved stability and learning of the neural network during backpropagation. One of the parallel paths may have a non-linear output, and one of the parallel paths may have a linear output. If backpropagation and learning through one of the paths (e.g., the non-linear path) result in a very high rate of change or no rate of change during learning, the learning may instead propagate through another parallel path (e.g., the linear path).

[0022] The method may include converting a second image to provide a corrected second image.

[0023] The present invention proposes a computer-implemented method for reconstructing a hyperspectral data cube from at least one encoded and / or spectrally clipped image and an unencoded and spectrally unclipped image of a scene, the method comprising: providing at least one encoded and / or spectrally clipped image and an unencoded and spectrally unclipped image of a scene to a neural network having an encoder-decoder structure with an encoder part and a decoder part; performing one or more first-order convolution operations using the encoder part; performing one or more transposed convolution operations using the decoder part; performing spatial adaptive normalization and / or inverse attention normalization on at least one output of one or more of the first-order convolution operations and / or transposed convolution operations using a representation of the encoded and / or spectrally clipped image and an unencoded and spectrally unclipped image of a scene, or an encoding scheme used to encode an unencoded and spectrally unclipped image of a scene; and including.

[0024] In at least one encoded and spectrally sheared image, either or both can be either a monochromatic imaging system, a filter-based imaging system, or have a high level of determinacy. The spectrally unsheared image can be either a monochromatic imaging system or a filter-based imaging system. The filter-based camera system can include a Bayer imaging system.

[0025] Reconstructing a hyperspectral data cube from an image corrected for parallax (e.g., misalignment, parallax, hardware defects, etc.) can provide improved hyperspectral reconstruction. In the context of tomographic reconstruction of a hyperspectral data cube, reconstructing a hyperspectral data cube from at least one encoded spectral shear image, including correcting for parallax, can allow the unique information in that field of view to be used independently from other fields of view before all fields of view are utilized in a combined reconstruction, while providing a common field of view basis for different fields of view.

[0026] According to one aspect, a computer-implemented method for reconstructing a hyperspectral data cube from encoded and spectrally sheared images is provided. The method includes providing an image to a neural network. The neural network has an encoder-decoder structure having an encoder part and a decoder part. The method includes performing one or more first convolutional operations using the encoder part. The method includes performing one or more transposed convolutional operations using the decoder part. The method includes performing spatial adaptive normalization and / or applying an inverse attention normalization block to at least an output of one or more of the first convolutional operations and / or transposed convolutional operations using a representation of a mask used to encode the image.

[0027] Spatial adaptive normalization applies the mask used to encode an image to the features extracted by a first-order convolution operation. SPADE forces the neural network to utilize encoding scheme information and thus can leverage the compression provided by the encoding scheme. Mask coercion is achieved through scaling (e.g., normalization) and biasing techniques. SPADE may have a unique effect in contexts where signals are masked and then dispersed before being detected (as a left shear measurement 601). By normalizing and biasing the encoding mask, it may be possible to make better use of the mask information in the reconstruction of the hyperspectral data cube.

[0028] Performing spatial adaptive normalization (SPADE) may involve performing a first quadratic convolution on the representation of the coding scheme and / or on any sheared or unsheared field of view. Performing SPADE may also involve performing a second quadratic convolution on the output of the first quadratic convolution. Performing SPADE may also involve performing an element-wise multiplication between the output of the second quadratic convolution and the output of a first-order convolution or transposed convolution. Performing SPADE may also involve summing the output of the second quadratic convolution with the output of the element-wise multiplication.

[0029] The method may include resizing the mask representation to match the resolution of the output of a linear convolution or transposed convolution operation.

[0030] Inverse attentional normalization may comprise a convolution or transpose convolution operation applied to the input features. Performing inverse attentional normalization may include one or more spatial adaptive normalization operations applied to the input features. The method may include one or more convolution operations applied sequentially to the result of the final spatial adaptive normalization operation.

[0031] The reverse attention normalization method may include either a sigmoid operation and / or a softmax activation operation applied to the result of the last convolution applied previously. This method may include element-wise multiplication between the output of the first convolution operation and the result of the activation operation. Performing reverse attention normalization may include one or more convolution operations and / or transpose convolution operations applied sequentially to the result of the element-wise multiplication.

[0032] The method may include performing one or more residual convolution operations using either or both of the encoder and decoder units. Residual convolution operations may enable the neural network to extract features from the results of preceding layers within the neural network. This may generate nonlinearity and complexity in the neural network, which may enable the neural network to identify these features in the data. Residual convolution operations may also aid in model learning through backpropagation.

[0033] Performing a residual convolution operation may include providing an input, performing one or more convolution operations, and combining the input and the outputs of a series of consecutive convolution operations.

[0034] Combining an input with the output of one or more convolution operations may include summing, concatenating, or multiplying the input and the output of one or more convolution operations, or a combination of such summing, concatenating, or multiplying operations.

[0035] The input may be, or comprise, either the output of a linear convolution operation or a transposed convolution operation, or the output of spatially adaptive normalization.

[0036] The method may include providing a skip coupling between the encoder unit and the decoder unit.

[0037] The method may include correcting the encoded and spectral shear images for disparity using the method of the third embodiment.

[0038] In a further embodiment, a computer implementation method is provided for training one or more neural networks from the first, second, or third embodiment. The method includes training the neural network using one or more of the following: i) L2 loss function, ii) Alignment loss function, iii) Compression measurement loss function, and iv) Relative normalized loss function.

[0039] The method may include training a neural network using a weighted sum of at least two different loss functions.

[0040] According to one embodiment, a computer implementation method for reconstructing a hyperspectral data cube is provided. The method includes acquiring a spectrally unsheared image of a scene. The method also includes acquiring at least one spectrally sheared image of the scene. The method further includes using the spectrally unsheared image and the at least one spectrally sheared image to reconstruct a hyperspectral data cube corresponding to the scene.

[0041] Using both at least one spectral sheared image of the scene and a spectrally unsheared image of the scene allows for improved reconstruction of the scene-specific hyperspectral datacube by providing additional information about the reconstruction. The spectrally unsheared image can also provide a simple visual comparison of the reconstructed hyperspectral datacube.

[0042] The method may include reconstructing a hyperspectral data cube using a neural network.

[0043] The method may include determining an estimated hyperspectral data cube of a scene using at least one spectral sheared image of the scene. The method may include providing the estimated hyperspectral data cube of the scene and an unsheared image of the scene to a neural network having an encoder-decoder structure having an encoder unit and a decoder unit. Reconstructing a hyperspectral data cube corresponding to the scene using the spectral unsheared image and at least one spectral sheared image may include determining an improved hyperspectral data cube of the scene from the estimated hyperspectral data cube and the unsheared image of the scene.

[0044] The estimated cube can arise from either a previous point step in the reconstruction model or from an external source.

[0045] In this context, an external source is defined as any hyperspectral dataset collected or estimated by means outside the scope of the reconstruction algorithm / model. This extension is possible because a secondary improved reconstruction (combined reconstruction) can work on any hyperspectral dataset, i.e., not just the results of previous reconstruction attempts. In addition, this can be further extended to include any set of features or data cubes generated by previous steps in the reconstruction, provided they have the expected shape.

[0046] Reconstructing an improved hyperspectral data cube from an estimated hyperspectral data cube can improve the inference time of the improved hyperspectral data cube (e.g., reduce reconstruction time) by separating the hyperspectral data cube reconstruction into two stages. The method can also provide an improved reconstructed hyperspectral data cube. Instead of starting with the detected raw data, information from both the estimated hyperspectral data cube and the spectrally unsheared image of the scene can be used to reconstruct the improved hyperspectral data cube. The estimated hyperspectral data cube may contain features or information already extracted or inferred from the detected raw data, which can be leveraged and combined with the spectrally unsheared image of the scene.

[0047] Alternatively, the method may be performed using multiple spectral shear images of the scene, for example, multiple estimated hyperspectral data cubes. The method may also be performed without using spectrally unsheared images of the scene. Multiple estimated hyperspectral data cubes may correspond to multiple different spectral shear fields of view of the scene. Performing tomographic reconstruction using multiple estimated hyperspectral data cubes may allow different information (e.g., different sets of extracted features from detected raw data, or information from multiple different fields of view of the scene) to be utilized to improve or enhance the reconstruction of the hyperspectral data cubes.

[0048] The method may include performing multiple convolution operations using an encoder unit. The method may also include performing multiple transposed convolution operations using a decoder unit.

[0049] The method may include performing multiple consecutive dilating-convolution operations using either an encoder or a decoder. The dilating ratio may increase with each consecutive dilating-convolution operation. Multiple consecutive dilating-convolution operations may provide a more compact neural network with virtually no loss in output resolution. A more compact neural network may improve the reconstruction speed of hyperspectral data cubes.

[0050] The method may include determining the estimated hyperspectral data cube using the method of the fourth or fifth embodiment.

[0051] The method does not necessarily include providing a neural network with mask information used to encode a spectral shear image of a scene. The method may include providing mask information indirectly to a neural network. The method may include providing mask information indirectly to a neural network via an estimated hyperspectral data cube determined using the method of the fifth embodiment.

[0052] Multiple consecutive convolution operations can be tightly coupled. Tight coupling between preceding, non-adjacent convolutional layers in a neural network can help stabilize learning. Alternatively or additionally, tight coupling can enable the transfer of higher-frequency features from previous, non-adjacent layers in the neural network. The output of each preceding tightly coupled convolution operation can be provided as input to a subsequent tightly coupled convolution operation. The output of each preceding tightly coupled convolution operation can be concatenated to provide input to a subsequent tightly coupled convolution operation.

[0053] The decoder unit may include a self-aware block configured to receive the final output of the encoder unit.

[0054] The method may include performing spatial adaptive normalization and / or inverse attentional normalization on the output of at least one of a convolution operation and / or transposed convolution operation using an unsheared image of the scene. SPADE and / or inverse attentional normalization using sheared or unsheared images of the scene may leverage the information provided by the unsheared image, thereby improving the spatial resolution of the improved hyperspectral data cube.

[0055] The method may include reconstructing a hyperspectral data cube using hierarchical channel reconstruction.

[0056] The method may include a relative normalization transformation in which a sigmoid operation can be applied to the reconstructed hypercube. The result of the sigmoid operation can be divided by its own reduced sum along its final axis.

[0057] The reduced total is determined as follows:

number

[0058] According to one embodiment, a computer implementation method for training a neural network of a fifth embodiment is provided. This method may include training the neural network using generative adversarial learning.

[0059] This method may involve training a neural network using cGAN, i.e., a conditional GAN ​​loss function.

[0060] In a further embodiment, a computer implementation method for training a neural network of the seventh or eighth embodiment is provided. The method may include training the neural network using one or more hyperspectral-specific loss functions.

[0061] The method may include training a neural network using at least one of the following: i) SSIM, i.e., the structural similarity exponential loss function, ii) SID, that is, spectral information divergence loss function, iii) HCR, i.e., hierarchical channel reconstruction loss function, and iv) Relative normalized loss function.

[0062] The method may include training a neural network using a weighted sum of at least two different loss functions.

[0063] According to the eighth aspect, a computer implementation method is provided for training a neural network of the third, fourth, or fifth aspect, separately from the neural network of the seventh aspect.

[0064] The method may include training a neural network to determine an improved hyperspectral data cube, separate from training a neural network to determine an estimated hyperspectral data cube.

[0065] The method may include training a neural network to determine an improved hyperspectral data cube using the method of the eighth or ninth embodiment. The method may also include training a neural network to determine an estimated hyperspectral data cube using the method of the sixth embodiment.

[0066] According to one embodiment, a computer or processor is provided that is configured to perform a method relating to any of the third to tenth embodiments, including any of its optional features.

[0067] The method described above may use the output from a hyperspectral imaging device comprising an input for receiving a light field from a scene, an encoder, at least one dispersive element, at least one array detector, and a processor. The encoder is configured to receive at least a portion of the light field from the input and transform it to provide first and second encoded light fields having different spatial patterns. At least one dispersive element is configured to apply a second spectral shear to the first and second encoded light fields, respectively, to provide first and second sheared light fields. At least one array detector is configured to detect the first and second sheared light fields. The processor is configured to process the output from at least one array detector to determine a data cube corresponding to a hyperspectral image of the scene.

[0068] The imaging device may be configured to acquire a data cube from a single shot (e.g., one frame of data from first and second array detectors). In some embodiments, the imaging device may be configured to acquire a data cube from multiple shots (e.g., from two shots, three or more shots).

[0069] Providing multiple encoding patterns within the detected light field enables tomographic reconstruction of the original hyperspectral data cube, thereby improving fidelity compared to methods that capture data based on a single encoding pattern.

[0070] The encoder may be configured to receive non-dispersive or non-shear light from the scene. The encoder may be configured such that at least a portion of the light field from the scene received by the encoder is substantially non-dispersive or non-shear. Dispersion of the light field from the scene is not required, or may not occur, before the light field from the scene is encoded. For example, the light field does not need to pass through a dispersive element such as a prism before it is received by the encoder.

[0071] At least one array detector may be configured to detect spectrally dispersed light. The shear light field detected by at least one array detector may be spectrally dispersed. The shear light field may not need to be recombined to form a spectrally nondispersive light field before detection by at least one array detector. The captured signal consists of the sum of multiple shifted images of the scene, each image may be shifted according to its wavelength.

[0072] In some embodiments, more than two encoded write fields may be provided and detected, for example, there may be four different encoded write fields (each possibly having a different encoding).

[0073] In some embodiments, a single detector and a reconfigurable encoder arrangement may exist. In these embodiments, the first and second spatial patterns are temporally separated, and the encoder can be reconfigured between detections of the first and second shear light fields. In these embodiments, a single dispersive element may be used (for example, between the reconfigurable encoder and the single detector).

[0074] In other embodiments, the encoder may be arranged to provide spatially separated first and second encoded write fields.

[0075] The detector resolution may be greater than or equal to the encoder resolution. The frame rate of hyperspectral image acquisition using the device of the first embodiment may be limited simply by the frame rate of the detector used (e.g., camera system or sensor). For example, if the detector frame rate is 200 Hz, the frame rate of the determined or reconstructed hyperspectral data cube may also be 200 Hz.

[0076] The first and second light fields may have complementary spatial patterns (regardless of whether the first and second encoded light fields are spatially or temporally separated). The complementary spatial patterns may be complementary random or pseudorandom spatial patterns.

[0077] The encoder may be configured to reflect the first and second encoded light fields in different directions.

[0078] Some embodiments may combine the spatial and temporal separation of encoded light fields by detecting both temporally and spatially separated encoded light fields (for example, by a reconfigurable encoder that provides spatially separated first and second light fields, and by collecting multiple shots from detectors having different encoder patterns).

[0079] Advantageously, the generation of first and second copies of the light field, encoded with complementary spatial patterns and separately sheared before detection, provides greater fidelity in data cube reconstruction. This is analogous to tomography, where the first and second copies effectively correspond to projections of the data cube along different directions, thereby providing more information about the data cube than when only a single sheared and encoded copy is used. The copies of the light field are from the same scene. Further segmentation of the light field from the scene is not required. The combination of encoding and shearing involving the detection of two or more copies may be referred to as compressed tomography.

[0080] At least one dispersive element may include a first dispersive element configured to apply a first spectral shear to a first encoded light field, and a second dispersive element configured to apply a second spectral shear to a second encoded light field.

[0081] At least one of the dispersion elements may be a transmissive dispersion element, or may include one. One or both of the first and second dispersion elements may be transmissive dispersion elements.

[0082] The first and second spectral shears may have different magnitudes.

[0083] The first and second spectral shears may have different spatial directions. The first and second dispersion elements may each be configured to spectrally shear the first and second encoded light fields in different directions. The first dispersion element may be configured to spectrally shear the first encoded light field in a first spatial direction, and the second dispersion element may be configured to spectrally shear the second encoded light field in a second spatial direction different from the first spatial direction.

[0084] The encoder may comprise a digital micromirror device, a static mask, and a liquid crystal device (e.g., liquid crystal on silicon).

[0085] The encoder may be or may comprise a transmissive encoder. Both the encoder and at least one dispersive element may be transmissive. The use of transmissive components may allow for easier miniaturization of hyperspectral imaging devices, particularly when performing compressed tomography.

[0086] The encoder may comprise a first encoder section configured to provide a first encoded light field, and a second encoder section configured to provide a second encoded light field. The first and second encoder sections may be arranged on or along first and second separate imaging paths, respectively. The first and second separate imaging paths may be parallel to each other. Separate imaging paths may eliminate the need for reflective elements such as beam splitters, thereby potentially enabling easier miniaturization of hyperspectral imaging devices.

[0087] The first and second dispersion elements may be arranged along the first and second separate imaging paths, respectively.

[0088] The hyperspectral imaging device may further include a spectral encoder configured to spectrally encode first and second shear light fields before detection by at least one array detector. This allows the hyperspectral imaging device to encode in both the spatial and spectral domains. This provides additional degrees of freedom in encoding, which may enable greater incoherence (randomness) in sampling the data cube, and subsequently improve the reconstruction of the data cube.

[0089] A spectral encoder may comprise a first spectral encoder unit configured to spectrally encode a first shear light field, and a second spectral encoder unit configured to spectrally encode a second shear light field. The first and second spectral encoder units may be arranged on or along separate first and second imaging paths, respectively. The spectral encoder may be a transmissive encoder.

[0090] At least one array detector may comprise first and second array detectors, each arranged to detect first and second shear light fields, respectively.

[0091] The hyperspectral imaging device may further comprise a beam splitter and a third array detector, the beam splitter being positioned between the input and the encoder, and configured to provide a portion of the light field to the third array detector and the remainder to the encoder.

[0092] A portion of the light field supplied to the third array detector may be non-shear (or spectrally non-dispersive).

[0093] The hyperspectral imaging device may further include a focusing or relay element positioned between the input and the encoder.

[0094] The focusing element can be positioned to project the scene onto the encoder.

[0095] At least one of the dispersion elements may include a concave grid.

[0096] At least one dispersion element may comprise a combination of a focusing element (e.g., refraction or reflection) and a planar grating. The planar grating may comprise a transmissive grating, a double amici prism, and the like.

[0097] At least one dispersive element may be positioned to project the first and second shear light fields onto at least one array detector, respectively. For example, the first and second dispersive elements may be configured to project the first and second shear light fields onto the first and second array detectors, respectively.

[0098] The encoder and at least one dispersion element can be integrated into a single component. This single component can be configured to provide first and second light fields that are both encoded and sheared. This may further facilitate the miniaturization of hyperspectral imaging devices.

[0099] An integrated encoder and at least one dispersive element may comprise an encoding pattern disposed on at least one dispersive element. The at least one dispersive element may be a diffraction grating. The encoding pattern may be lithographically printed on at least one dispersive element.

[0100] The processor may be configured to determine the data cube by solving a minimization problem.

[0101] Minimization problems can be equipped with regularizers that promote sparsity.

[0102] A processor can be configured to solve a minimization problem of the following form:

number

[0103] The processor may further be configured to process the output from a third detector along with the outputs from the first and second detectors to determine the data cube. In this embodiment, the processor may be configured to solve a minimization problem of the following form:

number

[0104] According to another aspect of this disclosure, a method for acquiring hyperspectral images is provided, and this method is: Receiving a light field from the scene, The light field is transformed to provide first and second encoded light fields having different spatial patterns. Applying spectral shear to the first and second encoded light fields respectively provides the first and second sheared light fields, To detect the first and second shear light fields and provide detection data, The detection data is processed to determine a data cube corresponding to the hyperspectral image of the scene, Includes.

[0105] Features of any embodiment (including optional features) may be combined with those of any other embodiment as needed. Features described with reference to the imaging device of the first embodiment may be used in the method of the second embodiment (for example, the method may acquire a data cube from a single shot, etc.). [Brief explanation of the drawing]

[0106] Exemplary embodiments are described only as illustrative examples with reference to the drawings. [Figure 1] This is a schematic diagram of a hyperspectral imaging device according to the present disclosure, employing a reflective encoder. [Figure 2] This is a schematic diagram of a hyperspectral imaging device according to another embodiment. [Figure 3] This is a schematic diagram of a hyperspectral imaging device according to an embodiment in which the encoder comprises multiple transmissive encoder units. [Figure 4] This is a schematic diagram of a hyperspectral imaging device according to an embodiment in which the encoder comprises multiple transmissive encoder units. [Figure 5] This is a schematic diagram of a hyperspectral imaging device according to an embodiment in which an encoder and a dispersive element are combined. [Figure 6] This is a block diagram of a computer implementation method for determining hyperspectral data cubes. [Figure 7] This is a block diagram of a conversion block that converts shear measurements and takes parallax into account. [Figure 7A] This is a block diagram of CONV_BLOCK. [Figure 8] This is a block diagram of a single shear reconstruction block. [Figure 8A] This is a block diagram of SPADE_BLOCK. [Figure 8B] This is a block diagram of RESIDUAL_CONV_BLOCK. [Figure 9A] A block diagram of the combined reconstruction blocks is shown, which determines the final hyperspectral data cube from at least one estimated hyperspectral data cube and a preview image. [Figure 9B] A block diagram of the combined reconstruction blocks is shown, which determines the final hyperspectral data cube from at least one estimated hyperspectral data cube and a preview image. [Figure 9C] The block diagram for DENSE_CONV_BLOCK is shown. [Figure 9D] The block diagram for DCP_BLOCK is shown. [Figure 9E] The block diagram for HCR_BLOCK is shown. [Figure 10A] This is a block diagram of a single shear reconstruction block in another embodiment using attention normalization. [Figure 10B] Figure 10A is a block diagram of the reverse caution block used in the single shear reconstruction block. [Figure 11] In another embodiment, this is a block diagram of a computer implementation method for determining a hyperspectral data cube. [Figure 12] This is a block diagram of relative transformation blocks that can be used after a combined reconstruction block to determine the final hyperspectral data cube from at least one estimated hyperspectral data cube and a preview image.

[0107] Please note that the figures are schematic and not drawn to scale. The same reference numerals are generally used to refer to corresponding or similar features in modified and different embodiments. [Modes for carrying out the invention]

[0108] Figure 1 shows a hyperspectral imaging device 100 according to an embodiment of the present disclosure. The hyperspectral imaging device 100 enables single-shot hyperspectral imaging. The hyperspectral imaging device 100 includes an input 102 for receiving a light field 104 from a scene 106, an encoder 108, a first dispersive element 110 and a second dispersive element 112, a first array detector 114 and a second array detector 116, and a processor 118.

[0109] Input 102 may have an aperture (or, for example, a slit) and is configured to direct light from the scene towards encoder 108.

[0110] Encoder 108 is configured to receive at least a portion of the light field 104 from the input and convert it to provide spatially separated first encoded light field 120 and second encoded light field 122 having unique and distinct binary spatial patterns. Encoder 108 may be a binary encoder, which comprises an array of reflective elements (e.g., mirrors) that direct light in either a first direction 120 or a second direction 122 different from the first direction 120. For example, encoder 108 may consist of a plurality of first type reflective elements and a plurality of second type reflective elements. There may be similar (e.g., equal) numbers of first and second type reflective elements, but this is not required. The first type of elements may be configured to reflect light incident on encoder 108 in the first direction 120. The second type of elements may be configured to reflect light incident on encoder 108 in the second direction 122. Light reflected in the first direction 120 comprises a first encoded light field, and light reflected in the second direction comprises a second encoded light field.

[0111] In some embodiments, the reflective elements may be fixed (for example, encoder 108 may have a fixed mirror array). In other embodiments, the patterns of the first and second types of reflective elements may be reconfigurable. For example, encoder 108 may have an array of movable micromirrors, such as a digital micromirror device (which can move quickly between a first angular position and a second angular position). A reconfigurable encoder 108 may be advantageous because additional information about a relatively slowly changing scene can be obtained by using different coding patterns to acquire a hyperspectral image cube and by combining the resulting data (i.e., by using multiple shots).

[0112] In other embodiments, the encoder 108 does not have to be a fully reflective type. For example, in some embodiments, a beam splitter may be used to provide light to the aperture of a first array and the aperture of a second array complementary to the first array. The encoder in this type of embodiment comprises the apertures of the first and second arrays. In some embodiments, a partially reflective encoder may be employed, in which a certain percentage (e.g., 50%) of the incident light is encoded and transmitted, and a certain percentage of the incident light is encoded and reflected.

[0113] Returning to Figure 1, the first and second dispersive elements 110 and 112 are arranged to apply the first and second spectral shears to the first and second encoded light fields, respectively, to provide the first sheared light field 124 and the second sheared light field 126. The dispersive elements 110 and 112 may comprise reflective dispersive elements such as diffraction gratings, but any dispersive element (including transmissive dispersive elements) can be used. It may be advantageous, but not essential, that the first and second spectral shears be different (e.g., positive and negative).

[0114] The first and second dispersion elements 110 and 112 may be curved diffraction gratings configured to project the first and second shear light fields onto their respective detectors. In other embodiments, the first and second dispersion elements 110 and 112 may each comprise a flat diffraction grating and a focusing element (e.g., a lens or mirror), the focusing element configured to project the shear light field onto the detector.

[0115] The first and second array detectors 114 and 116 are arranged to detect the first and second shear light fields, respectively. The processor 118 is arranged to process the outputs from the first and second detectors 114 and 116 to determine a data cube 128 corresponding to the hyperspectral image of the scene.

[0116] The two shear light fields 124 and 126 can be expressed as follows:

number

[0117] The hyperspectral data cube 128 can be reconstructed by the processor 118. The data acquisition process can be described as follows:

number

number

[0118] By using two detectors and employing a complementary coding scheme, single-shot hyperspectral images can be acquired without wasting incident light, enabling efficient tomographic reconstruction of the hyperspectral data cube. This is because the coding of the light detected by the first and second detectors is complementary.

[0119] Figure 2 shows a further embodiment 200, which also includes a beam splitter 130. Furthermore, lenses 134 and 136 are included (not all of which are labeled). The descriptions of elements with the same reference numbers in Figure 1 are similarly applicable to Figure 2.

[0120] The beam splitter 130 divides the light field 104 into a first portion 104a and a second portion 104b. The second portion 104b is directed to the encoder 108, while the first portion 104a is directed to a third array detector 132. The third array detector 132 may be configured to obtain a direct image (i.e., the sum of intensities from an unsheared image across the entire spectral range at each x,y pixel position). The signal detected by the third detector may be expressed as follows:

number

[0121] The data acquisition process using an additional third detector can be described as follows:

number

[0122] The reconstruction of the image data cube can be performed by solving a minimization problem (using the same notation as in (4)).

number

[0123] In the embodiment shown in Figure 2, a focusing lens 134 is shown that focuses the light from input 102 onto the encoder 108 and the third detector 132. This focusing lens can also be used in the embodiment of Figure 1 to focus the light onto the encoder 108. Although a refractive lens is shown, a focusing element with a reflector may be used instead.

[0124] The addition of a third detector that acquires direct images provides further information about the reconstruction of the hyperspectral data cube. These direct images are also direct and can be visually compared with the output data cube to provide confidence that they are correct.

[0125] Furthermore, two focusing lenses 136 are depicted in Figure 2 for each of the optical paths from the encoder 108 to the first and second array detectors 114 and 116. A similar arrangement of lenses can also be used in the embodiment of Figure 1. The lens between the encoder 108 and the first dispersive element 110 collimates the light from the encoder 108 to the first dispersive element 110. A further lens 136 between the first dispersive element 110 and the first array detector 114 focuses the light from the first diffracting element 110 to the first array detector 114. A similar arrangement is used for other detection paths (leading to the second array detector 116).

[0126] Preferably, the lenses coincide at a focal length f2, so that the distances from the dispersive elements 110, 112 to the encoder 108 and the array detectors 114, 116 are the same (the lenses are positioned midway between the encoder and the dispersive elements, and midway between the dispersive elements and the first array detectors).

[0127] While refractive lenses are depicted in the exemplary embodiment shown in Figure 2 (which may be readily available, compact, and low-cost), reflective elements may be used instead, which may have the advantage of not introducing any unwanted dispersion.

[0128] Figure 3 shows another embodiment of the hyperspectral imaging device 300. The descriptions of elements having the same or similar reference numbers in Figures 1 and 2 are similarly applicable to Figure 3. The hyperspectral imaging device 300 comprises an input 102 that receives a light field 104 from a scene 106, an encoder 308, a first dispersive element 310 and a second dispersive element 312, a first array detector 114, a second array detector 116, and a third array detector 132, and a processor 118. The hyperspectral imaging device 300 operates using the same principles as the hyperspectral imaging devices 100 and 200 described above, but utilizes transmissive elements instead of reflective elements. The use of transmissive elements can enable easier miniaturization (and potentially cost reduction) of the hyperspectral imaging device 300, particularly when performing compressed tomography, by avoiding the use of reflective elements (such as digital micromirror devices, beam splitters, and reflective diffraction gratings).

[0129] In the illustrated embodiment, the hyperspectral imaging device 300 comprises separate, isolated first imaging path 301a, second imaging path 301b, and third imaging path 301c from scene 106 to the respective array detectors 114, 116, and 132. In the illustrated embodiment, the separate imaging paths 301a, 301b, and 301c are parallel to each other to avoid the need for a beam splitter, which may further facilitate miniaturization of the hyperspectral imaging device 300. The directions of the arrows from input 102 along each imaging path 301a, 301b, and 301c are essentially schematic and do not necessarily indicate the physical paths of light through the hyperspectral imaging device 300.

[0130] The encoder 308 comprises a first encoder section 308a and a second encoder section 308b. The first encoder section 308a operates as part of a separate first imaging path 301a in relation to the first dispersive element 310 and the first array detector 314. In the illustrated embodiment, the imaging lens 340a is located between the input 102 and the first encoder section 308a, and the relay lens 342a is located between the first encoder section 308a and the first dispersive element 310. In other embodiments, the imaging lens 340a and the relay lens 342a may be omitted. Similarly, the second encoder section 308b operates as part of a separate second imaging path 301b in relation to the second dispersive element 312 and the second array detector 116. In the illustrated embodiment, the imaging lens 340b is positioned between the input 102 and the second encoder unit 308b, and the relay lens 342b is positioned between the second encoder unit 308b and the second dispersion element 312. In other embodiments, the imaging lens 340b and the relay lens 342b may be omitted.

[0131] The first encoder unit 108a and the second encoder unit 108b are each arranged to receive at least a portion of the light field 104 from the input 102 and convert it to provide the first encoded light field 120 and the second encoded light field 122, respectively. The first encoded light field 120 and the second encoded light field 122 have different spatial patterns. In the illustrated embodiment, the first encoder unit 308a and the second encoder unit 308b are transmissive encoders, for example, apertures or masks of the first and second arrays. In some embodiments, the first encoder unit 308a and the second encoder unit 108b are complementary to each other and generate the first encoded light field 120 and the second encoded light field 122 having complementary spatial patterns, but this is not required.

[0132] The first dispersive element 110 and the second dispersive element 112 are arranged to apply the first and second spectral shears to the first encoded light field 120 and the second encoded light field 122, respectively, to provide the first sheared light field 124 and the second sheared light field 126. In the illustrated embodiment, the first dispersive element 110 and the second dispersive element 112 are each transmission dispersive elements such as transmission diffraction gratings. It may be advantageous, but is not essential, that the first and second spectral shears be different (e.g., positive and negative).

[0133] The first array detector 114 and the second array detector 116 are arranged to detect the first shear light field 124 and the second shear light field 126, respectively. The processor 118 is arranged to process the outputs from the first detector 114 and the second detector 116 to determine a data cube 128 corresponding to the hyperspectral image of the scene. The processor 118 may reconstruct the hyperspectral data cube 128 according to equations (1) to (7) described above.

[0134] A separate third imaging path 301c extends from the input to a third array detector 132 that provides a direct image of scene 106, providing further information about the reconstruction of the hyperspectral data cube 128. The direct image is direct and can be visually compared with the output data cube 128 to provide confidence that the data cube 128 is correct. In the illustrated embodiment, an imaging lens 340c is located between the input 102 and the third array detector 132. In some embodiments, the imaging lens 340c may be omitted. In some embodiments, the third imaging path 301c may not be present or utilized.

[0135] Figure 4 shows a further embodiment of the hyperspectral imaging device 400. The hyperspectral imaging device 400 is substantially similar to the hyperspectral imaging device 300 described above, except that the imaging lens and relay lens are not depicted.

[0136] The hyperspectral imaging device 400 further comprises a spectral encoder 440. The spectral encoder 440 is configured to spectrally encode a first shear light field 124 and a second shear light field 126 before they are detected by the first array detector 114 and the second array detector 116, respectively. In the illustrated embodiment, the spectral encoder 440 is located after the respective dispersive elements 310, 312 in the first imaging path 301a and the second imaging path 301b. In the illustrated embodiment, similar to encoder 308, the spectral encoder 440 comprises a first spectral encoder unit 440a configured to spectrally encode the first shear light field 124, and a second spectral encoder unit 440b configured to spectrally encode the second shear light field 126. The first spectral encoder unit 440a operates as part of the first imaging path 301a, while the second spectral encoder unit 440b operates as part of the second imaging path 301b. In some embodiments, the first spectral encoder unit 440a and the second spectral encoder unit 440b are complementary to each other, but this is not essential.

[0137] In the illustrated embodiment, the spectral encoder 440 is a transmissive encoder, for example, an aperture of one or more arrays. As described above, the use of a transmissive component may allow for easier miniaturization of the hyperspectral imaging device 400. In other embodiments, the spectral encoder may be a reflective encoder, for example, a digital micromirror device.

[0138] The hyperspectral imaging device 300 described above encodes only the spatial domain using the encoder 308. The spectral encoder 440, in conjunction with the encoder 308, enables the hyperspectral imaging device 400 to encode in both the spatial and spectral domains. The additional degrees of freedom in encoding may allow for greater incoherence (randomness / orthogonality) in the sampling of the data cube 128, thereby improving reconstruction.

[0139] Alternatively, the spectral encoder 440 may be used independently of the encoder 308 (which may be omitted from the hyperspectral imaging device 400) to provide spectral coding only. It will also be understood that spectral encoders (such as spectral encoder 440) may be implemented in the hyperspectral imaging devices 100, 200 described above. The spectral encoder may be implemented as a single spectral encoder or as multiple spectral encoder units. For example, if the first coded light field 120 and the second coded light field 122 (resulting in the first shear light field 124 and the second shear light field 126) are temporally separated, a single spectral encoder 440 may be employed to spectrally encode the first shear light field 124 and the second shear light field 126.

[0140] In the embodiments described above, encoding and dispersion are performed by two separate, isolated components, namely encoders 108, 308 and at least one dispersion element 110, 112, 310, 312. Figure 5 shows an embodiment of the hyperspectral imaging device 500. The hyperspectral imaging device 500 is similar to the embodiments shown in Figures 3 and 4.

[0141] However, in the embodiment shown in Figure 5, the encoder 308 and at least one dispersive element 310, 312 are integrated into a single component, which is an integrated coding and dispersive element 550. The integrated coding and dispersive element 550 provides a first coding and sheared light field 552 and a second light field 554. In effect, the integrated coding and dispersive element 550 provides both the first coding light field 120 and the first sheared light field 124 as a single first coding and sheared light field 552 (correspondingly, the second coding light field 122 and the second sheared light field 126 as a single second coding and sheared light field 554).

[0142] In the illustrated embodiment, the first encoder unit 308a and the first dispersive element 310 on the first imaging path 301a are replaced by the first integrated coding and dispersive element 550a (correspondingly, the second encoder unit 308b and the second dispersive element 312 on the second imaging path 301b are replaced by the integrated coding and dispersive element 550b).

[0143] In the illustrated embodiment, the integrated encoder and dispersion element 550 is a transmissive component. In some embodiments, the integrated encoding and dispersion element 550 may comprise a dispersion element (e.g., a diffraction grating such as a transmissive diffraction grating) on ​​which an encoding pattern is arranged (e.g., using a lithography process).

[0144] The integrated coding and dispersive element 550 shortens the beam path within the separate imaging paths 301a and 301b of the hyperspectral imaging device 500, which may further facilitate miniaturization of the hyperspectral imaging device 500.

[0145] Optionally, a spectral encoder, such as the spectral encoder 440 described above, may be used in conjunction with an integrated encoding and dispersion element 550 for encoding in both the spatial and spectral domains.

[0146] Figure 6 illustrates an alternative method according to the present invention for reconstructing a hyperspectral data cube using the outputs from the aforementioned hyperspectral imaging devices 100-500 (which may also be applicable in other contexts). The alternative method involves using machine learning for hyperspectral data cube reconstruction rather than compressed sensing reconstruction.

[0147] Figure 6 shows the architecture of an embodiment of machine learning model 600 that reconstructs a hyperspectral data cube using the outputs from hyperspectral imaging devices 100-500.

[0148] Figure 11 shows the architecture of an embodiment relating to another aspect of a machine learning model 600 that reconstructs a hyperspectral data cube using the outputs from hyperspectral imaging devices 100-500.

[0149] In the illustrated embodiment, Model 600 takes a left shear measurement 601, a preview measurement 602, and a right shear measurement 603 as inputs. Left shear and right shear are merely labels for convenience for different shear fields, and it will be understood that the left shear and right shear measurements may have the same shear direction but different amounts of shear, or they may have the same shear and different coding. It is preferable that the shear directions are different and the spatial coding is different.

[0150] The left shear measurement 601 corresponds to the output from one of the first and second array detectors 114, 116, while the right shear measurement 603 corresponds to the output from the other of the first and second array detectors 114, 116. The preview measurement 602 corresponds to the output of the third array detector 132 (e.g., a direct image of the scene). The left shear measurement 601 and the right shear measurement 603 do not necessarily have to be obtained from the first and second detectors 114, 116 which are spatially positioned to the left and right of the central third array detector 132.

[0151] Each of the left shear measurement 601 and preview measurement 602 is passed to a transformation block 700a. Similarly, each of the right shear measurement 603 and preview measurement is passed to a transformation block 700b. Transformation blocks 700a and 700b are configured to determine and apply a transformation (e.g., a perspective transformation) to correct the parallax between each of the left shear measurement 601 and right shear measurement 603 and preview measurement 602. The measurements described may be captured using a monochrome imaging system and / or a filter array imaging system, including but not exclusive to a Bayer filter camera system.

[0152] Here, we will describe conversion block 700a in more detail.

[0153] Figure 7 shows an embodiment of the transformation block 700a. The transformation block 700a comprises a machine learning (ML) algorithm configured to determine the transformation and correct the disparity between the left shear measurement 601 and the preview measurement 602. Because the left shear measurement 601 is encoded and distributed, while the preview measurement is a direct image (e.g., not encoded and / or distributed), conventional analytical and computer vision methods for correcting disparity (e.g., disparity due to parallax or misalignment) cannot be applied.

[0154] In the illustrated embodiment, the ML algorithm is a neural network, but other ML algorithms may be used. In step 701, the left shear measurement 601 and the preview measurement 602 are concatenated. In step 702, the convolution block CONV_BLOCK_1 performs a convolution operation on the concatenated image data. In the illustrated embodiment, the left shear measurement 601 and the preview measurement are concatenated by stacking the two images. This is reflected in the input dimensions to CONV_BLOCK_1, which have an input depth of 2.

[0155] In the illustrated embodiment, CONV_BLOCK_1 comprises 20 convolutional filters, but any number of filters may be used. Each filter has a stride that allows CONV_BLOCK_1 to reduce the dimensionality of the data (known as downsampling). In the illustrated embodiment, each filter has a stride of 2, allowing the filter to reduce the dimensionality of the data by half. This is reflected in the output spatial dimensions of CONV_BLOCK_1, which are 296 pixels × 296 pixels, compared to the input spatial dimensions of 592 pixels × 592 pixels. Alternatively, each filter may have a different stride, allowing the dimensionality of the data to be reduced at different rates. The output depth of CONV_BLOCK_1 corresponds to the number of filters used (20 filters in the illustrated embodiment). The filters are used to extract features from the concatenated image data so that a transformation (e.g., a perspective transformation) can be determined.

[0156] The structure and operation of the convolution block CONV_BLOCK_1 are shown in more detail in Figure 7A. In step 720, the convolution operation is performed using the convolution module Conv2d. The convolution operation involves advancing the kernel across the input data by moving the stride amount between steps. With each step the kernel advances, a matrix multiplication is performed between the kernel and the input data over which the kernel overlaps. The results of the matrix multiplications are summed, and the summed value represents a single "pixel" in the output data. This process is repeated until the kernel has been moved or advanced across the entire input data. Each kernel has three dimensions: a planar dimension corresponding to the height and width of the kernel (e.g., "kernel_size x kernel_size x input_shape[3] x filter"), and a depth corresponding to the number of stack layers in the input data (e.g., "input_shape[3]" in "kernel_size x kernel_size x input_shape[3] x filter"). The fourth "dimension" represents the number of different kernels or filters advanced across the entire input data (e.g., "filters" in "kernel_size x kernel_size x input_shape[3] x filter"). The number of kernels or filters is equal to the number of stack layers in the output data of the convolution module Conv2d. For example, if 20 different kernels are advanced across the entire input data, the output data will have 20 stack layers. Each kernel or filter contributes one layer in the stack of layers that make up the output data of the convolution module Conv2d.

[0157] The size of the stride determines the dimensionality reduction of the input data achieved by the convolution operation. This is evident from the output dimensions from the convolution module Conv2d, which are "input_shape[0], input_shape[1] / / stride, input_shape[2] / / stride, filter". input_shape[1] and input_shape[2] represent the height and width of the input data. A stride of 2 for the kernel results in a 1 / 2 dimensionality reduction between the input and output data. The output dimension "filter" represents the number of stack layers in the output data, with each different filter running across the data corresponding to one of the stack layers.

[0158] In step 721, the batch normalization module Batch_normalisation normalizes the output of the convolution module Conv2d (step 720). Conventional batch normalization may be employed. The parameters "alpha" and "beta" shown in Figure 7A represent the initial values ​​of the variance and mean for batch normalization. The initial values ​​may be updated during training. In the illustrated embodiment, each layer of input depth is normalized using batch normalization, but it will be understood that batch normalization may be implemented differently in other embodiments.

[0159] In step 722, the activation module Linear_rectifier_unit applies the ReLU (rectifier linear unit) activation function to each "pixel" of the data output by the normalization module Batch_normalisation (step 721). Using the ReLU activation function, if a pixel value is less than 0, the ReLU function outputs a value of 0 for that pixel. If a pixel value is 0 or greater, the ReLU function outputs the value for that pixel. Alternatively, the activation module may employ a different activation function. The ReLU function of the activation module adds nonlinearity to the output of the convolution operation. The output of the activation module Linear_rectifier_unit contains the final output of the convolution block CONV_BLOCK_1.

[0160] Returning to Figure 1, in step 703, the output of CONV_BLOCK_1 (step 702) is passed to the convolution block CONV_BLOCK_2 and the convolution Conv2d.

[0161] CONV_BLOCK_2 is configured to further extract features from the data, further reduce the data's dimensionality, and perform a convolution operation similar to that performed by CONV_BLOCK_1. In the illustrated embodiment, CONV_BLOCK_2 comprises 20 convolution filters, each having a stride of 2, which reduces the data's dimensionality by half. Alternatively, each filter may have a different stride, and / or a different number of filters may be used.

[0162] The convolution module Conv2d is also configured to perform convolution operations. The convolution operations performed by Conv2d reduce the data dimensionality at the same rate as CONV_BLOCK_2 reduces the data dimensionality.

[0163] In the illustrated embodiment, the convolution module Conv2d is similar to CONV_BLOCK_1, but lacks the normalization module Batch_normalisation or the activation module Linear_rectifier_unit. The output of the convolution module Conv2d is neither normalized nor passed through an activation function. The convolution module Conv2d uses a kernel or filter with a height and width of 1 pixel × 1 pixels. The kernel is advanced across the data using a stride of 2. The 1 pixel × 1 pixel kernel size can improve computational efficiency. The convolution operation of Conv2d does not have to be intended to extract features like conventional convolution operations. Rather, the convolution operation of Conv2d ensures that the outputs of the convolution modules CONV_BLOCK_2 and Conv2d are the same size and shape. Alternatively, a pooling operation may be used instead of the convolution operation of Conv2d. However, a pooling operation does not utilize trainable weights or parameters. By reducing the data size through convolution operations using Conv2d, more weights and parameters are provided that can be trained to improve transformation decisions.

[0164] Providing a convolutional block CONV_BLOCK_2 with a nonlinear output and a convolutional module Conv2d with a linear output in parallel (via the activation module Linear_rectifier_unit) provides improved stability and learning of the neural network during backpropagation. If backpropagation and learning through CONV_BLOCK_2 (with a nonlinear output) results in a very high rate of change or no rate of change during learning, learning can instead be backpropagated through Conv2d, which has a simpler linear output. Different parallel paths can create differences in how gradients flow along the paths during backpropagation and may provide improved stability for backpropagation.

[0165] Returning to Figure 7, in step 704, the outputs of the convolution block CONV_BLOCK_2 and the convolution module Conv2d (step 703) are summed. In step 705, CONV_BLOCK_3 performs a convolution operation on the summed outputs of CONV_BLOCK_2 and Conv2d (step 704). CONV_BLOCK_3 is configured to further extract features from the data and further reduce the dimensionality of the data, performing a convolution operation similar to that performed by CONV_BLOCK_1 and CONV_BLOCK_2. In the illustrated embodiment, CONV_BLOCK_3 comprises 20 convolution filters, each having a stride of 2, which reduces the dimensionality of the data by half. Alternatively, each filter may have a different stride, and / or a different number of filters may be used. Alternatively, a different number of convolution blocks may be used in the transformation block 700a. For example, there may be one, two, four or more convolution blocks in the transformation block 700a.

[0166] In step 706, the pooling block AveragePooling2D performs a pooling operation on the output of CONV_BLOCK_3 (step 705). In the illustrated embodiment, the pooling operation is an average pooling operation. Alternatively, the pooling operation may be a maximum pooling operation, which obtains the maximum pixel value from each 2x2 pixel area. In the illustrated embodiment, AveragePooling2D is configured to obtain an average pixel value from each of a number of 2x2 pixel areas in the input data it receives. Alternatively, the pooling block AveragePooling2D may be configured to obtain an average pixel value from areas of different dimensions, for example, 3x3 pixels.

[0167] In step 707, the data reconstruction block Reshape reconstructs the output of the AveragePooling2D pooling block (step 706). In the illustrated embodiment, the reconstruction block Reshape is configured to reconstruct the data by incorporating the values ​​of each pixel in each of the 20 image stacks output by AveragePooling2D, each a 2D 37x37 pixel image (one image for each filter is used in the CONV_BLOCK_3 convolution operation), into a 1D column vector with 27,380 values. The exact number of values ​​in the column vector depends on the parameters of the preceding operation (e.g., the number of filters in each convolution operation, the filter stride, etc.).

[0168] In step 708, a first tightly coupled layer, Dense_layer_1, is provided. Dense_layer_1 has a number of nodes. In the illustrated embodiment, Dense_layer_1 has 54 nodes, but a different number of nodes may be used. Each node in Dense_layer_1 is coupled to each value in a one-dimensional column vector of the reconstructed data (output of step 707). In the illustrated embodiment, matrix multiplication is used to couple each node in Dense_layer_1 to each value in the one-dimensional column vector. The weights for the couple between each node and each value determine the output of each node in Dense_layer_1.

[0169] In step 709, a second tightly coupled layer, Dense_layer_2, is provided. Dense_layer_2 has a number of nodes. Each node in Dense_layer_2 is coupled to each node in Dense_layer_1. In the illustrated embodiment, matrix multiplication is used to couple each node in Dense_layer_2 to each node in Dense_layer_1. The weights for the couples between each node in Dense_layer_1 and Dense_layer_2 determine the output of each node in Dense_layer_2. In the illustrated embodiment, Dense_layer_2 has eight nodes. The eight nodes represent eight values ​​of a 3x3 affine transformation matrix that can be used to perform a transformation (e.g., a perspective transformation) on the left shear measurement 601. Since the mapping between the left shear measurement 601 and the preview measurement 602 can be performed using an affine transformation, the ninth value of the transformation matrix is ​​always 1 in the illustrated embodiment. The ninth value is appended to the eight node outputs of Dense_layer_2. Alternatively, one or more intermediate dense layers may be provided between Dense_layer_1 and Dense_layer_2. Alternatively, a single dense layer may be provided to extract the transformation parameters.

[0170] In step 710, the output of Dense_layer_2 is provided to a transformation matrix used to perform a transformation (e.g., a perspective transformation) on the left shear measurement 601. This transformation step provides a transformed left shear measurement 601a that provides an approximate inter-pixel mapping between the left shear measurement 601 and the preview measurement 602. The left shear measurement 601 is transformed to match the shape of the preview measurement 602. Transformation block 700b operates in a similar manner to determine a transformation (e.g., a perspective transformation) and provides a transformed right shear measurement 602a that provides an approximate inter-pixel mapping between the right shear measurement 603 and the preview measurement 602. Transformation block 700b may, for example, determine a transformation (e.g., a perspective transformation) that is effectively a mirror of the transformation (e.g., a perspective transformation) determined by transformation block 700a, depending on the arrangement of the components of the hyperspectral imaging devices 100-500.

[0171] In the hyperspectral imaging devices 100-500 described above, multiple fields of view of the same scene are acquired to enable tomographic reconstruction of hyperspectral data cubes corresponding to the scene. Using multiple fields of view can introduce differences between fields of view, for example, due to parallax (for multiple aligned parallel fields of view), misalignment (for example, between encoders and detectors used for each field of view), manufacturing tolerances, and the fact that light must pass through different components to reach each detector. Correcting these parallaxes enables improved (e.g., more accurate) hyperspectral data cube reconstruction. However, because light is encoded and dispersed before it is detected, directly finding the parameters of a transformation (e.g., perspective transformation) to correct for parallax at each detector is particularly difficult for hyperspectral imaging. The hyperspectral data cube needs to be reconstructed to determine the parallax, but a transformation is required to reconstruct the hyperspectral data cube. Existing compressed sensing and hyperspectral data cube reconstruction methods do not account for parallax and cannot reconstruct from different fields of view. Compressed sensing reconstruction methods also require a complete understanding of the mask features in the detectors. Furthermore, because one or more parts of the field of view are encoded and distributed, conventional parallax correction techniques (e.g., computer vision parallax correction techniques) cannot be applied.

[0172] Using machine learning techniques to determine a transformation (e.g., perspective transformation) to compensate for disparity (e.g., parallax or misalignment) between different fields of view can enable efficient iterative determination of the transformation. The use of machine learning techniques can also provide improved alignment robustness. As mentioned above, compressed sensing reconstruction methods require a complete prior understanding of the positions of encoder features in the detector, and misalignment between the mask and the sensor can make accurate reconstruction impossible. Using machine learning techniques to determine a transformation (e.g., perspective transformation) and provide a mapping to a common field of view base for different fields of view to compensate for arbitrary disparity (e.g., parallax or misalignment) means that a prior understanding of the difference is not required, and disparity is taken into account during the training process to determine the transformation. This may make it possible to accurately reconstruct the hyperspectral data cube even when the encoder is relayed to the sensor with an error of a few pixels. Accurate reconstruction of the hyperspectral data cube does not require the typically expensive and redundant alignment process required for hyperspectral imaging devices. The alignment robustness may also be advantageous in dealing with thermal expansion and / or deformation, vibration-induced noise, and overall robustness (for example, bumping or dropping a hyperspectral imaging device can typically impair alignment).

[0173] Although the transformation block 700a is presented as part of a larger system for hyperspectral data cube reconstruction, it will be understood that the transformation block 700a may be used independently or as part of a different system to correct disparity between direct (e.g., unencoded and non-spectrally dispersed) images and encoded and / or spectrally dispersed images.

[0174] Returning to Figure 6, the transformed left shear measurement 601a (output of transformation block 700a) is passed to the left single shear reconstruction block 800a. Similarly, the transformed right shear measurement 602a (output of transformation block 700b) is passed to the right single shear reconstruction block 800b. The single shear reconstruction blocks 800a and 800b are configured to reconstruct the hypercube from the transformed left and right shear measurements 601a and 602a.

[0175] Here, we will describe the single shear reconstruction block 800a in more detail.

[0176] Figure 8 shows an embodiment of a single shear reconstruction block 800a. In the illustrated embodiment, the single shear reconstruction block 800a includes a neural network. The neural network has an encoder-decoder structure comprising an encoder section 800c and a decoder section 800d.

[0177] In step 801, the convolution block CONV_BLOCK_1 of the encoder unit 800c performs a convolution operation on the transformed left shear measurement 601a. ​​CONV_BLOCK_1 of the encoder unit 800c is configured to perform a convolution operation similar to that performed by CONV_BLOCK_1, CONV_BLOCK_2, and CONV_BLOCK_3 of the transformation block 700a described above. In the illustrated embodiment, CONV_BLOCK_1 comprises 50 filters, but any preferred number of filters may be used. Each filter has a stride that allows CONV_BLOCK_1 to reduce the dimensionality of the data. In the illustrated embodiment, each filter has a stride of 2, but each filter may have a different stride.

[0178] CONV_BLOCK_1 is configured to extract features to aid in spectral reconstruction while reducing the dimensionality of the data (downsampling). This may allow for the extraction of features with low spatial and spectral frequencies. Downsampling can also increase computational efficiency, for example, by progressively reducing the amount of spectral and spatial data being trained and reconstructed.

[0179] In step 802, the SPADE block SPADE_BLOCK_1 performs spatial adaptive normalization (SPADE) on the output of CONV_BLOCK_1 (step 801). SPADE_BLOCK_1 is configured to apply the mask features used to encode the image of the left shear measurement 601 to the output of CONV_BLOCK_1.

[0180] The configuration and operation of SPADE_BLOCK_1 are shown in more detail in Figure 8A.

[0181] In step 830, the mask initially used to encode the image of the left shear measurement 601 is provided as a conditional input to SPADE_BLOCK_1. In step 831, if necessary, the mask code is resized to match the dimensions of the output of the preceding convolutional block CONV_BLOCK_1 (step 801) of encoder unit 800c.

[0182] In step 832, the convolution block CONV_BLOCK_1 of SPADE_BLOCK_1 performs a convolution operation on the (resized) mask, including normalization and activation steps (similar to steps 720-722 described above). In the embodiment shown in Figure 8, SPADE_BLOCK_1 of encoder unit 800c comprises 50 filters. The convolution operation of CONV_BLOCK_1 of SPADE_BLOCK_1 does not reduce the dimensions (height and width) of the resized conditional input data. CONV_BLOCK_1 of SPADE_BLOCK_1 generates a stack of output images corresponding to the number of filters used.

[0183] In step 833, the convolution module Conv2d_1 performs a convolution operation on the output of CONV_BLOCK_1 of SPADE_BLOCK_1 (step 832), similar to what is done by the convolution module Conv2d of the transformation block 700a (step 703) described above. No normalization or activation steps are performed. In the embodiment shown in Figure 8, Conv2d_1 of SPADE_BLOCK_1 comprises 50 filters.

[0184] In step 834, element-wise multiplication is performed between the output of Conv2d_1 (step 833) and the preceding output of CONV_BLOCK_1 (step 801) of encoder unit 800c.

[0185] In step 835, the convolution module Conv2d_2 performs a convolution operation on the output of CONV_BLOCK_1 (step 832) of SPADE_BLOCK_1. The convolution operation performed by Conv2d_2 is the same as the convolution operation performed by Conv2d_1 (step 833).

[0186] In step 836, the output of Conv2d_2 (step 835) and the output of element-wise multiplication (step 834) are summed. The summed output of step 836 forms the final output of SPADE_BLOCK_1 of encoder unit 800c. The final output of SPADE_BLOCK_1 has the same output dimensions as the output dimensions of CONV_BLOCK_1 (step 801) of encoder unit 800c.

[0187] Spatial adaptive normalization (SPADE), performed by SPADE_BLOCK_1 in the encoder unit, applies a mask used to encode the image of the left shear measurement 601 to the features extracted from the left shear measurement 601 by CONV_BLOCK_1 in the encoder unit 800c. SPADE forces the neural network to use the mask information and thus can leverage the compression provided by the mask. Mask coercion is achieved through a scaling and biasing technique that takes two inputs: an "original" input and a "conditional" input. The "original" input is the output of a preceding convolutional block (e.g., step 801) in the encoder unit 800c. The "conditional" input is the mask (which is resized as needed to match the dimensions of the "original" input).

[0188] The first convolution operation of SPADE_BLOCK_1 performed by CONV_BLOCK_1 extracts features from the "conditional" input. The second set of convolution operations of SPADE_BLOCK_1 performed by Conv2d_1 and Conv2d_2 generate the "scale" and "bias" from the extracted features.

[0189] The "scale" provided by element-wise multiplication in step 834 forces normalization of the weights between the mask and the features extracted via the convolution operation in step 801. The "bias" provided by summing in step 836 forces the mask to be adequately represented when compared to the features being masked.

[0190] SPADE blocks can have a unique effect in contexts where the signal is masked and then dispersed before being detected (as the left shear measurement 601). Normalizing and biasing the encoded mask may allow for more effective utilization of the mask information in the reconstruction of the hyperspectral data cube. However, SPADE blocks are not essential, and other techniques may be used in some embodiments.

[0191] In step 803, the residual convolution block RESIDUAL_CONV_BLOCK_1 performs a residual convolution operation on the output of SPADE_BLOCK_1 (step 802).

[0192] The configuration and calculations of RESIDUAL_CONV_BLOCK_1 are shown in more detail in Figure 8B.

[0193] In step 840, the input is provided to RESIDUAL_CONV_BLOCK_1. In the illustrated embodiment, the input comprises the output of SPADE_BLOCK_1 (step 802).

[0194] In step 841, the convolution block CONV_BLOCK_1 of RESIDUAL_CONV_BLOCK_1 performs a convolution operation on the input (step 840), including normalization and activation steps (similar to steps 720-722 described above). In the embodiment shown in Figure 8, RESIDUAL_CONV_BLOCK_1 of encoder unit 800c comprises 50 filters. The convolution operation of CONV_BLOCK_1 of RESIDUAL_CONV_BLOCK_1 does not reduce the dimensions (height and width) of the input data. CONV_BLOCK_1 of RESIDUAL_CONV_BLOCK_1 generates an output data volume having a depth equal to the number of filters used.

[0195] In step 842, the convolution block CONV_BLOCK_2 of RESIDUAL_CONV_BLOCK_1 performs a convolution operation on the output of CONV_BLOCK_1 of RESIDUAL_CONV_BLOCK_1 (step 841), including normalization and activation steps (similar to steps 720-722 above). The convolution operation of CONV_BLOCK_2 of RESIDUAL_CONV_BLOCK_1 does not reduce the data dimensions (height and width).

[0196] In step 843, the output and input (step 840) of CONV_BLOCK_2 (step 842) are summed. The summed output in step 843 provides the final output of RESIDUAL_CONV_BLOCK_1. The final output of RESIDUAL_CONV_BLOCK_1 has the same output dimensions as the output dimensions of CONV_BLOCK_1 and SPADE_BLOCK_1 of encoder unit 800c (step 801).

[0197] The residual convolution block RESIDUAL_CONV_BLOCK_1 is configured to enhance the features extracted by CONV_BLOCK_1 of the encoder unit 800c. The residual convolution block RESIDUAL_CONV_BLOCK_1 comprises a stack of convolutional blocks having a connection (sum in the illustrated embodiment) between the input of the residual convolutional block and the output of the final convolutional block in the residual convolutional block. The stack allows the neural network to extract features from the results of the preceding convolutional block CONV_BLOCK_1. This generates nonlinearity and complexity in the neural network, enabling it to identify these features in the data. The connection between the input and output of the residual convolutional block aids in model learning via backpropagation. However, the residual convolutional block is not mandatory, and in some embodiments, different methods may be used.

[0198] Steps 801-803 define a series of main blocks in encoder unit 800c, namely the convolution block, followed by the SPADE block, and then the residual convolution block. In the illustrated embodiment, steps 804-806 are substantially a mirror of steps 801-803. The main difference is that CONV_BLOCK_2 of encoder unit 800c further reduces the data dimension by half and has 100 filters instead of the 50 filters used by CONV_BLOCK_1 of encoder unit 800c. The series of steps, namely the convolution block, SPADE block, and residual convolution block, can be repeated any number of times as preferred. However, it will be understood that the different blocks do not need to be used in relation to one another, but can be used independently of one another or in different subcombinations.

[0199] In step 807, the transpose convolution block TransposeConv2d_2 of the decoder unit 800d performs a transpose convolution operation on the output of RESIDUAL_CONV_BLOCK_2 (step 806). The transpose convolution block is configured to increase the dimensionality or resolution of the data (i.e., upsample). In the illustrated embodiment, a skip coupling is provided between the transpose convolution block TransposeConv2d_2 of the decoder unit 800d and the corresponding convolution block CONV_BLOCK_2 of the encoder unit 800c. The skip coupling makes it possible to transfer high spatial frequency information during upsampling.

[0200] In step 808, the SPADE block SPADE_BLOCK_3 performs spatial adaptive normalization (SPADE) on the output of TransposeConv2d_2 (step 807). Similar to SPADE_BLOCK_1 in the encoder section, SPADE_BLOCK_3 is configured to ensure that the mask information is used appropriately.

[0201] In step 809, the residual convolution block RESIDUAL_CONV_BLOCK_3 performs a residual convolution operation on the output of SPADE_BLOCK_3 (step 808). The residual convolution block RESIDUAL_CONV_BLOCK_3 of the decoder unit 800d performs the same function as the residual convolution block RESIDUAL_CONV_BLOCK_1 of the encoder unit 800c.

[0202] In step 810, the transpose convolution block TransposeConv2d_1 performs a transpose convolution operation on the output of RESIDUAL_CONV_BLOCK_3 (step 809). In the illustrated embodiment, TransposeConv2d_1 restores the original resolution and dimensions of the transformed left shear measurement 601a.

[0203] In step 811, the convolution block CONV_BLOCK_4 of the decoder unit 800d performs a convolution operation on the output of TransposeConv2d_1 (step 810), including normalization and activation steps (similar to steps 720-722 described above). In the illustrated embodiment, the filter stride is set to 1, and as a result, the convolution operation does not reduce the dimensionality of the data. The convolution operation in step 811 extracts a set of single shear features 850a for the left shear measurement 601. These extracted features represent an initial estimation or trial reconstruction of the hyperspectral data cube based on the single shear measurement (left shear measurement 601). The extracted features are also provided as input to the combined reconstruction block 900 (described later), which helps pass the features to the combined reconstruction block 900 that encodes mask information.

[0204] A single shear reconstruction block 800a can provide independence between different shear fields (e.g., left shear measurement 601 and right shear measurement 603). The single shear reconstruction block 800a can allow Model 600 to independently utilize unique information in each shear field before utilizing the entire field in the combined reconstruction. The single shear reconstruction block 800a can enable control in the mask and independent shearing in each field, allowing for consideration of hardware defects. In addition, providing separate single shear reconstruction blocks for each shear field can enable a range beyond simple left and right shear measurements, allowing any combination of shear fields to be utilized.

[0205] While the single shear reconstruction blocks 800a and 800b are presented as part of a larger system for hyperspectral datacube reconstruction, it will be understood that each of the single shear reconstruction blocks 800a and 800b can be used independently to reconstruct a hyperspectral datacube from a single shear field of view of a scene. The single shear reconstruction blocks 800a and 800b may not require correction of the field of view (e.g., left or right shear measurements) (e.g., using transformation blocks 700a and 700b) before input to the single shear reconstruction blocks 800a and 800b. The single shear reconstruction blocks 800a and 800b can also be used as part of different hyperspectral datacube reconstruction systems and methods.

[0206] In the embodiment shown in Figure 8, the single shear reconstruction blocks 800a and 800b use spatial adaptive normalization (SPADE) to apply a mask used to encode the image to the features extracted by the main convolution operation.

[0207] In an alternative embodiment described with reference to Figure 10A, a single shear reconstruction block 1800a may be subjected to reverse attentional normalization.

[0208] Therefore, the single shear reconstruction block in Figure 10A differs from the single shear reconstruction block in Figure 8 mainly in that step 802, which applies SPADE_BLOCK_1, is replaced by step 1802, which has an inverse attention normalization block INVERTED_ATTENTION_BLOCK_1; step 805, which has SPADE_BLOCK_2, is replaced by step 1805, which has an inverse attention normalization block INVERTED_ATTENTION_BLOCK_2; and step 808, which has SPADE_BLOCK_3, is replaced by step 1808, which has an inverse attention normalization block INVERTED_ATTENTION_BLOCK_3.

[0209] In step 1802, the inverse attention normalization block INVERTED_ATTENTION_BLOCK_1 performs inverse attention normalization on the output of CONV_BLOCK_1 (step 801). INVERTED_ATTENTION_1 is configured to apply the features of the encoding scheme representation 1803, which is used to encode the image of the left shear measurement 601, to the output of CONV_BLOCK_1.

[0210] The coding scheme representation 1803 is defined as any array that represents or describes the coding and / or compression of an optical system. This may include a mask, a stack of shear masks, or a sensing matrix.

[0211] The structure and operation of INVERTED_ATTENTION_BLOCK_1 are shown in more detail in Figure 10B. The inverse attention block has the same function as the SPADE block described with reference to Figure 8A, but it will be understood that this block is not only a convolution block but also an attention mechanism for more effectively applying the mask.

[0212] In step 1830, the encoding scheme representation 1803, which was initially used to encode the image of the left shear measurement 601, is provided as a conditional input to INVERTED_ATTENTION_BLOCK_1. In step 1831, if necessary, the mask code is resized to match the dimensions of the output of the preceding convolutional block CONV_BLOCK_1 (step 801) of encoder unit 800c.

[0213] In step 1832, the convolution block CONV_BLOCK_1 of INVERTED_ATTENTION_BLOCK_1 performs a convolution operation on the (resized) mask, including normalization and activation steps (similar to steps 720-722 described above). In the embodiment shown in Figure 10B, INVERTED_ATTENTION_BLOCK_1 of encoder unit 800c comprises 50 filters. The convolution operation of CONV_BLOCK_1 of INVERTED_ATTENTION_BLOCK_1 does not reduce the dimensions (height and width) of the resized conditional input data. CONV_BLOCK_1 of INVERTED_ATTENTION_BLOCK_1 generates a stack of output images corresponding to the number of filters used.

[0214] In step 1833, the convolution module Conv2d_1 performs a convolution operation on the output of CONV_BLOCK_1 of INVERTED_ATTENTION_BLOCK_1 (step 832), similar to what is done by the convolution module Conv2d of the transformation block 700a (step 703) described above. No normalization or activation steps are performed. In the embodiment shown in Figure 10B, Conv2d_1 of INVERTED_ATTENTION_1 comprises 50 filters.

[0215] In step 1834, a sigmoid operation is performed on the output of convolution block 2, and in step 1835, element-wise multiplication is performed between the output of the sigmoid operation (step 1834) and the output of the preceding CONV_BLOCK_1 (step 801) of encoder unit 800c.

[0216] In step 1836, the convolution module Conv3d_ performs a convolution operation on the output of the element-wise multiplication of CONV_BLOCK_1 (step 1832) of INVERTED_ATTENTION_BLOCK_1. The convolution operation performed by Conv2d_2 is the same as the convolution operation performed by Conv2d_1 (step 833).

[0217] In step 1836, the output of Conv2d_2 (step 835) and the output of element-wise multiplication (step 834) are summed. The summed output of step 836 forms the final output of INVERTED_ATTENTION_BLOCK_1 of encoder unit 800c. The final output of INVERTED_ATTENTION_BLOCK_1 has the same output dimensions as the output dimensions of CONV_BLOCK_1 (step 801) of encoder unit 800c. The inverse attention normalization performed by INVERTED_ATTENTION_BLOCK_1 of encoder unit applies the encoded representation of the left shear measurement 601 to the features extracted from the left shear measurement 601 by CONV_BLOCK_1 of encoder unit 800c. Inverse attention normalization forces the neural network to use encoded scheme representation information and thus can leverage the compression provided by the mask. The coercion of the encoded scheme representation is achieved through scaling and biasing techniques that involve taking two inputs, namely the "original" input and the "conditional" input. The "original" input is the output of the preceding convolutional block in encoder section 800c (e.g., step 801). The "conditional" input is the encoding scheme representation (which is resized as needed to match the dimensions of the "original" input).

[0218] The first convolution operation in INVERTED_ATTENTION_BLOCK_1, performed by CONV_BLOCK_0, extracts features from the input. The second set of operations in INVERTED_ATTENTION_BLOCK_1 is performed by SPADE_BLOCK_1, which receives the conditional input and the result of CONV_BLOCK_1 and is responsible for coordinating the conditional input, in this case the scheme-coded representation, to the features extracted by CONV_BLOCK_1.

[0219] A description of SPADE block 1 can be found below. Following SPADE_BLOCK_1, a convolution operation is performed on the result of the spade block by CONV_BLOCK_1, which conditions the features returned by the spade block, enabling the application of operations on the final set. Subsequently, a sigmoid activation function, as shown below, is applied to the result of CONV_BLOCK_1, and then it is multiplied element by element with the result of CONV_BLOCK_1.

number

[0220] A key contribution in this reverse attention block arises from the sigmoid and element-wise multiplication operators combined to generate a local self-attention mechanism. This more efficiently applies the results of spatial adaptive normalization to the input features. This method is effective because it attempts to redistribute and attend (emphasize) feature information already present in the input features, rather than generating a new set of features.

[0221] This integration of localized attention is novel, and there is no prior art using this mechanism in SPADE-based conditioning. The reverse attention block effectively improves the SPADE block shown in Figure 8. The incorporation of the reverse attention block resulted in a 1.2 psnr improvement.

[0222] The first convolution operation of SPADE_BLOCK_1 performed by CONV_BLOCK_ extracts features from the “conditional” input. The second set of convolution operations of SPADE_BLOCK_1 performed by Conv2d_1 and Conv2d_2 generate “scales” and “biases” from the extracted features. The “scales” provided by element-wise multiplication in step 1834 enforce normalization of the weights between the mask and the features extracted via the convolution operation in step 801. The “biases” provided by summing in step 1836 enforce that the mask is adequately represented when compared to the features being masked.

[0223] The reverse attention normalization block may have a unique effect in contexts where the signal is masked and then dispersed before being detected (as the left shear measurement 601). Normalizing and biasing the encoding of the coded scheme representation may allow for more effective utilization of the scheme coding information in the reconstruction of the hyperspectral data cube. However, the reverse attention normalization block is not essential, and other techniques may be used in some embodiments.

[0224] Conv block 1 is the first operator that operates on the input feature, and the output of that feature is then passed to the spade block, which operates as described above. Together they are important contributions, the spade block enforces a conditional input and inverse attention, and the sigmoid and element-wise multiplication project the conditioned feature output by the spade block onto the feature resulting from conv block 1.

[0225] Steps 801, 802, and 803 in Figure 8, and steps 801, 802, and 803 in Figure 10A, respectively, define a series of main blocks in encoder unit 800c, namely the convolution block, followed by the SPADE block, the reverse attention normalization block, and the subsequent residual convolution block, respectively. In the illustrated embodiments, steps 804-806 are substantially mirror images of steps 801-803 in Figure 8, and steps 804, 805, and 806 in Figure 10A are substantially mirror images of steps 801, 80, and 803. The main difference is that CONV_BLOCK_2 of encoder unit 800c further reduces the data dimension by half and has 100 filters instead of the 50 filters used by CONV_BLOCK_1 of encoder unit 800c.

[0226] The sequence of steps, namely the convolution block, the SPADE or reverse attentional normalization block, and the residual convolution block, can be repeated any number of times as preferred. However, it will be understood that the different blocks do not need to be used in relation to one another, but can be used independently of one another or in different combinations of sub-blocks.

[0227] Single shear reconstruction blocks 800a and 1800a can provide independence between different shear fields (e.g., left shear measurement 601 and right shear measurement 603). Single shear reconstruction blocks 800a and 1800a can allow Model 600 to independently utilize unique information in each shear field before utilizing the entire field in a combined reconstruction. Single shear reconstruction blocks 800a and 1800a can enable control in the mask and independent shearing in each field, allowing for consideration of hardware defects. In addition, providing separate single shear reconstruction blocks for each shear field can enable a range beyond simple left and right shear measurements, allowing any combination of shear fields to be utilized.

[0228] While the single shear reconstruction blocks 800a, 800b, 1800a, and 1800b are presented as part of a larger system for hyperspectral datacube reconstruction, it will be understood that each of the single shear reconstruction blocks 800a and 800b can be used independently to reconstruct a hyperspectral datacube from a single shear field of view of a scene. The single shear reconstruction blocks 800a, 800b, 800a, 800b, 1800a, and 1800b may not require correction of the field of view (e.g., left or right shear measurements) (e.g., using transformation blocks 700a and 700b) before input to the single shear reconstruction blocks 800a and 800b. The single shear reconstruction blocks 800a and 800b can also be used as part of different hyperspectral datacube reconstruction systems and methods.

[0229] Returning to Figure 6, the outputs of the left and right single shear reconstruction blocks 800a and 800b, respectively, along with the preview measurements 602, are provided as inputs to the combined reconstruction block 900. The combined reconstruction block 900 is configured to reconstruct an improved hyperspectral data cube from these inputs.

[0230] Here, the combined reconstructed block 900 is described in more detail.

[0231] Figures 9A and 9B show embodiments of the combined reconstruction block 900. In the illustrated embodiments, the combined reconstruction block 900 comprises a neural network. The neural network has an encoder-decoder structure comprising an encoder section 900c and a decoder section 900d (similar to the neural network structure shown in the single shear reconstruction blocks 800a and 800b). The encoder section 900c is shown in Figure 9A, while the decoder section is shown in Figure 9B.

[0232] Referring to Figure 9A showing the encoder unit 900c, in step 901, the outputs 850a and 850b of the single shear reconstruction blocks 800a and 800b are summed. In step 902, the summed output from step 901 is concatenated with the preview measurement 602. In the illustrated embodiment, the summed output from step 901 and the preview measurement 602 are concatenated by stacking them together. This is reflected in the input dimension for CONV_BLOCK_0 in step 903, which has an input depth of 51.

[0233] In step 903, the convolution block CONV_BLOCK_0 of the encoder unit 900c performs a convolution operation on the concatenated output of step 902. The CONV_BLOCK_0 of the encoder unit 900c is configured to perform a convolution operation similar to that described in steps 720 to 722 above. In the illustrated embodiment, CONV_BLOCK_0 comprises 60 filters, but any preferred number of filters may be used. Each filter has a stride of 1. CONV_BLOCK_0 does not reduce the dimensions (height and width) of the data.

[0234] In step 904, the dense convolution block DENSE_CONV_BLOCK_1 performs a dense convolution operation on the output of CONV_BLOCK_0 (step 903).

[0235] The configuration and calculations of DENSE_CONV_BLOCK_1 are shown in more detail in Figure 9C.

[0236] In step 960, the convolution block CONV_BLOCK_1 of DENSE_CONV_BLOCK_1 performs a convolution operation on the output of CONV_BLOCK_1 of encoder unit 900c (step 903). The convolution operation does not reduce the dimensionality or resolution of the data. In the embodiment shown in Figure 9A, DENSE_CONV_BLOCK_1 comprises 60 filters, each with a stride of 1. The output of CONV_BLOCK_1 of DENSE_CONV_BLOCK_1 is a stack of layers, each layer corresponding to a different filter. The number of filters used in DENSE_CONV_BLOCK_1 is the same as the number of filters used in the preceding CONV_BLOCK_1 of encoder unit 900c to ensure that the data has the same dimensions.

[0237] In step 961, the output of step 903 is summed with the output of step 960.

[0238] In step 962, the convolution block CONV_BLOCK_2 of DENSE_CONV_BLOCK_1 performs a convolution operation on the summed output of step 961. Similar to step 960, CONV_BLOCK_2 does not reduce the data dimension or resolution. The number of filters used is the same as the number of filters used in the preceding CONV_BLOCK_0 of encoder unit 900c.

[0239] In step 963, the output of CONV_BLOCK_2 (step 962), the output of CONV_BLOCK_1 (step 960), and the preceding output of CONV_BLOCK_0 from encoder unit 900c are all added together.

[0240] In step 964, the convolution block CONV_BLOCK_3 of DENSE_CONV_BLOCK_1 performs a convolution operation on the summed output from step 963. Similar to steps 960 and 962, CONV_BLOCK_3 does not reduce the data dimension. The number of filters used is the same as the number of filters used in the preceding CONV_BLOCK_0 of encoder unit 900c.

[0241] In step 965, the outputs of CONV_BLOCK_3 (step 964), CONV_BLOCK_2 (step 962), CONV_BLOCK_1 (step 960), and the preceding output of CONV_BLOCK_0 from encoder unit 900c are all summed together. The summed output from step 965 provides the final output of DENSE_CONV_BLOCK_1.

[0242] The dense convolution block DENSE_CONV_BLOCK_1 comprises three stacked convolution blocks, but any preferred number of convolution operations (e.g., two, three, four or more) may be performed within the dense convolution block. Coupling is provided between each input and output of the stacked convolution block, as opposed to simply coupling with adjacent blocks (e.g., as in conventional convolution blocks). Dense convolution blocks can help stabilize learning. Alternatively or additionally, dense convolution blocks may enable the utilization of transfers of higher frequency features from previous non-adjacent blocks or layers within the neural network.

[0243] In step 905, the convolution block CONV_BLOCK_1 of the encoder unit 800c performs a convolution operation on the output of DENSE_CONV_BLOCK_1 (step 904). CONV_BLOCK_1 of the encoder unit 800c is configured to perform a convolution operation similar to that performed by CONV_BLOCK_1, CONV_BLOCK_2, and CONV_BLOCK_3 of the conversion block 700a described above. In the illustrated embodiment, CONV_BLOCK_1 comprises 60 filters, but any number of filters may be used. Each filter has a stride that allows CONV_BLOCK_1 to reduce the dimensionality of the data. In the illustrated embodiment, each filter has a stride of 2, but each filter may have a different stride.

[0244] Steps 906 and 908 are substantially mirrors of step 904, while steps 907 and 909 are substantially mirrors of step 905. The main difference is that CONV_BLOCK_2 (step 907) and CONV_BLOCK_3 (step 909) of encoder unit 900c further reduce the data dimension by half, and have 120 and 240 filters, respectively, instead of the 60 filters used by CONV_BLOCK_1 of encoder unit 900c. However, it will be understood that any suitable number of suitable filters may be used in CONV_BLOCK_2 and CONV_BLOCK_3 of encoder unit 900c. The filters used in each of CONV_BLOCK_2 and CONV_BLOCK_3 of encoder unit 900c may also reduce the data dimension using any suitable stride accordingly. The sequence of steps, i.e., convolutional blocks, dense convolutional blocks, may be repeated any suitable number of times (e.g., once, twice, three or more times). However, it will be understood that different blocks do not need to be used in relation to each other and can be used independently of each other. Dense convolution blocks are not required.

[0245] In step 910, the dilute convolution millmid (DCP) block DCP_BLOCK_1 of the encoder unit 900c performs at least one dilation convolution operation on the output of CONV_BLOCK_3 (step 909).

[0246] The configuration and calculations of DCP_BLOCK_1 are shown in more detail in Figure 9D.

[0247] In step 970, the dilated convolution block DILATED_CONV_BLOCK_1 performs a dilated convolution operation (also known as an atlas convolution) on the output of CONV_BLOCK_3 (step 909). The dilation rate of the filter or kernel used in the dilated convolution operation effectively spaces the kernels by a rate corresponding to the dilation rate. The dilated convolution operation can enable the kernel or filter to operate in a larger receptive field than, for example, a conventional convolution operation. The higher the dilation rate, the larger the receptive field in which the kernel operates. In the illustrated embodiment, the filter used in DILATED_CONV_BLOCK_1 uses a dilation rate of 1. A dilation rate of 1 corresponds to a kernel that is not spaced. There is no space between the elements of the kernel. Therefore, the kernel of DILATED_CONV_BLOCK_1 functions as a conventional convolution filter. In the illustrated embodiment, DILATED_CONV_BLOCK comprises 60 filters, but any preferred number of filters may be used. The filter is applied across the entire dataset using a stride of 1, while preserving the data's dimensions and resolution (without downsampling).

[0248] In step 971, the output of DILATED_CONV_BLOCK_1 is concatenated with the output of CONV_BLOCK_3 (step 909). In step 972, the convolutional block CONV_BLOCK_1 of DCP_BLOCK_1 performs a convolution on the concatenated output of step 971. In the illustrated embodiment, CONV_BLOCK_1 comprises 60 filters, but any preferred number of filters may be used. Each filter is advanced across the data with a stride of 1. CONV_BLOCK_1 is not configured to reduce the dimensionality (height and width) of the data.

[0249] In step 973, the dilated convolution block DILATED_CONV_BLOCK_2 performs a dilated convolution operation on the output of CONV_BLOCK_1 (step 972) of DCP_BLOCK_1. The dilated convolution operation performed by DILATED_CONV_BLOCK_2 is the same as that performed by DILATED_CONV_BLOCK_1. The main difference is that the filter's dilation ratio is increased to 2. A dilation ratio of 2 introduces a single space between each element of the filter. For example, a 3x3 pixel filter with a dilation ratio of 2 has the same receptive field as a 5x5 pixel filter, but uses only 9 filter elements. The filter is advanced across the data with a stride of 1 to preserve the data's dimension and resolution.

[0250] In step 974, the output of DILATED_CONV_BLOCK_2 is concatenated with the output of CONV_BLOCK_3 (step 909) and the output of DILATED_CONV_BLOCK_1 (step 970). In the illustrated embodiment, each output is concatenated by stacking the outputs, thereby increasing the depth of the data. In step 975, the convolutional block CONV_BLOCK_2 of DCP_BLOCK_1 performs a convolution operation on the concatenated output of step 974. In the illustrated embodiment, CONV_BLOCK_2 comprises 60 filters, but any preferred number of filters may be used. Each filter is advanced across the data using a stride of 1. CONV_BLOCK_2 is not configured to reduce the dimensions (height and width) of the data.

[0251] In the illustrated embodiment, steps 976–978, 979–981, and 982–984 are substantially mirror images of steps 970–972 and 973–975. The main difference is that the expansion rate for each consecutive expansion convolution block (steps 976, 979, and 982) increases with each subsequent expansion convolution block. In the illustrated embodiment, the expansion rates for DILATED_CONV_BLOCK_3, DILATED_CONV_BLOCK_4, and DILATED_CONV_BLOCK_5 are 4, 6, and 8, respectively. The expansion rates across the five expansion convolution blocks in the illustrated embodiment are 1, 2, 4, 6, and 8. However, instead, different increases in the expansion rate between consecutive expansion convolution blocks may be used (these increases in the expansion rate determine the expansion rate of the next expansion convolution block).

[0252] In the illustrated embodiment, the DCP block comprises a stack of consecutive dilated convolutional blocks. The dilation rate of the filter or kernel used increases with each consecutive dilated convolutional block. This allows the filter to function in an area of ​​data that becomes progressively larger with each consecutive dilated convolution operation. By increasing the dilation rate, the kernel can operate in a larger area than a conventional convolution operation using a kernel of the same size. Consecutive dilated convolutions with increasing dilation rates may allow for an increased receptive field of features using a more compact neural network than when undilated convolutions are used. This can be achieved with substantially the same resolution as undilated convolutions. A more compact neural network can improve reconstruction speed. However, the DPC block is not mandatory.

[0253] In the illustrated embodiment, the dilating convolutional blocks are tightly coupled. Each output of a preceding dilating convolutional block is provided as an input to the next dilating convolutional block. Coupling is provided between each input and output of the stacked convolutional blocks, as in contrast to simply coupling to adjacent blocks (e.g., as in conventional convolutional blocks). Tightly coupled dilating convolutional blocks can help stabilize learning. Tight coupling in DCP blocks can reduce the effect of resolution loss arising from stacked dilating convolutional blocks with continuously increasing dilating rates. Tight coupling can allow features captured at higher kernel resolutions to be maintained through successive dilating convolution operations and form part of the output of the DCP block. Resolution loss refers to the kernel resolution at different dilating rates. With increasing dilating rates, the kernel spreads out more with respect to the input but still has the same number of parameters. Thus, at lower dilating rates, higher resolution and higher frequency features are extracted, while at higher dilating rates, more semantic and lower frequency features are extracted. Tight coupling allows these different features to be combined.

[0254] Referring to FIG. 9B showing the encoder section 900d, in step 910, the output of DCP_BLOCK_1 (step 909) is passed to the self-attention block SELF_ATTENTION_BLOCK. In the illustrated embodiment, the self-attention block extracts three feature sets from the output of DCP_BLOCK_1, although any number of feature sets may be extracted. Two of the feature sets are combined using matrix multiplication to form an attention map. The attention map is then combined with a third feature set using element-wise multiplication to form a set of feature attention maps. The attention map functions to emphasize important features in the initial input to the self-attention block, which may effectively enable tight coupling from all of the individual features in the input to the self-attention block. This may help to emphasize regions within the overall extent of the feature map having important information and to reconstruct the hyperspectral data cube from a wide range of features. However, the self-attention block is not essential, and embodiments without this feature are possible.

[0255] In step 911, the transposed convolution block TransposeConv2d_3 performs a transposed convolution operation on the output of SELF_ATTENTION_BLOCK (step 910). The transposed convolution operation is substantially as described above for the decoder section 800d of the single shear reconstruction block 800a (steps 807, 810). The transposed convolution block is configured to increase the dimensionality or resolution of the data (known as upsampling). A skip connection may be provided between the transposed convolution block TransposeConv2d_2 of the decoder section 900d and the corresponding convolution block CONV_BLOCK_3 of the encoder section 900c. The skip connection enables the transfer of high spatial frequency information during upsampling.

[0256] In step 912, the SPADE block SPADE_BLOCK_3 performs spatial adaptive normalization on the output of TransposeConv2d_3 (step 911). The spatial adaptive normalization is substantially as described above for SPADE_BLOCK_1 of the single shear reconstruction block 800a (see step 802). However, instead of the mask or the encoder, the preview measurement 602 is used as the "conditional" input to SPADE_BLOCK_3 of the decoder unit 900d. The SPADE block utilizes the information provided by the preview measurement 602. The SPADE block uses the preview measurement 602 information to improve the spatial resolution of the reconstructed hyperspectral data cube.

[0257] In step 913, the dense convolution block DENSE_CONV_BLOCK_6 performs a dense convolution operation on the output of SPADE_BLOCK_3 (step 912). The dense convolution operation is substantially as described above for DENSE_CONV_BLOCK_1 of the encoder unit 900c.

[0258] In the illustrated embodiment, steps 914 - 916 and 917 - 919 are substantially the mirror of steps 911 - 913. The main difference is that TransposeConv2d_2 and TransposeConv2d_1 each further increase or restore the dimension of the data (e.g., height and width) by a factor of 2. In the illustrated embodiment, TransposeConv2d_2 and TransposeConv2d_1 each have 120 and 60 filters respectively, but any suitable number of filters may be used. A series of steps, i.e., the transposed convolution block, the SPADE block, the dense convolution block, may be repeated any suitable number of times (e.g., mirroring the number of downsampling convolution operations of the encoder unit 900c). However, it will be understood that the different blocks need not be used in relation to each other and may be used independently of each other. The SPADE block and the dense convolution block are not essential.

[0259] In step 920, the hierarchical reconstruction block (HCR block) HCR_BLOCK performs hierarchical channel reconstruction of the hyperspectral data cube.

[0260] The configuration and calculation of HCR_BLOCK are shown in more detail in Figure 9E.

[0261] In step 990, the convolution block CONV_BLOCK_1 of HCR_BLOCK performs a convolution operation on the output of DENSE_BLOCK_4 (step 919), including normalization and activation steps (similar to steps 720-722 described above). In the embodiment shown in Figure 9E, CONV_BLOCK_1 of HCR_BLOCK comprises 60 filters. Each filter is advanced across the data with a stride of 1. The convolution operation of CONV_BLOCK_1 of HCR_BLOCK does not reduce the dimensions (height and width) of the data. CONV_BLOCK_1 of HCR_BLOCK generates a stack of output images corresponding to the number of filters used.

[0262] In step 991, the convolution module Conv2d_1 performs a convolution operation on the output of CONV_BLOCK_1 (step 990) of HCR_BLOCK without any normalization or activation steps (see step 703). Conv2d_1 has 10 filters.

[0263] Each of the 10 filters corresponds to a spectral channel that formed part of the hyperspectral data cube being reconstructed.

[0264] In step 992, the outputs of CONV_BLOCK_1 and Conv2d_1 are concatenated. The concatenation forms a stack with a depth of 70. In step 993, the convolution block CONV_BLOCK_2 performs a convolution operation on the concatenated outputs from step 992. In the illustrated embodiment, CONV_BLOCK_2 comprises 60 filters. Each filter advances across the data with a stride of 1. The convolution operation of CONV_BLOCK_2 in HCR_BLOCK does not reduce the dimensions (height and width) of the data.

[0265] In step 994, the convolution module Conv2d_2 performs a convolution operation on the output of CONV_BLOCK_2 (step 993) of HCR_BLOCK without any normalization or activation steps. Conv2d_2 comprises 20 filters, each of which corresponds to a spectral channel that forms part of the hyperspectral data cube being reconstructed.

[0266] In step 995, the outputs of CONV_BLOCK_2 and Conv2d_2 are concatenated. The concatenation forms a stack with a depth of 80. In step 996, the convolution block CONV_BLOCK_3 performs a convolution operation on the concatenated outputs from step 995. In the illustrated embodiment, CONV_BLOCK_3 comprises 60 filters. Each filter advances across the data with a stride of 1. The convolution operation of CONV_BLOCK_3 does not reduce the dimensions (height and width) of the data.

[0267] In step 997, the convolution module Conv2d_3 performs a convolution operation on the output of CONV_BLOCK_3 of HCR_BLOCK. In the illustrated embodiment, Conv2d_3 comprises 40 filters. Each of the 40 filters corresponds to a spectral channel that forms part of the hyperspectral data cube to be reconstructed.

[0268] The outputs of Conv2d_1, Conv2d_2, and Conv2d_3, respectively, feature 10-channel, 20-channel, and 40-channel reconfigurations of hyperspectral data cubes corresponding to the scene.

[0269] The HCR block is configured to perform spectral reconstruction stepwise by increasing the number of channels reconstructed at each stage. In the illustrated embodiment, the HCR block is configured to reconstruct one-quarter of the total channels in the first stage. The exact number of channels is determined by the number of filters employed by the convolutional module used for reconstruction in the first stage. The HCR block is then configured to reconstruct the next quarter of the total channels in the second stage. The HCR block is then configured to reconstruct the total number of channels in the third stage. The reconstructed channels from the first stage are passed to the second stage to assist in channel reconstruction in the second stage. Similarly, the reconstructed channels from the second stage are passed to the third stage to assist in channel reconstruction in the third stage to form the entire reconstructed hyperspectral data cube. The output of the final stage of the HCR represents the final reconstructed hyperspectral data cube. The reconstructed channels in the intermediate stages of the HCR do not form part of the output reconstructed hyperspectral data cube. However, these intermediate outputs may be used during training of the combined reconstruction block 900 (described later).

[0270] The preview allows for the addition of another step to normalize the resulting reconstruction, as shown in step 1700 in Figure 11. The reconstruction resulting from the HCR block is normalized using a relative normalization transformation as shown in Figure 12. The process begins with a sigmoid operation 1702 applied to the reconstruction. The output of the sigmoid operation is then folded using a reduce_sum operation 1704, the result of which is the channel-by-channel sum of the results of the sigmoid operator. The output of the sigmoid operation is then divided by the output of the reduce_sum operation using a division operator 1706 that extends along a third axis of the input (described later). The result of the division operator is determined as the training loop output 1708.

number

[0271] Here,

number

[0272] Here, c is a hyperspectral reconstruction trial.

[0273] Additionally, the result of the division operator 1706 is also multiplied element by element by the non-shear measurement (1710). In this case, the non-shear measurement is defined as a preview measurement to generate the normalized output.

[0274] This relative transformation 1700 returns both the defined output 1712 and the training loop output 1708, but only the defined output is used in reconstruction, and only the training loop output is used in the training loop (as described later).

[0275] The output relative normalization 1700 converts the result of the reconstruction with respect to the intensity present in the non-shearing measurements. As a result of this, the reconstruction is bounded by the total force captured by the detector. Thereby, a direct intensity relationship between the reconstruction and the imaged scene can be developed. Thereby, valuable intensity information can be preserved during the reconstruction process.

[0276] Additionally, the relative transformation stabilizes and improves the performance of the training, which will be described in more detail later.

[0277] Normalization by preview results in a reconstruction for both the preview intensity and the spatial information (frequency) present in the preview.

[0278] This method resulted in an improvement of 3.5 psnr.

[0279] The output of step 920 (the reconstruction block in FIG. 9B) or the relative transformation step 1700 provides an improved hyperspectral data cube as the final output of the combined reconstruction block (in a specific example, there are three data cubes, each having a different spectral resolution (depth)).

[0280] Instead of starting from the detected raw data, the combined reconstruction block 900 tomographically reconstructs an improved hyperspectral data cube using at least one estimated hyperspectral data cube and preview measurements. In an exemplary embodiment, there are two estimated hyperspectral data cubes (from each single shear reconstruction block 800a, 800b) and preview measurements 602.

[0281] Therefore, the machine learning model 600 employs a two-step method for hyperspectral data cube reconstruction. The first stage of reconstruction includes correcting for differences between multiple fields of view and providing initial estimates for reconstructing hyperspectral data cubes from each shear field of view. The first stage of reconstruction is performed using transformation blocks 700a, 700b and single shear reconstruction blocks 800a, 800b or 1800a, 1800b. The second stage of reconstruction includes combining the estimated hyperspectral data cubes from the first stage and improving the combination to form the final reconstruction of the hyperspectral data cubes. The second stage of reconstruction is performed using the combined reconstruction block 900.

[0282] In the machine learning model shown in Figure 11, the first stage of reconstruction is performed using transformation blocks 700a and 700b, and single shear reconstruction blocks 800a, 800b or 1800a and 1800b. The second stage of reconstruction is performed using the combined reconstruction block 900 and the relative transformation block 1700.

[0283] Performing a two-stage reconstruction within a single Model 600 dramatically increases the speed of hyperspectral data cube reconstruction. The two-stage method also separates the final reconstruction from multiple individual fields of view (and any noise or disturbances within those fields, e.g., parallax, misalignment, or differences resulting from manufacturing tolerances). For example, for a hyperspectral data cube with dimensions of 512 pixels × 512 pixels × 40 pixels, conventional compressed sensing reconstruction takes longer than 20 seconds.

[0284] The aforementioned machine learning model 600 can reconstruct a hyperspectral data cube of the given dimensions in less than 0.1 seconds (on an NVIDIA RTX2080). This reduction in reconstruction time can enable "live" viewing of the reconstructed hyperspectral data cube and "live" interpretation of the hyperspectral data. In addition, the structure of model 600 (e.g., DCP block implementation) reduces the size of the model. This makes it possible to use model 600 across a wide range of platforms, including embedded platforms, and on mobile edge devices (e.g., NVIDIA Jetson), while maintaining performance (e.g., "live" viewing). Large, high-wattage GPUs are not required for inference on hyperspectral data cubes using model 600.

[0285] However, it will be understood that the combined reconstruction block 900 may also be used to reconstruct an improved hyperspectral data cube from a single shear field of view of the scene. The single shear field of view may be, or comprise, a direct measurement from the detector (e.g., the left shear measurement 601). Alternatively, the single shear field of view may be, or comprise, a single estimated hyperspectral data cube, e.g., a set of single shear features 850a, 850b output by one of the single shear reconstruction blocks 800a, 800b. The combined reconstruction block 900 may be used independently of the transformation blocks 700a, 700b, the relative normalization transformation, and / or the single shear reconstruction blocks 800a, 800b.

[0286] Model 600 is also trained using a two-step method. The first training step is responsible for training transformation blocks 700a, 700b and single shear reconstruction blocks 800a, 800b. The learning objective for this part of Model 600 is to transform the shear field (e.g., left shear measurement 601 and right shear measurement 603) by aligning the shear field with preview measurement 602, and to generate a reconstructed data cube. The second training step is responsible for training combined reconstruction block 900. The learning objective for this part of Model 600 is to reconstruct a high-quality hyperspectral data cube.

[0287] A two-step training method can constrain the model's learning to two independent objectives. This involves training each part of Model 600 separately. Different loss functions may be used to independently control the learning of each part of Model 600. A single model can then be trained both to transform the shear field and to reconstruct the hyperspectral data cube. Dividing the problem in this way improves the quality of the final hyperspectral data cube. Training two parts of a single model also improves the overall speed of both inference and training. Model 600 can perform image alignment by correcting disparity and end-to-end reconstruction while maintaining real-time performance (e.g., the ability to view the reconstructed hyperspectral data cube virtually in real-time or "raw").

[0288] With respect to the first training step, the objective of the first training step is to train the first part of model 600 (transformation blocks 700a, 700b and single shear reconstruction blocks 800a, 800b) to substantially enable inter-pixel mapping between shear fields 601, 603 and preview measurements 602, and to provide single shear reconstruction trials (estimated hyperspectral data cubes).

[0289] The first training step can be achieved by minimizing the loss between reconstruction trials (single shear features 850a, 850b), the desired hyperspectral data cube, and preview measurements 602.

[0290] The target hyperspectral data cube, shear measurements, and preview image can be determined using a composite dataset that simulates multiple fields of view of the scene via spectral dispersion elements and encoders. Parallax between disparities may be introduced. The composite dataset can be generated by assigning known spectra to different materials in a procedurally generated scene (e.g., a rendered / ray-traced composite field of view of the scene) with three-dimensional objects. Alternatively, actual hyperspectral data can be used as the target hyperspectral data cube and corresponding measurements in the various simulated detectors.

[0291] In the embodiment, the first training step may include using one or more of the following loss functions:

[0292] The L2 loss function can be used to train the first part of Model 600. The L2 loss function represents the mean squared error (MSE) between the single shear features 850a, 850b (estimated hyperspectral data cubes) and the target hyperspectral data cube. The loss function forces the single shear reconstruction trials to represent the target hypercube.

[0293] Additionally or alternatively, an alignment loss function may be used to train the first part of model 600. The alignment loss function represents the mean error between the folded single-shear reconstruction trials (single-shear features 850a, 850b) and the preview measurements 602. The folded single-shear reconstruction trials include a single value combination (e.g., sum or mean) of the spectral layer of the hyperspectral data cube for each spatial point of the hyperspectral data cube. The alignment loss function forces alignment between the reconstruction trials and the preview measurements 602. The alignment loss function also reduces errors in the energy distribution and / or concentration of the estimated hypercube. The alignment loss is specific to the method in the alignment of at least one shear field of view, which is corrected to match the preview (unsheared) image.

[0294] The compression measurement loss function represents the average difference between the target compression measurement (obtained from a simulation of the measurements of the target measured hyperspectral data cube, or simulated from a composite scene) and the compression shear measurement corresponding to the estimated hyperspectral data cube. The compression shear measurement corresponding to the estimated hyperspectral data cube can be determined by affine transforming the cube to mask, shear, and fold wavelengths.

[0295] The compressed measurement loss function forces a realistic alignment and transformation (e.g., fluoroscopic transformation) between the preview measurement 602 and the shear field. The compressed measurement loss is a loss function specific to hyperspectral imaging.

[0296] Training the first part of Model 600 may involve using a combination of loss functions, for example, a weighted sum of two or more loss functions (or all of the loss functions described above).

[0297] Regarding the second training step, the objective of the second training step is to train the second part of model 600 (combined reconstruction block 900), combine single shear reconstruction trials 850a and 850b, improve the results of the combination, and generate the final reconstruction of the hyperspectral data cube.

[0298] The second training step can be achieved by the final reconstruction of the hyperspectral data cube and matching the desired hyperspectral data cube. In embodiments, this can be achieved by using one or more of the following loss functions.

[0299] The Hierarchical Channel Reconstruction (HCR) loss function (see Reference 1) enforces inter-pixel similarity between the final reconstruction of the hyperspectral data cube and the target hyperspectral data cube. The HCR loss function utilizes HCR blocks in the combined reconstruction block 900.

[0300] The structural similarity index (SSIM) loss function (see reference 2) leverages the fact that adjacent hyperspectral channels share spatial features. The SSIM loss function can effectively confirm that very detailed features in distant (e.g., spatially / spectrally separated) parts of a hyperspectral data cube coincide with each other.

[0301] The relative normalization loss function, as described later, normalizes the spectral information present at each pixel location by normalizing using the folding of the target hyperspectral data cube. The relative normalization loss function uses the training loop output 1708 of the relative normalization transformation 1700 shown in Figure 12 above as the final reconstruction of the model.

number

[0302] Here,

number

[0303] Then, f represents the function to be minimized, x represents the target hyperspectral data cube, and r represents the reconstruction result, which is determined by Equation 1.

[0304] The second part of Model 600 can be trained using generative adversarial learning via a generative adversarial network (GAN). A conditional GAN ​​(cGAN) loss function may be used (see, for example, citation). The conditional GAN ​​allows the discriminator to observe the input from the generator. In this case, the GAN's generator comprises a combined reconstruction block 900. The cGAN loss function can suppress the reconstruction of unrealistic parallax-induced artifacts and further avoid the obscurity of the output reconstruction. A GAN is preferred, but not required, for training the second part of Model 600.

[0305] The spectral information divergence (SID) loss function allows for a more balanced penalty for noise and defects commonly associated with hyperspectral signals. These defects are typically not identified by spatial losses such as SSIM and MSE. The SID loss function calculates the distance between the probability distributions generated by two-pixel spectral signatures.

[0306] The SID loss function is defined as follows:

number

number

number

[0307] The appended claims are directed to specific combinations of features, but the scope of the disclosure of the present invention also includes any novel feature or any novel combination of features disclosed herein, whether explicitly or implicitly, or any generalization thereof, regardless of whether it relates to the same invention as that currently claimed in any claim and regardless of whether it alleviates any or all of the same technical problems as those of the present invention.

[0308] Features described in the context of another embodiment may also be provided in combination in a single embodiment. Conversely, various features described in the context of a single embodiment for the sake of brevity may also be provided separately or in any suitable sub - combination. The applicant hereby notifies herein that new claims may be constructed for such features and / or combinations of such features during the examination of the present application or any further application derived therefrom.

[0309] For the sake of completeness, it is also stated that the term "comprising" does not exclude other elements or steps, the term "a" or "an" does not exclude a plurality, and reference signs in the claims should not be construed as limiting the scope of the claims.

Claims

1. A method for reconstructing a hyperspectral data cube from at least one spectral shear image and an unencoded spectral unsheared image of a scene, wherein the at least one spectral shear image is either encoded or unencoded, and the method is The at least one spectral shear image and the unencoded spectral unshear image are provided to a neural network having an encoder-decoder structure with an encoder unit and a decoder unit. Using the encoder unit, one or more first-order convolution operations are performed. Using the decoder unit, one or more transpose convolution operations are performed. Using a representation of the encoding scheme used to encode the at least one spectral shear image and the unencoded spectral unsheared image of the scene, spatial adaptive normalization and / or inverse attentional normalization are performed on the output of at least one of the one or more first-order convolution operations and / or transposed convolution operations. Methods that include...

2. Performing the aforementioned spatial adaptation normalization means that Performing a first quadratic convolution operation on the mask or unencoded spectral unsheared image representation of the aforementioned scene, Performing a second quadratic convolution operation on the output of the first quadratic convolution operation, Performing element-wise multiplication between the output of the second quadratic convolution operation and the output of the first-order convolution operation or transposed convolution operation, The output of the second quadratic convolution operation is summed with the output of the element-wise multiplication, The method according to claim 1, including the method described in claim 1.

3. The method according to claim 1 or 2, wherein the inverse attention normalization includes a convolution operation or transposed convolution operation applied to the input features, and / or one or more convolution operations applied sequentially to the result of the final spatial adaptive normalization.

4. Performing the aforementioned reverse attention normalization includes performing one of the sigmoid operation and / or softmax activation operation applied to the result of the previously applied primary or secondary convolution operation, The method according to claim 1, wherein the method includes performing an element-wise multiplication between the output of the first-order convolution operation and the result of the softmax activation operation, and one or more convolution operations and / or transposed convolution operations are applied sequentially to the result of the element-wise multiplication.

5. The method according to any one of claims 1 to 4, further comprising resizing the representation of the mask to match the resolution of the output of the aforementioned first-order convolution or transposed convolution operation.

6. The method according to any one of claims 1 to 5, further comprising performing one or more residual convolution operations using one or both of the encoder unit and the decoder unit.

7. Performing the residual convolution operation described above means Providing input, Performing one or more consecutive convolution operations, The aforementioned input and the output of a series of consecutive convolution operations are combined, The method according to claim 6, including the method described in claim 6.

8. The method according to claim 7, wherein combining the input and the output of the series of consecutive convolution operations includes combining the input and the output of the series of consecutive convolution operations by summing, concatenating, or multiplying, or by any combination of the summing, concatenating, or multiplying operations.

9. The aforementioned input is Output of a linear convolution operation or transpose convolution operation, The method according to claim 7 or 8, wherein the output is either spatially adaptive normalization or residual coherence, and the input image is coded shear or uncoded unsheared.

10. The method according to any one of claims 1 to 9, further comprising providing a skip coupling between the encoder unit and the decoder unit.

11. Using a machine learning algorithm that includes a neural network, determine a transformation configured to map the first and second images to each other. The parallax is corrected by transforming one of the first image and the second image using the transformation determined above. The method further includes correcting the at least one spectral shear image for parallax, wherein the spectral shear image is encoded. The aforementioned method, The first image and the second image are concatenated to form concatenated image data, The concatenated image data is provided to the neural network, Performing one or more convolution operations using the aforementioned neural network, Using the aforementioned neural network, the transformation is determined from the output of one or more convolution operations. Using the determined transformation, transform one of the first image and the second image, The method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

12. The method according to any one of claims 1 to 11, wherein the reconstructed cube originates from either a preceding layer of the network or an external source.

13. The encoder unit is used to perform multiple first-order convolution operations, The decoder unit is used to perform multiple transpose convolution operations, The method according to any one of claims 1 to 12, further comprising:

14. The method according to any one of claims 1 to 13, further comprising performing a plurality of consecutive dilating and convolution operations using the encoder unit, wherein the dilating rate increases with each consecutive dilating and convolution operation at a rate greater than zero.

15. The method according to any one of claims 1 to 14, wherein multiple consecutive convolution operations are tightly coupled.

16. The method according to claim 15, wherein the output of each preceding tightly coupled convolution operation is provided as input to a subsequent tightly coupled convolution operation.

17. The method according to claim 16, wherein the outputs of each preceding tightly coupled convolution operation are coupled to provide the input for the subsequent tightly coupled convolution operation.

18. The method according to any one of claims 14 to 17, wherein the plurality of consecutive expansion and convolution operations are performed within the encoder unit.

19. The method according to any one of claims 1 to 18, wherein the decoder unit comprises a self-attention block configured to receive the final output of the encoder unit.

20. The method according to any one of claims 1 to 19, further comprising performing the spatial adaptive normalization on the output of at least one of the convolution operation and / or transposed convolution operation using the non-sheared image of the scene.

21. The method according to any one of claims 1 to 20, further comprising reconstructing the hyperspectral data cube using hierarchical channel reconstruction.

22. The method according to any one of claims 1 to 21, further comprising providing the neural network with one or more estimated hyperspectral data cubes.

23. The method according to any one of claims 1 to 22, wherein the output of the neural network is transformed by folding the neural network and a relative normalization operator on the result, and the representation of the result is re-mapping using one or more spectral shear or non-shear fields.

24. The method according to claim 23, wherein the relative normalization operator comprises a sigmoid operation applied to a reconstructed hypercube, the result of the sigmoid operation being divided by the reduced sum of the sigmoid operations themselves along the final axis of the sigmoid operation.

Images