Real-time multispectral systems and methods

Abstract

A system for detecting one or more target materials in an uncalibrated multispectral data cube comprising a collection of pixels, comprising: (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and determining, for at least one of the pixels, at least one material indicator indicative of the presence of a given one of the target materials at the pixel location, the machine learning model being trained using a labeled training data set comprising a plurality of training records, each training record comprising: (i) the training uncalibrated multispectral data cube; and (ii) a target material indicator associated with at least one pixel of the training uncalibrated multispectral data cube at the pixel location. and (B) an uncalibrated multispectral data cube; and determining, for at least one pixel of the uncalibrated multispectral data cube, the at least one substance indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated using a calibration process and an atmospheric simulator that simulates a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube.

Description

[Technical Field]

[0001] The present invention relates to the field of real-time multispectral and / or hyperspectral systems and methods. [Background technology]

[0002] Multispectral target detection systems use multiple image capture channels to capture a scene, each with a unique wavelength. These systems capture one or more target spectral signatures of detected objects within the scene. Multispectral target detection systems can be used in fields such as biomedical engineering, agriculture, weather forecasting, space exploration, homeland security, and military operations.

[0003] Current multispectral target detection solutions are unable to achieve remote sensing of a scene by applying on-the-fly atmospheric correction to captured multispectral imaging data, at least not at near real-time speeds. These current solutions require offline atmospheric correction and analysis of the captured multispectral imaging data, which is typically performed by human analysts and is time-consuming.

[0004] Additionally, current multispectral target detection solutions do not utilize ML and / or AI techniques that may be used to automate at least portions of the target detection process by relying on trained artificial intelligence (AI) and / or machine learning (ML) models in one or more aspects of the multispectral target detection process. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need in the art for novel real-time multispectral systems and methods. [Means for solving the problem]

[0006] According to a first aspect of the subject matter of the present disclosure, there is provided a multispectral potential target identification system, the system including: a multispectral sensor capable of capturing images in multiple imaging channels, each having a different wavelength range; one or more additional sensors; acquiring one or more target spectral signatures; activating the multispectral sensor, wherein the multispectral sensor is activated to operate in a wideband short wave infrared (SWIR) simple mode; and capturing a first field of view (FOV) by the multispectral sensor in the wideband SWIR simple mode. and a processing circuit configured to: determine a calculated exposure time for each imaging channel of the multiple imaging channels of the multispectral sensor based on observing a second FOV (first FOV), determine an atmospheric calibration matrix for the multispectral sensor based on input from a user; generate a multispectral data cube of the second FOV observed by the multispectral sensor utilizing the multispectral sensor, the calculated exposure time, and the atmospheric correction matrix, wherein generating the multispectral data cube includes radiometric calibration and multi-channel alignment; identify one or more potential targets utilizing the multispectral data cube, each target being a group of pixels identified in the multispectral data cube having a spectral signature corresponding to at least one of the acquired target spectral signatures, each target having a geographic location; and investigate one or more of the identified potential targets utilizing the one or more additional sensors.

[0007] According to a second aspect of the subject matter of the present disclosure, there is provided a method for multispectral potential target identification, the method including the steps of: acquiring, by a processing circuit, one or more target spectral signatures; activating, by the processing circuit, a multispectral sensor capable of capturing images in multiple imaging channels each having a different wavelength range, the multispectral sensor being activated to operate in a wideband shortwave infrared (SWIR) simple mode; determining, by the processing circuit, a calculated exposure time for each imaging channel of the multiple imaging channels of the multispectral sensor based on observing a first field of view (FOV) with the multispectral sensor in the wideband SWIR simple mode; and determining, by the processing circuit, an atmospheric calibration matrix for the multispectral sensor based on input from a user. and generating, by a processing circuit, a multispectral data cube of a second FOV observed by the multispectral sensor utilizing the multispectral sensor, the calculated exposure time, and an atmospheric correction matrix, wherein generating the multispectral data cube includes radiometric calibration and multi-channel alignment; identifying, by the processing circuit, one or more potential targets utilizing the multispectral data cube, each target being a group of pixels identified in the multispectral data cube having a spectral signature corresponding to at least one of the acquired target spectral signatures, each target having a geographic location; and investigating, by the processing circuit, one or more additional sensors, one or more of the identified potential targets.

[0008] According to a third aspect of the subject matter of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code including: acquiring, by a processing circuit, one or more target spectral signatures; activating, by the processing circuit, a multispectral sensor capable of capturing images in multiple imaging channels, each having a different wavelength range, the multispectral sensor being activated to operate in a wideband shortwave infrared (SWIR) simple mode; determining, by the processing circuit, a calculated exposure time for each imaging channel of the multiple imaging channels of the multispectral sensor based on observing a first field of view (FOV) with the multispectral sensor in the wideband SWIR simple mode; determining, by the processing circuit, an atmospheric calibration matrix for the multispectral sensor based on input from a user; and and investigating, by the processing circuit, one or more of the identified potential targets using one or more additional sensors.

[0009] According to a fourth aspect of the presently disclosed subject matter, there is provided a system for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, the calibrated multispectral data cube being generateable from the uncalibrated multispectral data cube by a calibration process, the system comprising: (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and generating a corresponding calibrated multispectral data cube, the machine learning model being trained using a labeled training data set including a plurality of training records, each training record including (i) a training uncalibrated multispectral data cube and (ii) a training calibrated multispectral data cube corresponding to the training uncalibrated multispectral data cube; and (B) a processing circuit configured to acquire the uncalibrated multispectral data cube; and generate the calibrated multispectral data cube using the machine learning model and the uncalibrated multispectral data cube.

[0010] In some cases, at least one of the training records is generated using an atmospheric simulator, which is capable of receiving (i) a calibrated multispectral data cube and (ii) one or more atmospheric conditions and generating an uncalibrated multispectral data cube corresponding to the calibrated multispectral data cube under those atmospheric conditions.

[0011] In some instances, two or more of the training records include different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

[0012] In some cases, the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

[0013] In some instances, the calibration process is an atmospheric calibration process.

[0014] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a Visual Geometry Group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0015] In some cases, the machine learning model is trained using reinforcement learning methods.

[0016] In some cases, the machine learning model further includes a vision transformer.

[0017] In some cases, the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

[0018] According to a fifth aspect of the subject matter of the present disclosure, there is provided a method for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, wherein the calibrated multispectral data cube can be generated from the uncalibrated multispectral data cube by a calibration process, the method including: acquiring, by a processing circuit, (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and generating a corresponding calibrated multispectral data cube, the machine learning model being trained using a labeled training data set including a plurality of training records, each training record including (i) a training uncalibrated multispectral data cube and (ii) a training calibrated multispectral data cube corresponding to the training uncalibrated multispectral data cube; and (B) the uncalibrated multispectral data cube; and generating, by the processing circuit, the calibrated multispectral data cube using the machine learning model and the uncalibrated multispectral data cube.

[0019] In some cases, at least one of the training records is generated using an atmospheric simulator, which is capable of receiving (i) a calibrated multispectral data cube and (ii) one or more atmospheric conditions and generating an uncalibrated multispectral data cube corresponding to the calibrated multispectral data cube under those atmospheric conditions.

[0020] In some instances, two or more of the training records include different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

[0021] In some cases, the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

[0022] In some instances, the calibration process is an atmospheric calibration process.

[0023] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0024] In some cases, the machine learning model is trained using reinforcement learning methods.

[0025] In some cases, the machine learning model further includes a vision transformer.

[0026] In some cases, the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

[0027] According to a sixth aspect of the presently disclosed subject matter, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being executable by at least one processing circuit of a computer to perform, by a processing circuit, a method including: acquiring, by the processing circuit, (A) a machine learning model capable of receiving an uncalibrated multispectral data cube and generating a corresponding calibrated multispectral data cube, the machine learning model being trained using a labeled training data set including a plurality of training records, each training record including (i) a training uncalibrated multispectral data cube and (ii) a training calibrated multispectral data cube corresponding to the training uncalibrated multispectral data cube; and (B) the uncalibrated multispectral data cube; and generating, by the processing circuit, the calibrated multispectral data cube using the machine learning model and the uncalibrated multispectral data cube.

[0028] According to a seventh aspect of the subject matter of the present disclosure, there is provided a system for automated generation of a multispectral labeled training data set, the multispectral labeled training data set including one or more training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) a calibrated training multispectral data cube, the calibrated training multispectral data cube being generateable from the uncalibrated training multispectral data cube by a calibration process, the system comprising: A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material class of which the element in the scene associated with the pixel is composed; and (B) a material database including a list of materials, at least one material being associated with (i) the material's spectral reflectance signature vector, (ii) the material's typical color, and (iii) a label indicating a material class of which the material is associated. (C) a material database, wherein the material database is associated with a material that is associated with a given scene; and (C) a heuristic table including one or more rules, each rule specifying a probability of the presence of a given material in the given scene based on characteristics of the given scene; capturing a two-dimensional (2D) image from a 3D model of the scene, the 2D image including a subset of pixels; and for at least one given pixel of the subset of pixels, querying the material database for a list of materials that may be a material having a material group for the given pixel; removing from the list of possible materials those materials whose probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model is less than a first threshold to obtain a revised list of possible materials; determining a matching material for the given pixel from the revised list of possible materials based on a match between a color of the given pixel and a typical color associated with a material from the revised list of possible materials; andgenerating a calibrated training multispectral data cube for the training record by selecting a portion of the calibrated training multispectral data cube at a given pixel location from the training multispectral data cube; and generating an uncalibrated training multispectral data cube for the training record using the calibrated training multispectral data cube and an atmospheric simulator, the atmospheric simulator being capable of receiving (i) the calibrated training multispectral data cube and (ii) one or more atmospheric conditions and generating an uncalibrated training multispectral data cube corresponding to the calibrated training multispectral data cube under those atmospheric conditions.

[0029] In some instances, two or more of the training records include different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

[0030] In some instances, the calibration process is an atmospheric calibration process.

[0031] In some cases, the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, or a viewing distance.

[0032] In some cases, the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

[0033] In some cases, the processing circuitry is further configured to add target pixels to the 3D model prior to capture of the 2D image.

[0034] In some instances, generating the uncalibrated training multispectral data cube further includes adding texture to the generated calibrated training multispectral data cube based on a texture associated with a corresponding captured 2D image.

[0035] In some instances, generating the uncalibrated training multispectral data cube further includes adding a simulated registration error.

[0036] In some instances, generating the uncalibrated training multispectral data cube further includes blurring at least one of the corresponding captured 2D images prior to generation.

[0037] In some instances, generating the uncalibrated training multispectral data cube further includes adding shot noise to at least one of the corresponding captured 2D images prior to generation.

[0038] According to an eighth aspect of the presently disclosed subject matter, there is provided a method for automated generation of a multispectral labeled training data set, the multispectral labeled training data set including one or more training records, each training record including (i) a training uncalibrated multispectral data cube and (ii) a training calibrated multispectral data cube, the training calibrated multispectral data cube being generateable from the training uncalibrated multispectral data cube by a calibration process, the method comprising: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which the element in the scene associated with the pixel is composed; and (B) a material database including a list of materials, at least one material being associated with (i) the material's spectral reflectance signature vector, (ii) the material's typical color, and (iii) a given material group of the material. (C) acquiring a heuristic table including one or more rules, each rule specifying a probability of the presence of a given material in the given scene based on characteristics of the given scene; capturing, by a processing circuit, a two-dimensional (2D) image from a 3D model of the scene, the 2D image including a subset of pixels; and, for at least one given pixel of the subset of pixels, querying, by the processing circuit, a material database for a list of materials that may be materials having a material group of the given pixel; removing, by the processing circuit, from the list of possible materials, materials whose probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model is less than a first threshold to obtain a revised list of possible materials; determining, by the processing circuit, a matching material for the given pixel from the revised list of possible materials based on a match between a color of the given pixel and a typical color associated with a material from the revised list of possible materials; and, by the processing circuit,generating a calibrated training multispectral data cube for the training record by selecting at least a portion of the spectral reflectance signature vectors associated with the matching materials to be part of the calibrated training multispectral data cube at a given pixel location; and generating, by a processing circuit, an uncalibrated training multispectral data cube for the training record using the calibrated training multispectral data cube and an atmospheric simulator, the atmospheric simulator being capable of receiving (i) the calibrated training multispectral data cube and (ii) one or more atmospheric conditions and generating an uncalibrated training multispectral data cube corresponding to the calibrated training multispectral data cube under those atmospheric conditions.

[0039] In some instances, two or more of the training records include different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

[0040] In some instances, the calibration process is an atmospheric calibration process.

[0041] In some cases, the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, or a viewing distance.

[0042] In some cases, the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

[0043] In some cases, the processing circuitry is further configured to add target pixels to the 3D model prior to capture of the 2D image.

[0044] In some instances, generating the uncalibrated training multispectral data cube further includes adding texture to the generated calibrated training multispectral data cube based on a texture associated with a corresponding captured 2D image.

[0045] In some instances, generating the uncalibrated training multispectral data cube further includes adding a simulated registration error.

[0046] In some instances, generating the uncalibrated training multispectral data cube further includes blurring at least one of the corresponding captured 2D images prior to generation.

[0047] In some instances, generating the uncalibrated training multispectral data cube further includes adding shot noise to at least one of the corresponding captured 2D images prior to generation.

[0048] According to a ninth aspect of the subject matter of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being configured to, by a processing circuit, generate a material database including: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a group of materials of which an element in the scene associated with the pixel is composed; and (B) a material database including a list of materials. (C) a heuristic table including one or more rules, each rule specifying a probability of the presence of a given material in a given scene based on characteristics of the given scene; and capturing, by a processing circuit, a two-dimensional (2D) image from the 3D model of the scene, wherein the 2D image is a 3D model of a pixel. generating, by the processing circuitry, a training calibrated multi-spectral data cube for the training record by, for at least one given pixel of the subset of pixels, querying, by the processing circuitry, a material database for a list of materials that are likely to be materials having a material group for the given pixel; removing, by the processing circuitry, materials from the list of likely materials that have a probability of being present in the 2D image less than a first threshold according to the rules of the heuristic table and the characteristics of the 3D model to obtain a revised list of likely materials; determining, by the processing circuitry, a matching material for the given pixel from the revised list of likely materials based on a match between the color of the given pixel and a typical color associated with a material from the revised list of likely materials; and selecting, by the processing circuitry, at least a portion of the spectral reflectance signature vector associated with the matching material to be part of the training calibrated multi-spectral data cube at the given pixel location;generating a training uncalibrated multispectral data cube for the training record utilizing the calibrated training multispectral data cube and an atmospheric simulator, the atmospheric simulator being capable of receiving (i) the training calibrated multispectral data cube and (ii) one or more atmospheric conditions and generating a training uncalibrated multispectral data cube corresponding to the calibrated training multispectral data cube under those atmospheric conditions;

[0049] According to a tenth aspect of the presently disclosed subject matter, there is provided a system for detecting one or more target materials in an uncalibrated multispectral data cube comprising a collection of pixels, the system comprising: (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and capable of determining, for at least one of the pixels, at least one material indicator indicative of the presence of a given one of the target materials at the location of the pixel, the machine learning model being trained using a labeled training data set comprising a plurality of training records, each training record comprising: (i) a training uncalibrated multispectral data cube; and (ii) a material indicator associated with at least one pixel of the training uncalibrated multispectral data cube. The system comprises a processing circuit configured to: (A) acquire a machine learning model including at least one training material indicator that indicates the presence of a target material at a pixel location; and (B) acquire an uncalibrated multispectral data cube; and determine, for at least one pixel of the uncalibrated multispectral data cube, at least one material indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated using a calibration process and an atmospheric simulator that simulates a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube.

[0050] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0051] In some cases, the machine learning model further includes a vision transformer.

[0052] In some cases, the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

[0053] According to an eleventh aspect of the subject matter of the present disclosure, there is provided a method for detecting one or more target materials in an uncalibrated multispectral data cube comprising a collection of pixels, the method comprising: (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and determining, for at least one of the pixels, at least one material indicator indicative of a presence of a given one of the target materials at the location of the pixel, the machine learning model being trained using a labeled training data set comprising a plurality of training records, each training record comprising: (i) the training uncalibrated multispectral data cube; and (ii) a material indicator associated with at least one pixel of the training uncalibrated multispectral data cube; The method includes acquiring (A) a machine learning model including at least one training material indicator that indicates the presence of a target material at a raw location; and (B) an uncalibrated multispectral data cube; and determining, by a processing circuit, for at least one pixel of the uncalibrated multispectral data cube, at least one material indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated by utilizing an atmospheric simulator to simulate a calibration process and a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube.

[0054] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0055] In some cases, the machine learning model further includes a vision transformer.

[0056] In some cases, the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

[0057] According to a twelfth aspect of the presently disclosed subject matter, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being configured to: (A) receive an uncalibrated multispectral data cube; and, for at least one pixel of the pixels, determine at least one material indicator indicative of a presence of a given target material of the target materials at the pixel location; the machine learning model is trained using a labeled training data set including a plurality of training records, each training record representing (i) the training uncalibrated multispectral data cube and (ii) a material indicator indicative of a presence of the target material at the pixel location, the material indicator being associated with at least one pixel of the training uncalibrated multispectral data cube. The method is executable by at least one processing circuit of a computer to perform a method including: acquiring (A) a machine learning model including at least one training substance indicator; and (B) an uncalibrated multispectral data cube; and determining, by the processing circuit, for at least one pixel of the uncalibrated multispectral data cube, at least one substance indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated by utilizing an atmospheric simulator to simulate a plurality of simulated uncalibrated multispectral data cubes through a calibration process and simulation of different atmospheric conditions for the calibrated multispectral cube.

[0058] According to a thirteenth aspect of the subject matter of the present disclosure, there is provided a system for empirical atmospheric calibration utilizing automatically identified objects in an image of a scene, the system comprising: (A) a machine learning model capable of receiving an image of the scene and identifying the presence of at least one of the automatically identified objects in the scene, where each identified object is associated with (i) a location in the scene and (ii) a predetermined representative reflectance spectral signature; and (B) a processing circuit configured to: acquire the image of the scene; identify the presence of the at least one of the automatically identified objects in the image of the scene using the machine learning model and the image of the scene; and calibrate an uncalibrated multispectral data cube associated with the scene based on the locations and the predetermined representative reflectance spectral signature associated with the automatically identified objects using an empirical atmospheric calibration process.

[0059] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0060] In some cases, the machine learning model further includes a vision transformer.

[0061] In some cases, the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

[0062] According to a fourteenth aspect of the subject matter of the present disclosure, there is provided a method for empirical atmospheric calibration utilizing automatically-identified objects in an image of a scene, the method including: acquiring, by a processing circuit, (A) a machine learning model capable of receiving an image of the scene and identifying the presence of at least one of the automatically-identified objects in the scene, wherein each identified object is associated with (i) a location in the scene and (ii) a predetermined representative reflectance spectral signature; and (B) the image of the scene; identifying, by the processing circuitry, the presence of at least one of the automatically-identified objects in the image of the scene using the machine learning model and the image of the scene; and calibrating, by the processing circuitry, an uncalibrated multispectral data cube associated with the scene based on the locations and the predetermined representative reflectance spectral signatures associated with the automatically-identified objects using an empirical atmospheric calibration process.

[0063] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0064] In some cases, the machine learning model further includes a vision transformer.

[0065] In some cases, the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

[0066] According to a fifteenth aspect of the subject matter of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being executable by at least one processing circuit of a computer to perform a method including: acquiring, by a processing circuit, (A) a machine learning model capable of receiving an image of a scene and identifying the presence of at least one object among the objects to be automatically identified in the scene, wherein each object to be identified is associated with (i) a location in the scene and (ii) a predetermined representative reflectance spectral signature; and (B) an image of the scene; identifying, by the processing circuit, the presence of at least one object among the objects to be automatically identified in the image of the scene utilizing the machine learning model and the image of the scene; and calibrating, by the processing circuit, an uncalibrated multispectral data cube associated with the scene based on the locations and the predetermined representative reflectance spectral signatures associated with the objects to be automatically identified using an empirical atmospheric calibration process.

[0067] According to a sixteenth aspect of the subject matter of the present disclosure, there is provided a system for determining an aligned multispectral data cube from one or more two-dimensional (2D) images of a scene, each 2D image captured at a different wavelength range, each image taken from a different viewpoint of the scene, and the aligned multispectral data cube is potentially generateable from the 2D images by an alignment process, the system comprising: (A) a machine learning model capable of receiving (a) a source 2D image of the 2D images of the scene and (b) a target 2D image of the 2D images of the scene and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model being trained using a training data set including a plurality of training records, each training record comprising: (i) a training source 2D image, (ii) a training target 2D image, and (iii) a training image to align the training source 2D image with the training target 2D image. The system comprises a processing circuit configured to: (A) acquire (B) given 2D images of a scene, each given 2D image having a different wavelength range and each image being taken from a different viewpoint of the scene; (C) determine at least one given flow map for at least one given source 2D image of the given 2D images and at least one given target 2D image of the given 2D images, the given flow map mapping changes to be made to pixels of the given source 2D image in order to align the given source 2D image with the given target 2D image by utilizing the machine learning model for the given source 2D image and the given target 2D image; and (D) generate an aligned multispectral data cube using the given source 2D image, the corresponding given flow map, and the remapping function.

[0068] In some cases, at least one given training record of the training data set comprises: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which the element in the scene associated with the pixel is composed; and (B) a materials database including a list of materials, at least one of which is associated with: (i) a material spectral reflectance signature vector; (ii) a material typical color; and (iii) a material database associated with a given material group of materials; and (C) a heuristic table including one or more rules, each rule specifying a probability of the presence of a given material in the given scene based on characteristics of the given scene; and capturing at least one 2D image from a 3D model of the scene, each 2D image being captured from a different viewpoint of the scene, each 2D image including a subset of pixels, each training 2D image being associated with a different wavelength range, obtaining (capturing) training source 2D images of the training records; for at least one training source 2D image, selecting a training target 2D image from among the 2D images captured from the 3D model to obtain a training target 2D image for the given training record; for at least one given pixel of the subset of pixels of the training source 2D image, querying a material database for a list of materials that may be materials having a material group for the given pixel; removing from the list of possible materials materials that have a probability of being present in the 2D image less than a first threshold according to the rules of the heuristic table and the characteristics of the 3D model to obtain a revised list of possible materials; determining a matching material for the given pixel from the revised list of possible materials based on a match between the color of the given pixel and a representative color associated with a material from the revised list of possible materials; selecting a corresponding portion of a spectral reflectance signature vector associated with the matching material according to the wavelength of the 2D image to be the portion at the given pixel location in the training source 2D image;and generating a training flow map for a given training recording that maps modifications made to pixels of the source 2D image to align the source 2D image with the target 2D image.

[0069] In some cases, at least one consecutive pair of 2D images of the scene overlap by more than an overlap threshold.

[0070] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0071] In some cases, the machine learning model further includes a vision transformer.

[0072] In some cases, the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, or a viewing distance.

[0073] According to a seventeenth aspect of the subject matter of the present disclosure, there is provided a method for determining an aligned multispectral data cube from one or more two-dimensional (2D) images of a scene, each 2D image captured in a different wavelength range, each image taken from a different viewpoint of the scene, and the aligned multispectral data cube can potentially be generated from the 2D images by an alignment process, the method comprising: (A) a machine learning model capable of receiving, by a processing circuit, (a) a source 2D image among the 2D images of the scene and (b) a target 2D image among the 2D images of the scene and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model being trained using a training data set including a plurality of training records, each training record comprising: (i) a training source 2D image, (ii) a training target 2D image, and (iii) a training source 2D image to align the training source 2D image with the training target 2D image. and (B) acquiring given 2D images of a scene, each given 2D image having a different wavelength range and each image taken from a different viewpoint of the scene; (C) determining, by a processing circuitry, at least one given flow map for at least one given source 2D image of the given 2D images and at least one given target 2D image of the given 2D images, the given flow map mapping changes to be made to pixels of the given source 2D image in order to align the given source 2D image with the given target 2D image by utilizing the machine learning model for the given source 2D image and the given target 2D image; and (D) generating, by the processing circuitry, an aligned multispectral data cube using the given source 2D images, the corresponding given flow map, and the remapping function.

[0074] In some instances, at least one given training record of the training data set includes: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which the element in the scene associated with the pixel is composed; and (B) a materials database including a list of materials, at least one of which is associated with (i) a material spectral reflectance signature base. (c) acquiring a material database including (i) a material vector associated with a given material group, (ii) a material typical color, and (iii) a material characteristic associated with a given material group; and (C) a heuristic table including one or more rules, each rule specifying a probability of the presence of a given material in a given scene based on characteristics of the given scene; and capturing at least one 2D image from a 3D model of the scene, each 2D image being captured from a different viewpoint of the scene, each 2D image including a subset of pixels, each training 2D image including a plurality of 2D images. The 2D images are associated with different wavelength ranges to obtain a training source 2D image for a given training record; for at least one training source 2D image, selecting a training target 2D image from among the 2D images captured from the 3D model to obtain a training target 2D image for the given training record; for at least one given pixel of the subset of pixels of the training source 2D image, querying a material database for a list of substances that may be a substance having a material group for the given pixel; removing from the list of possible substances substances whose probability of being present in the 2D image is less than a first threshold according to the rules of the heuristic table and the characteristics of the 3D model to obtain a revised list of possible substances; determining a matching substance for the given pixel from the revised list of possible substances based on a match between the color of the given pixel and a typical color associated with a substance from the revised list of possible substances; determining a corresponding portion of a spectral reflectance signature vector associated with the matching substance according to the wavelength of the 2D image;generating a training flow map for a given training recording by selecting a portion of a source 2D training image at a given pixel location, and generating a training flow map for a given training recording that maps changes made to pixels of the source 2D image to align the source and target 2D images.

[0075] In some cases, at least one consecutive pair of 2D images of the scene overlap by more than an overlap threshold.

[0076] In some instances, the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

[0077] In some cases, the machine learning model further includes a vision transformer.

[0078] In some cases, the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, and a viewing distance.

[0079] According to an eighteenth aspect of the subject matter of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being configured to, by a processing circuit, include: (A) a machine learning model capable of receiving (a) a source 2D image among the 2D images of a scene and (b) a target 2D image among the 2D images of the scene and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model being trained using a training data set including a plurality of training records, each training record including (i) a training source 2D image, (ii) a training target 2D image, and (iii) one or more training flow maps that map modifications to be made to pixels of the training source 2D image to align the training source 2D image with the training target 2D image; and (B) a given 2D image of a scene. The method is executable by at least one processing circuit of a computer to perform a method including: acquiring given 2D images, each given 2D image having a different wavelength range and each image taken from a different viewpoint of a scene; determining, by the processing circuitry, at least one given flow map for at least one given source 2D image of the given 2D images and at least one given target 2D image of the given 2D images, wherein the given flow map maps changes to be made to pixels of the given source 2D image to align the given source 2D image with the given target 2D image by utilizing a machine learning model for the given source 2D image and the given target 2D image; and generating, by the processing circuitry, an aligned multispectral data cube using the given source 2D images, the corresponding given flow map, and the remapping function.

[0080] In order to understand the subject matter of the present disclosure and to see how it may be carried out in practice, the subject matter will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0081] [Figure 1] 1 is a block diagram that schematically illustrates an example of a real-time multispectral system in accordance with the subject matter of the present disclosure. [Figure 2A] 1 is a schematic diagram of one possible exemplary configuration of a real-time multispectral system in accordance with the subject matter of the present disclosure. [Figure 2B] 1 is a schematic diagram of one possible exemplary design of a processing chain for a real-time multispectral system in accordance with the subject matter of the present disclosure. [Figure 3] 1 is a flowchart illustrating an example sequence of operations performed by a real-time multispectral system for target detection and further investigation in accordance with the subject matter of the present disclosure. [Figure 4] 1 is a flowchart illustrating an example series of operations performed by a real-time multispectral system with automatic exposure time calculation in accordance with the disclosed subject matter. [Figure 5] 1 is a flowchart illustrating an example series of operations performed by a real-time multispectral system for automatic atmospheric calibration matrix determination in accordance with the subject matter of the present disclosure. [Figure 6] 1 is a flowchart illustrating an example sequence of operations performed by a real-time multispectral system for multi-channel registration in accordance with the subject matter of the present disclosure. [Figure 7] 1 is a flowchart illustrating an example of a series of operations performed to automatically generate a calibrated multispectral data cube from an uncalibrated multispectral data cube in accordance with the disclosed subject matter. [Figure 8] 1 is a flowchart illustrating an example of a series of operations performed for the automated generation of a multispectral labeled training data set in accordance with the subject matter of this disclosure. [Figure 9]1 is a flowchart illustrating an example of a series of operations performed to detect one or more target materials in an uncalibrated multispectral data cube in accordance with the disclosed subject matter. [Figure 10] 1 is a flowchart illustrating an example sequence of operations performed for empirical atmospheric calibration utilizing automatically identified target objects in images of a scene, in accordance with the subject matter of this disclosure. [Figure 11] 1 is a flowchart illustrating an example of a series of operations performed to determine an aligned multispectral data cube from one or more two-dimensional images of a scene, in accordance with the subject matter of this disclosure. [Figure 12] FIG. 1 illustrates an example of steps performed to automatically generate a calibrated multispectral data cube from an uncalibrated multispectral data cube in accordance with the disclosed subject matter. [Figure 13] FIG. 1 illustrates an example of steps performed to determine an aligned multispectral data cube from one or more two-dimensional images of a scene, in accordance with the subject matter of this disclosure. [Figure 14] FIG. 10 illustrates another example of steps performed to determine an aligned multispectral data cube from one or more two-dimensional images of a scene, in accordance with the subject matter of this disclosure. [Figure 15] FIG. 1 illustrates an example of steps performed for the automated generation of a multispectral labeled training data set in accordance with the subject matter of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0082] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the subject matter of the present disclosure. However, it will be understood by those skilled in the art that the subject matter of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the subject matter of the present disclosure.

[0083] In the drawings and written description, the same reference numbers designate components that are common to different embodiments or configurations.

[0084] Unless otherwise specified, as will be apparent from the following description, it will be appreciated that the use of terms such as "obtaining," "identifying," "capturing," "investigating," "calculating," "generating," "determining," and the like throughout the description herein includes computer actions and / or processes that manipulate and / or transform data to be represented as physical quantities, e.g., electronic quantities, and / or other data that represent physical objects. The terms "computer," "processor," "processing resource," "processing circuitry," and "controller" should be interpreted broadly to cover any type of electronic device with data processing capability, including, by way of non-limiting example, personal desktop / laptop computers, servers, computing systems, communications devices, smartphones, tablet computers, smart televisions, processors (e.g., Digital Signal Processors (DSPs), Graphics Processing Units (GPUs), microcontrollers, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), etc.), a group of multiple physical machines that share the performance of various tasks, virtual servers co-resident on a single physical machine, any other electronic computing device, and / or any combination thereof.

[0085] Operations according to the teachings herein may be performed by a specially constructed computer for the desired purpose, or by a general-purpose computer specially configured for the desired purpose by a computer program stored on a non-transitory computer-readable storage medium. The term "non-transitory" is used herein to exclude transitory propagating signals, but in some instances to include any volatile or non-volatile computer memory technology suitable for the application.

[0086] As used herein, the phrases "for example," "such as," "for instance," and variations thereof describe non-limiting embodiments of the presently disclosed subject matter. Reference herein to "one case," "some cases," "other cases," or variations thereof means that a particular feature, structure, or characteristic described with respect to one or more embodiment(s) is included in at least one of the presently disclosed subject matter. Thus, appearances of the phrases "one case," "some cases," "other cases," or variations thereof do not necessarily refer to the same embodiment(s).

[0087] It will be appreciated that, unless otherwise stated, certain features of the presently disclosed subject matter, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the presently disclosed subject matter, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination.

[0088] In embodiments of the presently disclosed subject matter, fewer, more, and / or different steps than those shown in FIGS. 3-11 may be performed. In embodiments of the presently disclosed subject matter, one or more steps shown in FIGS. 3-11 may be performed in a different order and / or one or more groups of steps may be performed simultaneously. FIGS. 1-2 are schematic diagrams of system architectures according to embodiments of the presently disclosed subject matter. Each module in FIGS. 1-2 may be comprised of any combination of software, hardware, and / or firmware that performs the functions defined and described herein. The modules in FIGS. 1-2 may be concentrated in one location or distributed across two or more locations. In other embodiments of the presently disclosed subject matter, a system may include fewer, more, and / or different modules than those shown in FIGS. 1-2.

[0089] References in this specification to methods should apply mutatis mutandis to systems capable of carrying out the methods, and should apply mutatis mutandis to non-transitory computer-readable media storing instructions that, when executed by a computer, result in the performance of the methods.

[0090] References herein to a system should apply mutatis mutandis to methods that may be performed by the system, and should apply mutatis mutandis to non-transitory computer-readable media that store instructions that may be executed by the system.

[0091] References in this specification to non-transitory computer readable media should apply mutatis mutandis to a system capable of executing instructions stored on the non-transitory computer readable media, and should apply mutatis mutandis to a method that can be executed by a computer reading instructions stored on the non-transitory computer readable media.

[0092] With the above in mind, attention is now directed to FIG. 1, which illustrates a block diagram that generally illustrates an example of a real-time multispectral system in accordance with the subject matter of the present disclosure.

[0093] In accordance with the subject matter of the present disclosure, a real-time multispectral system 100 (interchangeably referred to herein as "multispectral system 100" or "system 100") can include a multispectral sensor 104 capable of capturing image data within specific wavelength ranges across the electromagnetic spectrum, such as visible (VIS), ultraviolet (UV), short-wave infrared (SWIR), near-infrared (NIR), middle-wave infrared (MWIR), long-wave infrared (LWIR), or combinations thereof. The wavelengths may be separated by multiple filters or detected using instruments sensitive to specific wavelengths, including frequencies of light beyond the visible light range, such as near-infrared, short-wave infrared, infrared, and ultraviolet. As a non-limiting example, the multispectral sensor 104 can capture images of a given scene or a given field of view (FOV) in multiple imaging channels, each with a different wavelength range, by utilizing a single sensor with at least one rotating filter wheel. The filter wheel is used to rotatably position filters having a given wavelength range in the sensor's imaging path to capture images in that wavelength range. The filter in the sensor's imaging path for a given time window and / or time interval is the active filter for that time window. The wheel is then rotated to position the next filter, which has a different wavelength range in the sensor's imaging path, allowing the sensor to capture images in different wavelength ranges. Rotating the filter wheel enables the multispectral sensor 104 to capture a series of images in multiple wavelength ranges over a given time frame. The rotating filter wheel can have a rotation axis parallel or perpendicular to the optical axis of the multispectral sensor 104. In some cases, the rotating filter wheel rotates around the detector of the multispectral sensor 104.The number of filters on the filter wheel determines the number of images in different wavelength ranges that can be captured by the multispectral sensor 104. Alternatively, the multispectral sensor 104 can include two or more filter wheels, which can be used to simultaneously combine different filters from each of the filter wheels. Alternatively, the multispectral sensor 104 can utilize one or more of the following filtering methods: (a) a Fabry-Perot interferometer (FPI), (b) a linear variable filter (LVF), and (c) a circular variable filter. Alternatively, the multispectral sensor 104 can be configured with multiple sensors, each capturing images of a scene in a different wavelength range, or with a combination of multiple sensors each with a filter that enables the sensor to capture a different wavelength range. Embodiments with a filter wheel allow the multispectral sensor 104 to have a smaller footprint than multispectral sensors 104 that use multiple sensors to capture images of a given scene in multiple wavelength ranges over a given time frame. Optionally, system 100 may be capable of sending different commands to the detectors of multispectral sensor 104 for each filter on the filter wheel. Optionally, system 100 may be capable of sampling multiple frames per filter (e.g., 1 to 50 frames per filter). Optionally, system 100 may be capable of calculating the average number of frames used for each filter on the filter wheel. Optionally, multispectral sensor 104 may be implemented using a shortwave infrared (SWIR) sensor capable of capturing a given scene at SWIR wavelengths. In some instances, multispectral sensor 104 may be implemented using a visible-SWIR (VIS-SWIR) sensor capable of capturing a given scene at visible and / or SWIR wavelengths.In some cases, the SWIR sensor or VIS-SWIR sensor can be a wide-field-of-view spectral SWIR sensor or a wide-field-of-view spectral VIS-SWIR sensor capable of capturing a given scene at a wide field-of-view angle of SWIR wavelengths or a wide field-of-view angle of VIS-SWIR wavelengths. Optionally, the multispectral sensor 104 can be coupled to a rotatable filter wheel that rotatably positions filters having a given wavelength range in the imaging path of the multispectral sensor 104 to capture images in that wavelength range. In some cases, the multispectral sensor 104 can include a wide-field SWIR simple mode (also referred to as a wide-field SWIR standard imaging mode) in which no filters are applied to the sensor, for example, by placing a non-filter portion of the filter wheel in the imaging path of the sensor. The multispectral sensor 104 can be used to capture spectral data that is utilized to generate a spectral data cube (also referred to herein as a “multispectral data cube”) of a given scene. The spectral data cube includes two spatial dimensions (x and y) and one spectral dimension, where the face of the cube is a function of the spatial coordinates and pixel resolution of the multispectral sensor 104 representing the captured scene, and the depth is a function of the wavelength ranges of the multiple imaging channels or multiple filters on a filter wheel used by the multispectral sensor 104. The spectral data cube provides a simple way to read, manipulate, analyze, and write data with two positional dimensions and one spectral dimension.

[0094] The process of capturing a spectral data cube may be as follows: The multispectral sensor 104 captures a raw spectral data cube. The raw spectral data cube includes gray level values ​​representing the scene captured by the multispectral sensor 104. The system 100 may perform radiometric correction on the raw spectral data cube to generate an emission spectral data cube. The information contained in the emission spectral data cube has physical meaning regarding the objects in the scene and their radiance power. The system 100 may then perform radiometric correction on the emission spectral data cube to generate a reflectance spectral data cube. The reflectance spectral data cube includes information regarding the reflectance of the objects in the scene, i.e., how reflective these objects are at each of the wavelengths captured by the multispectral sensor 104. Note that when referring to a spectral data cube herein, unless otherwise specified, the meaning refers to an emission spectral data cube.

[0095] The multispectral system 100 can analyze the spectral data cube to identify pixels in two spatial dimensions that correspond to the spectral signature of a given material type the system is attempting to detect within a given scene. A non-limiting example is a multispectral sensor 104 with a filter wheel containing 12 different filters. Such a multispectral sensor 104 would generate a spectral data cube for a given scene within a given time frame with a depth of 12. Each pixel on the face of the spectral data cube represents an area of ​​the scene and is associated with a 12-value vector. Each value in this vector is associated with the spectral data captured by the multispectral sensor 104 for that area of ​​the scene within that time frame, for the wavelength range associated with the corresponding filter on the filter wheel.

[0096] Optionally, multispectral system 100 may utilize machine learning (ML) and / or artificial intelligence (AI) models (also referred to herein as machine learning models) to perform at least a portion of the processes for capturing spectral data cubes, analyzing spectral data cubes, detecting targets, and / or any other aspects of the processes performed by multispectral system 100. These AI and / or ML processes may include, but are not limited to, a process for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, a process for detecting one or more target materials in an uncalibrated multispectral data cube, a process for empirical atmospheric calibration utilizing automatically identified objects of interest in images of a scene, a process for determining an aligned multispectral data cube from one or more two-dimensional images of a scene, etc. These AI and / or ML processes are described in further detail below.

[0097] System 100 can also automatically generate hyperspectral and / or multispectral training data sets. These automatically generated training data sets can be ground truth and / or labeled data sets, and system 100 automatically generates one or more pairs including an uncalibrated training multispectral data cube and a corresponding calibrated training multispectral data cube. The corresponding calibrated training multispectral data cube can be generated from the uncalibrated training multispectral data cube through a calibration process. Note that the generated training data sets can be used by system 100 itself, for example, to train a machine model that performs the automatic generation of calibrated multispectral data cubes from uncalibrated multispectral data cubes. The generated training data sets can also be used by other systems external to system 100 for various tasks, such as, for example, to train machine learning models that require hyperspectral and / or multispectral training data sets as part of the training process. The process of generating hyperspectral and / or multispectral training data sets is described in further detail below.

[0098] As described above, the multispectral sensor 104 can capture a raw multispectral data cube (also referred to herein as an “uncalibrated multispectral data cube”). The raw multispectral data cube can include gray level values ​​representing a given scene captured by the multispectral sensor 104. The system 100 can perform one or more calibration processes on the raw multispectral data cube. These can include radiometric correction of the raw multispectral data cube to generate an emission spectrum data cube, atmospheric correction of the emission spectrum data cube to generate a reflectance spectrum data cube, or a combination thereof. In some cases, the raw multispectral data cube can be obtained by utilizing other sensors and / or from another system external to the system 100. The calibration process can be performed by the system 100 employing an analytical calibration process using a given correction algorithm, as further detailed herein, particularly with reference to FIG. 4 . Additionally or alternatively, as detailed herein, the system 100 may perform calibration of the raw multispectral data cube by utilizing an atmospheric calibration based on a machine learning model.

[0099] Machine learning models are programs that can find patterns and make decisions from previously unseen data sets. They can solve problems that would be prohibitively expensive for human programmers to develop algorithms for; instead, the problem is solved by helping machines "discover" their own algorithms without needing to be explicitly told what to do by human-developed algorithms. Machine learning models have the ability to perform accurately on novel, unseen examples / tasks after going through a training dataset used to train the machine learning model. Training examples are drawn from some generally unknown probability distribution (thought to represent the emergent space), and the AI ​​and / or ML process builds a general model of this space that allows it to generate sufficiently accurate predictions when encountering novel examples.

[0100] A process for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube can utilize a machine learning model trained on a training data set that includes multiple pairs, each pair including an uncalibrated multispectral data cube and a corresponding calibrated multispectral data cube. The calibrated multispectral data cube can potentially be generated from the uncalibrated multispectral data cube through a calibration process (e.g., an atmospheric calibration process). The training data set includes ground truth (labeled) information from different scenes. At least a portion of the training data set can be automatically generated by utilizing an atmospheric simulator on another calibrated multispectral data cube. The automatic generation of training data can be used to expand and diversify input training and testing data for the machine learning model. The atmospheric simulator can receive a calibrated multispectral data cube and one or more atmospheric conditions (e.g., humidity level, sunlight angle, aerosol model, visibility score, or geographic area) and generate one or more uncalibrated multispectral data cubes corresponding to the calibrated multispectral data cube under those atmospheric conditions. System 100 can utilize the atmospheric simulator to generate two or more uncalibrated multispectral data cubes using different atmospheric conditions from a single given calibrated multispectral data cube, thereby facilitating the creation of training data set pairs from a small number of calibrated multispectral data cubes. A non-limiting example of such an atmospheric simulator is the Moderate Resolution Atmospheric Radiative Transfer (MODTRAN) computer code. Additionally, as further described herein, at least a portion of the training data set can be generated automatically by utilizing an automated process for generating multispectral labeled training data sets.

[0101] The trained machine learning model can receive a given uncalibrated multispectral data cube and generate a corresponding calibrated multispectral data cube. The generated corresponding calibrated multispectral data cube is equivalent to the result of applying an analytical calibration process to the given uncalibrated multispectral data cube. The machine learning model can have, as part of the training process, a loss function that represents the cost of inaccuracy in predictions. The loss function in the machine learning model can determine weights and biases for at least a portion of the pixels of the uncalibrated multispectral data cube.

[0102] Using a machine learning model to generate a calibrated multispectral data cube from an uncalibrated multispectral data cube has several advantages over more traditional methods that apply an analytical calibration process using a given correction algorithm to the uncalibrated multispectral data cube to generate a corresponding calibrated multispectral data cube. The machine learning model can perform atmospheric correction on the uncalibrated multispectral data cube to generate an atmospherically calibrated multispectral data cube that contains reflectance information about the reflectance of objects in the scene captured in the multispectral data cube, i.e., how reflective these objects are at each wavelength. In some cases, the atmospheric correction can include an inter-instrument matching correction to correct for mismatches between the multispectral sensor 104 and the known spectra of the captured materials. Additionally, because the machine learning model can operate at the pixel level, different optimization parameters (e.g., illumination angle, illumination distance) may be applied to different pixels of the uncalibrated multispectral data cube. In contrast, an analytical calibration process typically determines only one set of optimization parameters to be used for all pixels of the uncalibrated multispectral data cube.

[0103] The machine learning model may be constructed using one or more of model architectures including an encoder-decoder model, a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, a visual transformer model, or any other model architecture that may be used in building and training a machine learning model. In some cases, the machine learning model may be a neural network that processes video of the continuous multispectral data cube using one or more of the above-mentioned model architectures combined with one or more of the following architectures: a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), a visual transformer, a bidirectional encoder representations from transformers (BERT), a dilated convolutional network, a temporal convolutional network (TCN), an ordinary differential equation (ODE), etc. Training the machine learning model may include pruning. The machine learning model may also be trained using a reinforcement learning method. The machine learning model may further include a vision transformer. The vision transformer may enable the machine learning model to code one or more segments in the uncalibrated multispectral data cube.

[0104] A non-limiting example of steps and stages of a process for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube is shown in FIG. 12. In step 1202, at least one given uncalibrated multispectral data cube is used as input to the process. In step 1204, the given uncalibrated multispectral data cube is annotated by tagging and / or labeling elements with known spectral reflectances captured in one or more of the layers of the given uncalibrated multispectral data cube. The annotations associate spectral reflectance values ​​with the elements. In some cases, the tagging is performed automatically. In other cases, the tagging is performed by a human operator of system 100. In step 1206, the annotated given uncalibrated multispectral data cube undergoes a calibration process utilizing annotations of tagged substances with known spectral reflectances and data from a substances database in step 1208. The result in step 1210 is one or more calibrated multispectral data cubes. In step 1212, an atmospheric simulator is utilized to generate one or more uncalibrated multispectral data cubes (generated in step 1216) that correspond to the calibrated multispectral data cube under atmospheric conditions. In step 1214, a physical simulator may additionally and / or optionally be utilized on the calibrated multispectral data cube to add optical blur effects, add one or more alignment errors, and / or add shot noise when generating the one or more uncalibrated multispectral data cubes (generated in step 1216). The atmospheric simulator may receive the calibrated multispectral data cube and one or more atmospheric conditions (such as humidity level, sunlight angle, aerosol model, visibility score, or geographic area) and may generate one or more uncalibrated multispectral data cubes that correspond to the calibrated multispectral data cube under those atmospheric conditions.System 100 may utilize an atmospheric simulator to generate two or more uncalibrated multispectral data cubes with known target materials using different atmospheric conditions from a single, given calibrated multispectral data cube, thereby facilitating the creation of large training data sets from a small number of calibrated multispectral data cubes labeled with known target materials and their locations within the scene. A non-limiting example of such an atmospheric simulator is the Moderate Resolution Atmospheric Radiative Transfer (MODTRAN) computer code. Additionally, as further detailed herein, at least a portion of the training data set may be generated automatically by utilizing an automated process for generating multispectral labeled training data sets.

[0105] The process of automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube may be performed by system 100 as part of another process performed by system 100 (e.g., as part of a target identification and survey process performed by system 100). In other cases, the process of automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube may be performed by system 100 as a stand-alone process independent of other processes performed by system 100. The results of the process of automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube may also be used by other systems external to system 100 for various tasks.

[0106] The process of automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube is described in further detail herein, with particular reference to FIG.

[0107] System 100 can perform a target identification and reconnaissance process, as further described herein, with particular reference to FIG. 3 . The target identification and reconnaissance process can be performed by applying multiple analytical steps to a given uncalibrated multispectral data cube that maintains raw spectral data captured by a multispectral sensor (e.g., multispectral sensor 104). These multiple analytical steps can include radiometric correction of the uncalibrated multispectral data cube, atmospheric correction of the uncalibrated multispectral data cube, and target detection on the corrected multispectral data cube. At least some of these steps of the target identification and reconnaissance process can be performed by directly detecting target materials from the uncalibrated multispectral data cube by utilizing a machine learning model. The utilization of a machine learning model can omit at least some of the aforementioned steps, thereby saving computational time and reducing the likelihood of errors. A machine learning model used to directly detect target materials from a given uncalibrated multispectral data cube can receive the given uncalibrated multispectral data cube and determine at least one material indicator for at least one of the pixels of the given uncalibrated multispectral data cube. The material indicator indicates the presence of a given target material among the target materials at that pixel location. The process of detecting one or more target materials in the uncalibrated multispectral data cube enables semantic alignment of the target materials to pixels, and each pixel is mapped to a vector of target material indicators (e.g., for each given target material, a vector of binary indicators indicating whether the given target material is present in that pixel). Because the machine learning model operates at the pixel level, it can detect target materials occupying a single pixel in the uncalibrated multispectral data cube, which differs from analytical methods that typically require two or more pixels of a given target material for detection.In some cases, the machine learning model can detect target materials at sub-pixel dimensions: in these cases, the actual size of a given target material is smaller than the associated pixel, but because the machine learning model operates at the pixel level, the entire pixel is lit for a given target material to indicate its presence in the scene.

[0108] The process of detecting one or more target substances in an uncalibrated multispectral data cube can utilize a machine learning model trained on a labeled training data set, the labeled training data set including a plurality of training records, each training record including a training uncalibrated multispectral data cube and at least one training substance indicator associated with at least one pixel of the training uncalibrated multispectral data cube to indicate the presence of a target substance at that pixel. The training substance indicator can be, for example, a vector of binary indicators associated with corresponding pixels, each binary indicator indicating a given target substance. The training data set includes ground truth (labeled) information from different scenes. At least a portion of the training data set can be automatically generated by using an atmospheric simulator on another calibrated multispectral data cube with known target substance labels. The automatic generation of training data can be used to expand and diversify input training and test data for the machine learning model. The atmospheric simulator can receive a calibrated multispectral data cube and one or more atmospheric conditions (such as humidity level, sunlight angle, aerosol model, visibility score, or geographic area) and generate one or more uncalibrated multispectral data cubes corresponding to the calibrated multispectral data cube under those atmospheric conditions. System 100 can utilize the atmospheric simulator to generate two or more uncalibrated multispectral data cubes with known target materials using different atmospheric conditions from a single given calibrated multispectral data cube, thereby facilitating the creation of large training data sets from a small number of calibrated multispectral data cubes labeled with known target materials and their locations within the scene. A non-limiting example of such an atmospheric simulator is the Moderate Resolution Atmospheric Radiative Transfer (MODTRAN) computer code.Additionally, as further detailed herein, at least a portion of the training data set may be generated automatically by utilizing an automated process for generating multispectral labeled training data sets.

[0109] The machine learning model used to directly detect target materials from a given uncalibrated multispectral data cube may be constructed utilizing one or more of the following model architectures: an encoder-decoder model, a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, a visual transformer model, or any other model architecture that may be used in building and training a machine learning model. In some cases, the machine learning model may be a neural network that processes video of the continuous multispectral data cube using one or more of the above-mentioned model architectures combined with one or more of the following architectures: a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), a visual transformer, a bidirectional encoder representation with transformers (BERT), a dilated convolutional network, a time-series convolutional network (TCN), an ordinary differential equation (ODE), etc. Training the machine learning model may include pruning. The machine learning model may also be trained using a reinforcement learning method. The machine learning model may further include a vision transformer. The vision transformer may enable the machine learning model to code one or more segments in the uncalibrated multispectral data cube.

[0110] The process of detecting one or more target materials in an uncalibrated multispectral data cube may be performed by system 100 as part of another process performed by system 100 (e.g., as part of a target identification and investigation process performed by system 100). In other cases, the process of detecting one or more target materials in an uncalibrated multispectral data cube may be performed by system 100 as a stand-alone process independent of other processes performed by system 100. The results of the process of detecting one or more target materials in an uncalibrated multispectral data cube may also be used by other systems external to system 100 for various tasks.

[0111] The process of detecting one or more target materials in an uncalibrated multispectral data cube is described in further detail herein, with particular reference to FIG.

[0112] The target identification and survey process performed by system 100 includes multiple steps, as further detailed herein, with particular reference to FIG. 3 . Optionally, these steps may include determining an atmospheric calibration matrix for multispectral sensor 104 based on input from a user of system 100. A non-limiting example of such a user-dependent step is a step in which the user uses a control to move a cursor over an image captured by system 100 to indicate that the pixel corresponding to the cursor's location is a pixel of an object with known reflectance (e.g., indicating that the cursor is over a road, tree, gravel, etc.). System 100 can then determine the atmospheric calibration matrix using the known reflectances of these pixels in the image. At least a portion of this step of the user identifying one or more objects with known reflectances in the image of the scene can be performed by an empirical atmospheric calibration process using automatically identified objects in the image of the scene process, which can automatically identify calibration objects in the scene. The process of empirical atmospheric calibration using automatically identified objects in the image of the scene process can utilize a machine learning model. The use of a machine learning model omits at least some of the aforementioned steps, thereby saving computational time and reducing the likelihood of errors. The machine learning model used in the process for empirical atmospheric calibration utilizing automatically-identified target objects in an image of a scene can receive an image of a scene and identify the presence of at least one automatically-identified target object in the scene. The automatically-identified target object is associated with a given location in the scene and a predetermined representative reflectance spectral signature for that object. The system 100 can determine an atmospheric calibration matrix using the at least one automatically-identified target object in the image of the scene without relying on a user of the system 100 to manually select a calibration object.

[0113] A process for empirical atmospheric calibration utilizing automatically identified target objects within an image of a scene can identify the presence of one or more objects (e.g., a given road) within the image of the scene. These automatically identified target objects can be used by system 100 as calibration objects. Identification of these calibration objects can be achieved using a machine learning model and the image of the scene. The process can then utilize the empirical atmospheric calibration process to calibrate an uncalibrated multispectral data cube associated with the scene based on the locations and predetermined representative reflectance spectral signatures associated with the one or more identified calibration objects.

[0114] The machine learning model used to automatically identify objects in images of a scene may be constructed utilizing one or more of the following model architectures: an encoder-decoder model, a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, a visual transformer model, or any other model architecture that may be used in building and training a machine learning model. In some cases, the machine learning model may be a neural network that processes video of continuous multispectral data cubes using one or more of the above-mentioned model architectures combined with one or more of the following architectures: recurrent neural network (RNN), long short-term memory network (LSTM), gated recurrent unit (GRU), visual transformer, bidirectional encoder representation with transformers (BERT), dilated convolutional network, time-series convolutional network (TCN), ordinary differential equation (ODE), etc. Training the machine learning model may include pruning. The machine learning model may also be trained using reinforcement learning methods. The machine learning model may further include a vision transformer. The vision transformer may enable the machine learning model to code one or more segments within an image of a scene.

[0115] The process for empirical atmospheric calibration using automatically identified objects in images of a scene may be performed by system 100 as part of other processes performed by system 100 (e.g., as part of a target identification and survey process performed by system 100). In other cases, the process for empirical atmospheric calibration using automatically identified objects in images of a scene may be performed by system 100 as a stand-alone process independent of other processes performed by system 100. The results of the process for empirical atmospheric calibration using automatically identified objects in images of a scene may also be used by other systems external to system 100 for various tasks.

[0116] A process for empirical atmospheric calibration utilizing automatically identified objects of interest within images of a scene is described in further detail herein, with particular reference to FIG.

[0117] In some cases, the target identification and survey process performed by system 100 includes aligning multiple images to a single coordinate system that will be used to construct an uncalibrated multispectral data cube. As a non-limiting example, in cases where system 100 is moving while capturing images of a scene, or where multispectral sensor 104 is coupled to a rotating filter wheel that rotatably positions filters having a given wavelength range in the imaging path of multispectral sensor 104 and is used to capture images within that wavelength range, the captured images may have different angles of the scene and different capture times. Therefore, the captured images must be aligned and registered to a single coordinate system before generating the multispectral data cube.

[0118] Alignment of the captured images can be analytical, utilizing a multi-channel registration (MCR) algorithm, further detailed herein with particular reference to FIG. 6, that calculates a transformation matrix between pairs of captured images. The MCR algorithm is not limited in its ability to accurately perform parallax correction on the captured images. At least a portion of the alignment process can be performed by system 100 using AI and / or ML-based alignment. System 100 can execute an AI and / or ML-based process for determining an aligned multispectral data cube from one or more two-dimensional (2D) images of a scene. This AI and / or ML-based alignment process can better address parallax correction on the captured images. A process for determining an aligned multispectral data cube from one or more two-dimensional images of a scene can receive a source 2D image of a captured image of a scene and a target 2D image of the captured image, and can determine a corresponding flow map that maps changes to be made to pixels of the source 2D image to align the source 2D image with the target 2D image. Continuing with our non-limiting example above, the captured images can be a series of images captured by a multispectral sensor 104 coupled to a rotating filter wheel as the sensor 104 moves over the scene (in this example, the multispectral sensor 104 is housed on an airborne platform that flies over the scene). Due to the movement of the multispectral sensor 104 over the scene, each captured image has a different viewing angle of the scene. Note that at least one consecutive pair of captured images of the scene overlap by more than an overlap threshold.

[0119] The machine learning can receive source and target 2D images of the captured images and determine a flow map that maps changes to be made to pixels of the source 2D image to align the source 2D image with the target 2D image. This process can be performed for one or more of the images captured using the same target 2D image (e.g., the target 2D image can be an image captured at a midpoint in the capture of a series of images). The resulting flow map can be used by system 100 to generate a registration for all of the captured images by aligning the multiple source 2D images all with the same target 2D image. This can be used to generate a multispectral data cube from the aligned captured images.

[0120] Machine learning models are trained using a training data set that includes multiple training records. Each training record includes a training source 2D image, a training target 2D image, and one or more training flow maps that map changes to be made to pixels in the training source 2D image to align the training source 2D image with the training target 2D image. In some cases, the training data set may be automatically generated based on a three-dimensional (3D) model of a given scene. The 3D model may have one or more properties and may include a collection of pixels. At least one of the pixels may be associated with a color and a label indicating the material group of the element in the scene associated with that pixel. A non-limiting example of such a 3D model is a model used by a game engine (such as Unreal Engine) to render a game scene. In this example, the 3D model may be a 3D model of a field for growing strawberries. The 3D model may have properties such as a geographic area / region, a sunlight angle, a viewing distance, and other properties of the 3D model. A pixel associated with a strawberry in the model may be associated with a color (e.g., red) and a label for the fruit's material group. The training data set is also generated based on a material database and a heuristics table. The material database may include a list of materials. At least one material in the database may be associated with the material's spectral reflectance signature vector, a typical color for the material, and a given material group. For example, one material group may be soil. The soil group may be associated with several different materials registered in the database, such as farmland, sea sand, and desert sand. Each of these specific soils is associated with the material's spectral reflectance signature vector and a typical color for the material. The heuristics table includes one or more rules. Each rule defines the probability of the presence of a given material in a given scene based on the characteristics of the given scene. For example, the probability of desert sand being present in a scene with characteristics of a tropical jungle region is low. The training data set may be generated by simulating a "flyover" over the scene in the 3D model.The capture of a series of 2D images from the 3D model is simulated as if the multispectral sensor 104 were passing over a scene of the 3D model and capturing a series of 2D images. Each simulated captured 2D image may be simulated to have a given wavelength using the 3D model, the material database, and the heuristic table. The changes that need to be made to the pixels of the simulated captured 2D image to align the simulated captured 2D image with the simulated captured target 2D image may be precisely calculated from the 3D model itself, so that a flow map may be generated for at least one of the simulated captured 2D images relative to the selected simulated captured target 2D image based on knowledge of the 3D model. Generating a training data set may include repeating the following steps for a given pixel of at least one of the simulated captured 2D images: querying a material database for a list of materials that may be materials having the material group of the given pixel. According to the rules of the heuristic table and the characteristics of the 3D model, materials whose probability of being present in the given simulated captured 2D image is less than a threshold are removed from the list of possible materials to obtain a revised list of possible materials. A matching material is determined for the given pixel from the revised list of possible materials based on a match between the color of the given pixel and a typical color associated with a material from the revised list of possible materials, and a corresponding portion of a spectral reflectance signature vector associated with the matching material according to a wavelength in the given simulated captured 2D image at the given pixel location is selected. Generating the training data set may also include processing occluded pixels in the simulated captured 2D image by estimating the color of the occluded pixel based on one or more surrounding pixels located in the vicinity of the occluded pixel in the 3D model.

[0121] The machine learning model used to determine the aligned multispectral data cube from one or more two-dimensional images of the scene may be constructed utilizing one or more of the following model architectures: an encoder-decoder model, a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, a visual transformer model, or any other model architecture that may be used in building and training a machine learning model. In some cases, the machine learning model may be a neural network that processes video of continuous multispectral data cubes using one or more of the above-mentioned model architectures combined with one or more of the following architectures: recurrent neural network (RNN), long short-term memory network (LSTM), gated recurrent unit (GRU), visual transformer, bidirectional encoder representation with transformers (BERT), dilated convolutional network, time-series convolutional network (TCN), ordinary differential equation (ODE), etc. Training the machine learning model may include pruning. The machine learning model may also be trained using reinforcement learning methods. The machine learning model may further include a vision transformer. The vision transformer may enable the machine learning model to code one or more segments within a captured 2D image of a scene.

[0122] Two non-limiting examples of steps and stages of a process for determining a registered multispectral data cube from one or more two-dimensional images of a scene are shown in FIGS. 13 and 14. In step 1302 in FIG. 13, one or more frames of a scene are captured using the multispectral sensor 104. The capture may be performed, for example, while a platform to which the multispectral sensor 104 is attached is flying. A frame is captured for each wavelength of the sensor. Each frame has a different angle relative to the scene and targets within the scene due to the movement of the platform over the scene between frame captures. In step 1304, the multiple frames with different wavelengths are aligned to a single coordinate system. The result of aligning the multiple frames to a single coordinate system is one or more uncalibrated multispectral data cubes in step 1306. These uncalibrated multispectral data cubes undergo automatic calibration in step 1308 to obtain aligned, calibrated multispectral data cubes (e.g., by utilizing a process that automatically generates calibrated multispectral data cubes from uncalibrated multispectral data cubes). The aligned, calibrated multispectral data cubes may be used by system 100 to detect targets. Optionally, in FIG. 14 , a machine learning target detection process is used in step 1408 directly on the aligned, uncalibrated multispectral data cubes of step 1406.

[0123] The process of determining an aligned multispectral data cube from one or more two-dimensional images of a scene may be performed by system 100 as part of another process performed by system 100 (e.g., as part of a multi-channel alignment process performed by system 100 to create a multispectral data cube). In other cases, the process of determining an aligned multispectral data cube from one or more two-dimensional images of a scene may be performed by system 100 as a stand-alone process independent of other processes performed by system 100. The results of the process of determining an aligned multispectral data cube from one or more two-dimensional images of a scene may also be used by other systems external to system 100 for various tasks.

[0124] The process of determining an aligned multispectral data cube from one or more two-dimensional images of a scene is described in further detail herein, with particular reference to FIG.

[0125] As described above, the system 100 can automatically generate hyperspectral and / or multispectral training data sets. The multispectral training data set can include one or more training records, each of which includes an uncalibrated multispectral training data cube and a calibrated multispectral training data cube. The calibrated multispectral training data cube can be generated from the uncalibrated multispectral training data cube through a calibration process (e.g., an atmospheric correction calibration process). The multispectral training data set can be automatically generated based on a three-dimensional (3D) model of a given scene. The 3D model can have one or more properties and can include a collection of pixels. At least one of the pixels can be associated with a color and a label indicating the material group of which the element in the scene associated with that pixel is composed. Non-limiting examples of such 3D models include models used by game engines (e.g., Unreal Engine) to render game scenes. In this example, the 3D model can be a 3D model of a field for growing strawberries. The 3D model can have characteristics such as geographic area / region, sunlight angle, viewing distance, and other characteristics of the 3D model. A pixel associated with a strawberry in the model can be associated with a color (e.g., red) and a label for the fruit's material class. The generation of the training data set is also based on a material database and a heuristics table. The material database can include a list of materials. At least one material in the database can be associated with the material's spectral reflectance signature vector, a typical color for the material, and a given material class for the material. For example, one material class can be soil. The soil class can be associated with several different materials registered in the database, such as farmland, ocean sand, and desert sand. Each of these specific soils is associated with the material's spectral reflectance signature vector and a typical color for the material. The heuristics table includes one or more rules. Each rule specifies the probability of the presence of a given material in a given scene based on the characteristics of the given scene.For example, the probability of the presence of desert sand in a scene having the characteristics of a tropical jungle region is low.

[0126] The training data set may be generated by simulating the capture of 2D images from a 3D model of the scene. The simulated captured 2D image includes a subset of the pixels of the 3D model. For at least one given pixel of the subset of pixels, a training calibrated multispectral data cube of the training record is generated by: querying a materials database for a list of materials that may be materials having the material group of the given pixel; removing materials from the list of possible materials that have a probability of being present in the simulated captured 2D image below a threshold according to the rules of the heuristic table and the characteristics of the 3D model to obtain a revised list of possible materials; determining a matching material for the given pixel from the revised list of possible materials based on a match between the color of the given pixel and a representative color associated with a material from the revised list of possible materials; and selecting at least a portion of the spectral reflectance signature vector associated with the matching material to be part of the training calibrated multispectral data cube at the given pixel location. Then, generating uncalibrated training multispectral data cubes for the training records using the calibrated training multispectral data cube and an atmospheric simulator, where the atmospheric simulator can receive the calibrated training multispectral data cube and one or more atmospheric conditions and generate one or more uncalibrated training multispectral data cubes corresponding to the calibrated training multispectral data cube under those atmospheric conditions. The atmospheric conditions can include humidity level, sunlight angle, aerosol model, visibility score, geographic area, or any other condition associated with the atmosphere or its impact on the captured imagery. In some instances, multiple uncalibrated training multispectral data cubes can be generated using different atmospheric conditions from a single calibrated training multispectral data cube.In some cases, generating the uncalibrated training multispectral data cube further includes simulating various targets in the training data set by adding target pixels to the 3D model prior to capturing the 2D images. Generating the uncalibrated training multispectral data cube also includes adding texture to the generated calibrated training multispectral data cube based on the texture associated with the corresponding simulated captured 2D images. This may be accomplished by calculating average color values ​​for a given segment of pixels in the simulated captured 2D images and using these calculated average color values ​​for the pixels in the generated calibrated training multispectral data cube. In some cases, generating the uncalibrated training multispectral data cube further includes adding simulated registration errors. For example, simulating typical registration errors by randomly varying at least one channel relative to the other channels of the generated multispectral data cube. Generating the uncalibrated training multispectral data cube may also include blurring at least one of the corresponding simulated captured 2D images prior to generation. The blurring may be performed, for example, by using a point spread function (PSF) filter that mimics the physical blurring of the multispectral sensor 104, which may be based on the actual performance of the multispectral sensor 104. In some instances, generating the training uncalibrated multispectral data cube further includes adding shot noise to at least one of the corresponding simulated captured 2D images prior to generation.

[0127] A non-limiting example of steps and stages of a process for automatically generating hyperspectral and / or multispectral training data sets is shown in FIG. 15. In step 1502, a material database storing reflectance spectra, labels (material names), and representative colors of at least one type of material is used in conjunction with one or more heuristic tables from step 1504 and a 3D virtual world photorealistic engine (e.g., a game engine such as Unreal Engine) including the virtual world RGB frame, a distance map within the virtual world, a color map of scenes and objects within the virtual world, and label (object) masks (from step 1506) to populate the virtual world with materials. The heuristic table holds one or more rules that define the likelihood that a material type is present in a particular object and / or a particular terrain. In step 1510, a calibrated multispectral data cube is captured from the virtual world. In step 1512, an atmospheric simulator is utilized to generate one or more uncalibrated multispectral data cubes (generated in step 1516) that correspond to the calibrated multispectral data cube under atmospheric conditions. In step 1514, a physical simulator may additionally and / or optionally be utilized on the calibrated multispectral data cube to add optical blur effects, add one or more alignment errors, and / or add shot noise when generating the one or more uncalibrated multispectral data cubes (generated in step 1516). The atmospheric simulator may receive the calibrated multispectral data cube and one or more atmospheric conditions (such as humidity level, sunlight angle, aerosol model, visibility score, or geographic area) and may generate one or more uncalibrated multispectral data cubes that correspond to the calibrated multispectral data cube under those atmospheric conditions.System 100 may utilize an atmospheric simulator to generate two or more uncalibrated multispectral data cubes with known target materials using different atmospheric conditions from a single, given calibrated multispectral data cube, thereby facilitating the creation of large training data sets from a small number of calibrated multispectral data cubes labeled with known target materials and their locations within the scene. A non-limiting example of such an atmospheric simulator is the Moderate Resolution Atmospheric Radiative Transfer (MODTRAN) computer code. Additionally, as further detailed herein, at least a portion of the training data set may be generated automatically by utilizing an automated process for generating multispectral labeled training data sets.

[0128] It should be noted that the generated training data sets may be used by system 100 itself, for example, to train a machine model for the automated generation of calibrated multispectral data cubes from uncalibrated multispectral data cubes. The generated training data sets may also be used by other systems external to system 100 for various tasks, such as, for example, to train machine learning models that require hyperspectral and / or multispectral training data sets as part of the training process. The process of generating hyperspectral and / or multispectral training data sets is described in further detail below.

[0129] The process of generating hyperspectral and / or multispectral training data sets is described in further detail herein, with particular reference to FIG. 8.

[0130] Optionally, the multispectral system 100 may include additional sensors that can be used to further investigate a given scene, particularly potential targets identified by utilizing the spectral data cube generated by the multispectral sensor 104. These optional additional sensors may include a wide-range daylight sensor 106 capable of capturing a given scene at a wide viewing angle of visible wavelengths; a narrow-range daylight sensor 110 capable of capturing a given scene at a narrow viewing angle of visible wavelengths; a wide-range sensor may be a wide-range thermal sensor 114 capable of capturing a given scene at a wide viewing angle of infrared wavelengths or a wide-range ultraviolet sensor capable of capturing a given scene at a wide viewing angle of ultraviolet wavelengths; a narrow-range sensor may be a narrow-range thermal sensor 118 capable of capturing a given scene at a narrow viewing angle of infrared wavelengths or a narrow-range ultraviolet sensor capable of capturing a given scene at a narrow viewing angle of ultraviolet wavelengths; and a narrow-range SWIR sensor 108 capable of capturing a given scene at a narrow viewing angle of SWIR wavelengths.

[0131] Optionally, multispectral system 100 may also include a laser range finder 116. Laser range finder 116 may be used by system 100 to determine the distance to an object, such as one of the potential targets identified using the spectral data cube generated by multispectral sensor 104 and further investigated using one of the additional sensors described above. Optionally, multispectral system 100 may include a laser pointer 112. Laser pointer 112 may be used to mark potential targets at night.

[0132] Optionally, the multispectral system 100 may include a network interface 122. The network interface 122 (e.g., a network card, a Wi-Fi client, a Li-Fi client, a 3G / 4G client, or any other component) enables the system 100 to communicate over a network with external systems, such as a ground station system capable of remotely managing the system 100. The network interface 122 may handle inbound and outbound communications with such systems. For example, the system 100 may receive, through the network interface 122, multiple spectral target signatures, an exposure time transformation matrix, exposure times, reflectance values ​​for one or more known object types, an atmospheric calibration matrix, an image of the scene, a spectral data cube, identified potential targets and information about those targets, distances to the targets, etc.

[0133] The multispectral system 100 may further comprise or alternatively be associated with a data repository 120 configured to store data (e.g., a database, a storage system, a memory including read-only memory (ROM), random access memory (RAM), or any other type of memory, etc.). Some examples of data that may be stored in the data repository 120 include multiple spectral target signatures, an exposure-time transformation matrix, exposure times, reflectance values ​​for one or more known object types, an atmospheric calibration matrix, an image of the scene, a spectral data cube, identified potential targets and information about those targets, distances to the targets, etc. The data repository 120 may be further configured to enable retrieval and / or updating and / or deletion of the stored data. Note that in some cases, the data repository 120 may be distributed, in which case the system 100 has access to the stored information via, for example, a wired or wireless network to which the system 100 can connect (utilizing its network interface 122).

[0134] The multispectral system 100 further comprises a processing circuit 102. The processing circuit 102 may be one or more processing units (e.g., central processing units), microprocessors, microcontrollers (e.g., microcontroller units (MCUs)), (graphics processing units) GPUs, or any other computing devices or modules, including multiple and / or parallel and / or distributed processing units, adapted to independently or cooperatively process data for controlling and enabling operations on associated system 100 resources.

[0135] The processing circuit 102 includes modules such as a target identification and investigation module 124, an exposure time determination module 126, an atmospheric calibration module 128, a multi-channel registration module 130, a calibrated multispectral data cube generation module 132, a multispectral-labeled training data set generation module 134, a target material detection module in an uncalibrated multispectral data cube 136, an automatic identification of calibration objects module 138, and an aligned multispectral data cube determination module 140. The target identification and investigation module 124 is configured to perform a target identification and investigation process, as described in further detail herein, with particular reference to FIG. 3. The exposure time determination module 126 is configured to perform an exposure time determination process, as described in further detail herein, with particular reference to FIG. 4. The atmospheric calibration module 128 is configured to perform an automatic atmospheric calibration process, as described in further detail herein, with particular reference to FIG. 5. The multi-channel registration module 130 is configured to perform a multi-channel registration process, as described in further detail herein, with particular reference to FIG. 6. The calibrated multispectral data cube generation module 132 is configured to perform a process for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, as described in further detail herein, with particular reference to Figure 7. The multispectral labeled training data set generation module 134 is configured to perform a process for automatically generating a multispectral labeled training data set, as described in further detail herein, with particular reference to Figure 8. The target material detection in uncalibrated multispectral data cube module 136 is configured to perform a process for detecting one or more target materials in an uncalibrated multispectral data cube, as described in further detail herein, with particular reference to Figure 9.The automatic identification of calibration objects module 138 is configured to implement a process for empirical atmospheric calibration utilizing automatically identified objects in images of a scene, as described in further detail herein, with particular reference to Figure 10. The registered multispectral data cube determination module 140 is configured to implement a process for determining a registered multispectral data cube from one or more two-dimensional images of a scene, as described in further detail herein, with particular reference to Figure 11.

[0136] The real-time multispectral system 100 can be used to observe a given area. The system 100 scans the given area using an “area scan” or “step-and-steer” mode, which involves scanning the scene segment by segment (and merging / stitching the segments’ spatial data cubes into one overall data map cube representing the entire given area), allowing for controlled observation of the entire given area or zooming out to observe the entire given area within its FOV. In this mode, the system 100 can hold an observation grip over the given area. The “step-and-steer” mode can be implemented by the system 100 utilizing the multispectral sensor 104 and / or additional sensors to cover different segments of the given area (each segment fits within the size of the associated sensor’s FOV), with overlapping edges forming a mosaic (rows and columns) of segments that are stitched into one spectral data map cube. An example might be for system 100 to scan a given area with 5% overlap in both the X and Y axes. Once the section-by-section spectral data cube alignment process is complete, system 100 stitches the segments into an entire spectral data cube for the entire given area by aligning the overlapping sections. In another possible mode, system 100 can automatically initiate observation cycles and reduce the number of false alarms (false positives and false negatives) by using artificial intelligence (AI) and / or machine learning (ML) models for automatic determination of thresholds for potential target identification and / or AI and / or ML models for automatic spectral cube calibration.

[0137] It should be noted that the multispectral system 100 may operate as a standalone system without the need for the network interface 122 and / or the data repository 120. The addition of one or both of these elements to the system 100 is optional and not required, as the system 100 may operate in accordance with its intended use anyway. In some cases, some or all of the elements of the multispectral system 100 may be distributed. For example, additional sensors may be located at locations remote from the location of the processing circuit 102 and / or the location of the multispectral sensor 104. In these cases, the system 100 may utilize the network interface 122 to communicate and distribute information between the remotely located elements of the system 100. In some cases, the multispectral sensor 104 and one or more additional sensors may be observing the same given area in real time, and the system 100 may utilize these sensors by investigating potential targets within the FOV in real time or by observing images and / or video captured by the sensors at a later time. Optionally, this can also be done when an additional sensor is capable of observing a given area and scanning the FOV for potential targets, but is located at a different location than the location of the multispectral sensor 104 itself.

[0138] System 100 is a real-time solution that conducts surveys of additional potential targets by utilizing additional sensors that are pointed at and observe the same potential targets in real time. System 100's real-time solution significantly improves results by changing the parameters of the multispectral target detection system online. System 100's real-time solution provides a unique and robust solution for online target tracking. System 100's real-time solution is robust with respect to weather, sun direction, and atmospheric radiative transfer. It has automatic calibration corrections. In summary, System 100 is an online multispectral imaging data system that is significantly different from other offline solutions.

[0139] Attention is now directed to FIG. 2A, which is a schematic diagram of one possible exemplary configuration of a real-time multispectral system 100 in accordance with the subject matter of this disclosure.

[0140] In accordance with the subject matter of the present disclosure, the real-time multispectral system 100 can optionally be housed within an observation pod. The observation pod can be used as a standalone system and / or installed as a special-purpose payload on a platform, such as a personal platform, a guard tower as part of a wide-area border defense system, a ground vehicle, an airborne platform, or space. When the system 100 is installed in an observation pod on an airborne platform, the system 100 can detect potential targets within a given scene in real time without the need to download information from the airborne platform to a ground station for analysis by a human operator. The system 100 can maintain observation capability over a given observation area while the platform is in motion by controlling the movement of the lenses of the multispectral sensor 104 and additional sensors (e.g., the wide-area daylight sensor 106, the narrow-area daylight sensor 110, the wide-area thermal sensor 114, the narrow-area thermal sensor 118, the narrow-area SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.). In many cases, the various sensors on a pod or payload may have different FOVs, but the central FOVs of each sensor are aligned in the same direction, allowing the system to use different sensors for the same target location. The optical design of these lenses is optimized to minimize color distortion and eliminate spectral mixing across the FOV. In many cases, the system 100 is mounted on a gimbal support that allows it to compensate for platform movement while maintaining the sensor over a given observation area. In these cases, the system 100 utilizes a multispectral sensor 104 to capture a given scene in multiple wavelength ranges while the airborne platform flies over the given scene. The multispectral system 100 can generate a spectral data cube from the multispectral data captured by the multispectral sensor 104.The system 100 can analyze the spectral data cube to identify pixels in two spatial dimensions that correspond to the spectral signature of a given material type the system is attempting to detect in a given scene and mark those pixels as targets or potential targets. A spectral signature is the variation in a material's reflectance over a given wavelength range. Searching for target spectral signatures allows the system 100 to identify the object it is searching for in the scene. The system 100 can utilize the multispectral sensor 104 and additional sensors to detect and survey potential targets in real time. For example, the system 100 can be carried over a given observation area as part of a flight of an airborne platform. The system 100 can generate a spectral data cube for a given observation area using the multispectral sensor 104 and then use the additional sensors to survey and analyze suspected potential targets and identify potential targets in the spectral data cube while flying over the area, without having to download the captured material for analysis at a ground station. This is accomplished by system 100 performing real-time analysis of the spectral data cube along with input data from additional sensors to more accurately detect and identify potential targets. To support real-time target detection, system 100 utilizes one or more of the following processes as part of target identification, including an exposure time determination process further detailed herein with particular reference to Figure 4, an atmospheric calibration process further detailed herein with particular reference to Figure 5, and a multi-channel alignment process further detailed herein with particular reference to Figure 6.

[0141] A non-limiting example is a real-time multispectral system 100 used in an agricultural scenario. In such a scenario, the target spectral signature can be the spectral signature of a given agricultural product that the system 100 attempts to identify and further investigate. For example, the spectral signature of a strawberry. The strawberry's spectral signature is a vector associated with multiple values. Each value in this vector is associated with the spectral variation of the strawberry's reflectance or emissivity in a wavelength range that can be captured by the multispectral sensor 104. Another example is one in which the system 100 attempts to identify special leaves (e.g., artificial materials) that visually resemble all other leaves but differ at some invisible wavelengths; these special leaves have a unique reflectance that differs from normal leaves. In some cases, the system 100 is not provided with a specific target spectral signature. In these cases, the system 100 can determine whether one or more anomalies are present in the spectral data cube of a given scene. Such an anomaly is detected by the system 100 when one or more pixels in the spectral data cube have a spectral signature that is more than a threshold distance away from other pixels in the data cube. A non-limiting example is a system 100 used in the search and rescue (S&R) field. The system 100 may observe an area of ​​the ocean where several survivors are floating on the waves. The system 100 utilizes the multispectral sensor 104 to generate a spectral data cube of the observed area and uses it to detect pixels in the scene representing these survivors without acquiring their target spectral signatures—i.e., by simply detecting the survivors as anomalies against the background of the waves—and relay data regarding the survivors' locations to the S&R team. An anomaly would be detected even if the size of the survivors within the vast area of ​​the ocean were significantly smaller than the size of a single pixel.Another non-limiting example involves using the system 100 to detect potential moving targets. In this example, the system 100 is used as an observation solution in the field of homeland security. The system 100 may be deployed to observe a national border area or an area of ​​a protected facility. The target spectral signature is the material of clothing worn by an illegal intruder attempting to cross the protected border. The multispectral sensor 104 is utilized to generate a spectral data cube for the observed area of ​​the border and use it to detect pixels in the scene that represent the material of the clothing. If such a target is observed and then lost because it moves out of the scene, the system 100 can quickly reacquire the target by directing the optical line of sight of the multispectral sensor 104 to the area to which the target has moved and generating a new spectral data cube for the new area, thereby identifying the material of the target's clothing in the new area and allowing border patrol agents to maintain track of the target.

[0142] In some cases, newly identified potential targets within a scene represented by a spectral data cube may be added as new target spectral signatures for system 100 to store in data repository 120. System 100 may use these new target spectral signatures to attempt to detect corresponding potential targets in future spectral data cubes that it captures. Identification of such new potential targets may be performed automatically by system 100 or by a human analyst who is a user of system 100.

[0143] It should be noted that the system 100 can use the multispectral sensor 104 and the generated spectral data cube to identify multiple potential targets in a scene by analyzing one or more groups of pixels in the spectral data cube that have target spectral signatures. Additionally, the system 100 can search for potential targets (e.g., strawberries and blueberries) that correspond to two or more target spectral signatures in the same scene.

[0144] FIG. 2A illustrates non-limiting exemplary hardware that may be used to implement elements of system 100. Multispectral sensor 104 may be implemented by a wide-angle, high-definition (HD) SWIR camera with a 5-25 degree FOV and zoom capability. Wide-angle daylight sensor 106 may be implemented by a visible-NIR (VNIR) camera with a 5-25 degree FOV and zoom capability. Wide-angle thermal sensor 118 may be implemented by a mid-wave infrared (MWIR) camera with a 5-25 degree FOV and zoom capability. Narrow-angle daylight sensor 110, narrow-angle thermal sensor 118, and narrow-angle SWIR sensor 108 may all be implemented using spotter 202, which is an integrated telescope that provides imaging paths for the three narrow-angle sensors (narrow-angle daylight sensor 110, narrow-angle thermal sensor 118, and narrow-angle SWIR sensor 108). These narrow-angle sensors, also known as spotters, can be used by a user of system 100 to further investigate one or more detected potential targets by analyzing the spectral data cube captured by multispectral sensor 104. Multispectral sensor 104 can include a wide-angle SWIR simple mode, in which no filter is applied to the sensor, allowing system 100 to use multispectral sensor 104 as a wide-angle SWIR sensor. The use of imaging sensors (e.g., wide-angle daylight sensor 106, narrow-angle daylight sensor 110, wide-angle thermal sensor 114, narrow-angle thermal sensor 118, multispectral sensor 104, narrow-angle SWIR sensor 108, etc.) by system 100 with all sensors on the same optical line of sight can be configured such that when the zoom range of one wavelength series of imaging sensor (e.g., the zoom limit of wide-angle thermal sensor 114) is reached, system 100 automatically switches to the narrow-angle imaging sensor of the same series (continuing with this example, switching to narrow-angle thermal sensor 118).

[0145] 2A may optionally include a laser range finder 116, e.g., a narrow-beam laser designator and range finder having a laser beam wavelength of 1 to 1.6 micrometers. Optionally, a laser pointer 112 may also be included in system 100. A night vision goggle (NVG) laser pointer having a laser beam with a wavelength of 800 to 900 nanometers is one example of such a laser pointer 112 that may be part of system 100.

[0146] An environment may include two or more real-time multispectral systems 100. These systems 100 may utilize their respective network interfaces 122 to pass information between them. The information may include targets, target spectral signatures, exposure times, and atmospheric correction parameters. Sharing information may enable several systems 100 to collaborate in observing a given scene and identify potential targets by jointly investigating them.

[0147] Having described an exemplary configuration of the real-time multispectral system 100, attention is now directed to FIG. 2B, which is a schematic diagram of one possible exemplary design of a processing chain for a real-time multispectral system in accordance with the subject matter of the present disclosure.

[0148] System 100 utilizes a processing chain to process input from the detectors of multispectral sensor 104 and output the resulting images and / or video to a video output. Note that one or more of the steps of the processing chain described herein may be optional. Some of the steps may be performed in a different order than that shown in FIG. 2B.

[0149] To support operation with more than one filter, for example, when multispectral sensor 104 uses a filter wheel with one or more filters, system 100 utilizes a vector multiplexer with a different value register for each filter in the filter wheel. System 100 can use a general register block that supports insertion of a value associated with a corresponding filter, such that the value read from the vector multiplexer according to a selector becomes the value associated with the corresponding filter.

[0150] The processing chain can begin with the detector camera link 204, which is an interface to the detectors of the multispectral sensor 104. The output from the multispectral sensor 104 is used as input to the detector interface 206 through the detector camera link 204. The detector interface 206 can support two modes: (a) a normal simple mode (e.g., wideband SWIR simple mode) at, for example, 25 Hz; and (b) a cube sampling mode (e.g., sampling the cube every 1.5 seconds according to the rotational position of the filter wheel, which determines which filter with a given wavelength range is placed in the sensor's imaging path to capture an image in that wavelength range). In the simple mode, frames are sampled constantly. In the cube sampling mode, the detector interface samples input frames only when the filter wheel is ready. Information about the filter position (which filter on the filter wheel is currently in front of the detector) can be derived from the mechanism card of the multispectral sensor 104. Optionally, information regarding filter position may be obtained from software and / or firmware.

[0151] The radiometric calibration 208 step includes radiometric correction, non-uniformity correction (NUC), and bad pixel replacement (BPR). The radiometric calibration 208 step supports image integration (the summation of values ​​per pixel for a predefined number of frames) and allows radiometric calculations to be performed by the processing circuitry 102 (e.g., by executing a given software code). The image integration is determined based on the radiometric correction. This can be done by multiplying the raw spectral data cube by a corresponding radiometric correction matrix. The integration time can depend on the required filters of the filter wheel. In some cases, the system 100 can obtain multiple radiometric correction matrices. These predefined radiometric correction matrices correspond to a given exposure time. The system 100 can multiply each channel by a corresponding predefined radiometric correction matrix according to the exposure time used for that channel. These predefined radiometric correction matrices can be implemented by the system 100 as look-up tables. The system 100 can toggle in real time between multiple different calibration matrices (for different filters and different integration times). BPR can be performed using neighboring pixels from a given pixel window around the bad pixel. BPR can be performed after gain and offset correction. To support NUC operation with more than one filter, the system 100 utilizes the vector multiplexer described above to select the relevant values ​​in the gain and offset tables for each input image. The output statistics for NUC (mean and mean^2) are collected by sampling the results across the entire spectral data cube. The system 100 includes counters for the number of saturated pixels in each filter. This information is used to verify that the integration time does not exceed a time threshold. Optionally, the radiometric calibration 208 step can include white reference correction (WRC). WRC fixes the image of the spectral data cube to a reflectance image. This is performed based on gain and offset correction.

[0152] The alignment 210 step involves supporting alignment between all images sampled from one or more filters of the multispectral sensor 104 used to generate the spectral data cube. To perform alignment between the filter images that construct the spectral data cube, the system 100 utilizes a multichannel registration (MCR) algorithm. The MCR algorithm calculates a transformation matrix between pairs of images, even when the gray levels of the pair of images are not the same value or polarity, for example, when the pair of images have different dynamic ranges and contrast issues. The alignment operation performs a transformation matrix on one of the images to align each pair of images with the same scene, pixel by pixel. The system 100 can utilize a video formatter module to perform the transformation on the images. The alignment process can include the following steps: (a) creating descriptors for “interest points” on each image; (b) matching between the descriptors from the two images; (c) finding the transformation matrix between the two images (according to the location of the “interest points” in each image); and (d) performing an inverse transformation matrix on one of the images. This step can be implemented in software and / or firmware. Step (a) can use various detection modules, such as Harris Corner Detection, which finds interest points in each of the images by utilizing filters and Harris functions. If the overall result of the function for a particular pixel is greater than a threshold, that pixel is selected as an interest point. Another example is the use of an MCR Descriptor module, which builds a descriptor for each interest point that describes the pixel by all pixels in an NxN window around it. The descriptor contains information for each pixel around the interest point about whether the pixel is an edge and what the angle of the edge is.

[0153] The computation 212 step involves algorithms that are performed only after the spectral data cube is ready for analysis. When alignment is required, the computation 212 step is performed after the alignment 210 step is performed. The processing circuit 102 can read the relevant parameters of each algorithm from a first designated memory area of ​​the system 100, read the spectral data cube, perform the algorithm on the image of the spectral data cube, and store the results in a second designated memory area of ​​the system 100. Because pixels at the edge of the spectral data cube / image can harm the detection capabilities of various algorithms (e.g., because the values ​​of pixels at the edge are inaccurate due to optical effects on the image edges or because MCR causes the edges to not be part of the image for some bands), the system 100 defines an area of ​​interest (AOI) that cuts off the edges of the spectral data cube, and the algorithms only calculate pixels within this AOI window. The AOI does not affect the size of the spectral data cube and the corresponding detection layer, which is the same as the resolution of the detectors of the multispectral sensor 104. The AOI only affects the algorithm calculations (e.g., mean, covariance, pure algorithms, histograms, and detection decisions). Optionally, two or more algorithms can be performed on the same SLI (an SLI is a vector of 12 values ​​representing a spectral signature, e.g., the spectral signature of a known substance or known target), and the results can be compiled. The detection layer can be a heat map that tells the probability that each pixel is a target.

[0154] The detection algorithm may include one or more of a ratio algorithm, a Spectral Angle Mapper (SAM) algorithm, a Zero Mean Differential Area (ZDMA) algorithm, an Anomaly Detection (AD) algorithm, and a Match Filter (MF) algorithm.

[0155] The ratio algorithm is based on calculating the ratio between the values ​​of the same pixel in different filters (channels). The algorithm supports zero as input for each of the channels, and the same channel as input for two or more channels.

[0156] The SAM algorithm finds the "angle" between the target spectral signature (which is the spectral vector of the signature) and each of the pixel's spectral vectors. The spectral vector is a collection of values ​​for each filter that form a vector of values. The "angle" (also used as "distance") is the vector of value differences.

[0157] The ZDMA algorithm is similar to SAM, but finds the ZDMA between a target spectral signature "Fn" and a pixel spectral vector "Ln".

[0158] The AD algorithm uses the Reed-Xiaoli (RX) detector algorithm to detect spectral differences between the test pixel cube and the entire dataset. The anomaly detection calculation is based on the following steps: (a) calculating the covariance matrix, (b) calculating the inverse covariance matrix, and (c) calculating the anomaly detection equation.

[0159] The "detection layer index" can specify the detection layer index of a specific algorithm across all detection layers used on the spectral data cube. The index number allows the system 100 to set the color value of each algorithm for the output RGB image. For example, if the system 100 performs two SAM SLI algorithms and one ratio algorithm on the spectral data cube, a number (e.g., 0, 1, 2) is assigned to each algorithm, and then color values ​​for each of these algorithms are specified in the RGB module to assign different values ​​for each query. The entire spectral data cube is stored in memory, which the system 100 can use to extract the SLI of objects present in the scene. The SLI extraction process involves calculating the average gray level value within a specific area containing pixels completely covered by the object. The average value is calculated for each spectral band.

[0160] The detection layer 214 step involves creating a detection layer in the spectral data cube by adding one or more detection layers to the image. The detection layer calculation is based on a histogram of the results of each algorithm from the computation 212 step. The detection layer is generated by the system 100 after the algorithm results from the computation 212 step are prepared. The algorithm results indicate whether each pixel is a target or not. For many reasons, "holes" can form in a "large" target, where some pixels are determined to be targets and some pixels are not. This occurs because the detection algorithms used by the system 100 have specific threshold settings that limit the pixels that are counted as part of a target. When a threshold is set, some pixels in the target area may not exceed the threshold and therefore not be considered "part of the target." This creates "holes" within the actual area of ​​the target. The system 100 utilizes a blob merge algorithm to correct these holes. To correct these holes within the target, the system 100 can fuse marked pixels into a single target by applying a target pixel calculation. That is, whether these pixels exceed a threshold within a window around the target pixel. If so, all pixels within the window exceeding the threshold may likewise be marked as part of the target. Note that the applied calculation may be a local average calculation of the target pixels, taking into account the local average result of the algorithm indicating whether these pixels are targets or not. In other cases, the applied calculation may also include spatial calculations based on the contour of the target pixel. These spatial calculations may add neighboring pixels to the target even if they have not been identified as targets by the algorithm, provided that they comply with one or more metrics based on the spatial and / or spectral characteristics of these pixels. A non-limiting example may include pixels that share the same contour as a nearby target pixel.

[0161] The Red, Green, Blue (RGB) 216 step involves generating a pseudo-RGB image. The system 100 selects three images (wavelengths) from the spectral data cube, sets one of them as the red (RED) value of the RGB image, sets a second as the green (GREEN) value of the RGB image, and sets a third as the blue (BLUE) value of the RGB image. Typically, green wavelengths are selected as wavelengths with a high average value, and red and blue are selected on either side of the spectrum with significantly lower average values ​​relative to the green wavelengths. The selection of the three images from the spectral data cube can be done before the alignment 210 step or after the alignment 210 step.

[0162] The Zoom 218 step is based on a Video Formatter (VF) module. The VF module can operate in two modes: Gray mode or Y / C image mode. To generate a zoomed version of the "pseudo RGB" image generated by the RGB 216 step, the system 100 uses the VF module in Y / C image mode.

[0163] The video output 220 step outputs video from the system 100 onto a screen for viewing by a human user of the system 100. The output may be, for example, a High-Definition Serial Digital Interface (HD-SDI) at a rate of 25 Hz. In simple mode, the detector is synchronized to the HD-SDI output to prevent some output frames from being new and some from being old.

[0164] Referring to FIG. 3, a flow chart illustrating an example sequence of operations performed by a real-time multispectral system for target detection and further investigation in accordance with the subject matter of the present disclosure is shown.

[0165] Thus, the real-time multispectral system 100 may be configured to implement the target identification and investigation process 300, for example, using the target identification and investigation module 124. The multispectral system 100 detects potential targets in real time and has additional target investigation capabilities. The system 100 can utilize the multispectral sensor 104 and additional sensors to detect and investigate potential targets in real time within a given observation area. The system 100 uses the multispectral sensor 104 to generate a spectral data cube for a given observation area within a time window and detects potential targets within the spectral data cube by identifying one or more pixels having a spectral signature corresponding to one of one or more target spectral signatures. The system 100 can simultaneously image the scene during the time window using additional sensors (e.g., wide-area daylight sensor 106, zoom daylight sensor 110, wide-area thermal sensor 114, narrow-area thermal sensor 118, narrow-area SWIR sensor 108, laser rangefinder 116, etc.). The system 100 can grade identified target areas in the spectral data cube by utilizing artificial intelligence (AI) and / or machine learning (ML) models or by having the system user manually grade potential targets. Grading targets is the process by which the system 100 ultimately scores suspected targets on the level of confidence in the elements represented by the pixels in the scene identified as potential targets. The system 100 can grade targets using several methods, such as using logical decision trees, using a base ranking of the selection algorithm used, using the geometric dimensions of typical targets of this type and distance, and / or a combination of the above methods. Non-limiting examples include the system 100 generating a novel index that combines spectral characteristics along with the spatial and / or statistical behavior of the target and its surrounding environment.Target rankings may be related to the distance of each potential target from a cursor navigated by a user of system 100, the conspicuity of the target's identification within the spectral data cube, the size and behavior of the target, and / or atmospheric corrections for the target's pixels. System 100 enables further investigation of potential targets by utilizing additional sensors, for example, by capturing wide-angle images of the potential target in a given time window or by zooming in on the image of the potential target. Note that system 100 can detect multiple potential targets of one or more target types, such as multiple potential targets of the same type or multiple types of targets. This capability may be enabled by utilizing parallel logic designs of detection algorithms. These algorithms combine the spectral and spatial characteristics of potential targets to distinguish them from the background and reduce the number of false alarms. A geolocation process is performed for each detected target (above an automatically set threshold) to transfer the potential target's geographic coordinates to additional sensors in system 100 for further investigation.

[0166] To this end, multispectral system 100 acquires one or more target spectral signatures (block 302). Continuing with the non-limiting example above, system 100 acquires target spectral signatures that represent the reflectance of strawberries with respect to wavelengths that may be captured by multispectral sensor 104.

[0167] Once the target spectral signature is acquired, the system 100 activates the multispectral sensor 104, which can capture images in multiple imaging channels, each having a different wavelength range, and the multispectral sensor 104 is started to operate in the wide-area SWIR simple mode imaging channel (block 304). Optionally, the system 100 can be started with the wide-area daylight sensor 106 with medium zoom turned on, and / or the wide-area thermal sensor 114 with medium zoom turned on, and / or the multispectral sensor 114 with medium zoom turned on for one of the filters.

[0168] Optionally, the system 100 can switch to using the multispectral sensor 104 in a wide-area SWIR simple mode. The system 100 can determine a calculated exposure time for each of the multiple imaging channels of the multispectral sensor 104 based on the wide-area SWIR simple mode exposure time when observing the first FOV (block 306). The exposure time determined for the wide-area SWIR simple mode can be converted by the system 100 to the exposure time required for at least one other imaging channel of the multiple imaging channels by performing an exposure time determination process as described in further detail herein, particularly with reference to FIG. 4. The system 100 sets an optimal integration time (IT) for the “open” (broad-area) SWIR detector in simple mode at full frame rate. The IT of the multispectral channel is set by well-known methods of image auto-exposure, the selected integration time, and empirical, pre-measured factors. This ensures that the main channel of the multispectral sensor 104 is not saturated. This allows the system 100 to utilize the full capabilities of the detectors in the multispectral sensor 104. However, because some other channels may be underexposed or dark, a separate lookup table is used to set a specific IT for each channel to ensure the use of the dynamic range and optimized signal-to-noise ratio (SNR) for each channel. Note that the system 100 can determine the exposure times of the imaging channels based on a single generated spectral data cube for the first FOV. Recapturing the scene with the multispectral sensor 104 is not required to determine the exposure times of the imaging channels.The exposure time determination may be performed repeatedly or periodically by system 100, for example, system 100 may perform the exposure time determination process each time system 100 changes the FOV, and / or after a threshold time has elapsed since system 100 last set the exposure time, and / or before a spectral data cube is generated by multispectral sensor 104. Another option is for system 100 to determine that a new exposure time should be calculated by analyzing a captured image.

[0169] Following determination of the calculated exposure times for each of the imaging channels, system 100 may optionally determine an atmospheric calibration matrix for multispectral sensor 104 based on input from a user (block 308). As an example, a user of system 100 may utilize controls to move a cursor over an image captured by system 100 to indicate that a pixel corresponding to the cursor's location is for an object with known reflectance (e.g., indicating that the cursor is over a road, tree, gravel, etc.). The atmospheric calibration matrix for multispectral sensor 104 may be determined by performing an automatic atmospheric calibration process, as further detailed herein, with particular reference to FIG. 5.

[0170] After determining the atmospheric calibration matrix, the system 100 can be further configured to generate a multispectral data cube for a second FOV observed by the multispectral sensor 104 using the multispectral sensor 104, the calculated exposure time, and the atmospheric correction matrix, where generating the multispectral data cube includes radiometric calibration and multichannel alignment (block 310). Note that the second FOV is an area of ​​the scene in which the system 100 is attempting to identify targets. In some cases, the first FOV and the second FOV are the same. In some cases, the first FOV and the second FOV at least partially overlap, and in some cases, the first FOV and the second FOV are different, i.e., each observes a different scene. Multichannel alignment can be performed by performing a multichannel alignment process, as further detailed herein, particularly with reference to FIG. 6. The multi-channel alignment process allows the system 100, and in particular the multi-spectral sensor 104, to overcome the difficulties of capturing and creating spectral data cubes while moving. The multi-channel alignment process also allows the system 100 to determine the geographic location of identified targets using the generated multi-spectral data cubes.

[0171] Once the system 100 generates the multispectral data, the system 100 can be further configured to utilize the multispectral data cube to identify one or more targets, each target being a group of pixels identified in the multispectral data cube that have a spectral signature corresponding to at least one of the acquired target spectral signatures, and each target having a geographic location (block 312). The groups of pixels can be grouped into clusters, and a pixel cluster can be identified as a potential target. This can be done, for example, by identifying the object's centroid and marking suspected areas of potential targets by circling them (this is typically sufficient for a user of the system 100).

[0172] Targets can be identified using one or more predefined queries that are automatically executed by the system 100 against the spectral data cube. The queries can be dynamically defined by the operator of the system 100 as the analysis of the spectral data cube is performed. The system 100 can create a heat map of the pixels in the spectral data cube by coloring each pixel to indicate the distance of the pixel's spectral vector from the target spectral signature. The target threshold can be automatically changed by the system 100. Automatic thresholding can be implemented by the system 100 utilizing detection algorithms that each have a specific range of values. Some algorithms, such as SAM and ZMDA, are limited to a specific range of values. Other algorithms can have a very wide range of values. The system 100 generates a histogram of the actual values ​​of all pixels in the spectral data cube. Some algorithms use the minimum or maximum histogram value based on the fact that the target typically covers only a small portion of the scene image, and therefore the maximum or minimum histogram values ​​represent the scene. In other algorithms, such as anomaly detection or matched filtering (MF), where the resulting values ​​can span a very wide range, the maximum value derived from the histogram can be ignored. After setting the numerical range, the system 100 slices the entire range into steps and sets default initial values ​​for target representation on the display. A "blob" algorithm or other combination algorithms (such as decision trees) can also contribute to the automatic selection of thresholds for initial target representation.

[0173] One of the challenges in identifying targets within a multispectral data cube is pixel fragmentation. When using a histogram-based method to determine which pixels in a multispectral data cube are close enough to the target spectral signature to be considered potential target pixels, some target pixels may be farther away from the target spectral signature than a given distance threshold, resulting in gaps within the target area. The system 100 can utilize one or more cluster fusion algorithms to group pixels into clusters identified as targets. The cluster fusion algorithms can be based on the local average of pixels within the area of ​​the spectral data cube that contains the pixels. Some of these algorithms are based on the assumption that pixels within the target area are close to the target spectral signature even if they are below a given distance threshold. This assumption is used to calculate one or more metrics within the cluster area, calculating a local average that is higher than the average for areas of the spectral data cube that do not contain the target. The algorithm can scan the area of ​​clusters to identify a minimum bounding rectangle that circumscribes the area of ​​pixels with a high local mean, representing a smaller distance to the target spectral signature. The center of the bounding rectangle / box is used by the system 100 to determine the center of the target. In some instances, the cluster fusion algorithm is an iterative algorithm that iteratively searches for an area of ​​pixels whose bounding rectangle has an average that exceeds a mean distance threshold from the target spectral signature (i.e., is closer to the target spectral signature). Overlapping bounding rectangles are fused together. The system 100 can find the minimum bounding rectangle that circumscribes the fused rectangle and generate a bounding box that encompasses at least a portion of the fused rectangle. The bounding box is the identified target.

[0174] Continuing with the non-limiting example above, a first bounding box containing one group of pixels may be associated with a target spectral signature of strawberry and identified as strawberry in a second FOV, and another group of pixels contained within the second bounding box may be identified as blueberry in the second FOV. Note that the spectral data cube may be stored in the data repository 120 using a unique format that stores the identified potential targets as "layers" in the spectral data cube.

[0175] Optionally, the identified potential targets may be further investigated using one or more additional sensors (block 314). Multi-channel alignment of the spectral data cube allows for the geographic location of the targets to be determined. The geographic location may be used by system 100 to show the identified potential targets on the spectral data cube for images captured by the additional sensors. Continuing with our non-limiting example above, system 100 may show the identified potential strawberries (using their geographic location) on a daylight image. Using narrow-area daylight sensor 110, a user may zoom in on the identified potential strawberries and determine their visual quality.

[0176] It should be noted that, in addition to or as an alternative to the target identification and interrogation described above, system 100 can directly perform detection of one or more target materials in an uncalibrated multispectral data cube by utilizing a process for detecting one or more target materials in an uncalibrated multispectral data cube, as described in further detail herein with particular reference to FIG. 9. System 100 can also perform a process for empirical atmospheric calibration utilizing automatically identified objects of interest in images of a scene, as described in further detail herein with particular reference to FIG. 10. System 100 can also perform a process for determining an aligned multispectral data cube from one or more two-dimensional images of a scene, as described in further detail herein with particular reference to FIG. 11.

[0177] 3, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. It should also be noted that some of the blocks are optional (e.g., blocks 306, 308, and 314). It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is by no means binding, and the blocks may be performed by elements other than those described herein.

[0178] FIG. 4 is a flowchart illustrating an example sequence of operations performed by a real-time spectrum system with automatic exposure time calculations in accordance with the disclosed subject matter.

[0179] Thus, the real-time multispectral system 100 can be configured to implement the exposure time determination process 400, for example, using the exposure time determination module 126. The real-time multispectral system 100 can have adaptive real-time radiometric calibration. The system 100 can use different exposure times for each of the imaging channels of the multispectral sensor 104. The exposure time for each channel is determined according to a predefined empirical channel-average radiance level by assuming a common albedo (which is a measure of the diffuse reflection of solar radiation) and typical expected solar irradiance for each channel. The system adapts the radiometric calibration performed for each channel to the corresponding exposure time by dividing the radiometric measurement by the channel's adaptive integration time. In this way, the generated spectral data cube has an optimal, or at least improved, SNR while maintaining radiometric accuracy.

[0180] To this end, multispectral system 100 obtains an exposure time transformation matrix that defines the ratio between the exposure times of each of the multiple channels of the multispectral sensor (block 402). For example, the matrix may indicate that a given imaging channel has an exposure time that is four times the exposure time determined for the wideband SWIR simple mode imaging channel.

[0181] After obtaining the exposure time transformation matrix, system 100 determines a wide-area SWIR simple mode exposure time, which is an exposure time appropriate for a wide-area SWIR simple mode channel of the multiple channels of the multispectral sensor, based on observing the first FOV with the multispectral sensor in wide-area SWIR simple mode (block 406).

[0182] The system 100 may then calculate corresponding exposure times for at least some of the multiple channels of the multispectral sensor using the wideband SWIR simple mode exposure times and the exposure time transformation matrix (block 408).

[0183] Non-limiting examples of atmospheric correction processes performed by system 100 for each multispectral data cube generated by multispectral sensor 104 include one or more of the following steps: (a) dark image averaging, (b) dark image subtraction, (c) gain matrix resampling / averaging, (d) offset matrix resampling / averaging, and (e) radiometric calibration. Radiometric calibration is performed by system 100 for one or more pixels of the multispectral data cube retrieved by multispectral sensor 104 (this is typically performed for all pixels of the detector, or for a subset of the detector's pixels, excluding some edge or border pixels). Radiometric calibration may be performed on the multispectral sensor 104, including its filter wheel and / or one or more of the additional sensors, if present (e.g., wide-area daylight sensor 106, narrow-area daylight sensor 110, wide-area thermal sensor 114, narrow-area thermal sensor 118, narrow-area SWIR sensor 108, etc.). Radiometric calibration involves adjusting additional system 100 parameters, such as linearity, nonuniformity, and signal-to-noise ratio (SNR). For system 100 to analyze hyperspectral images, the captured multispectral data cube is converted from gray levels to radiometric measurements, after which atmospheric correction enables the final conversion to reflectance values. The resulting reflectance spectrum can then be analyzed by system 100 to identify or extract targets from the ambient spectrum. Radiometric calibration involves constructing a transformation matrix to the radiometric measurements. The radiation propagating through the system is affected by several parameters, such as the optical transmittance of the sensor of system 100, the quantum efficiency (QE) response of the detector, and the dark signal. The first two parameters multiply the signal and contribute to the gain of system 100, while the latter parameter is additive and defines the offset of system 100. The measurement setup can include an integrating sphere in front of the aperture of the sensor of system 100.Gain and offset matrices can be calculated from measurements performed at several signal levels, or alternatively, several iterations. The dark signal is the offset of the system 100. Subtraction of averaged dark images can be used to cancel the offset, leaving the gain matrix. The integrating sphere is also calibrated so its spectral radiometric measurements are known with high accuracy. The first assumption is that the system 100 is linear, and variations in integration time vary with the signal. While the integration time can be varied, the light level of the integrating sphere is limited to discrete values, which can introduce errors into the calibration compared to variations in integration time. Several images are captured by the system 100 for every integration time, allowing for image averaging and reducing the effects of random noise in the system 100. From the results of these measurements, the system 100 extracts the SNR level, system linearity, and nonuniformity correction, along with radiometric calibration.

[0184] It should be noted that, in addition to or as an alternative to at least a portion of the exposure time determination process 400 described above, the system 100 may perform calibration of the raw multispectral data cube by utilizing an atmospheric calibration based on a machine learning model, as further detailed herein with particular reference to FIG. 7.

[0185] 4, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0186] FIG. 5 is a flowchart illustrating an example series of operations performed by a real-time multispectral system for automatic atmospheric calibration matrix determination in accordance with the disclosed subject matter.

[0187] Thus, the real-time multispectral system 100 may be configured to perform an automatic atmospheric calibration process 500, for example, using the atmospheric calibration module 128. The multispectral system 100 has an automatic atmospheric calibration function. The system 100 uses calibration elements in a scene with known spectral reflectance functions. The calibration elements may be indicated by an automatic AI and / or ML discrimination model or by an operator or user of the system 100, who identifies the calibration elements in the scene and classifies the calibration objects as one of several predefined object types (e.g., the operator indicates an area of ​​the scene that is a paved road). The automatic AI and / or ML discrimination model may provide the operator of the system 100 with calibration material suggestions by thematically coloring parts of the scene. In this case, the AI ​​and / or ML model is pre-trained to segment the entire FOV into several phenomenal segments using labeled training data. The operator is then free to accept the suggestions or use a manual calibration material selection interface.

[0188] The system 100 automatically extracts the path radiance of the spectroradiometric measurements, and the selection / definition of the calibration material is performed by the operator or by AI and / or ML algorithms. This process can be controlled by automatic feedback on the quality of the calibration, which is presented to the operator on the screen as a coloring of the relevant area of ​​the calibration material (or by an AI and / or ML model analyzing the determined color). The automatic feedback on the quality of the calibration material can be based on an evaluation of the calibration material selected as a phenomenal segment. The spatial consistency of such a segment is calculated by the system 100 as the quantitative spectral difference between pixels that appear to belong to the same spatial phenomenon in the current FOV. Pixels with small differences are colored in real time, allowing the system 100 operator to visualize the selected calibration segment on the screen. The AI ​​and / or ML model performs an additional assessment of all determined phenomena as belonging to a predefined class of calibration material (e.g., vegetation, road, soil, rock). The system 100 can determine the atmospheric calibration for the entire spectral data cube or a portion of the spectral data cube. In some cases, specific targets may be enhanced and detected after atmospheric correction (ATC) based on specific calibration materials. For example, for best detection of a certain type or species of plant, atmospheric calibration for that specific plant may result in more accurate identification of the plant than calibration of the atmosphere with soil. Also, with AI and / or ML models, direct ATC may produce different results in different scenes, such as a specific target against a specific background at a specific range.

[0189] The system 100 may also be installed on an airborne platform. In some cases, the system 100 utilizes oblique imaging to capture a scene. The system 100 also addresses the issue of varying ranges for various pixels along the FOV and / or shaded areas of the scene. In these cases, the system 100 can determine atmospheric calibration for the entire spectral data cube or a portion of the spectral data cube. Partial atmospheric calibration is particularly useful for spectral scenes that are unevenly illuminated due to the nature of slant images. The division of the FOV into distance ranges according to the line-of-sight parameters of the current scene is performed directly from the scene using water vapor spectral absorption characteristics and the geometry of the camera model. The system 100 automatically performs atmospheric calibration of the imaged scene for all multispectral channels. In some cases, the system 100 may include AI and / or ML models capable of automatically performing direct atmospheric calibration on the spectral data cube. In some cases, the system 100 may include AI and / or ML models capable of automatically performing direct atmospheric calibration on the spectral data cube. Therefore, an AI neural network is trained using a large dataset of atmospherically corrected imagery from different scenes and some ground truth (labeled) information to build a reliable AI and / or ML model. The resulting trained model is fed as input into the system 100's bands and sensor metadata, and the output is corrected reflectance ready for target detection.

[0190] Another option is for system 100 to use AI and / or ML models to directly identify targets from radiance spectral data lacking atmospheric correction. This AI and / or ML model allows system 100 to directly detect targets in the radiance spectral data cube without atmospheric correction (ATC). This is done based on the physical behavior of the atmosphere and the expected relationship between path radiance values ​​of various channels. The trained AI and / or ML model can be pre-trained to detect all variations in the radiance spectral signature of a target of interest using simulations of all possible illumination conditions. Note that system 100 can employ AI and / or ML models for any of its sensors, i.e., for multispectral sensor 104 and / or additional sensors (e.g., wide-area daylight sensor 106, narrow-area daylight sensor 110, wide-area thermal sensor 114, narrow-area thermal sensor 118, narrow-area SWIR sensor 108, laser rangefinder 116, laser pointer 112, etc.).

[0191] To this end, the multispectral system 100 obtains reflectance values ​​for one or more known object types (block 502).

[0192] After obtaining reflectance values ​​for known object types, system 100 captures an image of the first FOV using the wideband SWIR simple mode imaging channel of the multispectral sensor (block 504).

[0193] The system 100 may then receive input from a user indicating one or more pixels of known reflectance that represent objects of one or more object types within the captured image (block 506).

[0194] After receiving input from the user, the system 100 identifies one or more dark pixels in the captured image (block 508).

[0195] The system 100 may determine an atmospheric calibration matrix for at least some of the imaging channels of the multispectral sensor based on the pixels of known reflectance, the dark pixels, and the captured image (block 510).

[0196] 5, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0197] FIG. 6 is a flowchart illustrating an example sequence of operations performed by a real-time multispectral system for multi-channel registration in accordance with the subject matter of this disclosure.

[0198] Thus, the real-time multispectral system 100 can be configured to perform the multichannel alignment process 600, for example, using the multichannel alignment module 130. The multispectral system 100 can eliminate or at least mitigate residual misalignment of the captured frames that make up the spectral data cube. An MCR algorithm can be used to eliminate spectral mixing and improve image stability during collection of the spectral data cube. The system 100 selects a reference frame, which is an image from one of the imaging channels of the multispectral sensor 104, a reference imaging channel, captured within a given time frame. The system 100 then aligns all other images from all other imaging channels, other than the reference imaging channel, within the given time frame to the reference frame. This is achieved in real time by dividing the computational tasks among the system's hardware, firmware, and software elements as follows: Harris corner detection is implemented by adaptive threshold matching in field-programmable gate array (FPGA) hardware using the Sobel rule for edge and corner matching. A Random Sample Consensus (RANSAC) algorithm is then used to iteratively calculate, test, and ultimately propose an optimal affine matrix using feedback residual errors. The MCR process is performed by finding edges in each image and aligning the edges between the captured image and the reference frame. For one or more of the images that have a contrast difference with the reference frame that exceeds a threshold, a mirror image of one or more of the images is used during the alignment process.

[0199] To this end, the multispectral system 100 captures a series of images of the second FOV using one or more of the imaging channels of the multispectral sensor over a given time frame (block 602).

[0200] After capturing the series of images, the system 100 selects a reference frame (block 604), which is an image from a reference imaging channel of one of the imaging channels used to capture the series of images within a given time frame.

[0201] The system 100 may then register at least some of the images captured in a given time frame by other imaging channels other than the reference imaging channel to the reference frame (block 606).

[0202] 6, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0203] FIG. 7 is a flowchart illustrating an example of a series of operations performed to automatically generate a calibrated multispectral data cube from an uncalibrated multispectral data cube in accordance with the disclosed subject matter.

[0204] Thus, the real-time multispectral system 100 may be configured to perform a process 700 for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, for example, using the calibrated multispectral data cube generation module 132. The multispectral system 100 may perform atmospheric calibration based on AI and / or ML models. The calibrated multispectral data cube may potentially be generated from the uncalibrated multispectral data cube by a calibration process, such as an atmospheric calibration process. The atmospheric calibration process may include inter-instrument matching corrections, if necessary.

[0205] To this end, multispectral system 100 may be configured to acquire (A) an uncalibrated multispectral data cube, a machine learning model capable of receiving an uncalibrated multispectral data cube and generating a corresponding calibrated multispectral data cube, the machine learning model being trained using a labeled training data set including a plurality of training records, each training record including (i) a training uncalibrated multispectral data cube and (ii) a training calibrated multispectral data cube corresponding to the training uncalibrated multispectral data cube (block 702). The acquired uncalibrated multispectral data cube may be captured by one or more multispectral sensors, such as multispectral sensor 104.

[0206] It should be noted that the machine learning model may be obtained by system 100 from an external source, e.g., a system external to system 100. In other cases, the machine learning model may be at least partially generated and / or trained by system 100 itself. The training data used to train the machine learning model may be obtained by system 100 from an external source, e.g., a system external to system 100. In some cases, at least a portion of the training data used to train the machine learning model may be generated automatically, e.g., using an automated process for generating multispectral labeled training data sets. The training data is labeled; that is, the training data includes ground truth, which is a pair of calibrated and uncalibrated training multispectral data cubes, and the calibrated training multispectral data cubes may potentially be generated from the uncalibrated training multispectral data cubes through a calibration process.

[0207] In some cases, at least one training record among the training records is generated using an atmospheric simulator. The atmospheric simulator can receive a calibrated multi-multispectral data cube and one or more atmospheric conditions and can generate an uncalibrated multispectral data cube corresponding to the calibrated multispectral data cube under the atmospheric conditions. In such cases, two or more training records among the training records can include different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions. Note that the one or more atmospheric conditions can include one or more of humidity level, sunlight angle, aerosol model, visibility score, or geographic area.

[0208] After obtaining the machine learning model and the uncalibrated multispectral data cube, the system 100 may generate a calibrated multispectral data cube utilizing the machine learning model and the uncalibrated multispectral data cube (block 704).

[0209] At least a portion of the process 700 for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube may additionally and / or alternatively be implemented by the system 100 using a database-based algorithm that searches a database of calibration parameters for calibrating the uncalibrated multispectral data cube. This algorithm may include, for example, the following steps: The database is pre-prepared with a large number of parameters. The system 100 searches the database for the best-matching resulting reflectance from among the possibilities in the database. The search may be performed pixel-by-pixel or macropixel-by-macropixel of the uncalibrated multispectral data cube. An extensive iterative process may be used to build a database of results expected to be measured using capture of the uncalibrated multispectral data cube by a multispectral or hyperspectral sensor (e.g., multispectral sensor 104). Local parameters, such as distance, reflectance level, oblique angle, and average environmental reflectance level, may be incremented in small steps and iterated to predict the expected captured radiance accordingly. Knowledge of global parameters such as visibility, humidity level, solar azimuth angle, geographic region, aerosol model, and path radiance can be the basis for calculating the predicted captured radiance described above using an appropriate atmospheric simulator, for example, the Moderate Resolution Atmospheric Radiative Transfer (MODTRAN) computer mode. For each iteration, the results of each iteration along with the guessed and predicted radiance values ​​used are recorded as records in a large database. During operation, distance maps and estimates of one or more global atmospheric values ​​can serve to eliminate irrelevant records in the database. An efficient search algorithm is used to best match the captured radiance values ​​for each pixel to the estimate across all relevant records. An averaging process can then be used on the relevant reflectance values ​​in the table to estimate reflectance values ​​and generate a calibrated multispectral data cube from the uncalibrated multispectral data cube.

[0210] 7, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0211] FIG. 8 is a flowchart illustrating an example of a series of operations performed for the automated generation of a multispectral labeled training data set in accordance with the subject matter of this disclosure.

[0212] Thus, the real-time multispectral system 100 may be configured to implement the automated multispectral labeled training data set generation process 800, for example, using the multispectral labeled training data set generation module 134. The multispectral system 100 may automatically generate hyperspectral and / or multispectral training data sets. The hyperspectral and / or multispectral labeled training data sets may include one or more training records. Each training record may include an uncalibrated training multispectral data cube and a calibrated training multispectral data cube, and the calibrated training multispectral data cube may be generated from the uncalibrated training multispectral data cube by a calibration process.

[0213] To this end, the multispectral system 100 may be configured to obtain (A) a three-dimensional (3D) model of a scene, the 3D model having one or more characteristics (such as a geographic area, a sunlight angle, or a viewing distance) and including a collection of pixels, at least one of which is associated with (a) a color and (b) a label indicating a material group of which the element in the scene associated with the pixel is made; (B) a material database including a list of materials, at least one of which is associated with (i) the material's spectral reflectance signature vector, (ii) the material's typical color, and (iii) a given material group of materials; and (C) a heuristic table including one or more rules, each rule specifying a probability of the presence of a given material in a given scene based on characteristics of the given scene (block 802). Note that in some instances, the calibration process is an atmospheric calibration process.

[0214] In some instances, two or more of the training records include different uncalibrated multispectral data cubes generated from the same calibrated multispectral under different atmospheric conditions, such as humidity levels, sunlight angles, aerosol models, visibility scores, or geographic areas.

[0215] After obtaining the 3D model of the scene, the material database, and the heuristics table, the system 100 can capture a two-dimensional (2D) image from the 3D model of the scene, where the 2D image includes a subset of pixels (block 804).

[0216] Using the captured 2D image, the system 100 can generate a training calibrated multispectral data cube for the training record by, for at least one given pixel of the subset of pixels, querying a material database for a list of materials that are likely to be materials having the material group of the given pixel; removing from the list of likely materials those materials whose probability of being present in the 2D image is less than a first threshold according to the rules of the heuristic table and the characteristics of the 3D model to obtain a revised list of likely materials; determining a matching material for the given pixel from the revised list of likely materials based on a match between the color of the given pixel and a typical color associated with a material from the revised list of likely materials; and selecting at least a portion of the spectral reflectance signature vector associated with the matching material to be part of the training calibrated multispectral data cube at the given pixel location (block 806).

[0217] After generating the calibrated training multispectral data cube for the training record, system 100 may generate an uncalibrated training multispectral data cube for the training record using the calibrated training multispectral data cube and an atmospheric simulator, which may receive (i) the calibrated training multispectral data cube and (ii) one or more atmospheric conditions, and may generate an uncalibrated training multispectral data cube corresponding to the calibrated training multispectral data cube under those atmospheric conditions (block 808).

[0218] 8, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is by no means binding, and the blocks may be performed by elements other than those described herein.

[0219] FIG. 9 is a flow chart illustrating an example series of operations performed to detect one or more target materials in an uncalibrated multispectral data cube in accordance with the disclosed subject matter.

[0220] Thus, the real-time multispectral system 100 may be configured to perform a process 900 for detecting one or more target materials in an uncalibrated multispectral data cube, for example, using the target material detection in uncalibrated multispectral data cube module 136. The multispectral system 100 may directly detect various target materials from the uncalibrated multispectral data cube (these are materials whose presence the system 100 may be used to identify in a scene).

[0221] To this end, the multispectral system 100 may be configured to acquire (A) an uncalibrated multispectral data cube and a machine learning model capable of receiving the uncalibrated multispectral data cube and determining, for at least one pixel of the pixels, at least one material indicator indicative of the presence of a given target material at the pixel location, the machine learning model being trained using a labeled training data set including a plurality of training records, each training record including (i) the uncalibrated training multispectral data cube and (ii) at least one training material indicator associated with at least one pixel of the uncalibrated training multispectral data cube indicative of the presence of a target material at the pixel location (block 902). Note that the machine learning model may be acquired by the system 100 from an external source, e.g., a system external to the system 100. In other cases, the machine learning model may be at least partially generated and / or trained by the system 100 itself. The training data used to train the machine learning model may be acquired by the system 100 from an external source, e.g., a system external to the system 100. In some cases, at least a portion of the training data used to train the machine learning model may be generated automatically, e.g., using an automated process for generating multispectral labeled training data sets. The training data may be labeled, i.e., the training data may include ground truth, which may be a pair of calibrated and uncalibrated training multispectral data cubes, potentially generated from the uncalibrated training multispectral data cubes by a calibration process.

[0222] After obtaining the machine learning model and the uncalibrated multispectral data cube, the system 100 determines at least one material indicator and a corresponding calibrated multispectral data cube for at least one of the pixels of the uncalibrated multispectral data cube, where the corresponding calibrated multispectral data cube is calculated by utilizing an atmospheric simulator to simulate a calibration process and a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube (block 904).

[0223] 9, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0224] FIG. 10 is a flowchart illustrating an example sequence of operations performed for empirical atmospheric calibration utilizing automatically identified target objects in images of a scene, in accordance with the subject matter of this disclosure.

[0225] Thus, the real-time multispectral system 100 may be configured to perform the empirical atmospheric calibration process 1000 utilizing automatically identified objects in an image of a scene, for example, using the automatic calibration object identification module 138. The multispectral system 100 may automatically identify one or more calibration objects in the scene. A calibration object is an object with known reflectance that can be used to calibrate an uncalibrated multispectral data cube.

[0226] To this end, multispectral system 100 may be configured to acquire (A) an image of a scene and a machine learning model capable of identifying the presence of at least one object of interest within the scene, where each object of interest is associated with (i) a location within the scene and (ii) a predetermined representative reflectance spectral signature, and (B) an image of the scene (block 1002). Note that the machine learning model may be acquired by system 100 from an external source, e.g., a system external to system 100. In other cases, the machine learning model may be at least partially generated and / or trained by system 100 itself. Training data used to train the machine learning model may be acquired by system 100 from an external source, e.g., a system external to system 100. In some cases, at least a portion of the training data used to train the machine learning model may be automatically generated, e.g., utilizing an automated process for generating multispectral labeled training data sets. The training data is labeled. That is, the training data includes ground truth, which is a pair of a calibrated training multispectral data cube and an uncalibrated training multispectral data cube, and the calibrated training multispectral data cube can potentially be generated from the uncalibrated training multispectral data cube by a calibration process.

[0227] After obtaining the machine learning model and the image of the scene, the system 100 may utilize the machine learning model and the image of the scene to identify the presence of at least one of the objects to be automatically identified in the image of the scene (block 1004).

[0228] After identifying the presence of at least one of the automatically-identified target objects in the image, the system 100 may utilize an empirical atmospheric calibration process to calibrate an uncalibrated multispectral data cube associated with the scene based on the location and predetermined representative reflectance spectral signature associated with the automatically-identified target object (block 1006).

[0229] 10, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0230] FIG. 11 is a flowchart illustrating an example series of operations performed to determine an aligned multispectral data cube from one or more two-dimensional images of a scene in accordance with the subject matter of this disclosure.

[0231] Thus, the real-time multispectral system 100 may be configured to perform a process 1100 for determining an aligned multispectral data cube from one or more two-dimensional images of a scene process, for example, using the aligned multispectral data cube determination module 140. The multispectral system 100 may perform AI and / or ML-based alignment. The system 10 may determine an aligned multispectral data cube from one or more 2D images of a scene, where at least some of the 2D images are captured in different wavelength ranges and at least some of the 2D images are taken from different viewpoints of the scene, and the aligned multispectral data cube may potentially be generated from the 2D images by the alignment process.

[0232] To this end, the multispectral system 100 may be configured to acquire (A) a machine learning model capable of receiving (a) a source 2D image among the 2D images of the scene and (b) a target 2D image among the 2D images of the scene and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model being trained using a training data set including a plurality of training records, each training record including (i) a training source 2D image, (ii) a training target 2D image, and (iii) one or more training flow maps that map modifications to be made to pixels of the training source 2D image to align the training source 2D image with the training target 2D image; and (B) given 2D images of the scene, each given 2D image having a different wavelength range and each image taken from a different perspective of the scene (block 1102). The training data set includes: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which the element in the scene associated with the pixel is composed; (B) a materials database including a list of materials, at least one material being associated with (i) the material's spectral reflectance signature vector, (ii) the material's typical color, and (iii) a given material group of the material; and (C) one or more capturing at least one 2D image from a 3D model of the scene, each 2D image being captured from a different viewpoint of the scene, each 2D image comprising a subset of pixels, and each training 2D image being associated with a different wavelength range, to obtain training source 2D images for the given training recording; and capturing, for the at least one training source 2D image,selecting a training target 2D image from among the 2D images captured from the 3D model to obtain a training target 2D image for a given training record; for at least one given pixel of the subset of pixels of the training source 2D image, querying a material database for a list of substances that may be materials having a material group for the given pixel; removing from the list of possible materials those substances whose probability of being in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model is less than a first threshold, to obtain a revised list of possible materials; and selecting a characteristic associated with the color of the given pixel and the material from the revised list of possible materials. determining a matching material for a given pixel from the modified list of possible materials based on a match between the material type and color; selecting a corresponding portion of a spectral reflectance signature vector associated with the matching material according to the wavelength of the 2D image to be the portion at the given pixel of the source training 2D image; and generating a training flow map for the given training record that maps changes to be made to the pixels of the source 2D image to align the source and target 2D images.

[0233] It should be noted that the machine learning model may be obtained by system 100 from an external source, e.g., a system external to system 100. In other cases, the machine learning model may be at least partially generated and / or trained by system 100 itself. The training data used to train the machine learning model may be obtained by system 100 from an external source, e.g., a system external to system 100. In some cases, at least a portion of the training data used to train the machine learning model may be generated automatically, e.g., using an automated process for generating multispectral labeled training data sets. The training data is labeled; that is, the training data includes ground truth, which is a pair of calibrated and uncalibrated training multispectral data cubes, and the calibrated training multispectral data cubes may potentially be generated from the uncalibrated training multispectral data cubes through a calibration process.

[0234] After obtaining the machine learning model and the given 2D images of the scene, the system 100 can determine at least one given flow map for at least one given source 2D image of the given 2D images and at least one given target 2D image of the given 2D images, where the given flow map maps changes to be made to pixels of the given source 2D image to align the given source 2D image with the given target 2D image by utilizing the machine learning model for the given source 2D image and the given target 2D image (block 1104).

[0235] After determining the given flow map for at least one given source 2D image of the given 2D images and for at least one given target 2D image of the given 2D images, the system 100 may generate an aligned multispectral data cube using the given source 2D image, the corresponding given flow map, and a remapping function (block 1106).

[0236] 11, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow diagrams are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0237] It is to be understood that the subject matter of the present disclosure is not limited in its application to the details set forth in the description contained herein or illustrated in the drawings. The subject matter of the present disclosure is capable of other embodiments and of being practiced and carried out in various ways. Accordingly, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting. Those skilled in the art will thus appreciate that the conception upon which the present disclosure is based may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out some of the purposes of the subject matter of the present disclosure.

[0238] It will also be understood that systems in accordance with the presently disclosed subject matter may be implemented, at least in part, as a suitably programmed computer. Similarly, the presently disclosed subject matter contemplates a computer program readable by a computer to perform the disclosed methods. The presently disclosed subject matter further contemplates a machine-readable memory tangibly embodying a program of instructions executable by a machine to perform the disclosed methods.

Claims

1. a multispectral sensor capable of capturing images in multiple imaging channels, each having a different wavelength range; one or more additional sensors; A processing circuit, acquiring one or more target spectral signatures; activating the multispectral sensor, wherein the multispectral sensor is activated to operate in a wideband shortwave infrared (SWIR) simple mode; determining a calculated exposure time for each imaging channel of the plurality of imaging channels of the multispectral sensor based on observing a first field of view (FOV) with the multispectral sensor in the wide-area SWIR simple mode; determining an atmospheric calibration matrix for the multispectral sensor based on input from a user; generating a multispectral data cube of a second FOV observed by the multispectral sensor utilizing the multispectral sensor, the calculated exposure time, and the atmospheric correction matrix, wherein the generation of the multispectral data cube includes radiometric calibration and multi-channel alignment; utilizing the multispectral data cube to identify one or more potential targets, each target being a group of pixels identified in the multispectral data cube having a spectral signature corresponding to at least one of the acquired target spectral signatures, each target having a geographic location; utilizing the one or more additional sensors to investigate one or more of the identified potential targets; and a processing circuit configured to: A multispectral potential target identification system comprising:

2. obtaining, by a processing circuit, one or more target spectral signatures; activating, by the processing circuitry, a multispectral sensor capable of capturing images in multiple imaging channels, each having a different wavelength range, wherein the multispectral sensor is activated to operate in a broadband shortwave infrared (SWIR) simple mode; determining, by the processing circuitry, a calculated exposure time for each imaging channel of the plurality of imaging channels of the multispectral sensor based on observing a first field of view (FOV) with the multispectral sensor in the wide-area SWIR simple mode; determining, by the processing circuitry, an atmospheric calibration matrix for the multispectral sensor based on input from a user; generating, by the processing circuitry, a multispectral data cube of a second FOV observed by the multispectral sensor utilizing the multispectral sensor, the calculated exposure time, and the atmospheric correction matrix, wherein the generation of the multispectral data cube includes radiometric calibration and multi-channel alignment; utilizing the multispectral data cube, by the processing circuitry, to identify one or more potential targets, each target being a group of pixels identified in the multispectral data cube having a spectral signature corresponding to at least one of the obtained target spectral signatures, each target having a geographic location; investigating, by the processing circuitry, one or more of the identified potential targets utilizing the one or more additional sensors; A multispectral potential target identification method including:

3. A non-transitory computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code comprising: obtaining, by a processing circuit, one or more target spectral signatures; activating, by the processing circuitry, a multispectral sensor capable of capturing images in multiple imaging channels, each having a different wavelength range, wherein the multispectral sensor is activated to operate in a broadband shortwave infrared (SWIR) simple mode; determining, by the processing circuitry, a calculated exposure time for each imaging channel of the plurality of imaging channels of the multispectral sensor based on observing a first field of view (FOV) with the multispectral sensor in the wide-area SWIR simple mode; determining, by the processing circuitry, an atmospheric calibration matrix for the multispectral sensor based on input from a user; generating, by the processing circuitry, a multispectral data cube of a second FOV observed by the multispectral sensor utilizing the multispectral sensor, the calculated exposure time, and the atmospheric correction matrix, wherein the generation of the multispectral data cube includes radiometric calibration and multi-channel alignment; utilizing the multispectral data cube, by the processing circuitry, to identify one or more potential targets, each target being a group of pixels identified in the multispectral data cube having a spectral signature corresponding to at least one of the obtained target spectral signatures, each target having a geographic location; investigating, by the processing circuitry, one or more of the identified potential targets utilizing the one or more additional sensors; A non-transitory computer-readable storage medium executable by at least one processing circuit of a computer to perform a method including:

4. 1. A system for automated generation of a multispectral labeled training data set, the multispectral labeled training data set comprising one or more training records, each training record comprising (i) an uncalibrated training multispectral data cube and (ii) a calibrated training multispectral data cube, the calibrated training multispectral data cube being generateable from the uncalibrated training multispectral data cube by a calibration process, the system comprising: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which an element in the scene associated with the pixel is composed; (B) a substance database including a list of substances, wherein at least one substance is associated with (i) a spectral reflectance signature vector of the substance, (ii) a typical color of the substance, and (iii) a given substance group of the substance; and (C) a heuristic table including one or more rules, each rule defining a probability of the presence of a given substance in a given scene based on characteristics of the scene; and capturing a two-dimensional (2D) image from the 3D model of the scene, the 2D image including the subset of pixels; For at least one given pixel of said subset of pixels, querying said substance database for a list of materials that may be materials of said material group for said given pixel; removing from the list of possible materials those materials whose probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model is less than a first threshold, to obtain a revised list of possible materials; determining a matching material for the given pixel from the modified list of possible materials based on a match between the color of the given pixel and the representative color associated with the material from the modified list of possible materials; and selecting at least a portion of the spectral reflectance signature vectors associated with the matching materials to be part of the training calibrated multispectral data cube at the given pixel location; generating the training calibrated multispectral data cube for the training recordings by performing generating the uncalibrated training multispectral data cube for the training recording using the calibrated training multispectral data cube and an atmospheric simulator, the atmospheric simulator being capable of receiving (i) the calibrated training multispectral data cube and (ii) one or more atmospheric conditions and generating the uncalibrated training multispectral data cube corresponding to the calibrated training multispectral data cube under the atmospheric conditions; A system comprising a processing circuit configured to:

5. 5. The system of claim 4, wherein two or more of the training records comprise different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

6. The system of claim 4 , wherein the calibration process is an atmospheric calibration process.

7. The system of claim 4 , wherein the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, or a viewing distance.

8. The system of claim 4 , wherein the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

9. The system of claim 4 , wherein the processing circuitry is further configured to add target pixels to the 3D model prior to the capture of the 2D image.

10. 5. The system of claim 4, wherein the generating the uncalibrated training multispectral data cube further comprises adding texture to the generated calibrated training multispectral data cube based on texture associated with a corresponding captured 2D image.

11. The system of claim 4 , wherein the generating the training uncalibrated multispectral data cube further comprises adding simulated registration errors.

12. 5. The system of claim 4, wherein the generating the uncalibrated training multispectral data cube further comprises blurring at least one of the corresponding captured 2D images prior to the generating.

13. 5. The system of claim 4, wherein said generating said uncalibrated training multispectral data cube further comprises adding shot noise to at least one of said corresponding captured 2D images prior to said generating.

14. 1. A method for automated generation of a multispectral labeled training data set, the multispectral labeled training data set comprising one or more training records, each training record comprising (i) an uncalibrated training multispectral data cube and (ii) a calibrated training multispectral data cube, the calibrated training multispectral data cube being generateable from the uncalibrated training multispectral data cube by a calibration process, the method comprising: The processing circuitry (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which an element in the scene associated with the pixel is composed; (B) a substance database including a list of substances, wherein at least one substance is associated with (i) a spectral reflectance signature vector of the substance, (ii) a typical color of the substance, and (iii) a given substance group of the substance; and (C) a heuristic table including one or more rules, each rule defining a probability of the presence of a given substance in a given scene based on characteristics of the scene; obtaining a capturing, by the processing circuitry, a two-dimensional (2D) image from the 3D model of the scene, the 2D image comprising the subset of pixels; For at least one given pixel of the subset of pixels, the processing circuitry: querying, by the processing circuitry, the material database for a list of materials that may be materials having the material group for the given pixel; removing, by the processing circuitry, from the list of possible materials, those materials that have a probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model that is less than a first threshold, to obtain a revised list of possible materials; determining, by the processing circuitry, a matching material for the given pixel from the modified list of possible materials based on a match between the color of the given pixel and the representative color associated with the material from the modified list of possible materials; and selecting, by the processing circuitry, at least a portion of the spectral reflectance signature vector associated with the matching material to be a portion of the training calibrated multispectral data cube at the given pixel location. generating the training calibrated multispectral data cube for the training recordings by performing generating, by the processing circuitry, the uncalibrated training multispectral data cube for the training recording using the calibrated training multispectral data cube and an atmospheric simulator, the atmospheric simulator capable of receiving (i) the calibrated training multispectral data cube and (ii) one or more atmospheric conditions, and generating the uncalibrated training multispectral data cube corresponding to the calibrated training multispectral data cube under the atmospheric conditions; A method comprising:

15. 15. The method of claim 14, wherein two or more of the training records comprise different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

16. The method of claim 14 , wherein the calibration process is an atmospheric calibration process.

17. The method of claim 14 , wherein the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, or a viewing distance.

18. 15. The method of claim 14, wherein the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

19. The method of claim 14 , further comprising adding target pixels to the 3D model prior to the capture of the 2D image.

20. 15. The method of claim 14, wherein the generating the uncalibrated training multispectral data cube further comprises adding texture to the generated calibrated training multispectral data cube based on texture associated with the corresponding captured 2D image.

21. The method of claim 14 , wherein the generating the training uncalibrated multispectral data cube further comprises adding simulated registration errors.

22. 15. The method of claim 14, wherein said generating said uncalibrated training multispectral data cube further comprises blurring at least one of said corresponding captured 2D images prior to said generating.

23. 15. The method of claim 14, wherein said generating said uncalibrated training multispectral data cube further comprises adding shot noise to at least one of said corresponding captured 2D images prior to said generating.

24. A non-transitory computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code comprising: The processing circuitry (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which an element in the scene associated with the pixel is composed; (B) a substance database including a list of substances, wherein at least one substance is associated with (i) a spectral reflectance signature vector of the substance, (ii) a typical color of the substance, and (iii) a given substance group of the substance; and (C) a heuristic table including one or more rules, each rule defining a probability of the presence of a given substance in a given scene based on characteristics of the scene; obtaining a capturing, by the processing circuitry, a two-dimensional (2D) image from the 3D model of the scene, the 2D image comprising the subset of pixels; For at least one given pixel of the subset of pixels, the processing circuitry: querying, by the processing circuitry, the material database for a list of materials that may be materials having the material group for the given pixel; removing, by the processing circuitry, from the list of possible materials, those materials that have a probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model that is less than a first threshold, to obtain a revised list of possible materials; determining, by the processing circuitry, a matching material for the given pixel from the modified list of possible materials based on a match between the color of the given pixel and the representative color associated with the material from the modified list of possible materials; selecting, by the processing circuitry, at least a portion of the spectral reflectance signature vectors associated with the matching materials to be part of the training calibrated multispectral data cube at the given pixel location; and generating, by the processing circuitry, the uncalibrated training multispectral data cube for the training recording using the calibrated training multispectral data cube and an atmospheric simulator, the atmospheric simulator being capable of receiving (i) the calibrated training multispectral data cube and (ii) one or more atmospheric conditions and generating the uncalibrated training multispectral data cube corresponding to the calibrated training multispectral data cube under the atmospheric conditions; generating the training calibrated multispectral data cube for the training recordings by performing A non-transitory computer-readable storage medium executable by at least one processing circuit of a computer to perform a method including:

25. 1. A system for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, wherein the calibrated multispectral data cube is generateable from the uncalibrated multispectral data cube by a calibration process, the system comprising: (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and generating the calibrated multispectral data cube, a machine learning model trained using a labeled training data set including a plurality of training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) a calibrated training multispectral data cube corresponding to the uncalibrated training multispectral data cube; (B) the uncalibrated multispectral data cube; and and generating the calibrated multispectral data cube using the machine learning model and the uncalibrated multispectral data cube; A system comprising a processing circuit configured to:

26. 26. The system of claim 25, wherein at least one of the training records is generated using an atmospheric simulator capable of receiving (i) a calibrated multispectral data cube and (ii) one or more atmospheric conditions and generating an uncalibrated multispectral data cube corresponding to the calibrated multispectral data cube under the atmospheric conditions.

27. 27. The system of claim 26, wherein two or more given ones of the training records comprise different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

28. 27. The system of claim 26, wherein the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

29. 26. The system of claim 25, wherein the calibration process is an atmospheric calibration process.

30. 26. The system of claim 25, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

31. 26. The system of claim 25, wherein the machine learning model is trained using reinforcement learning methods.

32. 26. The system of claim 25, wherein the machine learning model further comprises a vision transformer.

33. 26. The system of claim 25, wherein the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

34. 1. A method for automatically generating a calibrated multispectral data cube from an uncalibrated multispectral data cube, wherein the calibrated multispectral data cube can be generated from the uncalibrated multispectral data cube by a calibration process, the method comprising: The processing circuitry (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and generating the calibrated multispectral data cube, a machine learning model trained using a labeled training data set including a plurality of training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) a calibrated training multispectral data cube corresponding to the uncalibrated training multispectral data cube; (B) the uncalibrated multispectral data cube; and obtaining a generating, by the processing circuitry, the calibrated multispectral data cube using the machine learning model and the uncalibrated multispectral data cube; A method comprising:

35. 35. The method of claim 34, wherein at least one of the training records is generated using an atmospheric simulator capable of receiving (i) a calibrated multispectral data cube and (ii) one or more atmospheric conditions and generating an uncalibrated multispectral data cube corresponding to the calibrated multispectral data cube under the atmospheric conditions.

36. 36. The method of claim 35, wherein two or more given ones of the training records comprise different uncalibrated multispectral data cubes generated from the same calibrated multispectral data cube under different atmospheric conditions.

37. 36. The method of claim 35, wherein the one or more atmospheric conditions include one or more of a humidity level, a sunlight angle, an aerosol model, a visibility score, or a geographic area.

38. 35. The method of claim 34, wherein the calibration process is an atmospheric calibration process.

39. 35. The method of claim 34, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

40. 35. The method of claim 34, wherein the machine learning model is trained using a reinforcement learning method.

41. 35. The method of claim 34, wherein the machine learning model further comprises a vision transformer.

42. 35. The method of claim 34, wherein the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

43. A non-transitory computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code comprising: The processing circuitry (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and generating the calibrated multispectral data cube, a machine learning model trained using a labeled training data set including a plurality of training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) a calibrated training multispectral data cube corresponding to the uncalibrated training multispectral data cube; (B) the uncalibrated multispectral data cube; and obtaining a generating, by the processing circuitry, the calibrated multispectral data cube using the machine learning model and the uncalibrated multispectral data cube; A non-transitory computer-readable storage medium executable by at least one processing circuit of a computer to perform a method including:

44. 1. A system for determining a registered multispectral data cube from one or more two-dimensional (2D) images of a scene, each 2D image captured at a different wavelength range, each image taken from a different viewpoint of the scene, and wherein the registered multispectral data cube can potentially be generated from the 2D images by a registration process, the system comprising: (A) A machine learning model capable of receiving (a) a source 2D image of the 2D images of the scene and (b) a target 2D image of the 2D images of the scene, and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model is trained using a training data set including a plurality of training records, each training record including (i) a training source 2D image, (ii) a training target 2D image, and (iii) one or more training flow maps that map modifications to be made to pixels of the training source 2D image to align the training source 2D image with the training target 2D image; (B) given 2D images of the scene, each given 2D image having a different wavelength range and each image taken from a different viewpoint of the scene; and determining at least one given flow map for at least one given source 2D image of the given 2D images and for at least one given target 2D image of the given 2D images, the given flow map mapping changes to be made to pixels of the given source 2D image in order to align the given source 2D image with the given target 2D image by utilizing the machine learning model for the given source 2D image and the given target 2D image; generating the registered multispectral data cube using the given source 2D image, the corresponding given flow map, and a remapping function; A system comprising a processing circuit configured to:

45. At least one given training record of the training data set comprises: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which an element in the scene associated with the pixel is composed; (B) a substance database including a list of substances, wherein at least one substance is associated with (i) a spectral reflectance signature vector of the substance, (ii) a typical color of the substance, and (iii) a given substance group of the substance; and (C) a heuristic table including one or more rules, each rule defining a probability of the presence of a given substance in a given scene based on characteristics of the scene; and capturing at least one 2D image from the 3D model of the scene, each 2D image captured from a different viewpoint of the scene, each 2D image comprising a subset of the pixels, and each training 2D image associated with a different wavelength range to obtain the training source 2D image for the given training recording; selecting, for at least one training source 2D image, a training target 2D image from among the 2D images captured from the 3D model to obtain the training target 2D image for the given training recording; For at least one given pixel of the subset of pixels of the training source 2D images: querying said substance database for a list of materials that may be materials of said material group for said given pixel; removing from the list of possible materials those materials whose probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model is less than a first threshold, to obtain a revised list of possible materials; determining a matching material for the given pixel from the modified list of possible materials based on a match between the color of the given pixel and the representative color associated with the material from the modified list of possible materials; selecting a corresponding portion of the spectral reflectance signature vector associated with the matching material according to the wavelength of the 2D image to be the portion at the given pixel location in the training source 2D image; and generating a training flow map for the given training recording that maps modifications to be made to pixels of the source 2D image to align the source and target 2D images; generating the training flow map for the given training recording by:

45. The system of claim 44, wherein the system is generated by performing

46. 45. The system of claim 44, wherein at least one consecutive pair of the 2D images of the scene overlap by more than an overlap threshold.

47. 45. The system of claim 44, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

48. 45. The system of claim 44, wherein the machine learning model further comprises a vision transformer.

49. 46. ​​The system of claim 45, wherein the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, and a viewing distance.

50. 1. A method for determining a registered multispectral data cube from one or more two-dimensional (2D) images of a scene, each 2D image captured at a different wavelength range, each image taken from a different viewpoint of the scene, and wherein the registered multispectral data cube can potentially be generated from the 2D images by a registration process, the method comprising: The processing circuitry (A) A machine learning model capable of receiving (a) a source 2D image of the 2D images of the scene and (b) a target 2D image of the 2D images of the scene, and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model is trained using a training data set including a plurality of training records, each training record including (i) a training source 2D image, (ii) a training target 2D image, and (iii) one or more training flow maps that map modifications to be made to pixels of the training source 2D image to align the training source 2D image with the training target 2D image; (B) given 2D images of the scene, each given 2D image having a different wavelength range and each image taken from a different viewpoint of the scene; obtaining a determining, by the processing circuitry, at least one given flow map for at least one given source 2D image of the given 2D images and for at least one given target 2D image of the given 2D images, the given flow map mapping changes to be made to pixels of the given source 2D image in order to align the given source 2D image with the given target 2D image by utilizing the machine learning model for the given source 2D image and the given target 2D image; generating, by the processing circuitry, the registered multispectral data cube using the given source 2D image, the corresponding given flow map, and a remapping function; A method comprising:

51. At least one given training record of the training data set comprises: (A) a three-dimensional (3D) model of a scene, the 3D model having one or more properties and including a set of pixels, at least one of the pixels being associated with (a) a color and (b) a label indicating a material group of which an element in the scene associated with the pixel is composed; (B) a substance database including a list of substances, wherein at least one substance is associated with (i) a spectral reflectance signature vector of the substance, (ii) a typical color of the substance, and (iii) a given substance group of the substance; and (C) a heuristic table including one or more rules, each rule defining a probability of the presence of a given substance in a given scene based on characteristics of the scene; and capturing at least one 2D image from the 3D model of the scene, each 2D image captured from a different viewpoint of the scene, each 2D image comprising a subset of the pixels, and each training 2D image associated with a different wavelength range to obtain the training source 2D image for the given training recording; selecting, for at least one training source 2D image, a training target 2D image from among the 2D images captured from the 3D model to obtain the training target 2D image for the given training recording; For at least one given pixel of the subset of pixels of the training source 2D images: querying said substance database for a list of materials that may be materials of said material group for said given pixel; removing from the list of possible materials those materials whose probability of being present in the 2D image according to the rules of the heuristic table and the characteristics of the 3D model is less than a first threshold, to obtain a revised list of possible materials; determining a matching material for the given pixel from the modified list of possible materials based on a match between the color of the given pixel and the representative color associated with the material from the modified list of possible materials; selecting a corresponding portion of the spectral reflectance signature vector associated with the matching material according to the wavelength of the 2D image to be the portion at the given pixel location in the training source 2D image; and generating a training flow map for the given training recording that maps modifications to be made to pixels of the source 2D image to align the source and target 2D images; generating the training flow map for the given training recording by:

51. The method of claim 50, wherein the compound is generated by performing

52. 51. The method of claim 50, wherein at least one consecutive pair of the 2D images of the scene overlap by more than an overlap threshold.

53. 51. The method of claim 50, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

54. 51. The method of claim 50, wherein the machine learning model further comprises a vision transformer.

55. 51. The method of claim 50, wherein the one or more characteristics of the 3D model include one or more of a geographic area, a sunlight angle, and a viewing distance.

56. A non-transitory computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code comprising: The processing circuitry (A) A machine learning model capable of receiving (a) a source 2D image of the 2D images of the scene and (b) a target 2D image of the 2D images of the scene, and capable of determining a corresponding flow map that maps modifications to be made to pixels of the source 2D image to align the source 2D image with the target 2D image, the machine learning model is trained using a training data set including a plurality of training records, each training record including (i) a training source 2D image, (ii) a training target 2D image, and (iii) one or more training flow maps that map modifications to be made to pixels of the training source 2D image to align the training source 2D image with the training target 2D image; (B) given 2D images of the scene, each given 2D image having a different wavelength range and each image taken from a different viewpoint of the scene; obtaining a determining, by the processing circuitry, at least one given flow map for at least one given source 2D image of the given 2D images and for at least one given target 2D image of the given 2D images, the given flow map mapping changes to be made to pixels of the given source 2D image in order to align the given source 2D image with the given target 2D image by utilizing the machine learning model for the given source 2D image and the given target 2D image; generating, by the processing circuitry, the registered multispectral data cube using the given source 2D image, the corresponding given flow map, and a remapping function; A non-transitory computer-readable storage medium executable by at least one processing circuit of a computer to perform a method including:

57. 1. A system for empirical atmospheric calibration utilizing automatically identified objects of interest in an image of a scene, comprising: (A) a machine learning model capable of receiving the image of the scene and capable of identifying the presence of at least one object among the objects to be automatically identified within the scene, wherein each object to be identified is associated with (i) a location within the scene and (ii) a predetermined representative reflectance spectral signature; (B) the image of the scene; and and utilizing the machine learning model and the image of the scene to identify the presence of at least one object among the objects to be automatically identified in the image of the scene; calibrating an uncalibrated multispectral data cube associated with the scene based on the location and the predetermined representative reflectance spectral signature associated with the object to be automatically identified using an empirical atmospheric calibration process; A system comprising a processing circuit configured to:

58. 58. The system of claim 57, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

59. 58. The system of claim 57, wherein the machine learning model further comprises a vision transformer.

60. 58. The system of claim 57, wherein the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

61. 1. A method for empirical atmospheric calibration utilizing automatically identified objects of interest in an image of a scene, comprising: The processing circuitry (A) a machine learning model capable of receiving the image of the scene and capable of identifying the presence of at least one object among the objects to be automatically identified within the scene, wherein each object to be identified is associated with (i) a location within the scene and (ii) a predetermined representative reflectance spectral signature; (B) the image of the scene; and obtaining a identifying, by the processing circuitry, the presence of at least one object among the objects to be automatically identified in the image of the scene utilizing the machine learning model and the image of the scene; calibrating, by the processing circuitry, an uncalibrated multispectral data cube associated with the scene based on the location and the predetermined representative reflectance spectral signature associated with the object to be automatically identified using an empirical atmospheric calibration process; A method comprising:

62. 62. The method of claim 61, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

63. 62. The method of claim 61 , wherein the machine learning model further comprises a vision transformer.

64. 62. The method of claim 61 , wherein the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

65. A non-transitory computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code comprising: The processing circuitry (A) a machine learning model capable of receiving the image of the scene and capable of identifying the presence of at least one object among the objects to be automatically identified within the scene, wherein each object to be identified is associated with (i) a location within the scene and (ii) a predetermined representative reflectance spectral signature; (B) the image of the scene; and obtaining a identifying, by the processing circuitry, the presence of at least one object among the objects to be automatically identified in the image of the scene utilizing the machine learning model and the image of the scene; calibrating, by the processing circuitry, an uncalibrated multispectral data cube associated with the scene based on the location and the predetermined representative reflectance spectral signature associated with the object to be automatically identified using an empirical atmospheric calibration process; A non-transitory computer-readable storage medium executable by at least one processing circuit of a computer to perform a method including:

66. 1. A system for detecting one or more target materials in an uncalibrated multispectral data cube comprising a set of pixels, comprising: (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and determining, for at least one of the pixels, at least one material indicator indicative of a presence of a given one of target materials at the pixel location, the machine learning model comprising: the machine learning model is trained using a labeled training data set including a plurality of training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) at least one training substance indicator associated with at least one pixel of the uncalibrated training multispectral data cube, the training substance indicator indicating the presence of the target substance at the location of the pixel; (B) the uncalibrated multispectral data cube; and and determining, for at least one pixel of the pixels of the uncalibrated multispectral data cube, at least one material indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated by utilizing an atmospheric simulator that simulates a calibration process and a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube; A system comprising a processing circuit configured to:

67. 67. The system of claim 66, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

68. 67. The system of claim 66, wherein the machine learning model further comprises a vision transformer.

69. 67. The system of claim 66, wherein the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

70. 1. A method for detecting one or more target materials in an uncalibrated multispectral data cube comprising a set of pixels, comprising: The processing circuitry (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and determining, for at least one of the pixels, at least one material indicator indicative of a presence of a given one of target materials at the pixel location, the machine learning model comprising: the machine learning model is trained using a labeled training data set including a plurality of training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) at least one training substance indicator associated with at least one pixel of the uncalibrated training multispectral data cube, the training substance indicator indicating the presence of the target substance at the location of the pixel; (B) the uncalibrated multispectral data cube; and obtaining a determining, by the processing circuitry, for at least one pixel of the pixels of the uncalibrated multispectral data cube, at least one material indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated using a calibration process and an atmospheric simulator that simulates a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube; A method comprising:

71. 71. The method of claim 70, wherein the machine learning model is one or more of a convolutional neural network encoder-decoder model, a fully connected neural network model, a U-Net model, a U-Net++ model, a perceptron model, an inception model, a resnet model, a visual geometry group (VGG) model, an alexnet model, a densenet model, a mobilenet model, or a visual transformer model.

72. 71. The method of claim 70, wherein the machine learning model further comprises a vision transformer.

73. 71. The method of claim 70, wherein the uncalibrated multispectral data cube is captured by one or more multispectral sensors.

74. A non-transitory computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code comprising: The processing circuitry (A) a machine learning model capable of receiving the uncalibrated multispectral data cube and determining, for at least one of the pixels, at least one material indicator indicative of a presence of a given one of target materials at the pixel location, the machine learning model comprising: the machine learning model is trained using a labeled training data set including a plurality of training records, each training record including (i) an uncalibrated training multispectral data cube and (ii) at least one training substance indicator associated with at least one pixel of the uncalibrated training multispectral data cube, the training substance indicator indicating the presence of the target substance at the location of the pixel; (B) the uncalibrated multispectral data cube; and obtaining a determining, by the processing circuitry, for at least one pixel of the pixels of the uncalibrated multispectral data cube, at least one material indicator and a corresponding calibrated multispectral data cube, wherein the corresponding calibrated multispectral data cube is calculated utilizing a calibration process and an atmospheric simulator that simulates a plurality of simulated uncalibrated multispectral data cubes by simulating different atmospheric conditions for the calibrated multispectral cube; A non-transitory computer-readable storage medium executable by at least one processing circuit of a computer to perform a method including:

Images