Method and device for classification of samples using multimodalspectral knowledge distillation

Abstract

The present disclosure relates to a machine-learning training method and device of multimodal spectral knowledge distillation for classification of a target sample based on spectral response to interaction of said sample with light, comprising the steps of: receiving a first training dataset of instances, each instance comprising a first spectral response of a training sample acquired by a sensing device using a first spectroscopy technique, referred to as a teacher technique, and a second spectral response of the training sample acquired by a sensing device using a second spectroscopy technique, referred to as a student technique; clustering and labelling said instances by a first classification model of unsupervised-learning into one or more clusters using the first spectral response; and training a second classification model of supervised-learning with the labelled instances by using as training input the second spectral response and using as training output the cluster label for each said instance, wherein said cluster label is a class for said classification of a target sample.

Description

D E S C R I P T I O NMETHOD AND DEVICE FOR CLASSIFICATION OF SAMPLES USING MULTIMODAL SPECTRAL KNOWLEDGE DISTILLATIONTECHNICAL FIELD

[0001] The present disclosure relates to machine-learning multimodal spectral classification of target analytes or samples.BACKGROUND

[0002] Spectral imaging utilizes spatially referenced spectral signatures to create informative representations of samples and analytes in the form of spectral maps, offering access to information far beyond what traditional RGB images can offer. In the context of spectral imaging a wide range of techniques has been investigated, including Laser Induced Breakdown Spectroscopy (LIBS), Raman spectroscopy, Energy Dispersive X-Ray Spectroscopy and hyperspectral imaging (HSI), which are known to support technology applications in multiple sectors of industry, integrated online in the production cycle or offline for manual quality control.

[0003] For example, LIBS has been particularly successful in the metal industry for material sorting purposes and for process control in metallurgy, while also having promising results in mining, cork and wood sectors. In another example of spectral imaging, HSI is also well- established in a wide range of applications from food safety and quality control to mining. However, the inconsistency of results in complex samples, instrumental drifts, and the lack of interpretability of the data-driven models often hinders its reliability and motivates the pursuit for better diagnostic tools and strategies.

[0004] One of the approaches to increase the potential of individual imaging techniques and overcome its limitations is to combine them in multimodal spectral imaging solutions. Indeed, with the increasing accessibility to such techniques in academic and industrial environments, there has been interest in developing collaborative multimodal solutions, where information from distinct spectroscopy techniques may be combined to enhance the capabilities of the single modality systems. In principle, such approaches of multimodal sensing can be designed to solve some of the problems of the individual techniques, namely increasing the robustness andversatility of the systems, provided suitable methodologies and computational algorithms are developed for that purpose.

[0005] For example, previous attempts in multimodal imaging solutions have tested the combination of Raman and LIBS techniques, and of Raman and HSL For example EP1977205 discloses a method for analyzing a sample comprising the steps of illuminating a first region of the sample with a first illumination pattern, having an energy density sufficient to obtain a plurality of first sample photons from Raman spectroscopy, illuminating a second region of the sample with a second illumination pattern, having an energy density sufficient to obtain a plurality of second sample photons from laser induced breakdown spectroscopy, the first and the second regions of the sample being simultaneously illuminated and further processed.

[0006] US20120038908A1 on the other hand describes a two-step solution for Raman-LIBS system that aims at identifying regions of interest with one technique and performing a second analysis with the other. Similarly, but with distinct techniques, WO2022266779 describes a method of using LIBS technique for geological analysis resorting to hyperspectral cameras that are used to determine a region of interest and complement the data gathered by LIBS. The 2D maps coming from the two distinct spectra readings can be overlayed for further interpretation.

[0007] Other attempts also report on the combination of LIBS and HSI using the principle of data fusion for the analysis of copper concentrates and minerals showing significant improvements in the prediction capability and reproducibility, demonstrating the promising potential of these tandem solutions.

[0008] However, for these data fusion solutions the major limitation is that the results are as robust as the less robust (including but not limited to noise and instrumental drifts) technique utilized. This means that the level of detail and reproducibility of the technique is bounded by the capacities of the equipment used. Similarly, data fusion performance is also bounded in other qualitative aspects such as the speed of operation of the slowest technique, or the resolution of the coarser one. Additionally, the burden of extracting meaning from the combined results often falls on the user. As such, the mere combination of raw data coming from multiple sensors as done in the data fusion approach strongly limits performance of multimodal systems and thus is not effective for technology solutions based on spectral imaging.

[0009] These facts are disclosed in order to illustrate the technical problem addressed by the present disclosure.GENERAL DESCRIPTION

[0010] The present disclosure includes a multimodal spectral knowledge distillation (MSKD) method and system (or interchangeably, device) that uses one spectral imaging technique as an autonomous supervisor (a teacher) for a second spectral imaging technique (a student). The disclosed method and system leverage the higher performance and / or higher degree of interpretability of the teacher technique to first construct a unsupervised model for the classification of target analytes or samples based on its spectral information, subsequently using the prediction labels to construct a dataset to train a second classification model that uses only data from the student technique but trained in supervised manner, thus allowing to distill the knowledge between two spectral imaging modalities.

[0011] The present disclosure discloses an approach to multimodal spectral imaging by utilizing a knowledge distillation framework to deploy a new technique for the classification of unknown samples called Multimodal Spectral Knowledge Distillation (MSKD). The MSKD starts with the training of an unsupervised classification model using one spectroscopy technique (teacher) that provides soft or hard classification labels for each sample that are subsequently used to train the second spectroscopy technique (student). The student technique can thus learn a model with improved performance and / or interpretability benefiting from the use of supervised methodologies with a larger training dataset and / or suitable supervised dimensionality reduction algorithms, thus effectively capitalizing the multimodality through the teacher technique.

[0012] An object of the present disclosure is to provide a method to combine two spectral imaging techniques in a new technique called Multimodal Spectral Knowledge Distillation (MSKD). The method is designed to circumvent disadvantages of the prior art based on data fusion for unknown sample classification, in particular the need of acquiring spectral signals for every sample with each of the techniques at each spatial point, strongly limiting a multimodal spectral imaging solution to the lowest individual qualitative performance of the spectral techniques utilized, in terms but not limited to, speed of operation, spatial resolution, robustness to noise, and instrumental drifts.

[0013] The present disclosure comprises the use of a first spectral imaging technique system (to be called the teacher technique) to build a data-driven model for sample and / or target analyte classification in unsupervised manner. This teacher model is then able to produce autonomous soft (probability distribution over multiple classes) or hard (single class) predictive classification labels for the analyzed samples that can be used as training labels to feed a supervisedclassification algorithm for a second imaging technique (to be called the student technique) data in an autonomous manner. The final result is a second classification model that operates using only data obtained from the student spectral imaging technique but that was trained in supervised manner by knowledge distillation of the model obtained using the teacher spectral imaging technique.

[0014] In an embodiment, two spectral imaging techniques are selected from a plurality of spectroscopy techniques, and the plurality of spectroscopy techniques providing at least two distinct spectral information to define a teacher and student techniques. In this implementation, the technique selected as the teacher technique is the one that allows to construct an unsupervised classification model for the classification of samples providing better performances and / or interpretability - defined in this context as the technique with superior knowledge. The superior knowledge is application-specific and relates to the target analyte and samples intended to be classified. The technique selected as the student is the technique that the user wants to use classify samples in a possible application, which features in principle inferior knowledge to classify correctly the samples using an unsupervised classification model. Different spectral techniques are selected under different application scenarios to maximize efficiency of knowledge distillation practice for a possible implementation.

[0015] Examples of techniques that can be combined according to the present disclosure include Raman spectroscopy and LIBS, Raman spectroscopy and HSI, or LIBS and Laser-induced Fluorescence. These examples are not limiting to the combination of techniques as depending on the context and object under analysis, different techniques may serve as teacher and student technique, as described above.

[0016] In a particular embodiment of the present disclosure a multimodal imaging method combining Laser-induced breakdown spectroscopy (LIBS) and Hyperspectral imaging (HSI) data for mineral identification is disclosed.

[0017] In summary, the present disclosure relates a new approach to multimodal spectral imaging by utilizing a knowledge distillation framework to deploy a new technique for the classification of unknown samples called Multimodal Spectral Knowledge Distillation (MSKD). The MSKD method starts with the training of an unsupervised classification model using one spectroscopy technique (teacher) that provides soft or hard classification labels for each sample that are subsequently used to train the second spectroscopy technique (student). The student technique can thus learn a model with improved performance and / or interpretability benefitingfrom the use of supervised methodologies with a larger training dataset and / or suitable supervised dimensionality reduction algorithms, thus effectively capitalizing the multimodality through the teacher technique.

[0018] In particular, the present disclosure concerns a Multimodal Spectral Knowledge Distillation (MSKD) method for the combination of two imaging techniques, the method comprising the steps of i. choosing a teacher technique and a student technique according to each individual capacity to acquire data and / or interpretability of data for the classification task; ii. obtaining a training dataset comprising spectral signatures acquired from the teacher technique and the student technique for the same sample; iii.use the training dataset of step ii) to construct an unsupervised classification algorithm; iv.obtain classification labels for the spectral signatures of the training dataset of step ii); v.extract the set of features for running a supervised classifier containing a set of spectral features to classify the sample; vi. training the student technique by running the supervised classification algorithm on a computer using the predictions of the first imaging technique as soft labels for the second imaging technique; vii. obtaining a final supervised classification model working with data from the student technique only.

[0019] In particular, step iii) of constructing the unsupervised classification algorithm of the Multimodal Spectral Knowledge Distillation (MSKD) method comprises the steps of: i. Preprocessing the data with the operations of the teacher technique; ii. Extracting a set of features to be utilized for the development of the algorithm of the unsupervised classifier to be run in a computer; iii. Training the unsupervised classification algorithm; iv. Tuning the hyperparameters with a validation dataset.

[0020] In a preferred embodiment the teacher technique used in the Multimodal Spectral Knowledge Distillation (MSKD) method is selected from the group comprising Raman spectroscopy, Laser-induced breakdown spectroscopy LIBS, Hyperspectral imaging HIS (or interchangeably, HSI) and Laser-induced Fluorescence.

[0021] In another preferred embodiment the techniques combined used in the MultimodalSpectral Knowledge Distillation (MSKD) method are selected from the group comprising Ramanspectroscopy as teacher technique and LIBS as student technique, Raman spectroscopy as teacher technique and HSI as student technique, LIBS as teacher technique and Laser-induced Fluorescence as student technique and LIBS as teacher technique and HSI as student technique.

[0022] In a most preferred embodiment, the Multimodal Spectral Knowledge Distillation (MSKD) method combines Laser-induced breakdown spectroscopy (LIBS) and Hyperspectral imaging (HSI), the method comprising the following steps: i. obtaining a training dataset comprising spectral signatures acquired from LIBS technique and from HSI technique for the same sample; ii.use the training dataset of step i) to construct an unsupervised classification algorithm; iii. obtain classification labels for the spectral signatures of the training dataset of step ii); iv.extract the set of features for running a supervised classifier containing a set of spectral features to classify the sample; v.training the HSI technique by running the supervised classification algorithm on a computer using the predictions of LIBS as soft labels for the second imaging technique; vi. obtaining a final supervised classification model working with data from HSI technique only.

[0023] In a further preferred embodiment, the Multimodal Spectral Knowledge Distillation (MSKD) method, the supervised classification algorithm is a Partial Least Squares Discriminant Analysis algorithm or a Multilayer Perceptron Neural Network algorithm.

[0024] The present disclosure also relates to a system for combination of two imaging techniques by Multimodal Spectral Knowledge Distillation (MSKD), the system comprising spectral apparatus for acquisition of spectral signatures from a teacher technique; spectral apparatus for acquisition of spectral signatures from a student technique; a computer to run an unsupervised classification algorithm and a supervised classification algorithm; wherein the unsupervised classification algorithm teaches the supervised classification algorithm so that the student technique runs over the supervised algorithm autonomously.

[0025] In a preferred embodiment, for combination of two imaging techniques by Multimodal Spectral Knowledge Distillation (MSKD), Multimodal Spectral Knowledge Distillation (MSKD) of the present disclosure the teacher technique is selected from the group comprising Raman spectroscopy, LIBS, HIS and Laser-induced Fluorescence.

[0026] In a preferred embodiment, for combination of two imaging techniques by Multimodal Spectral Knowledge Distillation (MSKD), Multimodal Spectral Knowledge Distillation (MSKD) of the present disclosure the student technique is selected from the group comprising Raman spectroscopy, LIBS, HIS and Laser-induced Fluorescence.

[0027] In a most preferred embodiment, for combination of two imaging techniques by Multimodal Spectral Knowledge Distillation (MSKD), Multimodal Spectral Knowledge Distillation (MSKD) of the present disclosure, the techniques combined are selected from the group comprising Raman spectroscopy as teacher technique and LIBS as student technique, Raman spectroscopy as teacher technique and HSI as student technique, LIBS as teacher technique and Laser-induced Fluorescence as student technique and LIBS as teacher technique and HIS as student technique.

[0028] The present disclosure relates to a machine-learning training method of multimodal spectral knowledge distillation for classification of a target sample based on spectral response to interaction of said sample with light, comprising the steps of: receiving a first training dataset of instances, each instance comprising a first spectral response of a training sample acquired by a sensing device using a first spectroscopy technique, referred to as a teacher technique, and a second spectral response of the training sample acquired by a sensing device using a second spectroscopy technique, referred to as a student technique; clustering and labelling said instances by a first classification model of unsupervised-learning into one or more clusters using the first spectral response; and training a second classification model of supervised-learning with the labelled instances by using as training input the second spectral response and using as training output the cluster label for each said instance, wherein said cluster label is a class for said classification of a target sample.

[0029] In an embodiment, said labelling is soft labelling which comprises, for each instance, apportioning a probability of belonging to each of the clusters.

[0030] In an embodiment, said labelling is hard labelling which comprises, for each instance, allocating a single, definitive cluster.

[0031] In an embodiment, an intermediate step of adjusting said clustering and labelling by selecting a number of clusters that minimizes a predetermined loss criterium or criteria, such as minimizing the average silhouette cluster width (ASW), or equivalently maximizing apredetermined scoring function. For ASW in more detail see

[0010] , hereby incorporated in its entirety for the purpose of ASW calculation. Also, selecting a number of clusters that minimizes or minimizing an ASW-related index, or minimizing a cluster validation index to estimate an optimum number of clusters, can be used.

[0032] In an embodiment, the target sample is a mineral and the classification is mineral type, or the target sample is a wood sample and the classification is wood type, or the target sample is a wood sample and the classification is contamination level, or the target sample is a soil sample and the classification is soil type.

[0033] In an embodiment, the first spectroscopy technique and the second spectroscopy technique are independently selected from Raman spectroscopy, LIBS, HIS and Laser-induced Fluorescence.

[0034] In an embodiment, the first spectroscopy technique and the second spectroscopy technique are selected such that a selection parameter is higher for the first spectroscopy technique than for the second spectroscopy technique, in particular the selection parameter being classification knowledge, classification performance, classification knowledge.

[0035] In an embodiment, the first spectroscopy technique is ground-based and the second spectroscopy technique is satellite-based.

[0036] In an embodiment, the first spectroscopy technique and the second spectroscopy technique are, respectively, Raman spectroscopy and Laser-induced Breakdown Spectroscopy - LIBS, Raman spectroscopy and HyperSpectral Imaging - HIS, LIBS and Laser-induced Fluorescence, or LIBS and HIS.

[0037] The present disclosure also relates to a machine-learning classification method of multimodal spectral knowledge distillation for classification of a target sample based on spectral response to interaction of said sample with light, further comprising the step of using the trained second classification model to classify a spectral response of the target sample.

[0038] In an embodiment, the classification of the target sample does not comprise the use of the first spectroscopy technique on the target sample.

[0039] In an embodiment, the first spectroscopy technique and the second spectroscopy technique are imaging techniques and the classification of the target sample comprises the classification of a plurality of imaging areas of an acquired image of the target sample.

[0040] In an embodiment, the machine-learning training method further comprising the step of outputting the trained second classification model.

[0041] In an embodiment, the first classification model is a K-Means model.

[0042] In an embodiment, the second classification model is a Partial Least Squares Discriminant Analysis model or a Multilayer Perceptron Neural Network model.

[0043] It is also disclosed a trained supervised-learning classification model obtained by any of the disclosed methods.

[0044] It is also disclosed a device comprising a computer-readable medium comprising the trained supervised-learning classification model. In an embodiment, the device further comprises the sensing device using the second spectroscopy technique. In an embodiment the device further comprises the sensing device using the first spectroscopy technique.BRI EF DESCRI PTION OF THE DRAWI NGS

[0045] The following figures provide preferred embodiments for illustrating the disclosure and should not be seen as limiting the scope of invention.

[0046] Figure 1: Schematic representation of a typical pipeline of processing steps necessary to train an unsupervised model for sample classification using a single spectral imaging technique (left panel) or two spectral imaging techniques (right panel) using data fusion.

[0047] Figure 2: Schematic representation of an example of data fusion for mineral identification using LIBS and HIS.

[0048] Figure 3: Schematic representation of the workflow and processing pipeline of an embodiment of the Multimodal Spectral Knowledge Distillation methodology proposed.

[0049] Figure 4: Schematic representation of an example of the application of the Multimodal Spectral Knowledge Distillation for the combination of two techniques, namely LIBS and HIS, using LIBS to provide soft labels and train a supervised model taking only HSI data as its input, distilling the knowledge of LIBS to HSI technique.

[0050] Figure 5: Schematic representation of of data obtained using two distinct spectral imaging modalities. From left to right per column - Rock samples and indicative mineral zones. Spectral signature for LIBS and HSI, showcasing typical signature of distinct mineral types.

[0051] Figure 6: Schematic representation of the classification results of an unsupervised methodology for mineral identification using LIBS spectral imaging technique, followed by the class assignment step to the mineral type by leveraging LIBS interpretability, namely associating the presence of lines to the mineral type according to the table on the right-hand side.

[0052] Figure 7: Schematic representation of the results of a typical unsupervised training classification model using HSI only, and results obtained using HSI but trained with LIBS soft labels using the MSKD model.DETAILED DESCRI PTION

[0053] The present disclosure relates to a multimodal spectral knowledge distillation (MSKD) method and system that uses one spectral imaging technique (teacher) as a supervisor for a second spectral imaging technique (student). For the purpose of the present disclosure, teacher technique is synonymous of first technique, supervisor technique and of superior technique; and student technique is synonymous of second technique and inferior technique.

[0054] The present disclosure discloses a method of multimodal spectral imaging that utilizes a knowledge distillation framework to implement a new technique for the classification of unknown samples called Multimodal Spectral Knowledge Distillation (MSKD). The MSKD starts with the training an unsupervised classification model using one spectroscopy technique (teacher) that provides soft or hard classification labels for each sample that are subsequently used to train a second spectroscopy technique (student). The student technique can thus learn a model with improved performance and / or interpretability benefiting from the use of supervised methodologies with a larger training dataset and / or suitable supervised dimensionality reduction algorithms, thus effectively capitalizing the multimodality through the teacher technique.

[0055] The most typical sensor fusion approach to spectral imaging lays on a simple combination of the features extracted from the two techniques to train algorithms for identifying spatial regions of similar chemical content. This approach is interesting from an academic perspective but presents strong limitations for industrial applications. In particular, it may hinder the performance compared to the single modality system. For example, as it requires point-to-point matching of data, it is limited by the operation speed of the slower technique and the lower resolution of the two.

[0056] The multimodal spectral knowledge distillation method and system proposed in the present disclosure enables collaborative sensing of spectral imaging techniques, potentiating the individual advantages of the techniques while circumventing some the drawbacks of sensor fusion. The method of the present disclosure follows a knowledge distillation framework of artificial intelligence, where one model of superior knowledge is used to feed a simpler model with soft predictions, increasing the training dataset of the second model to improve its performance and transfer the knowledge between the first and second models.

[0057] It is well established in the art that a spectral imaging technique is considered of superior knowledge when it surpasses another technique in terms of knowledge and / or interpretability of the obtained spectral data according to the intended classification task. This other technique with less knowledge and / or interpretability is called inferior when compared against the first superior technique. Illustrative but non limiting examples of how a spectral technique can be considered superior in regard of a particular analysis objective is listed below:1. For a classification task comprising the detection of specific chemical elements, spectral techniques that enable the detection of elements (e.g., LIBS, X-Ray Fluorescence, ICP-MS) through the analysis and detection of associated spectral peaks on the obtained data may be considered of superior knowledge when compared against spectral techniques that provide data related with the molecular bands or structure (e.g., Raman Spectroscopy, Hyperspectral Imaging);2. For a classification task comprising the detection of specific molecules, spectral techniques that provide data related with the molecular bands or structure (e.g., Raman Spectroscopy, Hyperspectral Imaging) may be considered of superior knowledge when compared against spectral techniques that enable mostly the detection of elements (e.g., LIBS, X-Ray Fluorescence, ICP-MS).

[0058] Starting from a generic sample classification task to be solved with spectral imaging techniques, the method of the present disclosure comprises the following steps, in sequential order:1. Choosing two spectral imaging techniques from a plurality of spectroscopy techniques, and the plurality of spectroscopy techniques providing at least two distinct sets of spectral data plausible to contain information for classifying the target sample;2. Defining the teacher and student techniques:a. The teacher technique being the technique with superior knowledge over the sample as previously defined, i.e. allowing distinguishing samples with better performances and / or interpretability when used alone; b. The student technique being the technique ultimately intended to be used for sample classification due to its benefits in terms of, but not limited to, operating speed, accessibility, cost and running cost; Obtaining a training dataset, comprising spectral signatures obtained from both teacher and student techniques for the same object, representative of the classification task intended to solve. For the purpose of the present disclosure, a spectral signature is any sensed parameter, acquired by the spectral technique, that directly or indirectly characterizes the sample and consequently is used in the classification task. For example, for a geological sample, the spectral signature may be data acquired by both techniques coming from spatially distributed points of the sample; With the dataset obtained in step 3, the data acquired from the teacher technique is utilized to construct an unsupervised classification methodology following the steps below, and as illustrated in Figure 3, namely by: a. Preprocessing the data with the convenient operations depending on the spectroscopy technique utilized, which may include baseline removal or normalization, amongst others; b. Extracting the convenient set of features to be utilized for the development of the algorithm of the unsupervised classifier to be run in a computer, the algorithm preferably written in Python and / or C++ language, including but not limited to target peak areas, target peak intensity, total intensity or loadings of principal component analysis, chosen from the set of spectral features plausible to contain information capable of solving the classification task; c. Training an unsupervised classification algorithm amongst the class of existing classification algorithms, including but not limited to K-means method; d. Tuning the hyperparameters with a validation dataset; e. Obtaining a final unsupervised classification model working with data from the teacher technique; With the model for the teacher technique trained, the conditions are set for the knowledge distillation step to occur, as illustrated in Figure 3 and as follows:a. Obtaining classification labels for the samples of the training dataset using the previously trained teacher model, i.e. classifying each spatial point into a set of a number of distinct classes (e.g. Class 1, Class 2, etc.); b. Preprocessing the data with the convenient operations depending on the spectroscopy technique utilized. Preprocessing of the data includes but is not limited to baseline removal and / or normalization; c. Extracting the convenient set of features to be utilized for the development of the algorithm of the supervised classifier to be run in a computer, including but not limited to target peak areas, target peak intensity, total intensity or loadings of principal component analysis, chosen from the set of spectral features plausible to contain information capable of solving the classification task; d. Training a supervised classification algorithm using the predicted labels of the first imaging technique as soft labels for the second imaging technique. The algorithm can be selected from the group of existing classification algorithms, including but non-limited to Partial Least Squares Discriminant Analysis or Multilayer Perceptron Neural Network; e. Obtaining a final supervised classification model working with data from the student technique only.

[0059] The method described above allows simultaneously to generate and feed higher volumes of training data to the student technique (second imaging technique) in order to obtain better performances, as it happens in traditional single-modal knowledge distillation. Moreover, the multimodal spectral knowledge distillation framework no longer limits the multimodal solution to the slower speed or lower resolution, as the modality of higher throughput can be trained to achieve maximum performance.

[0060] Once training of the second technique is completed, it can be used following standard procedures of implementation and without additional training or the need to resort to any merging of data coming from the first technique. That is to say that after being trained, the second technique becomes autonomous in daily operational environment.

[0061] Non-limiting examples of the application of the method of the present disclosure include:1. A system for mineral type classification, trained using the combination of LIBS and HSI under the framework of MSKD: using the element detection capabilities of LIBS, one can identify minerals based on the presence / absence of a group of spectral emission lines characteristic of each element and characteristic of each mineral. The resulting unsupervisedmodel can then be used to construct a training dataset to train a classification model using a supervised machine learning algorithm, such as Partial Least Squares Determinant Analysis, for the HSI data, allowing to distill the knowledge of the model obtained with LIBS to the second model that uses only with HSI data, enabling higher processing throughput compared with a sensor fusion solution (as LIBS is typically slower than HSI techniques).2. A system to detect lead contaminated wood samples combining LIBS and HSI, trained under the framework of MSKD: using the element detection capabilities of LIBS, one can scan multiple wood samples to detect the presence of lead in wood samples based on the presence / absence of characteristic lead emission lines, thus allowing to train an unsupervised classification model to classify contaminated samples using as a unique feature the presence / absence of the lead spectral lines. Such interpretable model, trained with LIBS technique, can then be used to train an HSI model in supervised manner, for example a multilayer perceptron neural network, allowing to distill the knowledge of the model obtained with LIBS to a second model to be operated using only HSI data, again for higher processing throughput compared with a sensor fusion solution.3. A system to train remote sensing devices (e.g. hyperspectral cameras on satellites), using a spectroscopy technique of higher accessibility to train an unsupervised classification model, distilling the knowledge by supervised training of a second model using the same training dataset but with the data acquired from remote sensing device.

[0062] The following section describes a non-limiting preferred embodiment that exemplifies how the method can be used in the context of analysis of geological samples.

[0063] The following pertains to an example of an embodiment.

[0064] In a particular embodiment of the present disclosure, a LIBS system is used to train an HSI system for a specific task of mineral classification. Aligned with the proposed methodology, the LIBS data obtained for the training dataset will be first used to train an unsupervised classification algorithm using its superior knowledge and interpretability (e.g., relating elements present with each mineral type). The teacher model is subsequently used to teach a supervised classification algorithm working with HIS-only data obtained for the same samples, using the predictions of the LIBS-based model as soft labels for HSI in a knowledge distillation framework as described previously. This original approach allows feeding higher volumes of training data to the studenttechnique (HSI) and train it in autonomous yet supervised manner, allowing to achieve higher performances using HIS-only data than those obtained with an unsupervised approach.

[0065] This embodiment is not to be perceived as limiting but rather an experimental demonstration of the multimodal imaging system of the disclosure.

[0066] LIBS Imaging for mineral identification

[0067] LIBS is a spectroscopic technique that uses a focused high-intensity laser beam to ablate the sample surface and create an ionized plasma. The discrete lines of the plasma decay spectral signature may then be associated with specific transitions of atomic or ionic species, enabling to gather information on the sample chemical composition. As this element analysis occurs at the focal spot, typically at the sub-millimeter scale, LIBS can be turned into a spectral imaging technique by scanning the sample surface using a whisk broom technique, followed by signal processing and data analysis using suitable numerical routines.

[0068] Focusing on the geological samples, the differences in the characteristic spectral lines may assist in the identification of chemical elements at the sample surface, establishing a connection with the most probable mineral of its chemical composition as done in Capela (2023).

[0069] LIBS imaging features multiple advantages such as high dynamic range (most of the chemical elements can be observed), high sensitivity (most of the time in the ppm range), information redundancy (multiple lines for each element), high spatial resolution (up to micrometer scale), and versatility (remote operation in harsh environments).

[0070] The disadvantage of LIBS technique is that even operating at lKHz (typical systems work up to 100Hz), the whisk broom configuration translates into a slow technique, typically requiring hours to scan larger (e.g., square meter) samples.

[0071] HSI Imaging for mineral identification

[0072] HSI is a technique that gathers spectral data from the reflectance spectra of the target, typically from the visible to the short-wave infrared range (400nm to 3000 nm). In particular, the light radiation at specific wavelengths can be absorbed due to sub-molecular transitions, resulting in bending and stretching of molecular bonds and leading to the appearance of dips in the reflectance spectrum that are called absorption bands. Being associated with specific molecular bonds, different minerals may reveal distinct spectral signatures, thus allowing to perform qualitative mineral identification and analysis such as done in reference Cardoso- Fernandes (2021).

[0073] Compared with LIBS, one of the advantages of HSI is that it can explore a push broom scanning configuration, meaning that the map can be constructed by scanning the sample line by line, resulting in faster acquisition rates, a feature that is crucial for technological applications, from industrial online operations to aerial imaging.

[0074] The drawback of HSI is the lack of interpretability and limited knowledgebase with the information regarding the mineral type under consideration often convoluted and hard to interpret and extract. This means that the deployment of a mineral classification without prior information - i.e., in unsupervised manner - results for most of the cases into low performance models. Therefore, we can say that the HSI technique is plausible to provide information capable of distinguishing samples based on mineral types but the lack of interpretability of such information largely bounds the performance of the system.

[0075] Table 1 summarizes LIBS and HSI general parameters:Table 1: Comparison of typical parameters for LIBS and HSI

[0076] Multimodal Spectral Knowledge Distillation for mineral identification using HSI taught with LIBS

[0077] As presented, the use of LIBS to teach HSI for a mineral classification is a typical scenario which can benefit from the application of the disclosure. Indeed, HSI features a significantly higher throughput that is more compatible with online operation at industry level than LIBS. Yet, contrary to LIBS, it suffers from the drawback of lack of interpretability and convolution of the enclosed information, which strongly limits its performance when deploying an unsupervised classification procedure. The present disclosure includes the use the LIBS as a teacher, leveraging on its superior knowledge and interpretability to deploy an unsupervised classifier model,subsequently used to teach the HSI system in autonomous and supervised manner. This way, the knowledge of LIBS is distilled to HSI thus increasing the HSI performance without losing its higher throughput, thereby demonstrating the method of the application.

[0078] Application of the methodology

[0079] An experiment was conducted to demonstrate the advantageous results of multimodal spectral knowledge distillation technique for the classification of mineral samples. The LIBS system utilized consisted of a Nd:YAG laser, operating at a repetition rate of 1Hz and scanning in a spatial grid at the surface with regular intervals spaced of 1mm, with plasma emission being captured by spectrometers operating in the range of 200-900nm. The HSI data was collected using a Specim SWIR hyperspectral camera covering a range that spans from 1000-2400nm, with a resolution of 384 pixels per line, operated using a conveyor belt solution for spatial scanning operating at a velocity close to lOcm / s.

[0080] For the mineral classification tasks itself, three rock samples obtained from the same mining site were selected, both exhibiting a similar mineralogical composition (Figure 6 and Figure 7, first column). The samples are fragments of a Lithium-rich pegmatite vein located in the Central Iberian Zone of the Iberian Massif, which are known to be mostly composed of Lepidolite, Quartz, Albite, and Mica. One of the rocks was chosen as the training dataset, while the other was selected as the test dataset. Representative spectral signatures of each mineral for both LIBS and HSI are presented in Figure 5. For the application of the multimodal spectral knowledge distillation, we proceed following the steps of the methodology listed above.Choosing two spectral imaging technigues from a plurality of spectroscopy technigues, and the plurality of spectroscopy technigues providing at least two distinct spectral information plausible to contain information capable to classify the target samples;

[0081] For this application LIBS and HIS techniques were selected, for the reasons described above.

[0082] Defining the teacher and student techniques: the teacher technigue being the one with superior knowledge of the sample as previously defined, i.e., allowing distinguishing samples with better performances and / or interpretability when used alone;

[0083] LIBS is selected as the teacher technique, as it can identify minerals according to the presence / absence of chemical elements according to the Table 2.Table 2: Elements expected to be present in each mineralthe student technique, being the technique we ultimately intend to use for sample classification due to its benefits in terms, but not limited to, operating speed, accessibility, cost, running cost;

[0084] HSI is selected as the student technique as its higher throughput is highly desirable for industrial applications and it is plausible that it may contain the necessary information to classify minerals, but not in interpretable manner. obtaining a training dataset, comprising of spectral data obtained with both teacher and student techniques for the same set of spatially distributed points for a set of samples, representative of the task intended to solve;

[0085] For a suitable training set we opted for the rock region depicted in Figure 6, left column. In this region, the minerals appear to be well delimited, which will be instrumental to better interpreting the results qualitatively. For this training set, we acquired spectral images using both techniques as described above. To each point and each acquired spectra, the match between spatial points obtained with distinct techniques was done using a Kabsch-Umeyama algorithm to find a suitable set of transformation parameters for the translation, rotation, and scaling of the datasets. Furthermore, for achieving the point to-point match necessary, the lowest resolution of LIBS imaging is taken as the spatial mesh.With the dataset obtained, the data acquired from the teacher technique is utilized to construct an unsupervised classification methodology following the steps described in Figure 3, namely by:Preprocessing the data with the convenient operations depending on the spectroscopy technique utilized, which may include baseline removal or normalization, amongst others;

[0086] At this stage, each technique requires suitable pre-processing. For example, when using LIBS imaging the obtained signal contains not only the emission lines but also some background that results from Bremsstrahlung and recombination processes (continuous components). As this background has a non-constant distribution that influences emission lines in a non-homogeneous way, its removal is a crucial step to achieve correct line intensities. For the present case it was achieved using an Asymmetrical Least Squares Smoothing algorithm. Furthermore, a spatial Gaussian Filter was also applied to decrease the influence of possible contaminations and edge effects.Extracting the convenient set of features to be utilized for the deployment of the unsupervised classifier, including but not limited to target peak areas, target peak intensity, total intensity or loadings of principal component analysis, chosen from the set of spectral features plausible to contain information capable of solving the classification task;

[0087] LIBS spectrum features variations in spectral line intensities that may correspond to elements present in each mineral, such as Lithium (Li), Silicon (Si), Potassium (K), Sodium (Na), Aluminum (Al) and Rubidium (Rb) and as such, these were the selected lines used for feature extraction, gathering their intensity for each spatial point. Each feature is also scaled to its maximum absolute value, assuring in the process that the shape of the distribution is preserved.Training an unsupervised classification algorithm amongst the class of existing classification algorithms, including but non limited to K-means method;

[0088] Having the training dataset constructed after preprocessing and extracting relevant features from the LIBS-based signals, we are now in conditions to train a classification algorithm in unsupervised manner. A standard K-means algorithm is selected for this purpose. tuning the hyperparameters with a validation dataset;

[0089] The next task of the training stage is to select the number of clusters to be used during training in order to optimize for the best results. The optimal number of clusters was established by analyzing how the total cluster inertia varies with the number of clusters, estimating the ideal cluster number using an empirical elbow method, chosen to be five. obtaining a final unsupervised classification model working with data from the teacher technigue;

[0090] After the completion of the training stage of the teacher model we are in condition to generate predicted labels for each spatial point, classifying rock as observed in Figure 6. At this point, it should be noted that the classification only groups surface zones into mineral types but does not provide information on the nature of the mineral it corresponds to. While not mandatory and unnecessary, an additional label assignment stage can also be performed. For the present example, we performed a calculation of the centroids in the feature space presenting the results into the radar chart for the LIBS feature space (Figure 6). This enables a straightforward correspondence of the cluster with the centroid with non-zero features in the elements of interest - Table 2. For example, the cluster with Li and Rb features can be easily associated with the Lepidolite mineral in this context, which advocates the superior knowledge and interpretability of LIBS spectral imaging for the present case study.With the model for the teacher technigue trained, the conditions are set for the knowledge distillation step described in Figure 3 as following:Obtaining classification labels for the samples of the training dataset;

[0091] Done in previous step.Preprocessing the data with the convenient operations depending on the spectroscopy technigue utilized, which may include baseline removal or normalization, amongst others;

[0092] For preprocessing HSI data we applied a Savitsky Golay filter to remove noise, followed by hull quotient correction to remove the reflectance hull.Extracting the convenient set of features to be utilized for the deployment of the supervised classifier, including but not limited to target peak areas, target peak intensity, total intensity or loadings of principal component analysis, chosen from the set of spectral features plausible to contain information capable of solving the classification task;

[0093] Regarding HSI spectrum, each mineral is no longer associated with specific lines, but rather with distinct bends and dips of the reflectance curves, with the main sources of variability now originating from different slopes and depths of bands in the spectra. Therefore, it is decided to use all available data as features, bypassing this step.Training a supervised classification algorithm using the predictions of the first imaging technigue as soft labels for the second imaging technigue. The algorithm can be selected from the group of existing classification algorithms, including but non-limited to Partial Least Sguares Discriminant Analysis, Multilayer Perceptron Neural Network, amongst others;

[0094] For the present case it was opted for the use of Partial Least Sguares Discriminant Analysis for the supervised learning algorithm to be trained in the HSI feature space using the labels obtained using the teacher model for the training dataset. obtaining a final supervised classification model working with data from the student technigue only;

[0095] After the training procedure, we end up with a second classification model that uses only HSI data but trained in supervised manner, allowing to distill the knowledge between two spectral imaging modalities.

[0096] Results

[0097] The results obtained following the methodology described are presented in Figure 6 and Figure 7.

[0098] For comparison purposes, we added the results for an unsupervised classification model trained with HSI data. For this unsupervised model we have chosen to utilize a conventional principal component analysis (PCA) for dimensionality reduction, selecting the first four principal components that account for an explained variance ratio of 98%. Taking the scores as the extracted features, a standard scaling is then applied, applying the same K-means unsupervised clustering algorithm trained with these features to compare results obtained with and without MSKD (Figure 7, middle and right column).

[0099] The results obtained for the training dataset (as displayed in Figure 6), demonstrate that LIBS and HSI provide very distinct results when using unsupervised methodologies. Results obtained with LIBS are in line with what was expected from the sample figures. Regarding HSI, it is clear that the unsupervised PCA-based method has poor performance, having very littleagreement with mineral regions in the training and test sample. These results align with our prediction of LIBS as the technique with superior knowledge for the present case study.

[0100] Finally, overall results for HSI-based mineral identification using the MSKD methodology can be seen in Figure 7, demonstrating a near perfect classification match in the train dataset when comparing with the teacher model and with the expected mineral types. As such, the result enclosed illustrate well the feasibility and potential of the present disclosure.

[0101] In an embodiment, the multimodal Spectral Knowledge Distillation (MSKD) method for the combination of two imaging techniques, comprises the steps of: i) choosing a teacher technique and a student technique according to each individual capacity to acquire data and / or interpretability of data for the classification task; ii) obtaining a training dataset comprising spectral signatures acquired from the teacher technique and the student technique for the same sample; iii) use the training dataset of step ii) to construct an unsupervised classification algorithm; iv) obtain classification labels for the spectral signatures of the training dataset of step ii); v) extract the set of features for running a supervised classifier containing a set of spectral features to classify the sample; vi) training the student technique by running the supervised classification algorithm on a computer using the predictions of the first imaging technique as soft labels for the second imaging technique; vii) obtaining a final supervised classification model working with data from the student technique only.

[0102] In an embodiment, the step iii) of constructing the unsupervised classification algorithm comprises the steps of: preprocessing the data with the operations of the teacher technique; extracting a set of features to be utilized for the development of the algorithm of the unsupervised classifier to be run in a computer; training the unsupervised classification algorithm; tuning the hyperparameters with a validation dataset.

[0103] In an embodiment, the teacher technique is chosen from the group comprising Raman spectroscopy, LIBS, HSI and Laser-induced Fluorescence.

[0104] In an embodiment, the student technique is chosen from the group comprising Raman spectroscopy, LIBS, HSI and Laser-induced Fluorescence.

[0105] In an embodiment, techniques combined are chosen from the group comprising Raman spectroscopy as teacher technique and LIBS as student technique, Raman spectroscopy as teacher technique and HIS as student technique, LIBS as teacher technique and Laser-induced Fluorescence as student technique and LIBS as teacher technique and HIS as student technique.

[0106] In an embodiment, the Multimodal Spectral Knowledge Distillation (MSKD) method for the combination of Laser-induced breakdown spectroscopy (LIBS) and Hyperspectral imaging (HSI), comprises the steps of: i) obtaining a training dataset comprising spectral signatures acquired from LIBS technique and from HSI technique for the same sample; ii) use the training dataset of step i) to construct an unsupervised classification algorithm; iii) obtain classification labels for the spectral signatures of the training dataset of step ii); iv) extract the set of features for running a supervised classifier containing a set of spectral features to classify the sample; v) training the HSI technique by running the supervised classification algorithm on a computer using the predictions of LIBS as soft labels for the second imaging technique; vi) obtaining a final supervised classification model working with data from HSI technique only.

[0107] In an embodiment, the supervised classification algorithm is a Partial Least Squares Discriminant Analysis algorithm or a Multilayer Perceptron Neural Network algorithm.

[0108] It is also disclosed a system for combination of two imaging techniques by Multimodal Spectral Knowledge Distillation (MSKD), the system comprising: spectral apparatus for acquisition of spectral signatures from a teacher technique; spectral apparatus for acquisition of spectral signatures from a student technique; a computer to run an unsupervised classification algorithm and a supervised classification algorithm; wherein the unsupervised classification algorithm teaches the supervised classification algorithm so that the student technique runs over the supervised algorithm autonomously.

[0109] In an embodiment, the teacher technique is chosen from the group comprising Raman spectroscopy, LIBS, HIS and Laser-induced Fluorescence.

[0110] In an embodiment, the student technique is chosen from the group comprising Raman spectroscopy, LIBS, HSI and Laser-induced Fluorescence.

[0111] In an embodiment, the techniques combined are chosen from the group comprising Raman spectroscopy as teacher technique and LIBS as student technique, Raman spectroscopy as teacher technique and HSI as student technique, LIBS as teacher technique and Laser-induced Fluorescence as student technique and LIBS as teacher technique and HIS as student technique.

[0112] Flow diagrams of particular embodiments of the presently disclosed methods are depicted in figures. The flow diagrams illustrate the functional information one of ordinary skill in the art requires to perform said methods required in accordance with the present disclosure.

[0113] It will be appreciated by those of ordinary skill in the art that unless otherwise indicated herein, the particular sequence of steps described is illustrative only and can be varied without departing from the disclosure. Thus, unless otherwise stated the steps described are so unordered meaning that, when possible, the steps can be performed in any convenient or desirable order.

[0114] In the foregoing the preferred embodiments have been described and variants of the present invention have been suggested, but it should be understood that those skilled in the art will be able to make modifications and changes without thereby departing from the scope of protection as defined by the attached claims.

[0115] The term "comprising" whenever used in this document is intended to indicate the presence of stated features, integers, steps, components, but not to preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. The disclosure should not be seen in any way restricted to the embodiments described and a person with ordinary skill in the art will foresee many possibilities to modifications thereof. The above described embodiments are combinable. The following claims further set out particular embodiments of the disclosure.

[0116] Bibliography[1] S. Zhao, W. Song, Z. Hou, Z. Wang, Classification of ginseng according to plant species, geographical origin, and age using laser induced breakdown spectroscopy and hyperspectral imaging, Journal of Analytical Atomic Spectrometry 36 (2021) 1704-1711.[2] Y. Liu, S. Zhao, X. Gao, S. Fu, C. Song, Y. Dou, S. Song, C. Qi, J. Lin, Combined laser-induced breakdown spectroscopy and hyperspectral imaging with machine learning for the classification and identification of rice geographical origin, RSC advances 12 (2022) 34520-34530.[3] R. Fuentes, D. Luarte, C. Sandoval, A. K. Myakalwar, J. Alvarez, J. Yanez, D. Sbarbaro, Laser- induced breakdown spectroscopy and hyperspectral imaging data fusion for improved mineralogical analysis of copper concentrates, IFAC-PapersOnLine 55 (2022) 85-90.[4] S. Gupta, J. Hoffman, J. Malik, Cross modal distillation for supervision transfer, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2827-2836.[5] Generically, and aligned with established literature [ref - Zhao, Long, et al. "Knowledge as priors: Cross-modal knowledge generalization for datasets without superior knowledge." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020],[6] https: / / www.sciencedirect.com / science / article / pii / S0584854723001209, consulted on 17- 08-2023.[7] Capela, Diana, et al. "Robust and interpretable mineral identification using laser-induced breakdown spectroscopy mapping." Spectrochimica Acta Part B: Atomic Spectroscopy (2023): 106733.[8] Cardoso-Fernandes, Joana, et al. "Interpretation of the reflectance spectra of lithium (Li) minerals and pegmatites: A case study for mineralogical and lithological identification in the Fregeneda-Almendra Area." Remote Sensing 13.18 (2021): 3688.[9] Lopes, Tomas, et al. "Multimodal approach to mineral identification: merging Laser-induced breakdown spectroscopy with Hyperspectral imaging." Journal of Physics: Conference Series. Vol. 2407. No. 1. IOP Publishing, 2022.

[0010] Rousseeuw, P.J., 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53-65.

Claims

C L A I M S1. Machine-learning training method of multimodal spectral knowledge distillation for classification of a target sample based on spectral response to interaction of said sample with light, comprising the steps of: receiving a first training dataset of instances, each instance comprising a first spectral response of a training sample acquired by a sensing device using a first spectroscopy technique, referred to as a teacher technique, and a second spectral response of the training sample acquired by a sensing device using a second spectroscopy technique, referred to as a student technique; clustering and labelling said instances by a first classification model of unsupervised-learning into one or more clusters using the first spectral response; and training a second classification model of supervised-learning with the labelled instances by using as training input the second spectral response and using as training output the cluster label for each said instance, wherein said cluster label is a class for said classification of a target sample.

2. Method according to the previous claim wherein said labelling is soft labelling which comprises, for each instance, apportioning a probability of belonging to each of the clusters.

3. Method according to claim 1 wherein said labelling is hard labelling which comprises, for each instance, allocating a single cluster.

4. Method according to the previous claim comprising an intermediate step of adjusting said clustering and labelling by selecting a number of clusters that minimizes a predetermined loss criterium or criteria, in particular selecting a number of clusters that minimizes average silhouette cluster width, ASW.

5. Method according to any of the previous claims wherein the target sample is a mineral and the classification is mineral type, or the target sample is a wood sample and the classification is wood type, or the target sample is a wood sample and the classification is contamination level, or the target sample is a soil sample and the classification is soil type.

6. Method according to any of the previous claims wherein the first spectroscopy technique and the second spectroscopy technique are independently selected from Raman spectroscopy, LIBS, HIS and Laser-induced Fluorescence.

7. Method according to any of the previous claims wherein the first spectroscopy technique and the second spectroscopy technique are selected such that a selection parameter is higher for the first spectroscopy technique than for the second spectroscopy technique, in particular the selection parameter being classification knowledge, classification performance, classification knowledge.

8. Method according to the previous claim wherein the first spectroscopy technique is ground- based and the second spectroscopy technique is satellite-based.

9. Method according to any of the previous claims wherein the first spectroscopy technique and the second spectroscopy technique are, respectively, Raman spectroscopy and Laser- induced Breakdown Spectroscopy - LIBS, Raman spectroscopy and HyperSpectral Imaging - HSI, LIBS and Laser-induced Fluorescence, or LIBS and HSL10. Machine-learning classification method of multimodal spectral knowledge distillation for classification of a target sample based on spectral response to interaction of said sample with light, according to the previous claim further comprising the step of: using the trained second classification model to classify a spectral response of the target sample.

11. Method according to the previous claim wherein the classification of the target sample does not comprise the use of the first spectroscopy technique on the target sample.

12. Method according to any of the previous claims wherein the first spectroscopy technique and the second spectroscopy technique are imaging techniques and the classification of the target sample comprises the classification of a plurality of imaging areas of an acquired image of the target sample.

13. Method according to any of the claims 1-9 further comprising the step of outputting the trained second classification model.

14. Method according to any of the previous claims wherein the first classification model is a K- Means model.

15. Method according to any of the previous claims wherein the second classification model is a Partial Least Squares Discriminant Analysis model or a Multilayer Perceptron Neural Network model.

16. Trained supervised-learning classification model obtained by the method of the previous claim.

17. Device comprising a computer-readable medium comprising the trained supervised-learning classification model of the previous claim.

18. Device according to the previous claim further comprising the sensing device using the second spectroscopy technique.

19. Device according to the previous claim further comprising the sensing device using the first spectroscopy technique.

20. Computer configured to carry out the method of any of the claims 1-15.

21. Computer-readable medium comprising computer program instructions that when executed by a computer cause it to carry out the method of any of the claims 1-15.

Images