Multimodal machine learning system and method

Abstract

A multimodal machine learning system is disclosed for generating a multimodal input dataset. The system includes an encoder layer and a fusion layer. The encoder layer contains at least one unimodal analysis model for analyzing a plurality of data points from at least one full-spectral image to generate a decision value for the at least one full-spectral image. The fusion layer fuses the decision value of the at least one full-spectral image with at least one non-spectral data set to generate the multimodal input dataset, wherein the at least one non-spectral data set includes at least one clinical data set.

Description

Technical Field

[0001] This application relates to a multimodal machine learning system and method. Prior Technology

[0002] For some diseases, such as pancreatic ductal adenocarcinoma (PDAC), approximately 80% to 90% of patients are diagnosed at an advanced stage, and there is currently a lack of reliable early detection methods for these diseases.

[0003] Currently, machine learning models are used to assist in the early detection of diseases. However, most of these models only use a single type of input data as the basis for analysis to output detection results, resulting in insufficient accuracy. In addition, current machine learning models face great difficulty in analyzing different types of data simultaneously. For example, if one type of data has a few features while another type has a large number of features, it is difficult to analyze both types of data simultaneously using machine learning models.

[0004] Therefore, a novel multimodal machine learning system and method are needed to improve the above problems. Summary of the Invention

[0005] One objective of this application is to provide a multimodal machine learning system (second architecture) for generating a multimodal input dataset. The system includes an encoder layer and a fusion layer. The encoder layer includes at least one unimodal analysis model for analyzing a plurality of data points from at least one full-spectrum image to generate a decision value for the at least one full-spectrum image. The fusion layer is used to fuse the decision value from the at least one full-spectrum image with at least one non-spectral data to generate a multimodal input dataset, wherein the at least one non-spectral data includes at least one clinical data or protein data; wherein the multimodal input dataset is used as input data for a final decision layer, and the final decision layer analyzes the multimodal input dataset to generate a final decision value.

[0006] Another objective of this application is to provide a multimodal machine learning method. This method is implemented through a multimodal machine learning system (first architecture) to generate a multimodal input dataset. The multimodal machine learning system includes an encoder layer and a fusion layer. The method includes the steps of: analyzing a plurality of data points of at least one full-spectrum image using at least one unimodal analysis model of the encoder layer to generate a decision value for at least one full-spectrum image; and fusing the decision value of the at least one full-spectrum image with at least one non-spectral data point using the fusion layer to generate a multimodal input dataset, wherein the at least one non-spectral data point includes at least one clinical data point or protein data point; wherein the multimodal input dataset is used as input data for a final decision layer, and the final decision layer analyzes the multimodal input dataset to generate a final decision value.

[0007] Another objective of this application is to provide a multimodal machine learning system (second architecture) comprising an encoder layer, a fusion layer, and a final decision layer. The encoder layer includes at least one unimodal analysis model for analyzing a plurality of data points from at least one full-spectrum image to generate a decision value for the at least one full-spectrum image. The fusion layer is used to fuse the decision value from the at least one full-spectrum image with at least one non-spectral data to generate a multimodal input dataset, wherein the at least one non-spectral data includes at least one clinical data or protein data. The final decision layer includes a multimodal analysis model for analyzing the multimodal input dataset to generate a final decision value.

[0008] Another objective of this application is to provide a multimodal machine learning method. This method is implemented through a multimodal machine learning system (second architecture), which includes an encoder layer, a fusion layer, and a final decision layer. The method includes the steps of: analyzing a plurality of data points of at least one full-spectrum image using at least one unimodal analysis model in the encoder layer to generate a decision value for at least one full-spectrum image; fusing the decision value of the at least one full-spectrum image with at least one non-spectral data point using the fusion layer to generate a multimodal input dataset, wherein the at least one non-spectral data point includes at least one clinical data point or protein data; and analyzing the multimodal input dataset using a multimodal analysis model in the final decision layer to generate a final decision value.

[0009] Another objective of this application is to provide a multimodal machine learning system (third architecture) comprising a preprocessing module, a fusion layer, and a final decision layer. The preprocessing module performs a preprocessing procedure on at least one full-spectrum image to form at least one preprocessed full-spectrum image. The fusion layer is used to fuse at least a portion of the features of the preprocessed full-spectrum image with at least one non-spectral data to generate a multimodal input dataset, wherein the non-spectral data includes at least one clinical data or protein data. The final decision layer includes a multimodal analysis model for analyzing the multimodal input dataset to generate a final decision value.

[0010] Another objective of this application is to provide a multimodal machine learning method, executed through a multimodal machine learning system (third architecture), the multimodal machine learning system comprising a preprocessing module, a fusion layer, and a final decision layer, the method comprising the steps of: performing a preprocessing procedure on at least one full-spectrum image by the preprocessing module to form at least one preprocessed full-spectrum image; fusing at least a portion of the features of the preprocessed full-spectrum image with at least one non-spectral data by the fusion layer to generate a multimodal input dataset, wherein the non-spectral data includes at least one clinical data or protein data; and analyzing the multimodal input dataset by a multimodal analysis model of the final decision layer to generate a final decision value. Simple Explanation of the Diagram

[0011] Figure 1 is a system architecture diagram of a multimodal machine learning system according to an embodiment of this application. Figure 2 is a flowchart of the steps of a multimodal machine learning method according to an embodiment of this application. Figure 3A is a schematic diagram of the operation process of a multimodal machine learning system according to an embodiment of this application. Figure 3B is a schematic diagram of the training and operation flow of a multimodal machine learning system according to an embodiment of this application. Figure 3C is a schematic diagram of the data used in the training process of a multimodal machine learning system according to an embodiment of this application. Figures 4A to 4D are performance diagrams of multimodal machine learning systems according to various embodiments of this application (different multimodal input data sets). Figures 5A to 5D are performance diagrams of multimodal machine learning systems according to various embodiments of this application (different feature extraction layers 10). Figure 6 is a schematic diagram of the operation process of a multimodal machine learning system and a full-spectrum image acquisition system according to an embodiment of this application. Figure 7 is a system architecture diagram of a multimodal machine learning system according to an embodiment of this application. Figure 8 is a schematic diagram of the operation of a multimodal machine learning system according to an embodiment of this application. Figure 9 is a flowchart of the operation process of a preprocessing module according to an embodiment of this application. Figure 10 is a flowchart of the operation process of the feature selection layer according to an embodiment of this application. Figure 11 is a flowchart of the operation process of a clinical data screening module according to an embodiment of this application. Figure 12 is a schematic diagram of the operation process of a multimodal machine learning system and a full-spectrum image acquisition system according to an embodiment of this application. Implementation

[0012] The exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. Whenever possible, the same element symbols are used in the drawings and description to denote the same or similar parts.

[0013] Throughout this application, certain terms are used to refer to specific components. Those skilled in the art will understand that sensing device manufacturers may use different names to refer to the same components. This document is not intended to distinguish between components that have the same function but different names. In the following description and claims, words such as "containing," "comprising," and "including" are open-ended terms and should therefore be interpreted as "containing but not limited to...".

[0014] The terms "approximately," "substantially," or "roughly" are generally interpreted as being within 10% of a given value or range, or within 5%, 3%, 2%, 1%, or 0.5% of a given value or range. Furthermore, unless otherwise specified, the term "a value between a first value and a second value" is generally interpreted as "the first value ≤ the value ≤ the second value."

[0015] The ordinal numbers used in the specification and claims, such as "first" and "second," to modify elements do not inherently imply or represent any prior ordinal number for that element (or those elements), nor do they represent the order of one element with another, or the order of manufacturing processes. The use of these ordinal numbers is solely to clearly distinguish one named element from another with the same name. The claims and specification may not use the same terminology; therefore, a first element in the specification may be a second element in the claims.

[0016] In this application, the terms "given range is from the first value to the second value" and "given range falls within the range of the first value to the second value" indicate that the given range includes the first value, the second value, and other values ​​in between.

[0017] It should be understood that the features described below can be replaced, recombined, or mixed in several different embodiments to complete other embodiments without departing from the spirit of this application. Features between embodiments can be arbitrarily mixed and combined as long as they do not violate the spirit of the invention or conflict with it.

[0018] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It is understood that these terms, for example, as defined in commonly used dictionaries, should be interpreted as having a meaning consistent with the relevant art and the background or context of this application, and should not be interpreted in an idealized or overly formal manner, unless specifically defined in the embodiments of this application.

[0019] Furthermore, the term "adjacent" in the specification and claims is used to describe objects that are close to each other, and there may be contact or no contact between adjacent objects.

[0020] Furthermore, descriptions such as "when..." or "...when" in this application indicate states such as "at present, before, or after," and are not limited to simultaneous occurrences; this is stated in advance. Descriptions such as "set on..." in this application indicate the corresponding positional relationship between two elements, and do not limit whether the two elements are in contact, unless specifically limited; this is stated in advance. Moreover, when this application describes multiple functions, the use of the word "or" between functions indicates that the functions can exist independently, but does not preclude the simultaneous existence of multiple functions.

[0021] Figure 1 is a system architecture diagram of a multimodal machine learning system 1 according to an embodiment of this application. In one embodiment, the multimodal machine learning system 1 may have a first architecture, which includes a feature selection layer 10, an encoder layer 20, a fusion layer 30, and a final decision layer 40. The multimodal machine learning system 1 with the first architecture can fuse multiple different types of input data to form a multimodal input data set, analyze the multimodal input data set, and then output a final decision value. In one embodiment, the final decision value may be, for example, an auxiliary judgment information, which is related to whether a patient has PDAC or the probability of having PDAC; it should be noted that the final decision value of this application may also be other auxiliary judgment information, and is not limited to the judgment of PDAC. For ease of explanation, the following examples all use the final decision value output by the multimodal machine learning system 1 as being related to whether a patient has PDAC.

[0022] In another embodiment, the multimodal machine learning system 1 may have a second architecture, which includes an encoder layer 20 and a fusion layer 30, and may optionally include a feature selection layer 10. In this case, the final decision layer 40 may be located outside the multimodal machine learning system 1 of the second architecture, for example, in the cloud or on other electronic devices, and is not limited thereto. The multimodal machine learning system 1 of the second architecture can be used to generate a multimodal input data set, which can be transmitted to the final decision layer 40 and used as input data for the final decision layer 40, and is not limited thereto.

[0023] To make this application clear, the following description uses the first architecture multimodal machine learning system 1 (including feature selection layer 10, encoder layer 20, fusion layer 30 and final decision layer 40) as an example.

[0024] In one embodiment, the multimodal input data set may include one or more full-spectrum images of the patient and one or more non-spectral data. Details of the full-spectrum images and non-spectral data will be described below first.

[0025] Regarding full-spectrum images: In one embodiment, a full-spectrum image may be, for example, a spectral image obtained using a spectrometer. Each full-spectrum image may contain a large number of data points (e.g., hundreds, thousands, tens of thousands, hundreds of thousands, or more data points, and is not limited thereto), and each data point may be considered a feature, but is not limited thereto. In one embodiment, each full-spectrum image may be associated with a nuclear magnetic resonance (NMR) image of a patient's metabolic fingerprint, and is not limited thereto. The process of obtaining full-spectrum images will be described in subsequent paragraphs.

[0026] Regarding non-spectral data. In one embodiment, the type of non-spectral data may include one or more clinical data of the patient, which may include age, BMI, sex, smoking status (e.g., whether or not one smokes, duration of smoking, or smoking history, but not limited thereto), tumor marker index (CA19-9), glycated hemoglobin index (HbA1c), postprandial blood glucose index (GluAC), and carcinoembryonic antigen index (CEA), or any combination thereof, but not limited thereto.

[0027] In one embodiment, the non-spectral data may include data on one or more proteins of the patient, such as platelet-reactive protein-2 (TSP-2) and / or activin A, but is not limited thereto. In one embodiment, the serum concentrations of TSP-2 and / or Activin A can be measured using commercially available sandwich enzyme-linked immunosorbent assay kits, such as DTSP20 and DAC00B, but are not limited thereto. In one embodiment, the absorbance of TSP-2 at a wavelength of approximately 450 nanometers can be measured using, for example, a microplate spectrophotometer, where the absorbance is measured in ng / ml. Furthermore, the patient's serum sample may be diluted twofold to facilitate the measurement of Activin A, and then the absorbance of Activin A at a wavelength of approximately 450 nanometers can be measured again using a microplate spectrophotometer. The data for Activin A are expressed in pg / mg, and the data for Activin A can be standardized to form the total protein concentration in serum, expressed in mg / mL.

[0028] In one embodiment, measurements of non-spectral clinical data and / or protein data can be presented using means and standard deviations. Furthermore, clinical and protein data can be analyzed using statistical methods provided by GraphPad Prism 10.0. In one embodiment, TSP-2, CA19-9, and CEA levels in serum samples can be analyzed using the area under the curve (ROC-AUC) of MedCalc 22.023 statistical software to differentiate patient types (e.g., presence or absence of PDAC), but are not limited thereto. In one embodiment, the cutoff value for the TSP-2 biomarker can be determined by calculating the Youden index or achieving approximately 100% specificity. In one embodiment, significant differences can be calculated using the Student's T-test method, where the p-value is less than 0.05.

[0029] One of the features of this application is that the encoder layer 20 and fusion layer 30 of the multimodal machine learning system 1 can fuse full-spectral images with a large number of features with non-spectral data with a small number of features, and use the fused data set as input data for machine learning, thereby improving the accuracy of the analysis of the multimodal machine learning system 1. Existing technologies do not have the ability to fuse data with a large number of features with data with a small number of features.

[0030] The details of the multimodal machine learning system 1, feature selection layer 10, encoder layer 20, fusion layer 30 and final decision layer 40 will be explained next.

[0031] Regarding the multimodal machine learning system 1. In one embodiment, the multimodal machine learning system 1 may be, for example, a processor or an electronic device equipped with a processor. In another embodiment, the multimodal machine learning system 1 may also be, for example, one or more software or firmware, and is not limited thereto. In one embodiment, the feature selection layer 10, encoder layer 20, fusion layer 30, and final decision layer 40 may be, for example, functional modules whose functions can be implemented by the processor executing instructions in one or more computer program products, wherein the computer program products may be, for example, software or firmware, and may be stored in a non-transitory computer-readable medium, such as memory, hard disk, optical disk, USB flash drive, cloud hard drive, etc., and is not limited thereto.

[0032] Regarding feature selection layer 10: In one embodiment, feature selection layer 10 can be used to select a plurality of the most representative features from a large number of features (e.g., data points) in a full-spectrum image, thereby filtering out features with low relevance to the final decision and reducing the amount of data for subsequent processing. In one embodiment, feature selection layer 10 can perform feature selection in various suitable ways, such as the SelectBest algorithm or recursive feature elimination (RFE) algorithm, and is not limited thereto. In one embodiment, feature selection layer 10 can eliminate duplicate features through an iteration, and is not limited thereto.

[0033] Regarding encoder layer 20: In one embodiment, encoder layer 20 may include at least one unimodal analysis model 21. For each full-spectrum image, unimodal analysis model 21 may analyze a plurality of data points in the full-spectrum image to generate a decision value corresponding to the full-spectrum image. Here, the "decision value" is, for example, a radial basis function kernel value (RBF kernel value), but is not limited thereto. In one embodiment, unimodal analysis model 21 may be, for example, a support vector machine (SVM), but in another embodiment, unimodal analysis model 21 may be, for example, a neural network, such as a convolutional neural network or a random forest model, and is not limited thereto. In one embodiment, encoder layer 20 may simultaneously possess multiple unimodal analysis models 21, and each unimodal analysis model 21 may be used to analyze different types of full-spectrum images of the patient, such as different types of metabolic fingerprints, and is not limited thereto. In one embodiment, the data points of the full-spectrum image may be standardized using the StandardScaler function before entering encoder layer 20, and is not limited thereto. In one embodiment, the support vector machine may, for example, employ the scikit-learn suite (version 1.2.2), but is not limited thereto.

[0034] In one embodiment, the unimodal analysis model 21 is configured, for example, to analyze whether a patient has PDAC, but does not output whether the patient has PDAC. Instead, it outputs a decision value that corresponds to the feature in the full-spectrum image with the patient's PDAC status. For example, after analyzing the feature in the full-spectrum image, the unimodal analysis model 21 determines the correlation between the feature and PDAC and calculates a score as the decision value. If the decision value falls within a certain range, it indicates that the patient does not have PDAC (or has a low probability of having it). If the decision value falls within another range, it indicates that the patient has PDAC (or has a high probability of having it). Alternatively, the decision value may fall within a range that cannot be determined. In other words, the decision value here represents a score indicating the correlation between the feature of the full-spectrum image and PDAC, rather than a judgment result indicating whether the patient has PDAC.

[0035] The method of obtaining the decision values ​​will be further explained next. In one embodiment, the unimodal analysis model 21 itself has the function of outputting decision values. In another embodiment, the user can obtain the decision values ​​by decomposing the program of the unimodal analysis model 21 or by providing a subroutine. This application is not limited thereto.

[0036] Accordingly, through the operation of encoder layer 20, a large number of features (e.g., multiple data points) of each full-spectrum image can be transformed into a single feature (e.g., a single decision value).

[0037] Regarding fusion layer 30: In one embodiment, fusion layer 30 can be used to fuse decision values ​​from a patient's full-spectrum image with non-spectral data, thereby generating a multimodal input dataset. In one embodiment, fusion layer 30 can fuse decision values ​​from the full-spectrum image with non-spectral data using concatenation, but is not limited thereto. Here, "concatenation" is, for example, an operation in the field of machine learning used to connect features or information of different levels, channels, or types to form new features, but is not limited thereto. In one embodiment, non-spectral data can be standardized using the StandardScaler function before fusion, but is not limited thereto. In one embodiment, when fusion layer 30 fuses decision values ​​from the full-spectrum image with non-spectral data, data with missing values ​​can be removed to avoid affecting the analysis process.

[0038] Regarding the final decision layer 40: In one embodiment, the final decision layer 40 may include a multimodal analysis model 41 for analyzing the multimodal input dataset to produce a final decision value. In one embodiment, the final decision value is, for example, a binary outcome related to whether the patient has PDAC, such as a "yes" or "no" outcome, but is not limited thereto. In one embodiment, the multimodal analysis model 41 may be, for example, a support vector machine; in another embodiment, the multimodal analysis model 41 may be, for example, a neural network (e.g., but not limited to, convolutional neural networks or random forest models), but is not limited thereto. In one embodiment, the support vector machine may, for example, employ the scikit-learn suite (version 1.2.2), but is not limited thereto.

[0039] Based on the above features, the multimodal machine learning system 1 of this application can execute a multimodal machine learning method. Figure 2 is a flowchart of the steps of a multimodal machine learning method according to an embodiment of this application, and please also refer to Figure 1.

[0040] As shown in Figure 2, firstly, step A1 is executed, where the multimodal machine learning system 1 acquires at least one full-spectrum image of a patient. For example, a user can input the full-spectrum image into the multimodal machine learning system 1, or the instrument that generated the full-spectrum image can transmit the data to the multimodal machine learning system 1, and this is not limited to these methods. Next, step A2 is executed, where the feature selection layer 10 filters a plurality of data points from the at least one full-spectrum image and provides the filtered data points to the encoder layer 20. This step is used to reduce the number of features and can be selectively executed or not. Next, step A3 is executed, where at least one unimodal classification model 21 of the encoder layer 20 analyzes the filtered data points from the at least one full-spectrum image to generate at least one decision value. The details of this step can be found in the description of the encoder layer 20 in the preceding paragraph. Next, step A4 is executed, where the fusion layer 30 fuses the at least one decision value with the patient's non-spectral data to generate a multimodal input data set. The details of this step can be found in the description of the fusion layer 30 in the preceding paragraph. Next, step A5 is executed, where the multimodal classification model 41 of the final decision layer 40 analyzes the multimodal input data set to generate the final decision value. The details of this step can be found in the description of the final decision layer 40 in the preceding paragraph. Thus, the multimodal machine learning method can be understood.

[0041] The operation of the multimodal machine learning system 1 will now be illustrated with an example. Figure 3A is a schematic diagram of the operation of the multimodal machine learning system 1 according to an embodiment of this application, and please refer to Figures 1 and 2 simultaneously. In the example of Figure 3A, the encoder layer 20 may have two single-modal analysis models 21A and 21B. Two types of metabolic fingerprint full-spectrum images of the patient are input into the multimodal machine learning system 1, namely NMR-M spectrum and NMR-M+L spectrum, wherein single-modal analysis model 21A is used to analyze the NMR-M spectrum, and single-modal analysis model 21B is used to analyze the NMR-M+L spectrum.

[0042] As shown in Figure 3A, firstly, the NMR-M spectrum and the NMR-M+L spectrum are input into the multimodal machine learning system 1 (corresponding to step A1 in Figure 2). The NMR-M spectrum has a large number of data points (e.g., but not limited to 131,072), and the NMR-M+L spectrum also has a large number of data points (e.g., but not limited to 131,072). Next, the feature extraction layer 10 performs feature selection on the NMR-M spectrum and the NMR-M+L spectrum (corresponding to step A2 in Figure 2). The NMR-M spectrum uses the SelectBest algorithm for feature selection, where a plurality of data points (e.g., but not limited to 228) are selected. The NMR-M+L spectrum uses the RFE algorithm for feature selection, where a plurality of data points (e.g., but not limited to 12,945) are selected. In one embodiment, feature selection can be performed, for example, by iterating (e.g., but not limited to 4,315 times) to eliminate duplicate features, thereby reducing the number of features. Next, the single-modal analysis model 21A of encoder layer 20 analyzes the plurality of data points (e.g., but not limited to 228) of the NMR-M spectrum to generate one decision value corresponding to the NMR-M spectrum, while the single-modal analysis model 21B of encoder layer 20 analyzes the plurality of data points (e.g., but not limited to 12945) of the NMR-M+L spectrum to generate one decision value corresponding to the NMR-M+L spectrum (corresponding to step A3 in Figure 2). Next, fusion layer 30 fuses the decision value corresponding to the NMR-M spectrum, the decision value corresponding to the NMR-M+L spectrum, the patient's clinical data, and protein data to generate a multimodal input dataset (corresponding to step A4 in Figure 2). Next, the multimodal classification model 41 of final decision layer 40 analyzes the multimodal input dataset and outputs a final decision value, which corresponds to, but is not limited to, the outcome of whether the patient has pancreatic cancer. Thus, the operation of the multimodal machine learning system 1 and the multimodal machine learning method can be understood.

[0043] To enable the multimodal machine learning system 1 to analyze whether a patient has pancreatic cancer, it needs to be trained through a training process. Figure 3B is a schematic diagram of the training and operation flow of the multimodal machine learning system 1 according to an embodiment of this application. Please refer to Figures 1 to 3A simultaneously. As shown in Figure 3B, the untrained multimodal machine learning system 1 will be trained through the training process. After the training process is completed, the untrained multimodal machine learning system 1 is transformed into a trained multimodal machine learning system 1 and can execute its operation flow.

[0044] Next, the details of the training process of the multimodal machine learning system 1 will be described. Figure 3C is a schematic diagram of the data used in the training process of the multimodal machine learning system 1 according to an embodiment of this application. Please also refer to Figures 1 to 3B. As shown in Figure 3C, in one embodiment, the serum metabolic fingerprints, clinical data, and protein data of multiple (e.g., about 902) participants are used in the training process of the multimodal machine learning system 1. Some of these participants are control groups who have not yet developed PDAC (which may be high-risk individuals, or high-risk control (HRC), or healthy individuals, or health control (HC)), while the other part consists of PDAC patients at different stages. In one embodiment, of the data from the multiple participants (e.g., about 902), a portion of the data (e.g., about 350) can be used as the training set and cross-validation set in the training process of the unimodal analysis model 21A, unimodal analysis model 21B, and / or multimodal analysis model 41, while the other data can be used as the test set, but is not limited thereto. More specifically, in one embodiment, a portion of the data (e.g., approximately 350 participants) can be considered an exploratory cohort (training set), containing data on HRC patients and patients with PDAC stages I to IV. This exploratory cohort is then further divided into a training / interactive validation set (containing data on HRC patients and PDAC stages I / II) and a development test set (containing data on HRC patients and PDAC stages III / IV). Additionally, another portion of the data (e.g., approximately 552 participants) can be considered a validation cohort (blinded test set), which also contains data on HRC patients and patients with PDAC stages I to IV. In one embodiment, unimodal analysis models 21A and unimodal analysis models 21B, after training, can generate the aforementioned analytical data and output decision values, and multimodal analysis model 41, after training, can generate the aforementioned analytical data and output final decision values, but is not limited thereto.

[0045] In one embodiment, these training sets and cross-validation sets may be preliminarily classified using T-distributed stochastic neighbor embedding (t-SNE) before being input into the multimodal machine learning system 1 to visualize the cluster distribution in the metabolic fingerprint, but are not limited thereto.

[0046] Thus, the training process of the multimodal machine learning system 1 can be understood.

[0047] Next, some relevant experimental data of the multimodal machine learning system 1 are provided. In these experimental data, four metrics were used to evaluate the performance of the multimodal machine learning system 1 during the training phase, cross-validation phase, development testing phase, and blind testing phase, namely accuracy, specificity, sensitivity, and area under the receiver operating characteristic curve (ROC-AUC).

[0048] Figures 4A to 4D are performance diagrams of the multimodal machine learning system 1 according to various embodiments of this application. Figure 4A shows the performance during the training, cross-validation, development testing, and blind testing phases when the multimodal input data set includes decision values ​​corresponding to NMR-M spectra, decision values ​​corresponding to NMR-M+L spectra, and eight clinical data (including age, BMI, gender, smoking status, CA19-9, HbA1c, GluAC, and CEA). Figure 4B shows the performance during the training, cross-validation, development testing, and blind testing phases when the multimodal input data set includes decision values ​​corresponding to NMR-M spectra, decision values ​​corresponding to NMR-M+L spectra, and two clinical data (including age and CA19-9). Figure 4C shows the performance during the training, cross-validation, development testing, and blind testing phases when the multimodal input dataset includes decision values ​​corresponding to NMR-M spectra, decision values ​​corresponding to NMR-M+L spectra, and two clinical data (including gender and CA19-9). Figure 4D shows the performance during the training, cross-validation, development testing, and blind testing phases when the multimodal input dataset includes decision values ​​corresponding to NMR-M spectra, decision values ​​corresponding to NMR-M+L spectra, two clinical data (including age and CA19-9), and one protein data (Activin A).

[0049] In one embodiment, for the test set, the multimodal classification model used to analyze the multimodal input dataset to determine whether a patient has pancreatic cancer can achieve an AUC value of approximately 95%, as shown in Figures 4A to 4D, but is not limited thereto. This demonstrates that by fusing metabolic fingerprints, clinical data, and protein-related data as the multimodal input dataset, the analytical capabilities of machine learning can be effectively enhanced.

[0050] Furthermore, in one embodiment, the multimodal machine learning system 1 can select optimal hyperparameters by performing repeated computations on a cross-validation set. In one embodiment, the hyperparameter may range from 0 to 5000. In another embodiment, the hyperparameter may be 0.5, but is not limited thereto.

[0051] As shown in Figures 4A to 4D, the multimodal machine learning systems 1 in Figures 4A to 4D all have the required performance and are very suitable for assisting in the early diagnosis of pancreatic cancer. Among them, the multimodal machine learning system 1 in Figure 4D has the best accuracy and better ROC-AUC in the blind test phase, and it is not limited to this.

[0052] Figures 5A to 5D are performance diagrams of the multimodal machine learning system 1 according to various embodiments of this application. Figure 5A shows the performance of the feature extraction layer 10 in the training, cross-validation, development testing, and blind testing phases when the feature extraction layer 10 uses the SelectKBest algorithm to extract features from the NMR-M spectrum, and the multimodal input data set includes the decision value corresponding to the NMR-M spectrum and eight clinical data (including age, BMI, gender, smoking status, CA19-9, HbA1c, GluAC, and CEA). Figure 5B shows the performance of the feature extraction layer 10 in the training, cross-validation, development testing, and blind testing phases when the feature extraction layer 10 uses the RFE algorithm to extract features from the NMR-M spectrum, and the multimodal input data set includes the decision value corresponding to the NMR-M spectrum and eight clinical data (including age, BMI, gender, smoking status, CA19-9, HbA1c, GluAC, and CEA). Figure 5C shows the performance of the feature extraction layer 10 in the training, cross-validation, development testing, and blind testing phases when it uses the SelectKBest algorithm to extract features from the NMR-M+L spectrum, and the multimodal input data set includes the decision values ​​corresponding to the NMR-M+L spectrum and eight clinical data (including age, BMI, gender, smoking status, CA19-9, HbA1c, GluAC, and CEA). Figure 5D shows the performance of the feature extraction layer 10 in the training, cross-validation, development testing, and blind testing phases when it uses the RFE algorithm to extract features from the NMR-M+L spectrum, and the multimodal input data set includes the decision values ​​corresponding to the NMR-M+L spectrum and eight clinical data (including age, BMI, gender, smoking status, CA19-9, HbA1c, GluAC, and CEA).

[0053] As shown in Figures 5A to 5D, the multimodal machine learning system 1 in Figures 5A to 5D all have the required performance, and among them, Figure 5A (using the SelectKBest algorithm to extract features from the NMR-M spectrum) and Figure 5D (using the RFE algorithm to extract features from the NMR-M+L spectrum) have better performance, and this is not the only one.

[0054] Furthermore, the multimodal machine learning system 1 of this application can be used in conjunction with a full-spectrum image acquisition system 2, which can, for example, be used to acquire a full-spectrum image of a patient's metabolic fingerprint. Figure 6 is a schematic diagram of the operation process of the multimodal machine learning system 1 and the full-spectrum image acquisition system 2 according to an embodiment of this application. In one embodiment, the full-spectrum image acquisition system 2 may include an NMR spectrometer 220. After the full-spectrum image acquisition system 2 operates, it can generate a full-spectrum image, which can then be input into the multimodal machine learning system 1 for subsequent operations.

[0055] To obtain a full-spectrum image, a serum sample from the patient must first be obtained.

[0056] In one embodiment, the chemicals used to prepare serum samples may include, but are not limited to, sodium trimethylsilylpropionate-[2,2,3,3-²H₄](TSP) (98 atoms %D) (CAS# 24493-21-8), deuterium oxide (D₂O, 99.9 atoms %D) (CAS# 7789-20-0), disodium hydrogen phosphate (Na₂HPO₄) (CAS# 7558-79-4), and sodium azide (NaN₃) (CAS# 26628-22-8.

[0057] In one embodiment, the serum sample preparation process is as follows: Serum samples from all patients were prepared using serum separation tubes (BD Vacutainer Systems) from blood collected under fasting conditions, following a standard venous blood sampling protocol. First, the blood samples were centrifuged at 3000 g for 10 minutes at 4°C, and then stored at -80°C until use. Before 1H NMR spectroscopy analysis, the frozen serum samples were thawed on ice for 30 minutes and centrifuged at 13000 g for 10 minutes at 4°C. Serum samples were then prepared using 3 mm Ø, 5 times longer SampleJet NMR tubes (Brucker BioSpin GmbH). For standard sample preparation, 110 μL of serum was mixed with 110 μL of phosphate buffer (75 mM Na2HPO4, 2 mM NaN3, and 4.6 mM TSP D2O solution) and centrifuged for 10 seconds. Then, 200 μL of the mixture was transferred to a 3 mm NMR tube for subsequent analysis.

[0058] In one embodiment, details of the 1H NMR acquisition and data processing parameters are as follows: 1H NMR spectra were acquired using a 600MHz Bruker Avance III HD spectrometer equipped with a 5 mm BBI probe and a Bruker SampleJet robotic cooling system set to 4°C. Standard one-dimensional (1D) pulse sequences for solvent presaturation experiments included small molecule metabolites and lipoproteins (32 scans, zero fill to 128K index NMR-M+L spectra), while Carr-Purcell-Meiboom-Gill spin-echo experiments were used to identify small molecule metabolites (32 scans, zero fill to 128K index NMR-M spectra). Data were processed automatically using Bruker Topspin 3.6.2 and ICON NMR for phase fixing and baseline correction. A chemical shift reference (1H, δ 0.00) was provided by TSP.

[0059] In one embodiment, the data processing details for 1H NMR spectra used in machine learning are as follows: NMR spectra can be read using the nmrglue open-source software and converted into a digital format. In one embodiment, multiple types of spectra (e.g., NMR-M and NMR-M...) are used. [+]L) was used as input data for machine learning. In one embodiment, the spectral range of each NMR spectrum excluded the internal standard signal of 0 ppm and the noise region exceeding 10.0 ppm.

[0060] Thus, the process by which the full-spectrum image acquisition system 2 acquires NMR spectra can be understood.

[0061] The multimodal machine learning system 1 of this application may also have different implementations. Figure 7 is a system architecture diagram of the multimodal machine learning system 1 (third architecture) according to another embodiment of this application, and Figure 8 is a schematic diagram of the operation of the multimodal machine learning system 1 according to another embodiment of this application. Please also refer to Figures 1 to 6. Since some features of the embodiments of Figures 7 and 8 can be applied to the description of the embodiments of Figures 1 and 3A to 3C, the following description mainly focuses on the differences.

[0062] As shown in Figure 7, the third-architecture multimodal machine learning system 1 may include a feature selection layer 10, a fusion layer 30, a final decision layer 40, and a preprocessing module 50. Furthermore, the third and fourth-architecture multimodal machine learning systems 1 may optionally include a clinical data screening module 60; the following examples will use the inclusion of the clinical data screening module 60. Additionally, the third architecture may not include the encoder layer 20 described above. In one embodiment, the preprocessing module 50 and the clinical data screening module 60 may be, for example, functional modules, implemented as described above.

[0063] As shown in Figure 8, when System 1 is operating, firstly, when System 1 acquires a raw full-spectrum image of the patient (e.g., NMR-M spectrum and NMR-M+L spectrum), the preprocessing module 50 performs a preprocessing procedure on the raw full-spectrum image to convert it into a preprocessed full-spectrum image, thereby filtering out a large number of impurity signals (corresponding to step B1). Next, the feature selection layer 10 selects representative features from the features (e.g., data points) in the preprocessed full-spectrum image (corresponding to step B2). Furthermore, the clinical data screening module 60 screens the clinical data to select representative clinical data (corresponding to step B3). Then, the fusion layer 30 fuses the features of the preprocessed full-spectrum image with non-spectral data (e.g., screened clinical data and / or protein data) to generate a multimodal input dataset (corresponding to step B4). Next, the multimodal analysis model 41 of the final decision layer 40 can analyze the multimodal input data set (corresponding to step B5) to generate a final decision value, such as a binary outcome of whether the patient has PDAC.

[0064] In one embodiment, the third architecture multimodal analysis model 41 may include a TabPFN model 411. An untrained TabPFN model 411 can be transformed into a trained TabPFN model 411 through a training phase, thereby enabling the determination of whether a patient has PDAC based on the patient's corresponding multimodal input data set. In one embodiment, the training process of the TabPFN model 411 can refer to the training process of the first architecture multimodal analysis model 41, and therefore will not be described in detail. Furthermore, the number of training sets, interactive validation sets, development test sets, and blind test sets can be adjusted according to requirements.

[0065] Next, the details of the preprocessing procedure performed by the preprocessing module 50 will be explained. Figure 9 is a flowchart of the operation process of the preprocessing module 50 according to an embodiment of this application, and please also refer to Figures 7 and 8. As shown in Figure 9, firstly, step B11 is performed, and the preprocessing module 50 obtains the original full-spectrum image, which can be, for example, a signal map, where the X-axis is ppm (parts per million) and the Y-axis is density (intensity). Next, step B12 is performed, and the preprocessing module 50 performs a TSP normalization process on the full-spectrum image, for example, using the peak value of the TSP-d4 (3-(trimethylsilyl)propionic acid-d4 sodium salt) signal as a reference, and scaling each signal value (density) in the full-spectrum image. Next, step B13 is performed, and signal trimming is performed on the X-axis range (ppm) of the full-spectrum image to remove meaningless segments without metabolite signals. Next, step B14 is performed to remove water-related signals from the full-spectrum image. This transforms the full-spectrum image into a preprocessed full-spectrum image.

[0066] Regarding step B12, in one embodiment, after the full-spectrum image is positively normalized by TSP, the maximum value of the signal intensity (density) can be changed from 106 to 10-2, for example, and then each signal value is adjusted proportionally, but is not limited thereto.

[0067] Regarding step B13, in one embodiment, when the full-spectrum image is an NMR-M spectrum, the X-axis range is changed, for example, from -5 to 15 ppm to 0.13 to 11.13 ppm, thereby removing segments of the NMR-M spectrum that may contain less metabolite signal. In one embodiment, when the full-spectrum image is an NMR-M+L spectrum, the X-axis range is changed, for example, from -10 to 20 ppm to 0.11 to 10 ppm, thereby removing segments of the NMR-M+L spectrum that may contain less metabolite signal.

[0068] Regarding step B14, since water produces a strong signal in the full-spectrum image, but the water signal is generally not valuable for analysis, step B4 is used to remove the water signal. In one embodiment, the X-axis position corresponding to the water signal is 4.76 ppm.

[0069] In this way, impurity signals in the full-spectrum image can be removed, making it more suitable for analysis by machine learning systems. Furthermore, provided it is feasible, the first and second architectures of the aforementioned embodiments can also include a preprocessing module 50 to perform preprocessing procedures. Thus, the preprocessing procedure is understood.

[0070] Next, the operational details of the feature selection layer 10 will be explained. Figure 10 is a flowchart of the operation process of the feature selection layer 10 according to an embodiment of this application, and please also refer to Figures 7 and 8. Furthermore, since the TabPFN model 411 has an upper limit on the input data, the feature selection layer 10 is set to select a specific number of features, for example, it can be set to select about 500 features, preferably 200 to 250 features, but is not limited to this.

[0071] As shown in Figure 10, step B21 is first executed, where feature selection layer 10 performs a feature ranking step on the features of the preprocessed full-spectrum image. When the full-spectrum image is an NMR-M spectrum, the f_classif method can be used, while when it is an NMR-M+L spectrum, a simple linear SVM can be used. Next, feature selection layer 10 performs a feature selection step. When the full-spectrum image is an NMR-M spectrum, the SelectKBest method can be used, while when it is an NMR-M+L spectrum, the RFE algorithm can be used. Then, step B22 is executed, where feature selection layer 10 performs iterative calculations based on a set number of features (e.g., 200 to 250). Finally, step B23 is executed, where feature selection layer 10 uses UMAP (uniform manifold approximation and projection) to perform dimensionality reduction on the features in the iterative calculations. Next, step B24 is executed, where feature selection layer 10 calculates the Silhouette score for each feature. The Silhouette score can be, for example, a score indicating the presence or absence of cancer. Step B25 is then executed, selecting 200 to 250 features with the highest Silhouette scores as representative features.

[0072] Therefore, the operation of feature selection layer 10 can be understood.

[0073] Next, the operational details of the clinical data screening module 60 will be described. Figure 11 is a flowchart of the operation process of the clinical data screening module 60 according to an embodiment of this application, and please also refer to Figures 7 and 8.

[0074] As shown in Figure 11, step B31 is executed first, where the clinical data screening module 60 obtains various types of clinical data, such as age, BMI, smoking status, gender, CA19-9, HbA1C, CEA, and GluAC. Next, step B32 is executed, where the clinical data screening module 60 excludes types with missing data (NaN value) greater than 10%, thus removing types with excessive missing data. Next, step B33 is executed, where the clinical data screening module 60 removes types with features that show no significant difference in the F_Classif test (e.g., P value > 0.05). For example, if a certain type of feature is not significantly related to the presence or absence of cancer, that type will be removed. Next, step B34 is executed, where the clinical data screening module 60 performs UMAP projection on all remaining types for dimensionality reduction. Next, step B35 is executed, where the clinical data screening module 60 calculates the Silhouette score for each type, where the Silhouette score can be, for example, the score for the presence or absence of cancer. Next, step B36 is executed, and the clinical data screening module 60 selects 200 to 250 types with the highest Silhouette score as representative types of clinical data.

[0075] Therefore, the operation of the clinical data screening module 60 can be understood.

[0076] In one experimental example, the third-architecture multimodal machine learning system 1 achieved a prediction accuracy of approximately 0.91 to 0.95 for patients with PDAC, a sensitivity of approximately 0.89 to 0.96, a specificity of approximately 0.91 to 9.97, a PPV of approximately 0.92 to 0.97, an NPV of approximately 0.89 to 0.95, and an AUROC of approximately 0.98 to 0.99, all meeting the prediction requirements. This demonstrates that the third-architecture multimodal machine learning system 1 can provide high-accuracy predictions using only a small number of features (less than 500).

[0077] Furthermore, the third-architecture multimodal machine learning system 1 can also be used in conjunction with the full-spectrum image acquisition system 2. Figure 12 is a schematic diagram of the operation process of the multimodal machine learning system 1 and the full-spectrum image acquisition system 2 according to an embodiment of this application. The details are roughly the same as those in Figure 6, except that the data preprocessing module 50 and the clinical data screening module 60 are added, and the encoder layer 20 is not included.

[0078] In one embodiment, this application can determine whether a product falls within the scope of protection of this application by at least the presence or absence of components, component configuration, mechanical observation and / or operation mode of the product in dispute, and is not limited thereto.

[0079] Details or features among the various embodiments of this application may be freely combined and used as long as they do not violate the spirit of the invention or conflict with it. The numerical values ​​described in the various figures or embodiments are merely examples and are not intended to limit the application; modifications can be made as needed. Furthermore, the content of the training data used in the training process of various machine learning models in this application can be inferred from the operation process of already trained models (e.g., repeatedly performing this operation using a large amount of data and iteratively adjusting the model's parameters to improve predictive ability), and therefore will not be described in detail here.

[0080] The system and method of this application can solve the problems of the prior art.

[0081] The above embodiments are merely illustrative examples for ease of explanation. The scope of rights claimed in this application shall be determined by the claims of the patent application, and not limited to the above embodiments.

[0082] 1: Multimodal Machine Learning System 10: Feature Selection Layer 20: Encoder Layer 30: Fusion Layer 40: The final decision-making level 21, 21A, 21B: Single-modal analysis models 41: Multimodal Analysis Model A1~A5, B1~B4, B11~B14, B21~B25, B31~B36: Steps 2: Full-spectrum image acquisition system 220: NMR spectrometer 50: Preprocessing Module 60: Clinical Data Screening Module 411: TabPFN model

Claims

1. A multimodal machine learning system for generating a multimodal input dataset, comprising: an encoder layer (20) including at least one unimodal analysis model (21A, 21B) for analyzing a plurality of data points of at least one full-spectrum image to generate a decision value of the at least one full-spectrum image; and a fusion layer (30) for fusing the decision value of the at least one full-spectrum image with at least one non-spectral data to generate the multimodal input dataset, wherein the at least one non-spectral data includes at least one clinical data or protein data; wherein, The multimodal input data set is used as input data for a final decision layer (40), and the final decision layer (40) analyzes the multimodal input data set to generate a final decision value.

2. The multimodal machine learning system as claimed in claim 1, wherein the final decision value is associated with whether the source of the at least one full-spectrum image suffers from PDAC.

3. The multimodal machine learning system as claimed in claim 1, further comprising a feature selection layer (10) for filtering the plurality of data points of the at least one full-spectrum image and providing the filtered plurality of data points to the encoder layer (20).

4. The multimodal machine learning system as claimed in claim 1, wherein the at least one non-spectral data further comprises at least one protein data.

5. The multimodal machine learning system as claimed in claim 1, wherein the fusion layer (30) fuses the decision value of the at least one full-spectral image with the at least one non-spectral data in a concatenate data manner.

6. A multimodal machine learning method, executed through a multimodal machine learning system (1) to generate a multimodal input dataset, the multimodal machine learning system (1) comprising an encoder layer (20) and a fusion layer (30), the method comprising the steps of: analyzing a plurality of data points of at least one full-spectrum image using at least one unimodal analysis model (21A, 21B) of the encoder layer (20) to generate a decision value of the at least one full-spectrum image; and fusing the decision value of the at least one full-spectrum image with at least one non-spectral data using the fusion layer (30) to generate the multimodal input dataset, wherein the at least one non-spectral data comprises at least one clinical data or protein data; wherein, The multimodal input data set is used as input data for a final decision layer (40), and the final decision layer (40) analyzes the multimodal input data set to generate a final decision value.

7. The multimodal machine learning method as described in claim 6, wherein the final decision value is associated with whether the source of the at least one full-spectrum image suffers from PDAC.

8. The multimodal machine learning method as claimed in claim 6, further comprising the steps of: filtering the plurality of data points of the at least one full-spectrum image by means of a feature selection layer (10), and providing the filtered plurality of data points to the encoder layer (20).

9. The multimodal machine learning method as described in claim 6, wherein the at least one non-spectral data further comprises at least one protein data.

10. The multimodal machine learning method as claimed in claim 6, wherein the fusion layer (30) fuses the decision value of the at least one full-spectral image with the at least one non-spectral data in a data concatenation manner.

11. A multimodal machine learning system, comprising: an encoder layer (20) including at least one unimodal analysis model (21A, 21B) for analyzing a plurality of data points of at least one full-spectrum image to generate a decision value of the at least one full-spectrum image; a fusion layer (30) for fusing the decision value of the at least one full-spectrum image with at least one non-spectral data to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and a final decision layer (40) including a multimodal analysis model (41) for analyzing the multimodal input data set to generate a final decision value.

12. The multimodal machine learning system as claimed in claim 11, wherein the final decision value is associated with whether the source of the at least one full-spectrum image suffers from PDAC.

13. The multimodal machine learning system as claimed in claim 11, further comprising a feature selection layer (10) for filtering the plurality of data points of the at least one full-spectrum image and providing the filtered plurality of data points to the encoder layer (20).

14. The multimodal machine learning system as claimed in claim 11, wherein the at least one non-spectral data further comprises at least one protein data.

15. The multimodal machine learning system as claimed in claim 11, wherein the fusion layer (30) fuses the decision value of the at least one full-spectral image with the at least one non-spectral data in a concatenate manner.

16. A multimodal machine learning method, implemented through a multimodal machine learning system (1), the multimodal machine learning system (1) comprising an encoder layer (20), a fusion layer (30) and a final decision layer (40), the method comprising the steps of: analyzing a plurality of data points of at least one full-spectrum image using at least one unimodal analysis model (21A, 21B) of the encoder layer (20) to generate a decision value of the at least one full-spectrum image; fusing the decision value of the at least one full-spectrum image with at least one non-spectral data using the fusion layer (30) to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and analyzing the multimodal input data set using a multimodal analysis model (41) of the final decision layer (40) to generate a final decision value.

17. The multimodal machine learning method as described in claim 16, wherein the final decision value is associated with whether the source of the at least one full-spectrum image suffers from PDAC.

18. The multimodal machine learning method as claimed in claim 16, further comprising the steps of: filtering the plurality of data points of the at least one full-spectrum image by means of a feature selection layer (10), and providing the filtered plurality of data points to the encoder layer (20).

19. The multimodal machine learning method as described in claim 16, wherein the at least one non-spectral data further comprises at least one protein data.

20. The multimodal machine learning method as claimed in claim 16, wherein the fusion layer (30) fuses the decision value of the at least one full-spectral image with the at least one non-spectral data in a data concatenation manner.

21. A multimodal machine learning system, comprising: a preprocessing module (50) for performing a preprocessing procedure on at least one full-spectrum image to form at least one preprocessed full-spectrum image; a fusion layer (30) for fusing at least a portion of features of the at least one preprocessed full-spectrum image with at least one non-spectral data to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and a final decision layer (40) comprising a multimodal analysis model (411) for analyzing the multimodal input data set to generate a final decision value.

22. The multimodal machine learning system as claimed in claim 21, further comprising a feature selection layer (10) for filtering a plurality of data points of the at least one preprocessed full-spectrum image and providing the filtered plurality of data points to the fusion layer (30).

23. The multimodal machine learning system of claim 21, wherein the preprocessing procedure comprises: performing a TSP normalization process on the at least one full-spectrum image.

24. The multimodal machine learning system of claim 23, wherein the preprocessing procedure comprises: removing at least a portion of the signals from the at least one full-spectrum image that do not exhibit metabolite responses.

25. The multimodal machine learning system of claim 24, wherein the preprocessing procedure comprises: removing water-related signals from the signals of the at least one full-spectrum image.

26. A multimodal machine learning method, executed through a multimodal machine learning system (1), the multimodal machine learning system (1) comprising a preprocessing module (50), a fusion layer (30) and a final decision layer (40), the method comprising the steps of: performing a preprocessing procedure on at least one full-spectrum image by the preprocessing module (50) to form at least one preprocessed full-spectrum image; fusing at least a portion of features of the at least one preprocessed full-spectrum image with at least one non-spectral data by the fusion layer (30) to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and analyzing the multimodal input data set by a multimodal analysis model (411) of the final decision layer (40) to generate a final decision value.

27. The multimodal machine learning method as claimed in claim 26, further comprising the steps of: filtering a plurality of data points of the at least one preprocessed full-spectrum image by means of a feature selection layer (10), and providing the filtered plurality of data points to the fusion layer (30).

28. The multimodal machine learning method as described in claim 26, wherein the preprocessing procedure comprises: performing a TSP positive normalization on the at least one full-spectrum image.

29. The multimodal machine learning method of claim 28, wherein the preprocessing procedure comprises: removing at least a portion of the signals from the at least one full-spectrum image that do not exhibit metabolite responses.

30. The multimodal machine learning method as claimed in claim 29, wherein the preprocessing procedure comprises: removing water-related signals from the signals of the at least one full-spectrum image.

Images