Multimodal machine learning system and method

Abstract

A multimodal machine learning system generates a multimodal input data set. The multimodal machine learning system includes an encoder layer and a fusion layer. The encoder layer includes at least a single-modal classification model for analyzing multiple data points of at least one full spectrum image, so to generate at least one decision value of the at least one full spectrum image. The fusion layer is used to fuse the at least one decision value of the at least one full spectrum image with at least one non-spectral data including at least one clinical data or protein data, so as to generate the multimodal input data set.

Description

MULTIMODAL MACHINE LEARNING SYSTEM AND METHODBACKGROUND

[0001] Field of the Disclosure

[0002] The present disclosure relates to a multimodal machine learning system and method.

[0003] Description of Related Art

[0004] For certain diseases, such as pancreatic ductal adenocarcinoma (PDAC) in pancreatic cancer, there are about 80% to 90% of patients already in an advanced stage when diagnosed. Currently, these diseases lack reliable early detection methods.

[0005] At present, machine learning models have been used to assist in early detection of diseases, but most of these machine learning models only use a single type of input data as the basis for analysis to output detection results, resulting in that the accuracy is insufficient. In addition, it is very difficult for current machine learning models to analyze different types of data at the same time. For example, if a certain type of data has a small amount of features and another type of data has a large number of features, it will be difficult to analyze these two types of data at the same time through machine learning models.

[0006] Therefore, it is desired to provide a novel multimodal machine learning system and method to alleviate and / or obviate the above problems.SUMMARY

[0007] An object of the present disclosure is to provide a multimodal machine learning system (second architecture) for generating a multimodal input data set. The system comprises: an encoder layer including at least one single-modal analysis model for performing analysis on multiple data points of at least one full spectrum image to generate a decision value of the at least one full spectrum image; and a fusion layer for fusing the decision value of the at least one full spectrum image with at least one nonspectral data to generate the multimodal input data set, wherein the at least one nonspectral data includes at least one clinical data or protein data, wherein the multimodalinput data set is used as an input data of a final decision layer, and the final decision layer is used to analyze the multimodal input data set to generate a final decision value.

[0008] Another object of the present disclosure is to provide a multimodal machine learning method executed by a multimodal machine learning system (first architecture) including an encoder layer and a fusion layer to generate a multimodal input data set, the method comprising the steps of: performing analysis on multiple data points of at least one full spectrum image through at least one single-modal analysis model of the encoder layer to generate a decision value of the at least one full spectrum image; and fusing the decision value of the at least one full spectrum image with at least one non-spectral data through the fusion layer so as to generate the multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data, wherein the multimodal input data set is used as an input data for a final decision layer, and the final decision layer analyzes the multimodal input data set to generate a final decision value.

[0009] Another object of the present disclosure is to provide a multimodal machine learning system (second architecture), which comprises: an encoder layer including at least one single-modal analysis model for performing analysis on multiple data points of at least one full spectrum image to generate a decision value of the at least one full spectrum image; a fusion layer for fusing the decision value of the at least one full spectrum image with at least one non-spectral data to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and a final decision layer including a multimodal analysis model for analyzing the multimodal input data set to generate a final decision value.

[0010] Another object of the present disclosure is to provide a multimodal machine learning method executed by a multimodal machine learning system (second architecture) including an encoder layer, a fusion layer and a final decision layer, the method comprising the steps of: performing analysis on multiple data points of at least one full spectrum image through at least one single-modal analysis model of the encoder layer to generate a decision value of the at least one full spectrum image; fusing the decision value of the at least one full spectrum image with at least one non-spectral data through the fusion layer so as to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and performinganalysis on the multimodal input data set through a multimodal analysis model of the final decision layer to generate a final decision value.

[0011] Another object of the present disclosure is to provide a multimodal machine learning system (third architecture), which comprises a pre-processing module, a fusion layer, and a final decision layer. The pre-processing module performs a pre-processing procedure on at least one full spectrum image to form at least one pre-processed full spectrum image. The fusion layer is used to fuse at least a portion of the features of the pre-processed full spectrum image with at least one non-spectral data to generate a multimodal input data set, wherein the non-spectral data includes at least one clinical data or protein data. The final decision layer includes a multimodal analysis model to analyze the multimodal input data set to generate a final decision value.

[0012] Another object of the present disclosure is to provide a multimodal machine learning method, which is executed through a multimodal machine learning system (third architecture), wherein the multimodal machine learning system includes a pre-processing module, a fusion layer and a final decision layer, and the method comprises the steps of: performing a pre-processing procedure on at least one full spectrum image by the preprocessing module to form at least one pre-processed full spectrum image; fusing at least a portion of the features of the pre-processed full spectrum image with at least one nonspectral data by the fusion layer to generate a multimodal input data set, wherein the nonspectral data includes at least one clinical data or protein data; and analyzing the multimodal input data set by a multimodal analysis model of the final decision layer to generate a final decision value.

[0013] Other novel features of the disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS

[0014] FIG. 1 is a system architecture diagram of a multimodal machine learning system according to an embodiment of the present disclosure;

[0015] FIG. 2 is a flow chart illustrating the steps of a multimodal machine learning method according to an embodiment of the present disclosure;

[0016] FIG. 3 A is a schematic diagram of the operation process of the multimodal machine learning system according to an embodiment of the present disclosure;

[0017] FIG. 3B is a schematic diagram of the training and operation processes of a multimodal machine learning system according to an embodiment of the present disclosure;

[0018] FIG. 3C is a schematic diagram of the data used in the training process of a multimodal machine learning system according to an embodiment of the present disclosure;

[0019] FIG. 4A to FIG. 4D are performance diagrams of multi-modal machine learning systems according to multiple embodiments of the present disclosure (different multimodal input data sets);

[0020] FIG. 5A to FIG. 5D are performance diagrams of multimodal machine learning systems according to multiple embodiments of the present disclosure (different feature selection layers);

[0021] FIG. 6 is a schematic diagram of the operation process of the multimodal machine learning system and the full spectrum image acquisition system according to an embodiment of the present disclosure;

[0022] FIG. 7 is a system architecture diagram of a multimodal machine learning system according to an embodiment of the present disclosure;

[0023] FIG. 8 is a schematic diagram of the operation of a multimodal machine learning system according to an embodiment of the present disclosure;

[0024] FIG. 9 is a flowchart of the steps in the operation of the pre-processing module according to an embodiment of the present disclosure;

[0025] FIG. 10 is a flowchart of the steps in the operation of the feature selection layer according to an embodiment of the present disclosure;

[0026] FIG. 11 is a flowchart of the steps in the operation of the clinical data screening module according to an embodiment of the present disclosure; and

[0027] FIG. 12 is a schematic diagram of the operation of the multimodal machine learning system and the full spectrum image acquisition system according to an embodiment of the present disclosure.DETAILED DESCRIPTION OF EMBODIMENT

[0028] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and description to refer to the same or like parts.

[0029] Throughout the specification and the appended claims, certain terms may be used to refer to specific components. Those skilled in the art will understand that electronic device manufacturers may refer to the same components by different names. The present disclosure does not intend to distinguish between components that have the same function but have different names. In the following description and claims, words such as "containing" and "comprising" are open-ended words, and should be interpreted as meaning "including but not limited to".

[0030] The terms, such as "about", "substantially", or "approximately" are generally interpreted as within 10% of a given value or range, or as within 5%, 3%, 2%, 1% or 0.5% of a given value or range. In addition, unless otherwise specified, the term “a value between a first value and a second value” is generally interpreted as “first value < the value < second value”.

[0031] In the specification and claims, unless otherwise specified, ordinal numbers, such as “first” and “second”, used herein are intended to distinguish components rather than disclose explicitly or implicitly that names of the components bear the wording of the ordinal numbers. The ordinal numbers do not imply what order a component and another component are in terms of space, time or steps of a manufacturing method. Thus, what is referred to as a "first component" in the specification may be referred to as a "second component" in the claims.

[0032] In the present application, the terms "the given range is from the first numerical value to the second numerical value" and "the given range falls within the range from the first numerical value to the second numerical value" mean that the given range includes the first numerical value, the second value, and other values therebetween.

[0033] It is noted that the following are exemplary embodiments of the present application, but the present disclosure is not limited thereto, while a feature of some embodiments can be applied to other embodiments through suitable modification,substitution, combination, or separation. Tn addition, the present disclosure can be combined with other known structures to form further embodiments.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art related to the present disclosure. It can be understood that these terms, such as those defined in commonly used dictionaries, should be interpreted as having meaning consistent with the relevant technology and the background or context of the present disclosure, and should not be interpreted in an idealized or excessively formal way, unless there is a special definition in the embodiment of the present disclosure.

[0035] In addition, the term “adjacent” used herein may refer to describe mutual proximity and does not necessarily mean mutual contact.

[0036] In addition, descriptions such as "when" or "while" in the present disclosure refer to aspects such as "at the moment, before or after", but not limited to situations that occur at the same time. Similar descriptions such as "disposed on" in the present disclosure indicate the corresponding positional relationship between two components, but not limited to whether there is contact between the two components, unless otherwise specified. Furthermore, when multiple effects are provided in the present disclosure, if the word "or" is used between the effects, it means that the effects may exist independently, but it does not exclude that multiple effects may exist at the same time.

[0037] FIG. 1 is a system architecture diagram of a multimodal machine learning system 1 according to an embodiment of the present disclosure. In one embodiment, the multimodal machine learning system 1 may have a first architecture, and the first architecture includes a feature selection layer 10, an encoder layer 20, a fusion layer 30 and a final decision layer 40. The multimodal machine learning system 1 of the first architecture may fuse multiple different types of input data to form a multimodal input data set, analyze the multimodal input data set, and then output a final decision value. In one embodiment, the final decision value may be, for example, a kind of auxiliary determination information, which relates to whether a patient suffers from PDAC, or the possibility of suffering from PDAC. It should be noted that the final decision value of the present disclosure may also be other auxiliary determination information, but not limited to PDAC determination. For the convenience of explanation, in the following examples,the final decision value outputted by the multimodal machine learning system 1 relates to whether the patient suffers from PDAC.

[0038] In another embodiment, the multimodal machine learning system 1 may have a second architecture. The second architecture includes the encoder layer 20 and the fusion layer 30, and may optionally include the feature selection layer 10. At this moment, the final decision layer 40 may be provided outside the multimodal machine learning system 1 of the second architecture, such as on the cloud or other electronic devices, while it is not limited thereto. The multimodal machine learning system 1 of the second architecture may be used to generate a multimodal input data set, wherein the multimodal input data set may be transmitted to the final decision layer 40 and used as input data of the final decision layer 40, while it is not limited thereto.

[0039] In order to make the description of the present disclosure clear, in the following description, the multi-modal machine learning system 1 is exemplified by the first architecture (including the feature selection layer 10, the encoder layer 20, the fusion layer 30 and the final decision layer 40).

[0040] In one embodiment, the multimodal input data set may include one or more full spectrum images of the patient and one or more non-spectral data. The details of full spectrum images and non-spectral data are first described below.

[0041] Regarding the full spectrum image, in one embodiment, the full spectrum image may be, for example, various spectrum images obtained by using a spectrometer. Each full spectrum image may include a large number of data points (for example, hundreds, thousands, tens of thousands, hundreds of thousands, or more data points, but not limited thereto), each data point may be regarded as a feature, but it is not limited thereto. In one embodiment, each full spectrum image may be associated with a nuclear magnetic resonance (NMR) image of a metabolic fingerprint of the patient, while it is not limited thereto. The process of obtaining full spectrum images will be explained in the subsequent paragraphs.

[0042] Regarding non-spectral data, in one embodiment, the type of non-spectral data may include one or more clinical data of the patient, and the type of clinical data may include age, BMI, gender, smoking status (such as smoker or non-smoker, age of smoking, or smoking history, while it is not limited thereto), tumor marker index (CA19-9), glycated hemoglobin index (HbAl c), postprandial blood glucose index (GluAC), and carcinoembryonic antigen index (CEA), or any combination of the above, while it us not limited thereto.

[0043] In one embodiment, the non-spectral data may include one or more protein-related data of the patient, such as thrombospondin-2 (TSP-2) and / or activin A (Activin A), while it is not limited thereto. In one embodiment, serum levels of thrombospondin-2 (TSP-2) and / or Activin A were quantified using commercially available sandwich enzyme-linked immunosorbent assay kits (DTSP20 and DAC00B, respectively;Quantikine, R&D Systems, Inc., Minneapolis, MN, USA) based on the manufacturer’s instructions, while it is not limited thereto. Assays were performed in duplicate for both biomarkers. In one embodiment, the absorbance of TSP-2 was measured at 450 nm using a Multiskan™ GO Microplate Spectrophotometer (Thermo Scientific™, Waltham, MA, USA), and the results are presented as ng / mL. In addition, Activin A levels were measured after a twofold dilution of the serum samples, followed by absorbance measurement at 450 nm with the same spectrophotometer. Activin A data is presented as pg / mg, which was normalized to the total protein (mg / mL) in the serum sample.

[0044] In one embodiment, the measurement results for the clinical data and / or protein- related data of the non-spectral data are presented as the means and standard deviations. They were analyzed according to the statistical methods available in GraphPad Prism 10*0 (San Diego, CA). ROC-AUC analyses were performed for serum TSP-2, CA19-9, and CEA levels to discriminate PDAC patients from controls using MedCalc 22*023 statistical software (Mariakerke, Belgium). The cutoff values for the TSP-2 biomarker were determined by calculating the Youden index or achieving 100*0% specificity. Significant differences were calculated with a Student’s T-test with P-values < 0*05.

[0045] One of the features of the present disclosure is that the encoder layer 20 and the fusion layer 30 of the multimodal machine learning system 1 may fuse full spectrum images with a large number of features and non-spectral data with a small number of features, and the data set after fusion is used as input data for machine learning, thereby improving the accuracy of analysis of the multimodal machine learning system 1, while the existing technology has no way to fuse data with a large number of features and data with a small number of features.

[0046] Next, the details of the multimodal machine learning system 1, the feature selection layer 10, the encoder layer 20, the fusion layer 30 and the final decision layer 40 will be described.

[0047] Regarding the multimodal machine learning system 1, in one embodiment, the multi-modal machine learning system 1 may be, for example, a processor or an electronic device equipped with a processor and, in another embodiment, the multi-modal machine learning system 1 may also be, for example, one or more software or firmware, while it is not limited thereto. In one embodiment, the feature selection layer 10, the encoder layer 20, the fusion layer 30 and the final decision layer 40 may be, for example, functional modules. The functions of these functional modules may be realized by a processor executing instructions in one or more computer program products, and the computer program product may be, for example, software or firmware, and may be stored in a non- transitory computer-readable medium. The non-transitory computer-readable medium may be, for example, a memory, a hard disk, an optical disc, flash drive, cloud hard drive, etc., while it is not limited thereto.

[0048] Regarding the feature selection layer 10, in one embodiment, the feature selection layer 10 may be used to select the most representative features from a large number of features (such as data points) in the full spectrum image, thereby filtering out features that are less relevant to the final decision, and reducing the amount of data for subsequent processing. In one embodiment, the feature selection layer 10 may use various suitable methods for feature selection, such as the SelectBest algorithm or the recursive feature elimination algorithm (RFE), while it is not limited thereto. In one embodiment, the feature selection layer 10 may eliminate duplicate features through an iteration process, while it is not limited thereto.

[0049] Regarding the encoder layer 20, in one embodiment, the encoder layer 20 may include at least one single-modal analysis model 21. For each full spectrum image, the single-modal analysis model 21 may perform analysis on multiple data points in the full spectrum image, so as to generate a decision value corresponding to the full spectrum image, wherein the "decision value" is, for example, a radial basis function kernel value (RBF kernel value), while it is not limited thereto. In one embodiment, the single-modal analysis model 21 may be, for example, a support vector machine (SVM). However, inanother embodiment, the single-modal analysis model 21 may be a neural network, for example, a convolutional neural network or a random forest model, while it is not limited thereto. In one embodiment, the encoder layer 20 may be equipped with multiple single- modal analysis models 21 at the same time, and each single-modal analysis model 21 may be used to analyze different types of full spectrum images of the patient, such as different types of metabolic fingerprints, while it is not limited thereto. In one embodiment, before entering the encoder layer 20, the data points of the full spectrum image may be standardized through the StandardScaler function, while it is not limited thereto. In one embodiment, the support vector machine may, for example, adopt the scikit-leam kit (version 1.2.2), while it is not limited thereto.

[0050] In one embodiment, the single-modal analysis model 21 is configured, for example, to analyze whether the patient suffers from PDAC as the goal, but not using whether the patient suffers from PDAC as the output result, while it uses the decision value targeted for determining whether the patient corresponding to the features in the full spectrum image suffers from PDAC as the output result. For example, after the single-modal analysis model 21 analyzes the features in the full spectrum image, it will determine correlation between the features in the full spectrum image and the occurrence of PDAC, and calculate a score as the decision value. If the decision value is within a numerical range, it means that the patient does not suffer from PDAC (or has a low probability of suffering from PDAC). If the decision value is within another range of numerical value, it means that the patient suffers from PDAC (or has a high possibility of suffering from PDAC). Alternatively, the decision value may also be in a numerical range that cannot be used to determine whether the patient suffers from PDAC or not. In other words, the decision value here represents the score of the correlation between the features of the full spectrum image and the occurrence of PDAC, rather than the determination result of whether the patient suffers from PDAC.

[0051] Next, the method of obtaining the decision value will be further explained. In one embodiment, the single-modal analysis model 21 itself has the function of outputting decision values. In another embodiment, the user may obtain the decision value by disassembling the program of the single-modal analysis model 21 or providing a subroutine. However, the present disclosure is not limited thereto.

[0052] Accordingly, through the operation of the encoder layer 20, a large number of features (for example, multiple data points) of each full spectrum image may be converted into a single feature (for example, a single decision value).

[0053] Regarding the fusion layer 30, in one embodiment, the fusion layer 30 may be used to fuse the decision value of the patient's full spectrum image with the non-spectral data, thereby generating a multimodal input data set. In one embodiment, the fusion layer 30 may fuse the decision value of the full spectrum image with the non-spectral data by adopting a data concatenation manner, while it is not limited thereto, wherein "data concatenation" is, for example, an operation method in the field of machine learning, which is used to concatenate different levels, different channels or different types of features or information together to form new features, while it is not limited thereto. In one embodiment, before being fused, the non-spectral data may be standardized through the Standard Seal er function, while it is not limited thereto. In one embodiment, when the fusion layer 30 fuses the decision value of the full spectrum image and the non-spectral data, data with missing values may be removed to avoid affecting the analysis process.

[0054] Regarding the final decision layer 40, in one embodiment, the final decision layer 40 may include a multimodal analysis model 41 for analyzing the multi-modal input data set to generate the final decision value. In one embodiment, the final decision value is, for example, a binary result associated with whether the patient suffers from PDAC, such as a "yes" or "no" result, while it is not limited thereto. In one embodiment, the multimodal analysis model 41 may be, for example, a support vector machine. In another embodiment, the multimodal analysis model 41 may be, for example, a neural network (such as but not limited to a convolutional neural network or a random forest model), while it is not limited thereto. In one embodiment, the support vector machine may, for example, adopt the scikit-learn kit (version 1.2.2), while it is not limited thereto.

[0055] With the above features, the multimodal machine learning system 1 of the present disclosure may execute a multimodal machine learning method. FIG. 2 is a flow chart illustrating the steps of a multimodal machine learning method according to an embodiment of the present disclosure, and please refer to FIG. 1 at the same time.

[0056] As shown in FIG. 2, first, step Al is executed, in which the multimodal machine learning system 1 obtains at least one full spectrum image of a patient. For example, theuser may input the full spectrum image into the multimodal machine learning system 1, or the data is transmitted to the multimodal machine learning system 1 by an instrument that generates full spectrum images, while it is not limited thereto. Then, step A2 is executed, in which the feature selection layer 10 filters the multiple data points of the at least one full spectrum image and provides the filtered data points to the encoder layer 20. This step is used to reduce the number of features, and thus it can be performed or not performed. Then, step A3 is executed, in which at least one single-modal classification model 21 of the encoder layer 20 performs analysis on the filtered data points of the at least one full spectrum image to generate at least one decision value. For the content of this step, please refer to the description of the encoder layer 20 in the previous paragraphs. Then, step A4 is executed, in which the fusion layer 30 fuses at least one decision value with the patient's non-spectral data to generate a multimodal input data set. For the content of this step, please refer to the description of the fusion layer 30 in the previous paragraphs. Then, step A5 is executed, in which the multimodal classification model 41 of the final decision layer 40 performs analysis on the multimodal input data set to generate a final decision value. For the content of this step, please refer to the description of the final decision layer 40 in the previous paragraphs. Accordingly, the multimodal machine learning method can be understood.

[0057] Next, an example is given to illustrate the operation process of the multimodal machine learning system 1. FIG. 3 A is a schematic diagram of the operation process of the multimodal machine learning system 1 according to an embodiment of the present disclosure, and please refer to FIG. 1 and FIG. 2 at the same time. In the example of FIG. 3A, the encoder layer 20 may have two single-modal analysis models 21 A and 21B, and two types of metabolic fingerprint full spectrum images of the patient, which are NMR-M spectrum and NMR-M+L spectrum, are inputted into the multimodal machine learning system 1, wherein the single-modal analysis model 21A is used to analyze NMR-M spectrum, and single-modal analysis model 2 IB is used to analyze the NMR-M+L spectrum.

[0058] As shown in FIG. 3A, first, the NMR-M spectrum and the NMR-M+L spectrum are respectively inputted into the multimodal machine learning system 1 (corresponding to step Al of FIG. 2), wherein the NMR-M spectrum has a large number of data points(such as but not limited to 131072), and the NMR-M+L spectrum has a large number of data points (such as but not limited to 131072). Next, the feature selection layer 10 performs feature selection on the NMR-M spectrum and the NMR-M+L spectrum respectively (corresponding to step A2 in FIG. 2), wherein the NMR-M spectrum uses the SelectBest algorithm to perform feature selection, with which multiple data points (such as but not limited to 228) are selected, and the NMR-M+L spectrum uses the RFE algorithm to perform feature selection, with which multiple data points (such as but not limited to 12945) are selected. In one embodiment, in feature selection, duplicate features may be eliminated by performing iterations (for example, but not limited to 4315 times), thereby reducing the number of features. Then, the single-modal analysis model 21A of the encoder layer 20 performs analysis on the multiple data points (such as but not limited to 228) of the NMR-M spectrum to generate one decision value corresponding to the NMR-M spectrum, and the single-modal analysis model 2 IB of the encoder layer 20 performs analysis on the multiple data points (such as but not limited to 12945) of the NMR-M+L spectrum to generate one decision value corresponding to the NMR-M+L spectrum (corresponding to step A3 in FIG. 2). Next, the fusion layer 30 fuses the decision value corresponding to the NMR-M spectrum, the decision value corresponding to the NMR-M+L spectrum, the patient's clinical data and the protein-related data to generate a multimodal input data set (corresponding to step A4 in FIG. 2). Then, the multimodal classification model 41 of the final decision layer 40 performs analysis on the multimodal input data set and outputs a final decision value, wherein the final decision value corresponds to the result of whether the patient suffers from pancreatic cancer, while it is not limited thereto. Accordingly, the operation process of the multimodal machine learning system 1 and the multimodal machine learning method can be understood.

[0059] To acquire capability of analyzing whether the patient suffers from pancreatic cancer, the multimodal machine learning system 1 needs to be trained through a training process. FIG.3B is a schematic diagram of the training process and the operation process of the multimodal machine learning system 1 according to an embodiment of the present disclosure, and please refer to FIG. 1 to FIG. 3 at the same time. As shown in FIG. 3B, an untrained multimodal machine learning system 1 is trained through the training process.After the training process, the untrained multimodal machine learning system 1 becomes a trained multimodal machine learning system 1, and the operation process of the multimodal machine learning system 1 may be executed.

[0060] Next, the detail of the training process of the multimodal machine learning system 1 will be described. FIG. 3C is a schematic diagram of data used in the training process of the multimodal machine learning system 1 according to an embodiment of the present disclosure, and please refer to FIG. 1 to FIG. 3B at the same time. As shown in FIG. 3C, in one embodiment, data including serum metabolic fingerprints, clinical data and protein-related data of multiple (for example, about 902) participants are used in the training process of the multimodal machine learning system 1. Some of these participants are controls who have not yet suffered from PDAC (who may be high-risk individuals (for example, high risk control > HRC) or healthy individuals (for example, health control, HC)), and the other part is composed of patients with PDAC at different stages. In one embodiment, among the data of multiple (for example, about 902) participants, part of the data (for example, of about 350 participants) may be regarded as the training set and the cross-validation set in the training process of the single-modal analysis model 21A, the single-modal analysis model 21B and / or the multimodal analysis model 41, and the other data may be regarded as the test sets, but it is not limited thereto. More detail, in one embodiment, part of the data (for example, of about 350 participants) may be regarded as an exploration cohort (the training set) including data of patients with HRC and data of patients with PDAC I to PDAC IV, and then the exploration cohort is split into training / cross-validation sets including data of patients with HRC and data of patients with PDAC VII, and development test set including data of patients with HRC and data of patients with PDAC IIVIV. Besides, other part of the data (or example, of about 552 participants) may be regarded as a validation cohort (the blind-test set) including data of patients with HRC and data of patients with PDAC I to PDAC IV. In one embodiment, after training, the single-modal analysis model 21 A and the single- modal analysis model 21B may generate the aforementioned data analysis and final decision value output capability and, after training, the multimodal analysis model 41 may generate the aforementioned data analysis data and final decision value output capability, while it is not limited thereto.

[0061] In one embodiment, these training sets and cross-validation sets may be performed with preliminary classification by using T-distributed stochastic neighbor embedding (t-SNE) before being inputted to the multimodal machine learning system 1, so as to visualize the clustering distribution of metabolic fingerprints within the data, while it is not limited thereto.

[0062] Accordingly, the training process of the multimodal machine learning system 1 can be understood.

[0063] Next, some relevant experimental data of the multimodal machine learning system 1 are provided. In these experimental data, four types of indicators will be used to evaluate the performance of the training stage, cross-validation stage, development test stage and blind test stage of the multimodal machine learning system 1, which are the accuracy, specificity, sensitivity and area under the receiver operating characteristic curve (ROC-AUC), respectively.

[0064] FIG. 4A to FIG. 4D are performance diagrams of the multimodal machine learning system 1 according to multiple embodiments of the present disclosure. FIG. 4A shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the multimodal input data set includes the decision value corresponding to the NMR-M spectrum, the decision value corresponding to the NMR-M+L spectrum, and eight types of clinical data (including age, BMI, gender, smoking status, CAI 9-9, HbAlc, GluAC and CEA). FIG. 4B shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the multimodal input data set includes the decision value corresponding to the NMR-M spectrum, the decision value corresponding to the NMR-M+L spectrum and two types of clinical data (including age and CAI 9-9). FIG. 4C shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the multimodal input data set includes the decision value corresponding to the NMR-M spectrum, the decision value corresponding to the NMR-M+L spectrum and two types of clinical data (including gender and CAI 9-9). FIG. 4D shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the multimodal input data set includes the decision value corresponding to the NMR-M spectrum, thedecision value corresponding to the NMR-M+L spectrum, two types of clinical data (including age and CA19-9) and one type of protein-related data (Activin A).

[0065] In one embodiment, for the test set, the AUC value of the multimodal classification model that analyzes the multimodal input data set for distinguishing whether the patient suffers from pancreatic cancer may reach approximately 95%, as shown in FIG. 4A to 4D, while it is not limited thereto. It can be seen that, by fusing metabolic fingerprints, clinical data and protein-related data as a multimodal input data set, it is able to improve the analysis capabilities of machine learning.

[0066] In addition, in one embodiment, the multimodal machine learning system 1 may select an optimized hyperparameter by performing iteration on the cross-validation set. In one embodiment, the hyperparameter may be between 0 and 5000. In one embodiment, the hyperparameter may be 0.5, but it is not limited thereto.

[0067] As shown in FIG. 4A to FIG. 4D, the multimodal machine learning system 1 of FIG. 4A to FIG. 4D all has performance that meets the requirements, and thus is very suitable for assisting in the early diagnosis of pancreatic cancer, wherein the multimodal machine learning system 1 of FIG. 4D has the best accuracy and better ROC-AUC in the blind test stage, while it is not limited thereto.

[0068] FIG. 5A to FIG. 5D are performance diagrams of the multimodal machine learning system 1 according to multiple embodiments of the present disclosure. FIG. 5A shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the feature selection layer 10 uses the SelectKBest algorithm to extract features from the NMR-M spectrum, and the multimodal input data set includes the decision value corresponding to the NMR-M spectrum and eight types of clinical data (including age, BMI, gender, smoking status, CA19-9, HbAlc, GluAC and CEA). FIG. 5B shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the feature selection layer 10 uses the RFE algorithm to extract features from the NMR- M spectrum, and the multimodal input data set includes the decision value corresponding to the NMR-M spectrum and eight types of clinical data (including age, BMI, gender, smoking status, CAI 9-9, HbAlc, GluAC and CEA). FIG. 5C shows the efficacy performance under the training stage, cross-validation stage, development test stage andblind test stage when the feature selection layer 10 uses the SelectKBest algorithm to extract features from the NMR-M+L spectrum, and the multimodal input data set includes the decision value corresponding to the NMR-M+L spectrum and eight clinical data (including age, BMI, gender, smoking status, CAI 9-9, HbAlc, GluAC and CEA). FIG. 5D shows the efficacy performance under the training stage, cross-validation stage, development test stage and blind test stage when the feature selection layer 10 uses the RFE algorithm to extract features from the NMR-M+L spectrum, and the multimodal input data set includes the decision value corresponding to the NMR-M+L spectrum and eight clinical data (including age, BMI, gender, smoking status, CAI 9-9, HbAlc, GluAC and CEA).

[0069] As shown in FIG. 5A to FIG. 5D, the multimodal machine learning system 1 of FIG. 5A to FIG. 5D all has performance that meets the requirements, wherein FIG. 5A (using the SelectKBest algorithm to extract features of NMR-M spectrum) and FIG. 5D (using the RFE algorithm to extract features from the NMR-M+L spectrum) have better performance, while it is not limited thereto.

[0070] In addition, the multimodal machine learning system 1 of the present disclosure may be used in conjunction with a full spectrum image acquisition system 2. The full spectrum image acquisition system 2 may be used, for example, to acquire a full spectrum image of a patient's metabolic fingerprint. FIG. 6 is a schematic diagram of the operation process of the multimodal machine learning system 1 and the full spectrum image acquisition system 2 according to an embodiment of the present disclosure. In one embodiment, the full spectrum image acquisition system 2 may include an NMR spectrometer 220. After operation, the full spectrum image acquisition system 2 may generate a full spectrum image, and then the full spectrum image may be inputted into the multimodal machine learning system 1 for subsequent operations.

[0071] In order to obtain full spectrum images, the patient's serum sample must first be obtained.

[0072] In one embodiment, chemicals used to prepare serum samples may include Sodium trimethyl silyl propionate-[2,2,3,3-2H4] (TSP) (98 atom %D) (CAS# 24493-21-8), deuterium oxide (D2O, 99*9 atom %D) (CAS# 7789-20-0), sodium phosphate dibasic (Na2HPO4) (CAS# 7558-79-4), and sodium azide (NaN3) (CAS# 26628-22-8) purchasedfrom Merck (Darmstadt, Germany) and Sigma-Aldrich (St. Louis, Missouri, USA), respectively, while it is not limited thereto.

[0073] In one embodiment, the preparation process of serum samples is as follows: all serum samples were prepared from blood collected under fasting conditions using a serum-separating tube (BD Vacutainer Systems, Franklin Lakes, NJ, USA) following a standard venous blood sampling protocol; the samples were centrifuged at 3000g for 10 min at 4 °C, then stored at -80 °C until use; beforerH NMR spectroscopic analysis, frozen serum samples were thawed on ice for 30 min and centrifuged for 10 min at 13000g at 4 °C; serum samples were prepared using 3 mm 0, 5-long SampleJet NMR tubes (Brucker BioSpin GmbH, Rheinstetten, Germany); for standard sample preparation, 110 pL of serum was mixed with 110 pL phosphate buffer (75 mM Na2HPO4, 2 mM NaN3, 4*6 mM TSP in D2O), centrifuged for 10 s, and 200 pL was transferred to a 3 mm NMR tube for analyses.

[0074] In one embodiment, the details of ' H NMR acquisition and data processing parameters are as follows: the 'H NMR spectra were acquired using a 600 MHz Bruker Avance III HD spectrometer equipped with a 5 mm BBI probe and fitted with the Bruker SampleJet robot cooling system set to 4 °C; a standard one-dimensional (ID) pulse sequence with solvent presaturation experiment included both small molecule metabolites and lipoproteins (32 scans, zero-filled to 128K refers to NMR-M + L spectrum), whereas a Carr-Purcell-Meiboom-Gill spin-echo experiment was used to identify small molecule metabolites (32 scans, zero-filled to 128K refers to NMR-M spectrum); the data were processed in automation using a Bruker Topspin 3*6.2 and ICON NMR to achieve phasing and baseline correction; the TSP provided a chemical shift reference (TH, 5 0*00).

[0075] In one embodiment, the data processing details of the 'H NMR spectrum used in machine learning are as follows: the NMR spectrum may be read through the nmrglue open source software, and the data may be converted into a digital format. In one embodiment, multiple types of spectra (eg, NMR-M and NMR-M+L) are used as input data for machine learning. In one embodiment, the spectral range of each NMR spectrum excludes the internal standard signal at 0 ppm and the noise region over 10*0 ppm.

[0076] Accordingly, the process of obtaining the NMR spectrum by the full spectrum image acquisition system 2 can be understood.

[0077] The multimodal machine learning system 1 of the present disclosure may also have different implementations. FIG. 7 is a system architecture diagram of the multimodal machine learning system 1 (third architecture) according to another embodiment of the present disclosure, FIG. 8 is a schematic diagram of the operation of the multimodal machine learning system 1 according to another embodiment of the present disclosure, and please refer to FIG. 1 to FIG. 6 at the same time. Since some features of the embodiments of FIG. 7 and FIG. 8 are applicable to the description of the embodiments of FIG. 1 and FIG. 3A to FIG. 3C, the following description mainly focuses on the differences.

[0078] As shown in FIG. 7, the multimodal machine learning system 1 of the third architecture may include a feature selection layer 10, a fusion layer 30, a final decision layer 40 and a pre-processing module 50. In addition, the multimodal machine learning system 1 of the third architecture and the fourth architecture may optionally include a clinical data screening module 60, and the following examples are all exemplified by the embodiment including the clinical data screening module 60. In addition, the third architecture may not include the encoder layer 20 of the aforementioned paragraphs. In one embodiment, the pre-processing module 50 and the clinical data screening module 60 may be, for example, functional modules, and the implementation method is as described in the aforementioned paragraphs.

[0079] As shown in FIG. 8, when the system 1 is in operation, first, when the system 1 obtains an original full spectrum image of the patient (such as an NMR.-M spectrum and an NMR-M+L spectrum), the pre-processing module 50 may perform a pre-processing procedure on the original full spectrum image to convert the original full spectrum image into a pre-processed full spectrum image to filter out a large amount of impurity signals (corresponding to step Bl). Then, the feature selection layer 10 may select representative features from the features (such as data points) in the pre-processed full spectrum image (corresponding to step B2). In addition, the clinical data screening module 60 may filter the clinical data to select representative clinical data (corresponding to step B3). Then, the fusion layer 30 may fuse the features of the pre-processed full spectrum image withnon-spectral data (such as filtered clinical data and / or protein data) to generate a multimodal input data set (corresponding to step B4). Next, the multimodal analysis model 41 of the final decision layer 40 may analyze the multimodal input data set (corresponding to step B5) to generate a final decision value, such as a binary result of whether the patient suffers from PDAC.

[0080] In one embodiment, the multimodal analysis model 41 of the third architecture may include a TabPFN model 411. The untrained TabPFN model 411 may be converted into a trained TabPFN model 411 through a training phase, thereby determining whether a patient has PDAC based on the multimodal input data set corresponding to the patient. In one embodiment, the training process of the TabPFN model 411 may refer to the training process of the multimodal analysis model 41 of the first architecture and thus a detailed description is deemed unnecessart. Furthermore, the number of training sets, cross-validation sets, development test sets, and blind test sets may be adjusted as needed.

[0081] The details of the pre-processing procedure performed by the pre-processing module 50 are described below. FIG. 9 is a flowchart of the steps of the operation process of the pre-processing module 50 according to an embodiment of the present disclosure, and please refer to FIG. 7 to FIG. 8 at the same time. As shown in FIG. 9, first, step Bl 1 is executed, and the pre-processing module 50 obtains the original full spectrum image, wherein the full spectrum image may be, for example, a signal graph, and wherein the X- axis is ppm (parts per million) and the Y-axis is intensity. Then, step B12 is executed, and the pre-processing module 50 performs a TSP normalization process on the full spectrum image, for example, using the peak value of the TSP-d4 (3- (trimethylsilyl)propionic acid-d4 sodium salt) signal as a reference benchmark to scale each signal value (intensity) in the full spectrum image. Then, step B13 is executed to perform signal trimming on the X-axis range (ppm) in the full spectrum image to remove meaningless segments without metabolite signals. Next, step B14 is executed to remove the water signal from the full spectrum image, thereby converting the full spectrum image into a pre-processed full spectrum image.

[0082] Regarding step B12, in one embodiment, after the full spectrum image is subject to TSP normalization, the maximum value of the signal strength (intensity) may beconverted from 106to 10'2, for example, and then each signal value is adjusted according to the ratio, but it is not limited thereto.

[0083] Regarding step B13, in one embodiment, when the full spectrum image is an NMR-M spectrum, the range of the X-axis is changed from, for example, -5—15 ppm to 0.13-11.13 ppm, thereby removing sections of the NMR-M spectrum that may contain fewer metabolite signals. In one embodiment, when the full spectrum image is an NMR- M+L spectrum, the range of the X-axis is changed from, for example, -10-20 ppm to 0.11-10 ppm, thereby removing sections of the NMR-M+L spectrum that may contain fewer metabolite signals.

[0084] Regarding step B14, since water produces a strong signal in the full spectrum image, but the water signal is generally not analytically valuable, step B4 is used to remove the water signal. In one embodiment, the X-axis position corresponding to the water signal is 4.76 ppm.

[0085] In this way, the impurity signals in the full spectrum image may be removed, making it more suitable for analysis by a machine learning system. Furthermore, if feasible, the first and second architectures of the aforementioned embodiments may also include a pre-processing module 50 to perform a pre-processing procedure. Thus, the preprocessing procedure can be understood.

[0086] Next, the details of the operation of the feature selection layer 10 are described. FIG. 10 is a flowchart of the steps of the operation process of the feature selection layer 10 according to an embodiment of the present disclosure, and please refer to FIG. 7 and FIG. 8 at the same time. In addition, because the TabPFN model 411 has an upper limit on the input data, the feature selection layer 10 is configured to select features within a specific number range. For example, it may be configured to select approximately 500 features, preferably 200 to 250 features, but it is not limited thereto.

[0087] As shown in FIG. 10, step B21 is first executed, and the feature selection layer 10 performs a feature ranking step on the features of the pre-processed full spectrum image. When the full spectrum image is an NMR-M spectrum, the f_classif method may be used, and when the full spectrum image is an NMR-M+L spectrum, a simple linear SVM may be used. Next, the feature selection layer 10 performs a feature selection step. When the full spectrum image is an NMR-M spectrum, the SelectKBest method may be used, andwhen the full spectrum image is an NMR-M+L spectrum, the RFE algorithm may be used. Next, step B22 is executed, and the feature selection layer 10 performs an iterative operation based on a set number of features (for example, 200 to 250). Next, step B23 is executed, and the feature selection layer 10 uses UMAP (uniform manifold approximation and projection) on the features in the iterative operation to perform dimensionality reduction. Next, step B24 is executed, in which the feature selection layer 10 calculates a silhouette score for each feature. Here, the silhouette score may be, for example, a cancer score or a cancer-free score. Step B25 is then executed, in which 200 to 250 features with the largest silhouette scores are selected as representative features.

[0088] Accordingly, the operation of the feature selection layer 10 can be understood.

[0089] Next, the operation details of the clinical data screening module 60 are described.FIG. 11 is a flowchart of the operation process of the clinical data screening module 60 according to an embodiment of the present disclosure, and please refer to FIG. 7 to FIG. 8 at the same time.

[0090] As shown in FIG. 11, step B31 is executed first, in which the clinical data screening module 60 obtains various types of clinical data, such as age, BMI, smoking status, gender, CA19-9, HbAlC, CEA, GluAC, etc. Then step B32 is executed, in which the clinical data screening module 60 excludes types with missing data values (NaN value) greater than 10%, so that types with too much missing data will be removed. Then, step B33 is executed, in which the clinical data screening module 60 removes types of features that have no significant difference in the F Classif test method (for example, P value>0.05). For example, if a certain type of feature has no obvious relationship with cancer or no cancer, then the type will be removed. Then, step B34 is executed, in which the clinical data screening module 60 performs UMAP projection on all remaining types to perform dimensionality reduction. Then, step B35 is executed, and the clinical data screening module 60 calculates the silhouette score of each type, wherein the silhouette score may be, for example, a score for cancer or a score for no cancer. Then, step B36 is executed, in which the clinical data screening module 60 selects 200 to 250 types with the largest silhouette scores as representative clinical data types.

[0091] Accordingly, the operation of the clinical data screening module 60 can be understood.

[0092] In an experimental example, the prediction accuracy of the multimodal machine learning system 1 of the third architecture for whether a patient has PDAC is approximately between 0.91 and 0.95, its sensitivity is approximately between 0.89 and 0.96, its specificity is approximately between 0.91 and 9.97, its PPV is approximately between 0.92 and 0.97, its NPV is approximately between 0.89 and 0.95, and its AUROC is approximately between 0.98 and 0.99, all of which meet the prediction requirements. It can be seen that the multimodal machine learning system 1 of the third architecture may provide high-precision predictions using only a small number of features (less than 500).

[0093] In addition, the multimodal machine learning system 1 of the third architecture may also be used in conjunction with the full spectrum image acquisition system 2. FIG. 12 is a schematic diagram of the operation process of the multimodal machine learning system 1 and the full spectrum image acquisition system 2 according to an embodiment of the present disclosure. The details are substantially the same as those in FIG. 6, with the difference being the addition of the data pre-processing module 50 and the clinical data screening module 60, and the omission of the encoder layer 20.

[0094] In one embodiment, the present disclosure may at least determine whether the product in contention falls within the protection scope of the present disclosure based on the presence or absence of components, component configuration, mechanical observation and / or operation mode of the product in contention, while it is not limited thereto.

[0095] The details or features of the various embodiments of the present disclosure may be mixed and matched as long as they do not violate the spirit of the disclosure or conflict with each other. The numerical values recorded in the various figures or embodiments are for illustrative purposes only and are not intended to be limiting. In practice, they may be modified as needed. In addition, the content of the training data used in the training process of the various machine learning models in the present disclosure may be inferred by referring to the operation process of the trained model (for example, using a large amount of data to repeat the operation process and iteratively adjust the parameters of the model to improve the prediction ability), and thus a detailed description is deemed unnecessary.

[0096] Tn one embodiment, the present disclosure may at least determine whether the product in contention falls within the protection scope of the present disclosure based on the presence or absence of components, component configuration, mechanical observation and / or operation mode of the product in contention, while it is not limited thereto.

[0097] The details or features of the various embodiments of the present disclosure may be mixed and matched as long as they do not violate the spirit of the disclosure or conflict with each other. The numerical values recorded in the various figures or embodiments are for illustrative purposes only and are not intended to be limiting. In practice, they may be modified as needed. In addition, the content of the training data used in the training process of the various machine learning models in the present disclosure may be inferred by referring to the operation process of the trained model (for example, using a large amount of data to repeat the operation process and iteratively adjust the parameters of the model to improve the prediction ability), and thus a detailed description is deemed unnecessary.

[0098] With the system and method of the present disclosure, the problems of the prior art can be solved.

[0099] The aforementioned specific embodiments should be construed as merely illustrative, and not limiting the rest of the present disclosure in any way.

Claims

Claims1. A multimodal machine learning system (1) for generating a multimodal input data set, comprising: an encoder layer (20) including at least one single-modal analysis model (SVM1, SVM2) for performing analysis on multiple data points of at least one full spectrum image to generate a decision value of the at least one full spectrum image; and a fusion layer (30) for fusing the decision value of the at least one full spectrum image with at least one non-spectral data to generate the multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data, wherein the multimodal input data set is used as an input data of a final decision layer (40), and the final decision layer (40) is used to analyze the multimodal input data set to generate a final decision value.

2. The multimodal machine learning system (1) as claimed in claim 1, wherein the final decision value relates to whether a source of the at least one full spectrum image suffers from PDAC.

3. The multimodal machine learning system (1) as claimed in claim 1, further comprising a feature selection layer (10) for filtering the multiple data points of the at least one full spectrum image, and providing the filtered data points to the encoder layer (20).

4. The multimodal machine learning system (1) as claimed in claim 1, wherein the at least one non-spectral data further includes at least one protein-related data.

5. The multimodal machine learning system (1) as claimed in claim 1, wherein the fusion layer (30) uses data concatenation to fuse the decision value of the at least one full spectrum image with the at least one non-spectral data.

6. A multimodal machine learning method executed by a multimodal machine learning system (1) including an encoder layer (20) and a fusion layer (30) to generate a multimodal input data set, the method comprising the steps of: performing analysis on multiple data points of at least one full spectrum image through at least one single-modal analysis model (SVM1, SVM2) of the encoder layer (20) to generate a decision value of the at least one full spectrum image; and fusing the decision value of the at least one full spectrum image with at least one nonspectral data through the fusion layer (30) so as to generate the multimodal input data set,wherein the at least one non-spectral data includes at least one clinical data or protein data, wherein the multimodal input data set is used as an input data for a final decision layer (40), and the final decision layer (40) analyzes the multimodal input data set to generate a final decision value.

7. The multimodal machine learning method as claimed in claim 6, wherein the final decision value relates to whether a source of the at least one full spectrum image suffers from PDAC.

8. The multimodal machine learning method as claimed in claim 6, further comprising the step of filtering the multiple data points of the at least one full spectrum image through a feature selection layer (10), and providing the filtered data points to the encoder layer (20).

9. The multimodal machine learning method as claimed in claim 6, wherein the at least one non-spectral data further includes at least one protein-related data.

10. The multimodal machine learning method as claimed in claim 6, wherein the fusion layer (30) uses data concatenation to fuse the decision value of the at least one full spectrum image with the at least one non-spectral data11. A multimodal machine learning system (1), comprising: an encoder layer (20) including at least one single-modal analysis model (SVM1, SVM2) for performing analysis on multiple data points of at least one full spectrum image to generate a decision value of the at least one full spectrum image; a fusion layer (30) for fusing the decision value of the at least one full spectrum image with at least one non-spectral data to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data; and a final decision layer (40) including a multimodal analysis model (SVM3) for analyzing the multimodal input data set to generate a final decision value.

12. The multimodal machine learning system (1) as claimed in claim 11, wherein the final decision value relates to whether a source of the at least one full spectrum image suffers from PDAC.

13. The multimodal machine learning system (1) as claimed in claim 11, further comprising a feature selection layer (10) for filtering the multiple data points of the at least one full spectrum image, and providing the filtered data points to the encoder layer (20).

14. The multimodal machine learning system (1) as claimed in claim 11, wherein the at least one non-spectral data further includes at least one protein-related data.

15. The multimodal machine learning system (1) as claimed in claim 11, wherein the fusion layer (30) uses data concatenation to fuse the decision value of the at least one full spectrum image with the at least one non-spectral data.

16. A multimodal machine learning method executed by a multimodal machine learning system (1) including an encoder layer (20), a fusion layer (30) and a final decision layer (40), the method comprising the steps of performing analysis on multiple data points of at least one full spectrum image through at least one single-modal analysis model (SVM1, SVM2) of the encoder layer (20) to generate a decision value of the at least one full spectrum image; fusing the decision value of the at least one full spectrum image with at least one nonspectral data through the fusion layer (30) so as to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and performing analysis on the multimodal input data set through a multimodal analysis model (SVM3) of the final decision layer (40) to generate a final decision value.

17. The multimodal machine learning method as claimed in claim 16, wherein the final decision value relates to whether a source of the at least one full spectrum image suffers from PDAC.

18. The multimodal machine learning method as claimed in claim 16, further comprising the step of filtering the multiple data points of the at least one full spectrum image through a feature selection layer (10), and providing the filtered data points to the encoder layer (20).

19. The multimodal machine learning method as claimed in claim 16, wherein the at least one non-spectral data further includes at least one protein-related data.

20. The multimodal machine learning method as claimed in claim 16, wherein the fusion layer (30) uses data concatenation to fuse the decision value of the at least one full spectrum image with the at least one non-spectral data.

21. A multimodal machine learning system (1), comprising: a pre-processing module (50) for performing a pre-processing procedure on at least one full spectrum image to form at least one pre-processed full-spectrum image; a fusion layer (30) for fusing at least a portion of features of the at least one pre- processed full spectrum image with at least one non-spectral data to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and a final decision layer (40) including a multimodal analysis model (411) for analyzing the multimodal input data set to generate a final decision value.

22. The multimodal machine learning system (1) as claimed in claim 21, further comprising a feature selection layer (10) for filtering a plurality of data points of the at least one pre- processed full spectrum image and providing the filtered plurality of data points to the fusion layer (30).

23. The multimodal machine learning system (1) as claimed in claim 21, wherein the preprocessing procedure includes: performing a TSP normalization process on the at least one full spectrum image.

24. The multimodal machine learning system (1) as claimed in claim 23, wherein the preprocessing procedure includes: removing at least a portion of signal of the at least one full spectrum image that does not contain a metabolite reaction.

25. The multimodal machine learning system (1) as claimed in claim 24, wherein the preprocessing procedure includes: removing signal related to water from the signal of the at least one full spectrum image.

26. A multimodal machine learning method performed by a multimodal machine learning system including a pre-processing module (50), a fusion layer (30) and a final decision layer (40), comprising the steps of: performing a pre-processing procedure on at least one full spectrum image by the preprocessing module (50) to form at least one pre-processed full-spectrum image;fusing at least a portion of features of the at least one pre-processed full spectrum image with at least one non-spectral data by the fusion layer (30) to generate a multimodal input data set, wherein the at least one non-spectral data includes at least one clinical data or protein data; and analyzing the multimodal input data set by a multimodal analysis model (411) of the final decision layer (40) to generate a final decision value.

27. The multimodal machine learning method as claimed in claim 26, further comprising the steps of: filtering a plurality of data points of the at least one pre-processed full spectrum image by a feature selection layer (10), and providing the filtered plurality of data points to the fusion layer (30).

28. The multimodal machine learning method as claimed in claim 26, wherein the preprocessing procedure includes: performing a TSP normalization process on the at least one full spectrum image.

29. The multimodal machine learning method as claimed in claim 28, wherein the preprocessing procedure includes: removing at least a portion of signal of the at least one full spectrum image that does not have a metabolite reaction.

30. The multimodal machine learning method as claimed in claim 29, wherein the preprocessing procedure includes: removing signal related to water in the signal of the at least one full spectrum image.

Images