A multi-modal medical data fusion modeling method and device based on multi-task cascading

Abstract

The application is a kind of multi-modal medical data fusion modeling method and device based on multi-task cascade, comprising the following steps: learning and modeling a plurality of medical data modalities to obtain a plurality of single-modal medical data models; calibrating the prediction space output of the single-modal medical data model; cross-validation evaluation is carried out on the calibrated single-modal medical data model to obtain first evaluation data; priority strategy is formulated for the first evaluation data, multi-task cascade is carried out to obtain second evaluation data; and multi-modal fusion evaluation is carried out on the second evaluation data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer science and biomedical science, in particular to the field of multi-modal medical data modeling. BACKGROUND

[0002] Multi-omics technologies represented by second-generation gene sequencing have brought new diagnostic methods for single-gene genetic diseases, but the numerous clinical problems of complex diseases such as tumors have not been well solved. An important reason is that it is very difficult to associate high-dimensional gene molecular features with complex macroscopic phenotype features. In the current research mode, the association between the features of the body's inherent cells and organs is lost. With the development of high-content digital pathology technology, high-resolution clinical imaging technology, and multi-spectral imaging technology, the macroscopic image data representing cell and organ features are more abundant. Complex disease clinical research has entered the panoramic multi-modal era from a single modality, which is more in line with real-world clinical practice.

[0003] Electronic medical records, pathological images, and omics data constitute multi-modal data in medical treatment. The integration of these complex data can precisely profile the overall situation of patients from different dimensions and more accurately assess the state of patients, thereby making individualized medical treatment possible. Multi-modal data fusion has been confirmed in many studies to significantly improve the accuracy of disease typing, treatment response, survival prediction, and target prediction of tumors and other diseases. Multi-modal fusion decision algorithms can automatically extract and fuse features using complementary information from multi-modal data, explore and discover new markers and drug targets, and have important scientific significance and clinical value for guiding precise diagnosis and treatment of diseases such as tumors and new drug research and development.

[0004] General multi-modal fusion decision technology and research content includes: 1. Multi-modal representation: mainly studies how to abstract the semantic information contained in multiple modal data (structured data, images, videos, speech, text, etc.) into real-valued vectors; 2. Inter-modal mapping: mainly studies how to map the information in a particular modality data to another modality to discover the mapping relationship of knowledge between different modalities; 3. Alignment: mainly studies how to identify the correlation or correspondence of components and elements between different modalities, so as to make the learned multi-modal representation more accurate and also provide more detailed clues for multi-modal retrieval; 4. Fusion: mainly studies how to integrate models and features between different modalities to obtain consistent, common model outputs and more comprehensive features to improve model robustness; 5. Collaborative learning: mainly studies how to transfer the knowledge learned on information-rich modalities to information-poor modalities to assist each other in learning.

[0005] Multi-modal learning in medical scenarios faces its own challenges, one of which is the imbalance problem, i.e., the incompleteness problem of certain modal data: in the real world, there are various clinical detection methods, but due to the influence of sampling difficulty, economic factors and patient subjective will, the detection received by different patients is biased, resulting in the easy occurrence of missing of certain modal data, which is manifested as imbalance between positive and negative sample data and imbalance between different modal samples on the learning data set; however, general machine learning and deep learning methods cannot be trained under the condition that the features cannot be aligned, and the prediction accuracy of learning and training based on traditional modal splicing methods is low. SUMMARY

[0006] In view of the imbalance problem unique to medical multi-modal, the present application discloses a multi-modal medical data fusion modeling method based on multi-task cascading, comprising the following steps:

[0007] Learning and modeling multiple medical data modalities to obtain multiple single-modal medical data models;

[0008] Calibrating the prediction space output of the single-modal medical data model;

[0009] Cross-validation evaluation is performed on the calibrated single-modal medical data model to obtain first evaluation data;

[0010] A priority strategy is formulated for the first evaluation data, and multi-task cascading is performed to obtain second evaluation data;

[0011] The second evaluation data is subjected to multi-modal fusion.

[0012] In an embodiment of the present application, the step of learning and modeling multiple medical data modalities further comprises:

[0013] Modeling the medical data modal data using a machine learning model and / or a deep learning model;

[0014] Outputting a prediction result.

[0015] In an embodiment of the present application, the step of modeling the medical data modal data using a machine learning model and / or a deep learning model further comprises:

[0016] Different learning models are used for different medical data modalities.

[0017] In an embodiment of the present application, the step of calibrating the prediction space output further comprises:

[0018] Adding an untrustworthy and unpredictable identifier in the prediction result output by the single-modal medical data model.

[0019] In an embodiment of the present application, the step of performing cross-validation evaluation further comprises: using ten-fold, leave-one-out method for verification.

[0020] In an embodiment of the present application, the step of performing cross-validation evaluation further comprises:

[0021] The first evaluation data comprises: prediction coverage, prediction accuracy and ROC curve area, wherein,

[0022]

[0023]

[0024] COV represents prediction coverage, count(0,1) represents the number of cross-validation prediction results that are normal output results; count(0,1,UN,NA) represents the number of cross-validation prediction results containing untrusted and unpredictable identifiers;

[0025] ACG represents prediction accuracy, and correct(0,1) represents the number of correct cross-validation prediction results.

[0026] In an embodiment of the present application, the step of formulating a priority strategy and performing multi-task cascading further comprises:

[0027] According to the numerical value of any item in the first evaluation data, the priority is sorted;

[0028] According to the priority sorting, the prediction tasks are cascaded;

[0029] When the prediction result is an untrusted or unpredictable identifier during the multi-task cascading execution process, the current task prediction is stopped, and the next prediction task is performed.

[0030] In an embodiment of the present application, the step of performing multi-modal fusion on the second evaluation data further comprises:

[0031] The multiple modal sequences in the second evaluation data are combined to obtain combined data;

[0032] The prediction coverage, prediction accuracy and ROC curve area data of the combined data are serially arranged;

[0033] According to the serially arranged data, the multi-modal combination is evaluated, which is used to combine with external decision factors to recommend the most advantageous modal combination.

[0034] The present application also discloses a multi-modal medical data fusion modeling device based on multi-task cascading, comprising:

[0035] A single-modal medical data learning modeling module is configured to learn and model multiple medical data modalities.

[0036] A prediction space output calibration module is configured to calibrate the prediction space output of the single-modal medical data model.

[0037] A cross-validation evaluation module is configured to evaluate the cross-validation of the single-modal medical data model after calibration.

[0038] A multi-task cascading module is configured to prioritize the first evaluation data and perform multi-task cascading.

[0039] A multi-modal fusion evaluation module is configured to combine with external decision factors to recommend the most advantageous modal combination.

[0040] The application further discloses an electronic device, including a processor, a memory, the processor being connected with the memory, the memory storing a computer program, and the computer program being executed by the processor to implement the steps of any one of the above methods.

[0041] The application further discloses a storage medium storing a computer program, and the computer program being executed by the processor to implement the steps of the above method.

[0042] Based on the above, the application fully considers the problem that some modal data is prone to be missing, introduces the concept of prediction space in single-modal data and modeling, limits the model prediction space of different learning tasks, avoids the prediction of a single modal beyond the boundary, and improves the prediction accuracy of a single modal under data missing; the cross-validation performance evaluation between modalities is introduced, different modal cascading priority learning tasks are set under the traction of a target learning task, the cascading relationship definition between single-modal models is realized, the problem that training and prediction cannot be performed due to unbalanced modal data is avoided, the conflict of modal prediction results is solved by using the cascading relationship between modalities, and the accuracy of the final decision is optimized; by using the above cascading relationship, the modal accuracy under different combinations is evaluated by setting the cascading between multiple modalities, and the optimal detection combination is recommended by combining with external decision factors such as clinical detection cost, sampling difficulty, patient care and the like. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 FIG. 1 is a flowchart of a multi-modal medical data fusion modeling method based on multi-task cascading according to an embodiment of the application.

[0044] Figure 2 FIG. 4 is a schematic diagram of the cross-validation evaluation results of different modalities according to an embodiment of the application.

[0045] Figure 3 FIG. 5 is a schematic diagram of multi-modal fusion evaluation according to an embodiment of the application.

[0046] Figure 4 A block diagram of a multi-modal medical data fusion modeling device based on multi-task cascading in an embodiment of the present application.

[0047] Figure 5 A block diagram of an electronic device in an embodiment of the present application.

[0048] Figure 6 A block diagram of a medical device in an embodiment of the present application.

[0049] In the drawings:

[0050] 1: a multi-modal medical data fusion modeling device based on multi-task cascading

[0051] 2: a single-modal medical data learning modeling module

[0052] 3: a prediction space output calibration module

[0053] 4: a cross-validation evaluation module

[0054] 5: a multi-task cascading module

[0055] 6: a multi-modal fusion evaluation module

[0056] 100: an electronic device

[0057] 1001: a processor

[0058] 1002: a memory

[0059] 1003: a computer program

[0060] 200: a medical device DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the description of "one embodiment", "embodiment", "example embodiment" and the like in the specification means that the described embodiment can include specific features, structures or characteristics, but does not necessarily include these specific features, structures or characteristics. In addition, such description does not mean the same embodiment. Further, when a specific feature, structure or characteristic is described in combination with an embodiment, it is shown that such feature, structure or characteristic is combined into other embodiments within the knowledge of those skilled in the art, whether or not it is explicitly described.

[0062] Certain terms are used throughout the description and following claims to refer to particular modules, components or devices. As one skilled in the art will appreciate, the term "module", "component", or "device" can refer to a collection of one or more modules, components or devices. As used herein, the term "exemplary" merely means "serving as an example", "example" and not "preferred" or indicating a "preferred" implementation over other implementations. In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0063] In addition, numerous terms used in the specification and claims, are defined as follows. The singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. "Optional" or "optionally" means that the subsequently described event or circumstance can or can not occur, and that the description includes instances where the event or circumstance occurs and instances where it does not.

[0064] Reference will now be made to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a method for multi-modal medical data fusion modeling based on multi-task cascading according to an embodiment of the present application.

[0065] A method for multi-modal medical data fusion modeling based on multi-task cascading is disclosed, comprising the following steps:

[0066] Step S1: learning and modeling a plurality of medical data modalities to obtain a plurality of single-modal medical data models;

[0067] Step S2: calibrating the prediction space output of the single-modal medical data models;

[0068] Step S3: cross-validation evaluation of the calibrated single-modal medical data models to obtain first evaluation data;

[0069] Step S4: formulating a priority strategy for the first evaluation data, performing multi-task cascading to obtain second evaluation data;

[0070] Step S5: multi-modal fusion of the second evaluation data.

[0071] In an embodiment of the present application, the step of learning and modeling a plurality of medical data modalities further comprises:

[0072] modeling the medical data modality data using a machine learning model and / or a deep learning model;

[0073] Output the prediction result.

[0074] In an embodiment of the present application, the step of modeling the medical data modality data using a machine learning model and / or a deep learning model further comprises:

[0075] Different learning models are used for different medical data modalities.

[0076] The modality information in the medical data includes structured data, images, videos, speech, text, etc. Taking the learning and modeling of three different modalities of data, i.e., variant sites (M), variant signatures (S), and pathological images (P) in medical data, as an example, a multi-modal data fusion model supporting medical decision-making is constructed through the fusion learning of the three types of data.

[0077] First, single-modality learning modeling is performed to learn and model multiple medical data modalities to obtain multiple single-modality medical data models. That is, single-modality learning modeling is performed to model the association between the three modalities of data, i.e., variant sites (M), variant signatures (S), and pathological images (P), and clinical observation events by using a Bayesian probability model, a decision tree, an MLP, or other machine learning or deep learning models. Through cross-validation and other methods, the optimal prediction model in a single modality is obtained. Different models can be selected among different modalities to obtain better prediction performance at the level of a single modality.

[0078] Second, prediction space output calibration is performed. In an embodiment of the present application, the step of performing prediction space output calibration further comprises:

[0079] An untrustworthy and unpredictable identifier is added to the prediction result output by the single-modality medical data model.

[0080] For prediction space output calibration, due to the uniqueness of medical data, part of the data of each training instance may be missing. Therefore, in the original classification task, the output of the model that cannot be reached (i.e., beyond the model space) is increased. For example, the original classification task is {0, 1}, and the output under the new learning model is {0, 1, UN, NA}. Some strongly associated modalities do not have corresponding feature markers, thereby causing the model to be unpredictable or the prediction result to be untrustworthy. Therefore, the label UN is added to represent the uncertain prediction of the model, and the label NA is added to represent the unpredictability caused by the missing modality data. The latter two are considered as unpredictable.

[0081] Please refer to Figure 2 , Figure 2 The figure is a schematic diagram of the results of cross-validation and evaluation of different modalities in an embodiment of the present application.

[0082] Next, cross-validation is performed to evaluate different modalities. To facilitate fusion between different modalities, the performance of the model in each modality needs to be evaluated. For each modality model, methods such as 10-fold reduction and leave-one-out method are used to evaluate the accuracy of the predictable space, assessing the prediction accuracy of different models. Under cross-validation evaluation, the following calculations are performed respectively: Figure 2 The following indicators will be displayed.

[0083] In one embodiment of the present invention, the step of performing cross-validation evaluation further includes: performing validation using the 10-fold, leave-one-out method.

[0084] In one embodiment of the present invention, the step of performing cross-validation evaluation further includes:

[0085] The first evaluation data includes: prediction coverage, prediction accuracy, and the area under the ROC curve, wherein,

[0086]

[0087]

[0088] COV represents the prediction coverage, and count(0,1) indicates that the cross-validation prediction result is a normal output.

[0089] The number of untrusted and unpredictable identifiers in the cross-validation prediction results; count(0,1,UN,NA) represents the number of untrusted and unpredictable identifiers in the cross-validation prediction results.

[0090] ACG represents the prediction accuracy, and correct(0,1) represents the number of correct predictions obtained through cross-validation.

[0091] AUC stands for Area of ​​the ROC curve, used to evaluate the performance of classification models.

[0092] Then, the decision priorities for different modalities are cascaded.

[0093] The step of formulating a priority strategy and cascading multiple tasks further includes:

[0094] Prioritize and sort according to the numerical value of any item in the first evaluation data;

[0095] The prediction tasks are cascaded according to the priority order;

[0096] When the prediction result is identified as unreliable or unpredictable during the multi-task cascading execution process, the current prediction task is stopped, and the next prediction task is started.

[0097] For the above-mentioned three modal data of mutation sites (M), mutation signatures (S), and pathological images (P), the modal decision priority is M > S > P, and the modal with high priority is predicted first, and if a certain modal cannot be predicted, the next level model is predicted by cascade. The modal decision priority M > S > P is sorted from high to low according to the ACC in the cross-validation result, and other cross-validation indicators can also be used for strategy making.

[0098] Next, multi-modal fusion evaluation is performed. Please refer to Figure 3 , Figure 3 is a schematic diagram of multi-modal fusion evaluation in an embodiment of the present application.

[0099] The step of multi-modal fusion of the above-mentioned second evaluation data further comprises:

[0100] Combining the plurality of modal sequences in the above-mentioned second evaluation data to obtain combined data;

[0101] Serially arranging the prediction coverage, prediction accuracy, and ROC curve area data of the combined data;

[0102] According to the serially arranged data, the multi-modal combination is evaluated, which is used to recommend the most advantageous modal combination in combination with external decision factors.

[0103] In the above-mentioned multi-modal fusion evaluation process, 2n-1 modal sequences of N modalities are combined, cascaded and serially arranged, and the effect of the multi-modal combination model is evaluated, and the most advantageous modal combination is recommended. Through the comprehensive consideration of COV, ACC and AUC of MS, MP, SP and MSP, MSP, MP and SP have the highest COV; MS has the highest ACC, followed by MSP; MSP has the highest AUC, followed by MP. Without considering the cost of clinical detection, MSP is preferred, but in the case of considering the cost of clinical detection, MP can be selected. In summary, different modal combinations can be flexibly selected according to the above-mentioned priority strategy considering COV, ACC and AUC to select the most advantageous modal combination.

[0104] Those skilled in the art should understand that medical data has multiple modalities, and this paper takes three modal data of mutation sites (M), mutation signatures (S), and pathological images (P) as an example, and the present application is not limited to this. For other modal data, the above-mentioned method can also be used for multi-task cascaded multi-modal medical data fusion modeling.

[0105] Please refer to Figure 4 , Figure 4 is a block diagram of a multi-task cascaded multi-modal medical data fusion modeling device in an embodiment of the present application.

[0106] The application also discloses a multi-modal medical data fusion modeling device 1 based on multi-task cascading, comprising:

[0107] A single-modal medical data learning modeling module 2 is used for learning modeling of multiple medical data modalities.

[0108] A prediction space output calibration module 3 is used for calibrating the prediction space output of the single-modal medical data model.

[0109] A cross-validation evaluation module 4 is used for cross-validation evaluation of the single-modal medical data model after calibration.

[0110] A multi-task cascading module 5 is used for formulating a priority strategy for the first evaluation data and performing multi-task cascading.

[0111] A multi-modal fusion evaluation module 6 is used for combining with external decision factors to recommend the most advantageous modal combination.

[0112] The application also discloses an electronic device 100, comprising a processor 1001 and a memory 1002, wherein the processor 1001 is connected with the memory 1002, the memory 1002 stores a computer program 1003, and the computer program 1003 realizes the steps of any one of the above methods when executed by the processor 1001.

[0113] Please refer to Figure 5 , Figure 5 which is a block diagram of an electronic device in an embodiment of the application.

[0114] The processor 1001 described above can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor 1001 can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 1001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0115] Please refer to Figure 6 , Figure 6 which is a block diagram of an electronic device in an embodiment of the application.

[0116] The application further discloses a medical device 200, comprising a processor 1001 and a memory 1002, the processor 1001 is connected with the memory 1002, the memory 1002 stores a computer program, the medical device 200 comprises the multi-task cascade-based multi-modal medical data fusion modeling device 1, is used for learning modeling of multi-modal medical data, prediction space output calibration, multi-modal prediction result cross-validation evaluation, multi-task cascade, multi-modal fusion evaluation, and is combined with external decision factors in the disease treatment scheme process, and the most advantageous medical modal combination is recommended.

[0117] The application further discloses a storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the method.

[0118] The computer program executed by the processor can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the technical field.

[0119] To sum up, the application can have other various embodiments, and those skilled in the art can evolve various corresponding changes and modifications according to the application without departing from the spirit and essence of the application, but these corresponding changes and modifications should belong to the protection scope of the application.

Claims

1. A multimodal medical data fusion modeling method based on multi-task cascading, characterized in that, Includes the following steps: Single-modal learning modeling is performed on multiple medical data modalities to obtain multiple single-modal medical data models; The prediction spatial output of the single-modal medical data model is calibrated. Cross-validation was performed on the calibrated single-modal medical data model to obtain the first evaluation data; A priority strategy is established for the first evaluation data, and multiple tasks are cascaded to obtain the second evaluation data. The second evaluation data is then subjected to multimodal fusion evaluation.

2. The method as described in claim 1, characterized in that, The step of learning and modeling multiple medical data modalities further includes: The data of the medical data modality are modeled using machine learning models and / or deep learning models; Output the prediction results.

3. The method as described in claim 2, characterized in that, The step of modeling the medical data modality data using machine learning models and / or deep learning models further includes: Use the same or different learning models for different medical data modalities.

4. The method as described in claim 1, characterized in that, The step of performing prediction spatial output calibration also includes: Add unreliable and unpredictable labels to the prediction results output by the unimodal medical data model.

5. The method as described in claim 1, characterized in that, The steps for cross-validation evaluation further include: using the 10-fold or leave-one-out method for validation.

6. The method as described in claim 1 or 5, characterized in that, The steps for performing cross-validation evaluation also include: The first evaluation data includes: prediction coverage, prediction accuracy, and the area under the ROC curve, wherein, COV represents the prediction coverage, and count(0,1) indicates that the cross-validation prediction result is a normal output. The number; count(0,1,UN,NA) represents the number of untrusted and unpredictable identifiers included in the cross-validation prediction results; ACG represents the prediction accuracy, and correct(0,1) represents the number of correct predictions in the cross-validation.

7. The method as described in claim 6, characterized in that, The step of formulating a priority strategy and cascading multiple tasks further includes: Prioritize and sort according to the numerical value of any item in the first evaluation data; The prediction tasks are cascaded according to the priority order; When the prediction result is identified as unreliable or unpredictable during the multi-task cascading execution process, the current prediction task is stopped, and the next prediction task is started.

8. The method as described in claim 7, characterized in that, The step of performing multimodal fusion on the second evaluation data further includes: Multiple modal sequences in the second evaluation data are combined to obtain combined data; The prediction coverage, prediction accuracy, and ROC curve area data of the combined data are serialized. arrangement; The combination of the multiple modality sequences is evaluated based on the serially arranged data, and is used to recommend the most advantageous modality combination in combination with external decision factors.

9. A multimodal medical data fusion modeling device based on multi-task cascading, used to implement the method as described in any one of claims 1 to 8, characterized in that, include: The single-modal medical data learning and modeling module is used to learn and model multiple medical data modalities. The prediction spatial output calibration module is used to perform prediction spatial output calibration on the single-modal medical data model. The cross-validation evaluation module is used to perform cross-validation evaluation on the calibrated single-modal medical data model. Multi-task cascading module; Used to formulate a priority strategy for the first evaluation data and to perform multi-task cascading; The multimodal fusion evaluation module is used to recommend the most advantageous modal combination by combining it with external decision-making factors.

10. A medical device comprising a processor and a memory, the processor being connected to the memory, the memory storing a computer program, characterized in that, The medical device includes the apparatus as described in claim 9.

11. An electronic device comprising a processor and a memory, the processor being connected to the memory, the memory storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of any one of the methods as described in claims 1-8.

12. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images