A tumor classification method and system based on multimodal fusion

Abstract

The present invention proposes a tumor classification method and system based on multimodal fusion, comprising: constructing a multimodal graph based on multimodal images from the same user, where vertices in the multimodal graph are single-frame images within the multimodal image, and edges in the multimodal graph are matching edges between vertices of different modalities; using a tumor classification model to extract and fuse features from all matching edges in the multimodal graph to obtain a confidence score for each edge; selecting and constructing a trusted edge set based on the confidence score of each edge, and performing a weighted fusion of the confidence score of the trusted edge set and the user's clinical information to obtain a tumor recognition result based on the multimodal image map. This system can combine the user's imaging data from different modalities for high-precision tumor classification.

Description

Technical Field

[0001] This invention relates to a tumor classification method and system for multimodal fusion of medical images. Background Art

[0002] A tumor is a neoplasm formed by the proliferation of local tissue cells under the action of various tumorigenic factors. Different tumors can cause different harms to the human body. From the perspective of tumor categories, malignant tumors are extremely harmful to the human body and can even cause the death of patients. Although some benign tumors do not become malignant, rarely recur, and grow slowly, they still pose certain risks to the human body, especially when they grow in vital parts of the body, which can cause serious consequences. From the perspective of different subtypes of tumors, different types of tumors can cause certain harm to different parts of the human body. For example, schwannomas usually cause nerve compression, and lung tumors can cause respiratory diseases, etc. At present, in clinical practice, the classification of tumors is mainly based on the pathological diagnosis results of pathological sections. How to timely predict the type of tumor through patient imaging data at an early stage and formulate corresponding treatment plans is of great clinical significance for patient prognosis.

[0003] Due to the complexity of tumor pathology, even experienced radiologists or clinicians find it difficult to accurately determine the type of tumor a patient has directly from early imaging data. They often need to rely on the pathological diagnosis results of later pathological sections to determine the diagnosis. With the development of computer technology and image processing technology, auxiliary diagnosis technologies based on medical images have received increasing attention. Directly performing artificial intelligence to identify tumor types based on patient imaging data is extremely challenging. Most existing methods analyze tumors based on a single modality and lack effective fusion of multi-modal imaging data from patients. At the same time, there is currently a lack of relevant research and effective technical means to improve model classification performance in the case of small samples.

[0004] Currently, most methods for tumor classification using imaging data are based on machine learning or deep learning. Machine learning methods typically use hand-crafted features or extract image features using specialized medical image analysis software. Machine learning classifiers, such as support vector machines (SVMs) and k-means, are then used to classify tumors. Deep learning methods typically use complete scans or images of tumor regions as input and then classify tumors using models based on convolutional neural networks or attention mechanisms. Deep learning-based classification methods generally achieve better performance than machine learning-based methods.

[0005] Currently, most medical image analysis methods are based on single-plane or single-modality imaging data from a patient, failing to comprehensively and accurately characterize tumor information. In clinical applications, radiologists often utilize different scanning modalities (e.g., CT, MRI), different scanning planes (e.g., axial, sagittal, and coronal), and even different sequences (e.g., T1, T2) when performing tumor diagnosis. Sequences refer to different scanning sequences acquired using different scanning parameters, typically consisting of multiple consecutive frames. In addition, other data types, such as a patient's age and medical history, can be considered multimodal data. Doctors only make a final diagnosis after integrating this multimodal data. However, most current methods lack the ability to correlate and fuse data from different patient modalities, hindering the effective mining and utilization of multimodal data. Furthermore, most current deep learning methods rely on large amounts of labeled data, leaving much room for improvement in recognition performance for classification tasks with limited training data. Summary of the Invention

[0006] This method comprehensively considers patient data from different modalities, correlates and matches them, fully exploring the inherent connections between patients' multimodal data and deriving tumor classifications from these data. Furthermore, it utilizes unsupervised learning to construct a pre-trained model based on a large amount of source domain data and transfers this to downstream tumor classification tasks, further improving the model's classification performance.

[0007] By constructing a graph structure between patient data from different modalities, extracting and fusing features from the edges in the graph, and screening trusted edge sets and integrating clinical information at the decision-making level, this method can fully explore the relationships between imaging data from different modalities and fuse these data with clinical information, significantly improving the classification accuracy of multimodal imaging data. Furthermore, to address the limited number of labeled tumor samples, this method constructs a pre-trained model through unsupervised learning and migrates it to downstream tumor classification tasks, further improving the classification performance and accuracy of the model.

[0008] Specifically, the present invention proposes a tumor classification method based on multimodal fusion, which includes:

[0009] Step 1: Construct a multimodal graph based on multimodal images from the same user. The vertices in the multimodal graph are single-frame images in the multimodal image, and the edges in the multimodal graph are matching edges between vertices with different modalities.

[0010] Step 2: Use the tumor classification model to extract and fuse features of all matching edges in the multimodal graph to obtain the confidence score of each edge;

[0011] Step 3: Based on the confidence of each edge, a credible edge set is selected and constructed, and the confidence of the credible edge set and the user's clinical information are weightedly fused to obtain the tumor recognition result of the multimodal image mapping.

[0012] In the tumor classification method based on multimodal fusion, the training process of the tumor classification model includes:

[0013] Using unlabeled tumor imaging data, a self-supervised learning task is constructed to obtain an initial encoder that extracts common features of tumor images. This initial encoder uses the image reconstruction task as a pre-task and a feature extraction model as the encoder to extract features of the input image. The decoder then recovers the original image based on the features of the input image. The encoder is trained with the goal of extracting key features of the sample to enable the decoder to better restore the image. The trained encoder is then transferred to a multimodal fusion model.

[0014] Determine the tumor type and imaging modality to be identified, obtain multiple sets of multimodal images as training data, each set of training data comes from the same patient, and simultaneously obtain the pathological diagnosis data corresponding to the training data. Detect and locate the tumor area in the original data and store the corresponding tumor area location information;

[0015] According to the pathological diagnosis data, corresponding category information is attached to each edge in the multimodal graph as the label of the edge; the input of the multimodal fusion model is the edge in the multimodal graph, and each branch of the multimodal fusion model extracts features from the image corresponding to each vertex on the edge and fuses them at the feature space level; the fully connected network of the multimodal fusion model is used to predict the fused features, output its predicted probability, and calculate the loss of the model's predicted probability and the label corresponding to the edge through the loss function, and back-propagate the gradient to train the multimodal fusion model, and the trained multimodal fusion model is used as the tumor classification model.

[0016] In the tumor classification method based on multimodal fusion, there are matching edges only between vertices of different modalities in the multimodal graph.

[0017] The tumor classification method based on multimodal fusion, wherein step 3 includes the user's clinical information, including the user's age and medical history, obtaining the probability of the user suffering from each tumor category based on the clinical information, and performing weighted fusion with the confidence of the trusted edge set.

[0018] The present invention also proposes a tumor classification system based on multimodal fusion, which includes:

[0019] A graph construction module is used to construct a multimodal graph based on multimodal images from the same user, where the vertices in the multimodal graph are single-frame images in the multimodal image, and the edges in the multimodal graph are matching edges between vertices with different modalities;

[0020] The feature extraction and fusion module is used to extract and fuse features of all matching edges in the multimodal graph using the tumor classification model to obtain the confidence score of each edge;

[0021] The weighted fusion module is used to select and construct a credible edge set based on the confidence of each edge, and perform weighted fusion of the confidence of the credible edge set and the user's clinical information to obtain the tumor recognition result of the multimodal image mapping.

[0022] The tumor classification system based on multimodal fusion, wherein the training process of the tumor classification model includes:

[0023] Using unlabeled tumor imaging data, a self-supervised learning task is constructed to obtain an initial encoder that extracts common features of tumor images. This initial encoder uses the image reconstruction task as a pre-task and a feature extraction model as the encoder to extract features of the input image. The decoder then recovers the original image based on the features of the input image. The encoder is trained with the goal of extracting key features of the sample to enable the decoder to better restore the image. The trained encoder is then transferred to a multimodal fusion model.

[0024] Determine the tumor type and imaging modality to be identified, obtain multiple sets of multimodal images as training data, each set of training data comes from the same patient, and simultaneously obtain the pathological diagnosis data corresponding to the training data. Detect and locate the tumor area in the original data and store the corresponding tumor area location information;

[0025] According to the pathological diagnosis data, corresponding category information is attached to each edge in the multimodal graph as the label of the edge; the input of the multimodal fusion model is the edge in the multimodal graph, and each branch of the multimodal fusion model extracts features from the image corresponding to each vertex on the edge and fuses them at the feature space level; the fully connected network of the multimodal fusion model is used to predict the fused features, output its predicted probability, and calculate the loss of the model's predicted probability and the label corresponding to the edge through the loss function, and back-propagate the gradient to train the multimodal fusion model, and the trained multimodal fusion model is used as the tumor classification model.

[0026] In the tumor classification system based on multimodal fusion, there are only matching edges between vertices of different modalities in the multimodal graph.

[0027] The tumor classification system based on multimodal fusion, wherein the user's clinical information, including the user's age and medical history, is used to obtain the probability of the user suffering from each tumor category based on the clinical information, and then weightedly fused with the confidence of the trusted edge set.

[0028] The present invention also proposes a storage medium for storing a program for executing any one of the tumor classification methods based on multimodal fusion.

[0029] The present invention also proposes a client for use in any of the multimodal fusion-based tumor classification systems.

[0030] From the above scheme, it can be seen that the advantages of the present invention are:

[0031] The present invention uses deep learning technology to construct a graph structure between different modal data of patients, and then uses a multimodal fusion model to output the probability of each edge in the graph structure to obtain the probability of each tumor category, and combines it with the clinical information statistics module to perform patient-level fusion, and finally outputs the patient-level tumor recognition results. In addition, a pre-training model is constructed through unsupervised learning, and it is migrated to the downstream multimodal fusion model to perform the task of tumor category classification, which can further improve the model's tumor recognition performance. This technology will help to predict the tumor category based on imaging detection data in the early stages of the patient, thereby improving treatment efficiency and prognosis, and can also help improve the doctor's tumor category diagnosis accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall framework diagram of tumor classification based on multimodal imaging in the present invention. DETAILED DESCRIPTION

[0033] Aiming at the low robustness of existing image-based tumor classification technology, a tumor classification method and system based on multimodal fusion for medical images is proposed.

[0034] The present invention proposes a method for patient-level tumor discrimination based on multimodal imaging data and clinical information. First, a graph structure is constructed between the imaging data of different modalities of the patient. The vertices in the graph are single-frame imaging data, and the edges in the graph are matching edges between different modal data. For example, when the vertices are a frame of axial image and a frame of sagittal image, the edge between the two vertices is the matching edge of the two frames of image. Then, a multimodal fusion model is used to extract and fuse features of all matching edges in the graph to obtain the confidence of each edge. In the decision-level fusion stage, a number of credible edge sets are selected through a credible edge set screening strategy. These edge sets can better represent the matching information of tumor information. Then, the confidence of the credible edge set and the patient's clinical information are weightedly fused to obtain the final patient-level tumor recognition result. In addition to the above process, before training the multimodal fusion model, an unsupervised learning method is used to construct a pre-trained model on a large amount of tumor data, and then the trained model is migrated to the multimodal fusion model to further improve the performance of patient-level tumor classification. In order to achieve the above technical effects, such as Figure 1 As shown, the present invention includes the following key technical points:

[0035] Key point 1: Data collection and organization

[0036] First, we need to determine the tumor type to be identified, such as liver cancer, lung cancer, or breast cancer, as well as the imaging modality, such as CT or MRI, and the acquisition plane, such as axial or sagittal. We then determine which patients' data are needed and export their multimodal imaging data from the hospital information system as raw data. We also need to obtain the corresponding pathological diagnostic data for these patients.

[0037] Data curation involves reorganizing and reorganizing raw data according to specific rules to make it more suitable for training and testing deep learning models later on. These rules include integrating imaging data on a patient-by-patient basis, combining imaging data from different time periods for the same patient into a single directory. They also involve renaming the directory and file names of data within a particular patient directory according to specific naming rules. They also involve further summarizing each patient's pathological diagnosis results into specific tumor category labels, which can be binary classifications for benign and malignant tumors, or multi-classification tasks for multiple pathological subtypes. They also involve dividing patient data into training and test sets.

[0038] Key point 2, tumor area labeling

[0039] After collecting and organizing patient imaging data, tumor regions need to be detected and located. This can be done through either AI-based automatic tumor region detection and labeling or manual labeling. Manual labeling can utilize interactive labeling tools, where professionals use rectangular boxes or outlines to frame or outline each tumor region in the patient's imaging data, storing the corresponding tumor region location information. Labeling also involves further summarizing each patient's pathological diagnosis results into specific tumor category labels.

[0040] At the same time, the labeled data are divided into training sets and test sets based on patients.

[0041] Key point 3, construction of pre-training model

[0042] Due to the limited number of labeled training samples for tumor classification, it is difficult to train a robust deep network model. The present invention adopts a pre-training method, uses a large amount of unlabeled image data to learn, constructs a self-supervised learning task, and obtains an initial encoder to extract the common features of the image. All the image data of the tumor except the test set can be used as a training set, or the image data of other tumors can be used as a training set to construct a pre-training model. The present invention takes the image reconstruction task as a pre-task, and uses a specific model as an encoder to extract the features of the input image. The decoder then uses the features extracted by the encoder to restore the original image. The overall training goal is to enable the encoder to extract the key features of the sample so that the decoder can better restore the image. Finally, the trained encoder is migrated to the downstream tumor classification task for fine-tuning.

[0043] Key Point 4: Construction of Multimodal Graph Structure

[0044] In order to exploit the features of different modalities in the training set divided by key point 2, the patient's imaging data of different modalities is used as vertices. These vertices are connected using matching edges to construct a multimodal graph structure. It is important to note that edges only exist between vertices of different modalities, and these edges are all matching edges. Vertices of the same modality are not connected by edges. In addition, each vertex represents a single frame image.

[0045] Key Point 5: Training of Classification Models

[0046] Based on the training set divided in key point 2 and the pre-trained model and multimodal graph structure, the corresponding category information is attached to each edge in the graph structure according to the pathological results. This serves as the edge label. For example, if the patient's tumor category is schwannoma, then all edges in the graph structure will be classified as schwannoma. These edges in the graph structure and their corresponding labels are used for subsequent training of the multimodal fusion model.

[0047] The multimodal fusion model is a multi-branch network. For example, the first branch is a convolutional neural network branch, and the second branch is an attention branch. These branches can also be replaced with other models depending on the task. First, the pre-trained model is migrated to the multimodal fusion model. The input of the multimodal fusion model is the edges in the graph structure. Each branch extracts features from the image corresponding to each vertex on the edge, and then fuses them in the feature space. Next, a fully connected network is used to predict the fused features and output their predicted probabilities. The model's output probabilities and the labels corresponding to the edges are then used to calculate the loss, backpropagate the gradient, and update the multimodal fusion model parameters.

[0048] Key point 6, clinical information statistics module

[0049] The patient's clinical information, such as age and medical history, is helpful for diagnosis. The function of the clinical information statistics module is to convert textual clinical information data into the probability of the patient suffering from the corresponding tumor category. For example, the probability of suffering from a certain type of tumor in different age groups can be statistically calculated based on the patient's age information and the tumor category in the data set.

[0050] Key point 7, model testing

[0051] On the test set divided by key point 2, for each edge in the graph structure, the trained multimodal fusion model is used to obtain the probability of its corresponding category. This is then combined with the clinical information statistics module for patient-level fusion. First, the probabilities of all edges are weighted and summed with the probability of a patient suffering from a certain type of tumor, obtained from the clinical information statistics module, to obtain the new probability of each edge. Based on the probability value of each edge, the edges with the top K probabilities are selected as the credible edge set, i.e., the edges that the model considers to be the most reliable matches. Probabilities are then calculated for these matching edges to obtain the probability of the patient-level tumor category, and the category with the highest probability is selected as the patient's current tumor category.

[0052] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0053] Attach Figure 1 To illustrate the content of the present invention, a tumor classification method and system based on multimodal fusion for medical imaging includes the following steps and components:

[0054] Step 1: Data collection and organization

[0055] Data collection component 101:

[0056] First, the tumor type to be identified (such as liver cancer, lung cancer, or breast cancer) must be determined, as well as the imaging modality (such as CT or MRI) and imaging planes (such as axial and sagittal). Data from multiple patient modalities must be collected. Next, the required patient data must be determined, and the imaging data for these patients must be exported from the hospital information system as raw data. The corresponding pathological diagnosis results for these patients must also be obtained. To prevent the leakage of patient information, the exported data must be declassified. This typically involves anonymizing information such as the patient's name and hospitalization number.

[0057] Data sorting component 102:

[0058] Data organization refers to the re-induction and organization of raw data according to certain rules to make it more suitable for the training and testing of deep network models in the later stage. The rules include the integration of imaging data based on patients, that is, merging imaging examination data of the same patient at different time periods into one directory. The rules also include renaming the directory name and file name of the data under a patient directory according to certain naming rules. Patient imaging data is generally accessed in Dicom file format. Some fields in Dicom metadata can be used as attributes for folder naming. The folder naming format corresponding to different examinations of the same patient can be referred to as follows:

[0059] ID-gender-age-modality-posture-examination information-tumor type

[0060] For example, it contains seven attributes: ID (patient ID), gender (male or female), age (patient's age at the time of examination), modality (CT, MRI, or CT & MRI), posture (also known as plane), such as axial, sagittal, or coronal), and examination information from the Dicom metadata Study description field, including information such as T1, T2, and scan location. This information is stored in a digitally encoded format, for example, the second attribute 1 represents male and 0 represents female. Rules also include further summarizing patient pathology diagnosis results into specific tumor category labels.

[0061] The rules also include dividing the patient data into training and test sets, while ensuring that the patients included in the training and test sets do not overlap.

[0062] Step 2: Data Labeling

[0063] Tumor region annotation component 103:

[0064] After collecting and organizing the data, the tumor area needs to be labeled. This can be done by using an AI-based automatic tumor area detection and area labeling method, or by using a manual labeling method. Artificial intelligence methods, such as pre-labeling some tumor areas, use currently popular target detection frameworks such as FastRCNN to train the target detection model, and then automatically detect and locate the tumor area on the training set. The specific category of the tumor is determined based on the patient's pathological diagnosis results. The specific implementation method of manual labeling is as follows: the doctor observes the patient's image frame. If a tumor is present, a rectangular box is used in the labeling software to mark the tumor area. The specific category of the tumor is not marked. The specific category of the tumor is determined based on the patient's pathological diagnosis results.

[0065] Step 3: Construction of pre-trained model

[0066] Pre-training component 104 based on unsupervised learning:

[0067] Due to the limited number of labeled training samples for tumor classification, it is difficult to train a robust deep network model. The present invention adopts a pre-training method, uses a large amount of unlabeled image data to learn, constructs a self-supervised learning task, obtains an initial encoder, and extracts common features of the image. All the image data of the tumor except the test set can be used as a training set, or the image data of other tumors can be used as a training set to construct a pre-training model. The present invention takes the image reconstruction task as a pre-task, and the image reconstruction task can enable the encoder to better extract the features that play a decisive role in image reconstruction through the supervision of the decoder. Some branch networks of the multimodal fusion model are used as encoders, and upsampling, convolution operations, and upsampling are used as decoders to perform image reconstruction tasks, so that the encoder can extract more general high-level semantic features. It includes three parts: input layer Mask, encoder and decoder, and loss function.

[0068] (a) Input layer Mask

[0069] The input masking layer involves dividing the image into several h×w patches before feeding it into the deep learning model. It then masks out certain areas at a certain ratio, α. This allows the encoder to extract features that, when partial image information is missing, enable the decoder to recover the original image as much as possible. The better the image quality recovered by the decoder, the more representative the features extracted by the encoder are.

[0070] (b) Encoder and Decoder

[0071] The encoder used here is ResNet18, a convolutional neural network branch of the multimodal fusion model. ResNet18 has four stages, each extracting features at different scales. The features from the final stage of ResNet18 serve as the decoder input. To prevent the decoder from retaining strong memories of specific features and causing overfitting, the decoder uses a very simple structure: upsampling, convolution, and upsampling. The purpose of the two upsampling operations is to equalize the decoder output size with the original image size, and the purpose of the convolution operation is to add learnable parameters to the decoder. Finally, the decoder outputs a reconstructed image of the same size as the original image. This image is combined with the original image to calculate the reconstruction loss. The reconstruction loss function is described below.

[0072] (c) Loss function

[0073] To measure the quality of reconstruction, the reconstructed image and the original image can be compared pixel by pixel. The closer the pixel values ​​at the corresponding positions are, the better the reconstruction effect is. Based on this, the mean square error loss is used as the reconstruction loss between the reconstructed image and the original image, as shown in the following formula, where n represents the number of pixels. represents the reconstructed image, and y represents the original image. The better the reconstruction effect, the smaller the value of L. The optimization goal is to make L as small as possible.

[0074]

[0075] Step 4: Construction of multimodal graph structure

[0076] Multimodal data matching component 105 based on graph structure:

[0077] Suppose a patient has imaging data from three different modalities: M1, M2, and M3. M1 contains n frames of images, M2 contains m frames of images, and M3 contains k frames of images. First, the patient's imaging data from different modalities is divided into sets, such as set A for images from modality M1, set B for images from modality M2, and set C for images from modality M3. Each image frame in the set is used as a vertex in a graph to construct a graph structure. The connecting edges in the graph are called matching edges. These matching edges satisfy the relationship expressed by the following formula. That is, matching edges only exist between vertices of data from different modalities. The constructed graph structure contains a total of n × m × k matching edges. It should be noted that while three modalities are used as an example here, the proposed graph structure is not limited to this and can support more modalities.

[0078]

[0079] Step 5: Model training

[0080] Multimodal fusion model training component 106:

[0081] (a) Feature extraction and fusion

[0082] A multimodal fusion model based on attention learning and multiple branches is used to extract features from different modalities separately and then fuse them. First, a shared convolutional neural network branch is used to extract features from tumor areas of different modalities to obtain the modality M n The corresponding feature F n , where the convolutional neural network branch is a general ResNet model. The convolutional neural network branch can extract local features of the image, but lacks global information. Here, an auxiliary branch based on attention is used, modality M n Get feature L through the shared auxiliary branch n . Unlike models based entirely on Transformer, the attention-assisted branch proposed in this paper is very simple. First, the input tumor area is divided into 8×8 patches, and each patch is mapped into a vector space to obtain the encoding of each patch in the vector space. Then, the encoding of all patches in the current batch is normalized, and a multi-head attention module is used to associate global features to obtain patch areas with high response. Finally, the maximum pooling operation is used to retain the area with the largest response value as feature L.

[0083] For mode M n , through the above two branches, the features F will be obtained respectively n , L n , where F n From the convolutional neural network branch, L n From the attention auxiliary branch. Fusion is performed by the following formula, where Represents a channel splicing operation.

[0084]

[0085] Then, H undergoes a convolution operation, a global average pooling operation, and a full connection operation to obtain the final output result and calculate the cross entropy loss.

[0086] (b) Loss function

[0087] Cross entropy loss focuses on optimizing the model through a single edge in the graph, but does not consider the possible intrinsic correlation between different edges from the same patient, and may fall into a suboptimal solution. At the patient level, features from different modalities may have intrinsic similarities, and features of the same type may also have intrinsic similarities. Therefore, starting from the patient level, the modality similarity loss function PMSLoss and the intra-class similarity loss function PiTSLoss are proposed. These two loss functions are not calculated directly through Ground Truth, but by constructing a similarity matrix, which encourages the model to extract features that are as similar as possible for different modalities or the same category at the patient level. Taking PMSLoss as an example, during the training phase, for each Batch, it may contain data from the same patient. Assume that the set of patient IDs contained in the current Batch is {i 1 ,i 2 ,...,i m}∈ID, the features extracted by any branch in the multimodal fusion model for modalities M1 and M2 are T1 and T2. According to the patient ID set, the features extracted by the model for the corresponding samples of each patient can be obtained, as shown in the following formula.

[0088]

[0089] Where m represents the number of patients in a batch, k m -k m-1 Indicates that the id in a batch is i m The number of samples of patients with d is d, and d represents the length of the feature. T1 represents the feature of modality M1, and T2 represents the feature of modality M2. The multiple features of each patient in the modality are averaged by column to obtain the following matrix:

[0090]

[0091] in represents the patient-level modal feature matrix of modality M1, The patient-level modality feature matrix of modality M2 is represented by . According to the following formula, the feature similarity matrix of modality M1 and modality M2 can be calculated.

[0092]

[0093] The size of is m×m, where m is the number of patients in a batch. The diagonal values ​​represent the similarity of the features extracted by the model for the two modalities of the same patient. The optimization goal is to make the diagonal values ​​as close to 1 as possible and the values ​​at other positions as close to 0 as possible. The loss of mean square error is calculated with the unit matrix to obtain the final loss value.

[0094] The calculation process of PiTSLoss is basically the same as that of PMSLoss. The difference is that and After that, the two modal matrices are first averaged bit by bit to obtain a matrix containing both modal eigenvalues The reason for this is that there is already PMSLoss to constrain the model to extract features of different modalities. PiTSLoss only needs to care about the similarity between different categories. After that, perform self-similar matrix operations on it, as shown in the following formula.

[0095]

[0096] in, The value of the matrix diagonal is always 1, indicating the similarity of the matrix itself, and the value of the non-diagonal indicates the similarity between different categories. The optimization goal is to make the value of the non-diagonal as close to 0 as possible, that is, the similarity between the features of different categories is as small as possible. The loss of mean square error is calculated with the unit matrix to obtain the final loss value.

[0097] The final loss value of the multimodal fusion model based on attention learning and multi-branch during training is the average of cross entropy loss, PMSLoss, and PiTSLoss.

[0098] (c) Training parameters

[0099] The multimodal fusion model can be trained in a distributed manner across multiple graphics cards, for example, four 1080Ti graphics cards. The learning rate is 0.0002, stochastic gradient descent (SGD) is used as the optimizer, and the number of iterations is 20 epochs. The tumor regions are resized to 224×224 pixels, and the batch size is 32. The patch size in the auxiliary attention branch is 8, the dimension of the vector space mapping is 768, and the number of attention heads in the multi-head attention module is 4. These parameters can be adjusted accordingly based on the task.

[0100] Step 6: Clinical age information statistics

[0101] Clinical information statistics component 108:

[0102] For all patients in the training set, we can group them according to specific attributes (such as age, medical history) and the type of tumor they suffer from, and calculate the probability that a patient suffers from a certain type of tumor under a certain attribute. Taking age information as an example, assuming that the total number of tumor types in the training set is C n The patients are M, and those aged 30 to 40 are C. nThe number of patients with tumors is K. In the 30-40 age group, patients with C n The probability of tumor is After statistics, we can obtain the probability of patients in each age group suffering from a certain type of tumor.

[0103] Step 7: Testing the Model

[0104] Multimodal fusion model testing component 107 and patient-level fusion component 109:

[0105] During the testing phase, the multimodal fusion model predicts each of these matching edges, determining the probability of each matching edge belonging to a specific tumor type. For the same patient, the decision-layer fusion phase yields a set C, where each element represents the probability of a corresponding matching edge. The goal of the decision-layer fusion is to use set C to determine the patient's overall tumor type.

[0106] In clinical practice, for example, a patient's age is a valuable reference for tumor classification. Clinical information statistics component 108 obtains a set K of relationships between a patient's age and the probability of them having a particular tumor type. Each element in set C is weighted and summed with the probability of the patient having a particular tumor type at their current age using the following formula, where λ1 and λ2 are weight coefficients.

[0107] C i =λ1×C i +λ2×K i

[0108] Next, sort each element in set C from largest to smallest, and extract the top k elements with the highest probability values, forming set S. This represents the top k matching edges that the model considers to be most accurate. The probabilities of each category in these matching edges are then summed to obtain the cumulative probabilities of a particular tumor type across all edges. The tumor category with the largest cumulative sum is then selected as the current patient's tumor category.

[0109] Result output and storage components: 110

[0110] The patient's final tumor identification results are output and saved in a hard disk file to facilitate subsequent comparison of test set results or tumor prediction for new data.

[0111] In the above implementation, during training, a GTX 1080Ti GPU with 12GB of video memory and 64GB of RAM was selected, and an E5-2640v2 CPU was selected. The pre-trained model was trained for 100 epochs, and the multimodal fusion model was trained for 20 epochs, resulting in a pre-trained model and a multimodal fusion model that met convergence requirements.

[0112] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0113] The present invention also proposes a tumor classification system based on multimodal fusion, which includes:

[0114] A graph construction module is used to construct a multimodal graph based on multimodal images from the same user, where the vertices in the multimodal graph are single-frame images in the multimodal image, and the edges in the multimodal graph are matching edges between vertices with different modalities;

[0115] The feature extraction and fusion module is used to extract and fuse features of all matching edges in the multimodal graph using the tumor classification model to obtain the confidence score of each edge;

[0116] The weighted fusion module is used to select and construct a credible edge set based on the confidence of each edge, and perform weighted fusion of the confidence of the credible edge set and the user's clinical information to obtain the tumor recognition result of the multimodal image mapping.

[0117] The tumor classification system based on multimodal fusion, wherein the training process of the tumor classification model includes:

[0118] Using unlabeled tumor imaging data, a self-supervised learning task is constructed to obtain an initial encoder that extracts common features of tumor images. This initial encoder uses the image reconstruction task as a pre-task and a feature extraction model as the encoder to extract features of the input image. The decoder then recovers the original image based on the features of the input image. The encoder is trained with the goal of extracting key features of the sample to enable the decoder to better restore the image. The trained encoder is then transferred to a multimodal fusion model.

[0119] Determine the tumor type and imaging modality to be identified, obtain multiple sets of multimodal images as training data, each set of training data comes from the same patient, and simultaneously obtain the pathological diagnosis data corresponding to the training data. Detect and locate the tumor area in the original data and store the corresponding tumor area location information;

[0120] According to the pathological diagnosis data, corresponding category information is attached to each edge in the multimodal graph as the label of the edge; the input of the multimodal fusion model is the edge in the multimodal graph, and each branch of the multimodal fusion model extracts features from the image corresponding to each vertex on the edge and fuses them at the feature space level; the fully connected network of the multimodal fusion model is used to predict the fused features, output its predicted probability, and calculate the loss of the model's predicted probability and the label corresponding to the edge through the loss function, and back-propagate the gradient to train the multimodal fusion model, and the trained multimodal fusion model is used as the tumor classification model.

[0121] In the tumor classification system based on multimodal fusion, there are only matching edges between vertices of different modalities in the multimodal graph.

[0122] The tumor classification system based on multimodal fusion, wherein the user's clinical information, including the user's age and medical history, is used to obtain the probability of the user suffering from each tumor category based on the clinical information, and then weightedly fused with the confidence of the trusted edge set.

[0123] The present invention also proposes a storage medium for storing a program for executing any one of the tumor classification methods based on multimodal fusion.

[0124] The present invention also proposes a client for use in any of the multimodal fusion-based tumor classification systems.

Claims

1. A tumor classification method based on multimodal fusion, characterized in that: include: Step 1: Construct a multimodal graph based on multimodal images from the same user. The vertices in the multimodal graph are single-frame images in the multimodal image, and the edges in the multimodal graph are matching edges between vertices with different modalities. Step 2: Use the tumor classification model to extract and fuse features of all matching edges in the multimodal graph to obtain the confidence score of each edge; Step 3: Based on the confidence of each edge, a trusted edge set is selected and constructed. The confidence of the trusted edge set and the user's clinical information are weightedly fused to obtain the tumor recognition result of the multimodal image mapping; The training process of the tumor classification model includes: Using unlabeled tumor imaging data, a self-supervised learning task is constructed to obtain an initial encoder that extracts common features of tumor images. This initial encoder uses the image reconstruction task as a pre-task and a feature extraction model as the encoder to extract features of the input image. The decoder then recovers the original image based on the features of the input image. The encoder is trained with the goal of extracting key features of the sample to enable the decoder to better restore the image. The trained encoder is then transferred to a multimodal fusion model. Determine the tumor type and imaging modality to be identified, obtain multiple sets of multimodal images as training data, each set of training data comes from the same patient, and simultaneously obtain the pathological diagnosis data corresponding to the training data. Detect and locate the tumor area in the original data and store the corresponding tumor area location information; According to the pathological diagnosis data, corresponding category information is attached to each edge in the multimodal graph as the label of the edge; the input of the multimodal fusion model is the edge in the multimodal graph, and each branch of the multimodal fusion model extracts features from the image corresponding to each vertex on the edge and fuses them at the feature space level; the fully connected network of the multimodal fusion model is used to predict the fused features, output its predicted probability, and calculate the loss of the model's predicted probability and the label corresponding to the edge through the loss function, and back-propagate the gradient to train the multimodal fusion model, and the trained multimodal fusion model is used as the tumor classification model.

2. The tumor classification method based on multimodal fusion according to claim 1, characterized in that: In this multimodal graph, there are only matching edges between vertices of different modalities.

3. The tumor classification method based on multimodal fusion according to claim 1, characterized in that: Step 3 includes obtaining the user's clinical information, including the user's age and medical history, and obtaining the probability of the user suffering from each tumor category based on the clinical information, and performing weighted fusion with the confidence of the credible edge set.

4. A tumor classification system based on multimodal fusion, characterized in that: include: A graph construction module is used to construct a multimodal graph based on multimodal images from the same user, where the vertices in the multimodal graph are single-frame images in the multimodal image, and the edges in the multimodal graph are matching edges between vertices with different modalities; The feature extraction and fusion module is used to extract and fuse features of all matching edges in the multimodal graph using the tumor classification model to obtain the confidence score of each edge; The weighted fusion module is used to select and construct a credible edge set based on the confidence of each edge, and perform weighted fusion of the confidence of the credible edge set and the user's clinical information to obtain the tumor recognition result of the multimodal image mapping; The training process of the tumor classification model includes: Using unlabeled tumor imaging data, a self-supervised learning task is constructed to obtain an initial encoder that extracts common features of tumor images. This initial encoder uses the image reconstruction task as a pre-task and a feature extraction model as the encoder to extract features of the input image. The decoder then recovers the original image based on the features of the input image. The encoder is trained with the goal of extracting key features of the sample to enable the decoder to better restore the image. The trained encoder is then transferred to a multimodal fusion model. Determine the tumor type and imaging modality to be identified, obtain multiple sets of multimodal images as training data, each set of training data comes from the same patient, and simultaneously obtain the pathological diagnosis data corresponding to the training data. Detect and locate the tumor area in the original data and store the corresponding tumor area location information; According to the pathological diagnosis data, corresponding category information is attached to each edge in the multimodal graph as the label of the edge; the input of the multimodal fusion model is the edge in the multimodal graph, and each branch of the multimodal fusion model extracts features from the image corresponding to each vertex on the edge and fuses them at the feature space level; the fully connected network of the multimodal fusion model is used to predict the fused features, output its predicted probability, and calculate the loss of the model's predicted probability and the label corresponding to the edge through the loss function, and back-propagate the gradient to train the multimodal fusion model, and the trained multimodal fusion model is used as the tumor classification model.

5. The tumor classification system based on multimodal fusion according to claim 4, characterized in that: In this multimodal graph, there are only matching edges between vertices of different modalities.

6. The tumor classification system based on multimodal fusion according to claim 5, characterized in that: The user's clinical information, including the user's age and medical history, is used to obtain the probability of the user suffering from each tumor category, and the probability is weightedly fused with the confidence of the credible edge set.

7. A storage medium for storing a program for executing any one of the tumor classification methods based on multimodal fusion as claimed in claims 1 to 3.

8. A client for use in any one of the multimodal fusion-based tumor classification systems described in claims 4 to 6.

Images