Multi-modal model construction method and system for predicting efficacy of sorafenib in hepatocellular carcinoma

Abstract

The present invention provides a multi-modal model construction method and system for predicting the efficacy of sorafenib in hepatocellular carcinoma. The method comprises: step 1, collecting clinical information of a target patient, and generating a whole slide image; step 2, preprocessing clinical data, and retaining clinical features as input for a multi-modal deep learning model; step 3, preprocessing the whole slide image; step 4, constructing an image model, acquiring patch-level scores of the pathological image on the basis of the preprocessed image and by using different aggregation algorithms, and predicting the score of the whole pathological image to obtain best model features; step 5, constructing a multi-modal model, performing modal fusion on the best model features and the clinical features, and outputting an image-level or patient-level prediction result; and step 6, testing and evaluating the model. The present invention achieves bimodal input of a pathological image and clinical information, fully utilizes the complementarity of the two types of modal data, and thus improves prediction accuracy.

Description

A multimodal model construction method and system for predicting the efficacy of sorafenib in hepatocellular carcinoma Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and system for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma. Background Technology

[0002] Hepatocellular carcinoma (HCC) accounts for 75-85% of primary liver cancers and is one of the most common malignant tumors worldwide. Clinically, most HCC patients are diagnosed at an intermediate or advanced stage, missing the optimal window for surgical treatment. Molecularly targeted therapies have become the main treatment strategy for these patients. Sorafenib is an oral multi-kinase inhibitor with anti-proliferative and anti-angiogenic effects and is currently a first-line treatment for advanced liver cancer that is not treatable surgically. Its indications have been expanded to include: patients with stage C BCLC and stage A or B BCLC who cannot receive radical or local treatment due to disease progression, as well as the prevention of postoperative recurrence. However, the emergence of sorafenib resistance has significantly limited its efficacy. Therefore, effective screening of the population that will benefit from sorafenib treatment is of great importance for prolonging the survival of patients with advanced liver cancer.

[0003] Pathological images, as the gold standard for clinical diagnosis, contain a wealth of cellular-level information, such as the state and quantity of different cell types, the area and proportion of cancerous tissue, etc. The development of whole-slide image (WSI) technology has made computer-aided diagnosis possible, with diagnostic accuracy comparable to or even exceeding that of pathologists. However, its ability to predict deeper information, such as molecular, genetic, or prognostic information, has not yet reached the level of diagnostic or tissue identification.

[0004] Patent document CN113966999A discloses a method, device, and storage medium for predicting sorafenib resistance. The method includes: acquiring liver CT image data from a liver cancer patient and processing the liver CT image data to obtain a training sample dataset; establishing a composite neural network model and training the composite neural network model using the training sample dataset; acquiring liver CT images from the patient again, preprocessing the liver CT images again, and inputting them into the trained composite neural network model for processing to achieve sorafenib resistance prediction. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention.

[0005] Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma.

[0007] The method for constructing a multimodal model for predicting the efficacy of sorafenib in hepatocellular carcinoma, provided by the present invention, includes:

[0008] Step 1: Collect clinical information of the target patient and generate a full-view digital image based on the liver tissue removed during the surgery;

[0009] Step 2: Preprocess the clinical data, retaining clinical features as input for the multimodal deep learning model;

[0010] Step 3: Preprocess the full-view digital image;

[0011] Step 4: Construct an image model. Based on the preprocessed image, use different aggregation algorithms to obtain patch-level scores for the pathological image, predict the score of the entire pathological image, and obtain the optimal model features.

[0012] Step 5: Construct a multimodal model, fuse the optimal model features with clinical features, and output image-level or patient-level prediction results;

[0013] Step 6: Test and evaluate the model to obtain the recurrence risk profile of hepatocellular carcinoma patients after surgery using sorafenib targeted therapy.

[0014] Preferably, clinical data preprocessing includes: deleting variables with missing values ​​exceeding a preset range and variables without individual differences, and filling variables with missing values ​​within a preset range with the mean or median according to their discreteness type;

[0015] Image preprocessing includes: cutting a single pathological image of a patient into patches according to preset standards, performing quality control screening on each patch, including removing images with a background ratio higher than a preset threshold and images with artifacts; before inputting the patch into the multimodal deep learning model, performing staining normalization and data augmentation processing on the patch, including random flipping, rotation, adding Gaussian noise and random erasing.

[0016] Preferably, features belonging to different modalities are fused in spatial dimension by splicing, Hadamard product, and addition.

[0017] Preferably, the aggregation algorithm includes:

[0018] Global average: The recurrence probability is obtained by averaging the recurrence probabilities of all patches belonging to the same pathological image.

[0019] Pre_Score: Perform probability stitching on the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image. The feature dimension of each image is 1×i or 1×j.

[0020] Pre_Feature: For the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image, the feature dimensions of the second-to-last layer of the VGG19 model are concatenated. The second-to-last layer has a dimension of 4096, and the feature dimension of each image is i×4096 or j×4096. Then, the average value is taken on the patch dimension, and the final image feature is 1×4096.

[0021] Preferably, the area under the subject curve and the area under the precision-recall curve are used as primary evaluation criteria for model performance, and accuracy, precision, recall, and F1 score are used as secondary evaluation criteria for model performance.

[0022] The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma, provided by the present invention, includes:

[0023] Module M1: Collects clinical information of the target patient and generates a full-view digital image based on the liver tissue removed during the surgery;

[0024] Module M2: Preprocesses clinical data, retaining clinical features as input for the multimodal deep learning model;

[0025] Module M3: Preprocesses the full-view digital image;

[0026] Module M4: Constructs an image model. Based on the preprocessed image, it uses different aggregation algorithms to obtain patch-level scores for pathological images, predicts the score of the entire pathological image, and obtains the optimal model features.

[0027] Module M5: Constructs a multimodal model, fuses the optimal model features with clinical features, and outputs image-level or patient-level prediction results;

[0028] Module M6: Test and evaluate the model to obtain the recurrence risk profile of patients with hepatocellular carcinoma after surgery using sorafenib targeted therapy.

[0029] Preferably, clinical data preprocessing includes: deleting variables with missing values ​​exceeding a preset range and variables without individual differences, and filling variables with missing values ​​within a preset range with the mean or median according to their discreteness type;

[0030] Image preprocessing includes: cutting a single pathological image of a patient into patches according to preset standards, performing quality control screening on each patch, including removing images with a background ratio higher than a preset threshold and images with artifacts; before inputting the patch into the multimodal deep learning model, performing staining normalization and data augmentation processing on the patch, including random flipping, rotation, adding Gaussian noise and random erasing.

[0031] Preferably, features belonging to different modalities are fused in spatial dimension by splicing, Hadamard product, and addition.

[0032] Preferably, the aggregation algorithm includes:

[0033] Global average: The recurrence probability is obtained by averaging the recurrence probabilities of all patches belonging to the same pathological image.

[0034] Pre_Score: Perform probability stitching on the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image. The feature dimension of each image is 1×i or 1×j.

[0035] Pre_Feature: For the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image, the feature dimensions of the second-to-last layer of the VGG19 model are concatenated. The second-to-last layer has a dimension of 4096, and the feature dimension of each image is i×4096 or j×4096. Then, the average value is taken on the patch dimension, and the final image feature is 1×4096.

[0036] Preferably, the area under the subject curve and the area under the precision-recall curve are used as primary evaluation criteria for model performance, and accuracy, precision, recall, and F1 score are used as secondary evaluation criteria for model performance.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) In addition to using single pathological image data as input, this invention also incorporates patient-level clinical information, realizing dual-modal input, making full use of the complementarity of the two modal data, and improving prediction accuracy;

[0039] (2) In a weakly labeled dataset, this invention compares three methods of aggregating patch-level scores into image-level scores, and no longer simply uses global averaging, thereby improving the classification accuracy of the image model;

[0040] (3) This invention can make timely and accurate predictions of the efficacy of sorafenib in patients with hepatocellular carcinoma after surgery, seize the clinical treatment window, promptly carry out targeted therapy for patients with good prognosis, and promptly select other treatment options for patients with poor prognosis, reduce the risk of postoperative recurrence, and provide a reference for personalized treatment decisions. Attached Figure Description

[0041] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0042] Figure 1 is a schematic diagram of the multimodal deep learning model of the present invention for predicting the efficacy of sorafenib in hepatocellular carcinoma. Detailed Implementation

[0043] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0044] Example 1

[0045] This invention provides a method for constructing a multimodal deep learning model to predict the efficacy of sorafenib in hepatocellular carcinoma, the implementation process of which is as follows:

[0046] (1) Collect patients' clinical data and postoperative full-view digital images (WSI) of the primary liver cancer area to form a dataset. Divide the dataset into a training set and a test set at the patient level in a 7:3 ratio. The latter is used to evaluate the performance of the multimodal model proposed in this invention.

[0047] (2) Preprocessing of clinical and image data: Clinical data needs to be screened and filled with missing values, and Z-Score normalized before inputting into the model to eliminate imbalances caused by the magnitude of absolute values ​​between different variables. Pathological image data needs to be patched before input, with a standard of 224×224 and a stride of 128 to ensure the richness of the input image information. For each patch, those with a background ratio higher than 50% and those with artifacts are removed. In addition, Reinhard staining normalization is performed on the patches to eliminate differences in image color distribution caused by superior staining techniques and equipment. Data augmentation processing is performed on the patches in the training set, including random flipping, rotation, adding Gaussian noise, and random erasing according to a certain probability, but staining enhancement operations are excluded here.

[0048] (3) Image model construction process: As shown in Figure 1, VGG19 pre-trained on ImageNet is used as the deep learning model framework. The output is changed from 1000 to 2 to match the two prognostic states of relapse / non-relapse in patients. End-to-end training is performed, and the patch label is its subordinate image label. Relapse risk prediction is performed on the patch images. After the above preprocessing, each pathological image includes multiple patches. Regarding how to integrate these patch-level scores, three different aggregation methods are compared:

[0049] (a) Global averaging method: For each image, take the average score R of all patches it contains. Ave The final classification result is obtained by comparing it with the threshold t obtained on the training set that maximizes AUROC.

[0050] (b) Pre_Score method: For each image, select the recurrence probability of specific patches (such as the i patches with the highest recurrence risk or the j patches with the lowest recurrence risk) to form features. In this way, each image can be integrated into i-dimensional or j-dimensional features. Then, use a multiple perceptron (MLP) to perform image-level training again to obtain the final recurrence risk.

[0051] (c) Pre_Feature method: Similar to the above method, here a specific patch in the image is selected, but instead of using the recurrence probability to form the feature, the 4096-dimensional feature of the i or j patches in the penultimate layer of the VGG19 architecture is used. In this way, each image can construct an i×4096 or j×4096 feature matrix. The mean of the matrix is ​​taken in the first dimension and compressed into a 1×4096-dimensional feature. Then, the MLP is used to train the image again to obtain the final image-level recurrence risk.

[0052] In the methods described above, the MLPs are all neural networks containing two hidden layers.

[0053] (4) Multimodal model construction: The best-performing model features from the above methods were dimensionality-reduced and then subjected to Hadamard product with the same-dimensional features obtained through mapping from clinical features. This ensured that the performance improvement of the multimodal model was not due to an increase in the number of parameters. Furthermore, for comparison, we designed a set of experiments fusing image features with random data. For patients with multiple pathological images, we applied mean processing to the recurrence risk.

[0054] (5) Test set evaluation: The above methods were tested on data independent of the training set. The area under the curve of the subjects and the area under the curve of the precision-recall curve were used as the main evaluation criteria, and other criteria such as accuracy, precision, and F1 score were used as secondary evaluation criteria to obtain the recurrence risk performance of the present invention in patients with hepatocellular carcinoma who were treated with sorafenib after surgery within five years.

[0055] Example 2

[0056] Example 2 is a preferred example of Example 1.

[0057] I. Research Subjects

[0058] This study included 80 patients with hepatocellular carcinoma who underwent surgical resection and were treated with sorafenib postoperatively between January 2009 and August 2013. Each patient underwent two duplicate samplings, resulting in a total of 158 pathological images. The inclusion and exclusion criteria were as follows:

[0059] (1) The case was diagnosed as hepatocellular carcinoma;

[0060] (2) No history of other tumors at the time of diagnosis;

[0061] (3) All of them underwent surgical removal;

[0062] (4) Follow-up data within five years after surgery is available.

[0063] II. Experimental Methods

[0064] (1) Collect pathological image data and clinical data.

[0065] (2) The pathological images were acquired at high resolution using tissue microarrays (TMA) at 20x optical magnification. After processing, each image was 3352×3352 pixels, and after segmentation, each pathological image contained an average of 426 patches. After quality control, the patches underwent staining normalization and data augmentation on the training set, while only staining normalization was performed on the test set. The clinical information corresponding to the patients, after data processing, included 33 clinical features.

[0066] (3) First, clinical data was used to predict patient relapse, serving as a comparison for subsequent deep learning models. The machine learning methods selected included random forest, logistic regression, and SVM.

[0067] (4) The pathological image model was subjected to three patch aggregation methods: global averaging, Pre_Score, and Pre_Feature. The method and its corresponding specific patch that performed best on the test set were selected. Experiments showed that the AUROC obtained by using the Pre_Feature aggregation method with the top 5 single patch prediction scores on the pathological image was the best, followed by the Pre_Feature aggregation method with the top 25 patches. Considering that 5 patches are a small number and cannot fully represent the prognostic features of the pathological image, and may have a positive prediction tendency, the Pre_Feature aggregation method with the top 25 patches was selected for subsequent fusion with clinical features and prediction. The training process was divided into two steps: patch-level training and image-level training of the multilayer perceptron after saving the patch features. Specifically, in the first step of training, we set Epoch=100, used the cross-entropy loss function, and Adam as the optimizer to optimize the training process. At the same time, we used a learning rate decay strategy, and stipulated that the standard for early termination of training was that the loss on the validation set no longer decreased by more than 10 Epochs. In the second step of training, the loss function was binary classification BCELoss, and other settings were the same as above.

[0068] (5) In the training of the multimodal model, there is a serious imbalance in the number of modal features because the features from the images are 4096-dimensional while the clinical features are only 33-dimensional. Therefore, before fusion, dimensionality reduction was performed on the features from the pathological images, while upsampling was performed on the features from the clinical images. Both dimensionality reduction and upsampling operations were performed using trainable single-layer linear layers, mapping both to the same dimension before performing the Hadamard product, i.e., the dot product of corresponding positions of the two matrices, to complete the fusion of the two modal features. To ensure that the performance improvement brought by the multimodal model is not due to the increase in the number of parameters, the number of hidden layer units in the MLP of the multimodal model was kept consistent with the single-image model, which was 512 and 128 respectively. In addition, to prove that the multimodal model can indeed learn complementary information from clinical features, a model that fuses 33-dimensional random numbers with the image model was added for comparison. The results are shown in Appendix Table 2.

[0069] III. Experimental Results

[0070] (1) The comparison of the three aggregation methods is shown in Table 1. Among them, Pre_Feature performed better than the other two methods and was selected for fusion with clinical features;

[0071] (2) The comparison between the single-modal and multimodal models is shown in Table 2. After adding clinical features as input, the prediction performance is greatly improved. In particular, by comparing the multimodal model with random data, it can be seen that the improvement in the performance of the multimodal model is not due to the increase in the number of parameters, but because the model can indeed learn complementary knowledge from the bimodal data.

[0072] (3) The multimodal model ultimately predicted AUROC=0.804 and AUPR=0.932 at the patient level, which is better than the existing efficacy prediction.

[0073] Table 1: Comparison of three aggregation methods a, b, and c under the VGG19 model.

[0074] Table 2: Comparison of Single-Mode and Multi-Mode Models

[0075] Example 3

[0076] The present invention also provides a multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma. The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma can be implemented by executing the process steps of the multimodal model construction method for predicting the efficacy of sorafenib in hepatocellular carcinoma. That is, those skilled in the art can understand the multimodal model construction method for predicting the efficacy of sorafenib in hepatocellular carcinoma as a preferred embodiment of the multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma.

[0077] The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma according to the present invention includes: Module M1: collecting clinical information of target patients and generating a full-view digital image based on the liver tissue removed during surgery; Module M2: preprocessing the clinical data and retaining clinical features as input to the multimodal deep learning model; Module M3: preprocessing the full-view digital image; Module M4: constructing an image model, using different aggregation algorithms to obtain patch-level scores of pathological images based on the preprocessed images, predicting the score of the entire pathological image, and obtaining the optimal model features; Module M5: constructing a multimodal model, modally fusing the optimal model features with clinical features, and outputting image-level or patient-level prediction results; Module M6: testing and evaluating the model to obtain the recurrence risk performance after sorafenib targeted therapy in hepatocellular carcinoma patients after surgery.

[0078] Clinical data preprocessing includes: removing variables with missing values ​​exceeding a preset range and variables without individual differences; and imputing variables with missing values ​​within a preset range based on their discreteness type using the mean or median.

[0079] Image preprocessing includes: cutting a single pathological image of a patient into patches according to preset standards, performing quality control screening on each patch, including removing images with a background ratio higher than a preset threshold and images with artifacts; before inputting the patch into the multimodal deep learning model, performing staining normalization and data augmentation processing on the patch, including random flipping, rotation, adding Gaussian noise and random erasing.

[0080] Features belonging to different modalities are fused in spatial dimension through splicing, Hadamard product, and addition.

[0081] Aggregation algorithms include:

[0082] Global average: The recurrence probability is obtained by averaging the recurrence probabilities of all patches belonging to the same pathological image.

[0083] Pre_Score: Perform probability stitching on the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image. The feature dimension of each image is 1×i or 1×j.

[0084] Pre_Feature: For the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image, the feature dimensions of the second-to-last layer of the VGG19 model are concatenated. The second-to-last layer has a dimension of 4096, and the feature dimension of each image is i×4096 or j×4096. Then, the average value is taken on the patch dimension, and the final image feature is 1×4096.

[0085] The area under the receiver operating curve (AUC) and the area under the precision-recall curve (ARC) are used as primary evaluation criteria for model performance, while accuracy, precision, recall, and F1 score are used as secondary evaluation criteria.

[0086] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0087] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma, characterized in that, include: Step 1: Collect clinical information of the target patient and generate a full-view digital image based on the liver tissue removed during the surgery; Step 2: Preprocess the clinical data, retaining clinical features as input for the multimodal deep learning model; Step 3: Preprocess the full-view digital image; Step 4: Construct an image model. Based on the preprocessed image, use different aggregation algorithms to obtain patch-level scores for the pathological image, predict the score of the entire pathological image, and obtain the optimal model features. Step 5: Construct a multimodal model, fuse the optimal model features with clinical features, and output image-level or patient-level prediction results; Step 6: Test and evaluate the model to obtain the recurrence risk profile of hepatocellular carcinoma patients after surgery using sorafenib targeted therapy.

2. The method for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma according to claim 1, characterized in that, Clinical data preprocessing includes: removing variables with missing values ​​exceeding a preset range and variables without individual differences; and imputing variables with missing values ​​within a preset range based on their discreteness type using the mean or median. Image preprocessing includes: cutting a single pathological image of a patient into patches according to preset standards, performing quality control screening on each patch, including removing images with a background ratio higher than a preset threshold and images with artifacts; before inputting the patch into the multimodal deep learning model, performing staining normalization and data augmentation processing on the patch, including random flipping, rotation, adding Gaussian noise and random erasing.

3. The method for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma according to claim 1, characterized in that, Features belonging to different modalities are fused in spatial dimension through splicing, Hadamard product, and addition.

4. The method for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma according to claim 1, characterized in that, Aggregation algorithms include: Global average: The recurrence probability is obtained by averaging the recurrence probabilities of all patches belonging to the same pathological image. Pre_Score: Perform probability stitching on the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image. The feature dimension of each image is 1×i or 1×j. Pre_Feature: For the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image, the feature dimensions of the second-to-last layer of the VGG19 model are concatenated. The second-to-last layer has a dimension of 4096, and each image... The feature dimension is i×4096 or j×4096, and then the average value is taken over the patch dimension, resulting in a final image feature of 1×4096.

5. The method for constructing a multimodal model to predict the efficacy of sorafenib in hepatocellular carcinoma according to claim 1, characterized in that, The area under the receiver operating curve (AUC) and the area under the precision-recall curve (ARC) are used as primary evaluation criteria for model performance, while accuracy, precision, recall, and F1 score are used as secondary evaluation criteria.

6. A multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma, characterized in that, include: Module M1: Collects clinical information of the target patient and generates a full-view digital image based on the liver tissue removed during the surgery; Module M2: Preprocesses clinical data, retaining clinical features as input for the multimodal deep learning model; Module M3: Preprocesses the full-view digital image; Module M4: Constructs an image model. Based on the preprocessed image, it uses different aggregation algorithms to obtain patch-level scores for pathological images, predicts the score of the entire pathological image, and obtains the optimal model features. Module M5: Constructs a multimodal model, fuses the optimal model features with clinical features, and outputs image-level or patient-level prediction results; Module M6: Test and evaluate the model to obtain the recurrence risk profile of patients with hepatocellular carcinoma after surgery using sorafenib targeted therapy.

7. The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma according to claim 6, characterized in that, Clinical data preprocessing includes: removing variables with missing values ​​exceeding a preset range and variables without individual differences; and imputing variables with missing values ​​within a preset range based on their discreteness type using the mean or median. Image preprocessing includes: cutting a single pathological image of a patient into patches according to preset standards, performing quality control screening on each patch, including removing images with a background ratio higher than a preset threshold and images with artifacts; before inputting the patch into the multimodal deep learning model, performing staining normalization and data augmentation processing on the patch, including random flipping, rotation, adding Gaussian noise and random erasing.

8. The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma according to claim 6, characterized in that, Features belonging to different modalities are fused in spatial dimension through splicing, Hadamard product, and addition.

9. The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma according to claim 6, characterized in that, Aggregation algorithms include: Global average: The recurrence probability is obtained by averaging the recurrence probabilities of all patches belonging to the same pathological image. Pre_Score: Perform probability stitching on the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image. The feature dimension of each image is 1×i or 1×j. Pre_Feature: For the i patches with the highest recurrence probability or the j patches with the lowest recurrence probability belonging to the same pathological image, the feature dimensions of the second-to-last layer of the VGG19 model are concatenated. The second-to-last layer has a dimension of 4096, and the feature dimension of each image is i×4096 or j×4096. Then, the average value is taken on the patch dimension, and the final image feature is 1×4096.

10. The multimodal model construction system for predicting the efficacy of sorafenib in hepatocellular carcinoma according to claim 6, characterized in that, The area under the receiver operating curve (AUC) and the area under the precision-recall curve (ARC) are used as primary evaluation criteria for model performance, while accuracy, precision, recall, and F1 score are used as secondary evaluation criteria.

Images