Method and device for realizing renal cancer targeted drug postoperative prediction based on multi-modal data, processor and computer readable storage medium thereof

Abstract

The invention relates to a method for realizing kidney cancer targeted drug postoperative prediction based on multi-modal data. The method comprises the following steps: (1) collecting and preprocessing the multi-modal data; (2) carrying out feature extraction on the multi-modal data; (3) fusing and integrating the multi-modal data; (4) constructing a targeted drug curative effect prediction model; (5) training and optimizing the targeted drug curative effect prediction model; and (6) carrying out verification and performance evaluation on the targeted drug curative effect prediction model. The invention also relates to a device for realizing renal cancer targeted drug postoperative prediction based on multi-modal data, a processor and a computer readable storage medium thereof. According to the method, the device, the processor and the computer readable storage medium for realizing the postoperative prediction of the kidney cancer targeted drug based on the multi-modal data, the advanced deep learning method and the data fusion technology are utilized, the curative effect of the postoperative targeted drug of a kidney cancer patient can be accurately predicted, a solid support is provided for personalized treatment, and the clinical application prospect is wide. And the prediction precision and stability are improved.

Description

Technical Field

[0001] The present invention relates to the field of medicine, and particularly to the field of targeted drug therapy. Specifically, it refers to a method, device, processor and computer-readable storage medium for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data. Background Art

[0002] Renal cell carcinoma is a malignant tumor with a relatively high incidence rate. Its pathogenesis is complex and involves multiple molecular and cellular processes. In recent years, with the progress of medical technology, targeted drug therapy has become one of the important treatment methods for renal cell carcinoma. Targeted therapy can specifically inhibit the proliferation and metastasis of tumor cells, greatly improving the survival rate of patients. However, due to significant biological differences among renal cell carcinoma patients, there are large differences in the efficacy and prognosis of the same targeted drug. Therefore, how to accurately predict the response of patients to targeted drugs after surgery and help formulate personalized treatment plans has become a hot issue in the medical community and research field. Currently, most methods for predicting the efficacy of renal cell carcinoma after surgery rely on a single data modality, such as clinical data, gene data, or imaging data. For example, genomic data (such as gene mutations, mRNA expression, etc.) can reflect the molecular characteristics of patients' tumors and can predict the efficacy of drugs to a certain extent. However, its acquisition cost is high, and it is limited by the sample size, making it difficult to construct a prediction model with wide applicability. At the same time, the gene data is single and lacks a comprehensive description of the overall characteristics of the tumor. Medical imaging data (such as CT, MRI) can provide the spatial morphological characteristics and biological phenotypes of tumors, but the extraction of imaging features relies on traditional radiomics methods or shallow machine learning techniques, making it difficult to effectively capture the deep information in the imaging data. Moreover, a single imaging modality often cannot fully reveal the molecular and microscopic characteristics of tumors, resulting in insufficient prediction accuracy. Clinical data (such as patients' age, gender, surgical records, etc.) can provide background information for prediction, but most of them are high-level information and lack specific refined features for tumors, making it difficult to be used alone for accurate prediction. In addition, the integrated analysis of multi-modal data is an important trend in the current medical field. Especially in the prediction of the treatment effect of complex diseases such as renal cell carcinoma, multi-modal data includes imaging data, pathological images, genomic data, and clinical data, etc. Through the integrated analysis of multi-source information, the limitations of single-modal data can be made up for. However, due to significant differences in the sources and characteristics of each modal data (such as data formats, scale differences, and quality problems), cross-modal fusion technology faces challenges such as uneven data distribution and difficulty in effectively mining the associated features between modalities. Existing methods are mostly based on simple splicing or weighted averaging, making it difficult to fully utilize the complementarity between modalities. In addition, although deep learning technology has made certain progress in the fields of medical image analysis and genomic data mining in recent years, the convolutional neural network (CNN) based on deep learning can be used for feature extraction of imaging data, and the recurrent neural network (RNN) can be used for gene sequence analysis. However, these methods still face the problems of performance bottlenecks of single-modal deep learning models and insufficient model interpretability in practical applications, making it difficult to provide reliable reference information for clinicians. Multi-modal data fusion technology combines the advantages of different modal data, can characterize the biological behavior of tumors from multiple perspectives, and significantly improves the prediction ability.For example, by integrating the information reflecting the morphological characteristics of tumors in imaging data and the content revealing the molecular characteristics of tumors in genomic data, the limitations of single-modal data can be compensated; through deep learning models to extract and fuse features from various modal data, more comprehensive tumor features can be obtained, improving the accuracy of the prediction model; comprehensively using the multi-modal data of patients can also construct a prediction model that better fits individual characteristics, providing a scientific basis for the selection of targeted drugs and postoperative treatment. In summary, the current methods for predicting the postoperative efficacy of renal cancer still have obvious deficiencies in terms of accuracy, stability, and practicality. The comprehensive utilization of multi-modal data provides a new solution to solve this problem. However, how to efficiently fuse multi-modal data and construct a prediction model with high precision and high interpretability remains a major difficulty in current research and applications. Summary of the Invention

[0003] The object of the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a method, device, processor, and computer-readable storage medium for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data, which has good effects, good prediction ability, and a relatively wide range of applications.

[0004] To achieve the above object, the method, device, processor, and computer-readable storage medium for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data of the present invention are as follows:

[0005] The method for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data is mainly characterized in that the method includes the following steps:

[0006] (1) Collect multi-modal data and perform preprocessing;

[0007] (2) Extract features from multi-modal data;

[0008] (3) Fuse and integrate multi-modal data;

[0009] (4) Construct a prediction model for the efficacy of targeted drugs;

[0010] (5) Train and optimize the prediction model for the efficacy of targeted drugs;

[0011] (6) Verify and evaluate the performance of the prediction model for the efficacy of targeted drugs.

[0012] Preferably, the step (1) specifically includes the following steps:

[0013] (1.1) Collect multi-modal data of each patient, including imaging data and pathological data;

[0014] (1.2) Perform data preprocessing.

[0015] Preferably, step (1.2) specifically includes the following steps:

[0016] (1.2.1) Adjust the pixel sizes of the images and pathological images;

[0017] (1.2.2) Normalize each image based on the mean and standard deviation;

[0018] (1.2.3) Perform random transformations on the image data.

[0019] Preferably, step (1.2.3) specifically includes the following steps:

[0020] (1.2.3.1) Horizontally flip the image randomly;

[0021] (1.2.3.2) Rotate the image randomly;

[0022] (1.2.3.3) Randomly perturb the brightness, contrast, and saturation.

[0023] Preferably, step (2) specifically includes the following steps:

[0024] (2.1) Extract the features of the image data;

[0025] (2.2) Use the pre-trained ResNet50 model to extract the features of the pathological section images;

[0026] (2.3) Assign the feature extraction task to the GPU or CPU.

[0027] Preferably, step (2.1) specifically includes the following steps:

[0028] (2.1.1) Use the pre-trained ResNet50 model to extract the high-dimensional features of the image data;

[0029] (2.1.2) Use the pre-trained ImageNet weights to optimize the feature extraction effect and fine-tune the classification head.

[0030] Preferably, step (3) specifically includes the following steps:

[0031] (3.1) Concatenate the image features and pathological features into a single feature vector;

[0032] (3.2) Perform Z-score normalization on the concatenated feature vector;

[0033] (3.3) Save the normalized feature vector.

[0034] Preferably, step (4) specifically includes the following steps:

[0035] (4.1) Construct a classification model based on a fully connected network;

[0036] (4.2) Set hyperparameters;

[0037] (4.3) Obtain the loss function.

[0038] Preferably, the step (5) specifically includes the following steps:

[0039] (5.1) Use stratified sampling to divide the data into a training set and a test set, split and sample the data. Calculate the class weights to make the distribution of various samples in the training set and the test set uniform;

[0040] (5.2) Perform training in a loop, load the data, calculate multiple metrics for the training set and the test set in each round, and clear the cached video memory after each step;

[0041] (5.3) Set the learning rate scheduling strategy and perform learning rate scheduling.

[0042] Preferably, the step (6) specifically includes the following steps:

[0043] (6.1) Obtain the loss function and accuracy metrics of the training and test sets;

[0044] (6.2) Judge whether the metrics reach the set threshold. If so, save the model and perform result analysis.

[0045] The device for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data is mainly characterized in that the device includes:

[0046] A processor configured to execute computer-executable instructions;

[0047] A memory storing one or more computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data as described above is implemented.

[0048] The processor for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data is mainly characterized in that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data as described above is implemented.

[0049] The computer-readable storage medium is mainly characterized in that a computer program is stored thereon, and the computer program can be executed by a processor to implement each step of the method for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data as described above.

[0050] The method, device, processor and computer-readable storage medium for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention utilize advanced deep learning methods and data fusion technologies, can accurately predict the efficacy of postoperative targeted drugs for renal cancer patients, provide solid support for personalized treatment, and significantly improve the prediction accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a model structure diagram for extracting image and pathological data features of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention.

[0052] Figure 2 It is a flowchart of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention.

[0053] Figure 3 It is an architecture diagram of the targeted drug efficacy prediction model of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention.

[0054] Figure 4 It is a schematic diagram of the loss function curve in the training process of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention.

[0055] Figure 5 It is a schematic diagram of the accuracy curve in the training process of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention.

[0056] Figure 6 It is an interpretability diagram of the saved model of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention.

[0057] Figure 7 It is a flowchart of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] In order to more clearly describe the technical content of the present invention, the following will be further described in combination with specific embodiments.

[0059] The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data of the present invention includes the following steps:

[0060] (1) Collect multimodal data and perform preprocessing;

[0061] (2) Extract features from the multimodal data;

[0062] (3) Fuse and integrate the multimodal data;

[0063] (4) Construct a targeted drug efficacy prediction model;

[0064] (5) Train and optimize the targeted drug efficacy prediction model;

[0065] (6) Validate and evaluate the performance of the targeted drug efficacy prediction model.

[0066] As a preferred embodiment of the present invention, step (1) specifically includes the following steps:

[0067] (1.1) Collect multimodal data of each patient, including imaging data and pathological data;

[0068] (1.2) Perform data preprocessing.

[0069] As a preferred embodiment of the present invention, step (1.2) specifically includes the following steps:

[0070] (1.2.1) Adjust the pixel size of the imaging and pathological images;

[0071] (1.2.2) Normalize each image based on the mean and standard deviation;

[0072] (1.2.3) Perform random transformation on the image data.

[0073] As a preferred embodiment of the present invention, step (1.2.3) specifically includes the following steps:

[0074] (1.2.3.1) Perform random horizontal flipping on the image;

[0075] (1.2.3.2) Perform random rotation on the image;

[0076] (1.2.3.3) Perform random perturbation on the brightness, contrast, and saturation.

[0077] As a preferred embodiment of the present invention, step (2) specifically includes the following steps:

[0078] (2.1) Extract imaging data features;

[0079] (2.2) Use the pre-trained ResNet50 model to extract the features of the pathological section images;

[0080] (2.3) Assign the feature extraction task to the GPU or CPU.

[0081] As a preferred embodiment of the present invention, step (2.1) specifically includes the following steps:

[0082] (2.1.1) Use the pre-trained ResNet50 model to extract the high-dimensional features of the imaging data;

[0083] (2.1.2) Optimize the feature extraction effect using pre-trained ImageNet weights and fine-tune the classification head. As a preferred embodiment of the present invention, step (3) specifically includes the following steps:

[0084] (3.1) Concatenate the image features and pathological features into a single feature vector;

[0085] (3.2) Perform Z-score normalization on the concatenated feature vector;

[0086] (3.3) Save the normalized feature vector.

[0087] As a preferred embodiment of the present invention, step (4) specifically includes the following steps:

[0088] (4.1) Construct a classification model based on a fully connected network;

[0089] (4.2) Set hyperparameters;

[0090] (4.3) Obtain the loss function.

[0091] As a preferred embodiment of the present invention, step (5) specifically includes the following steps:

[0092] (5.1) Use stratified sampling to divide the data into a training set and a test set, split and sample the data. Calculate the class weights to make the distribution of various samples uniform in the training set and the test set;

[0093] (5.2) Perform training in a loop, load the data, calculate multiple metrics for the training set and the test set in each round, and clear the cache video memory after each step;

[0094] (5.3) Set the learning rate scheduling strategy and perform learning rate scheduling.

[0095] As a preferred embodiment of the present invention, step (6) specifically includes the following steps:

[0096] (6.1) Obtain the loss function and accuracy metrics of the training and test sets;

[0097] (6.2) Determine whether the metrics reach the set threshold. If so, save the model and perform result analysis.

[0098] The device for realizing postoperative prediction of renal cancer targeted drugs based on multi-modal data of the present invention, wherein the device includes:

[0099] A processor configured to execute computer-executable instructions;

[0100] A memory stores one or more computer-executable instructions, which, when executed by the processor, implement each step of the above method for predicting postoperative renal cancer targeted drugs based on multimodal data.

[0101] The processor for predicting postoperative renal cancer targeted drugs based on multimodal data of the present invention, wherein the processor is configured to execute computer-executable instructions, which, when executed by the processor, implement each step of the above method for predicting postoperative renal cancer targeted drugs based on multimodal data.

[0102] The computer-readable storage medium of the present invention stores a computer program thereon, and the computer program can be executed by a processor to implement each step of the above method for predicting postoperative renal cancer targeted drugs based on multimodal data.

[0103] In the specific implementation manner of the present invention, a technique for predicting the postoperative efficacy of renal cancer targeted drugs based on multimodal data is disclosed. This technique collects multimodal data of patients, including CT / MRI image data and pathological section images, and combines data preprocessing, feature extraction, and fusion analysis to effectively solve the limitations of single-modal data in efficacy prediction.

[0104] Aiming at the problems in the above research, in order to solve the problem of multimodal data processing in predicting the efficacy of targeted drugs after renal cancer surgery, a method for predicting efficacy based on multimodal data of images and pathology is provided.

[0105] This method includes the following steps:

[0106] S1. Collection and preprocessing of multimodal data;

[0107] S2. Feature extraction based on deep learning, using a pre-trained ResNet50 model to extract image and pathological features respectively;

[0108] S3. Data fusion and integration, using splicing and standardization techniques to construct a high-dimensional feature vector;

[0109] S4. Design and optimization of a targeted drug efficacy prediction model, improving the prediction performance based on a fully connected network;

[0110] S5. Optimize the model training process through stratified sampling and learning rate scheduling; S6. Evaluate the model performance using indicators such as AUC score.

[0111] A technique for predicting postoperative renal cancer targeted drugs based on multimodal data includes the following steps:

[0112] S1: Collection and preprocessing of multimodal data;

[0113] Further, the specific steps of S1 are as follows:

[0114] S1.1: Collection of multimodal data;

[0115] Collect the multimodal data of each patient, including:

[0116] Imaging data: including no more than 150 CT or MRI scan images, denoted as I image ∈R H×W×C , where H, W, and C are the height, width, and number of channels of the image respectively.

[0117] Pathological data: including no more than 5000 pathological section images, denoted as I pathology ∈R H×W×C .

[0118] S1.2: Data preprocessing:

[0119] S1.2.1: Image size adjustment:

[0120] All imaging and pathological images are uniformly adjusted to a size of 224×224 pixels, and the adjustment formula is:

[0121]

[0122] S1.2.2: Image normalization:

[0123] Based on the mean μ = [0.485, 0.456, 0.406] and standard deviation σ = [0.229, 0.224, 0.225] of ImageNet, each image is normalized according to the following formula:

[0124]

[0125] S1.2.3: Data augmentation:

[0126] Randomly transform the image data, and the specific operations include:

[0127] (1) Random horizontal flipping: Flip the image horizontally with a probability of 0.5:

[0128] I flipped (x, y) = I norm (W - x, y);

[0129] where W is the image width.

[0130] (2) Random rotation: Randomly rotate the image within the range of θ ∈ [-10°, 10°], and the rotation matrix is shown as follows:

[0131]

[0132] Color jittering: randomly perturbing brightness, contrast, and saturation:

[0133] I color = α·I norm + β;

[0134] where α and β are randomly generated adjustment factors.

[0135] S2: Feature extraction of multimodal data:

[0136] Further, the specific steps of S2 are as follows:

[0137] S2.1: Feature extraction of image data:

[0138] Using a pre-trained ResNet50 model to extract high-dimensional features of image data, the process includes:

[0139] Feature extraction:

[0140] F image = ResNet50(I norm )

[0141] where F image ∈ R 2048 represents the high-level semantic feature vector of image data.

[0142] Transfer learning: Using pre-trained ImageNet weights to optimize the feature extraction effect, fixing the weights of the convolutional layer and only fine-tuning the classification head.

[0143] S2.2: Feature extraction of pathological data:

[0144] Using a pre-trained ResNet50 model to extract the features of pathological section images:

[0145] F pathology = ResNet50(I pathology )

[0146] where F pathology ∈ R 2048 represents the high-level feature vector of pathological data.

[0147] S2.3: Computational hardware optimization:

[0148] Assign the feature extraction task to the GPU or CPU;

[0149] S3: Fusion and integration of multimodal data;

[0150] Further, the specific steps of S3 are as follows:

[0151] S3.1: Feature concatenation:

[0152] Concatenate the image features and pathological features into a single feature vector:

[0153] F combined =[F image , F pathology

[0154] where F combined ∈R 4096 .

[0155] S3.2: Feature standardization:

[0156] Perform Z-score standardization on the concatenated feature vector:

[0157]

[0158] where μ i is the mean and σ i is the standard deviation. After the standardization process, the feature values are distributed in a standard normal distribution with a mean of 0 and a variance of 1, which helps to improve the numerical stability of the model.

[0159] S3.3: Feature saving:

[0160] Save the standardized feature vector as a.npy format file.

[0161] S4: Construction of the targeted drug efficacy prediction model;

[0162] Furthermore, the specific steps of S4 are as follows:

[0163] S4.1: Model architecture:

[0164] Construct a classification model KidneyClassifier based on a fully connected network:

[0165] (1) Input layer:

[0166] The input is the fused feature vector F normalized , whose dimension is d input .

[0167] (2) Hidden layer (fc):

[0168] Perform a linear transformation on the input feature vector F normalized , add a bias, and output the result of the hidden layer after passing through LayerNorm and ReLU activation:

[0169] H1 = ReLU(LayerNorm(W1·F normalized + b1))

[0170] ​H2 = ReLU(LayerNorm(W2·H1 + b2))

[0171] H3 = ReLU(LayerNorm(W3·H2 + b3))

[0172] H4 = ReLU(LayerNorm(W4·H3 + b4))

[0173] Where: W 1，2，3，4 is the weight matrix of the hidden layer, b 1，2，3，4 is the bias vector of the hidden layer, H 1，2，3，4 is the output of the hidden layer.

[0174] (3) Output layer:

[0175] The output of the last layer is converted into a categorical probability distribution through the Softmax function:

[0176]

[0177] (4) Regularization and stability:

[0178] Dropout: Add a Dropout operation after the ReLU activation in each layer, which is used to randomly discard a certain proportion of neurons. The formula is:

[0179] H dropout = Dropout(H, p = 0.5);

[0180] where p represents the dropout probability, which is set to p = 0.5.

[0181] LayerNorm: Perform normalization after the output of each layer to ensure the stability of the feature distribution:

[0182]

[0183] where μ is the mean of the features, σ 2 is the variance of the features, and ∈ is a small constant to prevent the denominator from being zero.

[0184] S4.2: Hyperparameter setting:

[0185] Learning rate: 0.0005, using the Adam optimizer to adapt to different parameters during updates.

[0186] Batch size: Set to 8, processing small batches of data each time to make full use of computing resources and avoid out-of-memory errors.

[0187] Number of training epochs: The upper limit is set to 50, aiming to terminate unnecessary training as early as possible to save time and abort prematurely when the metrics reach the standard.

[0188] Gradient accumulation step: After this step, parameter updates are performed to reduce the negative impact of small batch sizes.

[0189] S4.3: Loss function:

[0190] Adopt Focal Loss:

[0191] FL(p t ) = -α t (1 - p t ) γ log(p t )

[0192] where α t is the class weight factor, set to 1, γ is the focusing factor, taking the value of 2, to ensure increased attention to difficult samples.

[0193] S5: Model training and optimization:

[0194] Furthermore, the specific steps of S5 are as follows:

[0195] S5.1: Data splitting and sampling:

[0196] Use stratified sampling to divide the data into a training set and a test set, ensuring that the samples of each class are evenly distributed in the training set and the test set, and use WeightedRandomSampler during training to correct the imbalance of sample classes:

[0197] For the sample set {x1, x2,..., x n}, each sample x i belongs to class y i ∈ {1, 2,..., C}, and its class weight is determined by the following formula:

[0198]

[0199] where: is the number of samples of class y i , ω i is the weight of y i . The fewer the samples of a class, the greater the weight, ensuring that the number of samples of different classes in each batch is more balanced, thereby improving the learning effect of the model on minority classes.

[0200] S5.2: Training loop:

[0201] Load data through DataLoader and implement progress monitoring in combination with tqdm. Calculate multiple metrics such as loss, accuracy, AUC, etc. for the training set and the test set in each round. Clear the cached video memory after each step to optimize resource usage.

[0202] S5.3: Learning Rate Scheduling:

[0203] Use LambdaLR to set the learning rate scheduling strategy. Gradually increase the learning rate through the linear scheduling warm-up stage and gradually decrease it as the training progresses.

[0204] S6: Model Validation and Performance Evaluation:

[0205] Furthermore, the specific steps of S6 are as follows:

[0206] S6.1: Metric Monitoring:

[0207] Obtain the loss function and accuracy metrics of the training and test sets for each epoch. Check whether the performance of the model on the test set reaches the set threshold (0.7). If it meets the standard, save the model.

[0208] S6.2: Model Saving and Result Analysis:

[0209] Save the optimal model when the model performance reaches the set threshold (such as AUC ≥ 0.7). And use the saved model to generate the interpretability graph of the model to detect whether the attention area of the model is effective.

[0210] The technical solution of the present invention is aimed at renal cancer, with the data source being CT / MRI images + pathological section images, and predicting the efficacy of postoperative targeted drugs. This technical solution uses the ResNet50 pre-trained model to extract image and pathological features, adopts feature splicing + Z-score standardization, introduces Focal Loss to solve class imbalance, and uses stratified sampling and learning rate scheduling.

[0211] The technical solution of the present invention discloses an end-to-end deep learning framework, which combines the ResNet50 pre-trained model with a fully connected network to achieve end-to-end learning from multi-modal data to efficacy prediction.

[0212] The technical solution of the present invention deeply mines the pathological section features, uses the pre-trained ResNet50 to extract the high-dimensional features of the pathological sections, and breaks through the subjective limitations of traditional pathological analysis.

[0213] The technical solution of the present invention realizes dynamic learning rate scheduling and video memory management, dynamically adjusts the learning rate during the training process and optimizes the use of video memory, and improves the model training efficiency.

[0214] For the specific implementation solution of this embodiment, reference can be made to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0215] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be referred to the same or similar content in other embodiments.

[0216] It should be noted that in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "a plurality of" refers to at least two.

[0217] Any process or method description shown in the flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0218] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known techniques in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0219] Those of ordinary skill in the art of the present technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0220] In addition, each functional unit in various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0221] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, etc.

[0222] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0223] The method, device, processor, and computer-readable storage medium for realizing the prediction after the operation of renal cancer targeted drugs based on multi-modal data of the present invention are adopted. By using advanced deep learning methods and data fusion technologies, it can accurately predict the efficacy of targeted drugs after the operation of renal cancer patients, provide strong support for personalized treatment, and significantly improve the prediction accuracy and stability.

[0224] In this specification, the present invention has been described with reference to its specific embodiments. However, it is obvious that various modifications and transformations can still be made without departing from the spirit and scope of the present invention. Therefore, the specification and the drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data, characterized in that, The method described above includes the following steps: (1) Collect multimodal data and perform preprocessing; (2) Extract features from the multimodal data; (3) Fuse and integrate the multimodal data; (4) Construct a targeted drug efficacy prediction model; (5) Train and optimize the targeted drug efficacy prediction model; (6) Validate and evaluate the performance of the targeted drug efficacy prediction model.

2. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 1, wherein, The specific steps of step (1) include the following steps: (1.1) Collect multimodal data for each patient, including imaging data and pathological data; (1.2) Perform data preprocessing.

3. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 2, wherein The specific steps of step (1.2) include the following steps: (1.2.1) Adjust the pixel size of the imaging and pathological images; (1.2.2) Normalize each image based on the mean and standard deviation; (1.2.3) Perform random transformation on the image data.

4. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 3, wherein The specific steps of step (1.2.3) include the following steps: (1.2.3.1) Perform random horizontal flipping on the image; (1.2.3.2) Perform random rotation on the image; (1.2.3.3) Perform random perturbation on the brightness, contrast, and saturation.

5. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 1, wherein The specific steps of step (2) include the following steps: (2.1) Extract imaging data features; (2.2) Use the pre-trained ResNet50 model to extract features of the pathological section images; (2.3) Assign the feature extraction task to the GPU or CPU.

6. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 5, wherein The specific steps of step (2.1) include the following steps: (2.1.1) Use the pre-trained ResNet50 model to extract high-dimensional features of the imaging data; (2.1.2) Optimize the feature extraction effect using the pre-trained ImageNet weights and fine-tune the classification head.

7. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 1, wherein The specific steps of step (3) include the following steps: (3.1) Concatenate the imaging features and pathological features into a single feature vector; (3.2) Perform Z-score normalization on the concatenated feature vector; (3.3) Save the normalized feature vector.

8. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 1, wherein, The specific steps of step (4) include the following steps: (4.1) Construct a classification model based on a fully connected network; (4.2) Set hyperparameters; (4.3) Obtain the loss function.

9. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 1, wherein The specific steps of step (5) include the following steps: (5.1) Use stratified sampling to divide the data into a training set and a test set, split and sample the data. Calculate the class weights to make the distribution of various samples in the training set and the test set uniform; (5.2) Perform training in a loop, load the data, calculate multiple metrics for the training set and the test set in each round, and clear the cached video memory after each step; (5.3) Set the learning rate scheduling strategy and perform learning rate scheduling.

10. The method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to claim 1, wherein, The specific steps of step (6) include the following steps: (6.1) Obtain the loss function and accuracy metrics of the training and test sets; (6.2) Judge whether the metrics reach the set threshold. If so, save the model and perform result analysis.

11. A device for realizing postoperative prediction of kidney cancer targeted drugs based on multimodal data, characterized in that, The device described above includes: A processor configured to execute computer-executable instructions; A memory that stores one or more computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to any one of claims 1 to 10 is realized.

12. A processor for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data, characterized in that, The processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to any one of claims 1 to 10 is realized.

13. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to realize each step of the method for realizing postoperative prediction of renal cancer targeted drugs based on multimodal data according to any one of claims 1 to 10.

Images