An immune checkpoint inhibitor response prediction system, method, apparatus and storage medium based on multi-modal biologically interpretable AI

Abstract

The application discloses an immune checkpoint inhibitor response prediction system, method and device based on a multi-modal biological interpretable AI and a storage medium, and the system comprises a multi-modal data acquisition module, a data quality control module, a multi-modal semantic coding module, a digital immune microenvironment construction module, a biological knowledge graph reasoning module, a cross-modal causal explanation module, an immune checkpoint inhibitor response prediction module, a treatment strategy recommendation module and a model continuous learning module. Through the fusion of multi-modal information such as pathological images, biological genomics, medical images and clinical data, the digital immune microenvironment model, the biological knowledge graph and the cross-modal causal explanation model are constructed, the comprehensive analysis of the tumor immune microenvironment, the multi-source biological characteristics and the interaction relationship thereof is realized, and the accuracy and reliability of the immune checkpoint inhibitor treatment response prediction are improved.

Description

Technical Field

[0001] This invention relates to the field of immune checkpoint inhibitor response prediction technology, and in particular to an immune checkpoint inhibitor response prediction system, method, device and storage medium based on multimodal biologically interpretable AI. Background Technology

[0002] With the widespread use of immune checkpoint inhibitors such as PD-1, PD-L1, and CTLA-4 in solid tumors, accurately identifying patients who benefit from immunotherapy has become an important research direction in precision medicine. Currently, clinical assessment of patients' response to immune checkpoint inhibitors is mainly based on biomarkers such as PD-L1 expression level, tumor mutation burden (TMB), and microsatellite instability (MSI). However, these indicators usually only reflect information about the local tumor or a single dimension, and it is difficult to comprehensively depict the complex relationship between the tumor immune microenvironment and multi-source biological characteristics. Therefore, there is still considerable room for improvement in predictive accuracy.

[0003] In addition, although existing AI-based immunotherapy response prediction methods can integrate some pathological images or omics information, most of them use black-box deep learning models, which lack clear biological interpretation of the prediction results and make it difficult to reveal the key molecular mechanisms and intercellular interactions that affect the efficacy of immunotherapy. This results in insufficient model credibility and is not conducive to clinical decision support and individualized treatment plan formulation.

[0004] In summary, this application proposes an immune checkpoint inhibitor response prediction system, method, device, and storage medium based on multimodal bio-interpretable AI. Summary of the Invention

[0005] Based on the technical problems existing in the background technology, the present invention proposes an immune checkpoint inhibitor response prediction system, method, device and storage medium based on multimodal biologically interpretable AI.

[0006] This invention proposes an immune checkpoint inhibitor response prediction system based on multimodal biologically interpretable AI, comprising a multimodal data acquisition module, a data quality control module, a multimodal semantic encoding module, a digital immune microenvironment construction module, a biological knowledge graph reasoning module, a cross-modal causal interpretation module, an immune checkpoint inhibitor response prediction module, a treatment strategy recommendation module, and a model continuous learning module.

[0007] The multimodal data acquisition module is connected to the data quality control module and is used to collect patient pathological image data, biological omics data, medical imaging data and clinical data, and form a unified patient dataset.

[0008] The data quality control module is connected to the multimodal semantic coding module and is used to complete data integrity detection, outlier handling, missing value completion, batch effect correction and data standardization.

[0009] The multimodal semantic coding module is connected to the digital immune microenvironment construction module and the biological knowledge graph reasoning module, and is used to extract pathological image features, biological omics features and clinical features, and establish a unified semantic feature representation;

[0010] The digital immune microenvironment construction module is connected to the cross-modal causal interpretation module, and is used to integrate spatial tissue structure information, immune cell spatial distribution information and biological omics information to construct a patient's digital immune microenvironment model.

[0011] The biological knowledge graph reasoning module is connected to the cross-modal causal interpretation module to establish a network of associations between genes, proteins, signaling pathways, immune cells, drugs, and clinical outcomes, and to perform knowledge reasoning to obtain biological association features.

[0012] The cross-modal causal explanation module is connected to the immune checkpoint inhibitor response prediction module and the treatment strategy recommendation module. It is used to integrate digital immune microenvironment features, biological knowledge graph reasoning results and multimodal semantic features to establish a causal reasoning model and analyze the key factors and action pathways that affect the treatment response of immune checkpoint inhibitors.

[0013] The immune checkpoint inhibitor response prediction module is connected to the treatment strategy recommendation module and the treatment strategy recommendation module, and is used to predict the patient's treatment response to immune checkpoint inhibitors, survival benefits and drug resistance risk based on causal reasoning results.

[0014] The treatment strategy recommendation module is connected to the model continuous learning module and is used to generate individualized immunotherapy plans based on response prediction results and knowledge reasoning results.

[0015] The model continuous learning module is used to update model parameters, biological knowledge graph, and causal reasoning model based on patient treatment results and follow-up data.

[0016] Preferably, the multimodal data acquired by the multimodal data acquisition module includes digital pathological slide data, DNA sequencing data, RNA expression data, protein expression data, spatial transcriptome data, medical imaging data, clinical examination data, and patient follow-up data.

[0017] Preferably, the digital immune microenvironment construction module is used to construct a patient's digital immune microenvironment model based on the pathological image features, biological omics features, and clinical features output by the multimodal semantic coding module, combined with the spatial distribution relationship of immune cells and tissue microenvironment information. It then calculates the degree of immune cell infiltration, the degree of immune activation, spatial heterogeneity, and the omics expression status to obtain the digital immune microenvironment index, the calculation formula of which is as follows:

[0018] ;

[0019] in Indicates the digital immune microenvironment index, Indicates the immune cell infiltration index, Indicates the immune activation index, Indicates spatial heterogeneity index, Represents the omics expression index. , , , Let each represent the weight coefficient corresponding to the indicator, and satisfy the following condition: ;

[0020] The Digital Immune Microenvironment Index serves as a quantitative representation of the patient's immune microenvironment status and is output to the cross-modal causal interpretation module for fusion analysis.

[0021] Preferably, the biological knowledge graph reasoning module is used to receive multimodal semantic features output by the multimodal semantic encoding module and digital immune microenvironment index output by the digital immune microenvironment construction module, and construct a heterogeneous biological knowledge graph including genes, proteins, signaling pathways, immune cells, immune checkpoint inhibitors, and clinical outcomes. By calculating the correlation strength between each node, it mines potential biological relationships affecting the treatment response of immune checkpoint inhibitors and outputs biological correlation reasoning features. The formula for calculating the node correlation strength is as follows:

[0022] ;

[0023] in This represents the strength of the association between node i and node j in a knowledge graph. Indicates the biological similarity between two nodes. This indicates the degree of association between two nodes in biological regulatory pathways. This indicates the degree of co-occurrence association between two nodes in a digital immune microenvironment. , , Let represent the weight coefficients of the corresponding indicators, and satisfy the following: Based on the node association strength, biological association inference features are generated, and the biological association inference features are output to the cross-modal causal explanation module.

[0024] Preferably, the cross-modal causal interpretation module is used to receive the digital immune microenvironment index output by the digital immune microenvironment construction module and the biological association reasoning features output by the biological knowledge graph reasoning module. It performs cross-modal fusion of pathological image features, biological omics features, digital immune microenvironment features, and biological association reasoning features to establish a multi-factor causal association model. It calculates the causal contribution of each influencing factor to the immune checkpoint inhibitor treatment response and outputs the key influencing factors and their corresponding causal interpretation results. The formula for calculating the causal contribution is as follows:

[0025] ;

[0026] in This represents the causal contribution of the i-th influencing factor. This represents the importance score of the i-th multimodal semantic feature. This represents the importance score of the i-th numerical immune microenvironment feature. This represents the importance score of the i-th biological association reasoning feature. , , Let represent the fusion weight coefficients of the corresponding features, and satisfy . ;

[0027] Based on the causal contribution, key driving factors affecting the response to immune checkpoint inhibitor therapy are identified, and corresponding causal explanation results are generated and output to the immune checkpoint inhibitor response prediction module.

[0028] This invention also proposes a method for predicting the response of immune checkpoint inhibitors based on multimodal biologically interpretable AI, comprising the following steps:

[0029] S1: Multimodal patient data aggregation: collecting patient pathological images, biological omics data, medical imaging data, and clinical data;

[0030] S2: Multimodal data quality control: Perform quality inspection, standardization processing, and multimodal alignment on the collected data;

[0031] S3: Multimodal semantic feature encoding: Extract pathological image features, biological omics features and clinical features respectively, and establish a unified semantic representation;

[0032] S4: Digital immune microenvironment modeling: Integrating spatial tissue structure information, spatial distribution information of immune cells, and omics information to construct a digital immune microenvironment model;

[0033] S5: Heterogeneous Biological Knowledge Graph Reasoning: Establish biological knowledge graphs and perform knowledge reasoning to obtain biological relationships between genes, proteins, signaling pathways, immune cells, and drugs;

[0034] S6: Cross-modal causal interpretable analysis: Integrating digital immune microenvironment, biological knowledge graph and multimodal semantic features, a cross-modal causal reasoning model is established to obtain the key factors and explanatory pathways affecting the response to immune checkpoint inhibitor therapy;

[0035] S7: Prediction of Immune Checkpoint Inhibitor Efficacy: Based on cross-modal causal inference results, predict patients' response to immune checkpoint inhibitor treatment, survival benefits, and drug resistance risks, and generate treatment response prediction results;

[0036] S8: Model self-learning and continuous optimization: Based on the actual treatment results and follow-up information of patients, the parameters of the multimodal semantic coding model, biological knowledge graph, cross-modal causal reasoning model and response prediction model are updated to achieve continuous model optimization.

[0037] Preferably, the execution logic steps of S4 are as follows:

[0038] S401: Multimodal spatial information mapping: Receives pathological image features, biological omics features and clinical features output by S3, spatially locates tumor regions, immune cell regions and stroma regions in pathological sections, and maps omics expression information to corresponding tissue regions, establishes the correspondence between pathological spatial structure and omics expression, forms a unified spatial biological information dataset, and provides basic data for digital immune microenvironment modeling.

[0039] S402: Spatial distribution analysis of immune cells: Based on the spatial bioinformatics dataset established in S401, the location coordinates and distribution density of different types of immune cells are identified, and the degree of local immune cell aggregation is calculated by combining the spatial proximity relationship between cells to obtain the spatial distribution characteristics of immune cells. Among them, immune cells include CD8⁺T cells, CD4⁺T cells, regulatory T cells, NK cells, dendritic cells and macrophages.

[0040] The formula for calculating the local immune cell aggregation index is: ,in Indicates the immune cell infiltration index, Indicates the total number of immune cells in the pathological area. This represents the spatial proximity weight between the i-th immune cell and the j-th immune cell. This represents the biological weight corresponding to the j-th type of immune cell; different immune cell types have different weights.

[0041] Spatial proximity weights are calculated using a distance decay method, and the calculation formula is as follows: ,in This represents the Euclidean distance between the i-th and j-th immune cells. This represents the spatial decay coefficient, which is used to adjust the range of influence between neighboring cells, ultimately yielding the spatial distribution characteristics of immune cells throughout the entire pathological region.

[0042] S403: Based on the spatial distribution characteristics of immune cells obtained in S402, and combined with pathological tissue structural characteristics and biological omics expression characteristics, the infiltration state, immune activation state, spatial heterogeneity state, and omics expression state of immune cells are quantitatively analyzed to calculate the digital immune microenvironment index. The calculation formula is as follows:

[0043] ;

[0044] in Indicates the digital immune microenvironment index, Indicates the immune cell infiltration index, Indicates the immune activation index, Indicates spatial heterogeneity index, Represents the omics expression index. , , , Let each represent the weight coefficient corresponding to the indicator, and satisfy the following condition: ;

[0045] S404: Construction of Digital Immune Microenvironment Model: Based on the digital immune microenvironment index calculated in S403, combined with the spatial distribution characteristics of immune cells and the topological relationship of pathological tissue, a digital immune microenvironment model of the patient is constructed; the digital immune microenvironment model can reflect the spatial interaction relationship between different immune cells, the interaction relationship between tumor tissue and immune cells, and the state of the local immune microenvironment.

[0046] S405: Digital Immune Microenvironment Feature Output: Based on the digital immune microenvironment model constructed in S404, extract the patient's digital immune microenvironment feature vector and output the digital immune microenvironment feature vector to the biological knowledge graph reasoning module for subsequent biological relationship reasoning and cross-modal causal interpretation analysis.

[0047] Preferably, the specific logical steps of S5 are as follows:

[0048] S501: Heterogeneous biological entity extraction: Receives the multimodal semantic features output by S3 and the digital immune microenvironment features output by S4, performs unified analysis on the gene, protein, signaling pathway, immune cell, immune checkpoint inhibitor and clinical outcome information in patient pathological images, biological omics data and clinical data, and establishes entity nodes respectively to form a heterogeneous biological entity set, providing a node foundation for the construction of biological knowledge graph;

[0049] S502: Construction of Biological Knowledge Graph: Based on the heterogeneous biological entity set obtained in S501, a patient-specific heterogeneous biological knowledge graph is constructed, using gene regulation relationships, protein-protein interaction relationships, signaling pathway relationships, immune cell regulation relationships, and drug action relationships as connecting edges between entities. Furthermore, attribute annotations are applied to each entity node in conjunction with digital immune microenvironment characteristics, forming a knowledge graph model that includes node attributes and relationships. The formula for calculating the association strength between nodes is as follows:

[0050] ;

[0051] in This represents the strength of the association between node i and node j in a knowledge graph. Indicates the biological similarity between two nodes. This indicates the degree of association between two nodes in biological regulatory pathways. This indicates the degree of co-occurrence association between two nodes in a digital immune microenvironment. , , Let represent the weight coefficients of the corresponding indicators, and satisfy the following: ;

[0052] The comprehensive correlation strength between each node is calculated based on the above formula to form a knowledge graph relationship matrix.

[0053] S503: Based on the knowledge graph relationship matrix established in S502, multi-level association reasoning is performed on nodes with association strength exceeding a set threshold to identify potential regulatory relationships between genes, proteins, signaling pathways, immune cells, and immune checkpoint inhibitors. This yields biological association pathways that influence the response to immunotherapy, and the credibility of each association pathway is calculated using the following formula:

[0054] ;

[0055] in Indicates the credibility of biological association pathways. This represents the overall association strength between two adjacent nodes in the path, and its value is derived from the association strength calculated by S502. This indicates the number of nodes contained in the current associated path. If the associated pathway is valid, it is retained as an effective biological regulatory pathway; otherwise, it is removed.

[0056] S504: Key Biological Association Feature Extraction: Based on the biological association paths obtained in S503, the frequency of each biological entity in all effective association paths is statistically analyzed, and the comprehensive contribution value of each biological entity to the immune checkpoint inhibitor treatment response is calculated in combination with the association path credibility. The calculation formula is as follows:

[0057] ;

[0058] in This represents the overall contribution value of the i-th biological entity. This indicates the credibility of the k-th valid association path. This indicates whether the i-th biological entity exists in the k-th associated path. If it exists, the value is 1; otherwise, the value is 0. n represents the total number of valid associated paths.

[0059] Based on the comprehensive contribution value sorted from largest to smallest, key genes, key proteins, key signaling pathways and key immune cells with high correlation to immune checkpoint inhibitor treatment response were screened, and biological correlation inference feature vectors were generated.

[0060] S505: Inference Result Output: Output the biological association inference feature vector generated in S504 to the cross-modal causal explanation module, and use it as an important input to the cross-modal causal explanation model to calculate the causal contribution of each biological feature to the immune checkpoint inhibitor treatment response.

[0061] This invention proposes an immune checkpoint inhibitor response prediction device based on multimodal bio-interpretable AI, comprising a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used to implement the method described above.

[0062] The present invention proposes a computer-readable storage medium, wherein the computer program, when executed by a processor, is used to implement the method described above.

[0063] Compared with existing technologies, the beneficial effects of this invention are:

[0064] 1. By integrating multimodal information such as pathological images, omics, medical imaging, and clinical data, and constructing a digital immune microenvironment model and heterogeneous biological knowledge graph, a comprehensive analysis of the complex relationships between the tumor immune microenvironment and multi-source biological characteristics is achieved. This overcomes the limitations of existing technologies that rely solely on single biomarkers such as PD-L1 expression, tumor mutation burden, or microsatellite instability for efficacy evaluation. It can more comprehensively and accurately predict patients' treatment response to immune checkpoint inhibitors, improving prediction accuracy and clinical applicability.

[0065] 2. By establishing a cross-modal causal explanation model, we integrate and analyze multimodal features, digital immune microenvironment features, and biological knowledge graph reasoning results to quantify the key driving factors and action pathways affecting the response of immune checkpoint inhibitor therapy. We also generate biologically based and interpretable analysis results, which overcomes the problems of insufficient interpretability and low model credibility of existing artificial intelligence prediction models. This provides clinicians with reliable efficacy prediction basis and personalized treatment decision support, and improves the clinical application value of the model.

[0066] This invention integrates multimodal information such as pathological images, omics, medical imaging, and clinical data to construct a digital immune microenvironment model, a biological knowledge graph, and a cross-modal causal explanation model. This enables a comprehensive analysis of the tumor immune microenvironment, multi-source biological characteristics, and their interactions. It not only improves the accuracy and reliability of predicting the response to immune checkpoint inhibitor therapy but also identifies key biological factors and mechanisms of action that affect treatment efficacy, enhancing the interpretability of prediction results and providing a scientific basis for the development of individualized immunotherapy plans and precision medicine decisions. Attached Figure Description

[0067] Figure 1 This is a block diagram of an immune checkpoint inhibitor response prediction system based on multimodal bio-interpretable AI proposed in this invention.

[0068] Figure 2 This is a flowchart of an immune checkpoint inhibitor response prediction method based on multimodal biologically interpretable AI proposed in this invention. Detailed Implementation

[0069] The present invention will be further explained below with reference to specific embodiments.

[0070] Example 1

[0071] Reference Figure 1 This embodiment proposes an immune checkpoint inhibitor response prediction system based on multimodal biologically interpretable AI, including a multimodal data acquisition module, a data quality control module, a multimodal semantic encoding module, a digital immune microenvironment construction module, a biological knowledge graph reasoning module, a cross-modal causal interpretation module, an immune checkpoint inhibitor response prediction module, a treatment strategy recommendation module, and a model continuous learning module.

[0072] The multimodal data acquisition module is connected to the data quality control module and is used to collect patient pathological image data, biological omics data, medical imaging data and clinical data, and form a unified patient dataset. The multimodal data collected by the multimodal data acquisition module includes digital pathological slide data, DNA sequencing data, RNA expression data, protein expression data, spatial transcriptome data, medical imaging data, clinical examination data and patient follow-up data.

[0073] The data quality control module is connected to the multimodal semantic coding module to complete data integrity detection, outlier handling, missing value completion, batch effect correction and data standardization to eliminate distribution differences between different data sources, obtain multimodal data in a unified format, and send the processed data to the multimodal semantic coding module.

[0074] The multimodal semantic coding module is connected to the digital immune microenvironment construction module and the biological knowledge graph reasoning module to extract pathological image features, biological omics features and clinical features, and to establish a unified semantic feature representation.

[0075] Among them, pathological image features include tumor tissue structure features, cell morphology features and spatial distribution features of immune cells; biological omics features include gene mutation features, RNA expression features, protein expression features and spatial transcriptome features; and clinical features include patient basic information, TNM stage, treatment plan and laboratory test indicators. After completing unified semantic encoding, the multimodal semantic features are sent to the digital immune microenvironment construction module and the biological knowledge graph reasoning module, respectively.

[0076] The digital immune microenvironment construction module is connected to the cross-modal causal interpretation module to integrate spatial tissue structure information, immune cell spatial distribution information and biological omics information to construct a patient's digital immune microenvironment model.

[0077] The digital immune microenvironment construction module is used to construct a digital immune microenvironment model of the patient based on the pathological image features, biological omics features, and clinical features output by the multimodal semantic coding module, combined with the spatial distribution relationship of immune cells and tissue microenvironment information. It also calculates the degree of immune cell infiltration, the degree of immune activation, spatial heterogeneity, and omics expression status to obtain the digital immune microenvironment index, the calculation formula of which is as follows:

[0078] ;

[0079] in Indicates the digital immune microenvironment index, Indicates the immune cell infiltration index, Indicates the immune activation index, Indicates spatial heterogeneity index, Represents the omics expression index. , , , Let each represent the weight coefficient corresponding to the indicator, and satisfy the following condition: Based on the calculated digital immune microenvironment index, a digital immune microenvironment model of the patient is constructed, and a digital immune microenvironment feature vector is generated. The digital immune microenvironment index and the digital immune microenvironment feature vector are output to the cross-modal causal interpretation module and sent to the biological knowledge graph reasoning module as the basis for knowledge reasoning.

[0080] The digital immune microenvironment index serves as a quantitative representation of the patient's immune microenvironment status and is output to the cross-modal causal interpretation module for fusion analysis.

[0081] The biological knowledge graph reasoning module is connected to the cross-modal causal interpretation module to establish a network of relationships between genes, proteins, signaling pathways, immune cells, drugs, and clinical outcomes, and to perform knowledge reasoning to obtain biological association characteristics.

[0082] The biological knowledge graph reasoning module receives multimodal semantic features output from the multimodal semantic encoding module and digital immune microenvironment indices output from the digital immune microenvironment construction module. It constructs a heterogeneous biological knowledge graph including genes, proteins, signaling pathways, immune cells, immune checkpoint inhibitors, and clinical outcomes. By calculating the correlation strength between nodes, it mines potential biological relationships influencing the response to immune checkpoint inhibitor therapy and outputs biological correlation reasoning features. The formula for calculating the node correlation strength is as follows:

[0083] ;

[0084] in This represents the strength of the association between node i and node j in a knowledge graph. Indicates the biological similarity between two nodes. This indicates the degree of association between two nodes in biological regulatory pathways. This indicates the degree of co-occurrence association between two nodes in a digital immune microenvironment. , , Let represent the weight coefficients of the corresponding indicators, and satisfy the following: Based on the node association strength, biological association inference features are generated and output to the cross-modal causal explanation module.

[0085] The cross-modal causal explanation module is connected to the immune checkpoint inhibitor response prediction module and the treatment strategy recommendation module. It is used to integrate digital immune microenvironment characteristics, biological knowledge graph reasoning results, and multimodal semantic features to establish a causal reasoning model, analyze the key factors and pathways affecting the treatment response of immune checkpoint inhibitors, and calculate the causal contribution of each influencing factor. The calculation formula is as follows:

[0086] ;

[0087] in This represents the causal contribution of the i-th influencing factor. Represents multimodal semantic feature scoring. Indicates the digital immune microenvironment characteristic score. This represents the score for reasoning features in biological knowledge graphs. , , These represent the fusion weight coefficients of the corresponding features. They are sorted from largest to smallest according to their causal contribution to identify the key driving factors affecting the response to immune checkpoint inhibitor treatment, generate the corresponding causal explanation results, and send the causal explanation results to the immune checkpoint inhibitor response prediction module and the treatment strategy recommendation module.

[0088] The immune checkpoint inhibitor response prediction module is connected to the treatment strategy recommendation module and is used to predict the patient's treatment response to immune checkpoint inhibitors, survival benefits, and drug resistance risk based on causal inference results. The comprehensive response score is calculated using the following formula:

[0089] ;

[0090] in This indicates the overall response score to immune checkpoint inhibitors. Indicates the digital immune microenvironment index, This represents the score for reasoning based on biological knowledge graphs. Indicates the causal contribution score. , , Let represent the weight coefficients of the corresponding indicators, and satisfy . When the overall response score is greater than or equal to the preset response threshold, the patient is determined to have a high probability of responding to immune checkpoint inhibitors; otherwise, the patient is determined to have a high risk of drug resistance, and the prediction result is sent to the treatment strategy recommendation module.

[0091] The treatment strategy recommendation module is connected to the model continuous learning module to generate individualized immunotherapy plans based on response prediction results and knowledge reasoning results, and output recommended drug combinations, recommendation basis and key biomarker information;

[0092] The model continuous learning module receives actual treatment results, imaging review results, and follow-up data from patients, calculates the prediction error between the predicted results and the actual treatment results, and incrementally updates the parameters in the digital immune microenvironment model, biological knowledge graph model, cross-modal causal interpretation model, and immune checkpoint inhibitor response prediction model based on the prediction error, thereby achieving model self-learning optimization. The updated model parameters are redeployed to the digital immune microenvironment construction module, biological knowledge graph reasoning module, cross-modal causal interpretation module, and immune checkpoint inhibitor response prediction module, forming a closed-loop operation mechanism of prediction, feedback, learning, and optimization.

[0093] In this embodiment, the multimodal data acquisition module first completes the acquisition of multi-source medical data of the patient. After unified preprocessing by the data quality control module, the multimodal semantic encoding module extracts unified semantic features and inputs them into the digital immune microenvironment construction module and the biological knowledge graph reasoning module, respectively. The digital immune microenvironment construction module constructs a digital immune microenvironment model of the patient, and the biological knowledge graph reasoning module completes the reasoning of biological relationships. Subsequently, the cross-modal causal interpretation module integrates the above results to generate key causal interpretation information. The immune checkpoint inhibitor response prediction module completes the efficacy prediction based on the causal interpretation results. The treatment strategy recommendation module outputs individualized treatment plans. The model continuous learning module iteratively updates the parameters of each model based on the actual treatment results of the patient, forming a closed-loop operation process of data acquisition, prediction analysis, result feedback and model optimization.

[0094] Example 2

[0095] Reference Figure 2 This embodiment proposes a method for predicting the response of immune checkpoint inhibitors based on multimodal biologically interpretable AI, including the following steps:

[0096] S1: Multimodal patient data aggregation: collecting patient pathological images, biological omics data, medical imaging data, and clinical data;

[0097] The specific logical steps of S1 are as follows:

[0098] S101: Collect digital pathological slide data, omics data, medical imaging data, and clinical diagnosis and treatment data from patients. The digital pathological slide data includes HE staining images and immunohistochemical images; the omics data includes DNA sequencing data, RNA expression data, protein expression data, and spatial transcriptome data; the medical imaging data includes CT images, MRI images, and PET-CT images; and the clinical diagnosis and treatment data includes basic patient information, TNM staging, previous treatment plans, laboratory test results, and follow-up data.

[0099] S102: Based on the patient's unique identifier, data from various sources are uniformly associated, and a unified patient data index is established according to the collection time, examination type, and data source to form a multimodal patient dataset, ensuring consistency and traceability between different modal data;

[0100] S103: Perform an integrity check on the patient multimodal dataset to determine whether the pathological images, biomics, medical images, and clinical data meet the preset data integrity requirements; if there is missing data, generate a data missing marker and send it to S2 for quality control;

[0101] S2: Multimodal data quality control: Perform quality inspection, standardization processing, and multimodal alignment on the collected data;

[0102] The specific logical steps of S2 are as follows:

[0103] S201: Based on the data distribution range corresponding to each modality, identify outliers, duplicate data and noisy data, and determine whether outliers need to be removed or corrected in accordance with medical rules.

[0104] S202: For data with missing values, estimate the missing values ​​based on historical data of similar patients and other modal data of the same patient to complete the data and improve the integrity of multimodal data;

[0105] S203: Standardize pathological images, biological omics data and clinical indicators to eliminate the impact of differences in different acquisition devices, detection platforms and units of measurement;

[0106] S204: Establish a unified timeline based on the data acquisition time of each modality, synchronize the data acquired at different times, form a unified time series patient dataset, and send it to S3;

[0107] S3: Multimodal semantic feature encoding: Extract pathological image features, biological omics features and clinical features respectively, and establish a unified semantic representation;

[0108] The specific logical steps of S3 are as follows:

[0109] S301: Extract tumor tissue structural features, cell morphology features, and spatial distribution features of immune cells from pathological images, and generate semantic vectors for pathological images.

[0110] S302: Extract DNA mutation features, RNA expression features, protein expression features, and spatial transcriptome expression features, and establish a unified omics semantic representation;

[0111] S303: Extract clinical features such as patient age, TNM stage, treatment plan, laboratory test results and follow-up information, and establish clinical semantic representation;

[0112] S304: Mapping pathological image semantics, biological omics semantics, and clinical semantics to a unified feature space yields a unified semantic feature vector, calculated as follows:

[0113] ;

[0114] in Represents a unified semantic feature vector. Representing the semantic features of pathological images, Represents semantic features in biological omics. Indicates clinical semantic features, , , These represent the corresponding feature weights;

[0115] S4: Digital immune microenvironment modeling: Integrating spatial tissue structure information, spatial distribution information of immune cells, and omics information to construct a digital immune microenvironment model;

[0116] S5: Heterogeneous Biological Knowledge Graph Reasoning: Establish biological knowledge graphs and perform knowledge reasoning to obtain biological relationships between genes, proteins, signaling pathways, immune cells, and drugs;

[0117] S6: Cross-modal causal interpretable analysis: Integrating digital immune microenvironment, biological knowledge graph and multimodal semantic features, a cross-modal causal reasoning model is established to obtain the key factors and explanatory pathways affecting the response to immune checkpoint inhibitor therapy;

[0118] S7: Prediction of Immune Checkpoint Inhibitor Efficacy: Based on cross-modal causal inference results, predict patients' response to immune checkpoint inhibitor treatment, survival benefits, and drug resistance risks, and generate treatment response prediction results;

[0119] The specific logical steps of S7 are as follows:

[0120] S701: Receives the key driving factors, explanatory paths, and causal contributions output by S6, and combines them with the digital immune microenvironment index and knowledge graph reasoning features to form a set of efficacy prediction features;

[0121] S702: Calculate the patient's comprehensive response score based on the efficacy prediction feature set. The calculation formula is as follows:

[0122] ;

[0123] in This indicates the overall response score. Indicates the digital immune microenvironment index, This represents the score for knowledge graph reasoning. Indicates cross-modal causal contribution score, , , These represent the corresponding weight coefficients;

[0124] S703: Based on the comparison of the comprehensive response score with the preset threshold, determine whether the patient can benefit from immune checkpoint inhibitor therapy, and predict the patient's objective response rate, survival benefit and drug resistance risk level.

[0125] S704: Outputs the patient's treatment response probability, survival benefit prediction results, drug resistance risk level, and recommended treatment basis, and sends them to the treatment strategy recommendation module;

[0126] S8: Model self-learning and continuous optimization: Based on the actual treatment results and follow-up information of patients, the parameters of the multimodal semantic coding model, biological knowledge graph, cross-modal causal reasoning model and response prediction model are updated to achieve continuous model optimization;

[0127] The specific logical steps of S8 are as follows:

[0128] S801: Collect post-treatment imaging results, laboratory test results, clinical efficacy evaluation results, and follow-up data of patients to establish a real efficacy dataset;

[0129] S802: Compare the predicted results with the actual treatment results, and calculate the model prediction error. The calculation formula is as follows:

[0130] ;

[0131] in Indicates the average prediction error. This represents the predicted response score for the i-th patient. This represents the actual treatment outcome for the i-th patient. Indicates the number of patients;

[0132] S803: Based on the prediction error, the parameters in the multimodal semantic coding model, biological knowledge graph model, cross-modal causal reasoning model and response prediction model are adaptively updated, and the weight parameters in the digital immune microenvironment model are updated simultaneously to improve the model prediction performance.

[0133] S804: Redeploy the updated model to S3, S4, S5, S6 and S7, continue to predict efficacy based on new patient data, realize continuous model learning, continuous optimization and dynamic iteration, and form a complete prediction-validation-feedback-optimization closed-loop operation mechanism.

[0134] In this embodiment, the execution logic steps of S4 are as follows:

[0135] S401: Multimodal spatial information mapping: Receives pathological image features, biological omics features and clinical features output by S3, spatially locates tumor regions, immune cell regions and stroma regions in pathological sections, and maps omics expression information to corresponding tissue regions, establishes the correspondence between pathological spatial structure and omics expression, forms a unified spatial biological information dataset, and provides basic data for digital immune microenvironment modeling.

[0136] S402: Spatial distribution analysis of immune cells: Based on the spatial bioinformatics dataset established in S401, the location coordinates and distribution density of different types of immune cells are identified, and the degree of local immune cell aggregation is calculated by combining the spatial proximity relationship between cells to obtain the spatial distribution characteristics of immune cells. Among them, immune cells include CD8⁺T cells, CD4⁺T cells, regulatory T cells, NK cells, dendritic cells and macrophages.

[0137] The formula for calculating the local immune cell aggregation index is: ,in Indicates the immune cell infiltration index, Indicates the total number of immune cells in the pathological area. This represents the spatial proximity weight between the i-th immune cell and the j-th immune cell. This represents the biological weight corresponding to the j-th type of immune cell; different immune cell types have different weights.

[0138] Spatial proximity weights are calculated using a distance decay method, and the calculation formula is as follows: ,in This represents the Euclidean distance between the i-th and j-th immune cells. This represents the spatial decay coefficient, which is used to adjust the range of influence between neighboring cells, ultimately yielding the spatial distribution characteristics of immune cells throughout the entire pathological region.

[0139] S403: Based on the spatial distribution characteristics of immune cells obtained in S402, and combined with pathological tissue structural characteristics and biological omics expression characteristics, the infiltration state, immune activation state, spatial heterogeneity state, and omics expression state of immune cells are quantitatively analyzed to calculate the digital immune microenvironment index. The calculation formula is as follows:

[0140] ;

[0141] in Indicates the digital immune microenvironment index, Indicates the immune cell infiltration index, Indicates the immune activation index, Indicates spatial heterogeneity index, Represents the omics expression index. , , , Let each represent the weight coefficient corresponding to the indicator, and satisfy the following condition: ;

[0142] S404: Construction of Digital Immune Microenvironment Model: Based on the digital immune microenvironment index calculated in S403, combined with the spatial distribution characteristics of immune cells and the topological relationship of pathological tissue, a digital immune microenvironment model of the patient is constructed; the digital immune microenvironment model can reflect the spatial interaction relationship between different immune cells, the interaction relationship between tumor tissue and immune cells, and the state of the local immune microenvironment.

[0143] S405: Digital Immune Microenvironment Feature Output: Based on the digital immune microenvironment model constructed in S404, extract the digital immune microenvironment feature vector of the patient, and output the digital immune microenvironment feature vector to the biological knowledge graph reasoning module for subsequent biological relationship reasoning and cross-modal causal interpretation analysis.

[0144] In this embodiment, the specific logical steps of S5 are as follows:

[0145] S501: Heterogeneous biological entity extraction: Receives the multimodal semantic features output by S3 and the digital immune microenvironment features output by S4, performs unified analysis on the gene, protein, signaling pathway, immune cell, immune checkpoint inhibitor and clinical outcome information in patient pathological images, biological omics data and clinical data, and establishes entity nodes respectively to form a heterogeneous biological entity set, providing a node foundation for the construction of biological knowledge graph;

[0146] S502: Construction of Biological Knowledge Graph: Based on the heterogeneous biological entity set obtained in S501, a patient-specific heterogeneous biological knowledge graph is constructed, using gene regulation relationships, protein-protein interaction relationships, signaling pathway relationships, immune cell regulation relationships, and drug action relationships as connecting edges between entities. Furthermore, attribute annotations are applied to each entity node in conjunction with digital immune microenvironment characteristics, forming a knowledge graph model that includes node attributes and relationships. The formula for calculating the association strength between nodes is as follows:

[0147] ;

[0148] in This represents the strength of the association between node i and node j in a knowledge graph. Indicates the biological similarity between two nodes. This indicates the degree of association between two nodes in biological regulatory pathways. This indicates the degree of co-occurrence association between two nodes in a digital immune microenvironment. , , Let represent the weight coefficients of the corresponding indicators, and satisfy the following: ;

[0149] The comprehensive correlation strength between each node is calculated based on the above formula to form a knowledge graph relationship matrix.

[0150] S503: Based on the knowledge graph relationship matrix established in S502, multi-level association reasoning is performed on nodes with association strength exceeding a set threshold to identify potential regulatory relationships between genes, proteins, signaling pathways, immune cells, and immune checkpoint inhibitors. This yields biological association pathways that influence the response to immunotherapy, and the credibility of each association pathway is calculated using the following formula:

[0151] ;

[0152] in Indicates the credibility of biological association pathways. This represents the overall association strength between two adjacent nodes in the path, and its value is derived from the association strength calculated by S502. This indicates the number of nodes contained in the current associated path. If the associated pathway is valid, it is retained as an effective biological regulatory pathway; otherwise, it is removed.

[0153] S504: Key Biological Association Feature Extraction: Based on the biological association paths obtained in S503, the frequency of each biological entity in all effective association paths is statistically analyzed, and the comprehensive contribution value of each biological entity to the immune checkpoint inhibitor treatment response is calculated in combination with the association path credibility. The calculation formula is as follows:

[0154] ;

[0155] in This represents the overall contribution value of the i-th biological entity. This indicates the credibility of the k-th valid association path. This indicates whether the i-th biological entity exists in the k-th associated path. If it exists, the value is 1; otherwise, the value is 0. n represents the total number of valid associated paths.

[0156] Based on the comprehensive contribution value sorted from largest to smallest, key genes, key proteins, key signaling pathways and key immune cells with high correlation to immune checkpoint inhibitor treatment response were screened, and biological correlation inference feature vectors were generated.

[0157] S505: Inference Result Output: Output the biological association inference feature vector generated in S504 to the cross-modal causal explanation module, and use it as an important input to the cross-modal causal explanation model to calculate the causal contribution of each biological feature to the immune checkpoint inhibitor treatment response.

[0158] In this embodiment, the specific logical steps of S6 are as follows:

[0159] S601: Cross-modal feature fusion: Receives the multimodal semantic features output by S3, the digital immune microenvironment features output by S4, and the biological association inference features output by S5. Performs unified mapping on pathological image features, biological omics features, clinical features, digital immune microenvironment features, and biological association inference features, and establishes a cross-modal fusion feature space to form a cross-modal causal analysis dataset, providing a unified data foundation for subsequent causal inference.

[0160] S602: Construction of Cross-Modal Causality: Based on the cross-modal causal analysis dataset obtained in S601, a cross-modal causal relationship model is established using multimodal semantic features, digital immune microenvironment features, and biological association inference features as causal variables, and the response to immune checkpoint inhibitor treatment as the target variable. The causal contribution of each influencing factor is calculated using the following formula:

[0161] ;

[0162] in This represents the causal contribution of the i-th influencing factor. Represents multimodal semantic feature scoring. Indicates the digital immune microenvironment characteristic score. This represents the score for reasoning features in biological knowledge graphs. , , These represent the fusion weight coefficients of the corresponding features. They are sorted from largest to smallest according to their causal contribution to identify the key driving factors affecting the response to immune checkpoint inhibitor treatment, generate the corresponding causal explanation results, and send the causal explanation results to the immune checkpoint inhibitor response prediction module and the treatment strategy recommendation module.

[0163] S603: Key Driver Identification: Based on the causal contribution calculated in S602, all influencing factors are sorted from high to low according to their contribution, and influencing factors with a causal contribution higher than a preset threshold are selected as key drivers; among them, key drivers include key genes, key proteins, key signaling pathways, key immune cells, and key pathological regions.

[0164] The key driver selection formula is as follows:

[0165] ;

[0166] in This indicates whether the i-th influencing factor is identified as a key driver. This represents the causal contribution of the i-th influencing factor. This indicates the key driver selection threshold. At that time, the corresponding factors will be added to the set of key driving factors;

[0167] S604: Based on the key driving factors obtained in S603 and combined with the biological correlation pathways obtained in S5, a correlation analysis is performed on the biological regulatory links corresponding to each key driving factor to generate explanatory pathways affecting the therapeutic response of immune checkpoint inhibitors, and the importance of each explanatory pathway is calculated using the following formula:

[0168] ;

[0169] in This indicates the importance of the p-th explanation path. This represents the causal contribution of the i-th key driving factor in the path. This indicates the number of key driving factors contained in the current explanatory path. The greater the importance of the explanatory path, the more significant its impact on the treatment response.

[0170] S605: Explanatory Analysis Results Output: Based on the importance of the explanatory paths obtained in S604, all explanatory paths are sorted to generate cross-modal causal explanation results. The key driving factors, corresponding explanatory paths, and the importance of each path are output. At the same time, the cross-modal causal explanation results are sent to the immune checkpoint inhibitor response prediction module to provide an interpretable basis for predicting patient treatment response, assessing survival benefits, and analyzing drug resistance risks.

[0171] Example 3

[0172] This embodiment proposes an immune checkpoint inhibitor response prediction device based on multimodal biologically interpretable AI, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, is used to perform the immune checkpoint inhibitor response prediction method based on multimodal biologically interpretable AI in any embodiment of the present invention. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor, etc.

[0173] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0174] Example 4

[0175] This embodiment proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it is used to perform the immune checkpoint inhibitor response prediction method based on multimodal biologically interpretable AI in any embodiment of the present invention.

[0176] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0177] In this case, the program code read from the storage medium itself can implement the function of any of the above embodiments, so the program code and the storage medium storing the program code constitute a part of the present invention;

[0178] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs; alternatively, program code can be downloaded from a server computer via a communication network;

[0179] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby achieving the function of any of the above embodiments.

[0180] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0181] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An immune checkpoint inhibitor response prediction system based on multimodal biologically interpretable AI, characterized in that, It includes a multimodal data acquisition module, a data quality control module, a multimodal semantic encoding module, a digital immune microenvironment construction module, a biological knowledge graph reasoning module, a cross-modal causal interpretation module, an immune checkpoint inhibitor response prediction module, a treatment strategy recommendation module, and a model continuous learning module; The multimodal data acquisition module is connected to the data quality control module and is used to collect patient pathological image data, biological omics data, medical imaging data and clinical data, and form a unified patient dataset. The data quality control module is connected to the multimodal semantic coding module and is used to complete data integrity detection, outlier handling, missing value completion, batch effect correction and data standardization. The multimodal semantic coding module is connected to the digital immune microenvironment construction module and the biological knowledge graph reasoning module, and is used to extract pathological image features, biological omics features and clinical features, and establish a unified semantic feature representation; The digital immune microenvironment construction module is connected to the cross-modal causal interpretation module, and is used to integrate spatial tissue structure information, immune cell spatial distribution information and biological omics information to construct a patient's digital immune microenvironment model. The biological knowledge graph reasoning module is connected to the cross-modal causal interpretation module to establish a network of associations between genes, proteins, signaling pathways, immune cells, drugs, and clinical outcomes, and to perform knowledge reasoning to obtain biological association features. The cross-modal causal explanation module is connected to the immune checkpoint inhibitor response prediction module and the treatment strategy recommendation module. It is used to integrate digital immune microenvironment features, biological knowledge graph reasoning results and multimodal semantic features to establish a causal reasoning model and analyze the key factors and action pathways that affect the treatment response of immune checkpoint inhibitors. The immune checkpoint inhibitor response prediction module is connected to the treatment strategy recommendation module and the treatment strategy recommendation module, and is used to predict the patient's treatment response to immune checkpoint inhibitors, survival benefits and drug resistance risk based on causal reasoning results. The treatment strategy recommendation module is connected to the model continuous learning module and is used to generate individualized immunotherapy plans based on response prediction results and knowledge reasoning results. The model continuous learning module is used to update model parameters, biological knowledge graph, and causal reasoning model based on patient treatment results and follow-up data.

2. The immune checkpoint inhibitor response prediction system based on multimodal biologically interpretable AI according to claim 1, characterized in that, The multimodal data acquisition module collects multimodal data including digital pathological slide data, DNA sequencing data, RNA expression data, protein expression data, spatial transcriptome data, medical imaging data, clinical examination data, and patient follow-up data.

3. The immune checkpoint inhibitor response prediction system based on multimodal biologically interpretable AI according to claim 2, characterized in that, The digital immune microenvironment construction module is used to construct a digital immune microenvironment model of the patient based on the pathological image features, biological omics features, and clinical features output by the multimodal semantic coding module, combined with the spatial distribution relationship of immune cells and tissue microenvironment information. It also calculates the degree of immune cell infiltration, the degree of immune activation, spatial heterogeneity, and omics expression status to obtain the digital immune microenvironment index, the calculation formula of which is as follows: ； in Indicates the digital immune microenvironment index, Indicates the immune cell infiltration index, Indicates the immune activation index, Indicates spatial heterogeneity index, Represents the omics expression index. , , , Let each represent the weight coefficient corresponding to the indicator, and satisfy the following condition: ; The Digital Immune Microenvironment Index serves as a quantitative representation of the patient's immune microenvironment status and is output to the cross-modal causal interpretation module for fusion analysis.

4. The immune checkpoint inhibitor response prediction system based on multimodal bio-interpretable AI according to claim 3, characterized in that, The biological knowledge graph reasoning module receives multimodal semantic features output by the multimodal semantic encoding module and digital immune microenvironment indices output by the digital immune microenvironment construction module. It constructs a heterogeneous biological knowledge graph including genes, proteins, signaling pathways, immune cells, immune checkpoint inhibitors, and clinical outcomes. By calculating the correlation strength between nodes, it mines potential biological relationships influencing the response to immune checkpoint inhibitor therapy and outputs biological correlation reasoning features. The formula for calculating the node correlation strength is as follows: ； in This represents the strength of the association between node i and node j in a knowledge graph. Indicates the biological similarity between two nodes. This indicates the degree of association between two nodes in biological regulatory pathways. This indicates the degree of co-occurrence association between two nodes in a digital immune microenvironment. , , Let represent the weight coefficients of the corresponding indicators, and satisfy the following: Based on the node association strength, biological association inference features are generated, and the biological association inference features are output to the cross-modal causal explanation module.

5. The immune checkpoint inhibitor response prediction system based on multimodal bio-interpretable AI according to claim 4, characterized in that, The cross-modal causal interpretation module receives the digital immune microenvironment index output by the digital immune microenvironment construction module and the biological association inference features output by the biological knowledge graph inference module. It performs cross-modal fusion of pathological image features, biological omics features, digital immune microenvironment features, and biological association inference features to establish a multi-factor causal association model. It calculates the causal contribution of each influencing factor to the immune checkpoint inhibitor treatment response and outputs the key influencing factors and their corresponding causal interpretation results. The formula for calculating the causal contribution is as follows: ； in This represents the causal contribution of the i-th influencing factor. This represents the importance score of the i-th multimodal semantic feature. This represents the importance score of the i-th numerical immune microenvironment feature. This represents the importance score of the i-th biological association reasoning feature. , , Let represent the fusion weight coefficients of the corresponding features, and satisfy . ; Based on the causal contribution, key driving factors affecting the response to immune checkpoint inhibitor therapy are identified, and corresponding causal explanation results are generated and output to the immune checkpoint inhibitor response prediction module.

6. A method for predicting the response of immune checkpoint inhibitors based on multimodal biologically interpretable AI, based on the system described in any one of claims 1-5, characterized in that, Includes the following steps: S1: Multimodal patient data aggregation: collecting patient pathological images, biological omics data, medical imaging data, and clinical data; S2: Multimodal data quality control: Perform quality inspection, standardization processing, and multimodal alignment on the collected data; S3: Multimodal semantic feature encoding: Extract pathological image features, biological omics features and clinical features respectively, and establish a unified semantic representation; S4: Digital immune microenvironment modeling: Integrating spatial tissue structure information, spatial distribution information of immune cells, and omics information to construct a digital immune microenvironment model; S5: Heterogeneous Biological Knowledge Graph Reasoning: Establish biological knowledge graphs and perform knowledge reasoning to obtain biological relationships between genes, proteins, signaling pathways, immune cells, and drugs; S6: Cross-modal causal interpretable analysis: Integrating digital immune microenvironment, biological knowledge graph and multimodal semantic features, a cross-modal causal reasoning model is established to obtain the key factors and explanatory pathways affecting the response to immune checkpoint inhibitor therapy; S7: Prediction of Immune Checkpoint Inhibitor Efficacy: Based on cross-modal causal inference results, predict patients' response to immune checkpoint inhibitor treatment, survival benefits, and drug resistance risks, and generate treatment response prediction results; S8: Model self-learning and continuous optimization: Based on the actual treatment results and follow-up information of patients, the parameters of the multimodal semantic coding model, biological knowledge graph, cross-modal causal reasoning model and response prediction model are updated to achieve continuous model optimization.

7. The method for predicting the response of immune checkpoint inhibitors based on multimodal biologically interpretable AI according to claim 6, characterized in that, The execution logic steps of S4 are as follows: S401: Multimodal spatial information mapping: Receives pathological image features, biological omics features and clinical features output by S3, spatially locates tumor regions, immune cell regions and stroma regions in pathological sections, and maps omics expression information to corresponding tissue regions, establishes the correspondence between pathological spatial structure and omics expression, forms a unified spatial biological information dataset, and provides basic data for digital immune microenvironment modeling. S402: Spatial distribution analysis of immune cells: Based on the spatial bioinformatics dataset established in S401, the location coordinates and distribution density of different types of immune cells are identified, and the degree of local immune cell aggregation is calculated by combining the spatial proximity relationship between cells to obtain the spatial distribution characteristics of immune cells. Among them, immune cells include CD8⁺T cells, CD4⁺T cells, regulatory T cells, NK cells, dendritic cells and macrophages. The formula for calculating the local immune cell aggregation index is: ,in Indicates the immune cell infiltration index, Indicates the total number of immune cells in the pathological area. This represents the spatial proximity weight between the i-th immune cell and the j-th immune cell. This represents the biological weight corresponding to the j-th type of immune cell; different immune cell types have different weights. Spatial proximity weights are calculated using a distance decay method, and the calculation formula is as follows: ,in This represents the Euclidean distance between the i-th and j-th immune cells. This represents the spatial decay coefficient, which is used to adjust the range of influence between neighboring cells, ultimately yielding the spatial distribution characteristics of immune cells throughout the entire pathological region. S403: Based on the spatial distribution characteristics of immune cells obtained in S402, and combined with pathological tissue structural characteristics and biological omics expression characteristics, the infiltration state, immune activation state, spatial heterogeneity state, and omics expression state of immune cells are quantitatively analyzed to calculate the digital immune microenvironment index. The calculation formula is as follows: ； in Indicates the digital immune microenvironment index, Indicates the immune cell infiltration index, Indicates the immune activation index, Indicates spatial heterogeneity index, Represents the omics expression index. , , , Let each represent the weight coefficient corresponding to the indicator, and satisfy the following condition: ; S404: Construction of Digital Immune Microenvironment Model: Based on the digital immune microenvironment index calculated in S403, combined with the spatial distribution characteristics of immune cells and the topological relationship of pathological tissue, a digital immune microenvironment model of the patient is constructed; the digital immune microenvironment model can reflect the spatial interaction relationship between different immune cells, the interaction relationship between tumor tissue and immune cells, and the state of the local immune microenvironment. S405: Digital Immune Microenvironment Feature Output: Based on the digital immune microenvironment model constructed in S404, extract the patient's digital immune microenvironment feature vector and output the digital immune microenvironment feature vector to the biological knowledge graph reasoning module for subsequent biological relationship reasoning and cross-modal causal interpretation analysis.

8. The method for predicting the response of immune checkpoint inhibitors based on multimodal biologically interpretable AI according to claim 7, characterized in that, The specific logical steps of S5 are as follows: S501: Heterogeneous biological entity extraction: Receives the multimodal semantic features output by S3 and the digital immune microenvironment features output by S4, performs unified analysis on the gene, protein, signaling pathway, immune cell, immune checkpoint inhibitor and clinical outcome information in patient pathological images, biological omics data and clinical data, and establishes entity nodes respectively to form a heterogeneous biological entity set, providing a node foundation for the construction of biological knowledge graph; S502: Construction of Biological Knowledge Graph: Based on the heterogeneous biological entity set obtained in S501, a patient-specific heterogeneous biological knowledge graph is constructed, using gene regulation relationships, protein-protein interaction relationships, signaling pathway relationships, immune cell regulation relationships, and drug action relationships as connecting edges between entities. Furthermore, attribute annotations are applied to each entity node in conjunction with digital immune microenvironment characteristics, forming a knowledge graph model that includes node attributes and relationships. The formula for calculating the association strength between nodes is as follows: ； in This represents the strength of the association between node i and node j in a knowledge graph. Indicates the biological similarity between two nodes. This indicates the degree of association between two nodes in biological regulatory pathways. This indicates the degree of co-occurrence association between two nodes in a digital immune microenvironment. , , Let represent the weight coefficients of the corresponding indicators, and satisfy the following: ; The comprehensive correlation strength between each node is calculated based on the above formula to form a knowledge graph relationship matrix. S503: Based on the knowledge graph relationship matrix established in S502, multi-level association reasoning is performed on nodes with association strength exceeding a set threshold to identify potential regulatory relationships between genes, proteins, signaling pathways, immune cells, and immune checkpoint inhibitors. This yields biological association pathways that influence the response to immunotherapy, and the credibility of each association pathway is calculated using the following formula: ； in Indicates the credibility of biological association pathways. This represents the overall association strength between two adjacent nodes in the path, and its value is derived from the association strength calculated by S502. This indicates the number of nodes contained in the current associated path. If the associated pathway is valid, it is retained as an effective biological regulatory pathway; otherwise, it is removed. S504: Key Biological Association Feature Extraction: Based on the biological association paths obtained in S503, the frequency of each biological entity in all effective association paths is statistically analyzed, and the comprehensive contribution value of each biological entity to the immune checkpoint inhibitor treatment response is calculated in combination with the association path credibility. The calculation formula is as follows: ； in This represents the overall contribution value of the i-th biological entity. This indicates the credibility of the k-th valid association path. This indicates whether the i-th biological entity exists in the k-th associated path. If it exists, the value is 1; otherwise, the value is 0. n represents the total number of valid associated paths. Based on the comprehensive contribution value sorted from largest to smallest, key genes, key proteins, key signaling pathways and key immune cells with high correlation to immune checkpoint inhibitor treatment response were screened, and biological correlation inference feature vectors were generated. S505: Inference Result Output: Output the biological association inference feature vector generated by S504 to the cross-modal causal explanation module, and use it as an important input to the cross-modal causal explanation model to calculate the causal contribution of each biological feature to the immune checkpoint inhibitor treatment response; The specific logical steps of S6 are as follows: S601: Cross-modal feature fusion: Receives the multimodal semantic features output by S3, the digital immune microenvironment features output by S4, and the biological association inference features output by S5. Performs unified mapping on pathological image features, biological omics features, clinical features, digital immune microenvironment features, and biological association inference features, and establishes a cross-modal fusion feature space to form a cross-modal causal analysis dataset, providing a unified data foundation for subsequent causal inference. S602: Construction of Cross-Modal Causality: Based on the cross-modal causal analysis dataset obtained in S601, a cross-modal causal relationship model is established using multimodal semantic features, digital immune microenvironment features, and biological association inference features as causal variables, and the response to immune checkpoint inhibitor treatment as the target variable. The causal contribution of each influencing factor is calculated using the following formula: ； in This represents the causal contribution of the i-th influencing factor. Represents multimodal semantic feature scoring. Indicates the digital immune microenvironment characteristic score. This represents the score for reasoning features in biological knowledge graphs. , , These represent the fusion weight coefficients of the corresponding features. They are sorted from largest to smallest according to their causal contribution to determine the key driving factors affecting the response to immune checkpoint inhibitor treatment, generate the corresponding causal explanation results, and send the causal explanation results to the immune checkpoint inhibitor response prediction module and the treatment strategy recommendation module. S603: Key Driver Identification: Based on the causal contribution calculated in S602, all influencing factors are sorted from high to low according to their contribution, and influencing factors with a causal contribution higher than a preset threshold are selected as key drivers; among them, key drivers include key genes, key proteins, key signaling pathways, key immune cells, and key pathological regions. The key driver selection formula is as follows: ； in This indicates whether the i-th influencing factor is identified as a key driver. This represents the causal contribution of the i-th influencing factor. This indicates the key driver selection threshold. At that time, the corresponding factors will be added to the set of key driving factors; S604: Based on the key driving factors obtained in S603 and combined with the biological correlation pathways obtained in S5, a correlation analysis is performed on the biological regulatory links corresponding to each key driving factor to generate explanatory pathways affecting the therapeutic response of immune checkpoint inhibitors, and the importance of each explanatory pathway is calculated using the following formula: ； in This indicates the importance of the p-th explanation path. This represents the causal contribution of the i-th key driving factor in the path. This indicates the number of key driving factors contained in the current explanatory path. The greater the importance of the explanatory path, the more significant its impact on the treatment response. S605: Explainable Analysis Results Output: Based on the importance of the explanatory paths obtained in S604, all explanatory paths are sorted to generate cross-modal causal explanation results. Key driving factors, corresponding explanatory paths, and the importance of each path are output. At the same time, the cross-modal causal explanation results are sent to the immune checkpoint inhibitor response prediction module to provide an explainable basis for predicting patient treatment response, assessing survival benefits, and analyzing drug resistance risks.

9. An immune checkpoint inhibitor response prediction device based on multimodal biologically interpretable AI, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the method described in any one of claims 6-8.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 6-8.

Images