Medical image-based tumor microenvironment state analysis method, system and device

Abstract

This application discloses a method, system, and device for analyzing the state of the tumor microenvironment based on medical images, relating to the field of medical image processing technology. First, by acquiring medical images and gene expression data of related tissues within those images, the medical images are preprocessed to obtain image patches, and semantic features of these patches are extracted. An algorithm is then used to calculate a predefined immune resistance mechanism that matches the gene expression data as the dominant immune resistance mechanism. This dominant immune resistance mechanism is then set as a supervisory label for the model, and a prediction model is constructed. The model is trained using a large amount of data, and the medical image to be analyzed is input into the trained prediction model to obtain the predicted category of the resistance mechanism. This application can accurately predict the resistance mechanism of pathological tissues based on medical images and their related gene expression data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method, system, and device for analyzing the state of tumor microenvironment based on medical images. Background Technology

[0002] In the cancer treatment system, individual immune resistance plays a crucial role in the treatment of advanced and recurrent cancer. According to relevant studies, some patients develop drug resistance during cancer treatment, which can lead to chemotherapy drugs being unable to effectively kill tumor cells and reduce treatment efficacy. The tumor microenvironment (TME) is a complex ecosystem composed of tumor cells and their surrounding matrix, immune cells, blood vessels, and signaling molecules. Various cells in the tumor microenvironment are involved in the formation of drug resistance. Especially in the treatment of immune checkpoint inhibitors (ICIs), the state of the TME directly determines the efficacy. Therefore, the TME plays a very important role in regulating tumor progression, metastasis, and drug treatment response.

[0003] In some related technologies, ICI response is usually judged through biological markers, but its predictive ability is limited. It is effective for some PD-L1 negative patients, but PD-L1 positive patients develop primary resistance during treatment. In addition, TME analysis methods based on gene expression profiles or public databases can estimate the level of immune cell infiltration, but still have the following problems: First, it relies on expensive and difficult-to-obtain transcriptome data; second, it only provides static cell proportions and cannot distinguish different resistance mechanisms such as T cell rejection, myeloid suppression, and antigen presentation defects; third, the results are out of touch with the actual pathological morphology and are difficult to apply to pathology-related research fields. Summary of the Invention

[0004] To address at least one of the problems mentioned in the background art, this application provides a method, system, and device for analyzing the state of the tumor microenvironment based on medical images. By processing pathological tissue images of tumors, an analysis model is constructed based on the processed pathological images and preset types of drug resistance mechanisms. A weakly supervised learning framework is adopted, and pathology-related gene sequences are used as labels to train the model, so that the drug resistance mechanism of individuals related to the pathological images can be accurately analyzed through the model.

[0005] The specific technical solutions provided in this application are as follows: Firstly, a method for analyzing the state of the tumor microenvironment based on medical images is provided, the method comprising: Acquire gene expression data of medical images and tissue samples containing the medical images; The preprocessed medical image is divided into image patches, and the semantic features of the image patches are extracted using a preset model; Obtain the dominant immune resistance mechanism that matches the gene expression data from the predefined immune resistance mechanisms, and set the dominant immune resistance mechanism as the dominant immune resistance mechanism label. Combine the semantic features and the dominant immune resistance mechanism label to construct an immune resistance mechanism prediction model. The immune resistance mechanism prediction model is trained by inputting the medical image to be analyzed into the trained immune resistance mechanism analysis model, so as to output the immune resistance mechanism prediction category.

[0006] In one specific embodiment, the medical image is a full slice image of the tissue sample, and the full slice image is subjected to color normalization processing to generate a color-normalized medical image; The Otsu algorithm is used to segment the color-normalized medical image into a foreground including tissue regions and a background including non-tissue regions to obtain the preprocessed medical image. The preprocessed medical image is divided into image blocks, the resolution of the preprocessed medical image is obtained, and the size of the image blocks is adjusted based on the resolution.

[0007] In one specific embodiment, the predefined immune resistance mechanism is set to include at least T cell rejection, myeloid suppression, antigen presentation defect, and tumor cell autonomous resistance. The gene set variation analysis method is used to calculate the activity score associated with each immune resistance mechanism in the predefined immune resistance mechanism; the immune resistance mechanism corresponding to the highest activity score is selected as the dominant immune resistance mechanism of the gene expression data; and all image patches segmented from the same medical image are set to share the same dominant immune resistance mechanism label.

[0008] In a specific embodiment, a drug resistance mechanism prediction model is constructed by combining the semantic features and the dominant immune drug resistance mechanism label, specifically including: A feature matrix is ​​constructed by combining the semantic features, and the spatial-semantic joint complexity is calculated based on the feature matrix. The number of output layers of the immune resistance mechanism prediction model is dynamically set based on the spatial-semantic joint complexity. Each immune resistance mechanism in the predefined immune resistance mechanism is set to generate a corresponding category attention weight. An attention-weighted covariance matrix is ​​constructed by combining the category attention weights of the same image patch and the feature vectors in the feature matrix. The attention-weighted covariance matrix is ​​vectorized and mapped through a learnable projection matrix to generate a feature representation of the immune resistance mechanism.

[0009] In one specific embodiment, the feature representation of the immune resistance mechanism is obtained, and the category attention weight associated with the feature representation of the immune resistance mechanism is extracted; The spatial coordinates of the image patch in the medical image are obtained, and the category attention weight associated with the immune resistance mechanism feature representation is mapped to the image plane in combination with the spatial coordinates to form a key region localization map. The key region localization map is used to indicate the key regions associated with the immune resistance mechanism feature representation.

[0010] In a specific embodiment, the calculation process of the spatial semantic joint complexity is expressed by formulas (1), (2), (3), and (4) as follows: (1) (2) (3) (4) in, Represents the joint complexity of spatial semantics; Indicates the normalization factor; Represent the covariance matrix; The feature matrix representing all image patches; This indicates the number of image blocks in a medical image; This represents the feature vector of the k-th image patch; This represents the spatial gradient of the k-th image patch. If image patch k has neighbors, calculate the gradient of image patch k with all its neighbors. Feature differences; if image patch k has no neighbors, set ; , Indicates learnable parameters; Represents the mean vector of H; The dimension of the feature vector; Wherein, the image block k is related to all its neighbors The characteristic differences are obtained through formula (5): (5) in, Represents the neighborhood set of image patch k; and Let represent the feature vectors of image patch k and its neighbor n, respectively; express and The Euclidean distance between two eigenvectors.

[0011] In a specific embodiment, a lightweight gating unit is used to control the number of output layers of the drug resistance mechanism prediction model, wherein the calculation formula (6) for the gating unit is as follows: (6) in, This represents the output of the gating unit, which is used to control the information flow intensity of the output layer of the drug resistance mechanism prediction model. Its value ranges from [0,1]. , These represent the learnable parameters, corresponding to the weight matrix and bias vector of the gated unit, respectively. This represents the Sigmoid activation function; Indicates the first The hidden state feature vector of the layer.

[0012] In a specific embodiment, the formula (7) for calculating the category attention weight of the k-th image patch of the i-th type of immune drug resistance mechanism in the l-th layer of the drug resistance mechanism prediction model is as follows: (7) in, The category attention weight represents the k-th image patch representing the i-th type of immune resistance mechanism in the l-th layer; , The first two layers of the drug resistance mechanism prediction model represent the weight matrix; N represents the parallel attention branches that the drug resistance mechanism prediction model splits into after the first two layers. The weight matrix representing the parallel attention branch; This represents the position code of the k-th image block; Represents the hyperbolic tangent activation function; Represents the Sigmoid function; This represents the element-wise multiplication of two matrices; The formula (8) for calculating the attention-weighted covariance matrix is ​​as follows: (8) in, Represents the attention-weighted covariance matrix of the i-th type of immune resistance mechanism; This represents the attention weight of the k-th image patch representing the i-th type of immune resistance mechanism in layer L; This represents the feature vector of the k-th image patch.

[0013] Secondly, a tumor microenvironment state analysis system based on medical images is provided to implement the tumor microenvironment state analysis method based on medical images as described above. The system includes: The input module is used to acquire gene expression data of medical images and tissue samples containing the medical images; The analysis module is used to divide the preprocessed medical image into image patches, extract the semantic features of the image patches using a preset model, and obtain the dominant immune resistance mechanism that matches the gene expression data in the predefined immune resistance mechanism, and set the dominant immune resistance mechanism as the dominant immune resistance mechanism label. The module combines the semantic features and the dominant immune resistance mechanism label to construct an immune resistance mechanism prediction model. The output module is used to train the immune resistance mechanism prediction model. The medical image to be analyzed is input into the trained immune resistance mechanism analysis model to output the immune resistance mechanism prediction category.

[0014] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above method steps.

[0015] The embodiments of this application have the following beneficial effects: 1. The solution provided in this application involves acquiring medical images and gene expression data of the corresponding tissue samples, processing the medical images and dividing them into several image blocks, extracting semantic features of the image blocks using a preset model, and forming a feature matrix by combining multiple semantic features. Simultaneously, matching calculations are performed between the gene expression data and four predefined types of immune resistance mechanisms to obtain the immune resistance mechanism with the highest matching degree to the gene expression data. This immune resistance mechanism is then set as the dominant immune resistance mechanism associated with the gene expression data. The semantic feature matrix and the dominant immune resistance mechanism are used as supervisory labels to construct and train a prediction model. The trained prediction model then predicts the drug resistance mechanism category of the medical image to be analyzed. Based on this solution, the pathological morphology of an individual can be deeply correlated with the immune resistance mechanism, thereby achieving efficient and accurate determination of the drug resistance mechanism of the individual in the medical image with unknown information. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of the application environment according to this application is shown; Figure 2 A schematic diagram of a tumor microenvironment state analysis method based on medical images according to this application is shown; Figure 3 This diagram illustrates the output process for predicting categories based on the immune resistance mechanisms described in this application. Figure 4 A schematic diagram of a tumor microenvironment state analysis system based on medical images according to this application is shown; Figure 5 A schematic diagram of a computer device according to this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The tumor microenvironment state analysis method based on medical images provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be a standalone server or a server cluster consisting of multiple servers.

[0020] In one embodiment, such as Figure 2 and Figure 3 As shown, a method for analyzing the state of the tumor microenvironment based on medical images is provided, and this method is applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps: Step 1: Obtain the medical image and the gene expression data of the tissue sample containing the medical image.

[0021] (1) Acquisition and processing of medical images Specifically, the medical images in this embodiment include tissue section images taken under a microscope, which have undergone H&E staining, immunohistochemical staining (IHC), and fluorescence in situ hybridization. The medical images in this embodiment are whole-section images of tissue samples, specifically digital images of whole sections stained with xylose-eosin (H&E), independent of immunohistochemical or fluorescence in situ hybridization results.

[0022] It should be noted that the medical images in this embodiment are derived from single-center or multi-center data collected and organized by clinical medical institutions, or from open-source datasets obtained from domestic and international public databases; the medical images include tumor-related diseases, and the types of tumors include, but are not limited to, one or more of the following: lung cancer, breast cancer, gastric cancer, esophageal cancer, colorectal cancer, pancreatic cancer, liver cancer, gallbladder cancer, endometrial cancer, cervical cancer, ovarian cancer, thyroid cancer, glioma, renal cell carcinoma, laryngeal cancer, bladder cancer, melanoma, lymphoma, prostate cancer, or adrenal cancer.

[0023] Before color normalization of the whole-slice images, the following steps are included: quality checking of the whole-slice images, and rejection of those that do not meet the quality standards, to ensure the reliability of the images selected for subsequent analysis. Specifically, this involves confirming that the whole-slice images are sharp, without blurring or defocusing; checking for overexposure or underexposure; verifying the absence of obvious artifacts or damaged areas; ensuring that the image colors are not distorted; and confirming that there is no physical or chemical damage on the whole-slice images that could hinder diagnosis. If any images do not meet the standards, these images are removed from the dataset or marked as invalid.

[0024] In one specific embodiment, due to differences in different scanning devices, imaging devices, staining batches, or laboratory conditions, there may be significant color differences between whole-slice images. To ensure the accuracy of subsequent analysis results, the whole-slice images need to be further processed, including: performing color standardization on the whole-slice images to generate color-standardized medical images; using the Otsu algorithm to segment the color-standardized medical images into a foreground including tissue regions and a background including non-tissue regions to obtain preprocessed medical images; dividing the preprocessed medical images into image patches, obtaining the resolution of the preprocessed medical images, and adjusting the size of the image patches based on the resolution.

[0025] The color standardization process for the whole-slice images includes: converting the original RGB image to optical density space to separate the absorption characteristics of the dyes; then, based on the staining distribution characteristics of the reference image, estimating and remapping the staining components of each image to be processed to ensure that the concentration distribution of the two main dyes, hematoxylin and eosin, is consistent with the reference standard; finally, the corrected optical density data is inversely transformed back to the RGB color space to generate a color-standardized pathological whole-slice image.

[0026] (2) Acquisition and processing of gene expression data In this embodiment, the gene expression cross-data originates from tumor tissue samples paired with the aforementioned whole-slice digital images. Fresh tumor tissue is obtained in clinical settings through surgical resection or biopsy, and each tissue sample is assigned a unique identifier to ensure data traceability throughout the entire process from collection to analysis. To ensure the integrity and stability of the gene expression data, tissue samples are immediately cryopreserved in liquid nitrogen after ex vivo and transported to a professional sequencing institution under cryogenic conditions. RNA extraction, library construction, and high-throughput sequencing are performed by a qualified third-party company. The returned sequencing data undergoes a standardized bioinformatics processing workflow, including raw data quality control filtering, sequence alignment to a reference genome, gene expression level quantification, batch effect correction, and subsequent functional annotation and pathway enrichment analysis to form a standardized gene expression matrix that can be used for model training.

[0027] Furthermore, to ensure the accuracy of the final prediction results in this embodiment, the data needs to be processed before being input into the model to ensure the consistency of the input data. Specifically, various visualization techniques are used to evaluate the quality of transcriptome data, including but not limited to UMAP (Uniform Manifold Approximation and Projection) or PCA (Principal Component Analysis), and to check for batch effects, outliers, or other systematic differences in gene expression data. UMAP, a nonlinear dimensionality reduction algorithm based on Riemannian geometry and algebraic topology, allows for intuitive observation of gene expression data quality and identification of potential batch effects or natural grouping among samples. If the sample points in the UMAP plot form obvious clusters, it may indicate the presence of batch effects or sample grouping; conversely, if the sample points are relatively evenly distributed, it indicates good data quality and the absence of significant batch effects.

[0028] PCA is a statistical method that transforms a set of potentially related variables into a set of linearly independent variables called principal components through linear transformation. PCA plots are used to identify outliers in gene expression data. If the points are distributed in obvious clusters, it may indicate batch effects or other systematic differences; if the points are relatively uniformly distributed and there are no obvious clusters, it indicates good data quality and no significant batch effects.

[0029] In one specific embodiment, the method further includes normalizing the gene expression data that has passed the quality assessment. If the gene expression data is not yet standardized, a base-2 logarithmic transformation is performed on the original count values ​​after incrementing by one. This transformation can effectively improve the distribution of gene expression values, reduce the dominant effect of highly expressed genes, stabilize variance, and make the data more suitable for calculating mechanism activity scores based on enrichment analysis. For data with outliers or values ​​outside the expected range, further necessary corrections or filtering are performed to ensure the accuracy of downstream immune resistance mechanism tag construction.

[0030] Step 2: Divide the preprocessed medical image into image blocks and extract the semantic features of the image blocks using a preset model.

[0031] The Otsu algorithm is used to segment color-normalized medical images, including: first, converting the color or grayscale medical image into a single-channel grayscale image suitable for processing; then, iterating through all possible thresholds and calculating the inter-class variance at each threshold; finally, selecting the threshold that maximizes the inter-class variance as the global threshold, thereby segmenting the whole-section pathological image into foreground (tissue region) and background (non-tissue region); the Otsu algorithm automatically determines the optimal threshold by maximizing the inter-class variance between the foreground and background to achieve effective image segmentation.

[0032] For example, the image segmentation process using the Otsu algorithm is as follows: Assume the image's grayscale range is [0, L-1], where L represents the total number of possible grayscale levels. Let the probability of a pixel appearing at each grayscale level i be... ,in , Let represent the number of pixels with gray level i, and N be the total number of pixels in the image. For each possible threshold T, the Otsu algorithm divides the image into two parts: one part contains all pixels with gray levels less than or equal to T (background), and the other part contains all pixels with gray levels greater than T (foreground). The core of the algorithm is to find an optimal threshold T such that the inter-class variance between these two parts is... It has reached its maximum value.

[0033] Given a threshold T, the inter-class variance of foreground and background Defined as formula (1): (1) in, Represents the inter-class variance of the foreground and background; and Let T and T represent the weights of pixels less than or equal to T and pixels greater than T, respectively, and they can be expressed as formula (2); T represents the threshold. and These represent the average gray values ​​of these two parts, respectively, and the average gray value is expressed by formula (3); by traversing all gray levels T, the average gray value is calculated at each step. And select the one that makes The largest value, T, is taken as the optimal threshold. The threshold T obtained in this way is the value that best distinguishes between the foreground and the background.

[0034] (1), (2) , (3) in, and These represent the weights of pixels less than or equal to T and the weights of pixels greater than T, respectively; T represents the threshold; i represents the gray level, with a value ranging from 0 to L-1. and These represent the average gray values ​​of these two parts, respectively.

[0035] After image segmentation, to avoid the high-resolution characteristics of histopathological images and resolution differences between histopathological slides from different sources affecting the segmentation, the size of the image blocks needs to be dynamically adjusted according to the actual resolution of the whole slide image. Specifically, a scaling strategy is used; for example, the image block size of a 40X image is set to twice that of a 20X image block. This ensures comparability in feature density and spatial information for images of different resolutions while maintaining the computational feasibility of the algorithm. The processed image block segmentation information is saved in a unified format; for example, the image blocks are saved in H5 format.

[0036] To efficiently extract pathologically relevant discriminative features from whole-slice images, this embodiment employs a large-scale self-supervised pre-trained pathological visual foundation model to encode deep features of image patches. This foundational model includes, but is not limited to, one or more of UNI, cTransPath, Prov-GigaPath, or Virchow2. These models can be pre-trained on massive amounts of unlabeled digitized pathological images, learning rich morphological semantics without relying on manual annotation. Their semantic features include, but are not limited to, key pathological features such as nuclear atypia, cell arrangement polarity, degree of stromal fibrosis, immune cell infiltration patterns, and the spatial relationship of the tumor-stromal interface.

[0037] It should be noted that each image patch is used to extract a corresponding semantic feature through a pre-defined model. This semantic feature is a high-dimensional feature vector, which includes microstructural information of the tissue and encodes its semantics within a larger tissue context. Simultaneously, the spatial coordinates of each image patch in the original whole-slice image are obtained. The extracted semantic feature vectors and their spatial coordinates are uniformly formatted into a structured data format to form a multi-instance input set. This multi-instance input set can be used as input information for subsequent model construction and supports the model in automatically focusing on pathological regions highly correlated with dominant immune resistance mechanisms even without pixel-level annotation. This achieves an end-to-end mapping from conventional whole-slice images to their specific resistance mechanisms.

[0038] Step 3: Obtain the dominant immune resistance mechanism that matches the gene expression data in the predefined immune resistance mechanisms, and set the dominant immune resistance mechanism as the dominant immune resistance mechanism tag.

[0039] It should be noted that the predefined immune resistance mechanisms in this embodiment include four resistance modes: T cell rejection, myeloid suppression, antigen presentation defect, and tumor cell autonomous resistance. Each of the above four resistance modes includes a set of immune resistance mechanism-related genes with well-defined functions and no overlap.

[0040] Specifically, the first type is T-cell rejection, driven by GF-β signaling activation and stromal fibrosis, which prevents CD8+ T cells from infiltrating the tumor parenchyma. The second type is myeloid suppression, characterized by the enrichment of M2 macrophages, myeloid-derived suppressor cells (MDSCs), and regulatory T cells (Tregs), forming a strongly immunosuppressive microenvironment. The third type is antigen-presenting defective, caused by the downregulation of key genes such as HLA-I molecules, B2M, and TAP1, which allows tumor cells to evade immune recognition. The fourth type is tumor cell autonomous resistance, caused by abnormal activation of tumor intrinsic signaling pathways such as WNT / β-catenin and PI3K / AKT, which prevents effective killing of tumor cells even in the presence of T-cell infiltration.

[0041] Each drug resistance pattern's gene set contains fifteen to fifty core genes, covering key nodes and effector molecules in the signaling pathway, ensuring high specificity in biological function. At the same time, non-specific genes that are commonly highly expressed in multiple mechanisms are excluded through cross-validation, thereby ensuring the mutual exclusivity of the four gene sets in terms of functional definition.

[0042] In one specific embodiment, a gene set variation analysis method is used to calculate the activity score associated with gene expression data and each drug resistance mechanism in the predefined immune drug resistance mechanism; the drug resistance mechanism corresponding to the highest activity score is selected as the dominant immune drug resistance mechanism of the gene expression data; and all image patches segmented from the same medical image are set to share the same dominant immune drug resistance mechanism label.

[0043] For example, gene expression data of the same individual is obtained, and the activity score of the individual in the four drug resistance patterns mentioned above is calculated using gene set variation analysis. This method quantifies the activity level of a specific biological process by assessing the relative enrichment of the target gene set in the whole-genome expression profile. Subsequently, the scores of the four categories are compared horizontally, and the immune resistance mechanism with the highest score is selected as the dominant immune resistance mechanism label for the patient. The dominant immune resistance mechanism label is a discrete categorical variable, rather than a continuous numerical value, which can be directly used for training subsequent weakly supervised multi-instance learning models. This method avoids reliance on expensive and difficult-to-routine detection methods such as single-cell sequencing, multiplex immunofluorescence, or PD-L1 IHC, significantly reducing the technical threshold and detection costs, and enabling accurate immunophenotyping to be promoted at the grassroots level.

[0044] It should be noted that in this embodiment, each whole-slice image is segmented into multiple image patches, and image patches within the same whole-slice image share the same dominant immune resistance mechanism label. The aforementioned semantic features and gene expression data are integrated to form a comprehensive dataset reflecting the tumor microenvironment. This comprehensive dataset is then divided into training, testing, and validation sets according to a preset ratio. For example, 80% is used as the training set, 10% as the testing set, and 10% as the validation set. This ensures model generalization ability while maximizing data utilization for model training. If the same pathological morphology contains multiple image patches, all image patches are forcibly assigned to the same subset to avoid information leakage and simulate generalization challenges in real clinical scenarios. Regarding data efficiency, the model's performance dependence on data volume is assessed by gradually reducing the size of the training dataset. For example, the proportion of the training dataset can be set in a decreasing pattern of 75%, 50%, 25%, and 10%. To ensure that model selection is not affected, the corresponding test set remains unchanged during each sampling.

[0045] Step 4: Construct an immune resistance mechanism prediction model by combining semantic features and dominant immune resistance mechanism labels.

[0046] Specifically, this includes step 4.1: constructing a feature matrix by combining the semantic features, calculating the spatial semantic joint complexity by combining the feature matrix, and dynamically setting the number of output layers of the immune drug resistance mechanism prediction model based on the spatial semantic joint complexity.

[0047] Specifically, a medical image is acquired and segmented into K image blocks. Semantic features are extracted from these blocks using a basic model to form a feature matrix. The spatial semantic joint complexity is then calculated based on the feature matrix. This design aligns the model complexity with the intrinsic complexity of the samples end-to-end, ensuring sufficient modeling of highly complex tumor regions while avoiding overfitting to homogeneous samples and wasting computational resources. This significantly improves the model's generalization ability in real clinical cohorts. The calculation process is illustrated by formulas (4), (5), (6), and (7) as follows: (4) (5) (6) (7) in, Represents the joint complexity of spatial semantics; Indicates the normalization factor; Represent the covariance matrix; The feature matrix representing all image patches; This indicates the number of image blocks in a medical image; This represents the feature vector of the k-th image patch; This represents the spatial gradient of the k-th image patch. If image patch k has neighbors, calculate the gradient of image patch k with all its neighbors. Feature differences; if image patch k has no neighbors, set ; , Indicates learnable parameters; Represents the mean vector of H; This represents the dimension of the feature vector.

[0048] Wherein, the image block k is related to all its neighbors The characteristic differences are obtained through formula (8): (8) in, Represents the neighborhood set of image patch k; and Let represent the feature vectors of image patch k and its neighbor n, respectively; express and The Euclidean distance between two eigenvectors.

[0049] In this embodiment, the semantic space joint complexity integrates the semantic differences between image patch features and the degree of drastic change in spatial morphology, thereby achieving multi-dimensional quantification of tumor heterogeneity. Furthermore, a depth mapping parameter function for complexity-driven parameters is introduced to convert the semantic space complexity into the maximum allowed number of layers in the attention network. The calculation formula (9) for the maximum allowed number of layers is as follows: (9) in, The maximum number of layers in an attention network whose semantic space complexity is dynamically determined. and Let represent the 5th percentile and 95th percentile of the joint semantic space complexity value of all samples in the training set, respectively. This indicates the maximum allowed number of attention layers in the attention network (a preset upper limit for the number of layers). This indicates the minimum number of attention layers allowed (the preset lower limit for the number of layers).

[0050] Step 4.2: Set the corresponding category attention weight for each of the predefined immune resistance mechanisms.

[0051] In the actual forward propagation of the model, the model does not enforce the maximum allowed layers of the attention network. Instead, it uses lightweight gating units for dynamic adjustment. The lightweight gating units control the number of output layers of the drug resistance mechanism prediction model. The calculation formula (10) for the gating units is as follows: (10) in, This represents the output of the gating unit, which is used to control the information flow intensity of the output layer of the drug resistance mechanism prediction model. Its value ranges from [0,1]. , These represent the learnable parameters, corresponding to the weight matrix and bias vector of the gated unit, respectively. This represents the Sigmoid activation function; Indicates the first The hidden state feature vector of a layer, that is, the feature representation from the previous network layer, is used as the input to the gating unit.

[0052] if If the result is positive, continue adding the next layer of attention modules; otherwise, terminate the inference early and use the output of the current layer as the final representation.

[0053] Furthermore, this embodiment employs a multi-class parallel attention mechanism, specifically learning independent attention weights for each of the four types of immune resistance mechanisms. This allows the model to simultaneously focus on pathological regions associated with different immune resistance mechanisms. By generating independent attention weights for each of the four resistance mechanisms, precise focusing on regions related to different biological mechanisms is achieved. Furthermore, through attention-weighted covariance matrices and their square-root normalization operations on the Riemannian manifold, a scale-invariant, batch-effect-robust slice-level representation is constructed. This strategy effectively enhances the feature consistency of cross-center and cross-device data, and is particularly suitable for the identification of low-abundance malignant regions and multi-center research scenarios.

[0054] The first in the drug resistance mechanism prediction model The formula (11) for calculating the category attention weight of the k-th image patch of the i-th drug resistance mode in the layer is as follows: (11) in, The category attention weight represents the k-th image patch representing the i-th type of immune resistance mechanism in the l-th layer; , The first two layers of the drug resistance mechanism prediction model represent the weight matrix; N represents the parallel attention branches that the drug resistance mechanism prediction model splits into after the first two layers. The weight matrix representing the parallel attention branch; This represents the position code of the k-th image block; Represents the hyperbolic tangent activation function; Represents the Sigmoid function; This indicates the element-wise multiplication of two matrices.

[0055] Step 4.3: Construct an attention-weighted covariance matrix by combining the category attention weights of the same image patch with the eigenvectors in the feature matrix.

[0056] Specifically, in this embodiment, a robust feature aggregation strategy based on attention-weighted second-order statistics is adopted. Specifically, after the dynamic multi-layer attention mechanism completes the inference of the final layer, for the corresponding immune resistance category, the model combines the category-specific attention weight of each image patch with the corresponding feature vector to construct an attention-weighted covariance matrix. The calculation formula (12) of the attention-weighted covariance matrix is ​​as follows: (12) in, Represents the attention-weighted covariance matrix of the i-th type of immune resistance mechanism; This represents the attention weight of the k-th image patch representing the i-th type of immune resistance mechanism in layer L; This represents the feature vector of the k-th image patch.

[0057] Step 4.4: Vectorize the attention-weighted covariance matrix using a learnable projection matrix to generate the feature representation of the immune resistance mechanism.

[0058] Specifically, the attention-weighted covariance matrix is ​​normalized by square root. This operation is implemented on the Riemannian manifold, which can effectively suppress the interference caused by the fluctuation of feature amplitude, while preserving the positive definiteness and geometric structure information of the original covariance matrix, and significantly enhancing the robustness of the model to data distribution shifts. The final feature representation represents the predicted category of immune drug resistance mechanism. The four feature representations are concatenated or input separately into the multi-task perceptual MLP classification head to output the predicted probabilities of the four drug resistance mechanisms. The feature representation calculation process is as follows: Formula (13) is as follows: (13) in, This represents a vectorization function used to convert a matrix into a vector. This represents the square root of the attention-weighted covariance matrix; This represents the learnable parameters.

[0059] In a specific embodiment, to more intuitively explain the morphological features associated with specific drug resistance clusters in the tumor microenvironment, an attention map is generated based on the aforementioned immune resistance mechanism prediction categories. Specifically, this includes: The immune resistance mechanism feature representation is obtained, and the category attention weight associated with the immune resistance mechanism feature representation is extracted; the spatial coordinates of the image patch in the medical image are obtained, and the category attention weight associated with the immune resistance mechanism feature representation is mapped to the image plane in combination with the spatial coordinates to form a key region localization map, which is used to indicate the key regions related to the immune resistance mechanism feature representation.

[0060] In this embodiment, for the category of the dominant prediction mechanism, the corresponding category-specific attention weight vector is extracted. Then, using the spatial coordinates of each image patch in the original H&E full slice, the weights are mapped back to the image plane to construct a low-resolution heatmap. The heatmap is then bilinearly interpolated and upsampled to the same size as the original full slice. Warm colors are used to encode high attention regions, and cool colors are used to encode low attention regions. Finally, a semi-transparent attention heatmap is generated over the H&E full slice image. This attention map does not require any pixel-level annotation and is automatically generated entirely by weak supervision signals.

[0061] Step 5: Train the immune resistance mechanism prediction model by inputting the medical image to be analyzed into the trained immune resistance mechanism analysis model to output the predicted immune resistance mechanism category.

[0062] Specifically, the model training process can refer to the construction process of the drug resistance mechanism prediction model mentioned above, obtain multiple datasets, and use the datasets to train the immune drug resistance mechanism prediction model. The entire training process achieves synergistic optimization of "microenvironment complexity - model architecture - attention focus - feature aggregation", breaking through the performance bottleneck of traditional fixed-depth models in tumors.

[0063] The model is optimized using the cross-entropy loss function. Given the significant class imbalance among the four types of immune resistance mechanisms in the clinical cohort, focus loss is further introduced to enhance the ability to identify rare resistance subtypes. The calculation formula for focus loss (14) is as follows: (14) in, Indicates focal loss; Indicates one-hot encoding; Indicates the model's predicted label; This represents the focus parameter; W represents the total number of categories, W=4; i represents the category index.

[0064] The above function formula automatically reduces the weight of easily classified samples, allowing the model to focus more on rare but clinically crucial drug-resistant subtypes, significantly improving the recall and classification accuracy of minority classes. The optimizer chosen is AdamW, which effectively alleviates overfitting in high-dimensional pathological feature spaces by decoupling weight decay from gradient updates. Combined with cosine annealing learning rate scheduling, this ensures stable convergence of the model with limited labeled data.

[0065] In a specific embodiment, standard multi-class evaluation metrics are used to evaluate model performance on an independent validation set. The main metrics include: overall accuracy, precision for each class, recall, F1-score, macro-average / weighted average F1-score, confusion matrix, and multi-class AUC-ROC curve. The calculation formulas (15) to (19) for the main metrics are as follows: (15) (16) (17) (18) (19) in, Indicates overall accuracy; Indicates the accuracy rate for each category; Indicates the F1 score; Indicates the true positive rate; Indicates the false positive rate; TP and TN represent correctly classified results. TP stands for True Positive, which means the number of positive samples that the model predicted as positive; TN stands for True Negative, which means the number of negative samples that the model predicted as negative; FP stands for False Positive, which means the number of negative samples that the model incorrectly predicted as positive; and FN stands for False Negative, which means the number of positive samples that the model incorrectly predicted as negative.

[0066] Furthermore, leveraging the predictive power of the dominant immune resistance mechanism in this embodiment for long-term clinical outcomes, this invention conducts survival analysis based on real follow-up data in an independent validation cohort. The specific implementation process is as follows: First, the trained weakly supervised multi-instance learning model was applied to all patients in the validation set who received immune checkpoint inhibitor therapy. Only their H&E whole-slice images were input, and the model output the dominant immune resistance mechanism category for each patient. This category was a four-choice discrete label, including: T-cell rejection, myeloid suppression, antigen-presenting defective, or tumor cell autonomous resistance.

[0067] Subsequently, follow-up information of corresponding patients was extracted from electronic medical record systems or clinical trial databases. Key variables included overall survival (OS), progression-free survival (PFS), and endpoint event status. Based on this, the Kaplan-Meier nonparametric estimation method was used to plot survival curves according to four types of drug resistance mechanisms, with time on the horizontal axis and cumulative survival probability on the vertical axis to visually demonstrate the differences in survival trends among different biological subgroups. To further quantify the statistical significance of differences between groups, a Log-rank test was performed; if the p-value was less than 0.05, it was determined that there were at least two groups with significant differences in survival distribution.

[0068] Furthermore, a Cox proportional hazards regression model was constructed, using one group (e.g., T-cell rejection) as a reference group, to calculate the hazard ratios (HRs) and their 95% confidence intervals (CIs) of the other three groups relative to the reference group. HR > 1 indicates that the resistance mechanism is associated with a higher risk of death or progression, while HR < 1 suggests a relatively better prognosis. Through the above analysis, not only was the proposed four-type resistance mechanism classification significantly correlated with actual patient survival outcomes, but the subtypes with the worst prognosis were also clearly identified. In other words, the proposed approach can provide clinicians with early warning of high-risk patients and support the development of more proactive individualized intervention strategies.

[0069] In one specific embodiment, to achieve accurate analysis of patient response outcomes after first-line immune checkpoint inhibitor (ICI) treatment, this protocol also constructs a response prediction model based on the aforementioned dominant immune resistance mechanism. Specifically, this includes: Combining the predicted categories of immune resistance mechanisms, the confidence distribution corresponding to predefined immune resistance mechanisms is calculated. A joint feature set is formed by integrating the confidence distribution, gene expression data, and clinical state variable data. A random forest classifier is used to analyze the joint feature set to obtain the average reduction in impurity caused by the features in the joint feature set in the decision tree. The average reduction in impurity is normalized to obtain the importance score of each feature, which is used to identify the factors with the strongest discriminative power for ICI response. Furthermore, the XGBoost gradient boosting decision tree algorithm is used to construct the final binary classification prediction model, which is used to analyze whether patients experience objective remission after receiving ICI treatment.

[0070] It should be noted that clinical state variables include, but are not limited to, one or more of the following: PD-L1 immunohistochemical score, tumor mutation burden (TMB), or ECOG performance status score; specifically, this XGBoost trains the model by minimizing the following objective function calculation formula (20). Through this objective function, XGBoost automatically balances model complexity and generalization ability while fitting the training data, thereby achieving high-precision prediction of immunotherapy response: (20) in, This represents the loss function, which is used to measure the predicted value. and the true value The differences between them; The regularization term, defined as follows, prevents overfitting by controlling the complexity of the tree. T represents the number of leaf nodes in the tree; Indicates the weight of the leaf node; and These represent hyperparameters that control the penalty intensity for the number of leaf nodes and the weight size, respectively.

[0071] In actual training, the model uses whether a patient achieves an objective response (OR) after receiving ICI treatment as a binary label (1 represents complete remission (CR) or partial remission (PR), and 0 represents stable disease (SD) or progressive disease (PD). End-to-end learning is performed using the aforementioned joint feature set. Through this modeling process, and XG-Boost's ability to effectively capture the complex interaction patterns between drug resistance probabilities and clinical variables without requiring manually pre-defined feature combination rules, the final output is the patient's response probability to first-line immune checkpoint inhibitor therapy. This result can be directly used for clinical risk analysis.

[0072] In one specific embodiment, the method further includes optimizing the response prediction model by employing a systematic hyperparameter optimization approach, specifically including: When training the XG-Boost model, five-fold cross-validation is performed on each candidate feature combination. In each fold validation, the dual optimization objectives are to minimize the logistic loss and maximize the AUROC, and the optimal number of boost rounds is dynamically selected, where the number of boost rounds is the number of trees.

[0073] It should be noted that logistic loss is a loss function that measures the quality of the output probability of a classification model. The degree to which it penalizes misclassification depends on the difference between the predicted probability and the actual label. The lower the logistic loss, the better the model performance. The logistic loss formula (21) is as follows: (twenty one) Where L represents the loss function; N represents the number of samples; This represents the true label of the i-th sample; This represents the probability that the model predicts the i-th sample belongs to the positive class.

[0074] Combining the above formulas, we can see that the lower the logistic loss value, the higher the consistency between the model's predicted probability and the true label, and the better the probability calibration effect. Furthermore, an early stopping mechanism is implemented: during cross-validation, if the AUROC metric on the validation set does not improve within 100 consecutive iterations, training is terminated early. This strategy effectively avoids overfitting to noise in the later stages of training and improves training efficiency.

[0075] In one specific embodiment, the method further includes constructing a combined intervention strategy model, extracting high-order spatial features from the input H&E whole-slice image, automatically identifying the patient's dominant immune resistance mechanism type, and then outputting the corresponding intervention strategy based on the combined intervention strategy model.

[0076] Specifically, mapping rules are set in the combined intervention strategy model. For example, T cell rejection corresponds to "PD-1 inhibitor + TGF-β inhibitor". The above mapping rules can be set according to the actual situation.

[0077] In one specific embodiment, this application also constructs a counterfactual image generation model, which can generate a counterfactual image after intervention based on a real whole-slice image as input. Specifically, firstly, a tissue component segmentation model is used to identify key regions; based on the drug resistance mechanism type output from the real whole-slice image, the key regions associated with the drug resistance mechanism type are subject to controllable morphological modifications to generate a pseudo-interventional image. Then, the real whole-slice image and the drug resistance mechanism category are used as input, and the pseudo-interventional image is used as the target output to construct training samples. The key regions are used to indicate pathologically relevant areas, such as stromal fibrosis areas and myeloid cell-rich areas, and the controllable morphological modifications include reducing stromal density and enhancing immune cell infiltration.

[0078] Specifically, the counterfactual image generation model includes a generator and a discriminator, used to achieve the task of "generating a virtual full-slice image of H&E after simulated intervention, given an original image and intervention conditions". The generator adopts a U-Net architecture, taking the original H&E full-slice image and encoded drug resistance mechanism categories as input, and outputting the counterfactual image after the simulated intervention strategy. The discriminator uses a Patch-GAN structure to determine whether the input image comes from the real distribution.

[0079] Furthermore, the training objective function in the training samples is given by formula (22) as follows: (twenty two) in, Indicates resistance to loss; and They actually come from the same real full-slice image. The real whole-slice image is used as a positive sample for the discriminator to learn the visual distribution of real pathological images; This represents the input to the generator, used to generate counterfactual images after simulating pseudo-intervention; This represents the generated counterfactual image. This represents a synthesized pseudo-label image. This represents the L1 pixel-level reconstruction loss. This represents the balance coefficient.

[0080] In one specific embodiment, to enhance the accuracy of the drug resistance reversibility score, the importance weights of feature dimensions related to the dominant immune drug resistance mechanism are extracted. For a preset category of immune drug resistance mechanism, a set of image patches located before the preset portion in descending order of attention weights is first identified. This set of image patches is designated as high-attention image patches, and the mean of the feature vectors of the high-attention image patches is calculated and normalized to obtain the feature weight vector. First and second global features of the real full-slice image and the counterfactual image are extracted respectively using the above feature extraction method. The differences between the first and second global features are weighted using the feature weight vector to calculate the drug resistance reversibility score. This drug resistance reversibility score can be used to indicate whether a clinical drug resistance state can be reversed. The higher the score, the greater the likelihood of reversal after treatment with the recommended combined intervention strategy, providing a more intuitive and quantitative basis for clinical decision-making.

[0081] For example, in the predefined immune resistance mechanism, the first... The calculation process of the characteristic weight vector of the immune resistance mechanism is shown in formulas (23) and (24) as follows: when When the T-cell rejection type is present, the weights of channels such as fibrosis and matrix density in the feature weight vector are significantly higher than those of other channels. (twenty three) (twenty four) in, This represents the first predefined immune resistance mechanism. Mechanisms of immune-like drug resistance; This represents the first predefined immune resistance mechanism. Feature weight vector of immune resistance mechanisms; The mean feature vector of the high attention image block is calculated using formula (24).

[0082] For example, the formula (25) for calculating the drug resistance reversibility score is as follows: (25) in, This represents the reversibility score of drug resistance; i represents the i-th type of immune resistance mechanism in the predefined immune resistance mechanisms. This represents the feature weight vector of the i-th type of immune resistance mechanism in the predefined immune resistance mechanisms; The first global feature representing the true full-slice image; This represents the second global feature of the counterfactual image; This indicates the element-wise multiplication of two matrices.

[0083] In this embodiment, the scheme acquires medical images and gene expression data of the corresponding tissue samples. The medical images are then processed and divided into several image blocks. A preset model is used to extract semantic features from the image blocks, and multiple semantic features are combined to form a feature matrix. Simultaneously, the gene expression data is compared with four predefined types of immune resistance mechanisms to determine the immune resistance mechanism with the highest matching degree. This immune resistance mechanism is then designated as the dominant immune resistance mechanism associated with the gene expression data. The semantic feature matrix and the dominant immune resistance mechanism are used as supervisory labels to construct and train a prediction model. The trained prediction model then predicts the drug resistance mechanism category for the medical images to be analyzed. Based on this scheme, the pathological morphology of an individual can be deeply correlated with immune resistance mechanisms, thereby achieving efficient and accurate determination of the drug resistance mechanism of an individual with unknown information in a medical image.

[0084] Example 2 Corresponding to the above embodiments, this application provides a tumor microenvironment state analysis system based on medical images, such as... Figure 4 As shown, the system includes: The input module is used to acquire gene expression data from medical images and the tissue samples in which the medical images are located. The analysis module is used to divide the preprocessed medical images into image patches, extract semantic features of the image patches using a preset model, and obtain the dominant immune resistance mechanism that matches the gene expression data in the predefined immune resistance mechanism, and set the dominant immune resistance mechanism as the dominant immune resistance mechanism label. The module combines the semantic features and the dominant immune resistance mechanism label to construct an immune resistance mechanism prediction model. The output module is used to train the immune resistance mechanism prediction model. The medical image to be analyzed is input into the trained immune resistance mechanism analysis model to output the predicted immune resistance mechanism category.

[0085] In one specific embodiment, the input module and the output module can be deployed on a smart terminal device, which includes, but is not limited to, the interface of a desktop computer, tablet, or smartphone; the analysis module can be deployed on a server or service cluster or other device with computing capabilities.

[0086] For specific limitations on the tumor microenvironment state analysis system based on medical images, please refer to the limitations on the tumor microenvironment state analysis method based on medical images above, which will not be repeated here.

[0087] The modules in the aforementioned tumor microenvironment state analysis system based on medical images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0088] Corresponding to the above embodiments, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above method steps.

[0089] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores medical image data and gene expression data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for analyzing the state of the tumor microenvironment based on medical images.

[0090] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for analyzing the state of the tumor microenvironment based on medical images, characterized in that, The method includes: Acquire gene expression data of medical images and tissue samples containing the medical images; The preprocessed medical image is divided into image patches, and the semantic features of the image patches are extracted using a preset model; Obtain the dominant immune resistance mechanism that matches the gene expression data from the predefined immune resistance mechanisms, and set the dominant immune resistance mechanism as the dominant immune resistance mechanism label. Combine the semantic features and the dominant immune resistance mechanism label to construct an immune resistance mechanism prediction model. The immune resistance mechanism prediction model is trained by inputting the medical image to be analyzed into the trained immune resistance mechanism analysis model, so as to output the immune resistance mechanism prediction category.

2. The method for analyzing the state of tumor microenvironment based on medical images according to claim 1, characterized in that, The method further includes: The medical image is a full slice image of the tissue sample. The full slice image is color-normalized to generate a color-normalized medical image. The color-normalized medical image is segmented into a foreground including tissue regions and a background including non-tissue regions to obtain the preprocessed medical image; The preprocessed medical image is divided into image blocks, the resolution of the preprocessed medical image is obtained, and the size of the image blocks is adjusted based on the resolution.

3. The method for tumor microenvironment state analysis based on medical images according to claim 1 or 2, characterized in that, The method further includes: The predefined immune resistance mechanisms are defined to include at least T-cell rejection, myeloid suppression, antigen-presenting defect, and tumor cell autonomous resistance. The gene set variation analysis method was used to calculate the activity score associated with the gene expression data and each of the predefined immune resistance mechanisms; The immune resistance mechanism corresponding to the highest activity score was selected as the dominant immune resistance mechanism in the gene expression data. All image blocks segmented from the same medical image are configured to share the same dominant immune resistance mechanism label.

4. The method for tumor microenvironment state analysis based on medical images according to claim 3, characterized in that, A drug resistance mechanism prediction model is constructed by combining the semantic features and the dominant immune drug resistance mechanism tags, specifically including: The semantic features are combined to form a feature matrix, and the spatial semantic joint complexity is calculated based on the feature matrix. The number of output layers of the immune drug resistance mechanism prediction model is dynamically set based on the spatial semantic joint complexity. Each of the predefined immune resistance mechanisms is configured to generate a corresponding category attention weight; An attention-weighted covariance matrix is ​​constructed by combining the category attention weights of the same image patch with the eigenvectors in the feature matrix; The attention-weighted covariance matrix is ​​vectorized using a learnable projection matrix to generate a feature representation of the immune resistance mechanism.

5. The method for tumor microenvironment state analysis based on medical images according to claim 4, characterized in that, The method further includes: Obtain the feature representation of the immune resistance mechanism, and extract the category attention weight associated with the feature representation of the immune resistance mechanism; The spatial coordinates of the image patch in the medical image are obtained, and the category attention weight associated with the immune resistance mechanism feature representation is mapped to the image plane in combination with the spatial coordinates to form a key region localization map. The key region localization map is used to indicate the key regions associated with the immune resistance mechanism feature representation.

6. The method for tumor microenvironment state analysis based on medical images according to claim 4, characterized in that, The calculation process of the joint spatial semantic complexity is expressed by formulas (1), (2), (3), and (4) as follows: in, Represents the joint complexity of spatial semantics; Indicates the normalization factor; Represent the covariance matrix; The feature matrix representing all image patches; This indicates the number of image blocks in a medical image; This represents the feature vector of the k-th image patch; This represents the spatial gradient of the k-th image patch. If image patch k has neighbors, calculate the gradient of image patch k with all its neighbors. Feature differences; if image patch k has no neighbors, set ; , Indicates learnable parameters; Represents the mean vector of H; This represents the dimension of the feature vector. Wherein, the image block k is related to all its neighbors The characteristic differences are obtained through formula (5): （5） in, Represents the neighborhood set of image patch k; and Let represent the feature vectors of image patch k and its neighbor n, respectively; express and The Euclidean distance between two eigenvectors.

7. The method for tumor microenvironment state analysis based on medical images according to claim 6, characterized in that, The method further includes: The number of output layers of the drug resistance mechanism prediction model is controlled by a lightweight gating unit, wherein the calculation formula (6) for the gating unit is as follows: （6） in, This represents the output of the gating unit, which is used to control the information flow intensity of the output layer of the drug resistance mechanism prediction model. Its value ranges from [0,1]. , These represent the learnable parameters, corresponding to the weight matrix and bias vector of the gated unit, respectively. This represents the Sigmoid activation function; Indicates the first The hidden state feature vector of the layer.

8. The method for tumor microenvironment state analysis based on medical images according to claim 7, characterized in that, The method further includes: The formula (7) for calculating the category attention weight of the k-th image patch of the i-th type of immune drug resistance mechanism in the l-th layer of the drug resistance mechanism prediction model is as follows: （7） in, The category attention weight represents the k-th image patch representing the i-th type of immune resistance mechanism in the l-th layer; , The first two layers of the drug resistance mechanism prediction model represent the weight matrix; N represents the parallel attention branches that the drug resistance mechanism prediction model splits into after the first two layers. The weight matrix representing the parallel attention branch; This represents the position code of the k-th image block; Represents the hyperbolic tangent activation function; Represents the Sigmoid function; This represents the element-wise multiplication of two matrices; The formula (8) for calculating the attention-weighted covariance matrix is ​​as follows: （8） in, Represents the attention-weighted covariance matrix of the i-th type of immune resistance mechanism; This represents the attention weight of the k-th image patch representing the i-th type of immune resistance mechanism in layer L; This represents the feature vector of the k-th image patch.

9. A tumor microenvironment state analysis system based on medical images, used to implement the tumor microenvironment state analysis method based on medical images according to any one of claims 1 to 8, characterized in that, The system includes: The input module is used to acquire gene expression data of medical images and tissue samples containing the medical images; The analysis module is used to divide the preprocessed medical image into image patches, extract the semantic features of the image patches using a preset model, and obtain the dominant immune resistance mechanism that matches the gene expression data in the predefined immune resistance mechanism, and set the dominant immune resistance mechanism as the dominant immune resistance mechanism label. The module combines the semantic features and the dominant immune resistance mechanism label to construct an immune resistance mechanism prediction model. The output module is used to train the immune resistance mechanism prediction model. The medical image to be analyzed is input into the trained immune resistance mechanism analysis model to output the immune resistance mechanism prediction category.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images