Gastric cancer immunotherapy curative effect prediction method based on multi-mode fusion

Abstract

The invention discloses a gastric cancer immunotherapy curative effect prediction method based on multi-modal fusion, which comprises the following steps of: extracting spatial and morphological characteristics of primary lesions and multiple metastases based on CT (Computed Tomography) images, and generating first microenvironment representation; then, the pathological section is segmented into a plurality of image blocks, cell-level instance extraction and hierarchical aggregation are completed through an improved UNETR model encoder, and second microenvironment representation is generated; constructing a clinical feature vector based on the clinical structured information, and executing cross-modal mapping with the two types of representations to obtain a third multi-modal fusion representation; and finally, constructing a joint sorting constraint through cross entropy loss and Cox regression loss in a training stage, and realizing unified optimization of short-term curative effect prediction and long-term survival trend modeling. The accuracy and the clinical stability of immunotherapy curative effect prediction of the advanced gastric cancer patient can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical artificial intelligence, in particular to a gastric cancer immunotherapy efficacy prediction method based on multi-modal fusion. BACKGROUND

[0002] Gastric cancer is one of the most common digestive tract malignancies worldwide. Patients with advanced gastric cancer generally face problems such as wide metastasis, short survival time, and significant heterogeneity of lesions. In recent years, immune checkpoint inhibitors (such as PD-1 inhibitors) combined with chemotherapy have been gradually applied to first-line or second-line treatment regimens for advanced gastric cancer. Some patients can achieve a high objective response rate. However, a large number of clinical follow-up results show that there are significant differences in the actual response of different patients to combined treatment, and the overall survival (OS) and progression-free survival (PFS) after treatment have not shown consistent improvement, indicating that traditional diagnostic methods cannot reflect the differences in the efficacy of combined treatment and its underlying biological mechanisms.

[0003] For treatment response prediction, existing technologies mainly rely on single-modal data analysis methods, such as imageomics models based on CT images, cytological or histological analysis models based on pathological sections, or statistical evaluation models based on clinical indicators. Due to significant differences in time scale, spatial resolution, and feature dimension between different data modalities, these single-modal methods cannot fully capture the complexity of gastric cancer. Relying solely on CT images can only reflect changes in organ-level structure and cannot describe cellular heterogeneity; although pathological sections can provide microscopic structural information, they cannot reflect important features such as whole-body tumor burden and changes in metastatic lesions; clinical structured indicators only provide limited representation capabilities and cannot support in-depth inference of the immune response state of patients.

[0004] With the development of deep learning and multi-modal modeling techniques, some studies have attempted to integrate image, pathology, and clinical information to improve prediction capabilities. However, existing multi-modal fusion models still have several technical bottlenecks, including: lack of a unified feature space between different modalities, making it difficult to effectively integrate cross-scale features through simple concatenation or weighting strategies; image models usually focus on primary lesions and do not systematically incorporate metastatic lesion information, making it impossible to reflect the overall burden of disease progression; pathology models are often based on slice patches for learning and do not fully utilize the structural differences at the cellular level, which can easily miss important expressions of the tissue microenvironment; in addition, most existing models are trained using binary classification loss and do not consider the inherent relationship between treatment response and survival risk, resulting in instability in the clinical significance of the prediction results.

[0005] In view of the above, the present application proposes a gastric cancer immunotherapy efficacy prediction method based on multi-modal fusion. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal fusion-based gastric cancer immunotherapy efficacy prediction method. The existing single-mode or simple multi-modal model cannot accurately depict the differences between macroscopic lesions and microscopic tissue structures of gastric cancer, and thus cannot accurately predict the real efficacy of PD-1 combined with chemotherapy.

[0007] In a first aspect, the present application provides a multi-modal fusion-based gastric cancer immunotherapy efficacy prediction method, comprising the following steps: S101: Extract the spatial and morphological information of primary lesions and metastatic lesions based on CT image data, construct a lesion feature set for representing the overall structure and texture features of the lesions, and form a first lesion representation accordingly; S102: Cut the pathological whole section image to obtain a target image block set, extract the cell-level structure features in the target image block set based on an improved UNETR model encoder, and form a second microenvironment representation through a multi-instance learning-based hierarchical aggregation method; S103: Construct a clinical feature vector based on clinical structured information, perform cross-modal mapping processing on the clinical feature vector, the first lesion representation, and the second microenvironment representation, and generate a third multi-modal fusion representation; S104: In the model training phase, the treatment response label and survival, progression time are used as labels, respectively, and the outputs of the efficacy prediction branch and the risk prediction branch are based on cross-entropy loss and Cox regression loss to form a joint ordering constraint, and the prediction network is updated through joint backpropagation; In the inference phase, the immunotherapy efficacy prediction result of a single patient is output.

[0008] As a preferred technical solution of the present application, the acquisition logic of the lesion feature set is: A pre-trained SwinUnetR model encoder is used to extract the deep feature vector of the CT image primary lesion, a pyradiomics library is used to extract the imageomics feature vector of the CT image primary lesion, such as tumor size and surface texture, and a 3D-ResNet-50 model is used to extract the CT image metastasis lesion feature vector; Using an intra-modal feature fusion method, the three groups of feature vectors are fused to obtain the first lesion representation.

[0009] As a preferred technical solution of the present application, the construction of the second microenvironment representation includes: The pathological whole section image is cut to obtain a target image block set; Each target image block is input into an UNETR model improved based on a Transformer structure to obtain cell-level instance features such as cell nucleus category, cell contour shape, and cell spatial coordinates; According to the cell spatial coordinates, the intercellular distance relationship is calculated, and a cell adjacency graph is constructed based on a preset neighborhood radius or relative distance, which is used to represent the local cell structure. The co-occurrence structure of the same type of cells and different types of cells is weighted and combined according to the adjacency relationship of the cell adjacency graph, a cell structure vector is generated, the cell structure vector is fused with the texture features of the corresponding image block to obtain the fusion features of the target image block, and the pathological whole slice image is divided into multiple bags through a multi-instance learning mode, and the patch-level features in the bag are aggregated to form bag features.

[0010] As a preferred technical solution of the application, the improved UNETR model comprises: The pathological whole slice image data is divided into a plurality of image blocks of a fixed size, and the image blocks are converted into token sequences through a PatchEmbedding module. The token sequence is input into an encoding structure composed of multiple layers of Transformer blocks, and multi-head self-attention operation and feedforward network operation are sequentially performed in each Transformer block to extract local and global spatial structure features.

[0011] As a preferred technical solution of the application, the improved UNETR model further comprises, at the output end of the encoding structure: A channel attention enhancement module performs global pooling and channel re-labeling on each channel feature to enhance the response capability of key cell regions. A position encoding adaptation module is used to support variable size pathological image inference in the non-training stage, and to avoid calculation errors caused by mismatched token sizes. A three-channel branch decoding structure is used to output a nucleus prediction map, a boundary map and a cell category map respectively, to improve the compatibility of the output results in the post-processing and evaluation stages.

[0012] As a preferred technical solution of the application, the acquisition logic of the third multi-modal fusion representation is: Based on the TNM stage, the number of metastatic sites, the ECOG score, the pathological score and the construction of the clinical feature vector, the clinical feature vector is input into an intra-modal alignment module to perform weighted rearrangement on the first lesion representation to form a clinically enhanced feature. The clinically enhanced feature and the second microenvironment representation are input into a cross-modal mapping unit to establish a bidirectional mapping relationship between the image modality and the pathology modality, and the third multi-modal fusion representation is generated.

[0013] As a preferred technical solution of the present application, the constructed joint ordering constraint comprises: Based on the survival time and progression time and the deletion state of each training sample, a survival ordering pair is constructed, the consistency deviation between the risk prediction output of two samples in each survival ordering pair and the survival time sequence is calculated, and an ordering constraint term is formed; The cross-entropy loss and the Cox regression loss are used jointly, and the prediction network is updated by mixed gradient back propagation.

[0014] As a preferred technical solution of the present application, the inference process comprises: In the inference stage, the pathological whole slice image is cut into patches, and after the token features are extracted by the last layer of the encoder, global average pooling is performed to obtain whole image level sparse WSI features; The pathological whole slice image is divided into multiple bags in a multi-instance learning manner, and self-attention aggregation is performed on the patch level features in the bag to form bag features; Based on the multi-bag feature aggregation, a whole image level pathological feature vector is obtained, and a third multi-modal fusion representation is generated together with the image modal and clinical feature vector.

[0015] In the above technical solution, the present application provides the technical effects and advantages: The present application extracts deep features, tumor volume, surface texture and other multi-source features from CT primary lesions and metastatic lesions to form a lesion level feature set representing the overall structure and texture properties of the lesion, forming a first lesion representation, so that the lesion load and dissemination state at the image level are comprehensively described. Secondly, cell level instance features such as cell nucleus category, contour shape and spatial distribution and corresponding texture features are obtained on the pathological whole slice image, and then the second microenvironment representation is obtained through feature aggregation and global pooling, so as to finely represent the tumor immune microenvironment state at the pathological level. Thirdly, the clinical structured information is encoded into a feature vector, which is input into the cross-modal mapping unit together with the first lesion representation and the second microenvironment representation to complete the alignment and deep fusion between the clinical, image and pathological modalities, and form the third multi-modal fusion representation. In the training stage, the ordering relationship between the risk prediction outputs and the real survival and progression order are introduced to form joint ordering constraints, so that the model can simultaneously optimize the consistency of short-term response and long-term survival trend. Therefore, the present application can significantly improve the accuracy and stability of immunotherapy efficacy prediction, and provide a reliable basis for individualized treatment decision. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0017] Figure 1 Flow chart of the gastric cancer immunotherapy efficacy prediction method of the present application; Figure 2 Schematic diagram of the Squeeze-and-Extraction Block structure of the present application; Figure 3 Schematic diagram of the model prediction process of the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will describe the technical solutions in the embodiments of the present application in more detail with reference to the drawings in the embodiments of the present application.

[0019] In the drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The described embodiments are part of the embodiments of the present application, not all of the embodiments. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. The embodiments of the present application will be described in detail below with reference to the drawings.

[0020] Embodiment 1 Please refer to Figure 1 As shown in the drawings, the present embodiment provides a gastric cancer immunotherapy efficacy prediction method based on multi-modal fusion, which is used for accurately predicting the efficacy of gastric cancer immunotherapy. The method integrates CT images, pathological whole slice images and clinical structured information, and adopts intra-modal joint enhancement and double ordering constraint for optimization.

[0021] Obtain multi-modal data of a gastric cancer patient, the multi-modal data comprising CT image data, pathological whole slice (WSI) image data and clinical structured information; wherein: the clinical structured information comprises TNM stage, metastatic site, ECOG score, pathological score, etc.

[0022] S101: Extract the spatial and morphological information of the primary lesion and the metastatic lesion based on the CT image data, construct a lesion feature set for representing the overall structural characteristics of the lesion, and form a first lesion representation accordingly.

[0023] Specifically, the preprocessing method of CT image data is as follows: First, the CT image data corresponding to the labeled abdominal primary lesion region of the advanced gastric cancer patient is segmented by using the SwinUnetR model. The SwinUnetR model performs full-supervised pre-training of primary lesion segmentation, so that the SwinUnetR model has the ability to segment gastric tissue and gastric cancer primary lesions. Second, the 3D-ResNet-50 model is used to detect the CT image data corresponding to the labeled metastatic lesion region of the advanced gastric cancer patient. The 3D-ResNet-50 model performs full-supervised pre-training of metastatic lesion detection, so that the 3D-ResNet-50 model has the ability to extract gastric cancer metastatic lesions. Then, after the pre-training of the SwinUnetR model and the 3D-ResNet-50 model is completed, the primary lesion features and the metastatic lesion features of the patient CT image data are extracted as intra-group modality 1 and intra-group modality 2, respectively, and the imageomics features of the CT image are extracted as intra-group modality 3 through the Pyradiomics library. That is, the intra-group modality 1, the intra-group modality 2 and the intra-group modality 3 together constitute the CT modality data.

[0024] Specifically, the preprocessing method based on CT image data extracts intra-group modality 1, intra-group modality 2 and intra-group modality 3, including the following steps: Based on the CT image data, the primary lesion region and the plurality of metastatic lesion regions are determined, the prediction mask of the primary lesion segmentation is obtained from the SwinUnetR decoder output, the imageomics features are extracted using the Pyradiomics library according to the prediction mask, and the primary lesion feature vector and the metastatic lesion feature vector are extracted from the encoder output of the SwinUnetR and the last convolutional layer output of the 3D-ResNet-50. Then, the first lesion representation is formed through splicing and the feature interaction mechanism of the Transformer encoder.

[0025] S102: The target image block set is obtained by patch processing the pathological whole section image, and the cell-level structure features and image block texture features in the target image block set are extracted based on the improved UNETR model. Then, the second microenvironment representation is formed through a hierarchical aggregation method based on multi-instance learning. Specifically, the construction process of the second microenvironment representation includes the following steps: First, the cell segmentation classification model is trained on the PanNuke dataset using a supervised learning strategy. The training image is input into the model in a 256x256 patch manner, and the cross-entropy loss and DICE loss are combined to optimize the nuclear segmentation, boundary detection and classification ability of the model. For each training image, it is input into the UNETR model improved based on the Transformer structure, which adopts the encoding structure based on ViT (Vision Transformer), divides the target image block into fixed scale patches, converts it into token sequence through the Patch Embedding module, and passes through the multi-layer self-attention module and the feedforward network for deep modeling to obtain the cell-level instance features across the region, including: cell nucleus category, cell contour shape and cell spatial coordinates.

[0026] It should be noted that, in order to realize the accurate identification and expression of cell-level structure in digital pathology sections, the present application learns from the principle of ViT (Vision Transformer), and the improved UNETR bottom architecture is used in the ViT model to process and model the pathological whole section image. The image segmentation framework based on Transformer integrates the local feature extraction ability of CNN and the global modeling ability of Transformer, and has the ability of cross-image region modeling and reasoning.

[0027] Specifically, the improved UNETR model includes: dividing the image into a plurality of fixed size image blocks (patches), and then flattening and mapping it into visual tokens, so as to model the global dependency relationship between tokens by means of the Transformer architecture, as shown in Figure 2 As shown in the figure, the pathological whole section image data is first divided into a plurality of small patches with a size of 256x256 pixels, and then converted into a token sequence with a dimension of by the Patch Embedding module (i.e. convolution operation with a step equal to the size of the patch), and input into a plurality of stacked Transformer blocks for deep modeling. Each layer of the Transformer block contains a self-attention module (Multi-head Self-Attention, MHSA) and a feedforward neural network (MLP), which can capture local and global spatial structure features.

[0028] In order to improve the modeling ability of complex tissue regions, the UNETR model further includes: Channel attention enhancement (SE Block) module: a Squeeze-and-Excitation module is introduced in the output stage of the Transformer encoder, as shown inFigure 3 As shown, global pooling and channel recalibration operations enhance the response capability of key cell regions and improve the model's ability to focus on lesion regions. Position encoding adaptation module: Adjust the model position encoding module to support inference computation under input sizes (such as 2048×2048, 1024×1024 patch) during the non-training stage, and avoid model crashes caused by token size mismatch. Three-channel branch decoding structure: A three-channel branch (nucleus prediction map, boundary map, and cell category map) is adopted in the decoder output stage to ensure the compatibility of the model structure in the post-processing and evaluation stages.

[0029] Based on the extracted cell spatial coordinates, the distance relationships between cells are calculated, and a cell adjacency graph is constructed based on a preset neighborhood radius or relative distance to characterize local cell structures. Based on the cell-level adjacency relationships, a hierarchical aggregation operation is performed, including: Based on adjacency relationships, the co-occurrence structures of similar and dissimilar cells are weighted and combined to generate cell structure vectors. The cell structure vectors are then fused with the texture features of the corresponding image patches to obtain the fusion features of the target image patches. Finally, the second microenvironment representation is formed by aggregating the features using a region-weighted approach.

[0030] The UNETR model employs a multi-task learning framework during training, using different loss functions for branches such as the cell nucleus region (binary map), boundary orientation map (HV map), and cell type map (type map) to comprehensively improve the model's performance in segmentation, boundary modeling, and cell recognition. The loss functions for each branch are set as follows: Channel branch 1 uses a nuclei binary map to detect the foreground and background (segmentation mask) of the cell nucleus, and employs three loss functions for joint optimization: a. Binary Cross Entropy (BCE) is used to handle binary classification problems and measures the distance between the model's predicted probability and the true label. ; in: It is the first The true labels of each sample It is the first Predicted probability for each sample This represents the total number of samples in the current training process. For the output of sigmoid: ; in: This indicates that the model is for the first... The original output value of the sample.

[0031] b. DICE Loss, emphasizing the imbalance problem between foreground and background, improving the detection ability of small targets; ; where ϵ is a small constant for numerical stability.

[0032] c. Focal Tversky Loss, used to improve the robustness of the model to imbalanced samples (such as small targets). Combines Tversky index and Focal strategy; ; ; where: an index for measuring the degree of overlap between the model's prediction and the true label, is the true label (0 or 1) of the th sample, is the prediction probability (between 0 and 1) of the th sample, represents true positive, when the true label is 1 and the prediction probability is close to 1; represents false positive, when the true label is 0 and the prediction probability is close to 1; represents false negative, when the true label is 1 and the prediction probability is close to 0; represents false positive weight, used to adjust the penalty weight coefficient of false positive. represents false negative weight, used to adjust the penalty weight coefficient of false negative; represents the final loss function value of Focal Tversky loss function, represents the focus parameter, used to adjust the loss weight, so that the model focuses on more difficult to distinguish samples or pixels during training.

[0033] Channel branch 2 uses horizontal and vertical direction maps (hv_map) to predict the horizontal and vertical direction information of the cell nucleus boundary, and two loss functions are used for cooperative optimization: a. Mean Squared Error (MSE) Loss, standard regression loss, used to evaluate the difference between continuous values: ; b. MSGE Loss (Mean Squared Gradient Error), especially for direction map training, measures the error between predicted gradient and true gradient to enhance boundary clarity: ; Where ∇ represents the gradient calculation operation under the Sobel or Scharr filter.

[0034] Channel branch 3 uses a cell type map (nuclei_type_map) to classify each pixel into different cell types (such as tumor cells, inflammatory cells, etc.) for multi-class segmentation: a. Categorical Cross Entropy (CCE), used for multi-class classification problems: ; Where: C is the number of categories. The true label for one-hot encoding The predicted probability after softmax; b. DICE Loss (Multi-class version): Calculates DICE separately for each class and takes the average. ; in: This represents the multi-class DICE loss, which is a loss function obtained by calculating the DICE coefficient for each class separately and then averaging it across all classes. This represents the number of categories, the total number of cell types the model needs to distinguish. This represents the category index, used to iterate from 1 to... All categories, This represents the pixel index, which is accumulated across all pixels in the image. Indicates the first The pixel in the first The true label for each category (0 or 1). Indicates the first The pixel in the first The predicted probabilities of each category are typically the output after applying Softmax. It represents a small constant, a very small positive number, used for numerical stability to prevent the denominator from being zero.

[0035] The total loss function (multi-task fusion) is ultimately composed of the weighted sum of each branch, with each loss term assigned a corresponding weight: ; in: Let T represent the total loss function, and T represent the set of output branches. This represents the branch index, which is a specific output branch in set T. This represents the set of loss functions used in this branch. Represents the loss index, which is a set. one of the specific loss terms in the loss function, denotes a weight of each loss term; denotes a loss function value, branch under the loss term calculation value.

[0036] After the UNETR-based cell classification and segmentation model is trained, the whole slice pathological image is cut into patches to obtain a target image block set, the UNETR encoder is used to extract the texture features of the target image block, the decoder extracts the cell-level instance features, the distance relationship between cells is calculated according to the cell spatial coordinates, and the cell adjacency graph is constructed based on the preset neighborhood radius or relative distance, which is used to represent the local cell structure; According to the adjacency relationship of the cell adjacency graph, the co-occurrence structure of the same type of cells and different types of cells is weighted and combined to generate a cell structure vector. The cell structure vector and the texture features of the corresponding image block are fused to obtain the fusion features of the target image block. And through multi-instance learning, the pathological whole slice image is divided into multiple bags, and the self-attention aggregation is performed on the patch-level features in the bag to form bag features. Based on the multi-bag feature aggregation, a whole image-level pathological feature vector is obtained to form a second microenvironment representation.

[0037] S103: Construct a clinical feature vector based on the clinical structured information, perform cross-modal mapping processing on the clinical feature vector, the first lesion representation and the second microenvironment representation, and generate a third multi-modal fusion representation.

[0038] Specifically, the clinical structured information includes TNM staging, number of metastatic sites, tumor burden indicators, ECOG scores, pathological scores, inflammation indicators, and previous treatment records of the patient, and other structured quantitative data for describing the patient's basic disease state. Based on the clinical structured information, a corresponding clinical feature vector is constructed through a multi-layer perceptron (MLP) to represent the overall clinical features of the patient.

[0039] After generating the clinical feature vector, the clinical feature, the first lesion representation and the second microenvironment representation are input into the cross-modal mapping unit, which includes a bilinear transformation layer or a cross-modal attention structure, to establish a learnable mapping relationship between the clinical, image and pathological modalities. By performing bidirectional mapping, cross-attention or bilinear fusion operations on the feature matrices of different modalities, the microscopic cell structure, tissue microenvironment state and macroscopic lesion structure features are aligned in a unified feature space.

[0040] After completing the cross-modal mapping, the fused features are aggregated and transformed to obtain a third multi-modal fusion representation that can simultaneously represent the macro-lesion state of the patient, the micro-tissue structure, and the clinical immune background, which is used for subsequent efficacy prediction and survival risk modeling.

[0041] In the model training phase, the treatment response label and the survival and progression time are used as labels to form a joint ordering constraint based on the cross-entropy loss and the Cox regression loss for the outputs of the efficacy prediction branch and the risk prediction branch, respectively. The efficacy prediction branch uses cross-entropy loss to optimize the binary treatment response discrimination, and the survival risk prediction branch constructs a risk set based on the follow-up time and the censoring state, and uses Cox regression loss to constrain the ordering consistency between the risk prediction output and the actual survival time and progression-free survival time. By jointly backpropagating the cross-entropy loss and the Cox regression loss, the model can simultaneously optimize short-term efficacy and long-term survival trends. In the inference phase, the immune therapy efficacy prediction result is output based on the multi-modal data of a single patient.

[0042] Further, the construction of the survival ordering pair includes: in the training phase, the training samples are ordered according to the survival time, and the risk comparison pairs are constructed for the uncensored samples in combination with the censoring state; the training samples are grouped according to different treatment schemes, and the adjacent sample pairs are constructed within the group according to the survival time, while different weights are set for the sample pairs in different groups, so that the contribution of each sample pair in the joint ordering constraint is consistent with the difference in treatment schemes.

[0043] To ensure the compatibility of the ordering constraint and the multi-modal fusion model, and to ensure the compatibility of the ordering constraint and the multi-modal fusion model at the feature expression level, the following is explained: In the training and inference phases, the whole slide image (WSI) is divided into non-overlapping patches (patch_overlap=0) of 256x256, each patch is independently predicted, and finally the whole image is spliced and restored. To extract the pathological modality for the sparse feature representation of the whole image level for multi-modal fusion, the token feature vector is extracted from the last layer of the model encoder, and the global average pooling is performed to obtain a WSI feature with a dimension of The WSI feature is input to the downstream multi-modal fusion module as the overall pathological representation of the patient, and can be used for efficacy prediction in combination with clinical information, CT modality, etc.

[0044] Since the pixel size of a pathological whole slide image (WSI) is usually several hundred million pixels, it cannot be directly input into a neural network for processing. Therefore, we use multi-instance learning to cut each pathological whole slide image (WSI) into several "bags" of 2048x2048 size. According to the Otsu binarization method, we set the threshold to judge the proportion of white background, extract the pathological tissue from the white background of the pathological whole slide image (WSI), and then cut each "bag" into several "image blocks" of 256x256 pixels. We use the first four parts of the trained cell nucleus segmentation model to extract key features. In addition, in order to fully consider the information interaction between different image blocks, we use a weighted aggregation method based on self-attention mechanism. The features of each "bag" come from the aggregation of the features of several "image blocks", and the features of each WSI come from the aggregation of the features of several "bags". Finally, we get a sparse feature vector of a WSI as the intra-group modal 1. ; ; wherein: represents the input feature matrix, represents in the real space, wherein is the number of Words in the Bag, is the feature dimension. represents the Word-level feature matrix after multi-head self-attention mechanism processing, which enhances the context-dependent relationship. represents the multi-head self-attention function, which is used to calculate the dependency between input features. represents the input Word feature vector: represents the feature vector of each image block (Word) segmented in the Bag.

[0045] The self-attention mechanism is represented as: ; ; ; ; wherein: represents the input feature matrix, in , and calculation, usually refers to , that is, the feature matrix input to the self-attention layer. represents the query matrix, which is obtained by multiplying the input feature by the learnable weight . denotes a key matrix, obtained by inputting the features multiplied by learnable weights obtained matrix. denotes a value matrix, obtained by inputting the features multiplied by learnable weights obtained matrix. , and denotes a learnable weight matrix, used to project the input features to , and the weight matrix of the space.

[0046] denotes an attention score, obtained by multiplying the transpose of and dividing by a scaling factor obtained original similarity score, denotes the feature dimension of the key matrix K, and the subscript k represents the dimension corresponding to K, not the matrix K itself.

[0047] denotes an attention weight, obtained by applying a Softmax function to the attention score , whose row sum is 1, denotes an activation function, used to convert the attention score into a probability distribution, denotes an attention output feature, obtained by multiplying the attention weight by the value matrix the final Word-level feature.

[0048] weighted sum to obtain bag representation: ; ; wherein: denotes a feature vector, which is the weighted sum of all Word features in the Bag, denotes the total number of image blocks (Words) contained in a single Bag, denotes the attention weight of the th Word in the Bag, denotes the output feature of the th Word in the Bag after self-attention mechanism processing, denotes the real space where the Bag representation is located, is the feature dimension; a matrix composed of the feature vectors of all Bags , Representation matrix The first in The feature vector of each Bag This represents the total number of Bags into which the data is divided.

[0049] Similarly, the representation of the final aggregated full-image level pathological feature vector WSI is obtained: ; in: Indicates the first In the bag, the first Attention weight of each word This represents the attention output features of the word. Indicates the first The attention weights of each bag are determined, and then Pyradiomics is used to extract omics features for each "bag". The intra-group modality of each WSI is obtained by average aggregation. The stitching method was used to analyze the internal features of the imaging modality and the pathological modality. After fusion, the classification features are aggregated into a cls token using a Transformer. A bilinear layer is then used to fuse the cls token information obtained from different modalities. The bilinear layer formula is expressed as: ; in: It is a two-modal cls token. It is the learnable weight matrix of the bilinear layer. It is the feature matrix after fusion.

[0050] The fused classification features are input into a linear layer for the final classification and risk outputs. The classification result indicates whether the patient has responded to drug treatment. The loss function is a weighted average of the binary cross-entropy loss (BCEWithLogits Loss) and the negative log-partial likelihood loss function of the Cox regression model (Cox Loss). The cross-entropy loss is used to optimize the classification result, while the Cox loss serves as a key parameter guiding whether the model's output risk result conforms to the medically defined risk level for survival.

[0051] The binary cross-entropy loss is as follows: ; in: The first output of the model represents the... The original logit value of each sample, Indicates the first The true label of each sample, where N represents the total number of samples in the current batch.

[0052] Cox Loss is represented as: ; in: Indicates the first The predicted risk score of each sample (represented by the output of the fully connected layer); Censoring indicator: 1 indicates no censoring (event observed), 0 indicates censoring; samples are sorted in ascending order of time, assuming the first... The sample corresponds to the first Short survival time; Indicates the first The risk set of n samples (representing all samples with a survival time ≥ n) (sample of samples) It is the sum of the exponential risks of all individuals in the risk set, and only applies to the uncensored samples. Calculate the log-likelihood and then divide by the number of uncensored samples. Normalize.

[0053] The overall loss function can be expressed as: ; in, This represents the final loss value used during model training, which is the weighted sum of all loss terms. This represents the binary cross-entropy loss, used to optimize classification results, and is a loss function indicating whether a patient is responding to drug treatment. The overall survival Cox loss represents the negative log-partial likelihood loss function based on the Cox regression model, used to guide the model's output of the risk outcome for overall survival (OS). The Cox loss represents the progression-free survival (PFS) loss, which is a negative log-partial likelihood loss function based on the Cox regression model. It is used to guide the model's output of the risk outcome of progression-free survival (PFS). This represents the overall survival loss weighting coefficient, used for balancing Contribution to the overall loss function, This represents the weighting coefficient for progression-free survival loss, used to balance its contribution to the overall loss function.

[0054] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gastric cancer immunotherapy efficacy prediction method based on multi-modal fusion, characterized in that, The method comprises the following steps: S101: Extract the spatial and morphological information of the primary lesion and the metastatic lesion based on the CT image data, construct a lesion feature set for representing the overall structure and texture features of the lesion, and form a first lesion representation accordingly; S102: Obtain a target image block set by patch processing of the pathological whole section image, extract cell-level structure features in the target image block set based on an improved UNETR model encoder, and form a second microenvironment representation by a hierarchical aggregation method based on multi-instance learning; S103: Construct a clinical feature vector based on clinical structured information, perform cross-modal mapping processing on the clinical feature vector, the first lesion representation, and the second microenvironment representation, and generate a third multi-modal fusion representation; S104: In the model training stage, the treatment response label and the survival and progression time are used as labels, respectively, and the outputs of the efficacy prediction branch and the risk prediction branch are combined based on cross-entropy loss and Cox regression loss to form a joint ranking constraint, and the prediction network is updated by joint back propagation; In the inference stage, the immunotherapy efficacy prediction result of a single patient is output.

2. The multi-modal fusion-based gastric cancer immunotherapy efficacy prediction method of claim 1, wherein, The acquisition logic of the lesion feature set is: A pre-trained SwinUnetR model encoder is used to extract a deep feature vector of the primary lesion of the CT image, a pyradiomics library is used to extract imageomics feature vectors such as tumor size and surface texture of the primary lesion of the CT image, and a 3D-ResNet-50 model is used to extract a metastatic lesion feature vector of the CT image; The three feature vectors are fused by an intra-modal feature fusion method to obtain a first lesion representation. 3.The multi-modal fusion based gastric cancer immunotherapy efficacy prediction method of claim 1, wherein, The construction of the second microenvironment representation comprises: The pathological whole section image is processed by patching to obtain a target image block set; Each target image block is input into an UNETR model improved based on a Transformer structure to obtain cell-level instance features such as cell nucleus category, cell contour shape, and cell spatial coordinates, and texture features of the corresponding image block are obtained through an UNETR model encoder; According to the cell spatial coordinates, the intercellular distance relationship is calculated, and a cell adjacency graph is constructed based on a preset neighborhood radius or relative distance to represent the local cell structure; The co-occurrence structure of homocellular and heterocellular cells is combined by weighting according to the adjacency relationship of the cell adjacency graph, a cell structure vector is generated, the cell structure vector and the texture features of the corresponding image block are fused to obtain the fusion features of the target image block, and the pathological whole section image is divided into multiple bags by a multi-instance learning method, and the patch-level features in the bag are aggregated by self-attention to form bag features; The whole image level pathological feature vector is aggregated based on multiple bag features to form a second microenvironment representation.

4. The multi-modal fusion-based gastric cancer immunotherapy efficacy prediction method of claim 3, wherein, The improved UNETR model comprises: The pathological whole section image data is divided into multiple image blocks of a fixed size, and the image blocks are converted into token sequences by a PatchEmbedding module; The token sequence is input into an encoding structure composed of multiple Transformer blocks, and in each Transformer block, multi-head self-attention operation and feedforward network operation are sequentially performed to extract local and global spatial structure features.

5. The multi-modal fusion-based gastric cancer immunotherapy efficacy prediction method of claim 4, wherein, The improved UNETR model further comprises, at an output end of the encoding structure: A channel attention enhancement module performs global pooling and channel recalibration on each channel feature to enhance the response capability of key cell regions; A position encoding adaptation module is used to support variable-size pathological image inference in the non-training stage, and to avoid calculation errors caused by token size mismatch; A three-channel branch decoding structure is used to output a nucleus prediction map, a boundary map and a cell category map respectively, to improve the compatibility of the output results in the post-processing and evaluation stages. 6.The multi-modal fusion based gastric cancer immunotherapy efficacy prediction method of claim 1, wherein, The third multi-modal fusion representation is obtained by: Constructing a clinical feature vector based on TNM staging, metastatic sites, ECOG scores and pathological scores; Inputting the clinical feature vector, the first lesion representation and the second microenvironment representation into a cross-modal mapping unit to establish a mapping relationship between the clinical, imaging and pathological modalities, and generating the third multi-modal fusion representation. 7.The multi-modal fusion based gastric cancer immunotherapy efficacy prediction method of claim 1, wherein, The construction of the joint ranking constraint includes: Based on the survival time and the censored state of each training sample, a survival ranking pair is constructed, the consistency deviation between the risk prediction output and the survival time order of the two samples in each survival ranking pair is calculated, and a ranking constraint term is formed; Cross-entropy loss and Cox regression loss are used jointly, and the prediction network is updated by mixed gradient backpropagation.

Images