Intelligent tumor infiltrating immune cell precise typing system
By using a multimodal fusion model and a multiscale deep network model, the problems of isolated modal data and multiscale complexity in traditional typing systems are solved, thereby improving the accuracy and stability of immune cell typing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- THE FIRST AFFILIATED HOSPITAL OF CHONGQING MEDICAL UNIVERSITY
- Filing Date
- 2026-06-09
- Publication Date
- 2026-07-07
AI Technical Summary
Traditional tumor-infiltrating immune cell typing systems rely on single-modality data or simple early fusion strategies, lack in-depth modeling of the complex interactions between multimodal data, struggle to capture the multidimensional nature of immune cell functional states, and are unable to cope with the multi-scale complexity of the tumor microenvironment, resulting in unstable typing performance.
A multimodal fusion model is adopted to integrate multimodal features, and a multi-scale deep network model is used for classification prediction. Biological prior knowledge is combined for modality alignment and interaction, and local details and global relationship modeling are integrated.
It improves the accuracy and generalization of immune cell typing, enhances the ability to identify complex immune cell subsets, and improves the stability of typing performance.
Smart Images
Figure CN122347997A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of tumor-infiltrating immune cell typing technology, specifically to an intelligent precision typing system for tumor-infiltrating immune cells. Background Technology
[0002] Precise typing of tumor-infiltrating immune cells refers to the automatic and accurate classification and identification of immune cells infiltrating tumor tissue and its surrounding microenvironment in digital pathological whole-slice images using technologies such as deep learning, computer vision, and image analysis. This system achieves intelligent typing of immune cell subtypes by learning multi-level information such as cell morphology, spatial distribution, molecular marker expression, and microenvironment structure. This provides objective and quantifiable scientific evidence for predicting tumor immunotherapy responses, prognostic assessment, and precision medicine, improving the consistency and efficiency of pathological analysis.
[0003] However, traditional tumor-infiltrating immune cell typing systems suffer from technical problems such as reliance on single-modality data or simple early fusion strategies, lacking in-depth modeling of the complex interactions between multimodal data, resulting in isolated morphological and molecular data, making it difficult to capture the multidimensional nature of immune cell functional states. Traditional tumor-infiltrating immune cell typing systems also suffer from technical problems such as difficulty in dealing with the inherent multi-scale complexity of the tumor microenvironment, inability to coordinate the correlation between abnormal local immune cell distribution and global immune response trends, potential omission of key immune escape drivers, and single feature representation level and insufficient modeling of immune cell spatial heterogeneity, thus limiting the stability of typing performance. Summary of the Invention
[0004] To address the above issues and overcome the shortcomings of existing technologies, this invention provides an intelligent, precise typing system for tumor-infiltrating immune cells. Traditional tumor-infiltrating immune cell typing systems rely on single-modality data or simple early fusion strategies, lacking in-depth modeling of the complex interactions between multimodal data. This results in morphological and molecular data remaining isolated, making it difficult to capture the multidimensional nature of immune cell functional states. This solution creatively employs a multimodal fusion model to integrate multimodal features, aligning with the nature of the tumor immune microenvironment being driven by morphological, molecular mechanisms, and clinical factors. Guided by prior biological knowledge, it deeply aligns and interacts data from different modalities, thereby revealing the intrinsic relationship between immune cell morphological characteristics and molecular functions, and improving feature representation. The complementary nature of traditional tumor-infiltrating immune cell typing systems is addressed by addressing their limitations in handling the inherent multi-scale complexity of the tumor microenvironment, their inability to coordinate the correlation between abnormal local immune cell distribution and global immune response trends, their potential to miss key immune escape drivers, and their simplistic feature representation and insufficient modeling of immune cell spatial heterogeneity, which restrict the stability of typing performance. This solution creatively employs a multi-scale deep network model as the typing prediction model. By simultaneously capturing subtle immune cell morphological patterns and the overall biological context, it can more comprehensively analyze the multi-scale features of immune cell typing, integrating local detail perception with global relationship modeling. This enhances the model's ability to identify complex immune cell subpopulations, improving the accuracy and generalization of typing.
[0005] The technical solution adopted by the present invention is as follows: The present invention provides an intelligent tumor infiltrating immune cell precise typing system, including a pathological data acquisition module, a data preprocessing module, a multimodal interaction module, a typing prediction model construction module, and an immune cell typing prediction module;
[0006] The pathological data acquisition module acquires data to obtain a raw dataset for classification prediction and sends the raw dataset for classification prediction to the data preprocessing module.
[0007] The data preprocessing module employs data preprocessing methods such as coarse data processing, data standardization, preliminary feature extraction, and dataset segmentation to obtain the current dataset to be fused, the fusion training set, and the fusion test set, and then sends the current dataset to be fused, the fusion training set, and the fusion test set to the multimodal interaction module.
[0008] The multimodal interaction module is used to fuse multimodal features, and by designing a multimodal fusion model, obtain the current dataset to be predicted, the prediction training set, and the prediction test set, and send the current dataset to be predicted to the immune cell typing prediction module, and send the prediction training set and the prediction test set to the typing prediction model construction module.
[0009] The typing prediction model construction module is used to construct a model required for predicting the typing of tumor-infiltrating immune cells. It constructs a multi-scale deep network model as the typing prediction model and sends the typing prediction model to the immune cell typing prediction module.
[0010] The immune cell typing prediction module specifically performs real-time tumor infiltrating immune cell typing prediction based on the typing prediction model to obtain immune cell typing prediction results.
[0011] Furthermore, in the pathological data acquisition module, the original dataset for classification prediction specifically includes a historical original dataset for classification prediction and a current original dataset for classification prediction. Both the historical original dataset for classification prediction and the current original dataset for classification prediction include pathological image data, genomic data, and clinical pathological data. The historical original dataset for classification prediction also includes tumor infiltrating immune cell classification annotation data.
[0012] Furthermore, in the data preprocessing module, the coarse data processing is used to ensure the reliability and consistency of the input data. Specifically, it removes obvious abnormal data and corrects technical deviations. Pathological image data uses a context-aware generative model to fill missing regions. Genomic data uses k-nearest neighbor imputation combined with pathway information to fill missing values. Clinical pathological data uses an iterative random forest algorithm to fill missing values.
[0013] The data standardization is used to eliminate technical batch effects and dimensional differences, specifically by processing different modal data separately through a distribution-aware standardization strategy.
[0014] The preliminary feature extraction is used to extract meaningful feature representations from the raw data. Specifically, the pathological image data is segmented into 256×256 pixel image blocks and features are extracted using EfficientNet. Genomic data is extracted using non-negative matrix factorization, and clinical pathological data is converted into structured feature vectors.
[0015] The dataset segmentation is used to obtain training data and test data, specifically by segmenting the original dataset of historical classification prediction.
[0016] The current classification prediction dataset is preprocessed through the coarse data processing, data standardization, and preliminary feature extraction to obtain the current dataset to be fused. The historical classification prediction dataset is preprocessed through the coarse data processing, data standardization, preliminary feature extraction, and dataset segmentation to obtain the fusion training set and the fusion test set.
[0017] Furthermore, in the multimodal interaction module, a model is used to construct the model required for fusing multimodal features and to process the dataset using the model. Specifically, a multimodal fusion model is designed, and the current dataset to be fused, the fusion training set, and the fusion test set are used as inputs to the multimodal fusion model to obtain the current dataset to be predicted, the prediction training set, and the prediction test set. The multimodal fusion model obtains modal interaction fusion features by extracting immune cell morphological anchor features and molecular functional anchor features and constructing an association matrix, as well as a cross-modal message passing mechanism.
[0018] The multimodal interaction module specifically includes dual anchor set construction, feature decoupling and purification, feature alignment mapping, cross-modal message passing, and model construction and training.
[0019] The construction of the dual anchor set, used to establish a modality alignment benchmark based on biological priors, includes the following steps:
[0020] Immune cell morphology anchor feature extraction is used to extract immune cell morphological features from pathological image data. Specifically, it involves using a pathological feature extraction network guided by predefined pathological criteria to process the pathological image data and obtain an immune cell morphology anchor feature set.
[0021] Molecular anchor feature extraction is used to extract function-related molecular features from genomic data. Specifically, it guides a molecular feature extraction network through predefined molecular biology functional criteria to process genomic data and obtain a molecular functional anchor feature set.
[0022] The dual anchor correlation matrix is constructed to establish the biological association between immune cell morphology anchors and molecular function anchors. Specifically, the correlation between anchors is determined by statistical correlation analysis and Fisher's exact test to obtain a binary dual anchor correlation matrix.
[0023] The feature decoupling and purification process is used to remove intramodal redundancy and noise, and extract core features. The steps include:
[0024] Pathological image feature decoupling is used to separate structural and noise information in pathological image features. Specifically, the pathological image features are encoded into structure-related latent variables by a variational autoencoder and reconstructed by a decoder to obtain the structural and noise features of the pathological image.
[0025] Genomic feature decoupling is used to separate pathway features and redundant features in genomic data. Specifically, it decomposes genomic features into pathway feature matrices and coefficient matrices through nonnegative matrix factorization to obtain genomic pathway features and genomic redundancy features.
[0026] Feature purification mapping is used to reduce the dimensionality and enhance the decoupled features. Specifically, it maps the structural features of pathological images and genomic pathway features through fully connected layers and nonlinear activation functions to obtain purified features of pathological images and genomics.
[0027] The feature alignment mapping is used to map features of different modalities and granularities to the same dimensional space. Specifically, it maps pathological image purification features and genome purification features to a unified semantic space through linear transformation combined with projection of anchor feature space, thereby obtaining pathological image alignment features and genome alignment features.
[0028] The cross-modal message passing is used to achieve deep interaction between multimodal features, and the steps include:
[0029] Attention weight calculation is used to calculate the interaction weights between features of different modalities. Specifically, it is obtained by calculating the similarity matrix of aligned features and combining it with the expanded dual anchor association matrix, and then processing it with the softmax activation function.
[0030] Bidirectional message passing is used to realize feature interaction between modalities. Specifically, it achieves feature interaction by multiplying the attention weight with the alignment feature and adding it to the original alignment feature, thereby obtaining pathological image interaction features and genome interaction features.
[0031] Iterative deepening of interaction is used to enhance the interaction effect between modalities. Specifically, it involves bidirectional message passing through multiple rounds of iterations, with the message passing intensity gradually decreasing with the number of iterations, to obtain deep interactive pathological image features and deep interactive genomic features. The message passing intensity gradually decreasing with the number of iterations is the ratio of the initial message passing intensity to the number of iterations.
[0032] Unified representation aggregation is used to aggregate the modal features of deep interaction into sample-level representations. Specifically, it processes the deep interaction features by applying global max pooling and global average pooling respectively and adds them together to obtain the final features. Then, it concatenates the final features of each modality and the element-wise product of the final features to obtain the modal interaction fusion features.
[0033] The construction and training of the model specifically involves constructing a multimodal fusion model by integrating the construction of the dual anchor set, the feature decoupling and purification, the feature alignment mapping, and the cross-modal message passing; training the model and verifying its performance based on the fusion training set and the fusion test set to obtain the multimodal fusion model.
[0034] The multimodal fusion model is used to perform multimodal interaction with the current dataset to be fused, the fusion training set, and the fusion test set as inputs to obtain the current dataset to be predicted, the prediction training set, and the prediction test set.
[0035] Furthermore, in the typing prediction model construction module, a model is used to construct the model required for predicting tumor-infiltrating immune cell typing based on modal interaction fusion features. Specifically, a multi-scale deep network model is constructed as the typing prediction model. The multi-scale deep network model outputs immune cell typing labels through multi-modal component attention interpretation combined with hierarchical feature enhancement.
[0036] The classification prediction model construction module specifically includes multimodal component interpretation, hierarchical feature enhancement, model output acquisition, and model construction and training.
[0037] The multimodal component interpretation is used to weight the importance of different components of the modal interaction fusion features, and the steps include:
[0038] Anchor-guided component weighting is used to determine the importance of different components based on prior biological knowledge. Specifically, component priority weights are obtained by aggregating and normalizing the prognostic values of immune cell morphology anchors and molecular function anchors.
[0039] The interpretation attention weight calculation is used to adjust the weights of modal interaction fusion feature components in combination with feature content. Specifically, it learns the dynamic weight of each component through a neural network and combines it with the component priority weight to obtain the interpretation attention weight.
[0040] Weighted feature interpretation is used to generate feature representations that highlight important information. Specifically, it involves multiplying the corresponding components of the interpretation attention and modal interaction fusion features, and then concatenating the weighted components to obtain the interpretation enhancement features.
[0041] The hierarchical feature enhancement is used to extract multi-level feature representations from the interpreted enhancement features. Specifically, it involves performing convolution and nonlinear transformation on the interpreted enhancement features through a multi-layer convolutional neural network to obtain local immune cell typing features. The interpreted enhancement features are then processed by a Transformer encoder to obtain global immune cell typing features. Finally, the local immune cell typing features, the global immune cell typing features, and the element-wise product of the two are concatenated to obtain multi-scale immune cell typing features.
[0042] The model output acquisition is used to obtain the model's prediction results. Specifically, it uses clinicopathological features as conditional embedding, performs feature mapping on multi-scale immune cell typing features using a fully connected layer, converts the mapping results into a probability distribution using a softmax activation function, and determines the corresponding immune cell typing label based on this probability distribution.
[0043] The construction and training of the model specifically involves integrating the multimodal component interpretation, the hierarchical feature enhancement, and the model output acquisition to construct a multi-scale deep network model. The model is then trained and its performance is verified based on the prediction training set and the prediction test set to obtain the multi-scale deep network model, which serves as the genotyping prediction model.
[0044] Furthermore, in the immune cell typing prediction module, the current dataset to be predicted is used as the input of the typing prediction model to perform tumor infiltrating immune cell typing prediction. The obtained immune cell typing labels are used as immune cell typing prediction results, and the patient's tumor immune microenvironment status is comprehensively evaluated based on the immune cell typing prediction results.
[0045] The beneficial effects achieved by the present invention using the above solution are as follows:
[0046] (1) In view of the technical problem that traditional tumor infiltrating immune cell typing systems rely on single-modality data or simple early fusion strategies and lack in-depth modeling of complex interactions between multimodal data, resulting in morphological information and molecular data being isolated and difficult to capture the multidimensional nature of immune cell functional state, this solution creatively adopts a multimodal fusion model to integrate multimodal features, which is consistent with the nature of the tumor immune microenvironment being driven by multiple factors such as morphology, molecular mechanisms and clinical factors. Under the guidance of biological prior knowledge, it can deeply align and interact data of different modalities, thereby revealing the intrinsic relationship between immune cell morphological features and molecular functions and improving the complementarity of feature representation.
[0047] (2) In view of the technical problems that traditional tumor infiltrating immune cell typing systems have, they are unable to cope with the inherent multi-scale complexity of the tumor microenvironment, cannot coordinate the relationship between abnormal local immune cell distribution and global immune response trends, may miss key immune escape driving factors, and have a single feature representation level and insufficient modeling of the spatial heterogeneity of immune cells, thus limiting the stability of typing performance, this solution creatively adopts a multi-scale deep network model as the typing prediction model. By simultaneously capturing subtle immune cell morphological patterns and the overall biological context, it can more comprehensively analyze the multi-scale features of immune cell typing, integrate local detail perception and global relationship modeling, enhance the model's ability to identify complex immune cell subpopulations, and improve the accuracy and generalization of typing. Attached Figure Description
[0048] Figure 1 A schematic diagram of the modules of an intelligent tumor-infiltrating immune cell precise typing system provided by the present invention;
[0049] Figure 2 This is a flowchart illustrating the data preprocessing module.
[0050] Figure 3 This is a flowchart illustrating the multimodal interaction module.
[0051] Figure 4 A flowchart illustrating the process of building a module for the fractal prediction model.
[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0053] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0054] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0055] Example 1, see Figure 1 The present invention provides an intelligent tumor-infiltrating immune cell precise typing system, including a pathological data acquisition module, a data preprocessing module, a multimodal interaction module, a typing prediction model construction module, and an immune cell typing prediction module;
[0056] The pathological data acquisition module acquires data to obtain a raw dataset for classification prediction and sends the raw dataset for classification prediction to the data preprocessing module.
[0057] The data preprocessing module employs data preprocessing methods such as coarse data processing, data standardization, preliminary feature extraction, and dataset segmentation to obtain the current dataset to be fused, the fusion training set, and the fusion test set, and then sends the current dataset to be fused, the fusion training set, and the fusion test set to the multimodal interaction module.
[0058] The multimodal interaction module is used to fuse multimodal features, and by designing a multimodal fusion model, obtain the current dataset to be predicted, the prediction training set, and the prediction test set, and send the current dataset to be predicted to the immune cell typing prediction module, and send the prediction training set and the prediction test set to the typing prediction model construction module.
[0059] The typing prediction model construction module is used to construct a model required for predicting the typing of tumor-infiltrating immune cells. It constructs a multi-scale deep network model as the typing prediction model and sends the typing prediction model to the immune cell typing prediction module.
[0060] The immune cell typing prediction module specifically performs real-time tumor infiltrating immune cell typing prediction based on the typing prediction model to obtain immune cell typing prediction results.
[0061] Example 2, see Figure 1 This embodiment is based on the above embodiment. In the pathological data acquisition module, the original dataset for classification prediction specifically includes the original dataset for historical classification prediction and the original dataset for current classification prediction. Both the original dataset for historical classification prediction and the original dataset for current classification prediction include pathological image data, genomic data and clinical pathological data. The original dataset for historical classification prediction also includes tumor infiltrating immune cell classification annotation data.
[0062] The pathological image data is used to capture the morphological characteristics of immune cells in the tumor microenvironment, specifically patient tumor pathological image data covering different time points obtained through high-resolution whole-slice scanning. The genomic data is used to characterize the functional state of immune cells at the microscopic level, specifically including gene expression profile data, gene mutation data of cells obtained through target region sequencing, epigenetic data, and pathway activity score data. The clinicopathological data is used to record individual patient and clinical progress characteristics, specifically including patient basic information data, tumor characteristic data, and patient treatment data. The patient basic information data specifically includes patient age data, patient gender data, and patient family medical history data. The tumor characteristic data specifically includes tumor stage data, tumor histological type data, and tumor differentiation degree data. The patient treatment data specifically includes patient chemotherapy and radiotherapy data, patient targeted therapy drug use data, and patient tumor-related surgical data.
[0063] Example 3, see Figure 1 and Figure 2This embodiment is based on the above embodiment. In the data preprocessing module, the coarse data processing is used to ensure the reliability and consistency of the input data. Specifically, it removes obvious abnormal data and corrects technical deviations. Pathological image data uses a context-aware generative model to fill missing regions. Genomic data uses k-nearest neighbor imputation combined with pathway information to fill missing values. Clinical pathological data uses an iterative random forest algorithm to fill missing values.
[0064] The data standardization is used to eliminate technical batch effects and dimensional differences, specifically by processing different modal data separately through a distribution-aware standardization strategy.
[0065] The preliminary feature extraction is used to extract meaningful feature representations from the raw data. Specifically, the pathological image data is segmented into 256×256 pixel image blocks and features are extracted using EfficientNet. Genomic data is extracted using non-negative matrix factorization, and clinical pathological data is converted into structured feature vectors.
[0066] The dataset segmentation is used to obtain training data and test data, specifically by segmenting the original dataset of historical classification prediction.
[0067] The current classification prediction dataset is preprocessed through the coarse data processing, data standardization, and preliminary feature extraction to obtain the current dataset to be fused. The historical classification prediction dataset is preprocessed through the coarse data processing, data standardization, preliminary feature extraction, and dataset segmentation to obtain the fusion training set and the fusion test set.
[0068] Example 4, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In the multimodal interaction module, a model is used to construct the model required to fuse multimodal features and to process the dataset using the model. Specifically, a multimodal fusion model is designed, and the current dataset to be fused, the fusion training set, and the fusion test set are used as inputs to the multimodal fusion model to obtain the current dataset to be predicted, the prediction training set, and the prediction test set. The multimodal fusion model obtains modal interaction fusion features by extracting immune cell morphological anchor features and molecular functional anchor features and constructing an association matrix, as well as a cross-modal message passing mechanism.
[0069] The multimodal interaction module specifically includes dual anchor set construction, feature decoupling and purification, feature alignment mapping, cross-modal message passing, and model construction and training.
[0070] The construction of the dual anchor set, used to establish a modality alignment benchmark based on biological priors, includes the following steps:
[0071] Immune cell morphology anchor feature extraction is used to extract morphological features of immune cells from pathological image data. Specifically, it involves processing the pathological image data through a pathological feature extraction network guided by predefined pathological criteria to obtain a set of immune cell morphology anchor features. The formula used is as follows:
[0072] ;
[0073] In the formula, This represents the characteristics of the morphological anchor point of the a-th immune cell. This represents a network function for extracting pathological features. This represents the image block-level feature matrix of the input pathological image data. The definition of the morphological anchor point of the a-th immune cell includes, but is not limited to, immune cell density, spatial distribution pattern of immune cells, and spatial relationship between immune cells and tumor cells. This represents the number of image patches associated with the morphological anchor point of the a-th immune cell. This represents a convolutional neural network function used for extracting morphological anchor features of immune cells. This represents the b-th image patch feature associated with the a-th immune cell morphological anchor point;
[0074] Molecular anchor feature extraction is used to extract function-related molecular features from genomic data. Specifically, it guides a molecular feature extraction network through predefined molecular biology functional criteria to process genomic data and obtain a set of molecular functional anchor features. The formula used is as follows:
[0075] ;
[0076] In the formula, This represents the characteristics of the a-th molecule's functional anchor point. This represents the molecular feature extraction network function, specifically a single-layer perceptron. This indicates the input of genomic data. The definition of the a-th molecular functional anchor includes, but is not limited to, the activity of immune-related pathways and the expression status of immune checkpoint molecules that correspond one-to-one with immune cell morphological anchors. This represents the linear transformation weight of the a-th molecular functional anchor point. This represents the linear transformation bias term for the a-th molecular functional anchor point. This represents a subset of genomic features associated with the a-th molecular functional anchor.
[0077] The dual anchor correlation matrix was constructed to establish the biological association between immune cell morphological anchors and molecular functional anchors. Specifically, the correlation between anchors was determined through statistical correlation analysis and Fisher's exact test, resulting in a binary dual anchor correlation matrix. The formula used is as follows:
[0078] ;
[0079] In the formula, The elements represent the dual anchor incidence matrix. Indicates an indicator function, This represents a function for calculating statistical correlation. This represents the characteristics of the c-th molecule's functional anchor point. This indicates a preset correlation threshold. This represents Fisher's exact test function. This indicates the preset Fisher exact test p-value threshold;
[0080] The feature decoupling and purification process is used to remove intramodal redundancy and noise, and extract core features. The steps include:
[0081] Pathological image feature decoupling is used to separate structural and noise information in pathological image features. Specifically, the pathological image features are encoded into structure-related latent variables by a variational autoencoder and reconstructed by a decoder to obtain the pathological image structural features and pathological image noise features.
[0082] Genomic feature decoupling is used to separate pathway features and redundant features in genomic data. Specifically, it decomposes genomic features into pathway feature matrices and coefficient matrices through nonnegative matrix factorization to obtain genomic pathway features and genomic redundancy features.
[0083] Feature purification mapping is used to reduce the dimensionality and enhance the decoupled features. Specifically, it maps the structural features of pathological images and genomic pathway features through fully connected layers and nonlinear activation functions to obtain purified features of pathological images and genomics. The formulas used are as follows:
[0084] ;
[0085] In the formula, Indicates the purification features of pathological images. Indicates genome purification characteristics, Represents the ReLU activation function. This represents the mapping weights for pathological image purification. Indicates the structural features of pathological images, This represents the pathological image purification mapping bias term. Represents the mapping weights for genome purification. Indicates genomic pathway characteristics, This represents the genome purification mapping bias term;
[0086] The feature alignment mapping is used to map features of different modalities and granularities to the same dimensional space. Specifically, it maps pathological image purification features and genome purification features to a unified semantic space through linear transformation combined with projection of the anchor feature space, resulting in pathological image alignment features and genome alignment features. The formula used is as follows:
[0087] ;
[0088] In the formula, This indicates alignment features of pathological images. Indicates genome alignment features, The weights represent the unified semantic space mapping of pathological images. Represents the weights of the genome unified semantic space mapping. This represents the bias term for the unified semantic space mapping of pathological images. This represents the bias term for the genome unified semantic space mapping. Indicates the pathological anchor point guiding weight. Indicates the molecular anchor guiding weight. This represents the immune cell morphology anchor matrix obtained based on the immune cell morphology anchor feature set. This represents the molecular functional anchor matrix obtained based on the molecular functional anchor feature set, where T represents the transpose operation;
[0089] The cross-modal message passing is used to achieve deep interaction between multimodal features, and the steps include:
[0090] Attention weight calculation is used to calculate the interaction weights between features of different modalities. Specifically, it involves calculating the similarity matrix of aligned features and combining it with the expanded dual anchor association matrix. After processing with the softmax activation function, the attention weight matrix is obtained. The formula used is as follows:
[0091] ;
[0092] In the formula, This represents the attention weight of pathological images to the genome. This represents the attention weight of the genome to pathological images. This represents the softmax activation function, and dim represents the dimension of the aligned feature. This represents the expanded dual anchor incidence matrix. This represents element-wise multiplication.
[0093] Bidirectional message passing is used to achieve feature interaction between modalities. Specifically, it achieves feature interaction by multiplying the attention weights with the alignment features and adding them to the original alignment features, thus obtaining pathological image interaction features and genome interaction features. The formulas used are as follows:
[0094] ;
[0095] In the formula, Representing interactive features of pathological images, Indicates genomic interaction characteristics, Indicates message passing strength. This function converts a vector into a diagonal matrix. Specifically, it maps each element of the input vector to the corresponding element on the main diagonal of the diagonal matrix, generating a diagonal matrix with the vector elements as the main diagonal elements. AO represents a vector with all elements equal to 1.
[0096] Iterative deepening of interaction is used to enhance the interaction effect between modalities. Specifically, it involves bidirectional message passing through multiple rounds of iterations, with the message passing intensity gradually decreasing with the number of iterations, to obtain deep interactive pathological image features and deep interactive genomic features. The message passing intensity gradually decreasing with the number of iterations is the ratio of the initial message passing intensity to the number of iterations.
[0097] Unified representation aggregation is used to aggregate the modal features of deep interactions into sample-level representations. Specifically, it processes the deep interaction features using global max pooling and global average pooling respectively, and then adds them together to obtain the final features. Subsequently, it concatenates the final features of each modality and the element-wise product of the final features to obtain the modal interaction fusion features. The formula used is as follows:
[0098] ;
[0099] In the formula, Indicates the final features of a pathological image. Indicates the final characteristics of the genome, Represents the global max pooling function. This represents the global average pooling function. Representing deep interactive pathological image features, Indicating deep interactive genomic features, This indicates modal interaction fusion features;
[0100] The construction and training of the model specifically involves constructing a multimodal fusion model by integrating the construction of the dual anchor set, the feature decoupling and purification, the feature alignment mapping, and the cross-modal message passing; training the model and verifying its performance based on the fusion training set and the fusion test set to obtain the multimodal fusion model.
[0101] The multimodal fusion model is used to perform multimodal interaction with the current dataset to be fused, the fusion training set, and the fusion test set as inputs to obtain the current dataset to be predicted, the prediction training set, and the prediction test set.
[0102] By performing the above operations, this solution addresses the technical problem of traditional tumor-infiltrating immune cell typing systems relying on single-modality data or simple early fusion strategies, lacking in-depth modeling of the complex interactions between multimodal data, resulting in isolated morphological and molecular data that are difficult to capture the multidimensional nature of immune cell functional states. This solution creatively adopts a multimodal fusion model to integrate multimodal features, which aligns with the nature of the tumor immune microenvironment being driven by multiple factors including morphology, molecular mechanisms, and clinical factors. Guided by prior biological knowledge, it can deeply align and interact data from different modalities, thereby revealing the intrinsic relationship between immune cell morphological features and molecular functions and enhancing the complementarity of feature representation.
[0103] Example 5, see Figure 1 and Figure 4 This embodiment is based on the above embodiment. In the typing prediction model construction module, a model is used to construct the model required for tumor infiltrating immune cell typing prediction based on modal interaction fusion features. Specifically, a multi-scale deep network model is constructed as the typing prediction model. The multi-scale deep network model outputs immune cell typing labels through multi-modal component attention interpretation combined with hierarchical feature enhancement.
[0104] The classification prediction model construction module specifically includes multimodal component interpretation, hierarchical feature enhancement, model output acquisition, and model construction and training.
[0105] The multimodal component interpretation is used to weight the importance of different components of the modal interaction fusion features, and the steps include:
[0106] Anchor-guided component weighting is used to determine the importance of different components based on prior biological knowledge. Specifically, it obtains component priority weights by aggregating and normalizing the prognostic values of immune cell morphology anchors and molecular functional anchors, using the following formula:
[0107] ;
[0108] In the formula, Indicates the priority weight of pathological images. Indicates genome priority weight, This indicates the joint priority weight, and Num represents the total number of immune cell morphological anchors. This represents the sigmoid activation function. This represents the prognostic value of the a-th immune cell morphology anchor point obtained based on prior knowledge. This represents the prognostic value of the a-th molecular functional anchor point obtained based on prior knowledge.
[0109] The interpretation attention weight calculation is used to adjust the weights of modal interaction fusion feature components in combination with feature content. Specifically, it learns the dynamic weight of each component through a neural network and combines it with the component priority weight to obtain the interpretation attention weight. The formula used is as follows:
[0110] ;
[0111] In the formula, This represents the attention weights for interpreting pathological images. Indicates attention weights in genome interpretation. This represents the joint interpretation attention weights, and le represents the learnable vector. This indicates the interpretation of attention map weights. This indicates the interpretation of the attention mapping bias term;
[0112] Weighted feature interpretation is used to generate feature representations that highlight important information. Specifically, it involves multiplying the corresponding components of the interpretation attention and modal interaction fusion features, and then concatenating the weighted components to obtain the interpretation enhancement features. The formula used is as follows:
[0113] ;
[0114] In the formula, This indicates enhanced interpretation features;
[0115] The hierarchical feature enhancement is used to extract multi-level feature representations from the interpreted enhancement features. Specifically, it involves performing convolution and nonlinear transformation on the interpreted enhancement features through a multi-layer convolutional neural network to obtain local immune cell typing features. The interpreted enhancement features are then processed by a Transformer encoder to obtain global immune cell typing features. Finally, the local immune cell typing features, the global immune cell typing features, and the element-wise product of the two are concatenated to obtain multi-scale immune cell typing features.
[0116] The model output acquisition is used to obtain the model's prediction results. Specifically, it uses clinicopathological features as conditional embedding, performs feature mapping on multi-scale immune cell typing features using a fully connected layer, converts the mapping results into a probability distribution using a softmax activation function, and determines the corresponding immune cell typing label based on this probability distribution.
[0117] The construction and training of the model specifically involves integrating the multimodal component interpretation, the hierarchical feature enhancement, and the model output acquisition to construct a multi-scale deep network model. The model is then trained and its performance is verified based on the prediction training set and the prediction test set to obtain the multi-scale deep network model, which serves as the genotyping prediction model.
[0118] By performing the above operations, this solution addresses the technical problems of traditional tumor-infiltrating immune cell typing systems, which struggle to cope with the inherent multi-scale complexity of the tumor microenvironment, cannot coordinate the correlation between abnormal local immune cell distribution and global immune response trends, may miss key immune escape drivers, and suffer from limited stability in typing performance due to their single feature representation level and insufficient modeling of immune cell spatial heterogeneity. This solution creatively employs a multi-scale deep network model as the typing prediction model. By simultaneously capturing subtle immune cell morphological patterns and the overall biological context, it can more comprehensively analyze the multi-scale features of immune cell typing, integrate local detail perception with global relationship modeling, enhance the model's ability to identify complex immune cell subpopulations, and improve the accuracy and generalization of typing.
[0119] Example 6, see Figure 1 This embodiment is based on the above embodiment. In the immune cell typing prediction module, the current dataset to be predicted is used as the input of the typing prediction model to perform tumor infiltrating immune cell typing prediction. The obtained immune cell typing labels are used as immune cell typing prediction results. The patient's tumor immune microenvironment status is comprehensively evaluated based on the immune cell typing prediction results.
[0120] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0121] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.
[0122] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. An intelligent system for precise typing of tumor-infiltrating immune cells, characterized in that: The system includes a pathological data acquisition module, a data preprocessing module, a multimodal interaction module, a typing prediction model construction module, and an immune cell typing prediction module; The pathological data acquisition module acquires data to obtain a classification prediction raw dataset, which specifically includes a historical classification prediction raw dataset and a current classification prediction raw dataset. The data preprocessing module employs data preprocessing methods such as coarse data processing, data standardization, preliminary feature extraction, and dataset segmentation to obtain the current dataset to be fused, the fusion training set, and the fusion test set. The multimodal interaction module is used to construct the model required for fusing multimodal features and to process the dataset using the model. Specifically, it designs a multimodal fusion model and takes the current dataset to be fused, the fusion training set, and the fusion test set as inputs to the multimodal fusion model to obtain the current dataset to be predicted, the prediction training set, and the prediction test set. The multimodal fusion model extracts immune cell morphological anchor features and molecular functional anchor features and constructs an association matrix, as well as a cross-modal message passing mechanism, to obtain modal interaction fusion features. The typing prediction model construction module is used to construct the model required for tumor infiltrating immune cell typing prediction based on modal interaction fusion features. Specifically, it constructs a multi-scale deep network model as the typing prediction model. The multi-scale deep network model outputs immune cell typing labels through multi-modal component attention interpretation combined with hierarchical feature enhancement. The immune cell typing prediction module takes the current dataset to be predicted as the input of the typing prediction model, performs tumor infiltrating immune cell typing prediction, and uses the obtained immune cell typing labels as the immune cell typing prediction results.
2. The intelligent tumor-infiltrating immune cell precise typing system according to claim 1, characterized in that: The multimodal interaction module specifically includes dual anchor set construction, feature decoupling and purification, feature alignment mapping, cross-modal message passing, and model construction and training.
3. The intelligent tumor-infiltrating immune cell precise typing system according to claim 2, characterized in that: The construction of the dual anchor set, used to establish a modality alignment benchmark based on biological priors, includes the following steps: Immune cell morphology anchor feature extraction is used to extract immune cell morphological features from pathological image data. Specifically, it involves using a pathological feature extraction network guided by predefined pathological criteria to process the pathological image data and obtain an immune cell morphology anchor feature set. Molecular anchor feature extraction is used to extract function-related molecular features from genomic data. Specifically, it guides a molecular feature extraction network through predefined molecular biology functional criteria to process genomic data and obtain a molecular functional anchor feature set. The dual anchor correlation matrix is constructed to establish the biological association between immune cell morphology anchors and molecular function anchors. Specifically, the correlation between anchors is determined by statistical correlation analysis and Fisher's exact test to obtain a binary dual anchor correlation matrix. The feature decoupling and purification process is used to remove intramodal redundancy and noise, and extract core features. The steps include: Pathological image feature decoupling is used to separate structural and noise information in pathological image features. Specifically, the pathological image features are encoded into structure-related latent variables by a variational autoencoder and reconstructed by a decoder to obtain the structural and noise features of the pathological image. Genomic feature decoupling is used to separate pathway features and redundant features in genomic data. Specifically, it decomposes genomic features into pathway feature matrices and coefficient matrices through nonnegative matrix factorization to obtain genomic pathway features and genomic redundancy features. Feature purification mapping is used to reduce the dimensionality and enhance the decoupled features. Specifically, it maps the structural features of pathological images and genomic pathway features through fully connected layers and nonlinear activation functions to obtain purified features of pathological images and genomics. The feature alignment mapping is used to map features of different modalities and granularities to the same dimensional space. Specifically, it maps pathological image purification features and genome purification features to a unified semantic space through linear transformation combined with projection of anchor feature space, thereby obtaining pathological image alignment features and genome alignment features. The cross-modal message passing is used to achieve deep interaction between multimodal features, and the steps include: Attention weight calculation is used to calculate the interaction weights between features of different modalities. Specifically, it is obtained by calculating the similarity matrix of aligned features and combining it with the expanded dual anchor association matrix, and then processing it with the softmax activation function. Bidirectional message passing is used to realize feature interaction between modalities. Specifically, it achieves feature interaction by multiplying the attention weight with the alignment feature and adding it to the original alignment feature, thereby obtaining pathological image interaction features and genome interaction features. Iterative deepening of interaction is used to enhance the interaction effect between modalities. Specifically, it involves bidirectional message passing through multiple rounds of iterations, with the message passing intensity gradually decreasing with the number of iterations, to obtain deep interactive pathological image features and deep interactive genomic features. The message passing intensity gradually decreasing with the number of iterations is the ratio of the initial message passing intensity to the number of iterations. Unified representation aggregation is used to aggregate the modal features of deep interaction into sample-level representations. Specifically, it processes the deep interaction features by applying global max pooling and global average pooling respectively and adds them together to obtain the final features. Then, it concatenates the final features of each modality and the element-wise product of the final features to obtain the modal interaction fusion features. The construction and training of the model specifically involves constructing a multimodal fusion model by integrating the construction of the dual anchor set, the feature decoupling and purification, the feature alignment mapping, and the cross-modal message passing; training the model and verifying its performance based on the fusion training set and the fusion test set to obtain the multimodal fusion model. The multimodal fusion model is used to perform multimodal interaction with the current dataset to be fused, the fusion training set, and the fusion test set as inputs to obtain the current dataset to be predicted, the prediction training set, and the prediction test set.
4. The intelligent tumor-infiltrating immune cell precise typing system according to claim 1, characterized in that: The classification prediction model construction module specifically includes multimodal component interpretation, hierarchical feature enhancement, model output acquisition, and model construction and training.
5. The intelligent tumor-infiltrating immune cell precise typing system according to claim 4, characterized in that: The multimodal component interpretation is used to weight the importance of different components of the modal interaction fusion features, and the steps include: Anchor-guided component weighting is used to determine the importance of different components based on prior biological knowledge. Specifically, component priority weights are obtained by aggregating and normalizing the prognostic values of immune cell morphology anchors and molecular function anchors. The interpretation attention weight calculation is used to adjust the weights of modal interaction fusion feature components in combination with feature content. Specifically, it learns the dynamic weight of each component through a neural network and combines it with the component priority weight to obtain the interpretation attention weight. Weighted feature interpretation is used to generate feature representations that highlight important information. Specifically, it involves multiplying the corresponding components of the interpretation attention and modal interaction fusion features, and then concatenating the weighted components to obtain the interpretation enhancement features. The hierarchical feature enhancement is used to extract multi-level feature representations from the interpreted enhancement features. Specifically, it involves performing convolution and nonlinear transformation on the interpreted enhancement features through a multi-layer convolutional neural network to obtain local immune cell typing features. The interpreted enhancement features are then processed by a Transformer encoder to obtain global immune cell typing features. Finally, the local immune cell typing features, the global immune cell typing features, and the element-wise product of the two are concatenated to obtain multi-scale immune cell typing features. The model output acquisition is used to obtain the model's prediction results. Specifically, it uses clinicopathological features as conditional embedding, performs feature mapping on multi-scale immune cell typing features using a fully connected layer, converts the mapping results into a probability distribution using a softmax activation function, and determines the corresponding immune cell typing label based on this probability distribution. The construction and training of the model specifically involves integrating the multimodal component interpretation, the hierarchical feature enhancement, and the model output acquisition to construct a multi-scale deep network model. The model is then trained and its performance is verified based on the prediction training set and the prediction test set to obtain the multi-scale deep network model, which serves as the genotyping prediction model.
6. The intelligent tumor-infiltrating immune cell precise typing system according to claim 1, characterized in that: Both the historical classification prediction dataset and the current classification prediction dataset include pathological image data, genomic data, and clinicopathological data. The historical classification prediction dataset also includes tumor infiltrating immune cell classification annotation data.
7. The intelligent tumor-infiltrating immune cell precise typing system according to claim 1, characterized in that: In the data preprocessing module, the coarse data processing is used to ensure the reliability and consistency of the input data. Specifically, it removes obvious abnormal data and corrects technical deviations. Pathological image data uses a context-aware generative model to fill missing regions. Genomic data uses k-nearest neighbor imputation combined with pathway information to fill missing values. Clinical pathological data uses an iterative random forest algorithm to fill missing values. The data standardization is used to eliminate technical batch effects and dimensional differences, specifically by processing different modal data separately through a distribution-aware standardization strategy. The preliminary feature extraction is used to extract meaningful feature representations from the raw data. Specifically, the pathological image data is segmented into 256×256 pixel image blocks and features are extracted using EfficientNet. Genomic data is extracted using non-negative matrix factorization, and clinical pathological data is converted into structured feature vectors. The dataset segmentation is used to obtain training data and test data, specifically by segmenting the original dataset of historical classification prediction. The current classification prediction dataset is preprocessed through the coarse data processing, data standardization, and preliminary feature extraction to obtain the current dataset to be fused. The historical classification prediction dataset is preprocessed through the coarse data processing, data standardization, preliminary feature extraction, and dataset segmentation to obtain the fusion training set and the fusion test set.