A multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling
By constructing a multimodal site joint characterization and introducing local microenvironmental difference enhancement and adaptive spatial bias modeling, the problems of insufficient local spatial information fusion and insufficient adaptive adjustment ability in existing methods are solved, and efficient characterization and classification of the complex heterogeneity inside tumor tissues are achieved.
Patent Information
- Application Number
- CN202611561172.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-10
- Publication Date
- 2026-11-17
AI Technical Summary
Existing spatial transcriptomics and pathological image multimodal analysis methods tend to lead to feature convergence when fusing local spatial information, making it difficult to characterize the complex local spatial heterogeneity within tumor tissues. Furthermore, they lack adaptive adjustment capabilities to adapt to the spatial distribution characteristics of different tissue samples, making it difficult to achieve collaborative modeling of multimodal features.
By constructing a multimodal site joint representation, introducing a local microenvironment difference enhancement mechanism and adaptive spatial bias modeling, and using a learnable distance adjustment parameter to adaptively adjust spatial distance constraints, the site representation after microenvironment enhancement is generated by fusing local microenvironment information and overall spatial structure information.
It improves the recognition ability and generalization performance of the multimodal tumor classification model, can adaptively characterize the heterogeneity of the local microenvironment inside the tumor tissue, adapt to the spatial structural differences of different tissue samples, and enhance the ability to classify and identify tumor tissues.
Smart Images

Figure CN123115876A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of bioinformatics computing, medical image analysis, and spatial multi-omics data analysis, specifically to a multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling. Background Technology
[0002] With the development of spatial multi-omics technologies, spatial transcriptomics can obtain gene expression information corresponding to different spatial sites while preserving the spatial location information of tissues, providing an important data foundation for studying changes in cellular composition, molecular state, and spatial heterogeneity within tissues. Meanwhile, pathological images can reflect cell arrangement, tissue regional distribution, and lesion-related morphological characteristics at the tissue morphology level. Therefore, combining spatial transcriptomics data with pathological image data allows for comprehensive analysis of tissue state at both the molecular expression and tissue morphology levels, providing data support for tumor classification and related intelligent medical analysis.
[0003] Currently, joint analysis methods based on spatial transcriptome data and pathological image data typically extract molecular expression features and visual features corresponding to spatial sites, and then use machine learning or deep learning models to predict tissue state. With the development of related technologies, some studies further consider the positional relationships between spatial sites and the complementarity between different modalities, enhancing the feature expression capabilities of spatial multi-omics data by constructing spatial correlations.
[0004] Existing spatial transcriptomics and pathological image multimodal analysis methods mainly fall into the following categories: some methods encode spatial sites as independent samples and complete classification prediction by fusing features from different modalities; some methods further utilize spatial neighborhood information to aggregate information between adjacent sites to enhance the ability to represent local spatial context; and some methods use graph structure learning or attention mechanisms to model the relationships between spatial sites to improve the ability to utilize tissue spatial information.
[0005] However, existing multimodal analysis methods for spatial transcriptomics and pathological images still have the following shortcomings: First, current local spatial information fusion methods usually rely on the aggregation of neighboring site information, which easily leads to convergence of features among different sites. This causes the specific expression information of the central site to be averaged by its neighbors, making it difficult to fully characterize the complex local spatial heterogeneity within tumor tissue. Second, existing spatial relationship modeling methods lack the ability to adaptively adjust the spatial constraints imposed on the strength of association between sites, making it difficult to adapt to the differentiated spatial distribution characteristics of different tissue samples and limiting the model's ability to represent complex tissue spatial structures. Finally, spatial transcriptomics data and pathological image data reflect tissue molecular expression information and tissue morphology information, respectively. Although existing methods can achieve multimodal feature fusion, there are still shortcomings in how to further combine the fused site features with local microenvironment information and overall spatial structure information for collaborative modeling. Summary of the Invention
[0006] The technical problem to be solved by this invention is to provide a multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling, which can make full use of spatial transcriptomic molecular expression information, pathological image tissue morphology information and spatial location information, adaptively characterize the heterogeneity of local microenvironment inside tumor tissue, and dynamically adapt to the spatial structural differences of different tissue samples, thereby improving the recognition ability and generalization performance of the multimodal tumor classification model.
[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is implemented as follows: A multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling includes the following steps: S1. Construction of multimodal site joint characterization: Obtain spatial transcriptome gene expression data, pathological image visual data and spatial site coordinate information corresponding to tissue samples, and establish multimodal site joint characterization that integrates molecular expression information, tissue morphology information and spatial location information based on the correspondence of spatial sites. S2. Enhancement of Local Microenvironment Differences: For the joint representation of the multimodal sites, a local reference region corresponding to the central site is determined based on spatial location information. The spatial site feature information within the local reference region is correlated and modeled to obtain a local context representation. The difference information between the joint representation of the central site and the local context representation is calculated to obtain difference features used to represent spatial heterogeneity. The joint representation of the central site, the local context representation, and the difference features are fused, and a site representation enhanced with microenvironment is generated through an adaptive weight adjustment mechanism. S3. Adaptive Spatial Bias Modeling: Based on the enhanced site representation of the microenvironment, a spatial bias matrix containing learnable distance adjustment parameters is constructed according to the distance relationship between spatial sites. The spatial distance constraint strength is adaptively adjusted during model training using the learnable distance adjustment parameters. The feature association information between sites is fused with the spatial bias matrix to obtain a comprehensive spatial association weight. Based on the comprehensive spatial association weight, spatial site information is aggregated to obtain global spatial fusion features. S4. Tumor classification prediction: Output the tumor-related classification results corresponding to the tissue samples based on the global spatial fusion features.
[0008] Furthermore, step S2, which enhances the local microenvironment differences, specifically includes the following sub-steps: S201. Construction of local reference region: Calculate the Euclidean distance between the sites based on the two-dimensional coordinate information of each spatial site, and filter the neighboring sites corresponding to the central site according to spatial proximity to form a local reference region. S202. Construction of local context representation: For the features of neighboring sites within the local reference region, the association weight of each neighboring site to the central site is calculated through an attention mechanism. The feature information of the neighboring sites is aggregated according to the weight to obtain the local context representation corresponding to the central site. S203. Differential Feature Extraction: Calculate the vector difference between the multimodal joint representation of the central site and the local context representation to obtain differential features used to represent the differences between the central site and the surrounding microenvironment. S204, Adaptive Fusion Enhancement: The joint representation of the central site, the local context representation, and the differential features are spliced and fused. A dynamic weight in the range of 0 to 1 is generated through an adaptive weight adjustment mechanism to adaptively adjust the contribution ratio of the three types of features and output the site representation after microenvironment enhancement.
[0009] Furthermore, in step S202, the calculation formula for the local context representation is as follows:
[0010]
[0011] in, , , Let i be the query vector, key vector, and value vector of the center site i and its neighboring site j, respectively. , , Let D be the learnable projective weight matrix, and D be the feature dimension scaling factor. Let j be the attention weight of neighboring site j to center site i. The set of effective neighboring sites corresponding to the central site i. This provides a basic characterization of the local microenvironment obtained through aggregation; In step S203, the formula for calculating the difference features is:
[0012] in, For multimodal joint characterization of the central site, These are the distinguishing features.
[0013] Furthermore, in step S204, the adaptive weight adjustment mechanism is implemented through a gating unit, and the calculation formula is as follows:
[0014]
[0015] in, , These are the weight parameters and bias parameters in the learnable weight mapping unit, respectively. It is a sigmoid nonlinear activation function. This represents the dynamically adjusted weights generated by the learnable weight mapping unit. This enhances the local microenvironment features in the final output.
[0016] Furthermore, step S3, adaptive spatial bias modeling, specifically includes the following sub-steps: S301, Feature Correlation Matrix Calculation: Based on the enhanced site representation of the microenvironment, the feature similarity between any two points is calculated through the self-attention mechanism to construct the feature correlation matrix between sites; S302, Adaptive Spatial Bias Generation: Calculate the Euclidean distance based on the relative positions between spatial sites, introduce a learnable distance adjustment parameter to construct a spatial bias matrix, and adaptively update the distance adjustment parameter as the model iterates and trains, dynamically adjusting the degree of attenuation of the spatial distance on the site association strength. S303, Multi-information association fusion: The feature association matrix between sites and the spatial bias matrix are added element by element to obtain a comprehensive association weight matrix of fused molecular expression association information and spatial location constraint information. After normalization, the features of all sites are weighted and aggregated to obtain global spatial fusion features. S304, Residual Normalization Output: The global spatial fusion features are residually connected with the input microenvironment enhancement site representation, and after layer normalization, the final multimodal global feature representation is output.
[0017] Furthermore, in step S302, the formula for calculating the spatial bias matrix is:
[0018] in, Let be the Euclidean spatial distance between site i and site j. This is a learnable distance adjustment parameter used to control the decay rate of spatial distance-to-site association strength. For site with site Gaussian space bias coefficients between them; In step S303, the formula for calculating the comprehensive correlation weight matrix is as follows:
[0019] in, B is the feature correlation matrix between sites, and A is the spatial bias matrix. This is a comprehensive correlation weight matrix.
[0020] Furthermore, step S1 specifically includes: Preprocessing of raw spatial transcriptome gene expression data and pathological image visual features includes missing data filtering, abnormal site removal, and invalid tissue region screening; Based on the spatial site coordinates, a mapping relationship between spatial transcriptome sites and corresponding regions of pathological images is established. Linear projections are performed on the spatial transcriptome molecular features and the pathological image visual features, respectively, mapping the two types of features to a hidden feature space of the same dimension. The mapped gene expression features are spliced and fused with pathological visual features to construct a multimodal site joint representation corresponding to each spatial site, while retaining the corresponding two-dimensional spatial coordinate information.
[0021] Furthermore, in step S304, the formula for calculating the residual normalization is:
[0022] in, V represents the spatial fusion features after global weighted aggregation, where V is the value feature matrix. The input is the microenvironment enhancement site characterization matrix. This is the final multimodal feature representation that integrates local microenvironment information and global spatial structure information.
[0023] Furthermore, the model training phase uses the following configuration: We use the cross-entropy loss function with class weights as the optimization objective, and combine it with a label smoothing strategy to reduce the risk of model overfitting. A gradient pruning strategy is adopted to stabilize the parameter update process, and a cosine annealing learning rate adjustment strategy is adopted to dynamically optimize the learning rate. A five-fold cross-validation method is used to divide the training set and the validation set. In each round of training, the model parameters corresponding to the best performance of the validation set are saved. The model's hidden layer dimension was set to 96, the Dropout rate was set to 0.1, the initial learning rate was set to 1e-4, and the total number of training iterations was set to 60.
[0024] Furthermore, the tumor-related classification results include at least one of the following: tumor disease classification results, tumor benign / malignant binary classification results, tissue / organ origin classification results, tumor subtype classification results, and tissue lesion status identification results; the method is applied to scenarios such as spatial multi-omics tissue sample intelligent classification analysis, tumor auxiliary diagnosis, and pathological analysis.
[0025] Beneficial effects: (1) This invention integrates spatial transcriptome molecular expression information, pathological image visual information and spatial site information to construct a spatial site-level multimodal joint representation, which improves the comprehensive utilization of tissue information from different sources; (2) This invention introduces the difference information between the central site and the local context region through the local microenvironment difference enhancement mechanism, which reduces the problem of the central site-specific molecular expression information being smoothed and weakened in the traditional neighborhood information aggregation process, and improves the model's ability to represent the local spatial heterogeneity of tumor tissue; (3) This invention introduces a spatial bias modeling mechanism with learnable distance adjustment parameters, which enables the spatial distance constraint strength to be dynamically adjusted according to the spatial distribution characteristics of tissue samples, thereby improving the model's adaptability to different tissue structure patterns; (4) This invention combines local microenvironment difference enhancement with global spatial bias modeling, and improves the spatial transcriptome and pathological image multimodal feature expression ability by simultaneously utilizing spatial site local association information and overall tissue spatial structure information; (5) This invention can be applied to the intelligent classification analysis of spatial multi-omics tissue samples, providing an effective calculation method for tumor-related tissue status discrimination and auxiliary analysis. Attached Figure Description
[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the steps of the multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling as described in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the overall algorithm of the multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling as described in this embodiment of the invention. Figure 3 This is a schematic diagram illustrating the principle of local microenvironmental difference enhancement in the multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling as described in an embodiment of the present invention. Figure 4This is a schematic diagram illustrating the principle of adaptive spatial bias modeling in the multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling as described in this embodiment of the invention. Detailed Implementation
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0028] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0029] To overcome the problems of insufficient characterization of local tissue microenvironment heterogeneity, difficulty in adapting spatial relationship constraints to differences in different tissue structures, and limited ability to synergistically utilize local and global spatial information in existing multimodal tumor analysis methods based on spatial transcriptomics and pathological images, this invention proposes a multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling.
[0030] This invention constructs a multimodal site joint representation that integrates spatial transcriptomic molecular expression information, pathological image tissue morphology information, and spatial location information. At the local level, it enhances the expression of spatial site microenvironment heterogeneity by using differential feature extraction and adaptive fusion mechanisms. At the global level, it optimizes spatial correlation by using a spatial bias modeling mechanism that includes learnable distance adjustment parameters. This achieves joint representation of local microenvironment information and overall spatial structure information, and ultimately completes the tumor-related classification and identification of tissue samples.
[0031] Example 1 See Figures 1-2 The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling in this embodiment includes the following steps: S1. Construction of joint characterization of multimodal sites: We acquire spatial transcriptome gene expression information, pathological image visual features, and spatial site coordinate information corresponding to tissue samples. Based on the spatial site correspondence, we associate different modal features and construct a multimodal site joint characterization that integrates molecular expression information, tissue morphology information, and spatial location information. Specifically, spatial transcriptomic data, pathological image visual features, and spatial site coordinate information corresponding to tissue samples are obtained. The spatial transcriptomic data includes gene expression matrices and two-dimensional spatial coordinate information corresponding to each spatial site, while the pathological image visual features are visual representations extracted from the regions corresponding to the spatial sites in the tissue slice images.
[0032] First, the raw spatial transcriptome data and pathological image visual features are preprocessed, including missing data filtering, abnormal site removal, and invalid tissue region screening, in order to reduce the impact of noise information on model training.
[0033] Furthermore, a mapping relationship between spatial transcriptome sites and corresponding regions in pathological images is established based on spatial site coordinates, and gene expression features at corresponding spatial locations are associated with visual features. To eliminate dimensional and semantic differences between different modal features, linear projections are performed on spatial transcriptome molecular features and pathological image visual features, mapping different modal features to a unified hidden feature space.
[0034] In this embodiment, the unified feature dimension is set to 96 dimensions. The mapped gene expression features are fused with pathological visual features to construct a multimodal joint representation corresponding to each spatial site, while retaining the corresponding two-dimensional spatial coordinate information. This ensures that the joint representation simultaneously includes molecular expression information, tissue morphology information, and spatial location information, providing spatial location priors for subsequent local microenvironmental difference enhancement and global spatial bias modeling.
[0035] S2, Enhanced differences in local microenvironments: For the multimodal site joint representation, a local reference region is constructed based on spatial location relationships, and the site information within the local region is correlated and modeled to obtain a local context representation. The difference information between the central site joint representation and the local context representation is calculated to obtain differential features used to describe local tissue heterogeneity. The central site features, local context features, and differential features are fused through an adaptive weight adjustment mechanism to generate a site representation enhanced with microenvironment. In step S2, tumor local microenvironment heterogeneity modeling is achieved through local microenvironment difference enhancement. Please refer to... Figure 3 Schematic diagram. Since a single spatial site feature primarily reflects the molecular expression state and tissue morphology information at the current location, it is difficult to fully describe the local tissue microenvironment features formed in the surrounding area. This invention first constructs a local reference region corresponding to the central site based on spatial location information, and performs correlation modeling on neighboring site information to obtain a local contextual representation reflecting the surrounding tissue state; further, it calculates the difference information between the joint representation of the central site and the local contextual representation to retain the specific expression characteristics of the central site relative to the surrounding region; finally, it fuses the central site features, local contextual features, and difference features through an adaptive weight adjustment mechanism to generate a site representation with enhanced microenvironment.
[0036] The specific implementation method is as follows: S201. Construction of Local Reference Region: Based on the two-dimensional coordinate information of each spatial point, the distance relationship between spatial points is calculated, and the local reference region corresponding to the central point is determined according to spatial proximity. Let the central point be... Its spatial coordinates are Select a set of neighboring sites associated with it based on spatial distance relationships. This provides a spatial scope for subsequent local context information modeling.
[0037] The specific calculation formula is as follows:
[0038] in, , Central sites Neighborhood sites Two-dimensional spatial coordinates, For two points, the Euclidean distance is given. Central site The corresponding set of effective neighboring sites.
[0039] S202. Construction of local context representation: For spatial sites within the local reference region, the importance of different neighboring sites to the central site is learned through feature association-modeling, and neighborhood information is aggregated according to association weights to obtain the local context representation corresponding to the central site.
[0040] The specific calculation formula is as follows:
[0041]
[0042]
[0043] in, , These are the multimodal original features of the central site and its neighboring sites, respectively. , , For learnable projective weight matrix, The scaling factor is the feature dimension. Neighborhood sites For the central site Attention weights This provides a basic characterization of the local microenvironment obtained through polymerization.
[0044] S203. Differential Feature Extraction: To avoid weakening the local feature difference information of the central site during the neighborhood information aggregation process and to improve the model's ability to represent spatially heterogeneous regions, this invention introduces differential branching. By calculating the difference between the original features of the central site and the neighborhood context representation, differential features used to describe the differences between the central site and the surrounding microenvironment are obtained and fused with the original site features and local context features. Feature optimization is then completed through a multilayer perceptron.
[0045] The specific calculation formula is as follows:
[0046] Furthermore, by fusing the joint representation of the central site, the local context representation, and the differential features, we obtain:
[0047] And it is optimized through the feature mapping unit:
[0048] in, These are the differential features between the central site and the local microenvironment, used to characterize the local heterogeneity of spatial sites; It is a fusion representation of original site features, local context features, and differential features; This is the fusion feature optimized by a multilayer perceptron.
[0049] S204. Adaptive Fusion Enhancement: To dynamically adjust the contribution ratios of the three types of information—joint representation of the central locus, local context representation, and differential features—in the final locus representation, this invention introduces a learnable weight mapping unit. A gated network generates dynamically adjustable weights ranging from 0 to 1, achieving adaptive fusion of different feature information. The specific calculation formula is as follows:
[0050]
[0051] in, , These are the weight parameters and bias parameters in the learnable weight mapping unit, respectively. It is a sigmoid nonlinear activation function. This represents the dynamically adjusted weights generated by the learnable weight mapping unit. This enhances the local microenvironment features in the final output.
[0052] Through the above processing, this module can introduce local spatial context information while preserving the characteristic information of the central site itself, and enhance the spatial heterogeneity representation ability by utilizing the difference features between the central site and the surrounding environment, thereby improving the ability to identify the feature differences of different tissue regions, and providing enhanced spatial site representation for subsequent global spatial bias modeling.
[0053] S3, Adaptive Spatial Bias Modeling: Based on the enhanced site characterization of the microenvironment, a spatial bias matrix containing learnable distance adjustment parameters is constructed according to the distance relationship between spatial sites. The spatial distance constraint strength is adaptively adjusted through the learnable distance adjustment parameters to obtain spatial association relationships that adapt to the spatial distribution characteristics of different tissues. The feature association information between sites and the spatial bias information are fused to generate a comprehensive spatial association weight. Based on the comprehensive spatial association weight, the spatial site information is aggregated to obtain a global spatial fusion feature. In step S3, global spatial structure modeling is completed using a spatial attention module enhanced with learnable Gaussian spatial bias. Please refer to... Figure 4 Structural principle. This module receives the local enhancement features output by the microenvironment encoding, fuses gene expression feature similarity and learnable Gaussian spatial distance information, and adaptively models the long-range spatial dependency of the entire slice. The specific implementation method is as follows: S301. Feature Projection and Semantic Association Calculation: Linear projection is performed on the enhanced features of the local micro-environment to generate query, key, and value vectors. The basic attention score, determined solely by differences in gene expression, is calculated through the vector inner product to uncover site-specific semantic relationships. The specific calculation formula is as follows:
[0054]
[0055] in, , , For the learnable projection weights of the global attention module, The scaling factor is the feature dimension. This represents the basic attention score matrix calculated based on the semantic association of site features.
[0056] S302. Construction of Learnable Spatial Bias: Based on the two-dimensional coordinates of spatial sites, the spatial distance between sites is calculated. A learnable distance adjustment parameter is introduced to construct a spatial bias matrix, which is used to characterize the decay law of spatial site association strength as spatial distance changes. The distance adjustment parameter can be adaptively updated during the model's iterative training process, thereby dynamically adjusting the spatial distance constraint strength to adapt the spatial association relationship to the spatial distribution characteristics of different tissue samples.
[0057] This embodiment uses a Gaussian kernel function to construct the spatial bias term, and the specific calculation formula is as follows:
[0058]
[0059] in, This is a learnable distance adjustment parameter used to control the degree of spatial distance attenuation. For site with site The Gaussian space bias coefficients between them.
[0060] S303. Spatial-Semantic Dual Information Fusion: The basic feature association score matrix and the spatial bias matrix are fused, and a comprehensive spatial association weight is obtained through a normalized exponential function. This comprehensive spatial association weight is then used to aggregate features from all spatial locations to obtain global spatial fusion features. The specific calculation formula is as follows:
[0061]
[0062] in, The final attention weight matrix is for fusing dual information. This represents the spatial features after global weighted aggregation.
[0063] S304, Residual Normalized Output: To enhance network training stability, the global spatial fusion features are residually connected to the input micro-environment enhancement site representations, and the final spatially optimized features are output through layer normalization. The specific calculation formula is as follows:
[0064] in, This is the final multimodal feature representation that integrates local microenvironment information and global spatial structure information.
[0065] S4. Tumor Classification Prediction: Based on the global spatial fusion features, the tumor-related classification results corresponding to the tissue samples are output.
[0066] In step S4, for the tissue sample to be tested, the corresponding spatial transcriptome gene expression data, pathological image visual features and spatial site coordinate information are first obtained, and data preprocessing and multimodal site joint characterization construction are completed according to the above steps.
[0067] Subsequently, the processed multimodal spatial data is input into the trained classification model, and local microenvironment difference enhancement, adaptive spatial bias modeling, and global spatial feature aggregation operations are performed in sequence. Finally, the classification prediction layer outputs the tumor category or disease status classification result of the corresponding tissue sample.
[0068] Model training configuration In this embodiment, the hidden layer dimension of the model is uniformly set to 96, the network dropout rate is set to 0.1, the basic initial learning rate is set to 1e-4, and the overall iterative training rounds are set to 60 rounds.
[0069] During the training phase, a cross-entropy loss function with class weights is used to mitigate the impact of differences in the number of samples from different classes on the model optimization process. Simultaneously, a label smoothing strategy is introduced to reduce the risk of overfitting and improve the model's generalization ability. Gradient pruning is used to stabilize parameter updates during training, and cosine annealing is employed to dynamically optimize the learning rate, ensuring stable convergence during model training. In each training round, the model parameters corresponding to the optimal performance on the validation set are saved for use in model inference during subsequent testing phases.
[0070] To verify the effectiveness of the method of this invention in the spatial transcriptome-pathological image multimodal tumor classification task, this embodiment uses a grouped five-fold cross-validation method to divide the training set and the validation set, so as to reduce the impact of sample partitioning randomness on the experimental results. The model uses accuracy (ACC), precision, recall, and F1 score as core evaluation indicators. The specific experimental results are shown in Table 1 below: Table 1. Evaluation results of the model of this invention in the multimodal tumor classification task.
[0071] Among them, accuracy is used to characterize the proportion of correctly classified samples in the overall model; precision is used to characterize the accuracy of the model in predicting positive samples; recall is used to characterize the model's ability to detect true positive samples; and F1 score is used to comprehensively evaluate the model's classification balance.
[0072] To verify the classification performance of the spatial multimodal fusion model proposed in this invention, comparative experiments were conducted on two public datasets, HEST-1K and STimage-1K4M. Four existing spatial multimodal methods—SPEMO, STPATH, StereoMM, and SpatialModal—were selected as baseline models. Accuracy, precision, recall, and F1 score were used as evaluation metrics; higher values for each metric indicate better model classification performance. The experimental results are shown in Table 2. Bold values represent the best results for each metric, and underlined values represent the second-best results. In the HEST-1K disease prediction, STimage-1K4M cancer binary classification, and tissue (organ) classification tasks, all metrics of this invention outperformed the baselines.
[0073] Table 2. Comparison of evaluation metrics between the model of this invention and mainstream spatial multimodal methods in multi-classification tasks.
[0074] The above multi-task comparison results demonstrate that, by introducing a local microenvironment difference enhancement module and an adaptive spatial bias attention module, this invention can effectively fuse multimodal information from spatial transcriptomics and pathological images, thereby better capturing the spatial heterogeneity characteristics of tissues. In multiple tasks, including disease prediction, cancer classification, and tissue / organ classification, the performance of this invention surpasses that of the existing similar methods listed, verifying the effectiveness of this technical solution.
[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling, characterized in that, Includes the following steps: S1. Construction of multimodal site joint characterization: Obtain spatial transcriptome gene expression data, pathological image visual data and spatial site coordinate information corresponding to tissue samples, and establish multimodal site joint characterization that integrates molecular expression information, tissue morphology information and spatial location information based on the correspondence of spatial sites. S2. Enhancement of Local Microenvironment Differences: For the joint representation of the multimodal sites, a local reference region corresponding to the central site is determined based on spatial location information. The spatial site feature information within the local reference region is correlated and modeled to obtain a local context representation. The difference information between the joint representation of the central site and the local context representation is calculated to obtain difference features used to represent spatial heterogeneity. The joint representation of the central site, the local context representation, and the difference features are fused, and a site representation enhanced with microenvironment is generated through an adaptive weight adjustment mechanism. S3. Adaptive Spatial Bias Modeling: Based on the enhanced site representation of the microenvironment, a spatial bias matrix containing learnable distance adjustment parameters is constructed according to the distance relationship between spatial sites. The spatial distance constraint strength is adaptively adjusted during model training using the learnable distance adjustment parameters. The feature association information between sites is fused with the spatial bias matrix to obtain a comprehensive spatial association weight, and the spatial site information is aggregated based on the comprehensive spatial association weight to obtain a global spatial fusion feature; S4. Tumor classification prediction: Output the tumor-related classification results corresponding to the tissue samples based on the global spatial fusion features.
2. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 1, characterized in that, The step S2, enhancing local microenvironment differences, specifically includes the following sub-steps: S201. Construction of local reference region: Calculate the Euclidean distance between the sites based on the two-dimensional coordinate information of each spatial site, and filter the neighboring sites corresponding to the central site according to spatial proximity to form a local reference region. S202. Construction of local context representation: For the features of neighboring sites within the local reference region, the association weight of each neighboring site to the central site is calculated through an attention mechanism. The feature information of the neighboring sites is aggregated according to the weight to obtain the local context representation corresponding to the central site. S203. Differential Feature Extraction: Calculate the vector difference between the multimodal joint representation of the central site and the local context representation to obtain differential features used to represent the differences between the central site and the surrounding microenvironment. S204, Adaptive Fusion Enhancement: The joint representation of the central site, the local context representation, and the differential features are spliced and fused. A dynamic weight in the range of 0 to 1 is generated through an adaptive weight adjustment mechanism to adaptively adjust the contribution ratio of the three types of features and output the site representation after microenvironment enhancement.
3. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 2, characterized in that, In step S202, the formula for calculating the local context representation is: in, , , Let i be the query vector, key vector, and value vector of the center site i and its neighboring site j, respectively. , , Let D be the learnable projective weight matrix, and D be the feature dimension scaling factor. Let j be the attention weight of neighboring site j to center site i. The set of effective neighboring sites corresponding to the central site i. This provides a basic characterization of the local microenvironment obtained through aggregation; In step S203, the formula for calculating the difference features is: in, For multimodal joint characterization of the central site, These are the distinguishing features.
4. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 2, characterized in that, In step S204, the adaptive weight adjustment mechanism is implemented through a gating unit, and the calculation formula is as follows: in, , These are the weight parameters and bias parameters in the learnable weight mapping unit, respectively. It is a sigmoid nonlinear activation function. This represents the dynamically adjusted weights generated by the learnable weight mapping unit. This enhances the local microenvironment features in the final output.
5. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 1, characterized in that, The adaptive spatial bias modeling step S3 specifically includes the following sub-steps: S301, Feature Correlation Matrix Calculation: Based on the enhanced site representation of the microenvironment, the feature similarity between any two points is calculated through the self-attention mechanism to construct the feature correlation matrix between sites; S302, Adaptive Spatial Bias Generation: Calculate the Euclidean distance based on the relative positions between spatial sites, introduce a learnable distance adjustment parameter to construct a spatial bias matrix, and adaptively update the distance adjustment parameter as the model iterates and trains, dynamically adjusting the degree of attenuation of the spatial distance on the site association strength. S303, Multi-information association fusion: The feature association matrix between sites and the spatial bias matrix are added element by element to obtain a comprehensive association weight matrix of fused molecular expression association information and spatial location constraint information. After normalization, the features of all sites are weighted and aggregated to obtain global spatial fusion features. S304, Residual Normalization Output: The global spatial fusion features are residually connected with the input microenvironment enhancement site representation, and after layer normalization, the final multimodal global feature representation is output.
6. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 5, characterized in that, In step S302, the formula for calculating the spatial bias matrix is: in, Let be the Euclidean spatial distance between site i and site j. This is a learnable distance adjustment parameter used to control the decay rate of spatial distance-to-site association strength. For site with site Gaussian space bias coefficients between them; In step S303, the formula for calculating the comprehensive correlation weight matrix is as follows: in, B is the feature correlation matrix between sites, and A is the spatial bias matrix. This is a comprehensive correlation weight matrix.
7. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 1, characterized in that, Step S1 specifically includes: Preprocessing of raw spatial transcriptome gene expression data and pathological image visual features includes missing data filtering, abnormal site removal, and invalid tissue region screening; Based on the spatial site coordinates, a mapping relationship between spatial transcriptome sites and corresponding regions of pathological images is established. Linear projections are performed on the spatial transcriptome molecular features and the pathological image visual features, respectively, mapping the two types of features to a hidden feature space of the same dimension. The mapped gene expression features are spliced and fused with pathological visual features to construct a multimodal site joint representation corresponding to each spatial site, while retaining the corresponding two-dimensional spatial coordinate information.
8. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 5, characterized in that, In step S304, the formula for calculating the residual normalization is: in, V represents the spatial fusion features after global weighted aggregation, where V is the value feature matrix. The input is the microenvironment enhancement site characterization matrix. This is the final multimodal feature representation that integrates local microenvironment information and global spatial structure information.
9. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 1, characterized in that, The following configuration is used during the model training phase: We use the cross-entropy loss function with class weights as the optimization objective, and combine it with a label smoothing strategy to reduce the risk of model overfitting. A gradient pruning strategy is adopted to stabilize the parameter update process, and a cosine annealing learning rate adjustment strategy is adopted to dynamically optimize the learning rate. A five-fold cross-validation method is used to divide the training set and the validation set. In each round of training, the model parameters corresponding to the best performance of the validation set are saved. The model's hidden layer dimension was set to 96, the Dropout rate was set to 0.1, the initial learning rate was set to 1e-4, and the total number of training iterations was set to 60.
10. The multimodal tumor classification method based on microenvironment enhancement and spatial bias modeling according to claim 1, characterized in that, The tumor-related classification results include at least one of the following: tumor disease classification results, tumor benign / malignant binary classification results, tissue / organ origin classification results, tumor subtype classification results, and tissue lesion status identification results; the method is applied to spatial multi-omics tissue sample intelligent classification analysis, tumor auxiliary diagnosis, and pathological analysis scenarios.
