Pathological image and omics fusion-based tumor microenvironment analysis method and system

Abstract

The application belongs to the technical field of tumor microenvironment analysis, and particularly relates to a tumor microenvironment analysis method and system based on pathological image and omics fusion, which comprises the following steps: acquiring pathological image and omics data of a tumor sample; after preprocessing, a multi-level structure graph is constructed through a pathological structure perception branch, and an omics function graph is constructed through an omics function coding branch; the two graphs are input into a cross-modal structure alignment module, and a cross-modal correlation between pathological nodes and omics nodes is established through bidirectional attention; a pathological-omics fusion graph is constructed based on the nodes and the correlation; a graph representation learning is performed on the structure gate hetero-graph Transformer to obtain comprehensive embedding representation; and finally, tumor microenvironment structure typing, molecular subtype prediction, treatment response evaluation or key structure contribution analysis results are output. The application realizes deep fusion of pathological morphology and molecular mechanism, and enhances tumor microenvironment structure perception ability and clinical interpretability.

Description

Technical Field

[0001] This invention belongs to the field of tumor microenvironment analysis technology, and particularly relates to a method and system for tumor microenvironment analysis based on the fusion of pathological imaging and omics. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, multimodal integration methods based on histopathological images and multi-omics data have made initial progress in tumor microenvironment analysis. However, most current multimodal methods, when integrating pathological images and multi-omics data, model pathological image features and omics features separately, lacking cross-modal information alignment, resulting in low quality of multimodal coding. At the same time, they do not make sufficient use of the spatial structural information in pathological images, making it difficult to effectively perceive the spatial heterogeneity of the microenvironment such as cell arrangement and invasion boundaries, and the characterization of cross-modal correlations is insufficient, failing to accurately capture the deep biological coupling between pathological morphology and molecular mechanisms.

[0004] Therefore, the problems of how to achieve deep integration of pathological images and omics information, enhance the ability to perceive the structure of the tumor microenvironment, and establish interpretable cross-modal associations urgently need to be solved. Summary of the Invention

[0005] To overcome the shortcomings of the existing technologies, this invention provides a tumor microenvironment analysis method and system based on the fusion of pathological imaging and omics. The aim is to construct a MOSAIC-TME-Net model that includes a pathological structure perception branch, a cross-modal structure alignment module, and a structure-gated heterogeneous graph Transformer module. This model enables cross-modal alignment and joint representation learning of spatial structural features of pathological images and molecular functional features of omics, enhancing the ability to perceive the structure of cell arrangement, tissue micro-region spatial topology, and overall sample heterogeneity within the tumor microenvironment. It also establishes an interpretable bidirectional correlation between pathological morphology and molecular mechanisms, thereby improving the accuracy and clinical interpretability of tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, and key structural contribution analysis.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: Firstly, a tumor microenvironment analysis method based on the fusion of pathological imaging and omics is disclosed, including: Pathological images and omics data of tumor samples were acquired and standardized respectively. A multi-level structural atlas was constructed through a pathological structure perception branch, and an omics functional atlas was constructed through an omics function coding branch. The two atlases were input into a cross-modal structure alignment module, and a cross-modal association between pathological structure nodes and omics functional nodes was established through a bidirectional attention mechanism to obtain alignment features. A pathology-omics fusion graph was constructed based on pathological structure nodes, omics functional nodes, and alignment features. The fusion graph is input into the Transformer module of the structure-gated heterogeneous graph for graph representation learning to obtain a comprehensive embedding representation; based on the comprehensive embedding representation, the results of tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment or key structure contribution analysis are output.

[0007] Furthermore, the pathological structure perception branch sequentially includes an image patch input layer, a shallow texture coding layer, a multi-scale feature extraction layer, a cross-scale feature fusion layer, a structural boundary enhancement layer, a regional feature enhancement layer, a regional discrimination layer, and a structured mask output layer.

[0008] Furthermore, the multi-scale feature extraction layer includes a small receptive field convolution branch, a medium receptive field convolution branch, a large receptive field dilated convolution branch, and a contextual pooling branch.

[0009] Furthermore, the cross-modal structure alignment module performs pathology-to-omics attention computation and omics-to-pathology attention computation, and introduces cross-modal consistency constraints. The cross-modal structure correlation between pathological structure nodes and omics functional modules is obtained by weighting feature similarity, task relevance, and stable correlation. The pathology-related omics-enhanced representation and the omics-related pathology-enhanced representation are generated by bidirectional attention weights, and the contrast constraint loss is used to narrow the cross-modal representation distance of the same sample and widen the cross-modal representation distance of different samples.

[0010] Furthermore, the nodes of the patho-omics fusion graph include pathological structure nodes and omics function nodes, and the edge set includes pathological internal structure edges, omics internal function edges, and patho-omics cross-modal edges.

[0011] Furthermore, the structure-gated heterogeneous graph Transformer module sequentially includes a node input embedding layer, a relation type encoding layer, a structure gating layer, a heterogeneous multi-head attention layer, a residual connection and normalization layer, a feedforward mapping layer, a multi-layer stacked update layer, a global pooling layer, and a task output layer.

[0012] Furthermore, the model is trained using a joint loss function, which includes task loss, structure recognition loss, fusion consistency loss, graph relation constraint loss, and regularization loss.

[0013] Secondly, a tumor microenvironment analysis system based on the fusion of pathological images and omics is disclosed, including: a data acquisition module, used to acquire pathological images and omics data of tumor samples, and perform standardized processing respectively; The atlas construction module is used to construct multi-level structural atlases through the pathological structure perception branch and to construct omics functional atlases through the omics functional coding branch. The cross-modal alignment module is used to input two types of maps into the cross-modal structure alignment module. Through a bidirectional attention mechanism, it establishes cross-modal associations between pathological structural nodes and omics functional nodes to obtain alignment features. The fusion graph construction module is used to construct pathology-omics fusion graphs based on pathological structural nodes, omics functional nodes, and alignment features. The graph representation learning module is used to perform graph representation learning on the fused graph input structure-gated heterogeneous graph Transformer module to obtain a comprehensive embedded representation; The analysis output module is used to output tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, or key structure contribution analysis results based on comprehensive embedding representation.

[0014] Thirdly, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the steps of the method described above.

[0015] Fourthly, a computer-readable storage medium is disclosed having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0016] The above one or more technical solutions have the following beneficial effects: This invention proposes a tumor microenvironment analysis method and system based on the fusion of pathological imaging and omics. By constructing a multi-level structural atlas of the tumor microenvironment composed of cell layer, micro-region layer and sample layer, it can systematically characterize the spatial relationships between tumor cells, stromal components, immune infiltration regions, tertiary lymphoid structures, necrotic regions and vascular-related regions, thereby enhancing the model's ability to model the structural heterogeneity of the tumor microenvironment.

[0017] This invention proposes the MOSAIC-TME-Net model, which performs cross-modal alignment and heterogeneous graph fusion of pathological image structural features and omics functional features, enabling the establishment of bidirectional correlations between pathological structural nodes and omics functional nodes. Compared to traditional single-modal analysis methods or simple feature stitching methods, this invention can more fully explore the intrinsic links between pathological morphological changes and molecular functional abnormalities, thereby improving the accuracy, stability, and interpretability of tumor microenvironment analysis.

[0018] This invention also employs a structure-gated heterogeneous graph Transformer module for adaptive modeling of multi-type nodes and multi-type edge relationships, enabling the model to dynamically focus on key cellular neighborhoods, tissue microregions, and omics pathways according to different tasks. This method is not only applicable to tumor microenvironment structure subtyping but can also be used for molecular subtype prediction, treatment response assessment, prognostic risk analysis, and interpretation of key microenvironment structures, demonstrating significant clinical auxiliary analytical value and promising prospects for widespread application.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart of the tumor microenvironment analysis method based on the fusion of pathological imaging and omics according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of cross-modal structure alignment and fusion graph representation learning according to Embodiment 1 of the present invention; Figure 3 This is a structural diagram of the tumor microenvironment analysis system based on the fusion of pathological imaging and omics, according to Embodiment 2 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Example 1 Existing technologies, when integrating pathological images and omics data, often model pathological image features and omics features separately, neglecting the information alignment between the two modalities. This means they fail to make the pathological structures and molecular features of the same patient as similar as possible in the latent space, while simultaneously minimizing the cross-modal features of different patients. Furthermore, existing methods underutilize the spatial structural information in pathological images, making it difficult to effectively perceive the spatial heterogeneity of the microenvironment, such as cell arrangement and infiltration boundaries. Moreover, the characterization of cross-modal associations is insufficient, failing to accurately capture the deep biological coupling between pathological morphology and molecular mechanisms, resulting in weak interpretability of the analysis results.

[0026] Therefore, directly splicing or simply fusing multimodal information makes it difficult to obtain high-quality cross-modal characterization, which limits the accuracy and clinical reliability of tumor microenvironment analysis. To address these technical issues, this application proposes a tumor microenvironment analysis method that integrates pathological imaging and omics information. By constructing the MOSAIC-TME-Net model, it achieves multi-level joint analysis of pathological structure and molecular function, which can enhance the ability to perceive tumor microenvironment structure and establish interpretable cross-modal associations, thereby improving the accuracy and interpretability of tumor microenvironment structure typing, molecular subtype prediction, and treatment response assessment.

[0027] In this embodiment, the MOSAIC-TME-Net model includes, in sequence from data input to result output, the following modules: data preprocessing and quality control module, pathological structure perception branch, omics function encoding branch, cross-modal structure alignment module, pathology-omics fusion graph construction module, structure-gated heterogeneous graph Transformer module, and task output and interpretation module.

[0028] The data preprocessing and quality control module is used to perform color normalization, tissue region extraction, background removal, image slicing, and effective image block selection on pathological images, and to handle missing values, standardize, and merge functional modules in omics data. The pathological structure perception branch extracts multi-scale morphological features from pathological image blocks, identifies tumor regions, stroma regions, immune infiltration regions, necrotic regions, vascular-related regions, and tertiary lymphoid structures, and generates structured masks and multi-level structural maps. The omics function encoding branch encodes omics variables such as transcriptomics, proteomics, mutations, methylation, or metabolomics into biologically functional components. The system includes functional node and module representations; a cross-modal structure alignment module for establishing bidirectional attention relationships between pathological structure nodes and omics functional nodes; a pathology-omics fusion graph construction module for unifying pathological structure nodes, omics functional nodes, homomodal edges, and cross-modal edges into a heterogeneous fusion graph; a structure-gated heterogeneous graph Transformer module for weighted information transfer and graph representation learning between multi-type nodes and multi-type edges; and a task output and interpretation module for outputting tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, and key structure contribution analysis results based on comprehensive embedding representation.

[0029] The connection order of the above modules is as follows: pathological images and omics data are first input into the data preprocessing and quality control module; the preprocessed pathological image blocks are input into the pathological structure perception branch to obtain pathological structure nodes, structured masks, and multi-level structure maps; the preprocessed omics data are input into the omics function coding branch to obtain omics function nodes and omics function maps; subsequently, the multi-level structure maps and omics function maps are jointly input into the cross-modal structure alignment module to obtain cross-modal association alignment features; then, the pathology-omics fusion graph construction module forms a unified pathology-omics fusion graph; finally, it is input into the structure-gated heterogeneous graph Transformer module for graph representation learning, and the final analysis results are obtained through the task output and interpretation module.

[0030] like Figure 1 As shown, this embodiment discloses a tumor microenvironment analysis method based on the fusion of pathological imaging and omics, including: Step S1: Input of pathological images and omics data; Multimodal data from multiple tumor samples to be analyzed are acquired, including pathological imaging data and omics data. For the first... For each tumor sample, the input data is represented as follows:

[0031] in, Indicates the first Pathological imaging data of each sample, Indicates the first The pathological imaging data includes, but is not limited to, HE-stained pathological slide images, immunohistochemical images, digital whole-slide images, local high-resolution pathological images, or microscopic pathological images; the omics data includes, but is not limited to, one or more of the following: transcriptome data, proteome data, gene mutation data, DNA methylation data, metabolome data, or immune repertoire data. Pathological images are used to characterize tissue morphology, cell distribution, spatial structure, and regional heterogeneity in the tumor microenvironment; omics data are used to characterize the molecular functional state, signaling pathway activity, immune regulatory characteristics, and abnormal molecular changes in the tumor microenvironment. The combined input of these two types of data provides a fundamental data source for subsequent pathological structure modeling, omics function modeling, and cross-modal fusion analysis.

[0032] Step S2: Pathological image preprocessing and omics standardization; The pathological imaging data and omics data obtained in step S1 were standardized to reduce the impact of differences in image quality, staining, batch effects, and omics dimensional differences on subsequent analyses. For the pathological imaging data... The process involves color normalization, tissue region extraction, background removal, image slicing, and effective image block selection to obtain a standardized set of pathological image blocks.

[0033] in, Indicates the first A collection of pathological image patches obtained after preprocessing a sample. Indicates the first The first sample A pathological image block, This indicates the number of valid pathological image patches retained from the sample. To remove pathological image patches with excessively high background ratios, blurred images, or abnormal staining, a pathological image patch validity index is defined:

[0034] in, Indicates the first In the nth sample The effective index of a pathological image block; This indicates the effective tissue proportion in the image patch; This indicates the sharpness index of the image patch; This indicates the staining stability index of the image patch; This indicates the proportion of blank background in the image block; , , , For weighting coefficients. Preferably, when When, remove the pathological image block; when If necessary, the pathological image block is retained for subsequent analysis. The retained pathological image block is input into the pathological structure perception branch for feature extraction and region recognition. The pathological structure perception branch includes a multi-scale feature extraction unit, a structural boundary enhancement unit, and a region discrimination unit.

[0035] For the The first sample For each pathological image block, initial features are first extracted through a shallow texture coding layer, which includes a convolutional layer, a normalization layer, and a nonlinear activation layer, to obtain basic morphological information such as cell nucleus morphology, staining distribution, local texture, and tissue edges.

[0036] Subsequently, the initial features are input into a multi-scale feature extraction layer. This multi-scale feature extraction layer includes a small receptive field convolution branch, a medium receptive field convolution branch, a large receptive field dilated convolution branch, and a contextual pooling branch. Specifically, the small receptive field convolution branch extracts cellular-level features such as cell nucleus size, cell density, and local staining texture; the medium receptive field convolution branch extracts micro-area-level features such as tumor cell clusters, stromal reaction zones, and immune cell aggregation zones; the large receptive field dilated convolution branch extracts tissue-level features such as infiltration boundaries, necrotic areas, vascular-related areas, and tertiary lymphoid structures; and the contextual pooling branch supplements the global tissue distribution information within the image patch.

[0037] The features output from each scale branch are concatenated and input into a cross-scale feature fusion layer, where channel weighting and convolutional compression are used to obtain fused features. After obtaining the fused features, a structural boundary enhancement layer enhances the structural boundaries in the pathological image. Specifically, the structural boundary enhancement layer calculates a boundary response map based on the fused features and combines image gradient information to enhance the boundaries of tumor-stromal junctions, immune infiltration edges, necrotic edges, and perivascular structures. The boundary response map is used to improve the distinguishability between adjacent microenvironment regions and reduce missegmentation of different tissue regions at the boundaries.

[0038] For omics data Missing value handling, batch effect correction, outlier handling, and standardization transformation are performed. For the first... Each omics feature has a standardized value expressed as:

[0039] in, Indicates the first In the nth sample One original omics feature value, and They represent the first The mean and standard deviation of each omics feature in the training samples To prevent the smoothing coefficient from being zero in the denominator, This represents the standardized omics feature values. Through the above processing, a set of pathological image patches with stable quality and uniform scale, as well as standardized omics features, are obtained, providing standardized input for the subsequent construction of multi-level structural maps and omics functional maps.

[0040] Step S3: Construction of multi-level structure maps and omics functional maps; Based on the standardized pathological image patch set and omics-normalized features obtained in step S2, a multi-level structural atlas characterizing the spatial heterogeneity of the tumor microenvironment and an omics functional atlas characterizing molecular functional associations are constructed. First, the pathological image patch set... The pathological structure perception branch is input. From the input end to the output end, the pathological structure perception branch sequentially includes: an image patch input layer, a shallow texture coding layer, a multi-scale feature extraction layer, a cross-scale feature fusion layer, a structural boundary enhancement layer, a region feature enhancement layer, a region discrimination layer, and a structured mask output layer.

[0041] Specifically, the image patch input layer is used to receive standardized pathological image patches after color normalization, tissue region extraction, background removal, and effective image patch filtering. And use it as the basic input for subsequent structure-aware analysis. For the first The first sample A pathological image block, whose input can be represented as:

[0042] in, Indicates the first A collection of pathological image patches for each sample. Indicates the first The first sample A pathological image block, This indicates the number of valid pathological image patches retained from the sample. The shallow texture encoding layer is used to perform initial feature encoding on the input pathological image patches, extracting low-level visual features such as cell nucleus morphology, staining intensity, local texture, and tissue edges. Its output shallow features can be represented as:

[0043] in, Indicates the first In the nth sample Shallow texture features of a pathological image patch This represents a shallow texture encoding function consisting of convolutional layers, normalization layers, and nonlinear activation layers. Subsequently, the shallow texture features... Input a multi-scale feature extraction layer. This layer includes a small receptive field convolution branch, a medium receptive field convolution branch, a large receptive field dilated convolution branch, and a contextual pooling branch. Specifically, the small receptive field convolution branch extracts cellular-level features such as cell nucleus size, cell density, and local staining texture; the medium receptive field convolution branch extracts micro-area-level features such as tumor cell clusters, stromal reaction zones, and immune cell aggregation zones; the large receptive field dilated convolution branch extracts tissue-level features such as invasion boundaries, necrotic areas, vascular-related areas, and tertiary lymphoid structures; and the contextual pooling branch supplements the global tissue distribution information within the image patch. The multi-scale feature extraction process can be represented as:

[0044] in, , , and These represent the features extracted by the small receptive field branch, the medium receptive field branch, the large receptive field branch, and the context pooling branch, respectively. , , and These represent the feature extraction functions for the corresponding branches. The cross-scale feature fusion layer is used to stitch together, weightedly fuse, and compress channels of pathological features at different scales to obtain a unified representation of the pathological structure.

[0045] in, This indicates the pathological structural features after fusion. This indicates a feature concatenation operation. This represents a cross-scale feature fusion function. This layer unifies cellular, micro-region, and tissue structure level information into a single pathological structural representation. The structural boundary enhancement layer enhances the tumor-stromal boundary, immune infiltration boundary, necrosis boundary, and vascular-related structural boundary. This layer calculates the boundary response map based on the fusion features and combines image gradient information to highlight the boundary regions between adjacent tissue structures.

[0046] in, Boundary response map representing a patch in a pathological image. Represents image gradient information, This represents the structural boundary enhancement function. This layer can reduce missegmentation of different microenvironment regions at the boundaries, improving the accuracy of structure recognition. The region feature enhancement layer is used to generate region attention weights based on the spatial distribution and morphological features of different microenvironment structures, and to perform weighted enhancement on tumor regions, stroma regions, immune infiltration regions, necrotic regions, vascular-related regions, and tertiary lymphoid structures respectively. The process can be represented as follows:

[0047]

[0048] in, Represents the region attention weight map. This represents the region attention generation function. This indicates an element-wise weighted operation. This represents the enhanced regional structural features. Higher weights are assigned to local regions that are strongly correlated with the current structural category, while lower weights are assigned to background regions, abnormally stained regions, or irrelevant tissue regions, thereby improving the stability of key microenvironment structure identification.

[0049] The region discrimination layer is used to determine the structure category of each pixel or local region based on the enhanced region structure features, and outputs a probability map of different microenvironment structure categories:

[0050] in, Indicates the first In the nth sample Structural category probability map of a pathological image patch This represents the weight matrix of the region discrimination layer. Indicates the bias term. This is used to output the probability of each structural category. Finally, the structured mask output layer generates a set of structured masks based on the structural category probability map output by the region discrimination layer, identifying tumor regions, stroma regions, immune infiltration regions, necrotic regions, vascular-related regions, and tertiary lymphoid structures, and outputs the set of structured masks:

[0051] in, Indicates the first In the nth sample A set of structured masks for each pathological image patch; , , , , , These represent the masks corresponding to the tumor region, stroma region, immune infiltration region, necrosis region, vascular-related region, and tertiary lymphoid structure region, respectively.

[0052] In the aforementioned pathological structure perception branch, the connection order between each layer is as follows: standardized pathological image blocks enter the shallow texture coding layer through the image block input layer to obtain initial texture features; the initial texture features enter the multi-scale feature extraction layer to obtain cellular, micro-region, tissue structure, and contextual features respectively; features at different scales are integrated into a unified pathological structure representation through the cross-scale feature fusion layer; the fused features are further input into the structural boundary enhancement layer and the regional feature enhancement layer to highlight key microenvironment regions and their boundaries; the enhanced regional features generate a structural category probability map through the regional discrimination layer; finally, various microenvironment structure masks are obtained from the structured mask output layer, providing nodes and regional basis for the subsequent construction of multi-level structural maps of the cellular layer, micro-region layer, and sample layer.

[0053] Step S4, Cross-modal structural alignment: Pathology-omics attention, omics-pathology attention, and consistency constraints; The multi-level structure map obtained in step S3 and omics functional map The input cross-modal structure alignment module establishes a bidirectional association between pathological structure nodes and omics functional nodes through pathology-to-omics attention, omics-to-pathology attention, and consistency constraints.

[0054] set up Indicates the first In the nth sample Feature representation of each pathological structural node. Indicates the first In the nth sample Aggregate representation of omics functional modules. The cross-modal structural correlation degree between pathological structural nodes and omics functional modules is defined as:

[0055] in, Indicates the first Pathological structural nodes in each sample With omics functional modules Cross-modal structural correlation between them; This represents the similarity calculation function; and These represent mapping matrices for pathological features and omics features, respectively. This indicates the task-related relationships between pathological structural nodes and omics functional modules. This indicates a stable correlation between the two. , , These are the weighting coefficients.

[0056] The pathology-to-omics attention is used to select relevant omics functional modules based on pathological structural features, and its attention weight is expressed as follows:

[0057] Based on this attention weight, an omics-enhanced representation of the pathological structural node associations is obtained:

[0058] Omics-to-pathological attention is used to reverse-locate relevant pathological structural nodes based on omics functional abnormalities, and its attention weight is expressed as follows:

[0059] Based on this attention weight, the pathological enhancement representation associated with the omics functional modules is obtained:

[0060] To ensure that the bidirectional attention results from pathology to omics and from omics to pathology remain consistent in the shared space, a cross-modal consistency constraint is introduced:

[0061] in, Indicates the first The aggregated representation of the pathological structures of each sample Indicates the first The aggregated representation of omics functions of a sample This represents the contrast constraint loss, used to narrow the patho-omics representation distance of the same sample and increase the cross-modal representation differences between different samples. Through this step, cross-modal association alignment features between pathological structural nodes and omics functional nodes are obtained, providing a basis for edge relationships in subsequent fusion graph construction.

[0062] Step S5: Construction of the pathology-omics fusion map; Based on the pathological structural nodes and omics functional nodes obtained in step S3, and the cross-modal association alignment features obtained in step S4, a unified pathology-omics fusion graph is constructed:

[0063] in, Indicates the first Pathogen-omics fusion map of individual samples; This represents the set of nodes in the fusion graph; Represents the set of edges in the fused graph; Represents a set of relation types. The set of nodes in a fusion graph is defined as:

[0064] in, Represents a set of pathological structural nodes. This represents the set of omics functional nodes. The set of edges in the fusion graph is defined as:

[0065] in, Represents the internal structure of a pathology, used to describe the spatial structural relationships between cells, regions, and image blocks; Indicates the functional edges within omics, used to describe the molecular functional relationships between genes, proteins, pathways, or functional modules; This represents a pathology-omics cross-modal edge, used to describe the cross-modal association between pathological structural nodes and omics functional nodes.

[0066] Preferably, cross-modal edges are selected based on the cross-modal structural correlation degree obtained in step S4:

[0067] in, Representing pathological structural nodes, Represents omics functional nodes. This represents the cross-modal edge filtering threshold. When... At that time, it was assumed that there was an effective cross-modal association between the pathological structural node and the corresponding omics functional node, and cross-modal edges were established in the fusion graph. The edge features of the cross-modal edges can be represented as:

[0068] in, Representing pathological structural nodes With omics functional nodes Cross-modal edge features between them. Through this step, pathological spatial structure relationships, omics functional relationships, and patho-omics cross-modal relationships are uniformly organized into the same heterogeneous fusion graph.

[0069] Step S6: Learning the Transformer representation of a structure-gated heterogeneous graph; The patho-omics fusion map constructed in step S5 The input structure-gated heterogeneous graph Transformer module performs graph representation learning on multiple types of nodes and edges to obtain sample-level comprehensive embedding representations.

[0070] First, initial mapping is performed on the different types of nodes in the fusion graph. For any node... Its initial node is represented as:

[0071] in, Represents a node The initial latent space representation; Represents the original features of a node; Indicates the node type; and These represent the mapping matrix and bias term for the corresponding node type, respectively.

[0072] Subsequently, the information transmission intensity of different types of edges is dynamically adjusted through structural gating units. For nodes... ,node and its relation types The structural gating weights are expressed as:

[0073] in, Represents a node With nodes In relation types Information transmission gating weights are applied below. Represents the Sigmoid function; Represents the gate weight matrix; Indicates the bias term; and They represent the first Layer nodes and nodes Feature representation; Representing relation type The encoded vector; This represents the structural, functional, or cross-modal correlation between nodes. In a heterogeneous multi-head attention layer, nodes... For neighboring nodes The attention weights are represented as follows:

[0074] in, Indicates the first Layer nodes For nodes In relation types Attention weights; Represents a node In relation types The set of neighboring nodes; and Representing relation types respectively The corresponding query matrix and key matrix; Represents the dimension of the latent space; Represents the relation type encoding bias term. Node The aggregated information is represented as:

[0075] in, Represents a node In the Information obtained from layer aggregation; Representing relation type The corresponding feature transformation matrix.

[0076] Furthermore, the node representation is updated through residual connections, layer normalization, and feedforward mapping:

[0077] in, Represents a node In the The updated representation of the layer; Presentation layer normalization operation; This represents a feedforward neural network. After updating the heterogeneous graph using a multi-layer gated Transformer, weighted pooling is performed on all nodes in the fused graph to obtain the [n]th [node]. The comprehensive embedding representation of each sample:

[0078] in, Indicates the first The comprehensive embedding representation of each sample, Represents a node After the first The updated node representation of the layer. Represents a node The pooling weights can be expressed as:

[0079] Through this step, the model can integrate pathological spatial structure information, omics functional network information, and patho-omics cross-modal coupling information in a unified graph structure, forming a sample-level comprehensive representation for subsequent analysis tasks.

[0080] like Figure 2 As shown, this embodiment uses the Transformer module of structure-gated heterogeneous graphs to perform graph representation learning on the pathology-omics fusion graph. Taking multi-level structure maps and omics function maps as inputs, the graphs undergo cross-modal structure alignment, pathology-omics fusion graph construction, and structure-gated heterogeneous graph representation learning, and finally output the comprehensive embedding representation and task analysis results.

[0081] The multi-level structural atlas includes a cell layer, a micro-region layer, and a sample layer, used to characterize cell arrangement, tissue micro-region spatial topology, and overall sample heterogeneity in the tumor microenvironment; the omics functional atlas includes genes, proteins, pathways, and functional modules, used to characterize molecular functional states, signaling pathway activity, and immune regulatory features.

[0082] The multi-level structural map and the omics functional map are jointly input into the cross-modal structural alignment module. This module establishes a bidirectional association between pathological structural nodes and omics functional nodes through pathology-to-omics attention computation and omics-to-pathology attention computation, and introduces consistency constraints to ensure that the bidirectional attention results from pathology to omics and from omics to pathology remain consistent in the shared space.

[0083] The Transformer module for structurally gated heterogeneous graphs includes, from input to output, a node input embedding layer, a relation type encoding layer, a structural gating layer, a heterogeneous multi-head attention layer, a residual connection and normalization layer, a feedforward mapping layer, a multi-layer stacked update layer, and a global pooling layer.

[0084] The node input embedding layer receives pathological structural node features and omics functional node features, mapping node features from different sources and dimensions to a unified latent space dimension. The relation type encoding layer assigns relation type codes to internal pathological structural edges, internal omics functional edges, and patho-omics cross-modal edges, enabling the model to distinguish spatial adjacency, functional association, and cross-modal coupling relationships represented by different edge types. The structure gating layer dynamically calculates information transfer weights based on node features, edge type features, and cross-modal structural association. The heterogeneous multi-head attention layer is used to manage interactions between different types of nodes and edges. Weighted information aggregation is performed; a residual connection and normalization layer fuses the original node representation with the aggregated information, and a feedforward mapping layer further performs nonlinear transformation; subsequently, the above-mentioned structural gating, heterogeneous attention aggregation, residual normalization, and feedforward mapping operations are repeatedly executed through multi-layer stacked update layers, enabling pathological structural nodes to fuse local tissue spatial information, omics functional nodes to fuse molecular functional network information, and cross-modal edges to transmit coupling information between pathological morphology and molecular function; finally, a global pooling layer performs weighted pooling on all pathological structural nodes and omics functional nodes to obtain a sample-level comprehensive embedding representation. This comprehensive embedding representation is used for subsequent tumor microenvironment structural typing, molecular subtype prediction, treatment response assessment, and key structural contribution analysis.

[0085] During this process, the information transmission between different types of nodes is regulated by the structural gating unit, and its working logic is as follows: First, the structure-gated heterogeneous graph Transformer module receives the pathology-omics fusion graph as input and processes the spatial structure information transmitted by the internal structure of the pathology, the molecular functional information transmitted by the internal function of the omics, and the pathology-omics association information transmitted across modalities.

[0086] Secondly, the structural gating unit dynamically calculates the information transmission weights of different types of edges and different neighbor nodes based on the relevance of the current task. Specifically, structural relationships that are more relevant to the task receive higher retention weights, thus playing a dominant role in information aggregation; while relationships that are more disruptive or less stable receive lower weights, and their information contribution is effectively suppressed.

[0087] Finally, through iterative updates of the multi-layer heterogeneous graph Transformer, the final representation of each node integrates its own features, features from its same-modal neighbors, and features from its cross-modal related neighbors. Based on this, the 1st... Comprehensive embedding representation of each sample The integrated embedded representation This reflects the combined state of the sample at the pathological structural level, spatial organization level, and omics functional level, which can be used for subsequent tumor microenvironment structure perception analysis tasks.

[0088] Step S7, Analysis Results Output: Microenvironment structure typing, molecular subtype prediction, treatment response assessment, and key structure contribution analysis; Based on the comprehensive embedding representation obtained in step S6 The output task is to determine at least one of the following: tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, and key structure contribution analysis. When the output task is a classification task, the prediction result is expressed as follows:

[0089] in, Indicates the first Classification prediction results for each sample; This represents the output layer weight matrix; Indicates the output layer bias term; This is used to output probability values ​​corresponding to different categories. When the task is tumor microenvironment structure typing... This indicates the probability that the sample belongs to different microenvironmental structure types; when the task is molecular subtype prediction... This represents the probability that the sample belongs to a different molecular subtype. When the output task is treatment response assessment, it can be represented as:

[0090] in, Indicates the first The probability that each sample belongs to a different treatment response category, such as major pathological remission, non-major pathological remission, complete pathological remission, partial remission, or no remission. When the output task is critical structure contribution analysis, the contribution of critical nodes is calculated based on structural gating weights, attention weights, and node representation contributions.

[0091] in, Represents a node Contribution to the final analysis results; Represents a node with neighboring nodes In the Layers, Relationship Types Attention weights; This represents the information transmission gating weight of the corresponding edge. According to... The size of the output can determine the pathological regions, key cell neighborhoods, tissue microregions, omics functional modules, or cross-modal associations that have a significant impact on the discrimination results.

[0092] Preferably, during the model training phase, a joint loss function is used for optimization to simultaneously constrain the prediction task, pathological structure recognition, cross-modal alignment, and the stability of fusion graph relationships:

[0093] in, This represents the total loss during model training; Indicates task loss; This represents the structural recognition loss; This indicates the loss of consistency during fusion; Represents graph relation constraint loss; Indicates the regularization loss; , , , and These represent the weight coefficients of each loss term, used to adjust the contribution of different training objectives to the total loss. Task Loss This is used to measure the difference between the model's output and the true labels. When the output task is a classification task, the task loss can be expressed as cross-entropy loss, as follows:

[0094] in, Indicates the sample number; Indicates the category number; Indicates the first The sample at the th Real labels in each category; The model predicts the first... The sample belongs to the first The probability of each category. When the output task is a continuous risk scoring or regression task, the task loss can also use the mean squared error loss. Structural recognition loss. This is used to constrain the accuracy of the pathological structure perception branch in identifying tumor regions, stroma regions, immune infiltration regions, necrotic regions, vascular-associated regions, and tertiary lymphoid structures. The structure recognition loss can be composed of both cross-entropy loss and Dice loss.

[0095] in, This represents pixel-level or region-level cross-entropy loss; The Dice loss measures the degree of overlap between the predicted mask and the true or weakly labeled structure mask. The Dice loss can be expressed as:

[0096] in, This represents the structural mask predicted by the model; This represents the corresponding real or weakly labeled structural mask; To prevent smoothing coefficients with zero denominators, a fusion consistency loss is applied. This is used to constrain the consistency between pathology-to-omics attention results and omics-to-pathology attention results in the cross-modal structure alignment module within the shared space. The fusion consistency loss can be expressed as:

[0097] in, This represents the embedded representation of pathological structural nodes after cross-modal alignment. This represents the embedded representation of omics function nodes after cross-modal alignment. Represents the squared Euclidean distance; This represents the contrast constraint loss, used to narrow the pathomic representation distance of the same sample and increase the cross-modal representation distance between different samples. (Figure: Relationship constraint loss) This is used to maintain the rationality of edge relationships and topological stability in the pathomics fusion graph. For any node in the fusion graph... and nodes The graph relation constraint loss can be expressed as:

[0098] in, Represents nodes in the fused graph With nodes The strength of structural correlation, functional correlation, or cross-modal correlation between them; Represents a node With nodes The similarity between the updated embeddings; and Representing nodes respectively and nodes The node representations are used to constrain the node representations learned by the model to maintain consistency with the pre-built graph relationships. Regularization loss. To constrain the size of model parameters and reduce the risk of overfitting, it can be expressed as:

[0099] in, This represents the set of trainable parameters in the model. Through the aforementioned joint loss function, the model can optimize the final prediction performance while maintaining the stability of pathological structure identification, cross-modal consistency of pathological and omics information, and the rationality of the topological relationships in the fusion graph, thereby improving the accuracy, robustness, and interpretability of tumor microenvironment analysis results.

[0100] Joint optimization can improve the overall robustness and interpretability of the model. After model training, the model network parameters, tumor microenvironment map construction rules, and fusion analysis parameters are saved to form a complete tumor microenvironment structure perception and analysis model. For newly input tumor samples to be analyzed, their pathological imaging data and omics data are preprocessed, map constructed, cross-modal aligned, and fusion graph representation learned, and then input into the trained model. This allows for the rapid output of tumor microenvironment structure classification, molecular subtype prediction, treatment response assessment, or key structure contribution analysis results, providing multi-level quantitative evidence for clinical decision support.

[0101] Specific application examples: The tumor microenvironment structure perception and analysis method provided in this embodiment was applied to the prediction of treatment response in non-small cell lung cancer and the auxiliary analysis of molecular subtypes in breast cancer.

[0102] Application Case 1: Multimodal Fusion Therapy Response Prediction Based on Non-Small Cell Lung Cancer Samples This application example focuses on non-small cell lung cancer patients. It uses the tumor microenvironment structure perception and analysis method proposed in this invention, which integrates pathological images and omics information, to predict the treatment response after neoadjuvant immunotherapy combined with chemotherapy.

[0103] Obtain multimodal data from non-small cell lung cancer samples. For the first... One non-small cell lung cancer sample, whose pathological imaging data is recorded as follows: Omics features are denoted as .

[0104] in, Indicates the first Pathological imaging data of a non-small cell lung cancer sample may include pre-treatment pathological images, during-treatment pathological images, or post-treatment pathological images. Indicates the first The omics characteristics of a non-small cell lung cancer sample can include features related to immune checkpoint pathways, antigen presentation pathways, T cell activation pathways, inflammatory response pathways, angiogenesis pathways, and immunosuppression pathways.

[0105] The pathological images of non-small cell lung cancer are input into the pathological structure perception branch to identify the tumor area, stroma area, immune infiltration area, necrosis area, vascular-related area and tertiary lymphoid structure area, and to construct a multi-level structural atlas of the tumor microenvironment of non-small cell lung cancer.

[0106] The set of pathological image blocks for non-small cell lung cancer is denoted as:

[0107] in, Indicates the first A collection of pathological image blocks from a non-small cell lung cancer sample; Indicates the first The first sample A pathological image block.

[0108] Accordingly, the effective index of the pathological image block is calculated for each pathological image block. :

[0109] in, Indicates the first In the first non-small cell lung cancer sample, the The effective index of a pathological image block; Indicates the effective organizational ratio; Indicates the clarity index; Indicates staining stability index; Indicates the background proportion; , , , These are the weighting coefficients.

[0110] Constructing a multi-level structural atlas of the non-small cell lung cancer tumor microenvironment based on structured mask images. :

[0111] in, This represents a cell layer atlas of non-small cell lung cancer. This represents a micro-layered map of non-small cell lung cancer. This represents a layer map of a non-small cell lung cancer sample.

[0112] Constructing a functional omics map of non-small cell lung cancer based on immunotherapy-related pathways and molecular functional relationships. For a certain omics functional module Its functional aggregation characterization is denoted as :

[0113] in, Indicates the first Aggregate characterization of omics functional modules in non-small cell lung cancer; This represents the set of omics features belonging to this functional module; Indicates the first In the first non-small cell lung cancer sample, the Standardized values ​​of omics features; This indicates the corresponding weight.

[0114] Furthermore, cross-modal structural correlations are established between pathological structural nodes and omics functional modules. :

[0115] in, Indicates the first The pathological structural node of non-small cell lung cancer and the first Cross-modal structural correlations among omics functional modules; Indicates similarity index; Indicates task-related metrics; Indicates a stable correlation indicator; , , These are the weighting coefficients.

[0116] The multi-level structural map of the non-small cell lung cancer tumor microenvironment was jointly embedded with immunotherapy-related omics features to construct a patho-omics fusion map structure. Based on this, a comprehensive sample embedding representation of non-small cell lung cancer was obtained through the structure-gated heterogeneous graph Transformer module. .

[0117] Treatment response prediction is performed using the comprehensive sample embedding representation of non-small cell lung cancer, and the output results are as follows. It can be represented as:

[0118] in, Indicates the first Treatment response prediction results for one non-small cell lung cancer sample; This represents the weight matrix of the treatment response prediction layer; Indicates the corresponding bias term; Indicates the first Embedded representation of a non-small cell lung cancer sample.

[0119] Based on the output, the predicted probability of the sample belonging to the categories of major pathological remission, non-major pathological remission, complete pathological remission, partial remission, or no remission can be obtained. Simultaneously, the key pathological regions and key omics functional modules most relevant to efficacy assessment can be output to assist in clinical interpretation.

[0120] Application Case 2: Multimodal Fusion Molecular Subtype-Assisted Analysis Based on Breast Cancer Samples: This application example uses breast cancer patients as subjects and applies the tumor microenvironment structure perception and analysis method that integrates pathological images and omics information proposed in this embodiment to assist in the analysis of breast cancer molecular subtypes.

[0121] Obtain multimodal data of breast cancer samples. For the first... One breast cancer sample, whose pathological imaging data is recorded as follows: Omics features are denoted as .in, Indicates the first Pathological imaging data of one breast cancer sample; Indicates the first The omics characteristics of a breast cancer sample can include features related to the HER2 signaling pathway, hormone receptor-related pathways, cell cycle pathways, immune regulation pathways, metabolic reprogramming pathways, and extracellular matrix remodeling pathways.

[0122] Breast cancer pathological images are input into the pathological structure perception branch to identify tumor cell-dense areas, stroma areas, immune infiltration areas, necrotic areas, vascular-related areas, and tertiary lymphoid structures, and to construct a multi-level structural atlas of the breast cancer tumor microenvironment.

[0123] The set of breast cancer pathological image blocks is denoted as:

[0124] in, Indicates the first A collection of pathological image patches from a breast cancer sample; Indicates the first The first sample A pathological image block.

[0125] Accordingly, the effective index of the pathological image block is calculated for each breast cancer pathological image block. :

[0126] in, Indicates the first In the breast cancer sample, the first The effective index of a pathological image block; Indicates the effective organizational ratio; Indicates the clarity index; Indicates staining stability index; Indicates the background proportion; , , , These are the weighting coefficients.

[0127] Constructing a multi-level structural atlas of the breast cancer tumor microenvironment based on structured mask images :

[0128] in, Represents a layered map of breast cancer cells; Represents a micro-layer map of breast cancer; This represents a layer map of breast cancer samples.

[0129] Constructing a breast cancer omics functional atlas based on pathways and functional relationships related to breast cancer molecular subtyping. For a certain omics functional module Its functional aggregation characterization is denoted as :

[0130] in, Indicates the first Aggregate characterization of several breast cancer omics functional modules; This represents the set of omics features belonging to this functional module; Indicates the first In the breast cancer sample, the first Standardized values ​​of omics features; This indicates the corresponding weight.

[0131] The multi-level structural map of the breast cancer tumor microenvironment was jointly embedded with molecular subtyping-related omics features to construct a patho-omics fusion map structure.

[0132] Furthermore, cross-modal structural correlations are established between pathological structural nodes and omics functional modules. :

[0133] in, Indicates the first The first pathological structural node of breast cancer and the first Cross-modal structural correlations among omics functional modules; Indicates similarity index; Indicates task-related metrics; Indicates a stable correlation indicator; , , These are the weighting coefficients. Based on this, the breast cancer sample embedding representation is obtained through the structure-gated heterogeneous graph Transformer module. .

[0134] The molecular subtype-assisted analysis was performed using the breast cancer sample embedding representation, and the output results were... It can be represented as:

[0135] in, Indicates the first Molecular subtype prediction results for one breast cancer sample; This represents the weight matrix of the molecular subtype prediction layer; Indicates the corresponding bias term; Indicates the first Embedded representation of a breast cancer sample.

[0136] Based on the output, the predicted probability of the sample belonging to HER2-positive breast cancer, triple-negative breast cancer, Luminal A, Luminal B, or other molecular subtypes can be obtained. Simultaneously, the key pathological regions and key omics functional modules most relevant to subtype identification can be output to assist in pathological analysis and mechanism research.

[0137] Example 2 Based on the method described in Implementation 1, such as Figure 3 As shown, the purpose of this embodiment is to provide a tumor microenvironment analysis system based on the fusion of pathological imaging and omics, including: The data acquisition module is used to acquire pathological images and omics data of tumor samples, and then perform standardized processing on them. The atlas construction module is used to construct multi-level structural atlases through the pathological structure perception branch and to construct omics functional atlases through the omics functional coding branch. The cross-modal alignment module is used to input two types of maps into the cross-modal structure alignment module. Through a bidirectional attention mechanism, it establishes cross-modal associations between pathological structural nodes and omics functional nodes to obtain alignment features. The fusion graph construction module is used to construct pathology-omics fusion graphs based on pathological structural nodes, omics functional nodes, and alignment features. The graph representation learning module is used to perform graph representation learning on the fused graph input structure-gated heterogeneous graph Transformer module to obtain a comprehensive embedded representation; The analysis output module is used to output tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, or key structure contribution analysis results based on comprehensive embedding representation.

[0138] Example 3 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0139] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium.

[0140] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0141] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0142] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0143] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A tumor microenvironment analysis method based on the fusion of pathological imaging and omics, characterized in that, include: Obtain pathological images and omics data of tumor samples, and perform standardized processing respectively; A multi-level structural atlas is constructed through a pathological structure perception branch, and an omics functional atlas is constructed through an omics functional coding branch. The two atlases are input into a cross-modal structure alignment module, and a cross-modal association between pathological structure nodes and omics functional nodes is established through a bidirectional attention mechanism to obtain alignment features. A pathology-omics fusion graph is constructed based on pathological structural nodes, omics functional nodes, and alignment features; The fused graph is input into the structure-gated heterogeneous graph Transformer module for graph representation learning to obtain a comprehensive embedded representation. Based on comprehensive embedding representation, output results for tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, or key structure contribution analysis.

2. The tumor microenvironment analysis method based on the fusion of pathological imaging and omics as described in claim 1, characterized in that, The pathological structure perception branch sequentially includes an image block input layer, a shallow texture coding layer, a multi-scale feature extraction layer, a cross-scale feature fusion layer, a structural boundary enhancement layer, a regional feature enhancement layer, a regional discrimination layer, and a structured mask output layer.

3. The tumor microenvironment analysis method based on the fusion of pathological imaging and omics as described in claim 2, characterized in that, The multi-scale feature extraction layer includes a small receptive field convolution branch, a medium receptive field convolution branch, a large receptive field dilated convolution branch, and a context pooling branch.

4. The tumor microenvironment analysis method based on the fusion of pathological imaging and omics as described in claim 1, characterized in that, The cross-modal structure alignment module performs pathology-to-omics attention computation and omics-to-pathology attention computation, and introduces cross-modal consistency constraints. The cross-modal structure correlation between pathological structure nodes and omics functional modules is obtained by weighting feature similarity, task relevance, and stable correlation. The pathology-related omics-enhanced representation and the omics-related pathology-enhanced representation are generated by bidirectional attention weights, and the contrast constraint loss is used to narrow the cross-modal representation distance of the same sample and widen the cross-modal representation distance of different samples.

5. The tumor microenvironment analysis method based on the fusion of pathological imaging and omics as described in claim 1, characterized in that, The nodes of the patho-omics fusion graph include pathological structure nodes and omics function nodes, and the edge set includes pathological internal structure edges, omics internal function edges, and patho-omics cross-modal edges.

6. The tumor microenvironment analysis method based on the fusion of pathological imaging and omics as described in claim 1, characterized in that, The structure-gated heterogeneous graph Transformer module sequentially includes a node input embedding layer, a relation type encoding layer, a structure gating layer, a heterogeneous multi-head attention layer, a residual connection and normalization layer, a feedforward mapping layer, a multi-layer stacked update layer, a global pooling layer, and a task output layer.

7. The tumor microenvironment analysis method based on the fusion of pathological imaging and omics as described in claim 1, characterized in that, It also includes training the model using a joint loss function, which includes task loss, structure recognition loss, fusion consistency loss, graph relationship constraint loss, and regularization loss.

8. A tumor microenvironment analysis system based on the fusion of pathological imaging and omics, characterized in that, include: The data acquisition module is used to acquire pathological images and omics data of tumor samples, and then perform standardized processing on them. The atlas construction module is used to construct multi-level structural atlases through the pathological structure perception branch and to construct omics functional atlases through the omics functional coding branch. The cross-modal alignment module is used to input two types of maps into the cross-modal structure alignment module. Through a bidirectional attention mechanism, it establishes cross-modal associations between pathological structural nodes and omics functional nodes to obtain alignment features. The fusion graph construction module is used to construct pathology-omics fusion graphs based on pathological structural nodes, omics functional nodes, and alignment features. The graph representation learning module is used to perform graph representation learning on the fused graph input structure-gated heterogeneous graph Transformer module to obtain a comprehensive embedded representation; The analysis output module is used to output tumor microenvironment structure typing, molecular subtype prediction, treatment response assessment, or key structure contribution analysis results based on comprehensive embedding representation.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the tumor microenvironment analysis method based on the fusion of pathological images and omics as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the tumor microenvironment analysis method based on the fusion of pathological images and omics as described in any one of claims 1-7.

Images