Space gene expression prediction method based on pathological image and multi-modal heterogeneous graph

Abstract

The invention relates to the technical field of gene expression prediction, and discloses a spatial gene expression prediction method based on a pathological image and a multi-modal heterogeneous graph. According to the method, a multi-modal heterogeneous graph containing a target slice edge, a cross-slice edge and a reference slice edge is constructed, and intra-slice space continuity and a cross-slice sharing mode are fused; designing a complementary mask strategy to generate an enhanced view, and enhancing the adaptability of the model to data defects; by means of CNDA bidirectional attention and CNAP multi-head pooling modules, precise cross-slice knowledge migration is achieved; and optimizing the multi-dimensional constraint model through a comprehensive loss function. According to the method, the association rule of tissue morphology and gene expression can be effectively captured, and the space consistency and accuracy of prediction are improved; the robustness and generalization of the model are enhanced, and data of different diseases and platforms are adapted.

Description

Technical Field

[0001] This invention relates to the field of gene expression prediction technology, and in particular to a spatial gene expression prediction method based on pathological images and multimodal heterogeneous maps. Background Technology

[0002] Histological images and spatial transcriptomics data are two key types of biomedical data. Histological images, obtained through tissue sections and microscopic imaging, provide detailed information on cellular and tissue structures. Traditional pathological image analysis relies heavily on visual observation and interpretation by professionals, which is subjective and time-consuming. On the other hand, spatial transcriptomics data provides locational information on gene expression within cells. Spatial transcriptomics methods can determine the expression levels of numerous genes in each cell within a tissue section, providing a detailed understanding of cellular function and state. These pathological images and spatial transcriptomics data are crucial for disease diagnosis, treatment strategy development, and understanding disease progression mechanisms. However, spatial transcriptomics data typically has high dimensionality and complex structures, and generating such data is very expensive, currently hindering large-scale application. In contrast, histological images are inexpensive and widely used in clinical practice. Therefore, using histological images and artificial intelligence to predict cellular gene expression levels from spatial transcriptomics data is a more economical approach, enabling the detection and diagnosis of gene expression in cancer cells.

[0003] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the present invention aims to provide a spatial gene expression prediction method based on pathological images and multimodal heterogeneous maps, which aims to solve the technical problems of large modal differences, high data noise, easy distortion of structural information, and inconsistent cross-modal clustering results in the processing of multi-omics data such as spatial transcriptomics and spatial proteomics.

[0005] The first aspect of this invention provides a spatial gene expression prediction method based on pathological images and multimodal heterogeneity maps, comprising: acquiring a dataset of target tissue slices and reference tissue slices from a publicly available database, the dataset including histological images, spatial transcriptome gene expression data, and location coordinates of spatial transcriptome sequencing points; and segmenting the histological images into image blocks of a preset size according to the location coordinates of the spatial transcriptome sequencing points to obtain a set P of image blocks of the target tissue slices. t Image patch set P with reference tissue slice rThe spatial transcriptome gene expression data were processed to screen for common highly variable genes in all tissue sections. The gene expression values ​​of the screened genes were normalized and converted to a natural logarithmic scale to obtain the gene expression matrix y of the target tissue section. t Gene expression matrix y of reference tissue section r A pre-trained pathology-based model was used to analyze the image patch set P. t and P r Feature extraction is performed to obtain the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r ; Gene expression matrix y based on the target tissue slice t And the image feature matrix zt, and the gene expression matrix y of the reference tissue slice. r and image feature matrix z r Construct a multimodal heterogeneous graph G; based on the multimodal heterogeneous graph G, execute a complementary masking strategy to generate two semantically complementary and topologically identical enhanced graph views G1 and G2; use a graph encoder with shared parameters to encode the enhanced graph views G1 and G2 respectively, the graph encoder including a GraphSAGE module, a cross-node bidirectional attention CNDA module, and a cross-node attention pooling CNAP module; construct a comprehensive loss function L. total The graph encoder is trained and optimized until the model converges to obtain an optimized model; the histological pathological image to be predicted is input into the optimized model, and the corresponding spatial gene expression prediction result is output.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the gene expression matrix y based on the target tissue slice is used. t And the image feature matrix zt, and the gene expression matrix y of the reference tissue slice. r and image feature matrix z r Constructing a multimodal heterogeneous graph G includes the following steps: Define a set of nodes V, where the set of nodes V is the target node V. t With reference node V r The union of the target node V t For each of the n spatial transcriptome sequencing points corresponding to the target tissue slice, the feature of each target node is a high-dimensional visual embedding vector of the target image patch extracted by a pre-trained UNI model. All the features of the target nodes constitute a target node feature matrix z with a dimension of n×1024. t The reference node V rFor each of the m spatial transcriptome sequencing points corresponding to a reference tissue slice, the feature of each reference node is the concatenation vector of the UNI visual embedding vector of the reference image patch and the normalized highly variable gene expression vector of that sequencing point. All reference node features constitute a reference node feature matrix h with dimensions m×(1024+1000). r 1000 represents the number of highly variable genes; define an edge set E, where the edge set E is the target slice edge E. TS Cross-slice edge E CS and reference slice edge E RS The union of these three types of edges involves: calculating the Euclidean distance between the sequencing point corresponding to each target node within the target slice and other target sequencing points; selecting the Q nearest neighboring sequencing points for each target node as neighbors and connecting the target node with its neighboring nodes to form a target slice edge set; calculating the cosine similarity between the visual embedding vector of each target node and the visual embedding vectors of all reference nodes; selecting the K reference nodes with the highest similarity for each target node and connecting the target node with the K reference nodes to form a cross-slice edge set; calculating the cosine similarity between the multimodal feature vector of each reference node and the multimodal feature vectors of other reference nodes; selecting the K other reference nodes with the highest similarity for each reference node and connecting the reference node with the K other reference nodes to form a reference slice edge set; and combining the node set V with the edge set E to construct a multimodal heterogeneous graph G.

[0007] Optionally, in a second implementation of the first aspect of the present invention, a complementary masking strategy is executed based on the multimodal heterogeneous graph G to generate two semantically complementary and topologically identical enhanced graph views G1 and G2, including the step of generating two binary mask matrices of dimension n×1024 for the feature matrix of the target node. and ; Through element-wise multiplication, the feature matrix z of the target node is transformed. t respectively with M t 1 M t 2 Multiplying them yields the two target node enhancement feature matrices z. t 1 z t 2 ; the reference node feature matrix h r respectively with M r 1 M r 2 Multiplying them together yields the two reference node enhancement feature matrices h. r 1 h r 2Keeping the node set V and edge set E of the multimodal heterogeneous graph G unchanged (with the same topology), only update the node feature matrix, and change z t 1 with h r 1 Row concatenation is used as the node feature matrix to form the first enhanced graph view G1; z t 2 with h r 2 Row concatenation serves as the node feature matrix, forming a second enhanced graph view G2. The two views are semantically complementary and topologically consistent.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the graph encoder includes a GraphSAGE module, a cross-node bidirectional attention (CNDA) module, and a cross-node attention pooling (CNAP) module. A graph encoder with shared parameters is used to encode the enhanced graph views G1 and G2 respectively, including the steps of: encoding the target slice edges E in the enhanced graph views respectively through the GraphSAGE module. TS and reference slice edge E RS Encoding is performed to obtain the coded features within the slice of the target node and the coded features within the slice of the reference node; bidirectional attention interaction between the target node and the reference node is established through the cross-node bidirectional attention CNDA module to update the cross-slice features of the target node and the reference node; the cross-node attention pooling CNAP module is used to aggregate the reference node features with multi-head attention, and combined with residual connections to obtain the final representation of the target node, and the gene expression prediction matrix of the target slice in the graph view is enhanced by outputting the MLP.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the GraphSAGE module is used to process the target slice edges E in the enhanced graph view. TS and reference slice edge E RS Encoding is performed to obtain the encoded features within the target node slice and the encoded features within the reference node slice, including the following steps: for each target slice edge E in the augmented graph view TS The mean aggregation method is used to average the enhanced features of each target node with the enhanced features of its neighboring target nodes to obtain aggregated features. These aggregated features are then nonlinearly transformed using a fully connected layer with ReLU activation to obtain the in-slice encoding features of the target node. For each reference slice edge E in the augmented graph view... RS Similarly, the reference node enhancement features are processed using the mean aggregation method, and then the encoded features within the reference node slice are obtained by transforming the fully connected layer with the ReLU activation function.

[0010] Optionally, in the fifth implementation of the first aspect of the present invention, a bidirectional attention interaction between the target node and the reference node is established through a cross-node bidirectional attention (CNDA) module to update the target node cross-slice features and the reference node cross-slice features, including the following steps: using the target node's slice-in-slice encoded features as a query vector and the reference node's slice-in-slice encoded features as a key vector and a value vector, projecting them onto a 256-dimensional space using a learnable weight matrix, calculating the scaled dot product attention weight of the query and key, and then multiplying the weight by the value vector to obtain the target node cross-slice attention features; using the reference node's slice-in-slice encoded features as a query vector and the target node's slice-in-slice encoded features as a key vector and a value vector, similarly projecting them, calculating the attention weight, and aggregating them to obtain the reference node cross-slice attention features; using a scatter mean operation, averaging the target node's slice-in-slice encoded features and the target node cross-slice attention features to update the target node cross-slice features; averaging the reference node's slice-in-slice encoded features and the reference node cross-slice attention features to update the reference node cross-slice features.

[0011] Optionally, in the sixth implementation of the first aspect of the present invention, the cross-node attention pooling (CNAP) module uses multi-head attention to aggregate the features of the reference node, combines residual connections to obtain the final representation of the target node, and outputs the gene expression prediction matrix of the target slice in the enhanced graph view through an MLP. The steps include: the cross-node attention pooling (CNAP) module is configured with h attention heads, each attention head projects the cross-slice features of the target node and the cross-slice features of the reference node into 32 dimensions, calculates the attention weight of each head and aggregates the features; the aggregated features of the h attention heads are concatenated and converted into a unified dimension through a learnable output projection matrix; then residual connections are performed with the cross-slice features of the target node, inputting into an MLP with two hidden layers, and outputting the gene expression prediction matrix of the target slice in the enhanced graph view.

[0012] A second aspect of the present invention provides a spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps, comprising: a dataset acquisition module for acquiring datasets of target tissue slices and reference tissue slices from a publicly available database, the datasets including histological images, spatial transcriptome gene expression data, and location coordinates of spatial transcriptome sequencing points; and a segmentation module for segmenting the histological images into image blocks of a preset size according to the location coordinates of the spatial transcriptome sequencing points, thereby obtaining a set P of image blocks of the target tissue slices. t Image patch set P with reference tissue slice r The gene expression data processing module is used to process the spatial transcriptome gene expression data, screen for common highly variable genes in all tissue sections, normalize the screened gene expression values ​​and transform them to the natural logarithmic scale to obtain the gene expression matrix y of the target tissue section.t Gene expression matrix y of reference tissue section r The feature extraction module is used to extract features from the image patch set P using a pre-trained pathology-based model. t and P r Feature extraction is performed to obtain the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r ; Multimodal heterogeneity map construction module, used for constructing gene expression matrices based on the target tissue slices. t And the image feature matrix zt, and the gene expression matrix y of the reference tissue slice. r and image feature matrix z r A multimodal heterogeneous graph G is constructed; a masking module is used to execute a complementary masking strategy based on the multimodal heterogeneous graph G to generate two semantically complementary and topologically identical enhanced graph views G1 and G2; an encoding module is used to encode the enhanced graph views G1 and G2 respectively using a graph encoder with shared parameters, the graph encoder including a GraphSAGE module, a cross-node bidirectional attention CNDA module, and a cross-node attention pooling CNAP module; a training module is used to construct a comprehensive loss function L. total The graph encoder is trained and optimized until the model converges to obtain an optimized model; the output module is used to input the histological pathological image to be predicted into the optimized model and output the corresponding spatial gene expression prediction results.

[0013] A third aspect of the present invention provides a spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the computer-readable instructions in the memory to cause the spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps to perform the various steps of the spatial gene expression prediction method based on pathological images and multimodal heterogeneity maps as described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-readable instructions that, when executed on a computer, cause the computer to perform the steps of the spatial gene expression prediction method based on pathological images and multimodal heterogeneous maps as described above.

[0015] Beneficial Effects: This invention achieves technological breakthroughs from multiple dimensions, including feature representation, robustness, knowledge transfer, and clinical adaptability, through core technologies such as multimodal heterogeneous graph construction, complementary masking strategies, and bidirectional attention pooling. The technological effects are significant and the logic is closed-loop. First, the structured design of the multimodal heterogeneous graph enables comprehensive capture of information within and between slices. By connecting neighboring sequencing points based on spatial Euclidean distance along the target slice edges, the spatial continuity of tissue morphology and gene expression is accurately restored, making the feature associations of adjacent regions more consistent with the natural distribution patterns of biological tissues. Cross-slice edges connect target and reference nodes based on visual feature similarity, establishing a knowledge transfer channel for gene expression in morphologically similar regions, allowing the target slice to fully leverage the "morphology-gene" association experience of the reference slice. Reference slice edges construct internal connections based on multimodal feature similarity, strengthening the consistency of global expression patterns in the reference library and providing stable and reliable knowledge support for cross-slice transfer. The synergistic effect of these three types of edges allows the model to capture both local spatial details and globally shared patterns, making gene expression prediction more in line with biomedical principles.

[0016] Secondly, the complementary masking strategy effectively enhances the model's adaptability to defects in real data. Addressing common issues in spatial transcriptome data such as gene deletions and sequencing noise, the model generates semantically complementary and topologically identical augmented views, simulating real-world scenarios of feature loss and data perturbation. This forces the model to move away from its dependence on a single feature dimension and learn more stable and universal representations. This design requires no additional data augmentation; feature masking alone strengthens the model's robustness, ensuring stable prediction results even in non-ideal data environments and avoiding feature distortion caused by data quality fluctuations.

[0017] Third, the CNDA and CNAP modules achieve accurate and efficient cross-slice knowledge transfer. The bidirectional attention mechanism of the CNDA module allows the target node to selectively filter useful information from the reference node, while the reference node can also adapt to the morphological features of the target slice, avoiding information imbalance and redundant interference in one-way knowledge transfer. The multi-head attention pooling design of the CNAP module aggregates reference features from multiple dimensions and retains the morphological information of the target node itself by combining residual connections. This fully utilizes the auxiliary value of the reference library without drowning out the specific features of the target slice, making the representation of the target node both universal and personalized, and greatly improving the accuracy of gene expression prediction.

[0018] Fourth, the multi-dimensional constraints of the comprehensive loss function ensure the overall reliability of the prediction results. The mean squared error loss ensures the numerical consistency between the predicted and the true values, the Pearson correlation coefficient loss strengthens the correlation of expression patterns, and the contrastive loss maintains the stability of the representation under different views. The synergistic optimization of these three factors allows the model to focus on both quantitative numerical accuracy and qualitative expression patterns, avoiding prediction bias caused by a single loss function. This ensures that the prediction results meet both numerical requirements and conform to the gene expression distribution patterns of biological tissues. Attached Figure Description

[0019] Figure 1 A flowchart of a spatial gene expression prediction method based on pathological images and multimodal heterogeneous maps provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the principle of the spatial gene expression prediction method based on pathological images and multimodal heterogeneous maps of the present invention.

[0021] Figure 3 To visualize the expression pattern of the marker gene SPINK5 on the cSCC dataset.

[0022] Figure 4 To visualize the expression pattern of the marker gene FASN on the HER2+ dataset.

[0023] Figure 5 To visualize the expression pattern of the marker gene CD24 on the Alex dataset.

[0024] Figure 6 To visualize the expression pattern of the marker gene BAMBI on the VisiumBC dataset.

[0025] Figure 7 This is a schematic diagram of the spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps provided in an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of the spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps provided in an embodiment of the present invention. Detailed Implementation

[0027] This invention provides a spatial gene expression prediction method based on pathological images and multimodal heterogeneity maps. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Most studies have not adequately addressed the inherent characteristics of spatial transcriptome (ST) data. On the one hand, ST data often suffers from sparsity and noise—gene expression may be absent or extremely low in certain spatial locations, and data quality is susceptible to experimental resolution, sample preparation, and batch effects. On the other hand, current methods primarily focus on modeling the spatial structure of individual tissue slices, neglecting the potential shared expression patterns between different slices. In fact, individuals with the same tissue type or disease often exhibit morphological and spatial gene expression similarities, while individual differences and disease progression introduce heterogeneity between samples. This diversity makes it difficult for models to learn generalizable representations from a single slice. Therefore, effectively integrating shared information between slices while simultaneously considering individual-specific differences is crucial for improving expression prediction performance and enhancing model generalization ability.

[0029] Based on this, the present invention provides a spatial gene expression prediction method based on pathological images and multimodal heterogeneity maps, such as... Figure 1 As shown, it includes the following steps: S10. Obtain a dataset of target tissue slices and reference tissue slices from a publicly available database. The dataset includes histological images, spatial transcriptome gene expression data, and the location coordinates of spatial transcriptome sequencing points. Specifically, this embodiment obtains target tissue slice datasets and reference tissue slice datasets from publicly available databases to meet the needs of model training and prediction. Both datasets must contain integrated information including histological pathological images, spatial transcriptomic gene expression data, and sequencing point location coordinates to ensure the feasibility of subsequent multimodal heterogeneous map construction and cross-slice knowledge transfer. Public databases specifically refer to authoritative public databases in the biomedical field, including but not limited to TCGA (The Cancer Genome Atlas), GEO (Gene Expression Omnibus), and SpatialDB.

[0030] In this embodiment, the target tissue slice refers to a tissue sample slice of the gene expression to be predicted, or a slice used as the object to be predicted during training (such as a sample slice in the HER2+ breast cancer dataset); the reference tissue slice refers to a slice that has complete spatial transcriptome gene expression data and belongs to the same tissue type or disease category as the target slice. For example, if the target slice is HER2+ breast cancer, the reference slice should include other HER2+ breast cancer, TNBC subtype breast cancer, and other similar tissue slices to provide reference information on the relationship between visual and gene expression.

[0031] S20. Based on the location coordinates of the spatial transcriptome sequencing points, the histological pathological image is segmented into image blocks of a preset size to obtain a set of image blocks P of the target tissue slice. tImage patch set P with reference tissue slice r ; In this embodiment, the histological pathology image is segmented into image blocks of uniform size according to the sequencing point location, realizing a one-to-one correspondence between sequencing points and image blocks, providing standardized input for subsequent visual feature extraction. For example, the preset size is preferably 224×224 pixels, which facilitates the use of the pre-trained pathological basic model UNI for subsequent feature extraction. The input specification of this model is 224×224 pixels. The uniform size allows for direct reuse of the pre-trained weights of UNI, avoiding feature extraction failure due to size mismatch. At the same time, the 224×224 pixel image block, combined with the physical resolution of the histological pathology image (e.g., 0.25μm / pixel), corresponds to an actual tissue area size of 56μm×56μm, which can completely contain the cell cluster corresponding to one sequencing point. It can capture cell morphology (e.g., cell nucleus size, staining intensity) and cover local tissue structures (e.g., tumor cell clusters, stromal cell distribution).

[0032] In this embodiment, the image segmentation process must strictly follow four steps: coordinate mapping, center alignment, edge filling, and quality verification to ensure that each image block corresponds precisely to the sequencing point. Specifically, firstly, the physical coordinates of the sequencing point and the physical resolution of the histological pathological image are obtained. The physical coordinates are converted into pixel coordinates, and the converted pixel coordinates are marked on the pathological image using an image annotation tool (such as QuPath). It is confirmed that the marked points are all located within the tissue region corresponding to the sequencing point. If there is a deviation (such as the marked points deviating from the tissue region), the physical resolution needs to be recalibrated. For the image of the target tissue slice, a 224×224 pixel square region is cropped from the image block of each sequencing point i, centered on its pixel coordinates, to ensure that the center of the image block is perfectly aligned with the coordinates of the sequencing point. When the cropped area exceeds the image boundary, a mirror filling strategy is used for filling. After segmentation, the image patch set P is processed. t (Target slice) and P r (Reference slice) Perform quality verification and remove image blocks with a blank area ratio >50% as well as image blocks that are blurry or unevenly colored.

[0033] S30. Process the spatial transcriptome gene expression data, screen for common highly variable genes in all tissue sections, normalize the screened gene expression values ​​and transform them to the natural logarithmic scale to obtain the gene expression matrix y of the target tissue section. t Gene expression matrix y of reference tissue section r ; In this embodiment, to address the three major challenges of high dimensionality, high sparsity, and sequencing depth differences in spatial transcriptome gene expression data, a three-step process—screening highly variable genes, normalization, and logarithmic transformation—is employed to obtain a gene expression matrix y suitable for model training. t (Target slice) and y r (Reference slice). The processing logic follows the principles of preserving key information, eliminating technical bias, and optimizing data distribution. Specifically, it involves screening highly variable genes: retaining 1000 genes with significant expression differences across different sequencing sites (these genes are more likely to be related to tissue morphology and cell function, such as the tumor marker HER2), and removing genes with constant or extremely low expression levels (such as non-functional variations in housekeeping genes) to reduce data dimensionality; normalization: eliminating expression value bias caused by differences in sequencing depth (total UMI count) across different sequencing sites (e.g., sequencing site A has a total UMI of 1000 and gene g expresses 10; sequencing site B has a total UMI of 2000 and gene g expresses 20, the actual expression levels are the same, but the original values ​​are different); and logarithmic transformation: converting the right-skewed expression data (most genes are low-expressed, a few genes are high-expressed) to a near-normal distribution to reduce the interference of extreme values ​​on model regression predictions.

[0034] S40. Using a pre-trained pathology-based model on the image patch set P t and P r Feature extraction is performed to obtain the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r ; Specifically, UNI (Underlying Pathology Model) is a pre-trained model (based on the ViT architecture) designed specifically for biomedical pathology images. It has been pre-trained on millions of clinical pathology images (such as HE staining and immunohistochemical staining images) and has learned general features of tissue morphology (such as cell arrangement patterns, nuclear morphology, and stroma structure), making it better able to capture subtle differences in pathological images than general image models (such as ResNet50 pre-trained on ImageNet). This embodiment uses the pre-trained underlying pathology model on the image patch set P. t and P r Feature extraction yields the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r .

[0035] S50, Gene expression matrix y based on the target tissue slice t and image feature matrix z t and the gene expression matrix y of the reference tissue section. r and image feature matrix z r Construct a multimodal heterogeneous graph G; The core of this step is to construct a multimodal heterogeneous graph G=(V,E) containing target nodes, reference nodes, and three types of edges, where V is the set of nodes (integrating visual and gene expression multimodal features), and E is the set of edges (capturing spatial continuity within slices and shared patterns between slices). This embodiment addresses the shortcomings of existing technologies that only model the spatial structure of a single slice and ignore cross-slice shared information. By fusing the multimodal features of target nodes and reference nodes and the topological connections of the three types of edges, it achieves dual modeling of spatial association within slices and knowledge transfer between slices. The constructed multimodal heterogeneous graph G provides structured multimodal information support for subsequent gene expression prediction.

[0036] In this embodiment, the core component of the multimodal heterogeneous graph includes: a node set V = V t ∪Vr, where V t For the target node (containing only visual features, gene expression needs to be predicted), V r Reference nodes (containing visual and gene expression multimodal features, providing known expression information); Specifically, the target node V t For each of the n spatial transcriptome sequencing points corresponding to the target tissue slice, the feature of each target node is a high-dimensional visual embedding vector of the target image patch extracted by a pre-trained UNI model. All the features of the target nodes constitute a target node feature matrix z with a dimension of n×1024. t ; Furthermore, the target node V t For each valid sequencing point of the target tissue slice (after screening in step S20), each target node contains only visual features (due to the gene expression y of the target slice). t (This is the object to be predicted and cannot be used in advance for node feature construction). Specific construction steps: Number of nodes: The number of target nodes is consistent with the number of effective sequencing points in the target slice, i.e., |V t |=n (n is the number of valid image blocks of the target tissue slice in step S20, corresponding to the number of valid sequencing points), each target node is denoted as v t (i) (i=1,2,…,n, where i is the sequencing point index); Node characteristics: Target node v t (i) The feature vector is the visual embedding z extracted in step S40. t (i) The dimension is d=1024 (the UNI model output embedding dimension is explicitly defined), i.e., v t (i) .feat=z t (i) ∈R1024, that is, all target node features constitute a target node feature matrix zt with dimension n×1024; Node Identifier: Each target node carries a unique identifier, including slice ID, sequencing point ID, and coordinate information (e.g., HER2+Sample1_Spot5 (100μm, 200μm)), which is used for node matching and result tracing during subsequent edge construction.

[0037] The reference node V r For each of the m spatial transcriptome sequencing points corresponding to a reference tissue slice, the feature of each reference node is the concatenation vector of the UNI visual embedding vector of the reference image patch and the normalized highly variable gene expression vector of that sequencing point. All reference node features constitute a reference node feature matrix h with dimensions m×(1024+1000). r 1000 represents the number of highly variable genes; Furthermore, the reference nodes correspond to valid sequencing points on the reference slices (after screening in step S20). Each reference node integrates visual features and gene expression features. Specific construction steps are as follows: Number of nodes: The number of reference nodes is consistent with the number of effective sequencing points of the reference tissue slice, i.e., |Vr|=m (m is the number of effective image blocks of the reference tissue slice in step S20), and each reference node is denoted as v. r (j) (j=1,2,…,m, where j is the sequencing point index); Multimodal feature stitching: reference node VR (j) The feature vector is the visual embedding z from step S40. r (j) The gene expression value y in step S30 r (j) splicing vector h r (j) The formula is: , where: z r (j) ∈R1024 (visual embedding, d=1024); y r (j) ∈R1000 (gene expression vector, M=1000, number of highly variable genes selected in step S30); the concatenated dimension is 1024+1000=2024, ensuring that it contains both tissue morphology information and known gene expression information. Node Identifier: The reference node identifier includes the reference tissue slice ID, sequencing point ID, coordinate information, and gene expression summary (e.g., cSCC_Sample3_Spot12 (150μm,300μm) [HER2:2.3, FASN:4.1]). The gene expression summary contains the expression values ​​of key genes, which facilitates the initial screening during subsequent cross-slice edge construction.

[0038] target node Vt With reference node V r The nodes are integrated into a total node set V. The two types of nodes are distinguished by node type labels (target nodes are labeled as target, and reference nodes are labeled as reference), which facilitates the subsequent graph encoder (such as the CNDA module) to process different types of nodes in a targeted manner. The node set is stored in the form of a node list, and each node entry contains an identifier, type, feature vector, and coordinates.

[0039] In this embodiment, the core component of the multimodal heterogeneous graph also includes: the edge set E = ∪ ∪ ,in, (Target slice edge) Captures the spatial continuity of sequencing points within the target slice. (Across slice edges) to achieve knowledge transfer between target and reference tissue slices. (Reference slice edge) Enhances the global expression pattern within the reference slice.

[0040] Specifically, the Euclidean distance between the sequencing point corresponding to each target node within the target slice and other target sequencing points is calculated. For each target node, the Q nearest neighbor sequencing points are selected as its neighbors, and the target node is connected to its neighbor nodes to form the target slice edge set. ; Calculate the cosine similarity between the visual embedding vector of each target node and the visual embedding vectors of all reference nodes. Select the K reference nodes with the highest similarity for each target node, and connect the target node to the K reference nodes to form a set of cross-slice edges. ; Calculate the cosine similarity between the multimodal feature vector of each reference node and the multimodal feature vectors of other reference nodes. Select the K other reference nodes with the highest similarity for each reference node, and connect the reference node to the K other reference nodes to form a set of reference slice edges. .

[0041] Furthermore, this embodiment is based on spatial distance ( ) or feature similarity ( , Establish node connections to ensure that the edge topology accurately reflects the relationships (spatial proximity or similar features) between nodes. The following details the construction process for each type of edge, combining formulas and specific implementation parameters (number of neighbors Q=5, number of reference nodes K=7): The target slice edge connects spatially adjacent target nodes within the target tissue slice. The core design principle of the target slice edge is that adjacent sequencing points are usually located in similar morphological regions and exhibit related gene expression patterns. Local associations within the target slice are modeled through spatial distance. Specific construction steps are as follows: To capture the spatial continuity within the target tissue, this invention connects neighboring sequencing points of the target slice based on Euclidean distance, that is, for any two target nodes v t (i) With v t (j) (i≠j), calculate the Euclidean distance d based on the physical coordinates of its sequencing points. ij , Where s t (i) =(x t (i) y t (i) Let be the physical coordinates of the target node i. It is the L2 norm (Euclidean distance).

[0042] For each target node v t (i) The formula for selecting the Q nearest target nodes as neighbors is as follows: in, This means selecting the Q nearest neighbor sequencing sites. For example, when Q=5, it can cover a sufficient local spatial region (capturing the morphological similarity of adjacent sequencing sites) while avoiding the increase in computational complexity caused by too many neighbors.

[0043] At this point, the target slice edge The set is defined as the set of undirected edges between the target node and its neighboring nodes, that is: Because spatial proximity relationships are symmetrical (node ​​i's neighbor k is also considered i as a neighbor), the edges are undirected and there is no need to distinguish the direction.

[0044] In this implementation, across slice edges Connecting target nodes with similar visual features to reference nodes is based on the morphological and gene expression similarities between slices of the same tissue type or disease. Through visual feature similarity matching, target nodes can obtain known gene expression information of morphologically similar regions from reference nodes, achieving cross-slice knowledge transfer. Specific construction steps are as follows: Image embedding for each target tissue slice This invention selects the K most similar reference slices for image embedding based on cosine similarity: in, Indicates the relationship with the first The set of K most similar reference sequencing sites for each target sequencing site. For example, when K=7, it can cover a sufficient number of similar reference regions (avoiding expression bias of a single reference node) and avoid information redundancy caused by too many reference nodes.

[0045] This embodiment defines the cross-slice edge set accordingly. Let be the set of directed edges between all target nodes and their matching reference nodes, i.e. The directionality of the edge needs to be clearly defined, pointing from the target node to the reference node. Subsequently, the CNDA module will perform attention calculations based on this direction to obtain information from the reference node.

[0046] In this embodiment, the reference slice edge set E RS Connecting reference nodes with similar multimodal features is based on the core design principle that there is a globally consistent morphological-expression association pattern among sequencing points within the reference slice. Through multimodal feature (visual + gene expression) similarity modeling, reference nodes can share global information, strengthening the consistency of their multimodal representations and providing a more reliable reference for cross-slice knowledge transfer. Specific construction steps are as follows: Embedded for each reference image patch In this embodiment, the top K most similar nodes of the same type are found based on cosine similarity: in, Let E be the set of neighboring reference node indices of node j. Based on this, this embodiment defines the reference slice edge E. RS Let be the set of undirected edges between all reference nodes and their neighboring nodes, i.e. .

[0047] Three types of edges E TS E CS E RS The edges are integrated into a total set E, distinguished by edge type label (E). TS Tags: TS, E CS Tags: CS, E RS The label RS will be used to selectively process information transmission for different types of edges in subsequent graph encoders (such as GraphSAGE and CNDA).

[0048] Finally, the node set V and the edge set E are integrated into a complete multimodal heterogeneous graph G=(V,E).

[0049] This embodiment uses E CS By connecting the target and reference nodes, the visual-gene expression association information of the reference slice is introduced into the target slice modeling, overcoming the limitations of single-slice modeling and enabling cross-slice knowledge transfer; through E... TS E captures local spatial relationships within the target slice. RS The global expression pattern within the reference slice is enhanced, and the combination of these two aspects allows node features to simultaneously encompass local details and global context, improving the robustness of feature representation. In this embodiment, the visual + gene expression multimodal features of the reference node are E. CSThe knowledge transfer of edges provides both morphological and representational basis, avoiding the expression prediction bias caused by relying solely on visual similarity; the topological structure (nodes + edges + type labels) of the multimodal heterogeneous graph in this embodiment is perfectly adapted to subsequent graph encoders such as GraphSAGE and CNDA, enabling information transmission to specifically handle different types of nodes and edges. Compared with unstructured feature stitching methods, the model training efficiency is effectively improved.

[0050] S60. Based on the multimodal heterogeneous graph G, execute a complementary masking strategy to generate two semantically complementary and topologically identical enhanced graph views G1 and G2. Specifically, such as Figure 2 As shown, the core of this step is the multimodal heterogeneous graph G constructed based on step S50. Two semantically complementary and topologically identical augmented graph views G1 and G2 are generated through a complementary masking strategy. The design goal is to address the common problems of feature sparsity, noise interference, and expression loss in spatial transcriptome (ST) data. By simulating augmented views with missing features, the model is forced to learn stable and consistent multimodal representations, thereby improving the model's robustness to noise in real data.

[0051] The core logic of complementary masking includes: topological invariance: the node set V and edge set E of the enhanced graph views G1 and G2 are completely consistent with the multimodal heterogeneous graph G (only node features are masked, and the topological structure remains unchanged), ensuring the stability of the topological relationship for information transmission; semantic complementarity: the node feature masks of the two views are complementary in dimension (i.e., the feature dimension of one view is masked, while the other view is not masked), ensuring that the model can learn complete information from different feature subsets; typological specificity: different masking strategies are designed for target nodes (visual features only) and reference nodes (multimodal features) to avoid the destruction of features of different types of nodes by a single mask.

[0052] For each enhanced graph view This invention constructs a binary mask matrix. and The mask ratios are determined by the hyperparameters. and The control determines the proportion of features to be masked. The two views are complementary in the feature dimension and satisfy the following condition: ; The target node feature matrix z is obtained through element-wise multiplication. t respectively with M t 1 M t 2 Multiplying them yields the two target node enhancement feature matrices z. t 1 z t 2 ; the reference node feature matrix hr respectively with M r 1 M r 2 Multiplying them together yields the two reference node enhancement feature matrices h. r 1 h r 2 Therefore, the formula for the enhanced feature matrix is: ; Keeping the node set V and edge set E of the multimodal heterogeneous graph G unchanged (with the same topology), only update the node feature matrix, and change z t 1 with h r 1 Row concatenation is used as the node feature matrix to form the first enhanced graph view G1; z t 2 with h r 2 Row concatenation is used as the node feature matrix to form a second enhanced graph view G2. The two views are semantically complementary and topologically consistent. The corresponding enhanced graph view formula is: Enhanced view G1=(V,E,Z) 1 ), where Z 1 =[z t 1 h r 1 ]∈R(n+m)×2024 (target node enhancement features are zero-padded to 2024 dimensions); Enhanced graph view view G2=(V,E,Z) 2 ), where Z 2 =[z t 2 h r 2 ]∈R(n+m)×2024.

[0053] This embodiment simulates feature loss and noise perturbation in ST data through a masking strategy, forcing the model to learn stable representations independent of specific feature dimensions; complementary masks of the two views ensure that the model can learn complete information from different feature subsets (such as views). Figure 1 A certain dimension of the visual features of the mask, view Figure 2 Retaining this dimension allows the model to learn the core information of this dimension by combining both approaches. This design enhances the representational power of node features. The enhanced view provides positive sample pairs (enhanced features of the same node in both views) for the contrastive loss in the subsequent step S80, enabling the model to optimize the representation through self-supervised learning without additional labeled data. In this embodiment, the enhanced graph view only masks node features, and the topology remains the same as the original. Figure 1This ensures that the information transmission logic of the subsequent graph encoder remains unchanged, avoiding fluctuations in model training caused by topology changes.

[0054] S70. The enhanced graph views G1 and G2 are encoded using a graph encoder with shared parameters. The graph encoder includes a GraphSAGE module, a cross-node bidirectional attention CNDA module, and a cross-node attention pooling CNAP module. This embodiment employs a graph encoder with shared parameters to encode the enhanced graph views G1 and G2, outputting node-level and graph-level representations. The core design principle of the graph encoder is to address the shortcomings of existing graph neural networks, which can only process isomorphic graphs, ignore node type differences, and have single information aggregation. Through a three-level structure of GraphSAGE neighborhood aggregation + cross-node bidirectional attention (CNDA) + cross-node attention pooling (CNAP), it achieves progressive encoding of multi-scale spatial information fusion, cross-modal node interaction, and global context integration, providing representations rich in spatial correlation and multimodal features for subsequent gene expression prediction.

[0055] Two enhanced graph views, G1 and G2, are encoded using the same set of graph encoder parameters (parameter sharing), ensuring the comparability of representations of the same node in different views (laying the foundation for the contrast loss calculation in step S80). Figure 2 As shown, the functional division of the three-level structure of the graph encoder is as follows: GraphSAGE module: The GraphSAGE module enhances the edge E of the target slice in the graph view. TS and reference slice edge E RS Encoding is performed to obtain the encoded features within the target node slice and the encoded features within the reference node slice. Specifically, for each target slice edge E in the augmented graph view... TS The mean aggregation method is used to average the enhanced features of each target node with the enhanced features of its neighboring target nodes to obtain aggregated features. Then, a nonlinear transformation is applied to the aggregated features through a fully connected layer with a ReLU activation function to obtain the encoded features within the slice of the target node. The expression is as follows: For each enhanced graph view, the reference slice edge E RS Similarly, the reference node enhancement features are processed using mean aggregation, and then transformed through a fully connected layer containing a ReLU activation function to obtain the encoded features within the reference node slice, the expression of which is: .

[0056] Cross-Node Bidirectional Attention (CNDA) Module: This module establishes bidirectional attention interaction between the target node and the reference node, updating the cross-slice features of both the target and reference nodes. Specifically, the target node's slice-encoded features are used as the query vector, and the reference node's slice-encoded features are used as the key and value vectors. These are projected onto a 256-dimensional space using a learnable weight matrix. The scaled dot product attention weights of the query and key are calculated, and then multiplied by the value vectors to obtain the target node's cross-slice attention features. Similarly, the reference node's slice-encoded features are used as the query vector, and the target node's slice-encoded features are used as the key and value vectors. After projection, attention weights are calculated and aggregated to obtain the reference node's cross-slice attention features. Furthermore, each target node will follow the reference edge set. The corresponding reference node performs a cross-node dual attention operation. It's important to note that the feature vector of each reference node is defined as: This approach combines visual features with gene expression features. Through dynamically calculated attention weights, the target node can selectively extract embedded morphological and gene expression knowledge from the reference node. ; ; ,in, This is a learnable weight matrix.

[0057] Symmetrically, the reference node also performs an attention operation on the visual embedding of the target node, thereby updating its own features: ; ; ,in, It is also a learnable weight matrix.

[0058] Finally, a scatter mean operation is used to average the encoded features within each target node slice with the cross-slice attention features of the target node, updating the target node's cross-slice features as follows: The average of the encoded features within each reference node slice and the cross-slice attention features of the reference node is used to update the cross-slice features of the reference node. .

[0059] Furthermore, this embodiment introduces a self-supervised contrastive learning objective that encourages the embeddings of the same node from two views to be as close as possible in the semantic space. The invention employs an asymmetric contrastive design: one view does not update its gradient when passing through the encoder, serving as a stabilizing objective; the other view passes through the same encoder and enables backpropagation. Subsequently, the invention minimizes the cosine distance between the corresponding node embeddings of the two views, thereby strengthening cross-view learning. Figure 1 Desire: ,in This represents the total number of nodes participating in the loss comparison, including the target node and the reference node.

[0060] The Cross-Node Attention Pooling (CNAP) module employs multi-head attention to aggregate reference node features, combines them with residual connections to obtain the final representation of the target node, and outputs the gene expression prediction matrix of the target slice in the enhanced graph view through an MLP. Specifically, the CNAP module has h attention heads, each projecting the cross-slice features of the target node and the cross-slice features of the reference node into a 32-dimensional plane, calculating the attention weight of each head and aggregating the features; concatenating the aggregated features of the h attention heads, and converting them into a unified dimension through a learnable output projection matrix; then performing a residual connection with the cross-slice features of the target node, inputting it into an MLP with two hidden layers, and outputting the gene expression prediction matrix of the target slice in the enhanced graph view.

[0061] Furthermore, the cross-node attention pooling (CNAP) module can adaptively integrate auxiliary information from reference nodes based on the contextual semantics of each target node. Formally, the CNAP module is defined as:

[0062]

[0063]

[0064] in, For the number of attention heads, Indicates the first One point of attention, For the first Learnable projection matrix for each attention head This is for outputting the projection matrix.

[0065] To more effectively model the contribution of reference nodes to gene expression prediction of target nodes and guide the model to rely more on the representation of the target nodes themselves when integrating features, this invention introduces residual connections in the fusion stage, ultimately outputting a gene expression prediction matrix for the target slice in the enhanced graph view: .

[0066] S80. Construct the comprehensive loss function L total The graph encoder is trained and optimized until the model converges, thus obtaining the optimized model. The optimization objective of this invention consists of three parts: mean squared error (MSE) loss, used to minimize the difference between the predicted and actual gene expression matrix values; Pearson correlation coefficient (PCC) loss, used to maximize the correlation between the predicted and actual gene expression matrix values; and contrast loss, used to constrain the representational consistency between enhanced views. The formulas are defined as follows:

[0067]

[0068]

[0069]

[0070] in, This represents the contrast loss of the target node between the two enhanced views; This represents the contrast loss of the reference node between the two enhanced views. This comprehensive loss function ensures the numerical accuracy of the prediction results, improves the correlation between the prediction and the true representation pattern, and enhances the robustness of the model to noise and data perturbations through contrastive learning.

[0071] As an example, during model training and optimization, the Adam optimizer is used, with a learning rate of 0.0001 and weight decay of 0.0001. During training, the graph encoder parameters corresponding to G1 are fixed (gradients are not updated), and only the graph encoder parameters and MLP parameters corresponding to G2 are updated. In each iteration, the preprocessed dataset is input, the comprehensive loss is calculated, and the parameters are updated through backpropagation. This iteration is repeated until the comprehensive loss value decreases by less than 1e over 10 consecutive training epochs. -6 Once the model has converged, the parameters at this point are saved as the optimized model.

[0072] S90. Input the histological pathology image to be predicted into the optimized model and output the corresponding spatial gene expression prediction results.

[0073] Specifically, the histological pathology image to be predicted needs to be preprocessed: obtain the spatial transcriptome sequencing point coordinates of the histological pathology image to be predicted, divide the image into 224×224 pixel blocks with each sequencing point coordinate as the center, and fill the image blocks with zeros at the boundaries; input the image blocks into the pre-trained UNI model, extract the high-dimensional visual embedding vector of each block, and form the image feature matrix of the image to be predicted.

[0074] Next, a simplified multimodal heterogeneous graph is constructed: the node set contains the target nodes of the image to be predicted (features are the image feature matrix vectors mentioned above) and the reference nodes of the training phase (features are the final encoded features of the reference nodes saved after training convergence); the edge set contains the target slice edges of the image to be predicted (connecting the top-5 neighboring nodes of each target node according to Euclidean distance) and cross-slice edges (connecting the top-7 reference nodes of each target node according to the cosine similarity of visual features), and the reference slice edges reuse the edge set of the training phase.

[0075] Finally, the prediction is executed and output: the simplified heterogeneous graph is input into the optimization model, the features are directly encoded through the GraphSAGE, CNDA, and CNAP modules, and then the MLP outputs 1000 highly variable gene expression prediction values ​​for each sequencing point in the image to be predicted, while also outputting the position coordinates of each sequencing point to form spatial gene expression prediction results.

[0076] In summary, the Spatial Gene Expression Prediction Method (SpaHGC) based on pathological images and multimodal heterogeneous maps proposed in this invention aims to jointly model the relationships between sequencing points within and between slices, thereby improving the accuracy of spatial gene expression prediction. SpaHGC first constructs a target slice map based on the spatial coordinates of sequencing points in the target slice to capture local spatial semantics. Subsequently, it connects the target sequencing points with sequencing points in reference slices through image feature similarity, constructing a cross-slice map. In this map, the features of reference nodes are composed of image features and corresponding gene expression profiles, enabling the model to capture shared expression patterns and achieve knowledge transfer. Furthermore, SpaHGC also constructs a reference slice map within each reference slice, capturing global expression relationships by calculating pairwise similarities of the spliced ​​features. To further capture shared expression patterns and suppress inter-slice differences, this invention introduces cross-node dual attention (CNDA) and cross-node attention pooling (CNAP) modules. These two modules can selectively update the features of different types of nodes and dynamically aggregate relevant reference node information, thereby achieving cross-slice knowledge transfer. In addition, to address the common issues of sparsity, noise, and missing expression in ST data, SpaHGC designed a node-type-specific feature masking strategy to simulate missing expression and sequencing errors. This mechanism generates semantically complementary graph views while maintaining structural consistency, significantly improving the robustness and generalization ability of the model.

[0077] The effectiveness of the spatial gene expression prediction method based on pathological images and multimodal heterogeneity maps in this invention is verified through specific experiments: This method collects and organizes human breast cancer datasets of different subtypes from three different platforms in public databases, a human squamous cell carcinoma dataset, and a human lymph node dataset.

[0078] The three breast cancer datasets are as follows: HER2+ dataset: HER2+ subtype, from ST platform, 36 samples in total; Alex dataset: TNBC subtype, from Visium platform, 4 samples in total; Visium BC dataset: HER2+ subtype, from the Visium platform, 3 samples in total; Parameter settings: In this invention, all learning rates are set to 0.0001, weight decay is set to 0.0001, the number of neighbor nodes is set to 5, the number of reference nodes is set to 7, the feature mask ratio of the target node is 0.8, and the reference node ratio is 0.9.

[0079] Implementation steps: Data Preparation: Obtain the histological images, cellular gene expression matrices of spatial transcriptomics sequencing points, and location coordinate matrices of the sequencing points provided in the dataset. Divide the histological images provided in the dataset into 224*224 image patches according to the location of the sequencing points for model input. Identify highly variable genes from the gene expression matrices and normalize them.

[0080] Feature extraction: Each image patch is input into a pre-trained pathological image model to obtain a high-dimensional visual feature vector. The visual features are then concatenated with gene expression features to construct the multimodal features of the reference node.

[0081] Heterogeneous graph construction: Based on spatial proximity, target slice edges are established within the target slice. Based on visual feature similarity, target spots are connected with reference sequencing points to construct cross-slice edges. Based on multimodal feature similarity, reference slice edges are established between reference sequencing points, ultimately forming a multimodal heterogeneous graph containing target and reference nodes.

[0082] Complementary Masking Enhancement: Complementary binary mask matrices are designed to mask the features of the target node and the reference node in different ways, generating two enhanced views. The two views are ensured to be complementary in the feature dimension to simulate feature loss and noise perturbation.

[0083] Graph Encoding and Information Interaction: The GraphSAGE module is used to encode the neighborhood information of edges between the target slice and the reference slice. A cross-node dual attention (CNDA) module is used to establish bidirectional information interaction between the target and reference nodes, capturing cross-slice shared patterns. A cross-node attention pooling (CNAP) module is used to adaptively fuse reference node information, improving the representation capability of the target node.

[0084] Optimization is achieved through a comprehensive loss function, which includes: mean squared error loss, Pearson correlation coefficient loss, and contrast loss between the target node and the reference node. The model parameters are iteratively updated until convergence.

[0085] Table 1 shows the performance comparison results of the present invention with 9 existing mainstream methods (STNet, HisToGene, His2ST, etc.) on five cancer datasets. The core evaluation metrics are the average Pearson correlation coefficient (PCC) and the root mean square error (RMSE).

[0086] Table 1 Performance Comparison Results

[0087] The data in Table 1 clearly demonstrates that the method of this invention (SpaHGC) outperforms all other methods in two key metrics across all datasets. Specifically, the PCC values ​​are the highest across all datasets, significantly higher than other comparative methods, indicating the strongest consistency between the gene expression patterns predicted by this invention and the actual biological patterns. The RMSE values ​​are the lowest (or close to the lowest) across all datasets, indicating the highest quantitative prediction accuracy and the smallest numerical bias. Furthermore, the method exhibits the best performance stability across datasets, platforms, and disease subtypes, with no significant performance fluctuations. This demonstrates that the generalization ability and adaptability of this invention far surpass existing methods, making it widely applicable to gene expression prediction scenarios in different types of cancer.

[0088] Figures 3-6 The image shows the results of visualizing the spatial expression patterns of marker genes on a specific dataset using the SpaHGC method of this invention, compared with existing methods such as STNet, HisToGene, and His2ST. Figure 3 To visualize the expression pattern of the marker gene SPINK5 on the cSCC dataset (human cutaneous squamous cell carcinoma dataset), SPINK5 is a skin keratinization-related gene and a core diagnostic biomarker for cSCC (SPINK5 expression levels are significantly higher in tumor regions than in normal skin tissue), from... Figure 3 As can be seen, the color distribution of the prediction results of this invention (SpaHGC) is highly consistent with the true values. The blue high-expression region accurately matches the tumor lesion area, and the fluorescent green low-expression region fits the normal tissue. The corresponding PCC value (0.84) is the highest among all methods. The prediction results of other comparative methods (such as STNet, HisToGene, and mclSTExp) have obvious deviations, such as discrete distribution of high-expression regions (such as STNet), color distribution completely opposite to the true values ​​(such as HisToGene), or extremely low PCC values ​​(such as M2OST, which is only 0.43), which cannot accurately restore the spatial expression pattern of the SPINK5 gene. This shows that the SpaHGC method of this invention can accurately capture the spatial expression characteristics of the diagnostic marker SPINK5 on the cSCC dataset, and the spatial consistency of the prediction results is highly consistent with the clinicopathological features.

[0089] Figure 4 To visualize the expression pattern of the marker gene FASN on the HER2+ dataset (a HER2-positive breast cancer dataset from the ST platform), and to further visualize this expression pattern, we need to analyze the expression of the FASN gene, a fatty acid synthase gene, which is a key biomarker for targeted therapy in HER2+ breast cancer (patients with high FASN expression are sensitive to fatty acid synthesis inhibitors). Figure 4 As can be seen, the high-expression regions (purple) of the true values ​​are distributed in an "island-like" pattern, corresponding to tumor cell clusters in breast tissue, while the low-expression regions (blue) are normal breast stroma and ductal tissue, spatially distributed as alternating tumor clusters and stroma. The prediction results of the method of this invention (SpaHGC) completely restore the island-like high-expression characteristics. The position and shape of the purple region almost completely overlap with the true values, and the corresponding PCC value (0.91) far exceeds all the comparison methods. The prediction results of other methods (such as STNet, HisToGene, EGGN) show regional shifts or loss of details: the high-expression regions of STNet are scattered in the stroma region (PCC=0.47), the high-expression regions of HisToGene have blurred boundaries (PCC=0.53), and although EGGN can capture the general area, the details are distorted (PCC=0.73). This shows that the method of this invention can accurately restore the spatial distribution details of FASN, a key therapeutic marker for HER2+ breast cancer, providing a reliable spatial molecular basis for the selection of targeted therapy.

[0090] Figure 5 To visualize the expression pattern of the marker gene CD24 on the Alex dataset (a dataset for triple-negative breast cancer (TNBC) using the Visium platform), this study aims to visualize the expression pattern of the marker gene CD24. The Alex dataset is a dataset for triple-negative breast cancer (TNBC) and CD24 is a tumor stem cell marker; its expression level is directly related to the invasiveness and chemotherapy resistance of TNBC (high expression indicates a poor prognosis). Figure 5 The results show that the high-expression region predicted by this invention (SpaHGC) accurately pinpoints the tumor core area, with the boundary perfectly matching the true value. The corresponding PCC value (0.59) is the only model among all methods that exceeds 0.5. The high-expression region of STNet is misclassified as the mesenchymal region (PCC=0.10), the high-expression region of HisToGene is punctately discrete (PCC=0.46), and the prediction result of EGGN is almost uncorrelated with the true value (PCC=0.04). This indicates that the method of this invention can accurately identify the spatial expression range of CD24, a tumor stem cell marker in TNBC, providing a key reference for prognostic assessment and treatment intensity selection.

[0091] Figure 6 To visualize the expression pattern of the marker gene BAMBI on the VisiumBC dataset (a HER2+ breast cancer dataset from the Visium platform), the BAMBI gene was analyzed. BAMBI is a negative regulator of the transforming growth factor β (TGF-β) signaling pathway, and its low expression suggests a high response rate to HER2-targeted drugs (such as trastuzumab). Figure 6The results show that the high expression region (fluorescent green) of the true value is concentrated in the normal breast ductal epithelium, while the tumor lesion region shows low expression (mainly blue). The spatial structure of "duct fluorescent green - tumor blue" is clear and completely matches the pathological histological structure. The prediction results of this invention (SpaHGC) perfectly reproduce this structure: the normal duct region maintains high fluorescent green expression, the tumor lesion region shows uniform low blue expression, and the branching morphology and position of the duct are consistent with the true value. The corresponding PCC value (0.71) far exceeds all comparison methods. STNet misclassifies the tumor lesion as high expression (PCC=0.06), HisToGene's prediction result shows irregular color variation (PCC=0.15), and M2OST cannot distinguish between ducts and tumors (PCC=0.35). This shows that the method of this invention can accurately reproduce the duct-tumor differential expression characteristics of the HER2+ breast cancer treatment response biomarker BAMBI, providing reliable spatial molecular evidence for targeted therapy response prediction.

[0092] The spatial gene expression prediction method based on pathological images and multimodal heterogeneity maps in the embodiments of the present invention has been described above. The spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 7 One embodiment of the spatial gene expression prediction device based on pathological images and multimodal heterogeneity maps in this invention includes: The dataset acquisition module 10 is used to acquire datasets of target tissue slices and reference tissue slices from a public database. The datasets include histological pathological images, spatial transcriptome gene expression data, and the location coordinates of spatial transcriptome sequencing points. Segmentation module 20 is used to segment the histological pathology image into image blocks of a preset size according to the location coordinates of the spatial transcriptome sequencing points, thereby obtaining a set of image blocks P of the target tissue slice. t Image patch set P with reference tissue slice r ; Gene expression data processing module 30 is used to process the spatial transcriptome gene expression data, screen for common highly variable genes in all tissue sections, normalize the screened gene expression values ​​and convert them to the natural logarithmic scale to obtain the gene expression matrix y of the target tissue section. t Gene expression matrix y of reference tissue section r ; Feature extraction module 40 is used to extract features from image patch set P using a pre-trained pathology-based model. t and P r Feature extraction is performed to obtain the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r ; Multimodal heterogeneity map construction module 50, used for gene expression matrix y based on the target tissue slice t And the image feature matrix zt, and the gene expression matrix y of the reference tissue slice. r and image feature matrix z r Construct a multimodal heterogeneous graph G; Masking module 60 is used to execute a complementary masking strategy based on the multimodal heterogeneous graph G to generate two semantically complementary and topologically identical enhanced graph views G1 and G2. Encoding module 70 is used to encode the enhanced graph views G1 and G2 respectively using a graph encoder with shared parameters. The graph encoder includes a GraphSAGE module, a cross-node bidirectional attention CNDA module and a cross-node attention pooling CNAP module. Training module 80 is used to construct the comprehensive loss function L. total The graph encoder is trained and optimized until the model converges, thus obtaining the optimized model. The output module 90 is used to input the histological pathology image to be predicted into the optimization model and output the corresponding spatial gene expression prediction results.

[0093] above Figure 7 The spatial gene expression prediction device based on pathological images and multimodal heterogeneous maps in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The spatial gene expression prediction device based on pathological images and multimodal heterogeneous maps in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0094] Figure 8 This is a schematic diagram of the structure of a spatial gene expression prediction device based on pathological images and multimodal heterogeneous mapping provided by an embodiment of the present invention. This spatial gene expression prediction device 100 based on pathological images and multimodal heterogeneous mapping can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 11 (e.g., one or more processors) and a memory 12, and one or more storage media 13 (e.g., one or more mass storage devices) for storing application programs 133 or data 132. The memory 12 and storage media 13 can be temporary or persistent storage. The program stored in the storage media 13 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the spatial gene expression prediction device 100 based on pathological images and multimodal heterogeneous mapping. Furthermore, the processor 11 may be configured to communicate with the storage media 13 and execute the series of instruction operations in the storage media 13 on the super-resolution gene expression map prediction device 100.

[0095] The spatial gene expression prediction device 100 based on pathological images and multimodal heterogeneity maps may also include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input / output interfaces 16, and / or one or more operating systems 131, such as Windows Server, MacOSX, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 8 The device structure shown does not constitute a limitation on the spatial gene expression prediction device 100 based on pathological images and multimodal heterogeneous maps, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0096] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a spatial gene expression prediction method based on pathological images and multimodal heterogeneous maps.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatial gene expression prediction method based on pathological images and multi-modal heterogeneous graphs, characterized in that, The method comprises the steps of: obtaining a data set of a target tissue slice and a reference tissue slice in a public database, the data set comprising a histopathological image, spatial transcriptome gene expression data and position coordinates of spatial transcriptome sequencing points; According to the position coordinates of the spatial transcriptome sequencing points, the histopathological image is segmented into image blocks of a preset size to obtain an image block set P of the target tissue section t and the image block set P of the reference tissue section r ; processing the spatial transcriptome gene expression data, screening high variable genes common in all tissue sections, normalizing the expression values of the screened genes and converting to natural logarithmic scale, to obtain the gene expression matrix y of the target tissue section t the gene expression matrix y of the reference tissue section r ; A pre-trained pathology-based model was used to analyze the image patch set P. t and P r Feature extraction is performed to obtain the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r ; a gene expression matrix y of the target tissue section t and an image feature matrix zt, and a gene expression matrix y of the reference tissue section r and an image feature matrix z r constructing a multi-modal heterogeneous graph G; performing a complementary masking strategy based on the multi-modal heterogeneous graph G to generate two enhanced graph views G1 and G2 that are complementary in semantics and identical in topology; encoding the enhanced graph views G1 and G2 respectively by using a graph encoder with shared parameters, the graph encoder comprising a GraphSAGE module, a cross-node bidirectional attention CNDA module and a cross-node attention pooling CNAP module; Constructing a comprehensive loss function L total The graph encoder is trained and optimized until the model converges, obtaining an optimized model; inputting a histopathological image to be predicted into the optimization model to output a corresponding spatial gene expression prediction result.

2. The method of claim 1, wherein the method is based on pathological images and multi-modal heterogeneous graphs for spatial gene expression prediction. a gene expression matrix y of the target tissue section t and an image feature matrix zt, and a gene expression matrix y of the reference tissue section r and an image feature matrix z r constructing a multi-modal heterogeneous graph G, comprising the steps of: Define a node set V, wherein the node set V is a target node V t The union set of the reference node V r t n spatial transcriptome sequencing points corresponding to the target tissue slice, each target node is characterized by a high-dimensional visual embedding vector of the target image block extracted by a pre-trained UNI model, and all target node features constitute a target node feature matrix z with a dimension of n x 1024 t ; the reference node V r m spatial transcriptome sequencing points corresponding to the reference tissue slice, the feature of each reference node being a splicing vector of the UNI visual embedding vector of the reference image block and the high-variable gene expression vector normalized with the sequencing point, and all reference node features constitute a reference node feature matrix h with a dimension of m x (1024 + 1000) r , and 1000 is the number of high-variable genes;​ define an edge set E, which is a target slice edge E TS , a cross slice edge E CS and a reference slice edge E RS The union of the three types of edges, wherein the Euclidean distance between each target node corresponding to the sequencing point and other target sequencing points in the target slice is calculated, the Q nearest neighbor nodes of each target node are selected as neighbors, the target node and its neighbor nodes are connected to form the target slice edge set; the cosine similarity between the visual embedding vector of each target node and the visual embedding vector of all reference nodes is calculated, the K reference nodes with the highest similarity are selected for each target node, and the target node and the K reference nodes are connected to form the cross slice edge set; the cosine similarity between the multi-modal feature vector of each reference node and the multi-modal feature vector of other reference nodes is calculated, the K other reference nodes with the highest similarity are selected for each reference node, and the reference node and the K other reference nodes are connected to form the reference slice edge set; combining the node set V and the edge set E to construct a multi-modal heterogeneous graph G.

3. The method of claim 2, wherein the method further comprises: performing a complementary masking strategy based on the multi-modal heterogeneous graph G to generate two enhanced graph views G1 and G2 that are complementary in semantics and identical in topology, comprising the steps of: Two binary mask matrices of n x 1024 dimensions are generated for the target node feature matrix and ; The target node feature matrix z t is multiplied by M t 1 , M t 2 respectively to obtain two target node enhanced feature matrices z t 1 , z t 2 ; the reference node feature matrix h r is multiplied by M r 1 , M r 2 respectively to obtain two reference node enhanced feature matrices h r 1 , h r 2 ; The node set V and the edge set E of the multi-modal heterogeneous graph G are kept unchanged (the same topology), and only the node feature matrix is updated, and z t 1 and h r 1 The row splicing is taken as the node feature matrix to form a first enhanced graph view G1; z t 2 and h r 2 The row splicing is taken as the node feature matrix to form a second enhanced graph view G2, and the two view semantics are complementary and the topologies are consistent.

4. The method of claim 3, wherein the method further comprises: The graph encoder comprises a GraphSAGE module, a cross-node bidirectional attention CNDA module and a cross-node attention pooling CNAP module, and the enhanced graph views G1 and G2 are encoded respectively by using a graph encoder with shared parameters, comprising the steps of: The target slice edge E TS and the reference slice edge E RS in the enhanced graph view are encoded by the GraphSAGE module respectively to obtain the intra-slice encoding features of the target node and the intra-slice encoding features of the reference node. establishing bidirectional attention interaction between the target node and the reference node through the cross-node bidirectional attention CNDA module to update the cross-slice features of the target node and the reference node; aggregating the reference node features by multi-head attention through the cross-node attention pooling CNAP module, combining residual connection to obtain the final representation of the target node, and outputting the gene expression prediction matrix of the target slice in the enhanced graph view through MLP.

5. The method of claim 4, wherein the method further comprises: The target slice edge E TS and the reference slice edge E RS in the enhanced graph view are encoded by the GraphSAGE module respectively to obtain the target node intra-slice encoded feature and the reference node intra-slice encoded feature, including the steps of: Edge E of the target slice in each augmented graph view TS The enhanced features of each target node and the enhanced features of the neighbor target nodes of the target node are element-wise averaged in a mean aggregation manner to obtain aggregated features, and the aggregated features are subjected to non-linear transformation through a fully connected layer with a ReLU activation function to obtain the intra-slice encoding features of the target node. For each augmented graph view, the reference slice edge E RS The reference node augmented features are also processed by mean aggregation, and then transformed by a fully connected layer with ReLU activation function to obtain the reference node intra-slice encoding features.

6. The method of claim 4, wherein the method further comprises: establishing bidirectional attention interaction between the target node and the reference node through the cross-node bidirectional attention CNDA module to update the cross-slice features of the target node and the reference node, comprising the steps of: projecting the intra-slice encoding features of the target node and the intra-slice encoding features of the reference node into a 256-dimensional space through a learnable weight matrix, calculating the scaled dot product attention weight of the query and the key, and then multiplying the weight and the value vector to obtain the cross-slice attention features of the target node; projecting the intra-slice encoding features of the reference node and the intra-slice encoding features of the target node into a 256-dimensional space through a learnable weight matrix, calculating the attention weight and aggregating to obtain the cross-slice attention features of the reference node; updating the cross-slice features of the target node by taking the mean of each target node intra-slice encoding feature and the cross-slice attention features of the target node using the scatter mean operation; and updating the cross-slice features of the reference node by taking the mean of each reference node intra-slice encoding feature and the cross-slice attention features of the reference node.

7. The method of claim 4, wherein the method further comprises: aggregating the reference node features by multi-head attention through the cross-node attention pooling CNAP module, combining residual connection to obtain the final representation of the target node, and outputting the gene expression prediction matrix of the target slice in the enhanced graph view through MLP, comprising the steps of: The cross-node attention pooling CNAP module is provided with h attention heads, each of which projects the target node cross-slice feature and the reference node cross-slice feature to 32 dimensions, calculates the attention weight of each head and aggregates the features; The aggregated features of the h attention heads are spliced and converted to a unified dimension through a learnable output projection matrix; then, residual connection is performed with the target node cross-slice feature, and the MLP with two hidden layers is input to output the gene expression prediction matrix of the target slice in the enhanced graph view.

8. An apparatus for spatial gene expression prediction based on pathological images and multi-modal heterogeneous graphs, comprising: Comprise: A data set acquisition module for acquiring a data set of a target tissue slice and a reference tissue slice in a public database, the data set comprising a histopathological image, spatial transcriptome gene expression data and position coordinates of spatial transcriptome sequencing points; The segmentation module is configured to segment the histopathological image into image blocks of a preset size according to the position coordinates of the spatial transcriptome sequencing points, so as to obtain an image block set P of the target tissue section t and the image block set P of the reference tissue section r . a gene expression data processing module, configured to process the spatial transcriptome gene expression data, screen high variable genes common in all tissue sections, normalize the gene expression values after screening and convert to natural logarithmic scale to obtain a gene expression matrix y of the target tissue section t a gene expression matrix y of the reference tissue section r ; The feature extraction module is used to extract features from a set of image patches P using a pre-trained pathology-based model. t and P r Feature extraction is performed to obtain the image feature matrix z of the target tissue slice. t Image feature matrix z of reference tissue slice r ; a multi-modal heterogeneous graph construction module configured to construct a multi-modal heterogeneous graph G based on a gene expression matrix y of the target tissue section t and an image feature matrix zt, and a gene expression matrix y of a reference tissue section r and an image feature matrix z r construct a multi-modal heterogeneous graph G; A mask module for performing a complementary masking strategy based on the multi-modal heterogeneous graph G to generate two enhanced graph views G1 and G2 that are semantically complementary and topologically identical; An encoding module for encoding the enhanced graph views G1 and G2 respectively using a graph encoder with shared parameters, the graph encoder comprising a GraphSAGE module, a cross-node bidirectional attention CNDA module and a cross-node attention pooling CNAP module; a training module configured to construct a comprehensive loss function L total training and optimizing the graph encoder until the model converges to obtain an optimized model; An output module for inputting a histopathological image to be predicted into the optimization model and outputting a corresponding spatial gene expression prediction result.

9. A device for spatial gene expression prediction based on pathological images and multi-modal heterogeneous graphs, characterized in that, Comprise a memory and at least one processor, the memory has computer readable instructions stored therein; The at least one processor calls the computer readable instructions in the memory to perform the steps of the spatial gene expression prediction method based on the pathological image and the multi-modal heterogeneous graph according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon computer-readable instructions, wherein, The computer readable instructions are executed by the processor to implement the steps of the spatial gene expression prediction method based on the pathological image and the multi-modal heterogeneous graph according to any one of claims 1-7.

Images