A multi-modal spatial domain identification method based on clustering guided gradient contrast learning

Abstract

The application discloses a kind of multi-modal spatial domain identification methods based on clustering guided gradient contrast learning, including step one, constructing dataset;Step two, constructing multi-modal spatial domain identification model;Step three, the multi-modal spatial domain identification model is trained using dataset, and multi-modal representation is obtained, and spatial domain identification result is obtained using unsupervised clustering method mclust clustering.The application increases the semantic consistency between the same cluster samples and the difference between different cluster samples by adjusting the distance between samples and different cluster centers;And an inter-modal attention fusion module is designed, the weight of different modal data information is adjusted, the adaptive fusion of multi-modal features is realized, and the quality of spatial domain identification is improved, with good robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biological information, and particularly relates to a multi-modal spatial domain identification method based on clustering guided gradient contrast learning. BACKGROUND

[0002] Cells in vivo form a complex ecosystem in the form of tissues, and different tissues have different cell compositions and biological functions. Spatial domain identification is committed to depicting the spatial structure of tissues, helping people better understand the coordinated manner of cells, the structure and functional changes of tissues in the process of growth, development, disease, aging, and the like, and has important significance.

[0003] Spatial transcriptome sequencing can measure gene expression information at different positions while preserving spatial position information, providing conditions for spatial domain identification. Existing spatial transcriptome sequencing mainly falls into two categories. The first category is image-based sequencing technology, represented by STARmap, seqFISH, MERFISH and the like. This kind of technology can achieve subcellular resolution, but can only measure part of the genes. The second category is next-generation sequencing-based technology, represented by 10x Visium, SLIDE-seq V2, Stereo-seq and the like. This kind of technology can measure gene expression in the whole genome range, but usually can only achieve multi-cell resolution. Spatial transcriptome data usually have paired pathology images, wherein the spatial transcriptome data reflect gene expression information at different positions, and the pathology images reflect morphological information of cells and tissues.

[0004] Early spatial domain identification is represented by K-Means, Louvain’s method, Seurat and the like. These methods do not use spatial information in the data, and the spatial domains identified are usually discontinuous. Subsequent methods model spatial information at different positions using graph neural networks, effectively solving the problem of discontinuous spatial domains. According to different learning methods, these methods can be divided into unsupervised learning methods and self-supervised learning methods. Unsupervised learning methods are represented by Gitto, SpaGCN, stLearn, BayesSpace, STAGATE and CCST. These methods have no supervision signal, and the accuracy of spatial domain identification is usually limited. In the self-supervised learning method, SpaceFlow maximizes the mutual information between the representation of a single spot and the average representation of all spots; ConST maximizes the mutual information between different spots at local-local, local-global and local-context levels;

[0005] GraphST utilizes contrastive learning to increase the similarity between a spot and its neighbors and decrease the similarity between a spot and its non-neighbors. By introducing self-supervised information, these methods can extract more representative representations, usually with better performance.

[0006] Depending on the use of data, existing methods can also be divided into single-modal methods and multi-modal methods. Single-modal methods such as SpaceFlow, GraphST, SEDR only use spatial transcriptome data, ignoring the complementary information of other modalities, limiting the performance of spatial domain identification. In multi-modal methods, SpaGCN converts the mean and variance of the pathology image block corresponding to the spot into location coordinates and combines the original location information in the spatial transcriptome data to construct a weighted graph; stLearn assumes that spots with similar morphology have more similar gene expression, and proposes to use pathology image features to correct gene expression data; conST realizes the fusion of pathology and spatial transcriptome representation through concatenation; stMDA first aligns the modalities of pathology and spatial transcriptome representation at the local and global levels, and then uses attention mechanism for modal fusion to obtain common representation; STAIG calculates the distance between different spots through pathology image representation, and further converts the distance into mask probability in copy generation. These methods integrate the complementary information in pathology images and spatial transcriptome data, and usually have better performance.

[0007] Although great progress has been made, there are still great limitations in existing methods. First, in terms of learning method, although existing methods introduce self-supervised learning techniques, they often rely on original contrastive learning methods to provide supervision information, in which positive samples are usually different copies of the same sample, and different samples are used as negative samples. However, in the task of spatial domain identification, the semantic information of samples in the same spatial domain is very close, and when these samples are used as negative samples, the model will pay too much attention to the individual differences of different samples, and ignore the common semantic information of samples in the same spatial domain. Second, pathology images and spatial transcriptome reflect different information, and the value of different samples is different, however, most of the existing multi-modal methods do not have adaptive ability, and it is difficult to model the differences between different samples.

[0008] Glossary:

[0009] Spot: data point in spatial transcriptome.

[0010] RetCCL: contrastive clustering model. SUMMARY

[0011] To solve the above problems, the application discloses a multi-modal spatial domain identification method based on clustering-guided gradient contrastive learning.

[0012] To achieve the above objectives, the technical solution of the present invention is as follows:

[0013] A multimodal spatial domain identification method based on clustering-guided gradient contrastive learning includes the following steps:

[0014] Step 1: Construct a dataset, which includes spatial transcriptome data and corresponding pathological data;

[0015] Step 2: Construct a multimodal spatial domain identification model; the multimodal spatial domain identification model includes, in sequence according to the data processing direction, a Graph construction module, a GraphMAE training module, a multimodal representation fusion module, and a cluster-guided gradient contrastive learning module;

[0016] The Graph building module is used to construct the spatial transcriptome graph G. st and pathological image G hist ;

[0017] The GraphMAE training module is used to train the input spatial transcriptome map G. st and pathological image G hist Obtaining pathological hidden space maps Reconstructed pathological map, spatial transcriptome hidden space map and reconstructed spatial transcriptome map The multimodal characterization fusion module is used to analyze the input pathological latent space map. Hidden space map of the transcriptome Obtaining multimodal characterization F m ;

[0018] Cluster-guided gradient contrastive learning module for multimodal representation F m Semantic information enhancement is performed to obtain the updated multimodal representation F. m' ;

[0019] Step 3: Train the multimodal spatial domain identification model using the dataset to obtain the updated multimodal representation F. m' Then, the updated multimodal representation F is processed using the unsupervised clustering method mclust. m' Clustering yields spatial domain identification results.

[0020] Further improvements include the following data processing procedure for the Graph building module:

[0021] Based on the positional correspondence between pathological images and spatial transcriptomes, pathological images are cropped into pathological image patches. Then, a pre-trained model is used to extract pathological image representations from these patches, and highly variable genes from the spatial transcriptome data are selected. Finally, based on spatial relationships, the K-nearest neighbor method is used to construct pathological maps G. hist Spatial transcriptome map G st; pathological graph G hist and spatial transcriptome graph G st Each node in the pathological graph G and the spatial transcriptome graph G corresponds to a spot, and the edges represent the K-neighborhood relationship. The nodes in the pathological graph G represent the pathological image features extracted by the pre-trained model, and the nodes in the spatial transcriptome graph G represent the selected high-variable genes.

[0022] Further improvement, the pre-trained model is RetCCL, and the K value of the K-neighborhood method is 3.

[0023] Further improvement, the GraphMAE training module comprises a pathological training module and a spatial transcriptome graph training module; the pathological training module and the spatial transcriptome graph training module are the same in structure and each comprises a masker, an encoder and a decoder; the masker is used for randomly masking the input graph to obtain a masked graph, and then the encoder extracts features from the masked graph to obtain a latent space graph, and the decoder decodes the latent space graph to obtain a reconstructed graph; the latent space graph comprises a pathological latent space graph and a spatial transcriptome latent space graph The reconstructed graph comprises a reconstructed pathological graph and a reconstructed spatial transcriptome graph

[0024] The GraphMAE training module is subjected to self-supervised training, and the loss function L of the self-supervised training is SCE which is a scaled cosine error:

[0025]

[0026] wherein V represents a set of nodes that are masked, x i represents the i-th node in the input latent space graph, z i represents the i-th node in the reconstructed graph, and γ is a hyperparameter; T represents matrix transposition, v i represents the i-th masked node; L SCE is the GraphMAE loss; the GraphMAE loss comprises is a pathological GraphMAE loss, and is a spatial transcriptome GraphMAE loss.

[0027] Further improvement, the encoder and the decoder each use one layer of graph attention network, and the feature dimension of the latent space is set to 128; the value of γ is 3.

[0028] Further improvement, the data processing method of the multi-modal feature fusion module is as follows:

[0029] The input pathological latent space graph and the spatial transcriptome latent space graph The corresponding node feature sets are respectively and The dimension of the representation is reduced to 1 dimension through two mapping networks, i.e.:

[0030]

[0031] where f hist and f st are vectors with length N, N is the number of nodes in the graph; M hist () represents a one-layer fully connected network of pathology, M st () represents a one-layer fully connected network of spatial transcriptome, and then the amount of information of the two modalities is measured by variance, i.e.:

[0032] v hist = Var(f hist ) (4)

[0033] v st = Var(f st ) (5)

[0034] Var() represents the calculation of variance, v hist represents the variance of the reduced pathology representation, and v st represents the variance of the reduced spatial transcriptome representation.

[0035] After normalization by Softmax, the modal weight is obtained:

[0036] [w hist ,w st ] = softmax([v hist ,v st ]) (6)

[0037] where w hist represents the weight of the pathology modality, w st represents the weight of the spatial transcriptome modality, and softmax() represents the normalization exponential function.

[0038] Finally, the multi-modal representation F m is obtained by multi-modal weighting:

[0039]

[0040] w hist represents the weight of the pathology modality, and w st represents the weight of the spatial transcriptome modality.

[0041] 7. The multi-modal spatial domain identification method based on cluster guided gradient contrast learning according to claim 1, wherein the data processing flow of the cluster guided gradient contrast learning module is as follows:

[0042] For the multi-modal representation Fm First, clustering is performed using K-Means, and the cluster label to which the ith sample belongs is τ i And the cluster center of each cluster is calculated:

[0043]

[0044] Where N n is the number of samples in cluster n, is the jth sample in cluster n; C n is the cluster center of cluster n; and for the ith multi-modal representation A random neighbor from the same cluster is selected as a positive sample, and the corresponding representation is The cluster center of all clusters is taken as a negative sample, and the corresponding representation is C n The set of all negative samples is S neg ; In order to increase the semantic consistency of intra-cluster samples and the semantic difference of inter-cluster samples, different weights are used to distinguish different negative samples, and for the cluster center of cluster n:

[0045]

[0046] Where β n is the weight corresponding to the cluster center of cluster n, and ε is a hyperparameter, taking a value between 0 and 1;

[0047] The loss function L Con of the cluster-guided gradient contrast learning module is:

[0048]

[0049] exp() represents the natural exponential function, and sim() is the similarity function; C neg represents the set of negative samples.

[0050] Further improvement, in step three, first pre-train the GraphMAE training module, and then train the multi-modal spatial domain identification model as a whole;

[0051] The loss function of pre-training is:

[0052]

[0053] α hist is the pathology GraphMAE loss weight, α st is the spatial transcriptome GraphMAE loss weight, is the pathology GraphMAE loss, is the spatial transcriptome GraphMAE loss;

[0054] The loss function L of the overall training is:

[0055]

[0056] alpha Con is a contrastive learning loss weight;

[0057] When the loss function L of the overall training is minimized, the updated multi-modal representation F is obtained m' Then, the updated multi-modal representation F is clustered by using an unsupervised clustering method mclust m' The spatial domain identification result is obtained by clustering.

[0058] Advantages of the present application:

[0059] The present application adjusts the distance between the sample and different cluster centers, increases the semantic consistency between the same cluster samples and the difference between different cluster samples, and designs a modal attention fusion module to adjust the weight according to the information amount of different modal data, realizes adaptive fusion of multi-modal features, and improves the quality of spatial domain identification, which has good robustness. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 It is a method principle diagram of the present application, wherein (a) pathology and spatial transcriptome Graph construction; (b) clustering guided contrastive learning; (c) modal attention fusion module.

[0061] Figure 2 It is a performance comparison diagram of single modal and multi-modal method of the present application.

[0062] Figure 3 It is an experimental result under different parameters.

[0063] Figure 4 It is a performance comparison diagram of the method of the present application and the GraphST method.

[0064] Figure 5 The results of three samples 151508, 151671 and 151673 are visualized. Wherein (a) is the Ground Truth of the 151508 sample, (b) is the identification result of the 151508 sample; (c) is the Ground Truth of the 151671 sample, (d) is the identification result of the 151671 sample; (e) is the Ground Truth of the 151673 sample, and (f) is the identification result of the 151673 sample. DETAILED DESCRIPTION

[0065] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application

[0066] 1. The method is as follows:

[0067] 1.1 Method overview

[0068] The principle of the proposed multi-modal spatial domain identification method based on clustering-guided contrastive learning is as shown in Figure 1 The method comprises four steps: (1) construction of pathology and spatial transcriptome maps; (2) GraphMAE-based pathology and spatial transcriptome encoder-decoder training; (3) multi-modal representation fusion; and (4) clustering-guided gradient contrastive learning.

[0069] 1.2 Graph construction

[0070] In order to better utilize spatial information, the present method intends to utilize a graph neural network, so first, a graph structure data is constructed for the pathology and spatial transcriptome data respectively. First, according to the position correspondence relationship of the pathology image and the spatial transcriptome, the pathology image is cropped into small image blocks; then, a pre-trained model is used to extract the representation of the pathology image block, and high variable genes in the spatial transcriptome data are selected; finally, according to the spatial position relationship, a K-nearest neighbor method is used to construct a pathology graph G hist and a spatial transcriptome graph G st Each node in the pathology graph and the spatial transcriptome graph corresponds to a spot, and the edge represents the K-nearest neighbor relationship. The node representation in the pathology graph is the pathology image representation extracted by the pre-trained model, and the node representation in the spatial transcriptome graph is the selected high variable gene.

[0071] In the specific implementation process, in order to cover the complete area of a spot, the size of the pathology image block is set to 100x100 μm 2 , the pre-trained pathology image model used is RetCCL, the K in the K-nearest neighbor is selected to be 3, and 3000 high variable genes are selected using seurat_v3.

[0072] 1.3 GraphMAE training

[0073] To obtain a good initial representation of pathological images and spatial transcriptome data, the method first uses a graph masked autoencoder (GraphMAE) to perform self-supervised training on data in two modalities. The model structures of the two modalities are the same, and the following is introduced taking the spatial transcriptome data as an example.

[0074] The spatial transcriptome GraphMAE includes three modules: a masker, an encoder, and a decoder. For the input spatial transcriptome graph G st , the masker first randomly masks the nodes in the graph to obtain Then, the encoder E st performs representation extraction on to obtain the latent space graph Finally, the decoder D st decodes the latent space graph to obtain the reconstructed spatial transcriptome graph

[0075] The loss function of GraphMAE is the scaled cosine error (SCE), which is calculated as follows:

[0076]

[0077] where V represents the set of masked nodes, x i represents the i-th node in the input spatial transcriptome graph G st , z i represents the i-th node in the reconstructed spatial transcriptome graph , and γ is a hyperparameter.

[0078] In the specific implementation process, the mask rate in the masker is set to 0.6, that is, 60% of the nodes are randomly selected for masking; the encoder and the decoder both use a layer of graph attentional network (GAT), and the feature dimension of the latent space is set to 128; the γ in the loss function is set to 3.

[0079] 1.4 Multi-modal representation fusion

[0080] Under different samples, the value of pathology and spatial transcriptome for spatial domain identification is different, in order to better model the difference between samples, the method uses a modal attention fusion module to adaptively adjust the weight of different modalities. The structure of the modal attention fusion module is as shown in Figure 1 (c). The corresponding node feature sets of the input pathological latent space graph and the spatial transcriptome latent space graph are and The dimension of the representation is reduced to 1 dimension through two mapping networks, i.e.:

[0081]

[0082] where f hist and f st are vectors with length N, N is the number of nodes in the graph. Then, the information amount of two modalities is measured by variance, i.e.:

[0083] v hist = Var(f hist ) (4)

[0084] v st = Var(f st ) (5)

[0085] After Softmax normalization, the modal weight is obtained:

[0086] [w hist ,w st ] = softmax([v hist ,v st ]) (6)

[0087] Finally, multi-modal weighting is performed:

[0088]

[0089] Through the above method, the method can adaptively adjust the modal weight according to the information amount of different modalities. In the specific implementation process, the mapping network M hist () and M st () are both one layer of fully connected network.

[0090] 1.5 Cluster-guided gradient contrast learning

[0091] In ordinary contrast learning, the positive sample is often different copies of the same sample, and different samples are set as negative samples. However, in the task of spatial domain identification, the semantic information of samples belonging to the same spatial domain is very close, which leads to the negative negative sample, affecting the model performance. To solve this problem, the method proposes cluster-guided contrast learning, which selects reliable negative samples to reduce the influence of negative negative samples.

[0092] For the obtained multi-modal representation F m , first use K-Means to cluster it, and the cluster label of the i-th sample is τ i , and the cluster center of each cluster is calculated:

[0093]

[0094] where N n is the number of samples in cluster n, is the jth sample in cluster n. For each sample, we represent We randomly select one of its co-cluster neighbors as the positive sample, corresponding representation is The cluster center of all clusters are regarded as negative samples, corresponding representation is C n , and the set of all negative samples is S neg . To increase the semantic consistency of intra-cluster samples and the semantic difference of inter-cluster samples, we distinguish different negative samples by different weights. Specifically, for sample i:

[0095]

[0096] where β n is the weight of the cluster center of cluster n, and ε is a hyperparameter, taking values between 0 and 1.

[0097] The loss function of clustering-guided contrastive learning is:

[0098]

[0099] In the specific implementation process, the number of clusters of K-Means is specified by prior knowledge.

[0100] 1.6 Training process

[0101] The training of the model is divided into two stages. In the first 500 epochs, only the encoders and decoders of pathology and spatial transcriptome are trained using GraphMAE, and the corresponding training loss is:

[0102]

[0103] where L and L are the reconstruction losses of pathology and spatial transcriptome, respectively, and α hist and α st are the corresponding weights, which are both 1 in this method. In the 500th epoch to the 1200th epoch, the entire model is trained, and the loss function at this time is:

[0104]

[0105] where L Con is the contrastive learning loss, and α Con is the contrastive learning loss weight, which is 0.05 in this method.

[0106] 2. Performance comparison and evaluation:

[0107] 2.1 Data and evaluation metrics

[0108] To evaluate the performance of the proposed method, we conduct experiments on the human dorsolateral prefrontal cortex (DLPFC) dataset. The DLPFC dataset is collected based on the 10x Visium technology, which contains 12 slides in total, with the number of spots ranging from 3460 to 4789 per slide, and a total of 33538 genes captured. Each slide in the DLPFC dataset is divided into 5 to 7 spatial domains, corresponding to multiple cortical layers and a white matter layer.

[0109] We use the Adjusted Rand Index (ARI) to evaluate the performance of the proposed method. The ARI takes into account the possibility of accidental agreement between two clusters. The ARI ranges from 0 to 1, where a higher ARI value closer to 1 indicates a more similar clustering result, and is calculated as follows:

[0110]

[0111] where TP is the number of spot pairs that belong to the same cluster in both the true labels and the predicted results, TN is the number of spot pairs that belong to different clusters in both the true labels and the predicted results, FN is the number of spot pairs that belong to the same cluster in the true results but belong to different clusters in the predicted results, and FP is the number of spot pairs that belong to different clusters in the true results but belong to the same cluster in the predicted results. E is the expected value of this metric, which is the value when clustering is completely random, and is calculated as follows:

[0112]

[0113] 2.2 Comparison of single-modality and multi-modality performance

[0114] We first compare the performance of the spatial transcriptome single-modality and the spatial transcriptome plus pathology multi-modality, and the results are shown in Figure 2 On most samples, the performance of the multi-modality method is better than that of the single-modality method. Specifically, the average ARI of the single-modality is 0.602, and the average ARI of the multi-modality is 0.622, which indicates that integrating pathology and spatial transcriptome data can improve the quality of spatial domain identification, and the proposed method can effectively integrate multi-modality information.

[0115] 2.3 Results of different parameters

[0116] In the proposed method, the weight of the negative samples in the same cluster is a key parameter, so we compare the performance of different weights, and the results are shown in Figure 3The proposed method can achieve good results under different weights. Specifically, when the weights are 0.5, 0.7 and 0.9, the corresponding average ARIs are 0.613, 0.622 and 0.608, respectively, which shows that the proposed method has good robustness to the weight of the same cluster negative samples.

[0117] 3.4 Comparison of different methods

[0118] We further compared the proposed method with the state-of-the-art method (GraphST), and the results are shown in FIG. 6. Figure 4 On most samples, the performance of the proposed method is better than that of GraphST. Specifically, the average ARI of GraphST is 0.558, and that of the proposed method is 0.622, which fully proves that the proposed method has excellent performance.

[0119] 3.5 Visualization results

[0120] In order to intuitively evaluate the spatial domain identification effect of the proposed method, we visualized the results of three samples 151508, 151671 and 151673, and the results are shown in FIG. 7. Figure 5 On the three samples, the proposed method can accurately achieve spatial domain identification, and has good consistency with the true label, which intuitively proves the performance of the proposed method.

[0121] Although the embodiments of the present application have been disclosed as above, they are not limited to the use listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application. Those skilled in the art can easily make other modifications, and therefore the present application is not limited to the specific details and shown herein without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A multi-modal spatial domain authentication method based on cluster-guided gradient contrastive learning, characterized in that, The method comprises the following steps: Step one, constructing a data set comprising spatial transcriptome data and corresponding pathological data; Step two, constructing a multi-modal spatial domain identification model; the multi-modal spatial domain identification model comprises, in sequence according to a data processing direction, a Graph construction module, a GraphMAE training module, a multi-modal representation fusion module and a cluster-guided gradient contrast learning module; The Graph construction module is configured to construct a spatial transcriptome graph G st and a pathology graph G hist ; The GraphMAE training module is configured to obtain a pathology latent space graph Gpath from the input pathology graph Gpath st and the pathology graph G hist obtain a pathology latent space graph Gpath the reconstructed pathology graph, the spatial transcriptomic latent space graph Gspat and the reconstructed spatial transcriptomic graph Gspat The multi-modal representation fusion module is configured to obtain a multi-modal representation F from the input pathology latent space graph Gpath and the spatial transcriptomic latent space graph Gspat obtain a multi-modal representation F m ; The cluster-guided gradient contrast learning module is applied to the multi-modal representation F m The semantic information is enhanced, and an updated multi-modal representation F m' ; Step three, training the multi-modal spatial domain identification model with the dataset to obtain an updated multi-modal representation F m' Then, using an unsupervised clustering method mclust to cluster the updated multi-modal representation F m' to obtain a spatial domain identification result The data processing flow of the cluster-guided gradient contrast learning module is as follows: For multi-modal representation F m , first clustering is performed using K-Means, the cluster label to which the ith sample belongs is τ i , and the cluster center of each cluster is calculated: where N n is the number of samples in cluster n, is the jth sample in cluster n; C n is the cluster center of cluster n; for the ith multi-modal representation randomly select one from the neighbors of the same cluster as the positive sample, corresponding representation is the cluster center of all clusters as negative samples, corresponding representation is C n , the set of all negative samples is S neg ; in order to increase the semantic consistency of the samples in the cluster and the semantic difference of the samples between the clusters, different weights are used to distinguish different negative samples, for the cluster center of cluster n: where β n is the weight corresponding to the cluster center of cluster n, and ε is a hyperparameter that takes values between 0 and 1. Loss function L of the cluster-guided gradient contrastive learning module Con is: exp() denotes the natural exponential function, sim() is a similarity function; V neg represents a set of negative samples.

2. The multi-modal spatial domain authentication method based on cluster guided gradient contrastive learning of claim 1, wherein, The data processing process of the Graph construction module is as follows: According to the position correspondence relationship between the pathological image and the spatial transcriptome, the pathological image is cropped into pathological image blocks; then the pathological image representation of the pathological image blocks is extracted through a pre-trained model, and high variable genes in the spatial transcriptome data are selected; finally, according to the spatial position relationship, a pathological graph G hist and a spatial transcriptome graph G st are constructed respectively by using a K nearest neighbor method; each node in the pathological graph G hist and the spatial transcriptome graph G st corresponds to a spot, an edge represents a K nearest neighbor relationship, the node representation in the pathological graph is the pathological image representation extracted by the pre-trained model, and the node representation in the spatial transcriptome graph is the selected high variable gene.

3. The multi-modal spatial domain qualification method based on cluster guided gradient contrast learning of claim 2, wherein, The pre-training model is RetCCL, and the K value of the K nearest neighbor method is 3.

4. The multi-modal spatial domain authentication method based on cluster guided gradient contrastive learning of claim 1, wherein, The GraphMAE training module comprises a pathology training module and a spatial transcriptome map training module; the pathology training module and the spatial transcriptome map training module are structurally identical, and each comprises a masker, an encoder and a decoder; the masker is used for randomly masking an input graph to obtain a mask graph, then the encoder extracts a feature of the mask graph to obtain a hidden space graph, and the decoder decodes the hidden space graph to obtain a reconstructed graph; the hidden space graph comprises a pathology hidden space graph and a spatial transcriptome hidden space graph The reconstructed graph comprises a reconstructed pathology graph and a reconstructed spatial transcriptome graph GraphMAE training module proceeds with self-supervised training, a loss function L of the self-supervised training SCE is the scaled cosine error: where V represents the set of nodes that are masked out, x i represents the i-th node in the input latent space graph, z i represents the i-th node in the reconstructed graph, γ is a hyperparameter; T represents matrix transpose, v i represents the i-th masked node; L SCE is the GraphMAE loss; the GraphMAE loss includes is the pathological GraphMAE loss and is the spatial transcriptome GraphMAE loss.

5. The multi-modal spatial domain authentication method based on cluster guided gradient contrastive learning of claim 4, wherein, Both the encoder and the decoder use one layer of graph attention network, the feature dimension of the hidden space is set to 128, and the value of γ is 3.

6. The multi-modal spatial domain authentication method based on cluster guided gradient contrastive learning of claim 1, wherein, The data processing method of the multi-modal representation fusion module is as follows: input pathological latent space map and spatial transcriptomic latent space map corresponding node feature sets are and are reduced to 1 dimension by two mapping networks, respectively, i.e.: where f hist and f st are vectors of length N, N is the number of nodes in the graph; M hist () represents the pathology one-layer fully connected network, M st () represents the spatial transcriptome one-layer fully connected network, and then the information amount of the two modalities is measured by variance, that is: v hist = Var(f hist ) (4) v st = Var(f st ) (5) Var() denotes computing variance, v hist Var() denotes computing variance, v st Var() denotes computing variance, v And after Softmax normalization, the modal weight is obtained: [w hist ,w st ] = softmax([v hist ,v st ]) (6) where wpath hist represents the weight of the pathology modality, wspac st represents the weight of the spatial transcriptome modality, and softmax() represents a normalized exponential function; Finally, the multi-modal representation F is obtained by multi-modal weighting m : w hist denotes the pathological modality weight, w st denotes the spatial transcriptomic modality weight.

7. The multi-modal spatial domain authentication method based on cluster guided gradient contrastive learning of claim 1, wherein, In step three, the GraphMAE training module is pre-trained first, and then the multi-modal spatial domain identification model is trained as a whole; The loss function of pre-training is: a hist is a pathology GraphMAE loss weight, a st is a spatial transcriptome GraphMAE loss weight, is a pathology GraphMAE loss, is a spatial transcriptome GraphMAE loss; The loss function L of the whole training is: a Con for contrastive learning loss weight; The updated multi-modal representation F is obtained when the loss function L of the whole training is minimized m' Then the updated multi-modal representation F is clustered by the unsupervised clustering method mclust m' The spatial domain identification result is obtained by clustering.

Images