Multi-modal joint optimization clustering method, system and device for spatial transcriptome

Abstract

The application discloses a multi-modal joint optimization clustering method, system and device for a spatial transcriptome, wherein the method comprises: multi-modal feature extraction and graph construction, cross-modal feature fusion, and contrast learning and joint optimization; the system comprises: a spatial graph construction module, a multi-modal feature fusion module, a joint optimization module, and a spatial domain division module. In the application, high-precision and robust biological tissue spatial functional domain division is realized, and the application is suitable for application scenarios such as tumor microenvironment analysis and developmental biology research.

Description

Technical Field

[0001] This invention relates to the field of bioinformatics and artificial intelligence, and more specifically to a multimodal joint optimization clustering method, system and device for spatial transcriptomics. Background Technology

[0002] Spatial transcriptomics technology has achieved groundbreaking progress in molecular biology and biomedical research. By integrating gene expression profiles with spatial location information, this technology provides a key tool for analyzing the tissue microenvironment. In particular, high-throughput spatial gene expression detection can simultaneously preserve the molecular characteristics of gene expression and the spatial distribution information of cells, providing high-dimensional data support for studies such as tumor heterogeneity analysis and developmental trajectory reconstruction. However, the complex multimodal characteristics of spatial transcriptomics data (including gene expression matrices, histopathological images, and spatial coordinates) make it difficult for traditional single-modal analysis methods to capture the complete biological meaning, especially in spatial functional region delineation tasks where boundary identification bias is prone to occur.

[0003] Multimodal clustering methods for spatial domain recognition have received widespread attention in recent years. These methods aim to integrate data from different modalities to improve the accuracy of spatial structure resolution. For example, stLearn uses graph neural networks to integrate gene expression and spatial neighborhood information, and uses manually defined image morphological features to assist in locating cell types; SpaGCN introduces graph convolutional networks to construct a joint gene expression-spatial topology model, and uses Gaussian kernel functions to quantify spatial adjacency relationships; GraphST proposes a cross-view contrastive learning framework that encodes spatial locations as graph structures and embeds them with gene expression. In addition, MODISCO coordinates the interaction between spatial coordinates and transcriptome features through a multi-head attention mechanism, while SPACE-GM uses variational autoencoders to jointly reconstruct gene expression and spatial distribution probabilities.

[0004] While existing methods attempt to integrate multimodal information, significant limitations remain in their processing flow: most models independently process gene expression, spatial location, and tissue image modalities, performing only simple concatenation or weighted averaging in the later stages of feature representation, failing to achieve dynamic interactive learning across modalities. This approach makes it difficult to deeply mine complementary information between modalities, especially failing to adaptively balance the semantic differences between data at different resolutions (such as single-cell-level image features and multi-cell-level spatial sites), ultimately limiting the accurate analysis of cellular microenvironment heterogeneity and the continuous identification of spatial functional boundaries. Summary of the Invention

[0005] The present invention provides a multimodal joint optimization clustering method, system and device for spatial transcriptomics that can achieve high-precision and robust spatial functional domain division of biological tissues, and can solve at least one of the above-mentioned technical problems.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A multimodal joint optimization clustering method for spatial transcriptomics includes the following steps: S1. Multimodal Feature Extraction and Graph Construction: Construct a multimodal feature embedding framework, extract hierarchical features from gene expression data through graph convolutional networks, capture local semantic features of histological images using pre-trained convolutional neural networks, and construct a dynamic weighted adjacency graph based on spatial coordinates to encode spatial proximity. S2, Cross-modal feature fusion: In the feature fusion stage, a cross-modal contrastive learning mechanism is designed to optimize modal alignment, and long-range dependencies are modeled by combining single-modal self-attention. S3. Contrastive Learning and Joint Optimization: For spatial clustering tasks, a joint objective function is proposed, which integrates zero-inflated negative binomial distribution reconstruction loss, graph topology constraints and contrastive loss, and uses graph embedding spatial K-means clustering algorithm to accurately partition the spatial domain.

[0007] Furthermore, S1 further includes: S1.1 Based on the spatial coordinate matrix, an adaptive weighted adjacency graph is constructed. Spatial proximity is encoded by the Gaussian kernel function. By setting the neighborhood radius threshold and distance decay parameter, a continuous representation of discrete adjacency relationships can be achieved with the help of a continuous weight transformation mechanism. Given a set of spatial locations and its corresponding coordinate matrix ,in, n This represents the total number of spatial sites in the tissue sample. d Representing the dimension of spatial coordinates, the adjacency matrix is ​​constructed using the radial basis function method. :

[0008] in, i and j These represent the site indices in the spatial site set. and Representing sites i and site j Spatial coordinate vectors in the tissue sample, Indicates the neighborhood radius threshold. This represents the distance attenuation parameter; S1.2. Graph convolution feature extraction is performed on the original gene expression matrix. Adjacency matrix with added self-loops and trainable weights are used to achieve hierarchical feature aggregation through a two-layer graph convolution network to output gene expression features. Original gene expression matrix ,in, n This represents the total number of spatial sites in the tissue sample. mTo represent the gene expression feature dimension corresponding to each spatial site, a self-loop adjacency matrix and its corresponding degree matrix are introduced, and a trainable parameter matrix is ​​defined. Hierarchical feature aggregation is achieved through a two-layer graph convolutional network, expressed as:

[0009]

[0010] Where A represents the original adjacency matrix. This represents the normalized adjacency matrix with self-loops added. Represents the adjacency matrix with self-loops The corresponding degree matrix, This represents the low-dimensional gene expression feature matrix output after graph convolutional encoding, where Encoder represents the multimodal graph convolutional encoding network. For activation functions; S1.3. Perform deep visual feature extraction on histological images, use a pre-trained convolutional neural network to extract local semantic information, and output a fixed-dimensional image feature vector. Given a histological image As the raw input, the convolutional neural network is trained using the ResNet-50 network architecture. The convolution operation process satisfies the following formula:

[0011] in, Represents the first visual neural network The output feature map of the layer, Represents the first in a visual neural network The trainable weight parameter matrix of the convolutional kernel. This represents the convolution operation. This represents a batch normalization operation used to extract local image features, which are then activated by ReLU to obtain a 2048-dimensional image feature vector. .

[0012] Furthermore, S2 further includes: S2.1, Splicing gene expression features and image features to generate an initial fusion representation; All spatial sites Image feature vectors Vertically stacked to form an image feature matrix Then combine the image feature matrix with the gene feature matrix. Perform column-wise concatenation to generate the initial fusion representation matrix. :

[0013] in, This represents a vector concatenation operation. n This represents the total number of spatial sites in the tissue sample. Dimensions representing gene expression characteristics Dimensions representing the features of a histological image; S2.2. Aggregate neighborhood information through multi-layer graph convolution operations to update the site feature representation, as expressed in the following expression:

[0014] in, Indicating the first in the fusion framework Multimodal joint feature representation of layers, Indicates the first The trainable parameter weight matrix of a layered graph convolutional network. This represents the normalized adjacency matrix with self-loops added. Represents a nonlinear activation function; and The current layer features are linearly transformed based on the adjacency matrix with added self-loops. Each site absorbs features from its neighboring sites, and the aggregation result is standardized before being output after nonlinear activation. S2.3 Introduce a graph attention mechanism to model global dependencies and dynamically calculate the similarity weights between sites; First, an attention mechanism is introduced for dynamic weight allocation to depict the nonlinear correlation characteristics among heterogeneous data, expressed as:

[0015]

[0016] in, Indicates attention weights, For normalized exponential functions, For a leaky linear rectified function, It is a learnable mapping vector. This represents a trainable shared weight parameter matrix. This represents the current target spatial location to be processed in the graph structure. Feature vector Indicates the spatial location relative to the target Neighborhood site feature vectors with spatial proximity relationships. This indicates the sites updated by the graph attention mechanism. Enhanced feature vectors, This indicates the relationship between the target spatial location and the adaptive weighted adjacency graph. A set of neighboring sites that have spatial proximity relationships; Secondly, this addresses the enhancement feature matrix generated in the feature fusion stage of a multi-layer graph convolutional network. The expression for the weighted fusion of gene and image features through a multi-head self-attention mechanism is as follows:

[0017]

[0018] in, Indicates the first The output of each attention head, This represents the output of the self-attention mechanism. , , They represent the first The trainable projective weight matrix for each query, key, and value. Let T be the dimension of the key vector, and T be the matrix transpose. This indicates a multi-head attention mechanism. This indicates a feature concatenation operation. For the total number of attention heads, This represents the output linear projection matrix after multi-head fusion.

[0019] Furthermore, S3 further includes: S3.1 Design a cross-modal contrastive loss function to dynamically select positive and negative sample pairs based on spatial proximity and feature similarity; splicing gene feature matrix The OK and image feature matrix The OK To form a joint representation vector To construct the contrastive loss function :

[0020] in, The cosine similarity function is used. For temperature coefficient, For site i The joint representation vector after splicing gene and image information. For anchor point samples Semantically consistent positive sample pairs This represents the total number of negative samples used in the comparison calculation. k Index for negative samples For anchor point samples The first semantic inconsistency kOne negative sample pair; S3.2, Loss of integrated gene expression remodeling Comparative loss and Graph Laplacian Regularization Constraints Construct a joint loss function for multiple tasks. The expression is:

[0021]

[0022]

[0023] in, , and Let represent the hyperparameter weights of the reconstruction loss, contrastive loss, and regularization term, respectively. n This represents the total number of spatial sites in the tissue sample. m This represents the dimension of gene expression characteristics corresponding to each spatial location. Represents the first gene in the original gene expression matrix. The locus The actual expression level of each gene , and Let these represent the mean, divergence parameter, and zero-inflation probability of the negative binomial distribution generated by the feature reconstruction decoding network, respectively. For indicator functions, The standard negative binomial probability density function, L Let A denote the graph Laplacian matrix, A denote the self-loop adjacency matrix, and D denote the degree matrix corresponding to A. The feature reconstruction decoding network consists of a parallel multilayer perceptron structure, and the decoding network uses a low-dimensional gene expression feature matrix obtained through graph convolutional encoding. As input, respectively map the mean matrix required for the zero-inflated negative binomial distribution to the output. , divergence matrix and the zero-inflation probability matrix The specific mapping relationship expression is as follows:

[0024]

[0025]

[0026] in, , and These represent multilayer perceptrons with independent weights. It is a natural exponential function. For activation functions; S3.3, Enhance feature discriminativeness through a difficult negative sample selection strategy, expressed as:

[0027] in, Indicates the determination site i with site j An indicator function for whether it is a hard negative sample. Indicates the first to be compared j Feature vectors of each site and These are all pseudo-labels for cell types generated by the model's pre-clustering in the current iteration round. This is an adaptive threshold.

[0028] A multimodal joint optimization clustering system for spatial transcriptomics, applicable to the multimodal joint optimization clustering method for spatial transcriptomics, includes: A spatial map construction module is used to acquire the original gene expression matrix, spatial coordinate matrix, and histological images from spatial transcriptome data. A multimodal feature fusion module, connected to the spatial graph construction module, is used to receive and splice gene expression features and image features to generate an initial fused representation; A joint optimization module, connected to the multimodal feature fusion module, designs a cross-modal contrastive loss function based on cross-modal interaction features to dynamically select positive and negative sample pairs; The spatial domain partitioning module, which is connected to the joint optimization module, achieves accurate spatial domain partitioning in the graph embedding space based on the K-means clustering algorithm and outputs the organizational functional region analysis results.

[0029] Furthermore, the spatial map construction module obtains the original gene expression matrix, spatial coordinate matrix, and histological image, and performs the following processing respectively: An adaptive weighted adjacency graph is constructed based on a spatial coordinate matrix, and spatial proximity is encoded by a Gaussian kernel function. The neighborhood radius threshold is adaptively adjusted according to the tissue density, and the distance decay parameter is set to 1.5 times the standard deviation of the Euclidean distance. A self-loop adjacency matrix is ​​added to the original gene expression matrix, and gene expression features are output through a two-layer graph convolutional network. A pre-trained ResNet-50 network was used to extract 2048-dimensional feature vectors from histological images. After convolutional layers and batch normalization, the image features were output.

[0030] Furthermore, in the generation of the initial fused representation in the multimodal feature fusion module, neighborhood information is first aggregated through multi-layer graph convolution operations, then a graph attention mechanism is introduced to model global dependencies, and features are fused through multi-head self-attention weighted fusion, and finally cross-modal interaction features are output.

[0031] Furthermore, in the joint optimization module, a multi-task joint loss function is constructed by integrating gene reconstruction loss, contrast loss, and graph Laplacian regularization constraint, and a difficult negative sample selection strategy is used to enhance feature discriminativeness.

[0032] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the above-described multimodal joint optimization clustering method for spatial transcriptomics.

[0033] The beneficial effects of this invention are reflected in: 1. By constructing an adaptive graph structure modeling module, the fusion of three modal data of gene expression, spatial coordinates and histological images was realized. The collaborative design of cross-modal contrastive learning mechanism and graph attention mechanism improved the semantic alignment capability of heterogeneous data and solved the problem of insufficient complementarity of multi-source information in traditional methods.

[0034] 2. By designing a joint optimization objective function, integrating zero-inflated negative binomial distribution reconstruction loss, graph topology constraints, and contrast loss, we address the challenges of gene expression sparsity and boundary ambiguity. The dynamic adjustment mechanism of adaptive neighborhood radius threshold and distance decay parameters enhances the model's adaptability to tissue density heterogeneity.

[0035] 3. Through joint verification using UMAP dimensionality reduction visualization and pseudo-temporal trajectory inference, the spatial domain divided by the algorithm is highly consistent with the tissue structure of H&E staining images. Differential gene enrichment analysis confirms the biological interpretability of the spatial domain function, providing a reliable tool for the study of molecular mechanisms in complex tissues. Attached Figure Description

[0036] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0037] Figure 1 This is a schematic diagram of the overall process of the multimodal joint optimization clustering method according to an embodiment of the present invention.

[0038] Figure 2 This is a detailed flowchart illustrating the multimodal joint optimization clustering method according to an embodiment of the present invention.

[0039] Figure 3This is a structural block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] It should be noted that the meaning of "and / or" throughout the text includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or a solution that simultaneously satisfies A and B. Furthermore, "multiple" refers to two or more. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0042] See Figure 1 and Figure 2 This invention provides a multimodal joint optimization clustering method for spatial transcriptomics, comprising the following steps: S1. Multimodal Feature Extraction and Graph Construction: Construct a multimodal feature embedding framework, extract hierarchical features from gene expression data through graph convolutional networks, capture local semantic features of histological images using pre-trained convolutional neural networks, and construct a dynamically weighted adjacency graph based on spatial coordinates to encode spatial proximity.

[0043] In this embodiment, S1 further includes: S1.1 Based on the spatial coordinate matrix, an adaptive weighted adjacency graph is constructed. Spatial proximity is encoded by the Gaussian kernel function. By setting the neighborhood radius threshold and distance decay parameter, a continuous representation of discrete adjacency relationships can be achieved with the help of a continuous weight transformation mechanism. Given a set of spatial locations and its corresponding coordinate matrix ,in, n This represents the total number of spatial sites in the tissue sample. d Representing the dimension of spatial coordinates, the adjacency matrix is ​​constructed using the radial basis function method. :

[0044] in, i and jThese represent the site indices in the spatial site set. and Representing sites i and site j Spatial coordinate vectors in the tissue sample, This represents the neighborhood radius threshold, which is adaptively adjusted based on tissue density. The distance decay parameter is set to 1.5 times the standard deviation of the Euclidean distance, and a continuous representation of discrete adjacency relationships is achieved by means of a continuous weight transformation mechanism. S1.2. Graph convolution feature extraction is performed on the original gene expression matrix. Adjacency matrix with added self-loops and trainable weights are used to achieve hierarchical feature aggregation through a two-layer graph convolution network to output gene expression features. Original gene expression matrix ,in, n This represents the total number of spatial sites in the tissue sample. m To represent the gene expression feature dimension corresponding to each spatial site, a self-loop adjacency matrix and its corresponding degree matrix are introduced, and a trainable parameter matrix is ​​defined. Hierarchical feature aggregation is achieved through a two-layer graph convolutional network, expressed as:

[0045]

[0046] Where A represents the original adjacency matrix. This represents the normalized adjacency matrix with self-loops added. Represents the adjacency matrix with self-loops The corresponding degree matrix, This represents the low-dimensional gene expression feature matrix output after graph convolutional encoding, where Encoder represents the multimodal graph convolutional encoding network. For activation functions; S1.3. Perform deep visual feature extraction on histological images, use a pre-trained convolutional neural network to extract local semantic information, and output a fixed-dimensional image feature vector. Given a histological image As the raw input, the convolutional neural network is trained using the ResNet-50 network architecture. The convolution operation process satisfies the following formula:

[0047] in, Represents the first visual neural network The output feature map of the layer, Represents the first in a visual neural network The trainable weight parameter matrix of the convolutional kernel. This represents the convolution operation. This represents a batch normalization operation used to extract local image features, which are then activated by ReLU to obtain a 2048-dimensional image feature vector. .

[0048] S2. Cross-modal feature fusion: In the feature fusion stage, a cross-modal contrastive learning mechanism is designed to optimize modal alignment, and long-range dependencies are modeled by combining single-modal self-attention.

[0049] In this embodiment, S2 further includes: S2.1, Splicing gene expression features and image features to generate an initial fusion representation; All spatial sites Image feature vectors Vertically stacked to form an image feature matrix Then combine the image feature matrix with the gene feature matrix. Perform column-wise concatenation to generate the initial fusion representation matrix. :

[0050] in, This represents a vector concatenation operation. n This represents the total number of spatial sites in the tissue sample. Dimensions representing gene expression characteristics Dimensions representing the features of a histological image; S2.2. Aggregate neighborhood information through multi-layer graph convolution operations to update the site feature representation, as expressed in the following expression:

[0051] in, Indicating the first in the fusion framework Multimodal joint feature representation of layers, Indicates the first The trainable parameter weight matrix of a layered graph convolutional network. This represents the normalized adjacency matrix with self-loops added. Represents a nonlinear activation function; and The current layer features are linearly transformed based on the adjacency matrix with added self-loops. Each site absorbs features from its neighboring sites, and the aggregation result is standardized before being output after nonlinear activation. S2.3 Introduce a graph attention mechanism to model global dependencies and dynamically calculate the similarity weights between sites; First, an attention mechanism is introduced for dynamic weight allocation to depict the nonlinear correlation characteristics among heterogeneous data, expressed as:

[0052]

[0053] in, Indicates attention weights, For normalized exponential functions, For a leaky linear rectified function, It is a learnable mapping vector. This represents a trainable shared weight parameter matrix. This represents the current target spatial location to be processed in the graph structure. Feature vector Indicates the spatial location relative to the target Neighborhood site feature vectors with spatial proximity relationships. This indicates the sites updated by the graph attention mechanism. Enhanced feature vectors, This indicates the relationship between the target spatial location and the adaptive weighted adjacency graph. A set of neighboring sites that have spatial proximity relationships; Secondly, this addresses the enhancement feature matrix generated in the feature fusion stage of a multi-layer graph convolutional network. The expression for the weighted fusion of gene and image features through a multi-head self-attention mechanism is as follows:

[0054]

[0055] in, Indicates the first The output of each attention head, This represents the output of the self-attention mechanism. , , They represent the first The trainable projective weight matrix for each query, key, and value. Let T be the dimension of the key vector, and T be the matrix transpose. This indicates a multi-head attention mechanism. This indicates a feature concatenation operation. For the total number of attention heads, This represents the output linear projection matrix after multi-head fusion.

[0056] S3. Contrastive Learning and Joint Optimization: For spatial clustering tasks, a joint objective function is proposed, which integrates zero-inflated negative binomial distribution reconstruction loss, graph topology constraints and contrastive loss, and uses graph embedding spatial K-means clustering algorithm to accurately partition the spatial domain.

[0057] In this embodiment, S3 further includes: S3.1 Design a cross-modal contrastive loss function to dynamically select positive and negative sample pairs based on spatial proximity and feature similarity; splicing gene feature matrix The OK and image feature matrix The OK To form a joint representation vector To construct the contrastive loss function :

[0058] in, The cosine similarity function is used. For temperature coefficient, For site i The joint representation vector after splicing gene and image information. For anchor point samples Semantically consistent positive sample pairs This represents the total number of negative samples used in the comparison calculation. k Index for negative samples For anchor point samples The first semantic inconsistency k One negative sample pair; S3.2, Loss of integrated gene expression remodeling Comparative loss and Graph Laplacian Regularization Constraints Construct a joint loss function for multiple tasks. The expression is:

[0059]

[0060]

[0061] in, , and Let represent the hyperparameter weights of the reconstruction loss, contrastive loss, and regularization term, respectively. n This represents the total number of spatial sites in the tissue sample. mThis represents the dimension of gene expression characteristics corresponding to each spatial location. Represents the first gene in the original gene expression matrix. The locus The actual expression level of each gene , and Let these represent the mean, divergence parameter, and zero-inflation probability of the negative binomial distribution generated by the feature reconstruction decoding network, respectively. For indicator functions, The standard negative binomial probability density function, L Let A denote the graph Laplacian matrix, A denote the self-loop adjacency matrix, and D denote the degree matrix corresponding to A. The feature reconstruction decoding network consists of a parallel multilayer perceptron structure, and the decoding network uses a low-dimensional gene expression feature matrix obtained through graph convolutional encoding. As input, respectively map the mean matrix required for the zero-inflated negative binomial distribution to the output. , divergence matrix and the zero-inflation probability matrix The specific mapping relationship expression is as follows:

[0062]

[0063]

[0064] in, , and These represent multilayer perceptrons with independent weights. It is a natural exponential function. For activation functions; S3.3, Enhance feature discriminativeness through a difficult negative sample selection strategy, expressed as:

[0065] in, Indicates the determination site i with site j An indicator function for whether it is a hard negative sample. Indicates the first to be compared j Feature vectors of each site and These are all pseudo-labels for cell types generated by the model's pre-clustering in the current iteration round. This is an adaptive threshold.

[0066] To verify the feasibility and superiority of this method, the present invention tests the performance of the method from the perspective of cell type clustering.

[0067] I. Dataset and Evaluation Metrics This invention was experimentally validated on the 10x Visium mouse brain coronal slice dataset, the Slide-seq mouse primary visual cortex dataset, and the human embryonic liver STereo-seq dataset.

[0068] The 10x Visium dataset contains 10 tissue slices, each containing 600-800 spatial sites, covering more than 15,000 gene expression profiles. It is accompanied by H&E staining images with a resolution of 5μm / pixel and precise spatial coordinates, and annotates 8 typical functional brain regions such as the cerebral cortex and hippocampus.

[0069] The Slide-seq dataset achieves single-cell spatial resolution. Gene expression features are reduced to 50 dimensions by principal component analysis, exhibiting significant spatial heterogeneity.

[0070] The human embryonic liver STereo-seq dataset integrates 10x single-cell RNA sequencing validation results, containing 20,123 cells, covering subcellular localization images and spatial coordinates, and systematically annotating regions of 6 key stages of embryonic liver development.

[0071] All datasets were manually labeled with real spatial domain labels to ensure the reliability of the experiments.

[0072] This invention uses three types of indicators to comprehensively evaluate algorithm performance: 1. Clustering accuracy metrics: ARI quantifies the similarity between the clustering label and the true label of any two site pairs, NMI measures the consistency of information between the clustering result and the true label, and AMI introduces the distribution of the true label to correct NMI; 2. Spatial structure retention index: Spatial neighborhood retention measures the probability that adjacent sites belong to the same cluster after clustering, and RC assesses the spatial distribution density; 3. Feature validity indicators: t-SNE dimensionality reduction visualization analysis of the separation degree of low-dimensional spatial feature clusters, and gene set enrichment analysis to verify the biological interpretability of spatial domain functions.

[0073] II. Analysis and Comparison of Experimental Results This method was systematically compared with several mainstream methods. The comparison methods included K-means, HC, DBSCAN, GCN clustering, GAT clustering, SCGNN, and multimodal fusion algorithms, including gene expression-only input, gene + spatial location input, and gene + image input.

[0074] K-means is a classic single-cell clustering method that divides single-cell data into K populations by randomly initializing K cluster centers, assigning each cell to the nearest center, updating the centers to the cluster mean, and iteratively optimizing until convergence.

[0075] HC uses the Ward minimum variance method as its clustering and merging strategy, and automatically determines the number of clusters using the Calinski-Harabasz index.

[0076] DBSCAN is based on the density reachability principle and sets the neighborhood radius. =0.3 spatial units, minimum sample size MinPts=5, to identify clusters of arbitrary shapes.

[0077] GCN clustering uses a two-layer graph convolutional network combined with the K-means algorithm, and only uses gene expression maps for feature aggregation.

[0078] GAT clustering integrates graph attention network and spectral clustering, constructs an adjacency matrix based on spatial distance, and dynamically allocates node weights.

[0079] SCGNN is a semi-supervised graph neural network that uses some real labels to initialize model parameters, enhancing clustering guidance.

[0080] Table 1. Performance Comparison of Non-Graph Basis Clustering Algorithms on the Visium Dataset

[0081] As can be seen from Table 1, traditional algorithms have blurred boundaries when processing elongated structures such as the hippocampus, with spatial signal-to-noise ratios all below 0.55, regional compactness not reaching the 0.7 threshold, and fragmented clustering results, failing to effectively integrate spatial correlation and multimodal data.

[0082] Table 2. Performance comparison of this method with other graph-based clustering algorithms on the Slide-seq dataset.

[0083] As shown in Table 2, the ARI of GCN clustering is 0.712, NMI is 0.765, SNR is 0.682, and the false discovery rate (GSEA FDR) is 0.082; the ARI of GAT clustering is 0.735, NMI is 0.781, SNR is 0.723, and GSEA FDR is 0.065; the ARI of SCGNN is 0.748, NMI is 0.790, SNR is 0.731, and GSEA FDR is 0.058; while the ARI of our proposed method is 0.789, NMI is 0.832, SNR is 0.815, and GSEA FDR is 0.032. Driven by the cross-modal contrastive learning mechanism, our proposed method reduces the GSEA FDR by 45% compared to SCGNN clustering.

[0084] Table 3. Performance comparison of our method and different modality fusion algorithms on the STereo-seq dataset.

[0085] As shown in Table 3, the ARI for gene expression input alone is 0.712 and the NMI is 0.765, indicating problems with key region identification, such as blurred hippocampal boundaries. The ARI for gene + spatial location input is 0.758 and the NMI is 0.802, but cortical layering is unclear. The ARI for gene + image input is 0.743 and the NMI is 0.791, showing fragmented nucleus morphology. In contrast, the three-modal fusion strategy of our proposed method achieves an ARI of 0.789 and an NMI of 0.832, with no significant errors. Our method dynamically integrates heterogeneous data through an attention-weighted mechanism, improving the ARI by 3.2% compared to earlier fusion strategies and the NMI by 2.7% compared to later fusion strategies.

[0086] A multimodal joint optimization clustering system for spatial transcriptomics, applicable to the multimodal joint optimization clustering method for spatial transcriptomics, includes: A spatial map construction module is used to acquire the original gene expression matrix, spatial coordinate matrix, and histological images from spatial transcriptome data. A multimodal feature fusion module, connected to the spatial graph construction module, is used to receive and splice gene expression features and image features to generate an initial fused representation; A joint optimization module, connected to the multimodal feature fusion module, designs a cross-modal contrastive loss function based on cross-modal interaction features to dynamically select positive and negative sample pairs; The spatial domain partitioning module, which is connected to the joint optimization module, achieves accurate spatial domain partitioning in the graph embedding space based on the K-means clustering algorithm and outputs the organizational functional region analysis results.

[0087] In this embodiment, the spatial map construction module obtains the original gene expression matrix, spatial coordinate matrix, and histological image, and performs the following processing respectively: An adaptive weighted adjacency graph is constructed based on a spatial coordinate matrix, and spatial proximity is encoded by a Gaussian kernel function. The neighborhood radius threshold is adaptively adjusted according to the tissue density, and the distance decay parameter is set to 1.5 times the standard deviation of the Euclidean distance. A self-loop adjacency matrix is ​​added to the original gene expression matrix, and gene expression features are output through a two-layer graph convolutional network. A pre-trained ResNet-50 network was used to extract 2048-dimensional feature vectors from histological images. After convolutional layers and batch normalization, the image features were output.

[0088] In this embodiment, the generation of the initial fused representation in the multimodal feature fusion module first aggregates neighborhood information through multi-layer graph convolution operations, then introduces a graph attention mechanism to model global dependencies, and finally outputs cross-modal interaction features through multi-head self-attention weighted fusion.

[0089] In this embodiment, the joint optimization module integrates gene reconstruction loss, contrast loss, and graph Laplacian regularization constraints to construct a multi-task joint loss function, and enhances feature discriminativeness based on a difficult negative sample selection strategy.

[0090] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the multimodal joint optimization clustering method for spatial transcriptomics described above.

[0091] See Figure 3 The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal joint optimization clustering method for spatial transcriptomics as described above.

[0092] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the multimodal joint optimization clustering method for spatial transcriptomics described above.

[0093] It is understood that the systems, devices and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the above-mentioned multimodal joint optimization clustering method for spatial transcriptomics.

[0094] It should be noted that those skilled in the art will understand that all or part of the steps implemented in the embodiments of the present invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented in hardware, it can be implemented entirely or partially by purchasing standard parts or modifications. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0095] In summary, this invention addresses key issues in existing technologies, such as low efficiency of cross-modal information fusion, insufficient spatial correlation modeling, and ambiguous boundary recognition. It proposes a multimodal joint optimization clustering method for spatial transcriptome data. Specifically, this method first constructs a multimodal feature embedding framework, extracts hierarchical features from gene expression data using a graph convolutional network, captures local semantic features of histological images using a pre-trained convolutional neural network, and constructs a dynamically weighted adjacency graph based on spatial coordinates to encode spatial proximity. In the feature fusion stage, a cross-modal contrastive learning mechanism is designed to optimize modality alignment, combined with single-modal self-attention to model long-range dependencies. For the spatial clustering task, a joint objective function is proposed, integrating zero-inflated negative binomial distribution reconstruction loss, graph topological constraints, and contrastive loss. Finally, the graph embedding spatial K-means algorithm achieves accurate spatial domain partitioning, providing an efficient solution for spatial domain analysis of complex biological tissues.

[0096] It should be understood that the examples and embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various modifications or changes based on them. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A multimodal joint optimization clustering method for spatial transcriptomics, characterized in that, Includes the following steps: S1. Construct a multimodal feature embedding framework, extract hierarchical features from gene expression data through graph convolutional networks, capture local semantic features of histological images using pre-trained convolutional neural networks, and construct a dynamic weighted adjacency graph based on spatial coordinates to encode spatial proximity. S2. In the feature fusion stage, a cross-modal contrastive learning mechanism is designed to optimize modal alignment, and long-range dependencies are modeled by combining single-modal self-attention. S3. For spatial clustering tasks, a joint objective function is proposed, which integrates zero-inflated negative binomial distribution reconstruction loss, graph topology constraints and contrast loss, and uses graph embedding spatial K-means clustering algorithm to accurately partition the spatial domain.

2. The multimodal joint optimization clustering method for spatial transcriptomics as described in claim 1, characterized in that, S1 further includes: S1.1 Based on the spatial coordinate matrix, an adaptive weighted adjacency graph is constructed. Spatial proximity is encoded by the Gaussian kernel function. By setting the neighborhood radius threshold and distance decay parameter, a continuous representation of discrete adjacency relationships can be achieved with the help of a continuous weight transformation mechanism. Given a set of spatial locations and its corresponding coordinate matrix ,in, n This represents the total number of spatial sites in the tissue sample. d Representing the dimension of spatial coordinates, the adjacency matrix is ​​constructed using the radial basis function method. : in, i and j These represent the site indices in the spatial site set. and Representing sites i and site j Spatial coordinate vectors in the tissue sample, Indicates the neighborhood radius threshold. This represents the distance attenuation parameter; S1.

2. Graph convolution feature extraction is performed on the original gene expression matrix. Adjacency matrix with added self-loops and trainable weights are used to achieve hierarchical feature aggregation through a two-layer graph convolution network to output gene expression features. Original gene expression matrix ,in, n This represents the total number of spatial sites in the tissue sample. m To represent the gene expression feature dimension corresponding to each spatial site, a self-loop adjacency matrix and its corresponding degree matrix are introduced, and a trainable parameter matrix is ​​defined. Hierarchical feature aggregation is achieved through a two-layer graph convolutional network, expressed as: Where A represents the original adjacency matrix. This represents the normalized adjacency matrix with self-loops added. Represents the adjacency matrix with self-loops The corresponding degree matrix, This represents the low-dimensional gene expression feature matrix output after graph convolutional encoding, where Encoder represents the multimodal graph convolutional encoding network. For activation functions; S1.

3. Perform deep visual feature extraction on histological images, use a pre-trained convolutional neural network to extract local semantic information, and output a fixed-dimensional image feature vector. Given a histological image As the raw input, the convolutional neural network is trained using the ResNet-50 network architecture. The convolution operation process satisfies the following formula: in, Represents the first visual neural network The output feature map of the layer, Represents the first in a visual neural network The trainable weight parameter matrix of the convolutional kernel. This represents the convolution operation. This represents a batch normalization operation used to extract local image features, which are then activated by ReLU to obtain a 2048-dimensional image feature vector. .

3. The multimodal joint optimization clustering method for spatial transcriptomics as described in claim 1, characterized in that, S2 further includes: S2.1, Splicing gene expression features and image features to generate an initial fusion representation; All spatial sites Image feature vectors Vertically stacked to form an image feature matrix Then combine the image feature matrix with the gene feature matrix. Perform column-wise concatenation to generate the initial fusion representation matrix. : in, This represents a vector concatenation operation. n This represents the total number of spatial sites in the tissue sample. Dimensions representing gene expression characteristics Dimensions representing the features of a histological image; S2.

2. Aggregate neighborhood information through multi-layer graph convolution operations to update the site feature representation, as expressed in the following expression: in, Indicating the first in the fusion framework Multimodal joint feature representation of layers, Indicates the first The trainable parameter weight matrix of a layered graph convolutional network. This represents the normalized adjacency matrix with self-loops added. Represents a nonlinear activation function; and The current layer features are linearly transformed based on the adjacency matrix with added self-loops. Each site absorbs features from its neighboring sites, and the aggregation result is standardized before being output after nonlinear activation. S2.3 Introduce a graph attention mechanism to model global dependencies and dynamically calculate the similarity weights between sites; First, an attention mechanism is introduced for dynamic weight allocation to depict the nonlinear correlation characteristics among heterogeneous data, expressed as: in, Indicates attention weights, For normalized exponential functions, For a leaky linear rectified function, It is a learnable mapping vector. This represents a trainable shared weight parameter matrix. This represents the current target spatial location to be processed in the graph structure. Feature vector Indicates the spatial location relative to the target Neighborhood site feature vectors with spatial proximity relationships. This indicates the sites updated by the graph attention mechanism. Enhanced feature vectors, This indicates the relationship between the target spatial location and the adaptive weighted adjacency graph. A set of neighboring sites that have spatial proximity relationships; Secondly, this addresses the enhancement feature matrix generated in the feature fusion stage of a multi-layer graph convolutional network. The expression for the weighted fusion of gene and image features through a multi-head self-attention mechanism is as follows: in, Indicates the first The output of each attention head, This represents the output of the self-attention mechanism. , , They represent the first The trainable projective weight matrix for each query, key, and value. Let T be the dimension of the key vector, and T be the matrix transpose. This indicates a multi-head attention mechanism. This indicates a feature concatenation operation. For the total number of attention heads, This represents the output linear projection matrix after multi-head fusion.

4. The multimodal joint optimization clustering method for spatial transcriptomics as described in claim 1, characterized in that, S3 further includes: S3.1 Design a cross-modal contrastive loss function to dynamically select positive and negative sample pairs based on spatial proximity and feature similarity; splicing gene feature matrix The OK and image feature matrix The OK To form a joint representation vector To construct the contrastive loss function : in, The cosine similarity function is used. For temperature coefficient, For site i The joint representation vector after splicing gene and image information. For anchor point samples Semantically consistent positive sample pairs This represents the total number of negative samples used in the comparison calculation. k Index for negative samples For anchor point samples The first semantic inconsistency k One negative sample pair; S3.2, Loss of integrated gene expression remodeling Comparative loss and Graph Laplacian Regularization Constraints Construct a joint loss function for multiple tasks. The expression is: in, , and Let represent the hyperparameter weights of the reconstruction loss, contrastive loss, and regularization term, respectively. n This represents the total number of spatial sites in the tissue sample. m This represents the dimension of gene expression characteristics corresponding to each spatial location. Represents the first gene in the original gene expression matrix. The locus The actual expression level of each gene , and Let these represent the mean, divergence parameter, and zero-inflation probability of the negative binomial distribution generated by the feature reconstruction decoding network, respectively. For indicator functions, The standard negative binomial probability density function, L Let A denote the graph Laplacian matrix, A denote the self-loop adjacency matrix, and D denote the degree matrix corresponding to A. The feature reconstruction decoding network consists of a parallel multilayer perceptron structure, and the decoding network uses a low-dimensional gene expression feature matrix obtained through graph convolutional encoding. As input, respectively map the mean matrix required for the zero-inflated negative binomial distribution to the output. , divergence matrix and the zero-inflation probability matrix The specific mapping relationship expression is as follows: in, , and These represent multilayer perceptrons with independent weights. It is a natural exponential function. For activation functions; S3.3, Enhance feature discriminativeness through a difficult negative sample selection strategy, expressed as: in, Indicates the determination site i with site j An indicator function for whether it is a hard negative sample. Indicates the first to be compared j Feature vectors of each site and These are all pseudo-labels for cell types generated by the model's pre-clustering in the current iteration round. This is an adaptive threshold.

5. A multimodal joint optimization clustering system for spatial transcriptomics, applicable to the multimodal joint optimization clustering method for spatial transcriptomics as described in any one of claims 1-4, characterized in that, include: A spatial map construction module is used to acquire the original gene expression matrix, spatial coordinate matrix, and histological images from spatial transcriptome data. A multimodal feature fusion module, connected to the spatial graph construction module, is used to receive and splice gene expression features and image features to generate an initial fused representation; A joint optimization module, connected to the multimodal feature fusion module, designs a cross-modal contrastive loss function based on cross-modal interaction features to dynamically select positive and negative sample pairs; The spatial domain partitioning module, which is connected to the joint optimization module, achieves accurate spatial domain partitioning in the graph embedding space based on the K-means clustering algorithm and outputs the organizational functional region analysis results.

6. The multimodal joint optimization clustering system for spatial transcriptomics as described in claim 5, characterized in that, The spatial map construction module obtains the original gene expression matrix, spatial coordinate matrix, and histological image, which are then processed as follows: An adaptive weighted adjacency graph is constructed based on a spatial coordinate matrix, and spatial proximity is encoded by a Gaussian kernel function. The neighborhood radius threshold is adaptively adjusted according to the tissue density, and the distance decay parameter is set to 1.5 times the standard deviation of the Euclidean distance. A self-loop adjacency matrix is ​​added to the original gene expression matrix, and gene expression features are output through a two-layer graph convolutional network. A pre-trained ResNet-50 network was used to extract 2048-dimensional feature vectors from histological images. After convolutional layers and batch normalization, the image features were output.

7. The multimodal joint optimization clustering system for spatial transcriptomics as described in claim 5, characterized in that, The generation of the initial fused representation in the multimodal feature fusion module first aggregates neighborhood information through multi-layer graph convolution operations, then introduces a graph attention mechanism to model global dependencies, and finally outputs cross-modal interaction features through multi-head self-attention weighted fusion.

8. The multimodal joint optimization clustering system for spatial transcriptomics as described in claim 5, characterized in that, In the joint optimization module, a multi-task joint loss function is constructed by integrating gene reconstruction loss, contrast loss and graph Laplacian regularization constraint, and a difficult negative sample selection strategy is used to enhance feature discriminativeness.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the multimodal joint optimization clustering method for spatial transcriptomics as described in any one of claims 1-4.

Images