Spatial Transcriptome Data Clustering Method Based on Progressive Learning and Multimodal Fusion

Abstract

The present invention relates to the technical field of spatial transcriptome data clustering analysis, and discloses a spatial transcriptome data clustering method based on progressive learning and multimodal fusion. The present invention uses a multi-layer graph convolutional network to capture the high-order structural relationship between each sample point, and then removes various noises in the gene expression data itself to the greatest extent through contrast learning and complementary masking mechanism; at the same time, through two cross-attentions, it cleverly achieves the maximum fusion of gene expression data, spatial position information and histological image information, that is, obtains the fused feature latent representation Latent. Compared with the mainstream spatial transcriptomics clustering analysis method, the present invention shows stronger ability and excellent clustering performance based on the fused feature latent representation Latent for clustering, and has significant robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of spatial transcriptome data clustering methods, and in particular to a spatial transcriptome data clustering method based on progressive learning and multimodal fusion. Background Art

[0002] Spatial transcriptomics (ST) is one of the most significant breakthroughs in bioinformatics in recent years. Unlike traditional single-cell transcriptome sequencing (scRNA-seq), ST not only provides gene expression information but also preserves the spatial relationships of cells within a tissue. This technology plays a key role in revealing intercellular interactions, tissue microenvironmental structure, and the spatial heterogeneity of disease. It has been widely applied in various fields, including cancer research, neuroscience, and developmental biology.

[0003] Identifying shared and specific spatial domains (clusters with similar spatial expression patterns) is a key task in the comprehensive analysis of ST datasets. Recent advances in spatially resolved transcriptomics have enabled comprehensive measurement of gene expression patterns while preserving the spatial context of the tissue microenvironment. However, due to limitations in sequencing technology, spatial transcriptomics data are highly sparse and exhibit complex noise patterns. Traditional clustering methods such as kmeans and hierarchical clustering, which rely on similarity metrics, are not well suited for clustering single-cell data.

[0004] To better identify the spatial domain of spatial transcriptomics data, deep clustering methods such as SpaGCN, GraphST, SEDR, and STAGATE have been proposed and used for tissue spatial domain clustering tasks, further utilizing the spatial location information of spatial transcriptomics data to cluster spatial transcriptomics data. However, histological images of spatial transcriptomics provide an intuitive, anatomical spatial context for gene expression data. This combination can help models more comprehensively understand the relationship between gene expression and tissue structure, but these methods have ignored this important modality of data.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a spatial transcriptome data clustering method based on progressive learning and multimodal fusion, aiming to solve the problem of poor clustering effect of existing spatial transcriptomics clustering analysis methods.

[0007] The first aspect of the present invention provides a spatial transcriptome data clustering method based on progressive learning and multimodal fusion, comprising: obtaining a histological image and its corresponding spatial transcriptome data; cutting the histological image into blocks according to the size of the sample points, and then extracting histological image feature information from each block through a medical image large model UNI. Calculate the spatial Euclidean distance between each sample point based on the spatial position information of each sample point, select the k nearest sample points of each sample point as neighbors, and construct a spatial adjacency matrix A; screen and regularize the gene expression data of the sample points in the spatial transcriptome data, select the gene expression matrix of the top 3000 highly variable genes for principal component analysis, and extract the top 200 principal component gene expression matrices as gene expression information X; randomly mask each row of the gene expression information X according to a predetermined ratio to obtain a masked gene expression matrix , and then perform complementary masking on the gene expression information X to obtain the complementary masked gene expression matrix ; Construct a neural network model, the neural network model includes a first encoder, a second encoder, a third encoder, a decoder, a first cross attention fusion module, a second cross attention fusion module, a progressive learning module and a contrastive learning module; with the spatial adjacency matrix A, the histological image feature information , masked gene expression matrix and complementary masked gene expression matrices As input data, the neural network model is trained with reconstruction loss and contrast loss as the total loss function to obtain a trained neural network model; the histological image to be analyzed and the corresponding spatial transcriptome data are preprocessed to obtain the histological image feature information , the spatial adjacency matrix A and the gene expression information X are input into the trained neural network model, and a fusion feature latent representation is output; cluster analysis is performed on the fusion feature latent representation, and a clustering result is output.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the spatial adjacency matrix, histological image feature information, mask gene expression matrix and complementary mask gene expression matrix are used as input data, and reconstruction loss and contrast loss are used as the total loss function to train the neural network model to obtain a trained neural network model, including the steps of: inputting the histological image feature information and the spatial adjacency matrix into the first encoder, and outputting the image feature information latent representation; inputting the mask gene expression matrix and the spatial adjacency matrix into the second encoder, and inputting the output mask gene latent representation into the third encoder after re-masking processing to obtain a re-masked gene latent representation; inputting the re-masked gene latent representation into the progressive learning module, and outputting the average re-masked gene latent representation. The method comprises the following steps: inputting the average re-masked gene latent representation and the image feature information latent representation into the first cross-attention fusion module, outputting a fused latent representation; inputting the first fused latent representation and the re-masked gene latent representation into the second cross-attention fusion module, outputting a fused feature latent representation; inputting the fused feature latent representation into the decoder, outputting a reconstructed feature representation; inputting the complementary masked gene expression matrix and the spatial adjacency matrix into the second encoder, outputting a complementary masked gene latent representation; inputting the complementary masked gene latent representation and the re-masked gene latent representation into the contrastive learning module, outputting the cosine similarity of the positive sample pair and the negative sample pair, and constructing a contrast loss based on the cosine similarity of the positive sample pair and the negative sample pair. ; Construct reconstruction loss based on the reconstruction feature representation ; Based on the reconstruction loss function And contrast loss function Constructing the total loss function The neural network model is trained to obtain a trained neural network model, wherein: and It is a hyperparameter that controls the proportion of reconstruction loss and contrast loss in the total loss.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the histological image feature information and the spatial adjacency matrix are input into the first encoder, and a latent representation of the image feature information is output, comprising the steps of: the first encoder including several layers of graph convolution layers, performing a convolution operation on the input histological image feature information and the spatial adjacency matrix through the several layers of graph convolution layers to obtain a latent representation of the image feature information, wherein the convolution operation expression is: ,in, is the feature representation of all sample points in the current layer, It represents the feature representation of all sample points after a convolution operation. , , I is the identity matrix, yes The degree matrix of is the activation function, is the trainable parameter matrix of the current layer.

[0010] Optionally, in a third implementation manner of the first aspect of the present invention, the masked gene expression matrix and the spatial adjacency matrix are input into the second encoder, and the output masked gene latent representation is input into the third encoder after re-masking to obtain a re-masked gene latent representation, including the steps of: the second encoder includes several layers of graph convolution layers, and the input masked gene expression matrix and the spatial adjacency matrix are convolved through the several layers of graph convolution layers to obtain the masked gene latent representation; the masked gene latent representation is re-masked, and the masked rows of the masked gene latent representation are the same as the masked rows in the masked gene table matrix to obtain a re-masked gene latent representation; the third encoder includes several layers of graph convolution layers, and the input re-masked gene latent representation is convolved through the several layers of graph convolution layers to obtain a re-masked gene latent representation.

[0011] Optionally, in a fourth implementation of the first aspect of the present invention, the re-masked gene latent representation is input into a progressive learning module, and the average re-masked gene latent representation is output, comprising the steps of: the progressive learning module increases the number of training rounds according to the spatial adjacency matrix A certain percentage of neighboring points are randomly selected for each sample point, and the re-masked gene latent representations of the neighboring points are averaged to obtain the average re-masked gene latent representation, where the percentage is set according to the following formula: ,in, The number of epochs required for training the neural network model, The current training round of the neural network model.

[0012] Optionally, in a fifth implementation of the first aspect of the present invention, the average re-masked gene latent representation and the image feature information latent representation are input into the first cross-attention fusion module, and a fused latent representation is output, including the steps of: the first cross-attention fusion module is used to capture the correlation between different feature spaces, and its calculation formula is as follows: , where Q, K, and V are query matrix, key matrix, and value matrix respectively. is a weight matrix, Representation matrix connection operation; input the average re-masked gene latent representation and the image feature information latent representation into the first cross-attention fusion module, and output a fused latent representation, wherein the image feature information latent representation serves as the data source of the query matrix Q, and the average re-masked gene latent representation serves as the data source of the key matrix K and the value matrix V.

[0013] Optionally, in a sixth implementation of the first aspect of the present invention, the complementary masked gene latent representation and the re-masked gene latent representation are input into the contrastive learning module, the cosine similarity of the positive sample pair and the negative sample pair is output, and a contrast loss is constructed based on the cosine similarity of the positive sample pair and the negative sample pair. , comprising the steps of: selecting corresponding sample rows in the complementary masked gene latent representation and the re-masked gene latent representation according to the masked sample rows in the re-masked gene latent representation to construct positive sample pairs, and randomly extracting sample rows that are inconsistent with the positive sample pairs from the complementary masked gene latent representation and the re-masked gene latent representation as negative sample pairs; according to the formula Calculate the cosine similarity of positive sample pairs and negative sample pairs, where (A, B) is a sample pair, Is the modulus symbol; by pulling the positive sample closer to the cosine similarity , push the negative sample to cosine similarity Contrastive loss is constructed in the following way .

[0014] The second aspect of the present invention provides a spatial transcriptome data clustering device based on progressive learning and multimodal fusion, comprising: a data acquisition module for acquiring histological images and their corresponding spatial transcriptome data; an image processing module for cutting the histological images into blocks according to the size of the sample points, and then extracting histological image feature information from each block through the medical image large model UNI. ; An adjacency matrix construction module is used to calculate the spatial Euclidean distance between each sample point based on the spatial position information of each sample point, select the k nearest sample points of each sample point as neighbors, and construct a spatial adjacency matrix A; a gene expression data processing module is used to screen and regularize the gene expression data of the sample points in the spatial transcriptome data, select the gene expression matrix of the top 3000 highly variable genes for principal component analysis, and extract the top 200 principal component gene expression matrices as gene expression information X; a masking module is used to randomly mask each row of the gene expression information X according to a predetermined ratio to obtain a masked gene expression matrix , and then perform complementary masking on the gene expression information X to obtain the complementary masked gene expression matrix ; A model construction module for constructing a neural network model, wherein the neural network model includes a first encoder, a second encoder, a third encoder, a decoder, a first cross attention fusion module, a second cross attention fusion module, a progressive learning module, and a contrastive learning module; a model training module for using the spatial adjacency matrix A, the histological image feature information , masked gene expression matrix and complementary masked gene expression matrices As input data, the neural network model is trained with reconstruction loss and contrast loss as the total loss function to obtain a trained neural network model; a feature output module is used to obtain the histological image feature information obtained after preprocessing the histological image to be analyzed and the corresponding spatial transcriptome data , the spatial adjacency matrix A and the gene expression information X are input into the trained neural network model to output the fusion feature latent representation; an aggregation module is used to perform cluster analysis on the fusion feature latent representation and output the clustering result.

[0015] A third aspect of the present invention provides a spatial transcriptome data clustering device based on progressive learning and multimodal fusion, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected through a line; the at least one processor calls the computer-readable instructions in the memory so that the spatial transcriptome data clustering device based on progressive learning and multimodal fusion performs the various steps of the spatial transcriptome data clustering method based on progressive learning and multimodal fusion as described above.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed on a computer, the computer executes the various steps of the spatial transcriptome data clustering method based on progressive learning and multimodal fusion as described above.

[0017] Beneficial Effects: The present invention provides a spatial transcriptome data clustering method based on progressive learning and multimodal fusion. First, by introducing a graph convolutional network as an encoder and decoder, the present invention allows the model to fully extract information from adjacent nodes to capture higher-order structural information between sample points. Gene expression information and image feature information are separately fused with structural information (spatial adjacency matrix), allowing the two modalities to first complete the extraction of their own spatial information. Second, when encoding gene expression information through the GCN, the present invention uses complementary masked data. Complementary data maximizes sample utilization and provides the model with different perspectives to learn its data structure. This makes it easier for the model to capture key patterns in the data and helps eliminate potential technical noise (such as sequencing errors) or biological noise. Furthermore, because the masking generates missing data, the complementary mask ensures that the model maintains spatial consistency during training. This ensures that the learned fused feature latent representations do not destroy the spatial structure due to the randomness of missing values, promoting the model to utilize more spatial information. Furthermore, the model learns to capture key patterns in the data even when some information is missing, thereby improving generalization and robustness. Third, the present invention designs a progressive learning module, which is a design based on the idea of designing from local to global. As the number of training rounds increases, this module will randomly provide adjacent node information from few to many for the next modal fusion. Due to the large modal differences between multimodal features, if all features are jointly modeled directly in one stage, the model may be difficult to converge or easily fall into local optimality. Randomly selecting from few to many can first perform preliminary alignment or pre-training of the representations of each part, and then gradually "unfreeze" or "combine" more elements, so that relatively stable and more controllable updates can be obtained at each stage. At the same time, it also avoids the model "thinking too much about the problem first and ignoring the global structure." Finally, the present invention also designs a dual cross-attention modal fusion mechanism, in which the first cross-attention fusion module enables image features to perceive the overall distribution or global semantics of gene expression in the "preliminary" stage, thereby "highlighting" the parts of image features that are highly correlated with overall gene expression, which is equivalent to a coarse-grained or "global" cross-modal fusion. The model first learns the correspondence between the overall image and the global information of gene expression, allowing the image representation to first integrate a macro-understanding of gene expression; the second cross-attention fusion module allows the model to "finely" align the image with specific dimensions or pathways of gene expression, thereby capturing more discriminative or more explanatory cross-modal patterns, which is equivalent to a fine-grained cross-modal fusion; on the premise that the model has obtained the global context in the first step, it can better grasp the correspondence between the various pathways / gene dimensions of gene expression and the local features of the image.The fusion feature latent representation obtained based on the model showed stronger ability and excellent clustering performance in cell clustering after being used for cluster analysis, and had significant robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flowchart of a spatial transcriptome data clustering method based on progressive learning and multimodal fusion provided by an embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the principle of the spatial transcriptome data clustering method based on progressive learning and multimodal fusion of the present invention.

[0020] Figure 3 Figure 3 is a diagram of the ARI and NMI clustering evaluation results of the DLPFC (human dorsolateral prefrontal cortex) multi-slice dataset under different methods, where A is the manually labeled classification of 151,675 DLPFC slices; B is a box plot of the clustered ARI and NMI evaluation results of all slices of the DLPFC multi-slice dataset under different methods (the method of the present invention, CCST, Deep ST, DiffusionST, GraphST, SEDR, SpaGCN, STAGATE); C is a visual display of the spatial domain clustering results of 151,675 DLPFC slices by each method.

[0021] Figure 4 The ARI and NMI clustering evaluation results of the single slice dataset under different methods are shown in Figure 2. Figure 4 The left side is the ARI and NMI clustering evaluation results of the BRCA (human breast cancer) single slice dataset under different methods. Figure 4 The middle right side shows the ARI and NMI clustering evaluation results of the MBA (rat brain) single slice dataset under different methods.

[0022] Figure 5 A schematic diagram of the structure of a spatial transcriptome data clustering device based on progressive learning and multimodal fusion provided by an embodiment of the present invention.

[0023] Figure 6 A schematic diagram of the structure of a spatial transcriptome data clustering device based on progressive learning and multimodal fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] An embodiment of the present invention provides a spatial transcriptome data clustering method based on progressive learning and multimodal fusion. The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0025] See also Figure 1 , Figure 1 The present invention provides a flow chart of a preferred embodiment of a spatial transcriptome data clustering method based on progressive learning and multimodal fusion, as shown in the figure, which includes the following steps:

[0026] S10, obtaining histological images and their corresponding spatial transcriptome data;

[0027] Specifically, histological images are usually obtained through microscopic imaging techniques (such as H&E staining, fluorescent staining, etc.), showing the morphological characteristics of cells and structures in tissue sections. Their function is to provide a spatial positioning "map" for the spatial analysis of subsequent gene expression data; the spatial transcriptome data is obtained through high-throughput sequencing technology, which records the gene expression spectrum of each sample point (spot) in the tissue section, and the position information (such as coordinates) of each sample point corresponds one-to-one to the physical position in the histological image. For example, a tissue section is placed on a Visium chip, and the surface of the chip is covered with sample points containing oligonucleotide probes. The oligonucleotide probes capture the mRNA released by the section and assign a unique spatial barcode to each sample point. The gene expression spectrum of each sample point is obtained by sequencing, and the expression data is mapped to the corresponding spatial coordinates using the probe barcode. The spatial transcriptome data can be obtained, which corresponds to the gene expression matrix (each row represents a gene, and each column represents a sample point); spatial coordinate data (x / y coordinates of each sample point).

[0028] S20, cutting the histological image into blocks according to the size of the sample points, and then extracting the histological image feature information from each block through the medical image large model UNI ;

[0029] Specifically, the histological image is converted according to the spatial position information corresponding to its sample points. As the center, the blocks are cut according to the size of the sample points, and then the image feature information of each small block is extracted through the medical image large model UNI. . For example, according to the coordinates of the spatial transcriptome data (such as the sample point position of Visium), the image blocks are cropped according to the actual size (such as 55 μm diameter) with each sample point as the center. In this embodiment, the medical image large model UNI is a general pre-training model designed for medical imaging. UNI incorporates prior knowledge of medical images (such as tissue staining patterns and cell arrangement patterns) in pre-training, and can better distinguish pathological features (such as tumor areas and inflammatory responses). UNI is more robust to common noise in medical images (such as uneven staining, slice folding) and artifacts (such as bubbles and knife marks); even if the blocks come from different laboratories or staining protocols (such as H&E vs. IHC), the features extracted by UNI remain consistent, reducing data batch effects.

[0030] S30, calculating the spatial Euclidean distance between each sample point based on the spatial position information of each sample point, selecting the k nearest sample points of each sample point as neighbors, and constructing a spatial adjacency matrix A;

[0031] Specifically, on the spatial transcription slice, the spatial Euclidean distance between sample points is calculated based on the spatial position information of each sample point, and the K nearest neighbor algorithm is used to ensure that each sample point has k (for example, k is 6-12) nearest sample points as neighbors, that is, the spatial adjacency matrix is obtained. On the nth spatial transcription slice there is a spatial adjacency matrix A n , satisfying that the jth sample point is a neighbor of the i-th sample point, then A nij =1, otherwise A nij =0, that is .

[0032] S40, screening and regularizing the gene expression data of the sample points in the spatial transcriptome data, selecting the gene expression matrix of the top 3000 highly variable genes for principal component analysis, and extracting the top 200 principal component gene expression matrices as gene expression information X;

[0033] Specifically, the sample points and genes without expression in the spatial transcriptome data were first screened out, and then the functions provided by the SCANPY library were used to regularize the gene expression data after screening. Then, the gene expression matrix of the top 3000 highly variable genes was selected. , and finally for this gene expression matrix Perform principal component analysis (PCA) and extract the first 200 principal components (PCs) gene expression matrix as the basic input of the model .

[0034] S50, randomly masking each row of the gene expression information X according to a predetermined ratio to obtain a masked gene expression matrix , and then perform complementary masking on the gene expression information X to obtain the complementary masked gene expression matrix ;

[0035] In this embodiment, the gene expression information X includes five sample points, namely, sample point 1, sample point 2, sample point 3, sample point 4, and sample point 5. Figure 2 As shown, first mask the gene expression corresponding to sample points 2 and 4 in the gene expression information X, and obtain the masked gene expression matrix ; Then the gene expressions corresponding to sample points 1, 3, and 5 in the gene expression information X are complementary masked to obtain the complementary masked gene expression matrix , at this time .

[0036] In this embodiment, complementary masked data is used when encoding gene expression information through GCN. Complementary masked data maximizes sample utilization and provides the model with different perspectives to learn its data structure. This makes it easier for the model to capture key patterns in the data and helps eliminate potential technical noise (such as sequencing errors) or biological noise. Furthermore, because the mask generates missing data, the complementary mask ensures that the model maintains spatial consistency during training, preventing the learned latent representations from disrupting the spatial structure due to the randomness of missing values, thus enabling the model to utilize more spatial information. At the same time, the model must learn to capture key patterns in the data even when some information is missing, thereby improving generalization and robustness.

[0037] S60: Construct a neural network model, wherein the neural network model includes a first encoder, a second encoder, a third encoder, a decoder, a first cross attention fusion module, a second cross attention fusion module, a progressive learning module, and a contrastive learning module;

[0038] S70, using the spatial adjacency matrix A and histological image feature information , masked gene expression matrix and complementary masked gene expression matrices As input data, the neural network model is trained with reconstruction loss and contrast loss as the total loss function to obtain a trained neural network model;

[0039] Specifically, if Figure 2As shown, the first encoder includes several layers of graph convolution layers. During the training process, the histological image feature information is converted into The spatial adjacency matrix A is input into the first encoder, and the input data is convolved through the several layers of graph convolution layers to obtain the latent representation of image feature information, where the convolution operation expression is: ,in, is the feature representation of all sample points in the current layer, It represents the feature representation of all sample points after a convolution operation. , , I is the identity matrix, yes The degree matrix of is the activation function, is the trainable parameter matrix of the current layer.

[0040] like Figure 2 As shown, the second encoder also includes several layers of graph convolution layers like the first encoder. During the training process, the mask gene expression matrix and the spatial adjacency matrix A are input into the second encoder, and the input masked gene expression matrix and the spatial adjacency matrix are convolved through the several layers of graph convolution layers to obtain a masked gene latent representation; the masked gene latent representation is then re-masked, and the masked rows of the masked gene latent representation are the same as the masked rows in the masked gene table matrix, to obtain a re-masked gene latent representation; the third encoder includes several layers of graph convolution layers, and the re-masked gene latent representation is input into the third encoder, and the input re-masked gene latent representation is again convolved through the several layers of graph convolution layers to obtain a re-masked gene latent representation. This embodiment can further improve the robustness of the model by re-masking the masked gene expression matrix. The obtained re-masked gene latent representation has multiple functions: it can serve as a positive sample for the contrastive learning module, as a data source for the progressive learning module, and can also be cross-fused with the subsequent fusion latent representation.

[0041] like Figure 2As shown, the progressive learning module is a dynamic information extraction module. Its idea comes from Curriculum Learning (CL). CL is a machine learning method inspired by the human learning process. Its core idea is that the model gradually learns from samples or tasks from simple to complex, from easy to difficult, rather than being directly exposed to all data. This strategy can improve the convergence speed and generalization ability of the model and avoid falling into local optimality. During training, the re-masked gene latent representation is input into the progressive learning module, and the average re-masked gene latent representation can be output. Specifically, the progressive learning module will increase the spatial adjacency matrix according to the increase in training rounds (epochs). A certain percentage (increasing with epoch) of neighboring points is randomly selected for each sample point, and the re-masked gene latent representations of these neighboring points are averaged as the condition for fusion with the latent representation of image feature information; the percentage is set according to the following formula: ,in, The number of epochs required for training the neural network model, The current training round of the neural network model. This design is to allow the model to extract as much information as possible from each neighboring point. Through a gradual rate, the model gradually adapts to the increase in information; and through random selection, the model does not rely solely on the information of a few specific neighboring points, thereby improving the robustness of the model. Finally, provide The 10% margin allows the model to adapt to the full amount of data input; after training is completed, the default model has adapted to the full amount of data input. Set the duration to 1 and never change it.

[0042] This embodiment designs a progressive learning module, which is a design based on the idea of designing from local to global. As the number of training rounds increases, this progressive learning module will randomly provide adjacent node information from small to large for the next modal fusion. Due to the large modal differences between multimodal features, if all features are jointly modeled directly in one stage, the model may be difficult to converge or easily fall into a local optimum. However, this embodiment can first perform preliminary alignment or pre-training on the representation of each part through random selection and from small to large, and then gradually "unfreeze" or "combine" more elements, so that relatively stable and more controllable updates can be obtained at each stage. At the same time, it also avoids the model "thinking too much about the problem first, but ignoring the global structure."

[0043] The latent representations of histological image feature information and gene expression data encoded separately are independent, so how to fuse the two parts of data is a very important issue. The fusion of multimodal data has always been a difficult task. Simply adding the two outputs to obtain a fused representation is a simple approach, but this approach may lose some important expression patterns or structural information, and may even cause a large number of irrelevant dimensions or mismatched noise features to be mixed together. Thanks to the widespread application of Transformer, this embodiment fuses the data of the two modalities through a cross-attention mechanism. Cross-attention will automatically assign higher weights to the parts with high correlation during the matching process, thereby playing a screening and weighting effect, reducing noise, greatly improving the fusion quality, and obtaining a richer representation.

[0044] The attention mechanism simply maps a query to a set of key-value pairs and produces an output, where the query, key, value, and output are all vectors. Specifically, the dot product of the query and key is calculated, and the softmax function is applied to obtain the weight associated with the value. This weight is then multiplied by the value to obtain the final result. To simplify the calculation, a set of key-value pairs is calculated simultaneously during the actual operation, converting vector calculations into matrix operations. The specific calculation method of the attention mechanism is as follows: ,in They are the query matrix, key matrix and value matrix.

[0045] On this basis, the present invention further introduces a multi-head attention mechanism, which can simultaneously focus on the information of each subspace at different locations in the data, so that the obtained attention fusion representation contains richer and more robust information. Perform M operations with different attention operations respectively, then execute the attention functions in parallel, connect them together and project them again to get the final result. The head operation is expressed as follows: ,in, Is the transformation matrix, responsible for projecting the corresponding three matrices respectively; the calculation of each multi-head attention (MultiHead) is as follows: ,in, is a weight matrix, Represents a matrix concatenation operation.

[0046] Cross-Attention is a special attention mechanism that is widely used in tasks such as multimodal learning, sequence alignment, and data fusion. This mechanism allows the query (Query, Q) of one data source to focus on the key (Key, K) and value (Value, V) of another data source, thereby establishing an interactive relationship between the two feature spaces. The main difference between cross-attention and self-attention is that the Q, K, and V of self-attention often come from the same data source, while the Q, K, and V of cross-attention come from different data sources. The first cross-attention fusion module and the second cross-attention module of the present invention both use a cross-attention mechanism to effectively capture the correlation between different modalities or different feature spaces.

[0047] like Figure 2 As shown, the first cross-attention fusion module is used to capture the correlation between different feature spaces. During training, the average re-masked gene latent representation and the image feature information latent representation are input into the first cross-attention fusion module, and a fused latent representation is output, wherein the image feature information latent representation serves as the data source of the query matrix Q, and the average re-masked gene latent representation serves as the data source of the key matrix K and the value matrix V.

[0048] The first fused latent representation and the re-masked gene latent representation are then input into the second cross-attention fusion module, which outputs a fused feature latent representation, Latent. The re-masked gene latent representation serves as the data source for the query matrix Q in the second cross-attention fusion module, and the first fused latent representation serves as the data source for the key matrix K and value matrix V in the second cross-attention fusion module. The fused feature latent representation, Latent, output from the second cross-attention process has important uses. First, during the training phase, it is used as decoder input to reconstruct the original gene expression. Once the training phase is complete, the output fused feature latent representation, Latent, can be used for downstream task analysis such as spatial domain recognition and visual clustering.

[0049] This embodiment designs a dual cross-attention modal fusion mechanism, in which the first cross-attention fusion module enables image features to perceive the overall distribution or global semantics of gene expression in the "preliminary" stage, thereby "highlighting" the parts of the image features that are highly correlated with the overall gene expression. This is equivalent to a coarse-grained or "global" cross-modal fusion. The model first learns the correspondence between the overall image and the global information of gene expression, allowing the image representation to first integrate a macro-understanding of gene expression; the second cross-attention fusion module allows the model to "finely" align the image with the specific dimensions or pathways of gene expression, thereby capturing more discriminative or more explanatory cross-modal patterns, which is equivalent to a fine-grained cross-modal fusion; on the premise that the model has obtained the global context in the first step, it can better grasp the correspondence between the various pathways / gene dimensions of gene expression and the local features of the image.

[0050] like Figure 2 As shown, during the training process, the fusion feature latent representation is input into the decoder, the reconstructed feature representation is output, and the reconstruction loss is constructed based on the reconstructed feature representation. Specifically, the decoder also consists of a multi-layer graph convolutional layer (GCN) as the backbone module, with the fused feature latent representation as the input of this component. The purpose is to reconstruct the original gene expression matrix and obtain the reconstructed feature representation. The process is as follows: , among which ; Once the reconstructed feature representation Z is obtained, it will be used to calculate the reconstruction loss In order to reconstruct the mask features from the given partially observable input features, this embodiment uses scaled cosine error (SCE) as the objective function. The normalized cosine error enhances the stability of embedding representation learning. Under the predefined scale factor γ, the similarity between the reconstructed prediction and the original input is calculated only on the masked gene. The reconstruction loss The mathematical formula is as follows: ,in are the sample points in the masked gene expression matrix, is the reconstructed feature representation corresponding to Each sample point, γ is fixed to 2 throughout the experiment to reduce the weight of each contribution from simple samples during training, and |V| represents the number of nodes in the mask set, means The transpose of .

[0051] like Figure 2 As shown, during the training process, the complementary mask gene expression matrix The spatial adjacency matrix is input into the second encoder to output a complementary masked gene latent representation; the complementary masked gene latent representation and the re-masked gene latent representation are input into the contrastive learning module, the cosine similarity of the positive sample pair and the negative sample pair is output, and the contrast loss is constructed based on the cosine similarity of the positive sample pair and the negative sample pair. . Specifically, contrastive learning is a self-supervised learning method that learns the similarities or differences between samples by constructing positive pairs and negative pairs. The goal is to make similar data points close in the representation space (latent space), and dissimilar data points far away. In this embodiment, the contrastive learning module has two data input sources, namely the complementary masked gene latent representation and the heavy masked gene latent representation. According to the masked sample rows in the heavy masked gene latent representation, the corresponding sample rows are selected from the complementary masked gene latent representation and the heavy masked gene latent representation to construct positive sample pairs. At the same time, sample rows that are inconsistent with the positive sample pairs are randomly extracted from the complementary masked gene latent representation and the heavy masked gene latent representation as negative sample pairs. As an example, if Figure 2 As shown, the masked sample behaviors in the re-masked gene latent representation are sample points 2 and 4, so the corresponding sample points 2 and 4 are selected from the complementary masked gene latent representation and the re-masked gene latent representation at the same time as the positive sample pair, and the sample rows that are inconsistent with the positive sample pair are randomly extracted from the complementary masked gene latent representation and the re-masked gene latent representation as the negative sample pair. For example, the sample point 1 in the complementary masked gene latent representation can be combined with any sample point in the re-masked gene latent representation as a negative sample pair. Sample point 2 in the latent representation can be combined with any sample point in the heavy-masked gene latent representation except sample point 2 as a negative sample pair, sample point 3 in the complementary masked gene latent representation can be combined with any sample point in the heavy-masked gene latent representation as a negative sample pair, sample point 4 in the complementary masked gene latent representation can be combined with any sample point in the heavy-masked gene latent representation except sample point 4 as a negative sample pair, and sample point 5 in the complementary masked gene latent representation can be combined with any sample point in the heavy-masked gene latent representation as a negative sample pair.

[0052] Furthermore, according to the formula Calculate the cosine similarity of the positive sample pair and the negative sample pair, where (A, B) Is the modulus symbol; by pulling the positive sample closer to the cosine similarity , push the negative sample to cosine similarity The contrast loss is constructed in a way.

[0053] The loss function plays a vital role in the training process of deep learning models. It is used to measure the gap between the model's prediction results and the true value and guide the optimization of model parameters. For the special design of this model, the total loss function consists of two parts: reconstruction loss and and Contrastive Loss , to ensure that the model can effectively interpolate missing spatial transcriptome data and learn reasonable spatial dependencies. In order to allow the model to learn spatial relationships, this embodiment introduces contrast loss, so that points with similar spatial positions or expression characteristics are close in the latent space, while points with different expression patterns are far away. The contrast loss is also divided into two parts, the first part is the cosine distance between positive sample pairs, and the second part is the cosine distance between negative sample pairs. By shortening the cosine distance between positive sample pairs , push away the cosine distance of negative samples This method ensures that the model can effectively distinguish different gene expression patterns while learning spatial structures; and reduces local noise interference by providing a complementary mask data source, thereby improving self-supervised learning capabilities and model generalization capabilities.

[0054] Since the core task of this model training is to interpolate missing gene expression values, a loss function that can measure the error between the predicted value and the true value is required. This embodiment uses scaled cosine error (SCE) to optimize the interpolation results. SCE loss measures directional consistency through cosine similarity, and dynamically adjusts sample weights in combination with an exponential weighting mechanism. It is robust to noise and amplitude changes, and focuses more on semantic alignment rather than precise numerical matching. It is very suitable for scenarios such as self-supervised learning. Therefore, this embodiment has a total loss function ,in, and It is a hyperparameter that controls the proportion of reconstruction loss and contrast loss in the total loss.

[0055] S80, pre-processing the histological image to be analyzed and the corresponding spatial transcriptome data to obtain histological image feature information , the spatial adjacency matrix A and the gene expression information X are input into the trained neural network model, and the fusion feature latent representation is output;

[0056] After the neural network model training is completed, negative samples will no longer be used, and there is no need to mask the gene expression information X. It is only necessary to preprocess the histological images to be analyzed and the corresponding spatial transcriptome data according to the same steps as the previous training steps to obtain the corresponding histological image feature information. , spatial adjacency matrix A and gene expression information X; then the histological image feature information , the spatial adjacency matrix A is input into the first encoder of the trained neural network model, the gene expression information X and the spatial adjacency matrix A are input into the second encoder of the trained neural network model, and are processed by each module in the trained neural network model in turn. Finally, after feature fusion by the second cross-attention fusion module, the fused feature latent representation Latent is output.

[0057] S90: Perform cluster analysis on the fusion feature latent representation and output a clustering result.

[0058] The present invention uses the medical image large model UNI, which has emerged in recent years, to extract image feature information and maximize the extraction of the features of histological image information; and uses multi-layer graph convolutional networks (GCN) to capture the high-order structural relationship between each sample point, and then removes various noises in the gene expression data itself to the greatest extent through contrast learning and complementary masking mechanism; at the same time, the histological image feature information and gene expression information are cleverly fused through two cross-attentions. The first cross-attention dynamically combines the adjacent gene latent representation of the sample point with the image feature latent representation as a condition, and the second cross-attention combines the first fusion result as a condition with the gene expression latent representation, thereby maximally integrating gene expression data, spatial location information and histological image information, that is, obtaining the fused feature latent representation Latent. In order to more accurately analyze the potential structure of spatial transcriptome data, this embodiment can use the mclust clustering method to cluster the fused feature latent representation Latent learned by the trained neural network model. mclust is a clustering method based on Gaussian Mixture Model (GMM), which can automatically select the optimal number of clusters in a high-dimensional latent space and provide the probability distribution structure of the data.

[0059] In a PyTorch-based deep learning environment, the following experiments compare the clustering performance of our proposed method (OURS) with other mainstream methods (CCST, Deep ST, DiffusionST, GraphST, SEDR, SpaGCN, and STAGATE). This example uses both multi-slice and single-slice datasets to evaluate the generalization and robustness of the algorithm. The experimental datasets are shown in Table 1. We provide a detailed evaluation of each data set using clustering performance metrics and graphical analysis.

[0060] Table 1 Statistics of spatial transcription data

[0061]

[0062] When evaluating unsupervised clustering algorithms, NMI (Normalized Mutual Information) and ARI (Adjusted Rand Index) are two commonly used metrics to measure the similarity between clustering results and true labels. The following details their definitions and calculation formulas:

[0063] 1. ARI (Adjusted Rand Index) is an indicator used to compare the similarity between two clustering results, taking into account the consistency between paired samples in the clustering results. Its calculation formula is as follows:

[0064] ,in, Indicates the The number of sample pairs that belong to the same cluster in the clustering results, Indicates the The number of sample pairs that belong to the same category in the true labels, Indicates that they are simultaneously assigned to The clustering results and The number of sample pairs with true labels, Indicates from Select from the elements The number of combinations of elements, is the total number of samples. The value of this indicator ranges between 0 and 1. The higher the value, the higher the similarity between the clustering result and the true label.

[0065] 2. NMI (Normalized Mutual Information) is an indicator used to evaluate clustering quality. It is based on the concept of information theory and measures the normalized value of the mutual information between the clustering results and the true labels. Its calculation formula is as follows:

[0066] ,in, represents the cluster set of clustering results, represents the number of clusters, Indicates the number of categories of the true label, Indicates that both belong to the cluster and the true label The number of samples, Represents clustering The number of samples in Represents the true label The number of samples in Indicates the total number of samples.

[0067] The above two indicators are values between 0 and 1. The higher the value, the higher the similarity between the clustering results and the true labels.

[0068] Figure 3 The chart in the figure is the ARI and NMI cluster evaluation results of DLPFC (human dorsolateral prefrontal cortex) multi-slice dataset under different methods, and Figure 3 The clustering results of its 151,675 slices are visualized in

[15] . Figure 3 Figure A shows the manually labeled classification of 151,675 slices of DLPFC; Figure B shows the box plot of the clustering ARI and NMI evaluation results of all slices of the DLPFC multi-slice dataset under different methods (the method of the present invention, CCST, Deep ST, DiffusionST, GraphST, SEDR, SpaGCN, and STAGATE); Figure C shows the visual display of the spatial domain clustering results of 151,675 slices of DLPFC for each method.

[0069] Figure 4 The ARI and NMI clustering evaluation results of the single slice dataset under different methods are shown in Figure 2. Figure 4 The left side is the ARI and NMI clustering evaluation results of the BRCA (human breast cancer) single slice dataset under different methods. Figure 4 The middle right side shows the ARI and NMI clustering evaluation results of the MBA (rat brain) single slice dataset under different methods.

[0070] from Figure 3 and Figure 4 The results show that compared with the mainstream spatial transcriptomics clustering analysis method, the proposed method shows stronger ability and excellent performance in cell clustering, and has significant robustness. Specifically, in order to quantify the overall performance of different methods in the spatial clustering task, we calculated the ARI and NMI of all 12 slices of the DLPFC dataset and visualized the results as follows: Figure 3Box plot of Figure B. Compared with seven state-of-the-art methods, our method achieves higher mean and median values in both metrics while exhibiting lower variance, indicating its superior consistency and hierarchical resolution across slices.

[0071] Figure 3 Figure C visually compares the spatial domain recognition results of different methods on 151,675 slices. The cluster boundaries obtained by our method closely align with the manually annotated layers, with clear transitions between layers and minimal noise interference. In contrast, some baseline methods (e.g., SEDR and SpaGCN) exhibit "mixed" or "cross-domain" clustering in boundary regions, resulting in significantly lower accuracy than our method.

[0072] In order to further demonstrate the robustness of the present invention, we also applied the present invention to more datasets BRCA ( Figure 4 Left), MBA ( Figure 4 Compared with the baseline method on the right side, the present invention still achieves the best performance on ARI and NMI, demonstrating significant robustness.

[0073] The above describes the spatial transcriptome data clustering method based on progressive learning and multimodal fusion in the embodiment of the present invention. The following describes the spatial transcriptome data clustering device based on progressive learning and multimodal fusion in the embodiment of the present invention. Figure 5 In one embodiment of the present invention, a spatial transcriptome data clustering device based on progressive learning and multimodal fusion includes:

[0074] A data acquisition module 10 is used to acquire histological images and their corresponding spatial transcriptome data;

[0075] Image processing module 20 is used to cut the histological image into blocks according to the size of the sample points, and then extract the histological image feature information from each block through the medical image large model UNI ;

[0076] An adjacency matrix construction module 30 is used to calculate the spatial Euclidean distance between each sample point based on the spatial position information of each sample point, select the k nearest sample points of each sample point as neighbors, and construct a spatial adjacency matrix A;

[0077] The gene expression data processing module 40 is used to screen and regularize the gene expression data of the sample points in the spatial transcriptome data, select the gene expression matrix of the top 3000 highly variable genes for principal component analysis, and extract the top 200 principal component gene expression matrices as gene expression information X;

[0078] The masking module 50 is used to randomly mask each row of the gene expression information X according to a predetermined ratio to obtain a masked gene expression matrix , and then perform complementary masking on the gene expression information X to obtain the complementary masked gene expression matrix ;

[0079] A model construction module 60 is used to construct a neural network model, wherein the neural network model includes a first encoder, a second encoder, a third encoder, a decoder, a first cross-attention fusion module, a second cross-attention fusion module, a progressive learning module, and a contrastive learning module;

[0080] Model training module 70, for using the spatial adjacency matrix A, histological image feature information , masked gene expression matrix and complementary masked gene expression matrices As input data, the neural network model is trained with reconstruction loss and contrast loss as the total loss function to obtain a trained neural network model;

[0081] Feature output module 80 is used to obtain the histological image feature information obtained by preprocessing the histological image to be analyzed and the corresponding spatial transcriptome data. , the spatial adjacency matrix A and the gene expression information X are input into the trained neural network model, and the fusion feature latent representation is output;

[0082] The aggregation module 90 is used to perform cluster analysis on the fusion feature latent representation and output a clustering result.

[0083] above Figure 5 From the perspective of modular functional entities, the spatial transcriptome data clustering device based on progressive learning and multimodal fusion in the embodiment of the present invention is described in detail. The spatial transcriptome data clustering device based on progressive learning and multimodal fusion in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0084] Figure 6This is a schematic diagram of the structure of a spatial transcriptome data clustering device based on progressive learning and multimodal fusion, provided by an embodiment of the present invention. This device 100 can vary significantly depending on configuration or performance. It may include one or more central processing units (CPUs) 11 (e.g., one or more processors), a memory 12, and one or more storage media 13 (e.g., one or more mass storage devices) storing application programs 133 or data 132. The memory 12 and storage medium 13 may be either transient or persistent storage. The program stored in the storage medium 13 may include one or more modules (not shown), each of which may include a series of instructions for the spatial transcriptome data clustering device 100 based on progressive learning and multimodal fusion. Furthermore, the processor 11 may be configured to communicate with the storage medium 13, executing the series of instructions stored in the storage medium 13 on the spatial transcriptome data clustering device 100 based on progressive learning and multimodal fusion.

[0085] The spatial transcriptome data clustering device 100 based on progressive learning and multimodal fusion may further include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input and output interfaces 16, and / or one or more operating systems 131, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 6 The device structure shown does not constitute a limitation on the spatial transcriptome data clustering device 100 based on progressive learning and multimodal fusion, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0086] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the steps of a spatial transcriptome data clustering method based on progressive learning and multimodal fusion.

[0087] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0089] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatial transcriptome data clustering method based on progressive learning and multimodal fusion, characterized in that: Including steps: Obtain histological images and their corresponding spatial transcriptome data; The histological image is cut into blocks according to the size of the sample points, and then the histological image feature information is extracted from each small block through the medical image large model UNI ; According to the spatial position information of each sample point, the spatial Euclidean distance between each two is calculated, the k nearest sample points of each sample point are selected as neighbors, and the spatial adjacency matrix A is constructed; The gene expression data of the sample points in the spatial transcriptome data were screened and regularized, the gene expression matrix of the top 3000 highly variable genes was selected for principal component analysis, and the top 200 principal component gene expression matrices were extracted as gene expression information X; The gene expression information X is randomly masked in each row according to a predetermined ratio to obtain a masked gene expression matrix , and then perform complementary masking on the gene expression information X to obtain the complementary masked gene expression matrix ; Constructing a neural network model, wherein the neural network model includes a first encoder, a second encoder, a third encoder, a decoder, a first cross-attention fusion module, a second cross-attention fusion module, a progressive learning module, and a contrastive learning module; The spatial adjacency matrix A and histological image feature information , masked gene expression matrix and complementary masked gene expression matrices As input data, the neural network model is trained with reconstruction loss and contrast loss as the total loss function to obtain a trained neural network model; The histological image feature information obtained by preprocessing the histological image to be analyzed and the corresponding spatial transcriptome data , the spatial adjacency matrix A and the gene expression information X are input into the trained neural network model, and the fusion feature latent representation is output; Perform cluster analysis on the fusion feature latent representation and output the clustering results.

2. The spatial transcriptome data clustering method based on progressive learning and multimodal fusion according to claim 1, characterized in that: The spatial adjacency matrix, histological image feature information, mask gene expression matrix, and complementary mask gene expression matrix are used as input data, and reconstruction loss and contrast loss are used as the total loss function to train the neural network model to obtain a trained neural network model, comprising the steps of: Inputting the histological image feature information and the spatial adjacency matrix into the first encoder, and outputting a latent representation of the image feature information; Inputting the masked gene expression matrix and the spatial adjacency matrix into the second encoder, and inputting the output masked gene latent representation into the third encoder after re-masking to obtain the re-masked gene latent representation; Inputting the re-masked gene latent representation into a progressive learning module, and outputting an average re-masked gene latent representation; Inputting the average re-masked gene latent representation and the image feature information latent representation into the first cross-attention fusion module, and outputting a fused latent representation; Inputting the once fused latent representation and the re-masked gene latent representation into the second cross-attention fusion module, and outputting a fused feature latent representation; Inputting the fused feature latent representation into the decoder and outputting the reconstructed feature representation; Inputting the complementary mask gene expression matrix and the spatial adjacency matrix into the second encoder, and outputting the complementary mask gene latent representation; The complementary masked gene latent representation and the re-masked gene latent representation are input into the contrastive learning module, the cosine similarity of the positive sample pair and the negative sample pair is output, and the contrastive loss is constructed based on the cosine similarity of the positive sample pair and the negative sample pair. ; Constructing a reconstruction loss based on the reconstructed feature representation ; Based on the reconstruction loss function And contrast loss function Constructing the total loss function The neural network model is trained to obtain a trained neural network model, wherein: and It is a hyperparameter that controls the proportion of reconstruction loss and contrast loss in the total loss.

3. The spatial transcriptome data clustering method based on progressive learning and multimodal fusion according to claim 2, characterized in that: Inputting the histological image feature information and the spatial adjacency matrix into the first encoder and outputting the latent representation of the image feature information comprises the steps of: The first encoder includes several graph convolution layers, which perform convolution operations on the input histological image feature information and the spatial adjacency matrix to obtain a latent representation of the image feature information, wherein the convolution operation expression is: ,in, is the feature representation of all sample points in the current layer, It represents the feature representation of all sample points after a convolution operation. , , I is the identity matrix, yes The degree matrix of is the activation function, is the trainable parameter matrix of the current layer.

4. The spatial transcriptome data clustering method based on progressive learning and multimodal fusion according to claim 3, characterized in that: The masked gene expression matrix and the spatial adjacency matrix are input into the second encoder, and the output masked gene latent representation is input into the third encoder after re-masking to obtain the re-masked gene latent representation, including the steps of: The second encoder includes a plurality of graph convolution layers, which perform convolution operations on the input mask gene expression matrix and the spatial adjacency matrix to obtain a mask gene latent representation; re-masking the masked gene latent representation, wherein the masked rows of the masked gene latent representation are the same as the masked rows in the masked gene table matrix, to obtain a re-masked gene latent representation; The third encoder includes several graph convolution layers, which perform convolution operations on the input re-masked gene latent representation to obtain the re-masked gene latent representation.

5. The spatial transcriptome data clustering method based on progressive learning and multimodal fusion according to claim 4, characterized in that: Inputting the re-masked gene latent representation into a progressive learning module and outputting the average re-masked gene latent representation comprises the following steps: The progressive learning module increases according to the number of training rounds and the spatial adjacency matrix A certain percentage of neighboring points are randomly selected for each sample point, and the re-masked gene latent representations of the neighboring points are averaged to obtain the average re-masked gene latent representation, where the percentage is set according to the following formula: ,in, The number of epochs required for training the neural network model, The current training round of the neural network model.

6. The spatial transcriptome data clustering method based on progressive learning and multimodal fusion according to claim 5, characterized in that: Inputting the average re-masked gene latent representation and the image feature information latent representation into the first cross-attention fusion module, and outputting a fused latent representation, comprising the steps of: The first cross-attention fusion module is used to capture the correlation between different feature spaces, and its calculation formula is as follows: , where Q, K, and V are query matrix, key matrix, and value matrix respectively. is a weight matrix, Represents a matrix concatenation operation; The average re-masked gene latent representation and the image feature information latent representation are input into the first cross-attention fusion module, and a fused latent representation is output, wherein the image feature information latent representation serves as the data source of the query matrix Q, and the average re-masked gene latent representation serves as the data source of the key matrix K and the value matrix V.

7. The spatial transcriptome data clustering method based on progressive learning and multimodal fusion according to claim 6, characterized in that: The complementary masked gene latent representation and the re-masked gene latent representation are input into the contrastive learning module, the cosine similarity of the positive sample pair and the negative sample pair is output, and the contrastive loss is constructed based on the cosine similarity of the positive sample pair and the negative sample pair. , including the steps of: According to the masked sample rows in the re-masked gene latent representation, corresponding sample rows are selected from the complementary masked gene latent representation and the re-masked gene latent representation to construct positive sample pairs, and sample rows that are inconsistent with the positive sample pairs are randomly extracted from the complementary masked gene latent representation and the re-masked gene latent representation as negative sample pairs; According to the formula Calculate the cosine similarity of positive sample pairs and negative sample pairs, where (A, B) is a sample pair, is the modulo symbol; By pulling the cosine similarity of the positive samples closer , push the negative sample to cosine similarity The contrast loss is constructed in a way.

8. A spatial transcriptome data clustering device based on progressive learning and multimodal fusion, characterized in that: include: Data acquisition module, used to obtain histological images and their corresponding spatial transcriptome data; Image processing module, used to cut the histological image into blocks according to the size of the sample points, and then extract the histological image feature information from each block through the medical image large model UNI ; The adjacency matrix construction module is used to calculate the spatial Euclidean distance between each sample point based on the spatial position information of each sample point, select the k nearest sample points of each sample point as neighbors, and construct the spatial adjacency matrix A; The gene expression data processing module is used to screen and regularize the gene expression data of sample points in the spatial transcriptome data, select the gene expression matrix of the top 3000 highly variable genes for principal component analysis, and extract the top 200 principal component gene expression matrices as gene expression information X; The mask module is used to randomly mask each row of the gene expression information X according to a predetermined ratio to obtain a masked gene expression matrix , and then perform complementary masking on the gene expression information X to obtain the complementary masked gene expression matrix ; A model construction module is used to construct a neural network model, wherein the neural network model includes a first encoder, a second encoder, a third encoder, a decoder, a first cross-attention fusion module, a second cross-attention fusion module, a progressive learning module, and a contrastive learning module; Model training module, used to use the spatial adjacency matrix A, histological image feature information , masked gene expression matrix and complementary masked gene expression matrices As input data, the neural network model is trained with reconstruction loss and contrast loss as the total loss function to obtain a trained neural network model; Feature output module, used to obtain the histological image feature information obtained after preprocessing the histological image to be analyzed and the corresponding spatial transcriptome data , the spatial adjacency matrix A and the gene expression information X are input into the trained neural network model, and the fusion feature latent representation is output; The aggregation module is used to perform cluster analysis on the fusion feature latent representation and output the clustering results.

9. A spatial transcriptome data clustering device based on progressive learning and multimodal fusion, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute each step of the spatial transcriptome data clustering method based on progressive learning and multimodal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the spatial transcriptome data clustering method based on progressive learning and multimodal fusion as described in any one of claims 1 to 7 are implemented.

Images