A histopathology spatial transcriptome prediction method based on cross-modal mask reconstruction and multi-task alignment modeling

Abstract

This invention relates to technologies such as deep learning, modality fusion, and spatial transcriptome prediction, and discloses a method for predicting histopathological spatial transcriptomes based on cross-modal mask reconstruction and multi-task alignment modeling. This method introduces a cross-modal mask reconstruction mechanism between image features and gene expression features, enabling the model to learn higher-order relationships between two modalities even with incomplete information. Simultaneously, by combining a multi-task joint optimization strategy involving contrastive learning and reconstruction tasks, the model not only obtains highly discriminative cross-modal representations but also improves its robustness to noise interference and data gaps through the reconstruction task. This addresses the technical problems of insufficient accuracy and modal interaction in existing methods for spatial transcriptome prediction.

Description

Technical Field

[0001] This invention belongs to the field of spatial transcriptome prediction, specifically involving deep learning and modality fusion, and particularly to a method for predicting histopathological spatial transcriptomes based on cross-modal mask reconstruction and multi-task alignment modeling. Background Technology

[0002] With the continuous development of spatial transcriptomics (ST) technology, we can comprehensively understand gene expression patterns within complex biological systems at the in situ level. This not only reveals the spatial relationships and interactions between cells but also helps us understand the functional structure and dynamic changes of tissues. Using ST technology, researchers can explore the molecular mechanisms of disease development and progression at a holistic level, providing a novel tool for disease mechanism research, drug development, and clinical diagnosis. However, despite the significant value of ST technology, its application faces certain challenges: firstly, the experimental costs are high, requiring expensive sequencing platforms and supporting imaging equipment for data acquisition; secondly, it demands a high level of expertise from researchers, often requiring interdisciplinary knowledge in biology and computer science. Therefore, the generation of large-scale spatial transcriptomics data using ST technology remains a constraint.

[0003] With the development of artificial intelligence technologies, especially machine learning and deep learning, it has become possible to predict gene expression from histopathological images. This not only reduces the high cost of spatial transcriptomics experiments but also makes full use of the large amount of pathological slide data routinely collected in clinical practice, enabling high-throughput inference of gene expression levels, thus providing new avenues for disease mechanism research and precision medicine. Existing studies typically employ multimodal information fusion strategies to combine spatial transcriptomics data with pathological images to model the complex relationship between the two. However, the mapping relationship between pathological images and gene expression is complex and highly nonlinear, influenced by factors such as tissue heterogeneity, sample source, and sequencing noise. Effectively modeling this cross-modal association remains a significant challenge.

[0004] Current research mainly focuses on two technical approaches. One approach directly predicts spatial transcriptome data from histopathological images, using convolutional neural networks (CNNs) or Vision Transformers to model histological image features and combining them with regression models to directly predict gene expression. This method can capture some expression-related structural information in the image, but it is insufficient in modeling global spatial dependencies. The other approach improves prediction performance by establishing a shared latent space between histopathological images and transcriptome features. This method emphasizes the consistency of cross-modal representations, but it still has shortcomings in the depth of multimodal information interaction and its ability to resist noise. Although these methods have improved prediction accuracy to some extent, some problems still exist: 1. Some methods rely too much on local image texture features, making it difficult to capture global spatial structural information; 2. The models are easily affected by data distribution shifts and sequencing noise, resulting in insufficient generalization performance; 3. The multimodal information of image-gene expression data is not effectively integrated, resulting in insufficient intermodal interaction. Therefore, how to achieve stronger generalization ability, more sufficient cross-modal interaction, and robustness to noise while ensuring prediction accuracy remains a core problem that urgently needs to be solved. Summary of the Invention

[0005] Current research methods still have shortcomings in modeling the relationship between histopathological images and spatial transcriptome features. This invention aims to address these problems by effectively integrating multimodal information from image-gene expression data while improving the accuracy of spatial transcriptome prediction. This invention proposes a histopathological spatial transcriptome prediction method based on cross-modal mask reconstruction and multi-task alignment modeling, with the goal of solving the technical problems of insufficient prediction accuracy and modal interaction in existing methods.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] 1) Obtain a histopathological dataset containing real data, including histological image data, spatial gene expression data, and spatial location coordinates. Then, divide the dataset accordingly, using leave-one-out cross-validation to divide the dataset into training and test sets to ensure the accuracy of gene expression data prediction. At the same time, preprocess the histological images and spatial gene expression data.

[0008] 2) To ensure consistency between the architecture of the image encoder in the contrastive learning task and the architecture of the cross-attention module in cross-modal mask reconstruction, a basic version of the Vision Transformer (ViT) is used as the visual encoder to obtain visual features; a Transformer encoder is used as the gene encoder to obtain gene expression features. For the contrastive learning task, image and gene expression data are fed into the visual encoder and gene encoder, respectively, to obtain visual and gene expression features.

[0009] 3) Project the extracted features into the same feature space. The optimization goal of this space is to maximize the cosine similarity of positive sample pairs and minimize the cosine similarity of negative sample pairs. The CLIP loss function is used and fine-tuned to optimize this task.

[0010] 4) For cross-modal mask reconstruction tasks, the process is performed at the image feature level, and the image features are masked according to the mask ratio.

[0011] 5) Perform cross-modal fusion of masked image features and gene expression features.

[0012] 6) The fused features are fed into the Transformer-based feature reconstruction module for image feature reconstruction.

[0013] 7) Use the mean squared error (MSE) loss function to compare the reconstructed image features with the real image features.

[0014] 8) Adaptively weightedly fuse the losses from 3) and 6) to obtain the parameters of the final model.

[0015] 9) In the inference stage, similarity-based retrieval is used to input test set images into the visual encoder to extract visual features. In the established shared feature space, spatial gene expression is inferred from the images.

[0016] Specifically, step 1) includes:

[0017] For histological images, corresponding image blocks are drawn based on the spatial coordinates of the sequencing sites (spots). Each image block is 224×224 in size to ensure a one-to-one correspondence between the image and the expression profile. Ultimately, each sample contains three types of information: image block, expression profile vector, and two-dimensional coordinate position.

[0018] For spatial gene expression data, a set of genes common to all tissue sections was extracted as candidate genes. Each sample was then standardized and logarithmically transformed, and the top 1000 highly variable genes (HVGs) were selected from each tissue section. Subsequently, officially provided marker genes were introduced and fused with the selected HVGs to construct a final gene subset. Based on this, the expression matrices of all samples were filtered, retaining only the expression profiles corresponding to the aforementioned gene subset, resulting in a uniform-dimensional highly variable gene expression matrix.

[0019] Specifically, step 2) includes:

[0020] Visual features are extracted from the extracted image patches using a pre-trained Vision Transformer encoder. Compared to traditional convolutional neural networks, the Vision Transformer can directly model global dependencies through its self-attention mechanism, making it more suitable for capturing long-range interactions between different spatial regions. It combines gene expression data with spatial location information and inputs this data into the Transformer encoder to extract gene expression features. It seamlessly integrates the gene expression features and spatial location features of spot through a self-attention mechanism.

[0021] Specifically, step 3) includes:

[0022] To map features from different modalities to the same feature space, a nonlinear projection module is added to each modal encoder to map high-dimensional representations to a unified low-dimensional embedding space, resulting in visual feature embeddings for contrastive learning. and gene expression feature embedding Meanwhile, the InfoNCE bidirectional contrast loss function is introduced to align the representations of image modalities and gene expression modalities.

[0023] To integrate positive sample pairs, a target matrix is ​​used, where diagonal elements represent positive sample pairs (labeled 1) and off-diagonal elements represent negative sample pairs (labeled 0). A cross-entropy loss function is introduced to calculate image loss and gene expression loss, ultimately yielding the contrastive learning loss.

[0024] Specifically, step 4) includes:

[0025] For a given input image feature, a feature-level masking strategy is adopted to randomly mask some feature dimensions according to a set masking ratio, and a learnable mask vector is introduced at these positions to replace the original feature values, thereby generating an image representation with missing information.

[0026] Specifically, step 5) includes:

[0027] masked image features Corresponding gene expression characteristics The inputs are shared into the cross-modal fusion module, guiding the model to perform semantic compensation using spatial transcriptome signals. To guide image reconstruction using gene expression information, a memory-enhanced cross-modal fusion module was designed. Unlike the traditional simple concatenation of visual features and gene expression features, this module introduces multi-head cross attention (MCA) to model the interaction between the two modalities at the shallow stage, enabling the model to more fully capture the cross-modal correspondences, thereby improving the ability to complete missing features.

[0028] Specifically, step 6) includes:

[0029] The fused features are fed into a Transformer-based feature reconstruction module for image feature reconstruction. Unmasked features serve as context input, while mask locations are replaced by learnable mask tokens. The model predicts missing parts by capturing correlations between features. This approach fully utilizes the global semantic information provided by cross-modal fusion and leverages the Transformer's self-attention mechanism to model dependencies between feature dimensions, thereby improving the reconstruction accuracy of masked features.

[0030] Specifically, step 7) includes:

[0031] The reconstructed image features from step 6) are compared with the real image features at the masked locations using mean squared error (MSE) loss. MSE is sensitive to continuous value bias and can effectively constrain the model's prediction error in missing regions, thereby improving the accuracy of the reconstruction results.

[0032] Specifically, step 8) includes:

[0033] Weighted joint optimization is performed on the losses obtained from steps 3) and 7), by setting... and Two parameters are used to dynamically balance the importance of the two tasks.

[0034] Specifically, step 9) includes:

[0035] The test set images are input into the visual encoder to extract their visual features. In the established unified feature space, the cosine similarity between the test image and all gene expression features is calculated. The gene expression value of the current spot is inferred by using the weighted distance of gene expression of the top k sequencing sites. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the method.

[0038] Figure 2 This is a schematic diagram of the model framework for this method.

[0039] Figure 3 This is a schematic diagram of the MeCoF module architecture in this method. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention proposes a spatial transcriptome prediction method for histopathology based on cross-modal mask reconstruction and multi-task alignment modeling. The main process of this method is as follows: Figure 1 As shown:

[0042] 1) Obtain a histopathological dataset containing real data, including histological image data, spatial gene expression data, and spatial location coordinates. Then, divide the dataset accordingly, using leave-one-out cross-validation to divide the dataset into training and test sets to ensure the accuracy of gene expression data prediction. At the same time, preprocess the histological images and spatial gene expression data.

[0043] 2) To ensure consistency between the architecture of the image encoder in the contrastive learning task and the architecture of the cross-attention module in cross-modal mask reconstruction, a basic version of the Vision Transformer (ViT) is used as the visual encoder to obtain visual features; a Transformer encoder is used as the gene encoder to obtain gene expression features. For the contrastive learning task, image and gene expression data are fed into the visual encoder and gene encoder, respectively, to obtain visual and gene expression features.

[0044] 3) Project the extracted features into the same feature space. The optimization goal of this space is to maximize the cosine similarity of positive sample pairs and minimize the cosine similarity of negative sample pairs. The CLIP loss function is used and fine-tuned to optimize this task.

[0045] 4) For cross-modal mask reconstruction tasks, the process is performed at the image feature level, and the image features are masked according to the mask ratio.

[0046] 5) Perform cross-modal fusion of masked image features and gene expression features.

[0047] 6) The fused features are fed into the transformer-based feature reconstruction module for image feature reconstruction.

[0048] 7) Use the mean squared error (MSE) loss function to compare the reconstructed image features with the real image features.

[0049] 8) Adaptively weightedly fuse the losses from 3) and 6) to obtain the parameters of the final model.

[0050] 9) In the inference stage, similarity-based retrieval is used to input test set images into the visual encoder to extract visual features. In the established shared feature space, spatial gene expression is inferred from the images.

[0051] Specifically, step 1) includes:

[0052] A histopathological dataset containing real-world data was obtained. This dataset includes four types of information: gene count matrix, histopathological slide images, spatial site files, and histological annotations for some samples. For histopathological images, image patches of 224×224 pixels were drawn based on the spatial coordinates of sequencing sites (spots) to ensure a one-to-one correspondence between images and expression profiles. For spatial gene expression data, a set of genes common to all tissue sections was extracted as candidate genes. Each sample was then standardized and logarithmically transformed, and the top 1000 highly variable genes (HVGs) were selected from each tissue section. Officially provided marker genes were then introduced and fused with the selected HVGs to construct a final gene subset. Based on this, the expression matrices of all samples were filtered, retaining only the expression profiles corresponding to the aforementioned gene subset, resulting in a uniform-dimensional highly variable gene expression matrix.

[0053] To improve the model's generalization ability, data augmentation operations are performed on image patches during the training phase, including color jittering, random horizontal flipping, and random rotation; while during the testing phase, the image patches are simply converted into tensors and input into the model. Ultimately, each sample contains three types of information: image patch, representation spectral vector, and two-dimensional coordinate position.

[0054] Further, step 2) includes:

[0055] Visual features are extracted from the extracted image patches using a pre-trained Vision Transformer encoder. Compared to traditional convolutional neural networks, the Vision Transformer can directly model global dependencies through its self-attention mechanism, making it more suitable for capturing long-range interactions between different spatial regions. It combines gene expression data with spatial location information and inputs this data into the Transformer encoder to extract gene expression features. It seamlessly integrates the gene expression features and spatial location features of spot through a self-attention mechanism.

[0056] Multi-head attention is an extension of the attention mechanism that enhances the model's ability to capture complex patterns and global information in the input sequence by simultaneously learning multiple sets of independent attention weights.

[0057]

[0058] This represents the weight matrix used to aggregate attention heads, where n represents the number of heads, and Q, K, and V represent the query, key, and value, respectively. The attention mechanism is defined as follows:

[0059]

[0060]

[0061] , and Represents the weight matrix. It is an attention map, and its shape is... , It is the value of the attention mechanism, where .

[0062] Further, step 3) includes:

[0063] To map features from different modalities to the same feature space, a nonlinear projection module is added to each modality encoder. This projection module consists of two fully connected layers, a GELU activation function, dropout, residual connections, and layer normalization. Through this module, high-dimensional representations are mapped to a unified low-dimensional embedding space, resulting in visual feature embeddings for contrastive learning. and gene expression feature embedding Meanwhile, the InfoNCE bidirectional contrast loss function is introduced to align the representations of image modalities and gene expression modalities.

[0064] First, the cosine similarity function between visual feature embeddings and gene expression feature embeddings is defined as follows:

[0065]

[0066]

[0067] To integrate positive sample pairs, a target matrix is ​​used, where the diagonal elements represent positive sample pairs (labeled as 1) and the off-diagonal elements represent negative sample pairs (labeled as 0).

[0068] The cross-entropy loss function (CE) is introduced to calculate image loss and gene expression loss, and the final contrastive learning loss is:

[0069]

[0070] Further, step 4) includes:

[0071] In the mask reconstruction task, a random mask is introduced into the feature vector output by the visual encoder to simulate feature loss and force the model to learn feature recovery capabilities. The specific process is as follows:

[0072] For a given input visual features Based on a set masking rate, image features are randomly masked. Inspired by MAE, a masking rate of 0.75 is used, and learnable mask vectors are introduced at the masking locations to replace them, thereby obtaining masked image features. .

[0073] Further, step 5) includes:

[0074] masked image features The corresponding gene expression features, along with the inputs, are fed into the cross-modal fusion module to guide the model in using spatial transcriptome signals for semantic compensation. To guide image reconstruction using gene expression information, a memory-enhanced cross-modal fusion module (MeCoF) was designed, with the following structure: Figure 2 As shown, unlike the traditional simple concatenation of visual features and gene expression features, this module introduces Multi-head Cross Attention (MCA) to model the interaction between the two modalities at the shallow level. This allows the model to more fully capture cross-modal correspondences, thereby improving the ability to complete missing features. MCA is defined as follows:

[0075]

[0076] , and This represents the weight matrix, where n represents the number of heads. A linear matrix representing multi-head attention.

[0077] Gene expression characteristics Image features after masking The input is fed into the multimodal cross-attention module (MCA) to obtain cross-modal intermediate representations. and Furthermore, a learnable memory matrix is ​​added to expand the keys and values. The fusion formula for the two modalities is as follows:

[0078]

[0079]

[0080] Obtain cross-modal fusion features .

[0081] Further, step 6) includes:

[0082] The fused features The images are fed into a transformer-based feature reconstruction module for feature reconstruction. Unmasked features serve as context input, while mask locations are replaced by learnable mask tokens. The model predicts missing parts by capturing correlations between features. This approach leverages the global semantic information provided by cross-modal fusion and utilizes the Transformer's self-attention mechanism to model dependencies between feature dimensions, thereby improving the reconstruction accuracy of masked features.

[0083] Further, step 7) includes:

[0084] Based on the reconstructed image features from step 6) Features of real images The mean squared error (MSE) loss is compared at the masked locations. MSE is sensitive to continuous value bias and can effectively constrain the model's prediction error in missing regions, thereby improving the accuracy of the reconstruction results.

[0085]

[0086] Specifically, step 8) includes:

[0087] Weighted joint optimization is performed on the losses obtained from steps 3) and 7), by setting... and Two parameters are used to dynamically balance the importance of the two tasks:

[0088]

[0089] Specifically, step 9) includes:

[0090] The test set images are input into a visual encoder to extract their visual features. Within an established unified feature space, the cosine similarity between the test image and all gene expression features is calculated. The weighted distance of gene expression at the top k sequencing sites (spots) is used to infer the gene expression value of the current spot. The formula for calculating the Euclidean distance between image visual features and gene expression features is as follows:

[0091]

[0092] Where n represents the dimension of the feature space.

[0093] The gene expression value of spot v is inferred using the weighted distance of the expression of the top k spot genes. The weights are defined as follows:

[0094]

[0095]

[0096] represent The observed gene expression values, where k is set to 200.

[0097] After steps 1) to 9) above, the model can predict the corresponding gene expression based on the input image. For performance evaluation, the Pearson correlation coefficient (PCC), mean squared error (MSE), and mean absolute error (MAE) are used to evaluate the prediction results.

[0098] The beneficial effects of this invention are as follows:

[0099] 1) Introducing a cross-modal mask reconstruction mechanism enables the model to learn higher-order relationships between image features and gene expression features even when some information is missing, thereby enhancing the effectiveness of modal interaction.

[0100] 2) The reconstruction task enables the model to have self-recovery capabilities under noisy and missing data conditions, significantly improving its adaptability to common technical noise and missing phenomena in real histopathological data.

[0101] 3) By adopting a joint optimization strategy of contrastive learning and reconstruction tasks, the interpretability and recovery ability of features are improved while maintaining the discriminativeness of the representation, and the predictive performance and robustness of the model are taken into account.

[0102] 4) This method can adapt to different types of spatial transcriptome data and is suitable for extension to spatial gene expression analysis of large-scale tissue sections, and has strong application value and scalability.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for predicting the spatial transcriptome in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling, characterized in that, Includes the following steps: 1) Obtain a histopathological dataset containing real data, including histological image data, spatial gene expression data, and spatial location coordinates. Then, divide the dataset accordingly, using leave-one-out cross-validation to divide the dataset into training and test sets to ensure the accuracy of gene expression data prediction. At the same time, preprocess the histological images and spatial gene expression data. 2) To ensure the consistency between the architecture of the image encoder in the contrastive learning task and the architecture of the cross-attention module in cross-modal mask reconstruction, the basic version of Vision Transformer (ViT) is used as the visual encoder to obtain visual features; The Transformer encoder is used as the gene encoder to obtain gene expression features. For the contrastive learning task, the image and gene expression data are fed into the visual encoder and gene encoder respectively to obtain visual features and gene expression features. 3) Project the extracted features into the same feature space. The optimization goal of this space is to maximize the cosine similarity of positive sample pairs and minimize the cosine similarity of negative sample pairs. The CLIP loss function is used and fine-tuned to optimize this task. 4) For cross-modal mask reconstruction tasks, the process is performed at the image feature level, and the image features are masked according to the mask ratio. 5) Perform cross-modal fusion of masked image features and gene expression features. 6) The fused features are fed into the Transformer-based feature reconstruction module for image feature reconstruction. 7) Use the mean squared error (MSE) loss function to compare the reconstructed image features with the real image features. 8) Adaptively weightedly fuse the losses from 3) and 7) to obtain the parameters of the final model. 9) In the inference stage, similarity-based retrieval is used to input the test set images into the visual encoder to extract visual features. In the established shared feature space, the task of inferring spatial gene expression from the images is completed.

2. The method for predicting spatial transcriptomes in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 1) includes: For histological images, drawing corresponding image patches based on the spatial coordinates of the sequencing spots, with each image patch being [size missing]. To ensure a one-to-one correspondence between images and expression profiles, each sample ultimately contains three types of information: image patch, expression profile vector, and two-dimensional coordinate position. For spatial gene expression data, a set of genes common to all tissue slices is extracted as candidate genes. Each sample is then standardized and logarithmically transformed, and the top 1000 highly variable genes (HVGs) are selected from each tissue slice. Officially provided marker genes are then introduced and fused with the selected HVGs to construct the final gene subset. Based on this, the expression matrices of all samples are filtered, retaining only the expression profiles corresponding to the aforementioned gene subset, resulting in a uniform-dimensional highly variable gene expression matrix.

3. The histopathological spatial transcriptome prediction method based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 2) includes: extracting visual features from the extracted image patches using a pre-trained VisionTransformer encoder. Compared to traditional convolutional neural networks, VisionTransformer can directly model global dependencies through a self-attention mechanism, making it more suitable for capturing long-range interactions between different spatial regions; and inputting gene expression data combined with spatial location information into the Transformer encoder to extract gene expression features, seamlessly integrating the gene expression features and spatial location features of spot through a self-attention mechanism.

4. The histopathological spatial transcriptome prediction method based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 3) includes: In order to map the features of different modalities to the same feature space, a nonlinear projection module is added to each modality encoder to map the high-dimensional representation to a unified low-dimensional embedding space. At the same time, the InfoNCE bidirectional contrast loss function is introduced to align the representations of image modality and gene expression modality.

5. The histopathological spatial transcriptome prediction method based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 4) includes: for the given input image features, a feature-level masking strategy is adopted to randomly mask some feature dimensions according to the set masking ratio, and learnable mask vectors are introduced at these positions to replace the original feature values, thereby generating an image representation with missing information.

6. The method for predicting spatial transcriptomes in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 5) includes: inputting the masked image features and gene expression features into the cross-modal fusion module to guide the model to use spatial transcriptome signals for semantic compensation.

7. The method for predicting spatial transcriptomes in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 6) includes: feeding the fused features into a Transformer-based feature reconstruction module for image feature reconstruction. The unmasked features serve as context input, while the mask positions are replaced by learnable mask tokens. The model predicts missing parts by capturing the correlations between features.

8. The method for predicting spatial transcriptomes in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 7) includes: comparing the image features reconstructed from step 6) with the real image features at the masked locations using mean squared error (MSE) loss. MSE is sensitive to continuous value bias and can effectively constrain the model's prediction error in missing regions, thereby improving the accuracy of the reconstruction results.

9. The method for predicting spatial transcriptomes in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 8) includes: performing weighted joint optimization on the losses obtained in steps 3) and 7), by setting... and Two weighting parameters are used to dynamically balance the importance of the two tasks.

10. The method for predicting spatial transcriptomes in histopathology based on cross-modal mask reconstruction and multi-task alignment modeling according to claim 1, characterized in that, Step 9) includes: inputting the test set images into the visual encoder to extract their visual features; calculating the cosine similarity between the test image and all gene expression features in the established unified feature space; and using the weighted distance of gene expression at the top k sequencing sites to infer the gene expression value of the current spot.

Images