Single-cell transcriptome-based model construction method based on mamba-transformer hybrid architecture

Abstract

The application provides a single-cell transcriptome basic model construction method based on a Mamba-Transformer hybrid architecture, and belongs to the technical field of artificial intelligence and bioinformatics. The technical scheme is as follows: quality control and standardization are performed on single-cell transcriptome data to obtain a standardized gene expression matrix; an input embedding containing gene identification, expression value marking and state label is constructed to form a cell-level continuous vector representation; a hybrid network architecture in which Mamba, Transformer and sparse hybrid experts are alternately stacked is constructed, combined with residual connection and layer normalization; joint pre-training is performed in combination with gene expression value prediction and cell representation learning tasks to obtain a basic model. The beneficial effects are as follows: the application overcomes zero inflation and technical noise by learning global context, improves signal-to-noise ratio and feature robustness, constructs a unified representation space to effectively strip batch effects, realizes precise alignment across platforms and modalities, and significantly reduces the dependence on artificial annotation for downstream tasks and saves computing resources by virtue of strong knowledge transfer capability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and bioinformatics technology, specifically relating to a method for constructing a basic model of single-cell transcriptome based on the Mamba-Transformer hybrid architecture. Background Technology

[0002] Single-cell transcriptome sequencing (scRNA-seq) technology digitally records the complex molecular regulatory networks within cells through gene-cell mapping, forming a high-dimensional feature space describing the microscopic state of life. With the continuous development of modern single-cell sequencing platforms, the generation of massive amounts of data means that single-cell data not only contains key information for cell type identification but also implicitly reveals intricate co-expression and regulatory relationships between genes. Therefore, deep learning models for single-cell data have become ideal tools for feature mining and pattern recognition. However, scRNA-seq data faces significant challenges, mainly in two aspects: First, due to the limitations of sequencing depth, the data often exhibits extreme sparsity and a "zero inflation" phenomenon, meaning that a large number of gene expression values ​​are incorrectly recorded as zero. This non-biological technical noise reduces the signal-to-noise ratio of the data, affecting the accuracy of subsequent analysis. Second, due to differences in different sequencing platforms, batches, and biological samples, batch effects inevitably exist in the data, which undermines the independent and identically distributed assumption of the data, increasing the difficulty of cross-platform and cross-tissue data integration and knowledge transfer.

[0003] Currently, the main technical challenge facing single-cell data analysis is that most existing deep learning methods employ a "train from scratch" strategy for single datasets, failing to effectively mine and reuse globally universal biological patterns from massive public transcriptome data. This isolated training strategy has drawbacks: firstly, it requires model re-initialization for each training task, leading to a huge waste of computational resources; secondly, the lack of global constraints from large-scale data makes it prone to overfitting when dealing with high-dimensional, sparse, and technically noisy single-cell data, lacking good cross-platform and cross-modal generalization capabilities. This means that traditional models still heavily rely on large amounts of expensive, high-quality manually labeled data when performing complex downstream tasks such as multi-omics integration, spatial topology analysis, and gene regulation modeling, making it difficult to guarantee accuracy in scenarios with scarce annotations, cold starts, or complex batch interference. Therefore, there is an urgent need for a single-cell data modeling solution that can overcome data silos, efficiently learn cross-platform and cross-modal universal representations, and possess stable transfer capabilities across diverse downstream tasks. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture. This method solves the problems of poor generalization and weak noise resistance caused by isolated training strategies in the prior art. It also solves the problem that traditional methods are unable to effectively extract global biological laws from massive single-cell data, resulting in insufficient robustness of the model when facing sparse and noisy data, and a lack of cross-platform and cross-modal knowledge transfer capabilities.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture, comprising the following steps: S1: Acquire and preprocess single-cell transcriptome datasets, the preprocessing including quality control and standardization of raw gene expression data to obtain a normalized gene expression matrix; S2: Construct an input embedding that includes gene identifiers, gene expression value markers, and status labels, and convert the discrete gene information of each cell into a continuous vector representation to form a cell-level input embedding; S3: Construct a hybrid network architecture, which includes a Mamba network module, a Transformer network module, and a sparse hybrid expert network module. The Mamba network module and the Transformer network module are stacked alternately, and the sparse hybrid expert network module is connected after every two network modules, and combined with residual connections and layer normalization. S4: The hybrid network architecture is trained using a pre-training strategy, which combines gene expression value prediction task and cell representation learning task to jointly optimize model parameters at the gene level and cell level, thereby obtaining a pre-trained basic model.

[0006] Furthermore, the quality control in step S1 includes: deleting cells with fewer than a preset threshold of genes, deleting cells with a mitochondrial gene ratio exceeding a preset percentage, and deleting genes whose expression levels in cells do not meet preset conditions; the standardization process includes at least one of normalization based on the total number, logarithmic transformation, and variance stabilization.

[0007] Furthermore, the steps for constructing the input embedding in step S2 are as follows: S21: Assign a unique gene identifier to each gene, construct a gene identifier sequence based on the list of non-zero expressed genes in the cell, and add special markers for aggregating cell characterization and filling sequence length; S22: Binning is performed on the non-zero gene expression values ​​of each cell, and the continuous expression values ​​are discretized into integer expression value labels with relative ordering significance; S23: Construct a state tag sequence for each cell to indicate state information; S24: The embedding of the gene identifier sequence, the embedding of the expression value label after mapping through a fully connected network, and the embedding of the state label sequence are added bit by bit to obtain the cell-level input embedding.

[0008] Furthermore, the binning process described in step S22 is based on the independent calculation of bin boundaries according to the expression value distribution of each cell, mapping non-zero expression values ​​to different bin intervals, so that the gene expression values ​​of different cells remain relatively consistent.

[0009] Furthermore, the Mamba network module in step S3 is constructed based on a state-space model and supports bidirectional feature extraction to simultaneously capture the forward and reverse information of the sequence; the Transformer network module performs global context modeling based on self-attention and cross-attention mechanisms; and the sparse hybrid expert network module uses a dynamic selection mechanism to adaptively activate some expert networks for computation based on input features.

[0010] Furthermore, the pre-training strategy described in step S4 includes: S41: Divide the data composed of cells into rows and partially mask the gene data within a single cell; S42: Perform gene expression value prediction task, predict the expression value of masked genes using known gene expression values ​​and contextual information, in order to learn the local regulatory relationships between genes; S43: Perform a cell characterization learning task, add a special marker at the beginning of the input sequence to aggregate global cell characterization, and estimate gene expression levels by the inner product of cell embedding and gene query vector; In the initial stage of pre-training, the warm-up training of step S42 is performed first, and then steps S42 and S43 are performed together for joint training.

[0011] Furthermore, in both the gene expression value prediction task and the cell characterization learning task, mean squared error is used as the loss function to measure the difference between the predicted and true expression values ​​of the masked gene, and the difference between the gene expression values ​​estimated based on cell characterization and the true expression values, respectively.

[0012] Furthermore, the pre-trained base model is used to perform any of the following downstream tasks: cross-conditional cell type annotation, cross-cancer zero-sample cell type annotation, cross-batch data integration and batch effect elimination, multi-omics data integration and alignment, spatial transcriptome topology analysis, and gene perturbation response prediction.

[0013] The present invention also includes an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for constructing a single-cell transcriptome basic model based on the Mamba-Transformer hybrid architecture.

[0014] The present invention also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture described above.

[0015] The beneficial effects of this invention are: The basic model based on large-scale single-cell transcriptome data pre-training (scMATE) proposed in this invention introduces a self-supervised learning paradigm, enabling it to mine high-order co-expression relationships between genes in massive unlabeled data and extract universal cell and gene embedding representations with strong transferability. The scMATE of this invention outperforms other mainstream methods in cross-platform and cross-modal integration and universal feature representation of single-cell and spatial transcriptome data. Furthermore, this invention demonstrates excellent adaptability and robustness in downstream tasks such as cell type annotation, multi-omics data fusion, spatial topology analysis, and complex gene regulation and perturbation prediction, highlighting the wide applicability and strong transferability of scMATE as a universal pre-trained basic model for single cells in computational biology. Compared with existing technologies, the single-cell transcriptome basic model construction method based on the Mamba-Transformer hybrid architecture described in this invention has the following technical features and beneficial effects: 1. Significantly improves data signal-to-noise ratio and feature robustness: By learning global context information from massive data, scMATE can reconstruct features that conform to biological laws for the random dropout event that is common in single-cell data, and successfully overcomes the interference of zero inflation and technical noise on feature extraction. 2. Strong cross-batch and cross-modal alignment capabilities: The unified characterization space constructed in this invention can effectively isolate the systematic batch effects brought about by multiple donors and multiple sequencing platforms; in the integration of transcription-protein, transcription-epigenetic and other multi-omics and the joint analysis of different spatial transcriptomics platforms, the model can accurately preserve the real tissue microenvironment topology and cellular heterogeneity. 3. Reduced reliance on annotations and significant savings in computational resources: Leveraging its powerful knowledge transfer capabilities, scMATE significantly reduces its dependence on high-quality manually labeled data in complex downstream tasks such as cell type annotation and gene perturbation response prediction. Even in small sample or complex dual-gene joint perturbation scenarios, it can still maintain high-accuracy prediction performance. This greatly reduces the need for expensive manually labeled data and effectively saves computational resources. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a diagram illustrating the overall framework of the scMATE model of this invention. Figure 2 Annotation results for cell types on the MS dataset; Figure 3 Annotate the confusion matrix diagram for cell types on the MS dataset; Figure 4 Annotation results for cell types on the Myeloid dataset; Figure 5 Annotate the confusion matrix diagram for cell types on the Myeloid dataset; Figure 6 Annotation results for cell types on the hPancreas dataset; Figure 7 Annotate the confusion matrix diagram for cell types on the hPancreas dataset; Figure 8 The results are shown in the graph for 25 epochs on the BMMC dataset; Figure 9 The results are plotted on the PBMC dataset after 25 epochs. Figure 10 The perturbation results for the first 20 genes in the Adamson dataset are shown in the figure. Figure 11 The image shows the perturbation results for the first 20 genes in the Norman dataset. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The following description, in conjunction with the accompanying drawings... Figure 1-11This paper further explains the method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture.

[0018] Example 1 This embodiment specifically illustrates the complete technical implementation process of the single-cell transcriptome basic model construction method based on the Mamba-Transformer hybrid architecture of the present invention. The following is combined with... Figure 1 The overall framework of the scMATE model is shown, and the data preprocessing, input embedding construction, hybrid network architecture, and pre-training strategy are explained in detail.

[0019] Constructing a high-quality pre-training dataset is crucial for improving model performance. The data used in this invention comes from the publicly available CELLxGENE single-cell data resource. This data includes samples from various high-throughput sequencing platforms, such as Illumina HiSeq, Ion Proton, MinION, and SeqCap, ensuring data diversity and covering different tissue types and sequencing protocols. To construct a comprehensive dataset supporting pre-training, this invention collected over 50 million single-cell samples, covering multiple tissue types including brain, lung, liver, kidney, and blood.

[0020] To ensure data quality and validity during dataset construction, the original gene expression matrix underwent rigorous quality control and standardization. The specific process is as follows: 1. Cell Screening: To remove abnormal data caused by sequencing errors or cell rupture, cells with fewer than 200 genes were first screened out. Simultaneously, cells with more than 20% mitochondrial genes were removed to avoid these cells being in a state of stress or apoptosis, thus ensuring data quality.

[0021] 2. Gene screening: To reduce the interference of technical noise, genes that are not expressed in most cells were removed during screening, while those that are expressed in multiple cells were retained, thus ensuring the validity and relevance of the data.

[0022] 3. Standardization: To reduce the differences in sequencing depth and between different batches, standardization techniques were adopted, including total number-based normalization, logarithmic transformation, and variance stabilization, to ensure the consistency and comparability of gene expression values, thereby improving the usability of the data.

[0023] In pre-trained models using single-cell sequencing data, a well-designed input embedding is crucial for constructing more accurate biological feature representations. Therefore, the first step in processing the training data is to convert the raw discrete data into continuous vectors, facilitating feature encoding and representation learning in the subsequent network structure. To accelerate training, scMATE restricts the input to non-zero expressed genes for each cell, thereby reducing computational overhead. To construct the embedding vectors for single-cell data, scMATE's input is divided into three parts, used to describe the gene ID, gene expression value, and state label, respectively. The specific construction process is as follows: (1) Gene ID Each gene is considered the smallest unit of information, similar to a word in natural language processing, and a gene ID represents the unique identity of a gene within the model. Therefore, a gene ID can be viewed as a token. During the pre-training initialization phase, a unique numerical identifier is assigned to each gene, and a unified gene vocabulary is constructed based on these identifiers. This method offers significant flexibility, enabling the integration of multiple research datasets from different gene sets. Specifically, by taking the union of genes from all studies, different gene tokens can be integrated into a general vocabulary. Furthermore, special markers are added to the vocabulary, such as... <cls>This indicates that all genes are aggregated into a single cellular representation. <pad>This indicates that the input will be padded to a fixed length. The gene marker sequence for each cell is represented as follows: Where N is the number of genes used to construct the cell input vector.

[0024] (2) Gene expression value markers Single-cell data preprocessing includes library size correction and log1p transformation; however, batch effects are still difficult to completely eliminate. To further mitigate gene expression differences caused by different sequencing platforms and data sources, scMATE uses an expression value binning mechanism, discretizing continuous expression values ​​into integer values ​​with relative ordering significance. Genes with non-zero expression are divided into B bins (1 to B) according to their original counts, with each bin containing the same number of genes. If a gene expression value is zero, the bin value is 0. Bin boundaries are calculated independently based on the expression value distribution of each cell; therefore, different cells may have different bin boundaries, thus maintaining relative consistency of expression values ​​across cells. The bin value of gene j in cell i is defined as follows: in, The original expression matrix, This represents the binned expression values. The final input vector of the binned expression values ​​for cell i is represented as: (3) Status label The gene ID sequence provides the model with genetic information, while the state label serves as additional input information. The state label is converted into unique numbers during word segmentation, and the state label sequence for each cell is represented as follows: (4) Intact cell embedding To improve the model's ability to represent cellular features and enhance its modeling stability for low-expression genes, scMATE combines gene IDs, expression values, and state labels to construct the input embedding of the pre-trained model. Specifically, gene IDs and state labels are mapped to a fixed-dimensional vector space through independent embedding layers, while expression value labels are non-linearly mapped through a fully connected network, thereby enhancing their expressive power. The embeddings of these three types of labels are integrated bit-by-bit to form a holistic embedding representation at the cell level. The input embedding representation for cell i is: in and For embedding layer, This is a fully connected network designed to process expression values. This input embedding design, through multi-level and multi-dimensional information integration, not only enhances the model's learning ability and stability but also provides stronger support for subsequent cell feature encoding, pre-training, and downstream tasks.

[0025] To effectively handle high-dimensional and sparse single-cell RNA sequencing data, the scMATE model employs an innovative hybrid architecture combining Mamba, Transformer, and MOE networks. This architecture fully leverages the advantages of Transformer in local sequence modeling and self-attention mechanisms, while introducing a sparse expert mechanism through MOE, effectively reducing computational complexity while maintaining model expressive power. The introduction of Mamba further enhances the model's performance in gene regulatory networks and cell type differential analysis, enabling it to achieve higher accuracy and efficiency in cell heterogeneity identification and gene function prediction. Through the sparse expert mechanism, scMATE can adaptively select the most relevant computational path when processing massive amounts of data, thereby optimizing resource utilization and ensuring efficient processing of large-scale datasets.

[0026] scMATE mainly consists of the following functional network modules: Mamba network module: This sequence modeling architecture, based on the State-Space Model (SSM), achieves linear time complexity through a dynamic selection mechanism, solving the quadratic complexity problem of the Transformer. It enables efficient processing of long sequence data, linear scalability, a selective mechanism, and hardware optimization. Furthermore, building upon standard Mamba, the model allows for adjustable parameters to control whether bidirectional Mamba technology is used during training. Bidirectional Mamba technology refers to the use of bidirectional feature extraction during the Mamba layer modeling process. This means that when processing the input sequence, not only the forward information of the sequence is considered, but also the backward information is captured simultaneously, comprehensively improving the sequence modeling effect. The Mamba input module of the Lth layer is defined as follows: Transformer network module: This module uses self-attention and cross-attention mechanisms to process interactions between all positions simultaneously. It perceives the positional relationships of all information from a near-global perspective. Within this architecture, it receives the output of the previous layer as input. The Transforem input module of the Lth layer is defined as follows: Sparse Hybrid Expert Network Module: A structure alternating between traditional feedforward neural networks and sparse hybrid expert networks is adopted, with a sparse MOE module following every two network layers. A dynamic selection mechanism is used to choose several experts for processing based on the input features. Each expert is an independent computational unit, focusing on specific feature processing. The MOE module output is represented as follows: in, This represents the activation probability of expert i, determined by the dynamic selection mechanism. Let represent the i-th expert network, and k represent the total number of activated experts. The sparse MoE network maintains high model representational power while reducing redundant computation by limiting the number of activated experts through a sparsity mechanism.

[0027] Residual connectivity and layer normalization network module: To ensure the stability of deep neural network training, the output of each submodule is typically normalized immediately after being connected via residuals. This operation helps mitigate the vanishing gradient problem common in deep networks and significantly accelerates model convergence. The following are the general steps of this process: The input x and the output F(x) of the submodule are added together via residual connection: Normalize y: α: Dynamic scaling parameter (learnable), controls the nonlinear compression strength of the input y; γ and β: Affine parameters (learnable), functioning the same as γ and β in the normalization layer, used to adjust the range and offset of the output; tanh function: replaces the traditional normalization operation of "subtract mean and divide variance", compressing the input to [-1,1] while maintaining the approximately linear properties of the central region.

[0028] Existing experiments have shown that using MAE as the pre-training task for Transformer and AR as the pre-training task for Mamba can maximize the performance of both networks. However, in fusion architectures, neither MAE nor AR alone fully realizes the potential of the fusion architecture. To better leverage the advantages of the Mamba and Transformer fusion architecture, this invention employs a pre-training strategy combining local MAE and AR. Local MAE is used to enable the Transformer module to learn effective local attention, while AR helps the Mamba module learn meaningful contextual information. Based on this idea, this invention introduces two pre-training tasks: gene expression value prediction and cell representation learning. The gene expression value prediction task focuses on learning the regulatory relationships between genes within a cell, allowing the model to better understand gene interactions and their roles in cellular states. The cell representation learning task focuses on learning global relationships between cells, helping the model capture the similarities and differences between different cells, thereby constructing a more comprehensive and accurate cell representation. Through these two different pre-training strategies, scMATE enables Mamba and Transformer to fully leverage their respective strengths in their respective tasks. Local MAE helps the Transformer capture local relationships between genes within a cell, while AR helps Mamba understand the cell's contextual information from a global perspective. Thus, scMATE not only excels in local modeling of cellular features but also demonstrates powerful capabilities in learning relationships at the global level. Ultimately, this optimized fusion architecture improves the model's performance on single-cell data and facilitates further optimization in feature modeling at various levels.

[0029] The pre-training strategy of scMATE is implemented as follows: Data splitting: Divide the scRNA-seq data into rows: Each line It consists of the following tokens: Masking and Prediction: Select a portion of the tokens from each row and mask them, then... Indicates line The index of the masked toekn. For a given row It is necessary to predict all masked tokens simultaneously. in This represents all rows preceding the i-th row.

[0030] The prediction of the token in the i-th row depends on all previous rows and the tokens visible in that row, which can be represented as: The mask design and prediction task are as follows: Masking strategy design First, the input embedding of each layer of the scMATE network is... This contains M tokens. The output of the stacked network modules can be simply represented as follows: Each token belongs to one of the following three categories: the first category is the starting position token, such as those used for learning cell representations. <cls>Each cell is similar to a "sentence" composed of genes, which characterizes... Gene-level representations obtained through aggregation learning Decision made <cls>A marker is added to the beginning of the input marker, and the final embedding at that position is extracted as a representation of the cell. The second category is known genes, which contain the expression values ​​that are not masked and their corresponding embeddings. The third category is unknown genes, whose expression values ​​are masked and cannot be known by the model, and whose true values ​​are the model's prediction targets.

[0031] Mamba uses an autoregressive training method that reconstructs expression values ​​row by row, referencing all previous rows when generating the next expression value. This helps the model learn the relationships between cell levels. Simultaneously, an attention mask matrix is ​​constructed within the Transformer layer. This mask matrix is ​​used to limit the attention connections between tokens in the Transformer. It ensures that each unknown token, when calculating attention, can only access known tokens, the starting token, and its own embedding, and cannot access the positions of other unknown tokens. In this way, the model generates and infers expression values ​​only based on known information.

[0032] Gene expression value prediction task To better learn local relationships between genes, scMATE employs a gene expression value prediction task. The goal of this task is to predict the expression values ​​of masked genes, enabling the model to reconstruct a complete cellular expression profile even with missing information. This design allows the model to better learn regulatory relationships between genes within local contexts, enhancing its understanding of gene expression dependencies. In this application, a subset of genes in the input cell expression sequence is first masked, with the expression values ​​of the masked genes set to null values, preventing the model from accessing this portion of the expression information. During the forward propagation phase, Mamba generates an estimate of the expression level at the current position based on all previous expression levels. In the Transformer, attention is calculated based on the attention mask, and features are extracted to output a representation for expression level estimation. Next, a multilayer perceptron network is used to predict the positions of the masked genes. The calculation formula can be simply expressed as: in, This represents the vector of cellular gene expression values ​​predicted by the model. To evaluate the error between the model's predictions and the actual values, the mean squared error is used as the loss function L, as shown in the following formula: in The set representing the output locations of unknown genes. These are the actual gene expression values ​​to be predicted. By reconstructing unknown gene expression values ​​from known gene expression values, the model enhances its ability to model potential regulatory relationships in the gene expression space.

[0033] Cell characterization learning task The goal of this task is to enable the model to learn and capture global and local features between cells, thereby generating effective cell representations, and based on this, to infer the expression values ​​of individual genes in the cells.

[0034] During pre-training, the model first undergoes a warm-up phase to stabilize parameters, where only gene expression value prediction is performed. At this stage, cell representation learning is not conducted; the focus is solely on learning the relationships between genes, aiding in rapid model learning. After the warm-up phase, joint training is performed on cell representation learning and gene expression value prediction tasks, allowing the model to optimize at both the cell and gene levels. In this task, the model calculates the global embedding vector for each cell. Map and place it at the beginning of the input sequence, as mentioned earlier. <cls>Tagging. This tag is used to store the overall cell characterization and plays a dominant role in subsequent cell-level prediction of gene expression throughout the cell. Simultaneously, a query vector is assigned to each gene. , representing the characteristic orientation of the gene in the expression space. The model calculates the inner product of the gene query vector and the cell representation vector to estimate the expression level of gene j in the cell. The simplified calculation formula is as follows: Similar to gene expression prediction tasks, cell characterization learning tasks use mean squared error as the loss function to measure the difference between expressed values ​​and true values. The loss function is defined as follows: in The set representing the output locations of unknown genes. It is the actual expressed value. These are model predictions.

[0035] This design, through joint training of masked prediction and cell representation, enables the model to simultaneously learn local co-expression relationships between genes and global cellular characteristics under self-supervised conditions. The masking mechanism facilitates the model's reconstruction of expression profiles with incomplete information, enhancing its ability to model gene regulation dependencies; the combination of cell embeddings and gene query vectors makes the predictions more biologically interpretable. This method not only improves the accuracy of expression value reconstruction and model robustness but also significantly enhances the discriminative power of cell representation and the generalization performance of downstream tasks.

[0036] decoding This invention utilizes a Transformer decoder, which can reconstruct regions based on encoder features by applying a decoder mask. In contrast, the Mamba decoder, due to its unidirectional scanning characteristic, struggles to reconstruct the entire local region simultaneously. The decoder employs a unique row-by-row decoding strategy, allowing autoregressive decoding of a single row of tokens at a time, enhancing the network's ability to capture local features and contextual relationships between regions.

[0037] This invention combines deep learning with bioinformatics, using deep learning methods to address the challenges of high-dimensional sparsity, excessive computational resource consumption for long-sequence modeling, and insufficient model generalization ability encountered during the pre-training process of massive single-cell transcriptome sequencing (scRNA-seq) data. Single-cell transcriptome sequencing technology can provide gene expression profile data at single-cell resolution, revealing not only intercellular heterogeneity but also recording complex molecular regulatory networks within cells. This enables researchers to better understand cell developmental trajectories, gene co-expression relationships, and potential pathogenic mechanisms of diseases.

[0038] Therefore, this invention selects pre-training datasets by collecting transcriptome data from different high-throughput sequencing platforms from the publicly available CELLxGENE single-cell data resource pool, using multi-platform, multi-tissue data to train the model. Specifically, these data include samples produced using different sequencing protocols and technologies such as Illumina HiSeq, Ion Proton, MinION, and SeqCap, ensuring the diversity of data sources. This invention has collected and integrated over 50 million single-cell samples, comprehensively covering various complex tissue types such as the brain, lungs, liver, kidneys, and blood. Since differences between different sequencing platforms, batches, and biological tissues introduce significant technical noise and batch effects, this highly heterogeneous massive dataset constitutes the benchmark data for training and evaluating the robustness of the model (scMATE) of this invention.

[0039] To fully utilize the massive amounts of data mentioned above, this invention constructs an innovative fusion network architecture—scMATE. During the data input phase, this invention performs rigorous quality control on the data and innovatively fuses gene IDs, binned expression value labels, and state labels through bit-by-bit addition to construct a more accurate continuous vector embedding. In its core architecture, scMATE deeply integrates a bidirectional state-space model (Mamba), a Transformer, and a sparse hybrid expert network (MoE). The bidirectional Mamba module efficiently captures complex dependencies in long sequences and reduces computational complexity; the Transformer module supplements global context awareness; and the MoE module significantly expands the model's parameter count while maintaining inference efficiency through dynamic selection mechanisms and load-balanced loss. In terms of training strategy, this invention employs a multi-task-based region mask autoregressive strategy, combining gene expression prediction and cell characterization learning tasks to promote joint optimization of the model at both the cellular and gene levels.

[0040] The data analysis focused on the performance of scMATE in overcoming batch effects and extracting long-sequence gene features. Specifically, scMATE, after large-scale pre-training, can be widely applied to downstream tasks of single-cell data, including but not limited to high-precision cell type annotation, gene perturbation response prediction, cross-sequencing platform data integration and denoising, and cell heterogeneity identification and deep gene regulatory mechanism mining, highlighting the method's broad applicability and excellent computational efficiency in processing massive single-cell data.

[0041] The overall framework of the model in this invention patent is as follows: Figure 1 As shown.

[0042] Example 2 This embodiment specifically illustrates the performance verification results and effect comparison of applying the pre-trained basic model constructed in Embodiment 1 to various downstream tasks. The following detailed explanation of the practical application effects of this invention is based on representative scenarios such as cross-disease state annotation of multiple sclerosis, zero-sample annotation of myeloid cells across cancer types, cross-batch integration of human pancreas, multi-omics data fusion and spatial topology analysis, and gene perturbation response prediction.

[0043] High-precision cell type annotation of single-cell transcriptome (scRNA-seq) data is a core step in understanding tissue biological heterogeneity and the mechanisms of cellular function and disease development. However, in practical applications, different physiological or pathological states can cause drastic changes in cellular gene expression patterns, making it difficult for models to maintain stable generalization ability under different conditions (such as from healthy to disease states). Therefore, accurate cross-condition cell type annotation is crucial for assessing whether a model has truly learned biologically meaningful universal cellular characterizations. By achieving high-quality cell annotation across healthy and disease states, it is possible to effectively overcome gene expression perturbations and technical noise caused by pathological states, and to accurately track specific cell subpopulations in highly heterogeneous pathological microenvironments. This provides a reliable computational foundation for studying the pathogenesis of complex diseases, discovering potential drug targets, and constructing high-resolution disease reference atlases.

[0044] To evaluate the generalization ability and feature extraction accuracy of the scMATE model under different health states, this invention selected a multiple sclerosis (MS) dataset from the EMBL-EBI database for cell type annotation. This dataset contains 9 healthy control samples and 12 MS patient samples. In the experimental design, all healthy samples were assigned to the training set, while MS patient samples were assigned to the test set, thus constructing a rigorous "cross-disease state" prediction task. To eliminate interference from label mismatch and ensure the objectivity of the evaluation, a set of disease-specific cell types (including specific B cell subsets, T cells, and oligodendrocyte-associated B cells) appearing only in MS samples were removed during data preprocessing. Ultimately, 7,884 cells were used for training, and 13,468 cells were used for testing. The prediction results were compared and evaluated with expert annotations provided by the original study.

[0045] like Figure 2 As shown, this invention compares the clustering distribution of real cell types (Cell Type) and the predictions from the scMATE model (Predictions) in a two-dimensional UMAP space. The results show that although the model was only trained on healthy samples, it exhibits excellent representation mapping ability on a highly heterogeneous MS patient test set. The predicted cell distribution clusters show a high degree of consistency with the clustering manifold of the real labels, and the boundaries of each cell subpopulation are clear and accurate. Furthermore, combined with… Figure 3 Quantitative analysis of the confusion matrix reveals that scMATE achieves extremely high prediction accuracy across a variety of complex neural cell subtypes. For example, in the classification of different cortical excitatory neurons, various interneurons (such as SST-expressing / VIP-expressing interneurons), astrocytes, endothelial cells, and oligodendrocyte C lineages, the diagonal values ​​of the matrix widely reach extremely high levels ranging from 0.8 to 1.0. Even among cell subtypes with extremely similar transcriptomic profiles, the model did not exhibit serious misclassification. This strongly demonstrates that the scMATE hybrid architecture, based on massive pre-training data, can successfully overcome expression perturbations introduced by disease states, exhibiting excellent noise resistance, generalization performance, and extremely high biological recognition accuracy in cross-scenario single-cell type annotation tasks.

[0046] Furthermore, precise typing of immune cells in the highly heterogeneous tumor microenvironment (TME) is crucial for elucidating tumor immune escape mechanisms, identifying potential drug targets, and developing targeted immunotherapies. Myeloid cells (such as various macrophages and dendritic cells) exhibit extremely high state continuity and plasticity across different organs and cancer types. Accurately aligning the states of immune cells from different tissue origins is essential for understanding the common regulatory mechanisms of tumorigenesis and development.

[0047] To focus on examining the model's transfer learning and generalization capabilities in the highly heterogeneous tumor microenvironment, this invention utilizes a myeloid cell dataset retrieved from the Gene Expression Omnibus (GEO) database. This dataset contains myeloid cell samples from nine different cancer types. In the experimental design, a strict cross-cancer zero-shot evaluation strategy was employed, meaning the training set completely excludes cancer type classification data from the test set: data from six cancer types (UCEC, PAAD, THCA, LYM, cDC2, Kidney) were selected to construct the training set, while the remaining three cancer types (MYE, OV-FTC, ESCA) served as the test set. After random subsampling and data balancing, the training set contained 9,748 cells. A two-dimensional UMAP dimensionality reduction plot was used to visually evaluate the alignment performance and cell annotation accuracy of each method. Figure 4 As shown, comparing the actual cell types in the original data with the prediction results of the scMATE model, it can be seen that even in highly complex and continuous myeloid cell populations, scMATE can still generate well-defined and highly consistent clustered manifolds. Whether it's monocytes (such as Mono_CD14, Mono_CD16), various tumor-associated macrophages (Macro_SPP1, Macro_C1QC, etc.), or dendritic cell subpopulations (cDC, pDC), the spatial distribution patterns of the predicted results are highly consistent with the actual labels. This indicates that even when faced with microenvironmental technical noise perturbations from unknown cancer types and completely different tissue origins, scMATE can still accurately capture the potential, cross-cancer-type-universal conserved transcriptional features of myeloid cells, demonstrating superior feature alignment stability and accuracy.

[0048] At the same time, further integration Figure 5 Quantitative analysis was performed. The results showed that scMATE achieved extremely high recognition accuracy across multiple highly granular myeloid cell subpopulations. For example, for common dendritic cell subtypes (such as cDC1_CLEC9A, cDC2_CXCR4hi, cDC2_IL1B, cDC2_CD1A) and specific macrophage subtypes (such as Macro_LYVE1, Macro_FN1), diagonal prediction accuracy reached a very high level of 0.8 to 1.0. Notably, myeloid cells in the tumor microenvironment typically exhibit continuous and highly overlapping differentiation states. Traditional algorithms are prone to subtype confusion when processing such data. However, the scMATE hybrid architecture, with its powerful long-sequence context awareness and efficient extraction of local features by a sparse hybrid expert network, successfully achieved excellent cross-cancer transfer learning on novel cancer types never before seen. This result not only demonstrates the model's strong zero-shot generalization ability but also showcases the important value of heterogeneous spatial alignment in analyzing the biological dynamics of immune cells across cancer types. The test set contained 3,430 cells.

[0049] In practical applications of single-cell sequencing data, significant systematic technical biases, known as the "batch effect," are often unavoidably introduced due to differences in laboratory operating procedures, sequencing platforms (technologies), and sample processing times. This non-biological factor often masks true cellular biological differences, leading to the failure of cross-dataset joint analyses. Therefore, the model's ability to effectively eliminate the batch effect and achieve high-quality integration and accurate annotation of data from multiple sources is a core indicator for evaluating its underlying architecture's noise resistance and characterization versatility.

[0050] To evaluate the ability of the scMATE model to eliminate batch effects, this invention utilizes the Human Pancreas dataset compiled by Chen et al. This dataset aggregates five independent scRNA-seq studies on the human pancreas and is a classic benchmark in bioinformatics for evaluating a model's ability to integrate data across batches. In the experimental design, this invention strictly divides the data according to its source (i.e., different study batches): data from a subset of studies is selected as the training set (10,600 cells, covering 13 cell types), while data from the remaining independent studies is used as the test set (4,218 cells, covering 11 cell types). This task is highly challenging, requiring the model not only to accurately identify complex pancreatic cell types but also to effectively overcome systematic biases introduced by different laboratories and sequencing technologies, even without prior knowledge of the test set batch distribution.

[0051] By creating dimensionality reduction visualizations, the cross-batch integration and annotation performance of the model was intuitively evaluated. For example... Figure 6 As shown, comparing the actual cell types in the original data with the predictions of the scMATE model, it is clear that the clustering topologies of the two are highly consistent in the two-dimensional UMAP space. Even when faced with completely unknown sequencing batches, scMATE can still accurately and closely map cells with the same biological attributes (such as purple alpha cells, green beta cells, and brown ductal cells), with clear boundaries between different cell populations and no batch separation due to technical bias. This intuitively proves that the model has successfully removed technical noise and extracted pure biological characteristics.

[0052] In addition, further integration Figure 7 Quantitative analysis was performed on the confusion matrix. The results showed that scMATE exhibited excellent classification accuracy for the vast majority of pancreatic cell types. Specifically, in the classification prediction of acinar cells, alpha cells, ductal cells, endothelial cells, and PP cells, the similarity and prediction accuracy (diagonal values ​​of the matrix) both reached a perfect level of 1.0; and the accuracy was also as high as 0.9 for beta cells, delta cells, macrophages, and epsilon cells.

[0053] These experimental results demonstrate that scMATE not only achieves high-precision cell type annotation but also possesses robust feature alignment and integration capabilities under complex or changing sequencing technology environments. Leveraging the global contextual constraints provided by large-scale pre-training, the model effectively overcomes systematic biases arising from different laboratories and sequencing technologies, further highlighting the significant value of this invention in intelligent mining and reference atlas construction of multi-source single-cell data across various scenarios.

[0054] In practical biological research, single transcriptome data often falls short of comprehensively depicting the complex regulatory states within cells. Effectively integrating single-cell transcriptome (RNA) data with multimodal data such as surface protein expression, chromatin accessibility and chromatin availability (ATAC), and spatial location information can help us gain a deeper understanding of cellular heterogeneity. However, inherent modal differences exist between different omics, often accompanied by their own unique systematic noise. Therefore, the model's ability to accurately uncover and align potential cross-modal biological associations while eliminating multi-platform and multi-donor batch effects is a crucial criterion for evaluating its underlying architecture's generalization ability.

[0055] To verify the cross-modal representation learning and spatial topology analysis capabilities of the scMATE model in multimodal data scenarios, this invention selected three representative benchmark datasets for evaluation: 1. BMMC Dataset (Transcriptome-Proteome Pairing): Contains bone marrow mononuclear cell samples from 12 healthy donors, simultaneously measuring the expression levels of 13,953 genes and the abundance of 134 surface proteins, covering 45 immune cell subtypes. This data is primarily used to evaluate the model's representational associations between the transcriptome and proteome and its ability to eliminate donor batch effects.

[0056] 2.10x Multiome PBMC Dataset (Transcriptome and Epigenetics Paired): Contains 9,631 PBMC cells from healthy donors, and provides transcriptome and chromatin accessibility (ATAC) information for analyzing model performance in transcriptional regulation and epigenetic signal integration.

[0057] 3. Fetal Lung Spatial Transcriptome Dataset: This dataset contains samples from two different spatial technology platforms, Visium and Xenium, during the GW15 developmental stage. A total of 3,745 spatial loci were screened to observe the model's representational stability and cross-platform alignment capabilities in spatial structural data.

[0058] In the three tasks mentioned above, this invention compared and analyzed scMATE with mainstream single-cell analysis tools and multi-omics integration algorithms in the current field (including Seurat V4, scGlue, scTranslator, BayesSpace, SpaSEG and the general large model scGPT). The relevant quantitative evaluation results are shown in Tables 1 to 3.

[0059] First, in the CITE-seq protein-transcriptome alignment task (see Table 1), scMATE demonstrated relatively stable cross-modal integration capabilities. Particularly noteworthy is its performance on the Adjusted Rand Index (ARI), a measure of cell type cluster consistency, where scMATE achieved a score of 0.912, significantly higher than the general pre-trained model scGPT (0.725) and the specialized cross-modal prediction model scTranslator (0.625). This indicates that the cell representations generated by scMATE maintained a high degree of consistency with the original biological classifications. Furthermore, scMATE also achieved high scores on the ASWbatch and GraphConn metrics, reflecting batch mixing, objectively demonstrating that the model achieved a good balance between batch mixing and structure preservation during the integration of data from 12 different donors.

[0060] Table 1 Comparison of experimental results on the BMMC dataset Secondly, scMATE also achieved stable results when facing the epi-transcriptome integration challenge of 10x Multiome PBMC data (see Table 2). On the clustering consistency indices NMI (0.812) and ARI (0.847), scMATE's scores were similar to existing multi-omics integration methods and slightly higher than scGLUE and scGPT. Notably, scMATE achieved the highest score of 0.688 on the mean silhouette coefficient (ASWcell) indices, which measure the density within cell clusters and the degree of separation between clusters. This result demonstrates that even with significant modal barriers between RNA and ATAC data, the model can still project them into a unified feature space and form a clearly distinguishable biological representation structure.

[0061] Table 2 Comparison of experimental results on the PBMC dataset Finally, for the fetal lung spatial transcriptome dataset containing spatial location information (see Table 3), scMATE also demonstrated competitiveness in spatial topology modeling. On the spatial clustering consistency indices NMI (0.936) and ARI (0.976), the model of this invention outperformed the classic algorithm BayesSpace and was in the same tier or slightly higher than SpaSEG and scGPT. More importantly, on the batch correlation index ASWbatch, scMATE achieved a superior performance of 0.693. This indicates that when integrating data from Visium and Xenium platforms with drastically different underlying principles, scMATE can mitigate the impact of differences in technology platforms to a certain extent, while properly preserving the continuity and topological associations of the internal spatial structure of the tissue microenvironment.

[0062] Table 3 Comparison of experimental results on the fetal lung dataset In summary, scMATE demonstrates comprehensive adaptability in multi-omics integration and biological clustering analysis tasks. Whether in transcription-protein association mapping, deep transcription-epimetic integration, or spatial topological analysis of complex tissues, the framework proposed in this patent achieves a good balance between batch correction and preservation of true biological signals. Considering various metrics, scMATE performs at a similar or even superior level to existing dedicated tools or general pre-trained models in different task scenarios, providing a reliable computational method for decoding multi-dimensional life maps.

[0063] Gene perturbation prediction is one of the most challenging downstream tasks in single-cell data analysis. By predicting the overall changes in the cell transcriptome after a specific gene is knocked out or repressed, we can intuitively test whether the model has truly learned the underlying complex gene regulatory network. To evaluate scMATE's capabilities in this area, this invention selected two classic benchmark datasets based on CRISPR perturbation (CRISPRi) technology: the Adamson dataset and the Norman dataset. Both datasets are constructed based on controlled gene editing experiments and are derived from the K562 leukemia cell line, but they differ in the complexity of the perturbations. The Adamson dataset mainly records the transcriptome response under 87 single-gene perturbation conditions; while the Norman dataset is larger in scale, containing not only 105 single-gene perturbations but also introducing 131 combinations of dual-gene joint perturbations. This data setup, progressing from simple to complex, provides a reliable experimental basis for systematically examining the model's response prediction capabilities under different conditions.

[0064] On the Adamson dataset for single-gene perturbations (see Table 4), scMATE demonstrated excellent fitting performance. On the Pearson metric, which measures overall genome-wide expression correlation, scMATE achieved a score of 0.991. More importantly, on the Pearson_de_delta metric, which specifically measures the model's ability to capture the magnitude of expression changes (i.e., the difference between the actual and predicted values), scMATE achieved a score of 0.831, leading all compared models. This indicates that in single-gene perturbation scenarios, the model can not only grasp the global expression trend but also accurately quantify the magnitude of changes in differentially expressed genes after perturbation.

[0065] Table 4 Comparison of experimental results on the Adamson dataset Subsequently, this invention underwent more rigorous testing on the Norman dataset, which includes dual-gene joint perturbations (see Table 5). Since dual-gene perturbations are often accompanied by complex nonlinear genetic effects (such as epistasis), transcriptome change patterns are more difficult to predict. Experimental results show that even in more complex scenarios, scMATE maintains relatively stable performance. On the Pearson (0.985) and Pearson_delta (0.556) metrics, which reflect global expression structure and overall changes, scMATE outperforms other baseline models, including GeneCompass. Objectively speaking, although scMATE's score (0.644) is slightly lower than scGPT (0.702) in focusing on specific changes in differentially expressed genes (Pearson_de_delta), its overall prediction level is still in the same tier as other advanced algorithms. This subtle emphasis on metric performance reflects, to some extent, the characteristics of scMATE's underlying hybrid architecture—this architecture, when integrating long-range inter-gene dependence on local contextual information, tends to maintain the coherence and robustness of global expression patterns.

[0066] Table 5 Comparison of experimental results on the Norman dataset The results from the two datasets demonstrate that scMATE possesses reliable transcriptome change modeling capabilities under both single-gene perturbation and complex dual-gene combined perturbation conditions. This indicates that the pre-training framework can effectively extract regulatory patterns between genes, providing powerful tool support for complex biomedical applications such as gene function analysis and drug target screening.

[0067] In summary, traditional "de novo" analysis methods based on single datasets are insufficient to meet the demands of modern computational biology for in-depth data mining, given the prevalent high-dimensional sparsity, extreme zero-inflation technical noise, and complex batch effects arising from cross-platform and cross-modal data in single-cell and spatial transcriptome data. To address this, this invention proposes and constructs a single-cell general foundational model (scMATE) based on large-scale data pre-training. This framework successfully transforms massive transcriptome data into highly universal cell and gene embedding representations by introducing a self-supervised learning paradigm and an innovative hybrid network architecture.

[0068] In a series of rigorous benchmark tests, scMATE demonstrated stable and comprehensive downstream task adaptability: First, in multi-omics integration and spatial topology analysis scenarios, scMATE effectively overcomes the modal barriers between different underlying sequencing technologies. Whether performing joint mapping of transcriptome and proteome (CITE-seq), transcriptome and epigenome (10xMultiome), or processing data fusion from different spatial transcriptome platforms (Visium and Xenium), this model can completely eliminate interference from multiple donors and multiple platform batches while accurately preserving the true cellular heterogeneity and spatial topological coherence of the tissue microenvironment. Its overall performance meets or exceeds that of existing dedicated integration tools.

[0069] Secondly, in complex gene regulation modeling and perturbation prediction tasks, scMATE further demonstrated its ability to analyze underlying gene interaction networks. Faced with single-gene and even complex dual-gene joint perturbations mediated by CRISPR interference technology, the model not only keenly captures the dynamic changes in genome-wide expression trends under controlled conditions, but also provides reasonable quantitative fitting of the response amplitudes of differentially expressed genes, validating its structural advantages in integrating long-range inter-gene dependencies and local contextual information.

[0070] In summary, the scMATE model proposed in this invention breaks through the limitations of data silos and redundant computational consumption in previous single-cell data analysis, providing a unified computational foundation with both high robustness and strong transferability. This framework not only provides a highly reliable computational solution for analyzing complex, multi-dimensional, cross-modal biomaps, but also lays a solid algorithmic foundation for subsequent biomedical applications such as cell development trajectory inference, analysis of complex disease pathogenesis mechanisms, drug target screening, and personalized medication evaluation. It has significant scientific value and broad prospects for engineering transformation.

[0071] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.< / cls> < / cls> < / cls> < / pad> < / cls>

Claims

1. A method for constructing a basic single-cell transcriptome model based on a Mamba-Transformer hybrid architecture, characterized in that, The steps are as follows: S1: Acquire and preprocess single-cell transcriptome datasets, the preprocessing including quality control and standardization of raw gene expression data to obtain a normalized gene expression matrix; S2: Construct an input embedding that includes gene identifiers, gene expression value markers, and status labels, and convert the discrete gene information of each cell into a continuous vector representation to form a cell-level input embedding; S3: Construct a hybrid network architecture, which includes a Mamba network module, a Transformer network module, and a sparse hybrid expert network module. The Mamba network module and the Transformer network module are stacked alternately, and the sparse hybrid expert network module is connected after every two network modules, and combined with residual connections and layer normalization. S4: The hybrid network architecture is trained using a pre-training strategy, which combines gene expression value prediction task and cell representation learning task to jointly optimize model parameters at the gene level and cell level, thereby obtaining a pre-trained basic model.

2. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 1, characterized in that, The quality control in step S1 includes: deleting cells with fewer than a preset threshold of genes, deleting cells with a mitochondrial gene ratio exceeding a preset percentage, and deleting genes whose expression levels in cells do not meet preset conditions; the standardization process includes at least one of normalization based on the total number, logarithmic transformation, and variance stabilization.

3. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 1, characterized in that, The steps for constructing the input embedding in step S2 are as follows: S21: Assign a unique gene identifier to each gene, construct a gene identifier sequence based on the list of non-zero expressed genes in the cell, and add special markers for aggregating cell characterization and filling sequence length; S22: Binning is performed on the non-zero gene expression values ​​of each cell, and the continuous expression values ​​are discretized into integer expression value labels with relative ordering significance; S23: Construct a state tag sequence for each cell to indicate state information; S24: The embedding of the gene identifier sequence, the embedding of the expression value label after mapping through a fully connected network, and the embedding of the state label sequence are added bit by bit to obtain the cell-level input embedding.

4. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 3, characterized in that, The binning process described in step S22 is based on the independent calculation of bin boundaries according to the expression value distribution of each cell, mapping non-zero expression values ​​to different bin intervals, so that the gene expression values ​​of different cells remain relatively consistent.

5. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 1, characterized in that, The Mamba network module described in step S3 is built based on a state-space model and supports bidirectional feature extraction to simultaneously capture the forward and reverse information of the sequence; the Transformer network module performs global context modeling based on self-attention and cross-attention mechanisms. The sparse hybrid expert network module uses a dynamic selection mechanism to adaptively activate a portion of the expert network for computation based on the input features.

6. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 1, characterized in that, The pre-training strategy described in step S4 includes: S41: Divide the data composed of cells into rows and partially mask the gene data within a single cell; S42: Perform gene expression value prediction task, predict the expression value of masked genes using known gene expression values ​​and contextual information, in order to learn the local regulatory relationships between genes; S43: Perform a cell characterization learning task, add a special marker at the beginning of the input sequence to aggregate global cell characterization, and estimate gene expression levels by the inner product of cell embedding and gene query vector; In the initial stage of pre-training, the warm-up training of step S42 is performed first, and then steps S42 and S43 are performed together for joint training.

7. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 6, characterized in that, In both the gene expression value prediction task and the cell characterization learning task, mean squared error is used as the loss function to measure the difference between the predicted and actual expression values ​​of the masked gene, and the difference between the gene expression values ​​estimated based on cell characterization and the actual expression values, respectively.

8. The method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture according to claim 1, characterized in that, The pre-trained base model is used to perform any of the following downstream tasks: cross-conditional cell type annotation, cross-cancer zero-sample cell type annotation, cross-batch data integration and batch effect elimination, multi-omics data integration and alignment, spatial transcriptome topology analysis, and gene perturbation response prediction.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for constructing a single-cell transcriptome basic model based on the Mamba-Transformer hybrid architecture as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for constructing a basic single-cell transcriptome model based on the Mamba-Transformer hybrid architecture as described in any one of claims 1 to 8.

Images