BERT-based cell type deconvolution method and system

Abstract

The invention discloses a BERT-based cell type deconvolution method and a BERT-based cell type deconvolution system, which effectively overcome the problems of data noise and unstable gene expression in single cell transcriptome sequencing data, enhance the adaptability to gene-gene interaction in complex single cell data, effectively capture gene-gene complex relationship, and improve the accuracy of gene-gene deconvolution. The problem of dependence on hypervariant gene selection and data dimension reduction in a traditional method is solved, and the generalization ability and robustness of the model are remarkably improved. By means of the design of gene embedding, unstable gene expression is converted into more stable feature expression, the accuracy and robustness of cell type recognition are remarkably improved, and high performance is shown in complex heterogeneous cell sample annotation.

Description

Technical Field

[0001] The present invention relates to the field of bioinformatics, and more specifically, to a BERT-based cell type deconvolution method and system. Background Art

[0002] In the past few years, sequencing-based spatial transcriptomics has emerged as a powerful tool that can directly measure genome-wide gene expression in tissue sections with spatial information. This technology provides unprecedented opportunities to explore the details of tissue biology. Many in situ capture technologies, such as Spatial Transcriptomics (ST) and 10X Genomics Visium, use wells with a diameter of 55-100 microns to record the location of mRNA, so each well may cover multiple cells (1 to 20) at the same time, and these cells may be homogeneous or heterogeneous. Other high-resolution methods, such as Slide-seq and HDST, can increase the resolution to the level of close to single cells. However, because the fixed beads or wells used in these technologies cannot completely match the cell boundaries, the measured gene expression is often a mixture of multiple cells. This intra-well heterogeneity has a great impact on the accuracy of unsupervised clustering.

[0003] In order to better understand the distribution of cell types in spatial transcriptomics, many methods have attempted to integrate spatial transcriptomics data with scRNA-seq data. Commonly used methods include gene signature scoring and deconvolution. Gene signature scoring methods perform enrichment analysis through gene expression features of single-cell transcriptome sequencing (scRNA-seq), but their application is limited by the incomparability of scores between spatial locations or slices. Deconvolution methods estimate the specific cell type proportions at each spatial location by applying regression models or fitting probability distributions. Most existing deconvolution methods rely on marker genes inferred from reference single-cell data, which may be affected by high dropout rates and unnecessary gene expression fluctuations. In addition, marker genes selected based on the researcher's prior knowledge may be biased and may not reliably identify certain unique or new cell types. If the expression information of the marker gene set cannot be effectively combined, the accuracy of the cell type obtained by deconvolution will be difficult to guarantee.

[0004] Compared with methods that rely on marker genes, correlation-based methods infer cell types by comparing the correlation between the dataset to be annotated and the reference dataset. However, such methods are susceptible to technical batch effects. Although methods are available for correcting batch effects, it remains challenging to distinguish true biodiversity from technical differences and retain important biological variations. Correlation measures, such as cosine similarity and Pearson correlation coefficient, are also questionable in their robustness and effectiveness when dealing with high-dimensional and sparse single-cell data.

[0005] Supervised or semi-supervised methods transfer the annotation information of labeled data to unlabeled data by learning the pattern of gene expression profiles through models. The main advantage of these methods is that they do not rely on manually selected marker genes and are highly robust to noise and data variability. However, their limitation is that the data usually needs to be screened for highly variable genes (HVGs) and principal component analysis (PCA) dimensionality reduction before entering the model. The parameter settings of HVGs vary depending on the data set and batch, which hinders the generalization and versatility of such methods. In addition, the screening of HVGs is based on expression variance, which often makes the dominant cell type occupy the largest variance and may ignore the key genes of rare cell types. As a linear dimensionality reduction method, PCA inevitably loses some important information, such as high-dimensional features or cell type-unique gene expression. In addition, considering only HVGs may ignore the interactions and co-occurrence relationships between genes, especially the interactions between highly variable genes and non-highly variable genes. Summary of the invention

[0006] In order to overcome the defects of the prior art, such as lack of marker gene sets, improper handling of batch effects, and failure to effectively utilize gene interactions, the present invention provides a BERT-based cell type deconvolution method and system, which obtains an overall understanding of gene interactions by pre-training on a large amount of unlabeled single-cell data, and performs cell type deconvolution on user-specific spatial transcriptomics data in supervised fine-tuning, thereby improving the generalization and robustness of the model.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0008] The present invention provides a cell type deconvolution method based on BERT, comprising:

[0009] Obtain unlabeled single-cell data, labeled single-cell data, and spatial transcriptional data of the same tissue;

[0010] Preprocessing unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing;

[0011] Input the preprocessed unlabeled single-cell data of the same tissue into the constructed BERT model for pre-training to obtain a pre-trained BERT model;

[0012] The pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model;

[0013] The preprocessed spatial transcription data of the same tissue are input into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

[0014] Preferably, preprocessing the unlabeled single cell data, labeled single cell data, and spatial transcription data of the same tissue to obtain the unlabeled single cell data, labeled single cell data, and spatial transcription data of the same tissue after preprocessing includes:

[0015] Clean, filter and reduce noise on unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue, remove problematic genes from unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue, and filter genes expressed in less than a preset number of cells;

[0016] The unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue after cleaning, filtering and noise reduction are normalized to adjust the gene expression levels of the unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue to the same level.

[0017] Preferably, the pre-processed unlabeled single-cell data of the same tissue is input into the constructed BERT model for pre-training to obtain the pre-trained BERT model, including:

[0018] Constructing a BERT model, the model comprising an input embedding unit, an encoder unit, and a prediction unit connected in sequence;

[0019] The unlabeled single-cell data of the same tissue after preprocessing are randomly masked to obtain the masked gene expression values;

[0020] Set the reconstruction loss function, calculate the reconstruction loss function value according to the masked gene expression value and the unmasked gene expression value, and adjust the parameters of the BERT model;

[0021] When the reconstruction loss function value reaches a preset value or the preset number of training times is reached, the pre-trained BERT model is obtained.

[0022] Preferably, the input embedding unit comprises a first expression embedding layer, a gene embedding layer and a second expression embedding layer arranged in parallel; the encoder unit comprises a first encoder subunit and a second encoder subunit arranged in parallel;

[0023] The output end of the first expression embedding layer is connected to the input end of the first encoder subunit; the output end of the gene embedding layer is connected to the input ends of the first encoder subunit and the second encoder subunit; the output end of the second expression embedding layer is connected to the output end of the second encoder subunit;

[0024] The prediction unit includes a reconstruction layer, a convolution layer and a fully connected classifier; the output end of the first encoder subunit is connected to the input end of the reconstruction layer; the output end of the second encoder subunit is connected to the input end of the convolution layer; the output end of the convolution layer is connected to the input end of the fully connected classifier.

[0025] Preferably, the first encoder subunit comprises a first encoder layer, a second encoder layer, a third encoder layer, a fourth encoder layer, a fifth encoder layer and a sixth encoder layer connected in sequence;

[0026] An output terminal of the sixth encoder layer is connected to an input terminal of the reconstruction layer;

[0027] The second encoder subunit includes a seventh encoder layer, an eighth encoder layer, a ninth encoder layer, a tenth encoder layer, an eleventh encoder layer and a twelfth encoder layer connected in sequence;

[0028] The output end of the twelve encoder layers is connected to the input end of the convolutional layer;

[0029] The encoder layer includes a first self-attention layer, a second self-attention layer, a third self-attention layer, a fourth self-attention layer, a fifth self-attention layer, a sixth self-attention layer, a seventh self-attention layer, an eighth self-attention layer, a ninth self-attention layer, and a tenth self-attention layer connected in sequence.

[0030] Preferably, the reconstruction loss is:

[0031]

[0032] Where M is the number of single cells; N is the number of masked gene expression values; y i,j is the true expression value of gene j in cell i, p i,j is the predicted expression value of gene j in cell i.

[0033] Preferably, the pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model, including:

[0034] The single-cell data with labels of the same tissue after preprocessing are tested. If the gene name of the single-cell data is the gene name of the NCBI gene database, otherwise the single-cell data is converted to homologous genes and then input into the pre-trained BERT model;

[0035] Input the labeled single-cell data of the same tissue after detection into the pre-trained BERT model to obtain the predicted cell type label of the single cell;

[0036] Setting the prediction loss, calculating the prediction loss function value according to the predicted cell type label and the true cell type label of the single cell, and adjusting the parameters of the BERT model;

[0037] When the prediction loss function value reaches a preset value or the preset number of training times is reached, the trained BERT model is obtained.

[0038] Preferably, the prediction loss function is:

[0039]

[0040] Where M is the number of single cells; Z i is the true cell type label of single cell i, q i Predicted cell type label for single cell i.

[0041] Preferably, the preprocessed spatial transcription data of the same tissue is input into the trained BERT model for cell type deconvolution to obtain a deconvolution result, including:

[0042] The pre-processed spatial transcription data of the same tissue were input into the trained BERT model to obtain the score matrix of each cell type in each single cell;

[0043] A softmax operation is performed on the score matrix of each cell type in each single cell to obtain the ratio of various cell types in each single cell, and the cell type ratio is used as the deconvolution result.

[0044] The present invention also provides a BERT-based cell type deconvolution system for implementing the above method, comprising:

[0045] The data acquisition module acquires unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue;

[0046] A preprocessing module preprocesses the unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain the unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing;

[0047] The pre-training module inputs the pre-processed unlabeled single-cell data of the same tissue into the BERT model for pre-training to obtain a pre-trained BERT model;

[0048] The model training module inputs the pre-processed labeled single-cell data of the same tissue into the pre-trained BERT model for secondary training to obtain a trained BERT model;

[0049] The deconvolution module inputs the preprocessed spatial transcription data of the same tissue into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

[0050] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0051] This paper proposes a BERT-based cell type deconvolution method and system, which effectively overcomes the problems of data noise and unstable gene expression in single-cell transcriptome sequencing data, enhances the adaptability to gene-gene interactions in complex single-cell data, effectively captures the complex relationship between genes, solves the dependence on high-variable gene selection and data dimensionality reduction in traditional methods, and significantly improves the generalization ability and robustness of the model. Through the design of gene embedding, unstable gene expression is converted into a more stable feature representation, which significantly improves the accuracy and robustness of cell type identification, and shows higher performance in the annotation of complex heterogeneous cell samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flow chart of the BERT-based cell type deconvolution method described in Example 1;

[0053] Figure 2 Flow chart of the BERT-based cell type deconvolution method described in Example 2;

[0054] Figure 3 Schematic diagram of the structure of the BERT model described in Example 2;

[0055] Figure 4 is a schematic diagram of the structure of the encoder subunit described in Example 2;

[0056] Figure 5 Schematic diagram of the structure of the BERT-based cell type deconvolution system described in Example 3. DETAILED DESCRIPTION

[0057] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0058] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0059] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0060] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0061] Example 1

[0062] This embodiment provides a cell type deconvolution method based on BERT, such as Figure 1 As shown, including:

[0063] Obtain unlabeled single-cell data, labeled single-cell data, and spatial transcriptional data of the same tissue;

[0064] Preprocessing unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing;

[0065] Input the preprocessed unlabeled single-cell data of the same tissue into the constructed BERT model for pre-training to obtain a pre-trained BERT model;

[0066] The pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model;

[0067] The preprocessed spatial transcription data of the same tissue are input into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

[0068] This embodiment effectively overcomes the problems of data noise and unstable gene expression in single-cell transcriptome sequencing data, enhances the adaptability to gene-gene interactions in complex single-cell data, effectively captures the complex relationship between genes, solves the dependence on high-variable gene selection and data dimensionality reduction in traditional methods, and significantly improves the generalization ability and robustness of the model. Through the design of gene embedding, unstable gene expression is converted into a more stable feature representation, which significantly improves the accuracy and robustness of cell type identification, and shows higher performance in the annotation of complex heterogeneous cell samples.

[0069] Example 2

[0070] This embodiment provides a cell type deconvolution method based on BERT, such as Figure 1 As shown, including:

[0071] Obtain unlabeled single-cell data, labeled single-cell data, and spatial transcriptional data of the same tissue;

[0072] Preprocessing unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing;

[0073] Input the preprocessed unlabeled single-cell data of the same tissue into the constructed BERT model for pre-training to obtain a pre-trained BERT model;

[0074] The pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model;

[0075] The preprocessed spatial transcription data of the same tissue are input into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

[0076] Preprocessing the unlabeled single cell data, labeled single cell data, and spatial transcription data of the same tissue to obtain the unlabeled single cell data, labeled single cell data, and spatial transcription data of the same tissue after preprocessing, including:

[0077] Clean, filter and reduce noise on unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue, remove problematic genes from unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue, and filter genes expressed in less than a preset number of cells;

[0078] The unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue after cleaning, filtering and noise reduction are normalized to adjust the gene expression levels of the unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue to the same level.

[0079] The pre-processed unlabeled single-cell data of the same tissue is input into the constructed BERT model for pre-training to obtain a pre-trained BERT model, including:

[0080] Constructing a BERT model, the model comprising an input embedding unit, an encoder unit, and a prediction unit connected in sequence;

[0081] The unlabeled single-cell data of the same tissue after preprocessing are randomly masked to obtain the masked gene expression values;

[0082] Set the reconstruction loss function, calculate the reconstruction loss function value according to the masked gene expression value and the unmasked gene expression value, and adjust the parameters of the BERT model;

[0083] When the reconstruction loss function value reaches a preset value or the preset number of training times is reached, the pre-trained BERT model is obtained.

[0084] The input embedding unit includes a first expression embedding layer, a gene embedding layer, and a second expression embedding layer arranged in parallel; the encoder unit includes a first encoder subunit and a second encoder subunit arranged in parallel;

[0085] The output end of the first expression embedding layer is connected to the input end of the first encoder subunit; the output end of the gene embedding layer is connected to the input ends of the first encoder subunit and the second encoder subunit; the output end of the second expression embedding layer is connected to the output end of the second encoder subunit;

[0086] The prediction unit includes a reconstruction layer, a convolution layer and a fully connected classifier; the output end of the first encoder subunit is connected to the input end of the reconstruction layer; the output end of the second encoder subunit is connected to the input end of the convolution layer; the output end of the convolution layer is connected to the input end of the fully connected classifier.

[0087] The first encoder subunit comprises a first encoder layer, a second encoder layer, a third encoder layer, a fourth encoder layer, a fifth encoder layer and a sixth encoder layer connected in sequence;

[0088] An output terminal of the sixth encoder layer is connected to an input terminal of the reconstruction layer;

[0089] The second encoder subunit includes a seventh encoder layer, an eighth encoder layer, a ninth encoder layer, a tenth encoder layer, an eleventh encoder layer and a twelfth encoder layer connected in sequence;

[0090] The output end of the twelve encoder layers is connected to the input end of the convolutional layer;

[0091] The encoder layer includes a first self-attention layer, a second self-attention layer, a third self-attention layer, a fourth self-attention layer, a fifth self-attention layer, a sixth self-attention layer, a seventh self-attention layer, an eighth self-attention layer, a ninth self-attention layer, and a tenth self-attention layer connected in sequence.

[0092] The reconstruction loss is:

[0093]

[0094] Where M is the number of single cells; N is the number of masked gene expression values; y i,jis the true expression value of gene j in cell i, p i,j is the predicted expression value of gene j in cell i.

[0095] The pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model, including:

[0096] The single-cell data with labels of the same tissue after preprocessing are tested. If the gene name of the single-cell data is the gene name of the NCBI gene database, otherwise the single-cell data is converted to homologous genes and then input into the pre-trained BERT model;

[0097] Input the labeled single-cell data of the same tissue after detection into the pre-trained BERT model to obtain the predicted cell type label of the single cell;

[0098] Setting the prediction loss, calculating the prediction loss function value according to the predicted cell type label and the true cell type label of the single cell, and adjusting the parameters of the BERT model;

[0099] When the prediction loss function value reaches a preset value or the preset number of training times is reached, the trained BERT model is obtained.

[0100] The prediction loss function is:

[0101]

[0102] Where M is the number of single cells; Z i is the true cell type label of single cell i, q i Predicted cell type label for single cell i.

[0103] The pre-processed spatial transcription data of the same tissue is input into the trained BERT model for cell type deconvolution to obtain the deconvolution results, including:

[0104] The pre-processed spatial transcription data of the same tissue were input into the trained BERT model to obtain the score matrix of each cell type in each single cell;

[0105] A softmax operation is performed on the score matrix of each cell type in each single cell to obtain the ratio of various cell types in each single cell, and the cell type ratio is used as the deconvolution result.

[0106] In a specific embodiment, Figure 2 As shown, the following steps are included:

[0107] S1: Data collection and preprocessing: Collect millions of unlabeled scRNA-seq data of various cell types from different sources, obtain public spatial transcriptome data, and preprocess the spatial transcriptome data. The spatial transcriptome data includes the gene expression data of each spot and the spatial location information of each spot. The gene expression data includes gene expression matrix and metadata, etc.

[0108] Preprocessing includes: first, cleaning, filtering and noise reduction of data, eliminating problematic genes that appear in a small number of samples, filtering genes expressed in less than 200 cells, which may be caused by technical errors or other non-biological factors. Next, normalization and standardization are performed to adjust the gene expression of all samples to the same level to eliminate technical deviations between different samples.

[0109] S2: Model construction: BERT is a language model trained through a bidirectional Transformer architecture, focusing on understanding context, and is widely used in natural language understanding tasks such as question answering, sentiment analysis, and sentence classification. Here we also use its main architecture for single-cell analysis, using transformer as the model backbone. The main modifications are replacing the standard Transformer encoder with an efficient Performer encoder, and using gene expression embeddings as token embeddings and position embeddings.

[0110] like Figure 3 As shown, the BERT model includes an input embedding module, an encoder module and a classification module. The input embedding module includes gene embedding and expression embedding. The encoder module is composed of multiple Performer encoder layers, and the classification module includes a convolutional layer and a fully connected classifier. The input embedding module processes the gene expression data into gene embedding and expression embedding, and the embedded features are input into the encoder module to extract features layer by layer, and the output of the encoder module is input into the classification module. Among them, the gene expression data first generates a feature vector for each gene through gene embedding, and the expression value is discretized to generate expression embedding, and then the two embeddings are spliced ​​and input into the Performer encoder module for multi-head attention feature extraction. The encoder module includes multiple Performer encoder layers, which are connected in sequence, and each layer performs deep representation learning on the input features. The encoder output extracts local features through the convolutional layer, and generates a predicted probability distribution of cell types through a fully connected classifier.

[0111] Finally, the outputs of the classification module are aggregated through post-processing steps such as softmax to generate the final cell type annotation results.

[0112] Performer encoder: Since the BERT model is based on the Transformer as the basic unit, its computational complexity grows quadratically, so its scalability for long sequences is limited. However, the number of genes in scRNA-seq data may exceed 20,000, which makes it difficult for conventional Transformers to handle such large-scale inputs. To solve this problem, a matrix decomposition version of the Transformer, namely Performer, is used to expand the processing capability of sequence length. The attention mechanism formula in the Transformer is:

[0113] Att(Q,K,V)=D -1 (QK T )V,D=diag(QK T 1 L )

[0114] Among them, Q, K, V are the linear transformations of the input X, which can be expressed as:

[0115] Q=W q X,K=W k X,V=W v X

[0116] W Q, W K ,W V is the weight matrix of the model, which is a learnable parameter; 1 L is a vector of all ones of length L; diag(·) is a diagonal matrix with the input vector as its diagonal.

[0117] The attention matrix formula in Performer is as follows:

[0118]

[0119] Among them, Q' and K' are Q and K through the function The processed result is as follows:

[0120]

[0121] function is defined as:

[0122]

[0123] Here c is a positive number, ω is a random feature matrix, and m is the dimension of the matrix. Through this feature mapping and matrix decomposition, Performer reduces the complexity of attention calculation from the traditional quadratic complexity to linear complexity, making it possible to effectively process inputs of tens of thousands of genes. Figure 4As shown in the figure, in this model, 6 Performer encoder layers are used, each of which contains 10 attention heads. Each attention head independently calculates a set of attention weights and models the relationship between genes. Each encoder layer performs deep feature extraction on the input features through a multi-head attention mechanism and a feedforward neural network, and gradually learns the global representation of gene expression patterns and gene-gene relationships.

[0124] Gene embedding and expression embedding: BERT's embedding in text analysis is generally word embedding, the purpose is to understand the context. In order to enable BERT to identify cell types, gene expression information is used as BERT's token embedding. Since shuffling the input columns does not change their meaning, absolute positions are meaningless for genes. Instead, we use gene2vec to generate gene embeddings, which can be regarded as relative embeddings that capture the semantic similarity between any two genes.

[0125] Calculate gene embedding and expression embedding: First, use gene2vec to obtain the embedding vector generated by each gene, thereby representing the identity of each gene and treating it as the relative embedding of the model. Since BERT's token embedding is a discrete variable, and single-cell gene expression is a continuous variable, it needs to be converted into a discrete variable. The specific approach is to use the bag-of-words technology in the field of natural language processing (NLP) to bin the expression of the gene (which can be considered as the gene transcription frequency of each cell), and convert it into a 200-dimensional vector as the token embedding of the model.

[0126] S3: Model pre-training: Collect public single-cell RNA sequencing (scRNA-seq) databases and resources, large scRNA-seq databases (Human Cell Atlas, Tabula Muris, etc.) or public scRNA-seq data of various biological species, including humans and other model organisms (such as mice, zebrafish). Normalize the data to reduce the impact of sequencing depth, and perform high noise processing to remove low-quality cells and technical noise. Process the data according to the above-mentioned gene embedding and expression embedding methods, input the model for self-supervised learning, randomly mask part of the input, and use the remaining part to predict the masked content.

[0127] Masking strategy: Randomly select non-zero gene expression values ​​for masking (taking into account the dropout phenomenon of single-cell data); the masking ratio is usually 15%-20% to ensure that the model can both learn data features and make predictions.

[0128] The model predicts the original expression value of the masked gene expression based on the unmasked gene expression information. The objective function uses the cross entropy loss as the reconstruction loss function:

[0129]

[0130] Where M is the number of cells, N is the number of gene expression values ​​to be masked, and y i,j is the true expression value of gene j in cell i, p i,j is the predicted expression value in cell i.

[0131] This stage mainly enables the model to learn the global and local characteristics of gene expression patterns, capture the co-expression relationships between genes and complex biological patterns. This general pre-training model lays a solid foundation for downstream supervised learning tasks (such as cell type annotation), while greatly reducing the amount of label data and computational overhead required for fine-tuning.

[0132] S4: The core of fine-tuning is the supervised learning of the classification head, and the model is further optimized to adapt to the cell type annotation task through labeled data. In the fine-tuning stage, the pre-trained weights of scBERT are loaded and used as the initial model parameters. These pre-trained weights have learned common gene expression patterns and gene-gene interactions from large-scale unlabeled data through self-supervised learning. In order to prevent the model from overfitting on small-scale data, you can choose to freeze the first few layers of the encoder and only update the weights of the last few layers and the classification head. If the scale of labeled data is large, choose to update the parameters of the entire model.

[0133] First, check whether the single-cell dataset to be fine-tuned meets the input of the model, where the gene name should be the gene name of the NCBI gene database. Otherwise, perform homologous gene conversion on it. Taking mouse genes as an example, you can extract the corresponding human homologous genes in the mouse gene list by calling the Ensembl REST API, build a mapping dictionary, and replace the mouse gene name with the human gene name. Otherwise, perform homologous gene conversion on it. Taking mouse genes as an example, you can extract the corresponding human homologous genes in the mouse gene list by calling the Ensembl REST API, build a mapping dictionary, and replace the mouse gene name with the human gene name.

[0134] The data input here is consistent with the pre-training stage, both of which are gene expression matrices for each cell, but no masking is required. After the genes and expressions are embedded, the encoder will extract features, and the output is a 200-dimensional feature corresponding to each gene. These features are then convolved in one dimension to extract more abstract features. The convolution layer here can effectively aggregate feature interaction information between different genes. For the classification head, we use a three-layer fully connected network. Each layer uses nonlinear activation functions (such as ReLU) and Dropout to prevent overfitting, and maps the convolution layer output to the cell type probability distribution. The objective function used in supervised learning here is the cross entropy loss function, which is the prediction loss of the cell type label. The specific formula is as follows:

[0135]

[0136] Z i is the true cell type label of cell i, q i is the predicted label of cell i.

[0137] S5: Deconvolution operation: Deconvolution of spatial transcription data: Obtain spatial transcription data of the same tissue as the fine-tuned single-cell data above, preprocess it, delete cells with total gene expression below a certain threshold (such as less than 500 UMIs) and genes expressed in too few cells (such as genes with expression ratios below 1%), and filter outliers such as cells with extremely high expression levels (possibly double cells); then normalize and logarithmize, and standardize the gene expression value of each cell by the total expression (such as CPM or TPM) to reduce the impact of sequencing depth and alleviate the high dynamic range of expression. Use the fine-tuned model to perform the same operation on the spatial transcription data, and perform a softmax operation on the score matrix of each cell type in a single spot to obtain the proportion of various cell types in the spot, which we use as the final deconvolution result.

[0138] Tests are conducted on public datasets to evaluate the performance of the proposed method in deconvolution of spatially transcribed data.

[0139] In order to better demonstrate the effectiveness of the method of this application, this method was applied to fine-tune the single-cell data of the mouse brain in Cell2location, and verified using real data with labels. The prediction results include major cell type annotations and fine cell type annotations. The main cell types are divided into 8 types, such as excitatory neurons (Ext), oligodendrocytes (Oligo), endothelial cells (Endo), etc.; fine cell types are further precise subdivisions of the main cell types, such as Ext_L25, Ext_L56, Ext_Thal_1, Ext_L5_1, Ext_Thal_2, Ext_L23, etc. There are more than 50 subtypes, and the final evaluation is based on the AUC value. The data format here is as follows Figure 2 As shown in the figure. The fine-tuned model is applied to the data, and the cell type distribution matrix of each spot is output. It is compared with the ground truth, and the final AUC value is 0.95. Compared with other algorithms, even the Cell2location algorithm itself, this model has achieved good results, which verifies that the model can significantly improve the accuracy and robustness of cell type recognition by training on large-scale unlabeled single-cell data.

[0140] Example 3

[0141] This embodiment also provides a BERT-based cell type deconvolution system for implementing the method described in Embodiment 1 or Embodiment 2, such as Figure 5 As shown, including:

[0142] The data acquisition module acquires unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue;

[0143] A preprocessing module preprocesses the unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain the unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing;

[0144] The pre-training module inputs the pre-processed unlabeled single-cell data of the same tissue into the BERT model for pre-training to obtain a pre-trained BERT model;

[0145] The model training module inputs the pre-processed labeled single-cell data of the same tissue into the pre-trained BERT model for secondary training to obtain a trained BERT model;

[0146] The deconvolution module inputs the preprocessed spatial transcription data of the same tissue into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

[0147] The same or similar reference numerals correspond to the same or similar components;

[0148] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0149] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A cell type deconvolution method based on BERT, characterized in that: include: Obtain unlabeled single-cell data, labeled single-cell data, and spatial transcriptional data of the same tissue; Preprocessing unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing; Input the preprocessed unlabeled single-cell data of the same tissue into the constructed BERT model for pre-training to obtain a pre-trained BERT model; The pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model; The preprocessed spatial transcription data of the same tissue are input into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

2. The BERT-based cell type deconvolution method according to claim 1, characterized in that: Preprocessing the unlabeled single cell data, labeled single cell data, and spatial transcription data of the same tissue to obtain the unlabeled single cell data, labeled single cell data, and spatial transcription data of the same tissue after preprocessing, including: Clean, filter and reduce noise on unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue, remove problematic genes from unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue, and filter genes expressed in less than a preset number of cells; The unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue after cleaning, filtering and noise reduction are normalized to adjust the gene expression levels of the unlabeled single-cell data, labeled single-cell data and spatial transcription data of the same tissue to the same level.

3. The BERT-based cell type deconvolution method according to claim 1, characterized in that: The pre-processed unlabeled single-cell data of the same tissue is input into the constructed BERT model for pre-training to obtain a pre-trained BERT model, including: Constructing a BERT model, the model comprising an input embedding unit, an encoder unit, and a prediction unit connected in sequence; The unlabeled single-cell data of the same tissue after preprocessing are randomly masked to obtain the masked gene expression values; Set the reconstruction loss function, calculate the reconstruction loss function value according to the masked gene expression value and the unmasked gene expression value, and adjust the parameters of the BERT model; When the reconstruction loss function value reaches a preset value or the preset number of training times is reached, the pre-trained BERT model is obtained.

4. The BERT-based cell type deconvolution method according to claim 3, characterized in that: The input embedding unit includes a first expression embedding layer, a gene embedding layer, and a second expression embedding layer arranged in parallel; the encoder unit includes a first encoder subunit and a second encoder subunit arranged in parallel; The output end of the first expression embedding layer is connected to the input end of the first encoder subunit; the output end of the gene embedding layer is connected to the input ends of the first encoder subunit and the second encoder subunit; the output end of the second expression embedding layer is connected to the output end of the second encoder subunit; The prediction unit includes a reconstruction layer, a convolution layer and a fully connected classifier; the output end of the first encoder subunit is connected to the input end of the reconstruction layer; the output end of the second encoder subunit is connected to the input end of the convolution layer; the output end of the convolution layer is connected to the input end of the fully connected classifier.

5. The BERT-based cell type deconvolution method according to claim 4, characterized in that: The first encoder subunit comprises a first encoder layer, a second encoder layer, a third encoder layer, a fourth encoder layer, a fifth encoder layer and a sixth encoder layer connected in sequence; An output terminal of the sixth encoder layer is connected to an input terminal of the reconstruction layer; The second encoder subunit includes a seventh encoder layer, an eighth encoder layer, a ninth encoder layer, a tenth encoder layer, an eleventh encoder layer and a twelfth encoder layer connected in sequence; The output end of the twelve encoder layers is connected to the input end of the convolutional layer; The encoder layer includes a first self-attention layer, a second self-attention layer, a third self-attention layer, a fourth self-attention layer, a fifth self-attention layer, a sixth self-attention layer, a seventh self-attention layer, an eighth self-attention layer, a ninth self-attention layer, and a tenth self-attention layer connected in sequence.

6. The BERT-based cell type deconvolution method according to claim 3, characterized in that: The reconstruction loss is: Where M is the number of single cells; N is the number of masked gene expression values; y i,j is the true expression value of gene j in cell i, p i,j is the predicted expression value of gene j in cell i.

7. The BERT-based cell type deconvolution method according to claim 1, characterized in that: The pre-processed labeled single-cell data of the same tissue is input into the pre-trained BERT model for secondary training to obtain a trained BERT model, including: The single-cell data with labels of the same tissue after preprocessing are tested. If the gene name of the single-cell data is the gene name of the NCBI gene database, otherwise the single-cell data is converted to homologous genes and then input into the pre-trained BERT model; Input the labeled single-cell data of the same tissue after detection into the pre-trained BERT model to obtain the predicted cell type label of the single cell; Setting the prediction loss, calculating the prediction loss function value according to the predicted cell type label and the true cell type label of the single cell, and adjusting the parameters of the BERT model; When the prediction loss function value reaches a preset value or the preset number of training times is reached, the trained BERT model is obtained.

8. The BERT-based cell type deconvolution method according to claim 7, characterized in that: The prediction loss function is: Where M is the number of single cells; Z i is the true cell type label of single cell i, q i Predicted cell type label for single cell i.

9. The BERT-based cell type deconvolution method according to claim 1, characterized in that: The pre-processed spatial transcription data of the same tissue is input into the trained BERT model for cell type deconvolution to obtain the deconvolution results, including: The pre-processed spatial transcription data of the same tissue were input into the trained BERT model to obtain the score matrix of each cell type in each single cell; A softmax operation is performed on the score matrix of each cell type in each single cell to obtain the ratio of various cell types in each single cell, and the cell type ratio is used as the deconvolution result.

10. A BERT-based cell type deconvolution system, used to implement the method according to any one of claims 1 to 9, characterized in that: include: The data acquisition module acquires unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue; A preprocessing module preprocesses the unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue to obtain the unlabeled single-cell data, labeled single-cell data, and spatial transcription data of the same tissue after preprocessing; The pre-training module inputs the pre-processed unlabeled single-cell data of the same tissue into the BERT model for pre-training to obtain a pre-trained BERT model; The model training module inputs the pre-processed labeled single-cell data of the same tissue into the pre-trained BERT model for secondary training to obtain a trained BERT model; The deconvolution module inputs the preprocessed spatial transcription data of the same tissue into the trained BERT model for cell type deconvolution to obtain the deconvolution results.

Images