A joint analysis method of single-cell transcriptome data and text descriptions based on multimodal language models

Abstract

The present invention relates to the technical field of cell data analysis and discloses a joint analysis method of single-cell transcriptome data and text descriptions based on a multimodal language model. The method comprises the following steps: obtaining a single-cell RNA sequencing expression matrix and a corresponding cell text description; preprocessing the single-cell RNA sequencing expression matrix and the corresponding cell text description; constructing a multimodal dataset; and achieving a deep fusion of gene expression data and text knowledge by constructing a dual model and a cross-modal projection module, thereby avoiding the limitations of a single modality. During preprocessing, gene expression values ​​and index sequences are retained, the coarse coding method is changed, and the accuracy of cell type recognition is improved. Based on a pre-training strategy of contrastive learning, matching learning, and a cross-modal projection module, fine-grained cross-modal information interaction and sharing is achieved, and the effects of cross-modal tasks such as text generation into cells or cell generation into text are optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of cell data analysis, and in particular to a method for jointly analyzing single-cell transcriptome data and text descriptions based on a multimodal language model. Background Art

[0002] With the rapid development of single-cell RNA sequencing (scRNA-seq) technology, researchers can accurately measure the gene expression profile of each cell at single-cell resolution. This type of high-throughput data provides key information for revealing cellular functional heterogeneity, biological processes, and disease mechanisms. However, how to efficiently understand and interpret massive amounts of single-cell expression data remains a major challenge facing the field of bioinformatics.

[0003] To address this issue, the recent rise of single-modal pre-trained language models (PLMs) has achieved remarkable results in multiple areas, including text generation and understanding, image recognition, and gene sequence modeling. Although PLMs are currently being applied to single-cell analysis tasks, such as cell type annotation and pseudocell generation, they still have many limitations, as follows:

[0004] 1. Most current methods can only process single-modal data (such as text or expression matrices) and lack effective mechanisms to integrate textual knowledge with cellular expression data. This limits the model's deep understanding and generalization capabilities of cellular function, status, and classification.

[0005] 2. Currently, when trying to convert expression data into "cell sentence" text, only very little gene information is retained, resulting in the complete discarding of expression values. This crude encoding method leads to a significant decrease in the accuracy of cell type identification;

[0006] 3. Currently, most methods use simple strategies such as splicing and matching to connect text and cell data, which cannot achieve fine-grained information interaction and sharing, resulting in poor overall results in cross-modal generation tasks (such as text-to-cell or cell-to-text). Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention provides a method for jointly analyzing single-cell transcriptome data and text descriptions based on a multimodal language model to solve the above problems.

[0008] The above technical objectives of the present invention are achieved through the following technical solutions:

[0009] A method for jointly analyzing single-cell transcriptome data and text descriptions based on a multimodal language model, comprising:

[0010] S1: Obtain the single-cell RNA sequencing expression matrix and the corresponding cell text description, preprocess the single-cell RNA sequencing expression matrix and the corresponding cell text description, and construct a multimodal dataset;

[0011] S2: Based on the multimodal dataset, a dual-model and cross-modal projection module is constructed. The dual-model includes: a cell-based pre-trained language model and a text-based pre-trained language model.

[0012] S3: The dual models are pre-trained for alignment based on contrastive learning and matching learning, while the cross-modal projection module is pre-trained based on the cell description generation loss and pseudo-cell generation loss.

[0013] S4: In pre-training, different strategies are used for parameter adjustment at different stages, while text-based pseudo-cell generation and cell type automatic annotation tasks are implemented.

[0014] Furthermore, the single-cell RNA sequencing expression matrix and the corresponding cell text descriptions were preprocessed to construct a multimodal dataset, including:

[0015] The single-cell RNA sequencing expression matrix was analyzed to obtain the The original expression vector of each cell, and Construct the corresponding original gene index sequence;

[0016] Process the text description to get each cell The text token sequence;

[0017] The gene expression value of each cell was normalized and processed to obtain the The final expression vector for each cell;

[0018] The first The final expression vector of each cell is paired with the text token sequence, and a multimodal dataset is constructed, which is divided into a training set, a validation set, and a test set.

[0019] Furthermore, based on the multimodal dataset, a dual model is constructed, including:

[0020] According to the gene embedding matrix Map the gene index sequence of each cell to obtain the gene matrix;

[0021] Based on the gene matrix, a cell pre-training language model is constructed.

[0022] Furthermore, based on the multimodal dataset, a dual model is constructed, which also includes:

[0023] For the first Map the text token sequence of each cell to obtain the corresponding tokenID vector;

[0024] According to the word embedding matrix, the tokenID vector is converted into an embedding matrix, and a text pre-training language model is constructed.

[0025] Furthermore, based on the multimodal dataset, a cross-modal projection module is constructed, including:

[0026] Set K trainable query vectors and form a query vector matrix;

[0027] Based on the query vector matrix and the The hidden matrix of each cell output constructs a cross-modal projection module;

[0028] The cell pre-trained language model, text pre-trained language model and cross-modal projection module are integrated to obtain a unified multimodal framework.

[0029] Furthermore, the dual models are pre-trained for alignment based on contrastive learning and matching learning, including:

[0030] Based on the training set, for each sample , obtain the global representation of the cell through the cell pre-training language model;

[0031] Based on the training set, for each sample , obtain the global representation of the text through the text pre-training language model;

[0032] The contrastive learning loss is obtained by performing contrastive learning on the cell global representation and the text global representation.

[0033] Furthermore, the dual models are pre-trained for alignment based on contrastive learning and matching learning, which also includes:

[0034] For each positive pair in the training set batch, randomly select a sample as a negative sample pair;

[0035] Concatenate the cell global representation and the text global representation to obtain the matching probability;

[0036] Based on the negative sample pairs, positive and negative sample pairs are constructed, and the matching probability is combined to determine whether the cell and the text are paired, and the matching learning loss is obtained;

[0037] The contrastive learning loss and matching learning loss are calculated to obtain the total loss of the first stage.

[0038] Furthermore, the cross-modal projection module is pre-trained based on the cell description generation loss and pseudo-cell generation loss, including:

[0039] Analyze B paired samples based on the dual model to obtain a first hidden representation matrix and a second hidden representation matrix;

[0040] Analyze the first hidden representation matrix and generate the cell description generation loss;

[0041] Analyze the second hidden representation matrix to generate pseudo cell generation loss;

[0042] The cell description generation loss and pseudo cell generation loss are calculated to obtain the total loss of the second stage.

[0043] Furthermore, in pre-training, different strategies are used for parameter adjustment at different stages, including:

[0044] In the first stage, the parameters of the text pre-trained language model are frozen and the mapper of the cross-modal projection module is adjusted;

[0045] In the second stage, the projection parameters of the text pre-trained language model and the mapper of the cross-modal projection module are adjusted based on LoRA technology.

[0046] Furthermore, text-based pseudo-cell generation and cell type automatic annotation tasks are implemented, including:

[0047] The concatenated vectors of the cell pre-trained language model and the text pre-trained language model are input into the MLP classifier, and the predicted label of the cell type is obtained through the softmax function.

[0048] In summary, the present invention mainly has the following beneficial effects:

[0049] By constructing a multimodal dataset, the cell pre-trained language model and the text pre-trained language model are used to process the gene expression matrix and text description respectively, and the cross-modal information interaction is realized through the cross-modal projection module. Specifically, the dual models are aligned and pre-trained with the help of contrastive learning and matching learning, so that the cell representation and the text representation form a semantic association in the cross-modal space, avoiding the limitations of single modal information. This mechanism can simultaneously utilize the numerical features of gene expression and the semantic knowledge of text description to achieve a deep understanding of cell function, status and classification, significantly improving the model's generalization ability in tasks such as cell type annotation and pseudo-cell generation, and solving the problem of one-sided understanding caused by modal isolation in traditional methods.

[0050] By normalizing and transforming the single-cell RNA sequencing expression matrix in the preprocessing stage, the gene expression value of each cell is retained, and the original gene index sequence is constructed for each cell. This process retains both the numerical characteristics of gene expression (such as the level of expression) and the structural information of the gene (such as the index sequence), avoiding the loss of key information in traditional encoding methods. By pairing the normalized expression vector with the text token sequence, the model can simultaneously learn the quantitative information of gene expression and the qualitative knowledge of the text description, making the generated cell representation more comprehensive. In cell type recognition, it can avoid classification bias caused by missing information, significantly improve recognition accuracy, and solve the performance degradation problem caused by rough encoding methods.

[0051] Fine-grained information interaction is achieved through a cross-modal projection module and a bidirectional generation loss function. Specifically: a projection module is constructed using a trainable query vector matrix and a hidden matrix to capture the deep correlation between gene expression and text description. The cell description generation loss and the pseudo-cell generation loss are used to force the model to learn semantic consistency during cross-modal generation. In the pre-training stage, contrastive learning and matching learning are used to further align the global representations of cells and text, making cross-modal information interaction more complete. This breaks through the limitations of simple strategies and achieves a deep fusion of gene expression numerical features and text semantic features. In tasks such as text-to-cell generation or cell-to-text generation, it can generate results that are more in line with biological logic, thereby improving the effectiveness of cross-modal generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a step diagram of the method for jointly analyzing single-cell transcriptome data and text descriptions based on a multimodal language model of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] refer to Figure 1 , a method for jointly analyzing single-cell transcriptome data and text descriptions based on a multimodal language model, including:

[0055] S1: Obtain the single-cell RNA sequencing expression matrix and the corresponding cell text description, preprocess the single-cell RNA sequencing expression matrix and the corresponding cell text description, and construct a multimodal dataset;

[0056] S2: Based on the multimodal dataset, a dual-model and cross-modal projection module is constructed. The dual-model includes: a cell-based pre-trained language model and a text-based pre-trained language model.

[0057] S3: The dual models are pre-trained for alignment based on contrastive learning and matching learning, while the cross-modal projection module is pre-trained based on the cell description generation loss and pseudo-cell generation loss.

[0058] S4: In pre-training, different strategies are used for parameter adjustment at different stages, while text-based pseudo-cell generation and cell type automatic annotation tasks are implemented.

[0059] By constructing a multimodal dataset and integrating single-cell RNA sequencing expression matrices with text descriptions, we can break through the limitations of a single modality and enhance the model's deep understanding of cell function, status, and classification. At the same time, we can improve data processing methods to avoid the loss of key information such as gene expression values ​​during data conversion, and improve the accuracy of cell type identification. With the help of dual models and cross-modal projection modules, combined with training strategies such as comparative learning and matching learning, we can achieve fine-grained information interaction and sharing, significantly improving the effectiveness of cross-modal generation tasks.

[0060] In one case of this embodiment, the single-cell RNA sequencing expression matrix and the corresponding cell text description are preprocessed to construct a multimodal dataset, including:

[0061] The single-cell RNA sequencing expression matrix was analyzed to obtain the The original expression vector of each cell, and Construct the corresponding original gene index sequence, specifically including: downloading human tissue-derived scRNA from the public single-cell database (CELLxGENE) seq data snapshot, the original data is usually provided in sparse matrix or AnnData format, convert it into dense matrix form: cells, define their original expression vector, which is a column vector containing elements (M represents the total number of genes), each element is the The number of reads of genes in the raw count matrix, The original expression vector of each cell contains the first element to the Mth element in sequence; collect the gene names appearing in all cells, remove duplicates and construct a vocabulary so that its number is the same as Consistent, for each cell Construct the corresponding original gene index sequence, which is a vector containing several elements, where each element belongs to the constructed gene vocabulary. The gene index sequence of each cell can be unified using the same gene vocabulary, and the index order remains fixed according to the order in the vocabulary and does not change with the specific cell;

[0062] Process the text description to get each cell The text token sequence includes: Each cell in CELLxGENE is accompanied by certain annotation fields, such as "cell_type", "tissue_origin", etc. The above cell type strings are mapped to standardized CellOntology terms, and the same-name entries are searched on Wikipedia to obtain richer functional and structural descriptions. First, we extract the cell types annotated in the CELLxGENE platform. Then, we query the canonical definition of cell types in OBOFoundry. Then, we obtain the complete paragraph description of the same cell type from the Wikipedia API. We merge the OBOFoundry definition with the Wikipedia description to form a continuous text. For the entire concatenated text, we convert it to lowercase, remove HTML tags and special symbols, and use the same tokenizer as the TextPLM used to obtain the text token sequence for each cell i:

[0063] ;

[0064] Where, Represents the text token sequence of each cell i, where T is the fixed maximum sequence length. If the original text is too long, it is truncated to the first T tokens. If it is insufficient, a special filler is added at the end.

[0065] The gene expression value of each cell was normalized and processed to obtain the The final expression vector of each cell is obtained by performing logarithmic transformation on the normalized cells to reduce the numerical difference between high-expression genes and low-expression genes. The final expression vector of each cell is expressed as:

[0066] ;

[0067] Where, Indicates the The final expression vector for each cell;

[0068] The first The final expression vector of each cell is paired with the text token sequence, and a multimodal dataset is constructed. The multimodal dataset is divided into a training set, a validation set, and a test set. Specifically, the batch number and cell type label of each cell are recorded, and the cell sample is The corresponding final expression vectors, token sequences, batch numbers, and cell type labels were paired and combined to obtain a multimodal dataset, which was then divided into training set, validation set, and test set according to the cell type label at 80%, 10%, and 10%.

[0069] The data preprocessing strategy retains complete gene expression information and avoids the loss of key data. By constructing a unified gene vocabulary and a fixed index order, the original expression matrix is ​​converted into a structured expression vector. Combined with logarithmic transformation and normalization processing, it not only retains the numerical characteristics of gene expression but also reduces data noise, providing the model with richer and more accurate input, thereby greatly improving the accuracy of cell type identification. On the other hand, the construction of a multimodal dataset realizes the deep fusion of gene expression data and text knowledge. By pairing text descriptions with expression vectors, a complementary multimodal representation is formed. This fusion breaks through the limitations of single-modal analysis, enabling the model to combine the empirical characteristics of expression data to achieve a more comprehensive and in-depth understanding of cell function, status and classification, and enhance the generalization ability of the model.

[0070] By retaining the batch numbers and type labels of cells in the multimodal dataset, it not only supports the model learning the intrinsic correlation between modalities, but also effectively identifies and controls the experimental batch effects, improves the reliability of the analysis results, and at the same time, the proportional division of training sets, validation sets, and test sets ensures the adequacy of model training and the objectivity of evaluation. This fine-grained data processing and division method enables the subsequent cross-modal projection module to better capture the complex relationship between gene expression and text description, realize the deep sharing and interaction of the two modal information, and thus significantly improve the effect of cross-modal generation tasks.

[0071] In one case of this embodiment, a dual model is constructed based on a multimodal dataset, including:

[0072] According to the gene embedding matrix The gene index sequence of each cell is mapped to obtain the gene matrix, which specifically includes: The gene index sequence of each cell is retrieved from the gene embedding matrix by looking up the table, and the corresponding low-dimensional gene embedding vector is obtained. The final expression vector of each cell is linearly transformed and a bias term is added to obtain the expression value projection vector. The low-dimensional gene embedding vector of each gene and its corresponding expression value projection vector are added element by element to obtain the cell embedding vector of the fused gene. The cell embedding vectors of all genes in the cell are stacked vertically according to the order of genes in the original gene index sequence to construct a cell embedding matrix, which is the gene matrix.

[0073] According to the gene matrix, a cell pre-training language model is constructed, specifically including: the base of the cell pre-training language model adopts a generative pre-training model based on Transformer, including Layer Transformer encoder structure, each layer of encoder consists of multi-head self-attention and fully connected feedforward network, and stabilizes the training process through residual connection and layer normalization. In the calculation, each layer of encoder first maps the input gene matrix into query vector, key vector and value vector respectively, calculates the dependency between genes through the multi-head self-attention mechanism, generates attention output, and the output is residually connected and layer normalized with the input, and then processed by the feedforward network (including two layers of linear transformation and GELU activation function), and then residually connected and layer normalized again to obtain the output of the layer. After layer encoding, the final hidden representation matrix of the gene matrix is ​​obtained. If you want to obtain a global representation of a single cell, you can perform an average pooling operation on the matrix in the gene dimension, average the hidden vectors of all genes, and generate a cell vectorized representation. This representation integrates the expression information of all genes in the cell and the interaction relationship between genes, which can be used for downstream tasks such as cell type annotation and cross-modal alignment.

[0074] By fusing gene embedding with expression values, the gene index sequence is mapped into a low-dimensional embedding vector and added element-by-element to the expression value projection vector. This not only preserves the structural relationship between genes, but also integrates the numerical characteristics of gene expression, avoiding the problem of expression value loss in traditional methods and providing a more comprehensive and accurate cell representation for subsequent analysis. On the other hand, the cell pre-training language model adopts a multi-layer Transformer encoder structure, which captures the complex dependencies between genes through a multi-head self-attention mechanism. Combined with residual connections and layer normalization to stabilize the training process, it can effectively learn the collaborative patterns of gene expression in cells. This method breaks through the limitations of single-modal analysis and realizes deep modeling of gene expression data. The generated cell vector representation integrates gene expression information and the interaction between genes, providing more representative features for downstream tasks such as cell type annotation and cross-modal alignment, significantly improving the model's understanding of cell function, state and classification, and promoting the efficient interpretation and application of single-cell expression data.

[0075] In one case of this embodiment, building a dual model based on a multimodal dataset further includes:

[0076] For the first The text token sequence of each cell is mapped to obtain the corresponding tokenID vector. Specifically, for the text token sequence of the i-th cell, the index mapping is performed through the dedicated vocabulary of the Llama-2 pre-trained language model. The vocabulary contains the mapping rules of all unique words in the text. Each token will be converted into a corresponding numerical ID to form a tokenID vector, in which each element corresponds to a vocabulary index in the vocabulary.

[0077] According to the word embedding matrix, the tokenID vector is converted into an embedding matrix, and a text pre-training language model is constructed, specifically including: the word embedding matrix is ​​of dimension V d matrix, V is the size of the gene vocabulary, each row corresponds to the initial vector representation of a gene in the vocabulary, and the pre-trained scGPT is used for training. By finding the row vector corresponding to each tokenID in the word embedding matrix, the discrete ID is converted into a continuous semantic vector. The embedding vectors of all tokens are stacked vertically in the original sequence order to form a text embedding matrix with a dimension of T×d (d is the hidden layer dimension). This matrix integrates the semantic information of the text and serves as the input of the subsequent text pre-training language model, providing a basic representation for the text pre-training language model to understand the text semantics and conduct cross-modal interaction. In the model architecture design, TextPLM (text pre-training The base of the language model is Llama-2. You can choose the open source version with 7 billion parameters (70B) or less than 7 billion parameters (such as 7B) according to your needs. The model adopts a decoder-only generative Transformer architecture and has been pre-trained. Each layer of the Llama-2 network adopts an autoregressive (causal) multi-head self-attention mechanism. Its core feature is to ensure that the calculation of each token depends only on the context information on its left (that is, the sequence content before the token position) through a masking matrix (mask), thus conforming to the temporal logic of natural language generation. The text pre-training language model as a whole includes Layer network, the calculation process of each layer is as follows: the input text embedding matrix is ​​generated into query vector, key vector and value vector respectively through linear projection, and then the masked softmax function is applied to calculate the attention weight so that the current position can only focus on the semantic information of the left context. The calculated attention output is residually connected with the input and layer normalized, and then the features are further extracted through a feedforward network composed of two layers of linear transformation and GELU activation function. Finally, the output of this layer is obtained through residual connection and layer normalization. After layer encoding, the Llama-2 model A text input generates a hidden representation matrix of dimension T×d, which contains the deep semantic representation of each token in the text sequence and its context, which is used for subsequent cross-modal mapping and text generation tasks.

[0078] The semantic integrity of biological text is preserved through text mapping, and the text token sequence is converted into a numerical ID vector through the Llama-2 dedicated vocabulary. The text embedding matrix is ​​then generated in combination with the pre-trained scGPT. This method not only utilizes the semantic understanding ability of large-scale language models, but also integrates gene vocabulary information through the word embedding matrix to make the text representation more in line with the actual context. This process avoids the problem of information loss in traditional methods, provides the model with a richer and more accurate text semantic foundation, and significantly improves the ability to understand text information such as cell type descriptions and functional annotations.

[0079] By ensuring the temporal nature of semantic dependencies through a masking matrix, the model can capture contextual relationships in the text and generate deep semantic representations that conform to the logic of natural language processing. It can also effectively integrate hierarchical information in cell text descriptions, such as cell type definitions, functional descriptions, and tissue origins. The generated hidden representation matrix contains deep semantic information of the text sequence, providing high-quality text representations for subsequent cross-modal mapping tasks. Combined with the cell pre-trained language model, it achieves a deep fusion of gene expression data and textual knowledge, breaking through the limitations of single-modal analysis and significantly improving the model's understanding and generalization capabilities of cell functions, states, and classifications, facilitating cross-modal generation tasks.

[0080] In one case of this embodiment, a cross-modal projection module is constructed based on a multimodal dataset, including:

[0081] Set K trainable query vectors and construct a query vector matrix. Specifically, the cross-modal projection module includes a Q-Former for cell-to-text mapping and a Cross-Attention layer for text-to-cell mapping. The Q-Former serves as the cell-to-text mapping module, and the Cross-Attention layer serves as the text-to-cell mapping module, forming a query vector matrix of dimension K × d.

[0082] Based on the query vector matrix and the The hidden matrix of each cell output is used to construct a cross-modal projection module, which includes: Cell-to-Text mapping module: ... The cell hidden matrix is ​​used as the key vector and value vector of the cross-attention mechanism, and the query vector matrix is ​​used as the query vector. The cross-layer attention network is calculated layer by layer: each layer first generates a query vector, a key vector, and a value vector by linear projection of the input query matrix and the cell hidden matrix, and uses the softmax function to calculate the attention weight so that the query vector focuses on the key features in the cell representation. After the attention calculation, the output result is residually connected with the input and layer normalized, and then further features are extracted through the feedforward network. The above process is repeated until completion Layer calculation, the cross-attention module finally outputs a query embedding matrix with a dimension of K×d. This matrix is ​​used as the pre-semantic prefix of the text pre-trained language model. After being spliced ​​with the initial token embedding of the text input, it drives Llama-2 to perform autoregressive text generation, realizing the conversion from cell expression data to natural language description; Text-to-Cell mapping module: The text pre-trained language model is mapped to the first The hidden representation matrix generated by the text input generates the query vector, key vector and value vector required for the cross-attention mechanism through linear transformation. The linear projection process is implemented through a trainable weight matrix to adapt the text features to the computational requirements of the cell representation space. In order to match the input dimension (M) of the cell encoder, M trainable gene query vectors are defined to form a query matrix P with a dimension of M×d, which is used to extract conditional information related to gene expression from text semantics. Its initial value is generated by random initialization and adaptively optimized during the training process. An S-layer cross-attention network is used to map text to cell features: when calculating each layer, the query vector, key vector and value vector obtained by text projection are used as the input of the cross-attention, and the query matrix P is used as the query vector input of the current layer. The product attention mechanism calculates the interaction weight between text semantics and gene query vectors, so that the gene query vector focuses on the semantic information related to cell function in the text. After each attention calculation, residual connection and layer normalization are performed to stabilize the training, and then features are further extracted through the feedforward network. The above process is repeated until the S-layer calculation is completed, and finally a pseudo-gene embedding matrix with a dimension of M×d is output. This matrix incorporates text semantic information and can be used as a conditional input for the text pre-trained language model to generate cell expression values. The generated pseudo-gene embedding matrix is ​​fused with the original gene embedding matrix by element-wise addition or splicing to form a cell feature input containing text semantic conditions, driving the text pre-trained language model to generate a pseudo-cell expression spectrum that conforms to the text description, realizing the cross-modal generation task of "text→cell";

[0083] The cell pre-trained language model, text pre-trained language model and cross-modal projection module are integrated to obtain a unified multimodal framework, specifically including: the multimodal framework adopts a dual-tower parallel architecture, and the dual towers are divided into a left tower and a right tower. Among them, the left tower represents the cell pre-trained language model, which is responsible for processing cell expression vectors, extracting cell features through gene embedding and Transformer encoder, and outputting gene-level hidden representation matrix and cell-level global representation vector. The hidden representation matrix retains the interaction information between genes, and the global representation is the overall semantic vector of the cell; the right tower represents the text pre-trained language model, receives text token sequences, uses pre-trained decoding Transformer to extract text semantic features, and outputs token-level hidden representation matrix and text global representation vector. The latter integrates the overall semantic information of the text sequence; during parallel computing, the dual towers independently maintain their respective Transformer layers and parameters to ensure that the feature extraction processes of the cell modality and the text modality do not interfere with each other, while providing basic representation for cross-modal mapping;

[0084] In the Cell-to-Text description generation process, the input cell expression vector is encoded by CellPLM to generate a gene-level hidden matrix. The Q-Former module performs cross-attention calculations on the gene-level hidden matrix to extract the K-dimensional query embedding (which can be regarded as the text mapping feature of the cell semantics). The query embedding is then concatenated with the initial token embedding of the actual text description to form a text input matrix that integrates the cell features. This matrix is ​​then fed into the Llama-2 decoder. Llama-2 uses an autoregressive mechanism to predict the subsequent text token by token, ultimately generating a natural language description that matches the cell expression.

[0085] In pseudo-cell generation (Text-to-Cell), the calculation process is as follows: after the input text token sequence is encoded by TextPLM, a text hidden representation matrix is ​​generated. Through the Text-to-Cell mapping module, the text hidden representation and the M-dimensional trainable gene query vector are cross-attentionally calculated to generate virtual gene embeddings (cell feature mapping that integrates text semantics). The virtual gene embeddings are merged with the original gene expression embeddings to form a cell feature input with text conditions, which is then encoded by CellPLM to output a pseudo-cell expression vector that conforms to the text description.

[0086] The bidirectional mapping mechanism of the cross-modal projection module breaks through the limitations of traditional methods. The Cell-to-Text mapping module converts cell expression data into text semantic prefixes through Q-Former, driving Llama-2 to generate natural language descriptions, and fully retaining the association between gene expression information and cell function. The Text-to-Cell mapping module converts text semantics into a pseudo-gene embedding matrix through a cross-attention network, generating a pseudo-cell expression spectrum that conforms to the text description. This bidirectional conversion mechanism realizes in-depth interaction and analysis of data from two modalities.

[0087] By combining a dual-tower parallel architecture with cross-modal projection, an efficient unified multimodal framework was constructed. The independent feature extraction of the dual towers ensures the professionalism of the cell and text modalities, while the cross-modal projection module achieves semantic alignment of the two modalities by sharing the query vector matrix and gene query vector, avoiding the information loss of traditional splicing strategies and adaptively optimizing cross-modal mapping through trainable parameters, significantly improving the accuracy of cross-modal generation tasks. In cell description generation, the model can generate detailed cell function descriptions based on expression data, and in pseudo-cell generation, it can accurately reconstruct gene expression profiles based on text descriptions, ensuring fine-grained information interaction and sharing.

[0088] In one case of this embodiment, alignment pre-training of the dual models is performed based on contrastive learning and matching learning, including:

[0089] Based on the training set, for each sample , the global representation of the cell is obtained by pre-training the language model of the cell, specifically including: randomly extracting the cell containing Small batches of cell samples, each sample corresponds to a set of paired cell expression data and text descriptions, for each sample , its cell expression vector is input into the cell pre-trained language model. After being processed by the Transformer encoder layer inside the model, the hidden representation of the gene dimension is pooled to generate a cell global representation vector of dimension d, which integrates the overall expression characteristics of the cell;

[0090] Based on the training set, for each sample , the global representation of the text is obtained by pre-training the language model, specifically including: randomly extracting the text containing Small batches of cell samples, each sample corresponds to a set of paired cell expression data and text descriptions, for each sample , the corresponding text token sequence is input into the text pre-trained language model, the text semantics is encoded through the decoding Transformer layer, and then a pooling operation is performed to generate a text global representation vector of dimension d, which summarizes the overall semantic information of the text;

[0091] Perform contrastive learning on the cell global representation and the text global representation to obtain contrastive learning loss, specifically including: making the paired normalized cell global representation Global representation of text In the same batch, they attract each other, but repel each other with the representations of other samples; for any two samples and The representation vector of , calculates its similarity, and the calculation formula is as follows:

[0092] ;

[0093] Where, Indicates the Cell characterization of the first sample and the The similarity between the text representations of samples is used to measure the vector angle between the two modal semantic spaces. The larger the value, the closer the semantics. Represents the normalized The global representation of the text of samples, Mathematical symbol for cosine similarity;

[0094] For all positive pairs in a batch and negative pairs Calculate the contrast loss and set the temperature coefficient to , the calculation formula is as follows:

[0095] ;

[0096] Where, represents the cell-context contrastive learning loss, represents the similarity score of the positive sample pair (i.e., paired cell and text), represents the similarity score between text sample j and cell sample i, represents the exponential function, Represents a logarithmic function.

[0097] Through dual-model alignment pre-training based on contrastive learning and matching learning, the cell and text pre-training language models are first used respectively to accurately extract the global representation of cells and the global representation of text, retaining multi-dimensional information. In contrastive learning, paired representations are made to attract each other and unpaired representations are made to repel each other. With the help of cosine similarity and contrast loss optimization, cross-modal deep alignment is achieved, which can effectively integrate text knowledge and cell expression data, avoid the loss of coarse information encoding, enable the interactive sharing of fine-grained information, improve the understanding and generalization ability of cell functions, and provide a new path to solve the problem of efficient understanding of single-cell expression data, so as to achieve the effect of optimizing cell analysis tasks.

[0098] In one case of this embodiment, performing alignment pre-training on the dual models based on contrastive learning and matching learning further includes:

[0099] For each positive pair in the training set batch, randomly select a sample as a negative sample pair, specifically including: for each positive pair in the training set batch , randomly selected from other Extract one from the text , construct a negative pair ;

[0100] The cell global representation and the text global representation are spliced ​​to obtain the matching probability, specifically including: Global representation of text Splice and input into a matching classifier to get the matching probability;

[0101] Based on the negative sample pairs, construct positive and negative sample pairs, and combine the matching probability to determine whether the cell and the text are paired, and obtain the matching learning loss, which specifically includes: setting the positive label to , the negative label is , the calculation formula is as follows:

[0102] ;

[0103] Where, Represents cells Text matching learning loss, which is used to calculate the difference between the model prediction and the true label, represents the normalized similarity function, Represents the normalized text feature representation of the sample that forms a negative pair with the i-th sample. For each , randomly sample one As negative sample index;

[0104] The contrastive learning loss and matching learning loss are calculated to obtain the total loss of the first stage, which specifically includes: weighting contrastive learning and matching learning to obtain the total loss of stage one , the calculation formula is as follows:

[0105]

[0106] Where, is a hyperparameter, , used to balance the contrast loss and matching loss.

[0107] Through dual-model alignment pre-training, a deep cross-modal fusion of cell expression data and textual knowledge is achieved. Compared with single-modal processing methods, it uses contrastive learning to make paired cells and text representations attract each other, which can effectively integrate gene expression features and text semantic information, avoiding the limitations of cell function understanding caused by single-modal analysis. At the same time, the expression value information of the gene dimension is retained in the process of generating the global representation of the cell, which changes the traditional rough method of discarding expression values ​​during encoding, so that the multi-dimensional features required for cell type identification can be retained, and the integrity and accuracy of data encoding are improved.

[0108] Through the joint optimization of contrastive learning and matching learning, the limitations of simple splicing strategies have been broken through, and cross-modal fine-grained information interaction has been achieved. The semantic space constraints of positive and negative samples in contrastive learning, combined with the probabilistic judgment of the matching classifier on the representation after splicing, can enable the semantic association between cells and text to be accurately mapped in the vector space, promote the improvement of cross-modal generation tasks (such as text generating cells or cells generating text), enhance the model's ability to generalize and understand cell status and function, and provide a new path for the efficient analysis of massive single-cell data, effectively optimizing the performance of analysis tasks such as cell type annotation and pseudo-cell generation.

[0109] In one case of this embodiment, the cross-modal projection module is pre-trained based on the cell description generation loss and the pseudo-cell generation loss to improve the bidirectional generation capability, including:

[0110] Analyzing B paired samples based on the dual model to obtain a first hidden representation matrix and a second hidden representation matrix, specifically comprising: inputting the cell data in each paired sample into the cell pre-trained language model to obtain a first hidden representation matrix that can reflect cell characteristics; inputting the text data in each paired sample into the text pre-trained language model to obtain a second hidden representation matrix that reflects text characteristics;

[0111] The first hidden representation matrix is ​​analyzed to generate the cell description generation loss, specifically including: using the first cell hidden representation matrix as the key vector and value vector of the cross-attention mechanism, and using K trainable query vectors as query vectors. Through cross-attention calculation, the cell semantic features related to text generation are extracted to generate a query embedding matrix of dimension K×d. The query embedding matrix is ​​concatenated with the initial token embedding matrix of the real text description to form a decoder input matrix that fused the cell features. The decoder input matrix is ​​sent to the Llama-2 decoding Transformer, and the subsequent text is predicted token by token through the autoregressive mechanism. For each position in the real text token sequence, the cell pre-trained language model calculates the probability of generating the corresponding token. The cross-entropy loss function is used to measure the difference between the probability distribution of the token generated by the cell pre-trained language model and the real text sequence. By minimizing this loss, the parameters of the cross-modal mapping module and the text pre-trained language model are optimized. The average cross-entropy loss is calculated for all token positions of B samples in the batch to generate the cell description generation loss.

[0112] The second hidden representation matrix is ​​analyzed to generate pseudo-cell generation loss, specifically including: using the second hidden representation matrix as the key vector and value vector of the cross-attention mechanism, and using M trainable gene query vectors as query vectors, and mapping the text semantic features into a virtual gene embedding matrix with a dimension of M×d through multi-layer cross-attention calculation of the Text-to-Cell mapping module, and fusing the original gene embedding matrix and the virtual gene embedding matrix by element-wise addition, and then converting them into the initial feature matrix with additional text conditions through linear mapping. This operation integrates the text semantic constraints with the original gene expression information to form conditional features that meet the input requirements of the cell pre-training language model, ensuring that the generated pseudo-cell expression spectrum is consistent with the semantics of the text description, and inputting the fused initial feature matrix into the first layer of the cell pre-training language model. The model performs Transformer encoder processing (each layer includes a self-attention mechanism and a feedforward network) to extract the regulatory interaction features between genes and generate the final hidden matrix. The matrix is ​​linearly mapped and nonlinearly activated to convert the abstract features into numerical expression values. The reconstructed log-normalized expression vector is output, which is the pseudo-cell expression spectrum generated by the model based on the text description. For the M gene expression values ​​of each sample in the batch, the sum of the squares of the differences between the generated values ​​and the true values ​​is calculated and then averaged. This average is the pseudo-cell generation loss.

[0113] The cell description generation loss and pseudo cell generation loss are calculated to obtain the total loss of the second stage, which specifically includes: weighted summing the cell description generation loss and pseudo cell generation loss to obtain the total loss of stage 2 , the calculation formula is as follows:

[0114] ;

[0115] Where, and Represents the corresponding cell description generation loss and loss of pseudocell generation The hyperparameters of , .

[0116] Through the joint optimization of cell description generation loss and pseudo-cell generation loss, the bidirectional generation capability of the cross-modal projection module is significantly improved. In the cell-to-text direction, the cross-attention mechanism is used to extract cell semantic features, and combined with the Llama-2 decoding Transformer to achieve token-by-token text generation, it can accurately retain the multi-dimensional information in the cell expression data, avoiding the problem of discarding expression values ​​when converting expression data into text by traditional methods. In the text-to-cell direction, text semantics are mapped to the gene expression space through multi-layer cross-attention calculations. The generated pseudo-cell expression spectrum is highly consistent with the true expression value, ensuring the effective integration of text semantic constraints and gene expression information. This bidirectional generation capability enables the model to achieve accurate conversion between the two modalities of cells and text.

[0117] Through pre-training of the cross-modal projection module, the limitations of traditional splicing and matching strategies have been broken through, and fine-grained information interaction between cells and text data has been achieved. In the process of cell description generation, the cross-attention mechanism can selectively focus on cell features related to text generation, improving the accuracy of text generation. In the process of pseudo-cell generation, multi-layer cross-attention calculations can capture the regulatory interaction relationship between genes, making the generated expression spectrum more consistent with biological laws.

[0118] In one case of this embodiment, different strategies are used for parameter adjustment at different stages in pre-training, including:

[0119] In phase one, the parameters of the text pre-trained language model are frozen, and the mapper of the cross-modal projection module is adjusted. Specifically, the pre-trained scGPT weights are used in the cell pre-trained language model, and all parameters can be frozen or some layers can be fine-tuned. The text pre-trained language model is initialized to the original Llama-2 weights, all of which are frozen and not involved in phase one training to maintain the text pre-training capability. During initialization, all cross-attention and FFN parameters in the cell-to-text mapping module are loaded from BioMedBERT and all are fine-tuned to adapt to the cell characteristics. The parameters of the text-to-cell mapping module are randomly initialized, frozen, and not updated.

[0120] In the second stage, the projection parameters of the text pre-trained language model and the mapper of the cross-modal projection module are adjusted based on LoRA technology. Specifically, all parameters in the cell pre-trained language model can be frozen or allowed to be fine-tuned according to experimental requirements. If you want to retain the cell representation ability learned after the first stage of alignment, you can choose to freeze it, otherwise you can make a small amount of fine-tuning; based on LoRA technology, most of the original parameters in the text pre-trained language model are frozen, and only the relevant parameters of the training are retained; fine-tuning is continued in the Cell-to-Text mapping module to maintain coordination with the TextPLM decoder; all fine-tuning begins in this stage in the Text-to-Cell mapping module to learn the mapping relationship from text to cell features.

[0121] Through a phased parameter adjustment strategy, efficient optimization and performance improvement of cross-modal models are achieved. In the first phase, the parameters of the text pre-trained language model are frozen to avoid forgetting the pre-trained knowledge. At the same time, the Cell-to-Text mapping module is fine-tuned to adapt it to the cell feature representation, ensuring the initial fusion of cell expression data and text knowledge, and solving the limitations of single modality processing. The cell pre-trained model can be selectively frozen or fine-tuned, which provides flexible optimization space for cell feature extraction while retaining knowledge, avoiding the loss of genetic information caused by traditional encoding methods.

[0122] Through lightweight parameter optimization and fine-grained interaction enhancement, in the second stage, most parameters of the text model are frozen with the help of LoRA technology, and only the projection-related parameters are fine-tuned. While reducing the computational cost, fine-grained interaction between text semantics and cellular features is achieved. The text-to-cell mapping module is fully fine-tuned in this stage. By learning the mapping relationship from text to cellular features, it breaks through the limitations of simple splicing strategies and promotes the deep alignment of gene expression values ​​and text semantics in cross-modal generation tasks.

[0123] In one aspect of this embodiment, text-based pseudo-cell generation and cell type automatic annotation tasks are implemented, including:

[0124] The concatenated vector of the cell pre-trained language model and the text pre-trained language model is input into the MLP classifier, and the predicted label of the cell type is obtained through the softmax function. Specifically, The original expression vector of each cell is input into the cell pre-trained language model to obtain a global embedding , and query embedding is obtained through Q-Former , and then extract text features through the text pre-training language model If only cell expression information is used for annotation, it can be directly used ; If combined with text knowledge, and Spliced ​​into ;

[0125] Add a multi-layer perceptron (MLP) classifier on top of the global embedding. The specific calculation formula is as follows:

[0126] ;

[0127] Where, Represents the multi-layer perceptron classifier for the The output vector of cells, MLP represents a multi-layer perceptron (fully connected neural network), represents the symbol for the set of real numbers, Can be selected as or ;

[0128] ;

[0129] ;

[0130] Where, Represents the probability distribution vector of the i-th cell belonging to each type, with a value range between 0 and 1. represents the activation function, Indicates that the i-th cell is predicted to be The probability of the class, C represents the total number of cell types, represents the index of C, Indicates that the multi-layer perceptron (MLP) classifier processes the When there are cells, the output vector corresponds to The original output value of the class;

[0131] The formula for calculating the predicted type label of a cell is as follows:

[0132] ;

[0133] Where, represents the predicted type label of the cell, Indicates The category corresponding to the maximum value.

[0134] By inputting the concatenated vectors of the cell pre-trained language model and the text pre-trained language model into the MLP classifier, the limitations of single-modal data processing are broken, the fusion of text knowledge and cell expression data is achieved, the model's understanding and generalization capabilities of cell functions, etc. are enhanced, and the rough encoding method of converting expression data into "cell sentence" text is changed, avoiding the loss of genetic information and expression values, and improving the accuracy of cell type identification; breaking through the simple text and cell data connection strategy, achieving fine-grained information interactive sharing, optimizing the cross-modal generation task effect, and thus being able to efficiently process massive single-cell expression data.

[0135] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for joint analysis of single-cell transcriptome data and text descriptions based on a multimodal language model, characterized in that: include: S1: Obtain the single-cell RNA sequencing expression matrix and the corresponding cell text description, preprocess the single-cell RNA sequencing expression matrix and the corresponding cell text description, and construct a multimodal dataset, including: Analyze the single-cell RNA sequencing expression matrix to obtain the original expression vector of the i-th cell, and construct the corresponding original gene index sequence for each cell i; Process the text description to obtain the text token sequence of each cell i; Normalize and change the gene expression value of each cell to obtain the final expression vector of the i-th cell; Pair the final expression vector of the i-th cell with the text token sequence, construct a multimodal dataset, and divide the multimodal dataset into training set, validation set, and test set; S2: Based on multimodal datasets, a dual model and cross-modal projection module are constructed. The dual model includes: a cell pre-trained language model and a text pre-trained language model. Based on multimodal datasets, a dual model is constructed, including: Map the gene index sequence of the i-th cell according to the gene embedding matrix to obtain the gene matrix; Based on the gene matrix, a cell pre-training language model is constructed; Map the text token sequence of the i-th cell to obtain the corresponding tokenID vector; Convert the tokenID vector into an embedding matrix based on the word embedding matrix and build a text pre-training language model; Based on the multimodal dataset, a cross-modal projection module is constructed, including: Set K trainable query vectors and form a query vector matrix; Construct a cross-modal projection module based on the query vector matrix and the hidden matrix output by the i-th cell; Integrate the cell-based pre-trained language model, text-based pre-trained language model, and cross-modal projection module to obtain a unified multimodal framework. S3: Alignment pre-training of the dual models is performed based on contrastive learning and matching learning. At the same time, the cross-modal projection module is pre-trained based on the cell description generation loss and pseudo-cell generation loss, including: Analyze B paired samples based on the dual model to obtain a first hidden representation matrix and a second hidden representation matrix; Analyze the first hidden representation matrix and generate the cell description generation loss; Analyze the second hidden representation matrix to generate pseudo cell generation loss; Calculate the cell description generation loss and pseudo cell generation loss to get the total loss of the second stage; S4: In pre-training, different strategies are used for parameter adjustment at different stages, including: In the first stage, the parameters of the text pre-trained language model are frozen and the mapper of the cross-modal projection module is adjusted; In the second phase, the projection parameters of the text pre-trained language model and the mapper of the cross-modal projection module are adjusted based on LoRA technology; Simultaneously, text-based pseudo-cell generation and cell type automatic annotation tasks are implemented.

2. The method for joint analysis of single-cell transcriptome data and text description based on a multimodal language model according to claim 1, characterized in that: Alignment pre-training of the two models is performed based on contrastive learning and matching learning, including: Based on the training set, for each sample i, the cell global representation is obtained through the cell pre-training language model; Based on the training set, for each sample i, a global representation of the text is obtained through the text pre-trained language model; The contrastive learning loss is obtained by performing contrastive learning on the cell global representation and the text global representation.

3. The method for joint analysis of single-cell transcriptome data and text description based on a multimodal language model according to claim 2, characterized in that: Alignment pre-training of the two models based on contrastive learning and matching learning also includes: For each positive pair in the training set batch, randomly select a sample as a negative sample pair; Concatenate the cell global representation and the text global representation to obtain the matching probability; Based on the negative sample pairs, positive and negative sample pairs are constructed, and the matching probability is combined to determine whether the cell and the text are paired, and the matching learning loss is obtained; The contrastive learning loss and matching learning loss are calculated to obtain the total loss of the first stage.

4. The method for joint analysis of single-cell transcriptome data and text description based on a multimodal language model according to claim 1, characterized in that: Text-based pseudocell generation and cell type automatic annotation tasks include: The concatenated vectors of the cell pre-trained language model and the text pre-trained language model are input into the MLP classifier, and the predicted label of the cell type is obtained through the softmax function.

Images