Transformer-based prediction method for CRISPR / Cas9 single guide RNA targeting activity

Abstract

The present invention discloses a method for predicting CRISPR / Cas9 single-guide RNA targeting activity based on Transformer, and belongs to the technical field of gene editing. The Transformer of the present invention uses a self-attention mechanism to learn global architecture information in a sequence, and calculates the relationship between each base and all other bases. The process can be captured through the interaction between the bases, and the same sequence feature is mapped to multiple subspaces to obtain multiple vector space representations, which is convenient for the fusion and learning of context data when calculating attention. In addition, the present invention uses a multi-head mechanism to learn the expression of the sequence vector expression in different subspaces, thereby ensuring the efficiency and accuracy of calculation and learning. In addition, in order to avoid gradient disappearance or information loss caused by a deep network, a dynamic residual structure adds the output of each module, which can extract feature information faster and enhance the model expression ability, thereby facilitating the effective solution of the problem of long sequence remote feature capture while ensuring parallel calculation.

Description

Technical Field

[0001] The present invention relates to the field of gene editing technology, and in particular to a method for predicting the targeting activity of CRISPR / Cas9 single guide RNA based on Transformer. Background Art

[0002] The CRISPR / Cas9 system derived from Streptococcus pyogenes has been widely used in the field of genome editing. The Cas9 nuclease can be guided by sgRNA and recognize the protospacer adjacent motif (PAM) sequence and introduce site-specific DNA double-strand breaks in the target region, thereby achieving site-specific editing of the genome. The success of the CRISPR / Cas9 system in prokaryotic host genetic engineering depends largely on the activity of sgRNA. However, the difference in activity between different sgRNAs still has significant limitations, resulting in unstable targeting efficiency. Therefore, maximizing the activity of the target site, that is, accurately predicting the targeting activity of sgRNA, will help improve the safety of experiments based on the CRISPR system.

[0003] In order to determine the effectiveness of sgRNA target identification and prediction, previous work has designed various computational model tools; the existing tools can be roughly divided into two categories: one is a scoring-based method, which usually scores the contribution of sgRNA activity according to specific genomic environmental factors, or conducts a thorough search of the target sequence by locating the PAM sequence. However, as the size of the data set continues to increase, this rule-based sorting method will take up too much computing resources and it is difficult to improve the prediction accuracy; the other is a learning-based method, which trains the model and predicts the cutting efficiency of sgRNA by integrating different features that affect the targeted knockout effect of sgRNA. In addition, when the sgRNA in the data set When the number of gRNAs reaches a certain level, models based on deep learning are usually superior to traditional machine learning. At present, there are already some deep learning model design schemes to predict sgRNA activity. For example, Chinese Patent No.: CN111613274A discloses a CRISPR / Cas9sgRNA activity prediction method based on deep learning, which combines the CNN mechanism with the attention mechanism to construct an interpretable model that can capture the intrinsic characteristics of Cas9-sgRNA binding and cutting. However, due to the limitations of CNN and attention mechanisms, it still cannot capture contextual sequence information in both directions and cannot solve the problem of long-distance feature calculation.

[0004] Therefore, a new prediction method is needed to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to solve the defects in the prior art and propose a Transformer-based CRISPR / Cas9 single guide RNA targeting activity prediction method.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The Transformer-based CRISPR / Cas9 single guide RNA targeting activity prediction method includes the following steps:

[0008] (1) Obtaining a public data set: Obtaining a public data set containing sgRNA sequences and indel frequencies; the data set includes Sniper-Cas9, SpCas9-NG, xCas9, and HypaCas9;

[0009] (2) sgRNA sequence encoding: preprocessing the public dataset, using complementary bases to represent the original sgRNA sequences in the public dataset, and encoding the sgRNA sequences in the public dataset using a specific encoding method to generate sample sets for different stages of the model, and dividing them into training sets and test sets with preset ratios;

[0010] (3) Constructing a prediction model: constructing a Transformer model based on a dynamic residual structure, inserting the training set as input data into the model for training, and outputting a trained prediction model according to preset indicators;

[0011] (4) Model application: The sgRNA sequence to be tested is input into the trained prediction model to predict the indel frequency.

[0012] Furthermore, the sgRNA sequence in step (1) consists of 23 bases, including a 20-base pre-spacer sequence and a 3-base PAM sequence.

[0013] Furthermore, the preprocessing in step (2) is used to normalize the indel frequency, limit it to the range of (0, 1), and use it as a label for the corresponding sequence.

[0014] Furthermore, the specific process of sgRNA sequence encoding in step (2) is as follows:

[0015] (2.1) The sgRNA sequence is regarded as text, and a dictionary is constructed using the Tokenizer method, which includes the mapping of four different bases A, T, C, and G to their corresponding numerical values, forming {A:1, T:2, C:3, G:4};

[0016] (2.2) adding a flag bit SOS at the beginning of each sgRNA, and expanding the dictionary to add the element {SOS:0}; converting the single base in the sgRNA sequence into a corresponding positive integer according to the dictionary to realize the vectorization of the sgRNA sequence;

[0017] (2.3) Finally, the vectorized sgRNA sequences were used as sample sets and shuffled, and then divided into training set and test set in a ratio of 85%:15%.

[0018] Furthermore, the specific process of constructing the prediction model is as follows:

[0019] (3.1) inputting the training set, i.e., the vectorized sgRNA sequence, into the Transformer layer to capture the sgRNA sequence features, and obtaining the output of the Transformer part;

[0020] (3.2) Flattening the output of the Transformer part to obtain a two-dimensional vector of n*3072, that is, generating a flat;

[0021] (3.3) Input the flat layer into a fully connected layer with a parameter dimension of 64, use ReLU as the activation function, and name the layer dense1;

[0022] (3.4) Input the dense1 into the Dropout layer with a parameter of 0.4 and name it dropout1;

[0023] (3.5) Input the dropout1 into a fully connected layer with a parameter dimension of 32, use ReLU as the activation function, and name the layer dense2;

[0024] (3.6) The dense2 output is input again into the Dropout layer with a parameter of 0.4 and named dropout2;

[0025] (3.7)Finally, the output of the dropout2 layer is input to the output layer with a parameter of 1 to obtain the corresponding predicted indel frequency and perform the loss function calculation.

[0026] Furthermore, the capturing of the sgRNA sequence features includes:

[0027] (3.1a) The training set, i.e., the vectorized sgRNA sequence, is input into the Embedding layer. Each base in the sgRNA sequence is converted into a corresponding vector representation. The length of this vector is determined by the parameter embedding-size, and each dimension represents the attribute of the base.

[0028] (3.1b) The serial numbers 1-23 are used to represent the position data of the bases in the sgRNA. The position data is also vectorized by the Embedding method. The position vector is added to the base vector to obtain the final representation vector. The formula is as follows:

[0029] W=(ω1,ω2,…,ω m )

[0030] p=(p1,p2,…,p m )

[0031] e=(ω1+p1,ω2+p2,…,ω m +p m )

[0032] Where: W is the base vector; p is the position vector; e is the final representation vector;

[0033] (3.1c) The final representation vector is input into the encoder and decoder respectively, and the multi-head attention module in the encoder and decoder is used for attention representation and concatenation to obtain the self-attention matrix in multiple sub-expression spaces. The multi-head attention module is obtained by concatenating 8 scaled dot product attentions, and its calculation formula is as follows:

[0034]

[0035] Where: Q, K, V represent matrices, and they are base vectors combined with position information through the learnable weight matrix W Q ,W k ,W v Multiply by; d k is the dimension of matrix K;

[0036] (3.1d) The self-attention matrices in the multiple sub-expression spaces are input into a feedforward neural network, which is composed of multiple layers of fully connected layers and uses ReLU as the activation function. The calculation formula is as follows:

[0037] FF=ReLU(xW+b)

[0038] Where: x is the output of the previous layer, W and b are the weight and bias of the fully connected layer respectively;

[0039] (3.1e) Residual connections and layer normalization are used for each multi-head attention module and feedforward neural network, and the distribution of base vectors is adjusted to avoid gradient vanishing / explosion or covariate shift problems. The calculation formula is as follows:

[0040] y=LN(F(x)+x)

[0041] Where: F(·) function is the multi-head attention function, LN is the layer normalization function, and the specific calculation method is as follows:

[0042]

[0043] Where: μ and δ are the sample mean and standard deviation respectively, γ and β represent the trainable parameter vectors of scaling and translation respectively;

[0044] (3.1f) An Encoder-Decoder Attention module is added between the multi-head attention module and the feedforward neural network module in the Decoder part. At the same time, the multi-head attention module uses the base vector with position information added as the input of the Encoder part, and linearly transforms the original base vector:

[0045]

[0046] Where: x is the original input sequence, d k is the dimension of matrix K;

[0047] (3.1g) The Encoder-Decoder Attention module uses the output of the Encoder and the output of the previous multi-head attention module in the Decoder as the input matrix for self-attention calculation, and introduces a dynamic residual structure in the Decoder module. The output of each neural module is accumulated and constrained using the weight of the module as the final output of the Decoder part. The output of each neural module can be expressed as:

[0048] out=LN(F(y)+y)

[0049] Where F(·) is the feedforward neural network function, LN is the layer normalization function, y is the output of the multi-head attention module, and combined with the normalization calculation for the multi-head attention module in step (3.1e), the output out is derived with respect to the input x:

[0050]

[0051] in and The calculation formulas are:

[0052]

[0053]

[0054] It can be seen that in the process of back propagation calculation of the neural network to update the weight matrix, there is a cumulative multiplication of multiplication factors, so there is still a possibility of gradient vanishing; in order to solve this problem, the present invention introduces a dynamic residual structure in the Decoder module, accumulates the output of each neural module, and uses the weight of the module for constraint as the output of the final Decoder part, which can prevent the gradient of the loss function from disappearing as the depth of the model increases;

[0055] (3.1h) Combine the high-level vector representation with the low-level vector representation to prevent information loss and enhance the high-dimensional representation learned by the base vector in different spaces. The specific calculation process is as follows:

[0056]

[0057] Where: Block i is the output of each module, W i is the weight corresponding to the module;

[0058] (3.1i) The output of the Decoder part is sent to a shallow fully connected neural network to obtain the output of the Transformer part.

[0059] Furthermore, the loss function is MSE, which continuously updates the model weights and continuously fits the true values ​​through loss calculation and gradient back propagation of the predicted value and the true value, thereby improving the generalization ability and robustness of the model. The calculation formula of MSE is as follows:

[0060]

[0061] Where: n is the size of the set; y i is the true value; is the test value.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The Transformer-based CRISPR / Cas9 single-guide RNA targeting activity prediction method proposed in the present application uses a self-attention mechanism through the Transformer to learn the global architecture information in the sequence and calculate the relationship between each base and all other bases. While maintaining a high degree of attention to local features, base units with greater correlation will have a higher weighted attention score; in this way, the interaction between bases can be captured by the process, and the same sequence features are mapped to multiple subspaces to obtain multiple vector space representations, which facilitates the fusion and learning of context data when calculating attention, and effectively alleviates the defect that the attention mechanism cannot capture context sequence information in both directions; in addition, on the basis of parallel processing of sequences, the vector expression of the sequence is learned through a multi-head mechanism to express the sequence in different subspaces, so as to prevent the local information weight from being too large, give the model a higher generalization ability, and ensure the efficiency and accuracy of calculation and learning; in order to avoid the gradient disappearance or information loss caused by deep networks, the dynamic residual structure adds the output of each module, which can extract feature information faster and enhance the model expression ability, which is conducive to effectively solving the problem of long sequence remote feature capture while ensuring parallel computing, and makes up for the defects of CNN. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0065] Figure 1 This is a flow chart of the model training process in the present invention;

[0066] Figure 2 It is a structural schematic diagram of the model training process flow in the present invention;

[0067] Figure 3 Schematic diagram of the overall architecture of the model in the present invention combined with the Transformer of dynamic residual. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0069] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0070] The purpose of this invention is to explore the spatial interaction relationship between bases in sgRNA sequences by using a Transformer-based dynamic residual structure deep learning method, so as to continuously improve the prediction efficiency of sgRNA on-target activity. Studies have shown that the influence of PAM upstream bases on sgRNA activity is greater than that of PAM distal regions. Transformer consists of a self-attention module and a feed-forward neural network. The self-attention mechanism is composed of a network, and self-attention is an attention mechanism that associates different positions of a single sequence to calculate the representation of the sequence. Based on the self-attention mechanism, Transformer maintains a high interest in the bases at the upstream (PAM-proximal) position of the PAM sequence while avoiding information loss in sequence calculation. It is a suitable strategy for optimizing sgRNA design and improving the efficiency of sgRNA prediction. At the same time, in addition to paying close attention to high-contribution sequence regions, Transformer can also perform bidirectional attention capture, that is, obtain the contribution of all contextual features for a certain position information, fully explore the correlation between data content context, and effectively alleviate the defect that the attention mechanism cannot capture contextual sequence information in both directions. In addition, while ensuring parallel computing, Transformer also effectively solves the problem of long sequence remote feature capture, making up for the defects of CNN.

[0071] Transformer consists of two parts: Encoder and Decoder. Each part contains a multi-layer unit module composed of a self-attention mechanism and a feedforward neural network. The sequence features are learned by stacking the modules. However, this architecture will also cause the gradient vanishing problem caused by the back propagation of the loss function. In order to solve this problem, a dynamic residual structure is adopted to capture the output of each module and integrate it into the final prediction result, while avoiding the loss of sequence information.

[0072] In one embodiment, referring to Figure 1-2 , provides a Transformer-based CRISPR / Cas9 single guide RNA targeting activity prediction method, including the following steps:

[0073] (1) Obtaining a public data set: Obtaining a public data set containing sgRNA sequences and indel frequencies; the data set includes Sniper-Cas9, SpCas9-NG, xCas9, and HypaCas9;

[0074] Specifically, the sgRNA sequence consists of 23 bases, including a 20-base pre-spacer sequence and a 3-base PAM sequence.

[0075] (2) sgRNA sequence encoding: preprocessing the public dataset, using complementary bases to represent the original sgRNA sequences in the public dataset, and encoding the sgRNA sequences in the public dataset using a specific encoding method to generate sample sets for different stages of the model, and dividing them into training sets and test sets with preset ratios;

[0076] Specifically, the preprocessing is used to normalize the indel frequency, limit it to the range of (0, 1), and use it as a label of the corresponding sequence;

[0077] Specifically, the specific process of the sgRNA sequence encoding is as follows:

[0078] (2.1) The sgRNA sequence is regarded as text, and a dictionary is constructed using the Tokenizer method, which includes the mapping of four different bases A, T, C, and G to their corresponding numerical values, forming {A:1, T:2, C:3, G:4};

[0079] (2.2) adding a flag bit SOS at the beginning of each sgRNA, and expanding the dictionary to add the element {SOS:0}; converting the single base in the sgRNA sequence into a corresponding positive integer according to the dictionary to realize the vectorization of the sgRNA sequence;

[0080] (2.3) Finally, the vectorized sgRNA sequences were used as sample sets and shuffled, and then divided into training set and test set in a ratio of 85%:15%.

[0081] (3) Constructing a prediction model: constructing a Transformer model based on a dynamic residual structure, inserting the training set as input data into the model for training, and outputting a trained prediction model according to preset indicators;

[0082] In one embodiment, referring to Figure 3 , the specific process of constructing the prediction model is as follows:

[0083] (3.1) inputting the training set, i.e., the vectorized sgRNA sequence, into the Transformer layer to capture the sgRNA sequence features, and obtaining the output of the Transformer part;

[0084] Specifically, the capture of the sgRNA sequence features includes:

[0085] (3.1a) The training set, i.e., the vectorized sgRNA sequence, is input into the Embedding layer. Each base in the sgRNA sequence is converted into a corresponding vector representation. The length of this vector is determined by the parameter embedding-size, and each dimension represents the attribute of the base.

[0086] (3.1b) The serial numbers 1-23 are used to represent the position data of the bases in the sgRNA. The position data is also vectorized by the Embedding method. The position vector is added to the base vector to obtain the final representation vector. The formula is as follows:

[0087] W=(ω1,ω2,…,ω m )

[0088] p=(p1,p2,…,p m )

[0089] e=(ω1+p1,ω2+p2,…,ω m +p m )

[0090] Where: W is the base vector; p is the position vector; e is the final representation vector;

[0091] (3.1c) The final representation vector is input into the encoder and decoder respectively, and the multi-head attention module in the encoder and decoder is used for attention representation and concatenation to obtain self-attention matrices in multiple sub-expression spaces. Specifically, the input sequence is mapped to 8 different sets of Q, K and V matrices, and 8 different attention representations are obtained respectively. Finally, the different representations are concatenated. The multi-head attention module is obtained by concatenating 8 scaled dot product attentions. The calculation formula is as follows:

[0092]

[0093] Where: Q, K, V represent matrices, and they are base vectors combined with position information through the learnable weight matrix W Q ,W k ,W v Multiply by; d kis the dimension of matrix K (set to 256 in the experiment); since the Q, K and V matrices represent the same input sequence in three different vector subspaces, the mutual contribution within the sequence can be effectively learned in the process of multiplying the Q and K vectors. k The calculation results are scaled to prevent the calculated contribution of individual words from being too high, which will lead to excessive variance and long gradients that are not conducive to optimization. After scaling, the data distribution is smoothed and then multiplied with the V matrix to obtain the contribution score of each position feature on all features of the sequence.

[0094] (3.1d) The self-attention matrices in the multiple sub-expression spaces are input into a feedforward neural network, which is composed of multiple layers of fully connected layers and uses ReLU as the activation function. The calculation formula is as follows:

[0095] FF=ReLU(xW+b)

[0096] Where: x is the output of the previous layer, W and b are the weight and bias of the fully connected layer respectively;

[0097] (3.1e) Residual connections and layer normalization are used for each multi-head attention module and feedforward neural network, and the distribution of base vectors is adjusted to avoid gradient vanishing / explosion or covariate shift problems. The calculation formula is as follows:

[0098] y=LN(F(x)+x)

[0099] Where: F(·) function is the multi-head attention function, LN is the layer normalization function, and the specific calculation method is as follows:

[0100]

[0101] Where: μ and δ are the sample mean and standard deviation respectively, y and β represent the trainable parameter vectors of scaling and translation respectively;

[0102] (3.1f) An Encoder-Decoder Attention module is added between the multi-head attention module and the feedforward neural network module in the Decoder part. At the same time, the multi-head attention module uses the base vector with position information added as the input of the Encoder part, and linearly transforms the original base vector:

[0103]

[0104] Where: x is the original input sequence, d k is the dimension of matrix K;

[0105] In one embodiment, the Encoder-Decoder Attention module uses the output of the Encoder and the output of the pre-order multi-head attention module in the Decoder as the input matrix for self-attention calculation. Since the present invention aims to learn the feature relationship in different spaces of sgRNA through a deep neural network, but the sgRNA sequence itself has no real semantic relationship, a comparative experiment is conducted in this section to demonstrate whether the pre-order output in the Decoder part is used as the Q or V matrix. The specific data is shown in the following table:

[0106] Table 1: Preamble output comparison

[0107]

[0108] The experiment was evaluated using two data sets, xCas9 and HypaCas9, under two evaluation indicators. Spearman represents the Spearman correlation coefficient, which is used to measure the degree of correlation of hierarchical sequencing variables. The calculation formula is as follows:

[0109]

[0110] Where: n is the number of samples in the test data set, the i-th predicted value After sorting with the true value y, we get and y′, the difference between them is d i ;

[0111] Pearson represents the Pearson correlation coefficient, which is used to measure the linear relationship between interval variables. The calculation formula is as follows:

[0112]

[0113] In the formula: cov(·) represents the covariance between the true value and the predicted value, and δ is the standard deviation;

[0114] The above experimental results show that under the system of cross-evaluation of two datasets and two evaluation indicators, the overall performance of using the previous output in the Decoder as the Q matrix of the next layer for self-attention calculation is better than that of using the V matrix for calculation. In the xCas9 dataset, the maximum difference is 0.7% when Pearson is used as the evaluation indicator, and the average performance of the former is 0.5 better than the latter.

[0115] (3.1g) The Encoder-Decoder Attention module uses the output of the Encoder and the output of the previous multi-head attention module in the Decoder as the input matrix for self-attention calculation, and introduces a dynamic residual structure in the Decoder module. The output of each neural module is accumulated and constrained using the weight of the module as the final output of the Decoder part. The output of each neural module can be expressed as:

[0116] out=LN(F(y)+y)

[0117] Where F(·) is the feedforward neural network function, LN is the layer normalization function, y is the output of the multi-head attention module, and combined with the normalization calculation for the multi-head attention module in step (3.1e), the output out is derived with respect to the input x:

[0118]

[0119] in and The calculation formulas are:

[0120]

[0121]

[0122] It can be seen that in the process of back propagation calculation of the neural network to update the weight matrix, there is a cumulative multiplication of multiplication factors, so there is still a possibility of gradient vanishing. In order to solve this problem, the present invention introduces a dynamic residual structure in the Decoder module, accumulates the output of each neural module, and uses the weight of the module for constraint as the output of the final Decoder part, which can prevent the gradient of the loss function from disappearing as the depth of the model increases;

[0123] (3.1h) Combine the high-level vector representation with the low-level vector representation to prevent information loss and enhance the high-dimensional representation learned by the base vector in different spaces. The specific calculation process is as follows:

[0124]

[0125] Where: Block i is the output of each module, W i is the weight corresponding to the module;

[0126] In one embodiment, a comparative experiment was conducted on the introduction of the dynamic residual structure using two data sets, SpCas9 and Sniper-Cas9, to detect the effect of the dynamic residual structure. The specific experimental results are as follows:

[0127] Table 2: Effect of detecting dynamic residual structure

[0128]

[0129] In order to ensure the validity of the experimental results, in addition to the two indicators of Spearman and Pearson described in Table 1, the R2 indicator is introduced for comprehensive evaluation to evaluate the goodness of fit of the regression model coefficients. The calculation formula is as follows:

[0130]

[0131] Where: y i is the true value of the ith time, is the predicted value for the i-th time, is the average of the true values;

[0132] The above experimental results show that for the SpCas9 dataset, the three evaluation indicators after the introduction of dynamic residuals are better than those without the introduction of dynamic residual structures, among which the R2 indicator is 8.9% higher than the latter; the Sniper-Cas9 dataset also has the same performance, and the R2 indicator is 11.3% higher than the result without the introduction of dynamic residual structures. The cross-comparison average is 3.5% higher after the introduction of dynamic residual structures than before, indicating that the introduction of dynamic residual structures can effectively improve the activity prediction efficiency of sgRNA.

[0133] (3.1i) The output of the Decoder part is sent to a shallow fully connected neural network to obtain the output of the Transformer part.

[0134] (3.2) Flattening the output of the Transformer part to obtain a two-dimensional vector of n*3072, that is, generating a flat;

[0135] (3.3) Input the flat layer into a fully connected layer with a parameter dimension of 64, use ReLU as the activation function, and name the layer dense1;

[0136] (3.4) Input the dense1 into the Dropout layer with a parameter of 0.4 and name it dropout1;

[0137] (3.5) Input the dropout1 into a fully connected layer with a parameter dimension of 32, use ReLU as the activation function, and name the layer dense2;

[0138] (3.6) The dense2 output is input again into the Dropout layer with a parameter of 0.4 and named dropout2;

[0139] (3.7) Finally, the output of the dropout2 layer is input to the output layer with a parameter of 1 to obtain the corresponding predicted indel frequency and perform the loss function calculation;

[0140] Specifically, the loss function is MSE, which continuously updates the model weights and continuously fits the true values ​​through loss calculation and gradient back propagation of the predicted value and the true value, thereby improving the generalization ability and robustness of the model. The calculation formula of MSE is as follows:

[0141]

[0142] Where: n is the size of the set; y i is the true value; is the test value.

[0143] (4) Model application: The sgRNA sequence to be tested is input into the trained prediction model to predict the indel frequency.

[0144] In one embodiment, the optimizer of the model is set to Adam, the learning rate is set to an initial value of 0.0001, the batch size in training is 64, and after the training is completed, the test is performed on the divided test set; the experimental results show that the Decoder module in the Transformer architecture makes its pre-order output as a Q matrix compared to a V matrix in both Spearman and Pearson indicator dimensions, and the Spearman indicator on the xCas9 data set is improved by 1.5%. After the dynamic residual structure is introduced, the average performance of this method on the test set is 1.3% higher than the existing optimal prediction algorithm. It can be seen that the method of the present invention can more accurately predict the activity of CRISPR / Cas9 single guide RNA and has a strong generalization ability.

[0145] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Transformer-based method for predicting the targeting activity of CRISPR / Cas9 single guide RNA, characterized in that: The steps include: (1) Obtaining a public data set: Obtaining a public data set containing sgRNA sequences and indel frequencies; the data set includes Sniper-Cas9, SpCas9-NG, xCas9, and HypaCas9; (2) sgRNA sequence encoding: preprocessing the public dataset, using complementary bases to represent the original sgRNA sequences in the public dataset, and encoding the sgRNA sequences in the public dataset using a specific encoding method to generate sample sets for different stages of the model, and dividing them into training sets and test sets with preset ratios; (3) Constructing a prediction model: constructing a Transformer model based on a dynamic residual structure, inserting the training set as input data into the model for training, and outputting a trained prediction model according to preset indicators; (4) Model application: inputting the sgRNA sequence to be tested into the trained prediction model to predict the indel frequency; The specific process of sgRNA sequence encoding in step (2) is as follows: (2.1) The sgRNA sequence is regarded as text, and a dictionary is constructed using the Tokenizer method, which includes the mapping of four different bases A, T, C, and G to their corresponding numerical values, forming {A:1, T:2, C:3, G:4}; (2.2) adding a flag bit SOS at the beginning of each sgRNA, and expanding the dictionary to add the element {SOS:0}; converting the single base in the sgRNA sequence into a corresponding positive integer according to the dictionary to realize the vectorization of the sgRNA sequence; (2.3) Finally, the vectorized sgRNA sequences are used as sample sets and shuffled, and divided into training sets and test sets in a ratio of 85%:15%; The specific process of constructing the prediction model is as follows: (3.1) Inputting the training set and the vectorized sgRNA sequence into the Transformer layer to capture the sgRNA sequence features, and obtaining the output of the Transformer part; (3.2) Flatten the output of the Transformer part to obtain a two-dimensional vector of n*3072, generating a flat; (3.3) Input the flat layer into a fully connected layer with a parameter dimension of 64, use ReLU as the activation function, and name the layer dense1; (3.4) Input the dense1 into the Dropout layer with a parameter of 0.4 and name it dropout1; (3.5) Input the dropout1 into a fully connected layer with a parameter dimension of 32, use ReLU as the activation function, and name the layer dense2; (3.6) The dense2 output is input again into the Dropout layer with a parameter of 0.4 and named dropout2; (3.7) Finally, the output of the dropout2 layer is input to the output layer with a parameter of 1 to obtain the corresponding predicted indel frequency and perform the loss function calculation; The capture of the sgRNA sequence features includes: (3.1a) The training set and the vectorized sgRNA sequence are input into the Embedding layer. Each base in the sgRNA sequence is converted into a corresponding vector representation. The length of this vector is determined by the parameter embedding-size, and each dimension represents the attribute of the base respectively; (3.1b) The serial numbers 1-23 are used to represent the position data of the bases in the sgRNA. The position data is also vectorized by the Embedding method. The position vector is added to the base vector to obtain the final representation vector. The formula is as follows: W=(ω1,ω2,…,ω m ) p=(p1,p2,…,p m ) e=(ω1+p1,ω2+p2,…,ω m +p m ) Where: W is the base vector; p is the position vector; e is the final representation vector; (3.1c) The final representation vector is input into the encoder and decoder respectively, and the multi-head attention module in the encoder and decoder is used for attention representation and concatenation to obtain the self-attention matrix in multiple sub-expression spaces. The multi-head attention module is obtained by concatenating 8 scaled dot product attentions, and its calculation formula is as follows: Where: Q, K, V represent matrices, and they are base vectors combined with position information through the learnable weight matrix W Q ,W k ,W v Multiply by; d k is the dimension of matrix K; (3.1d) The self-attention matrices in the multiple sub-expression spaces are input into a feedforward neural network, which is composed of multiple layers of fully connected layers and uses ReLU as the activation function. The calculation formula is as follows: FF=ReLU(xW+b) Where: x is the output of the previous layer, W and b are the weight and bias of the fully connected layer respectively; (3.1e) Residual connections and layer normalization are used for each multi-head attention module and feedforward neural network, and the distribution of base vectors is adjusted to avoid gradient vanishing / explosion or covariate shift problems. The calculation formula is as follows: y=LN(F(x)+x) Where: F(·) function is the multi-head attention function, LN is the layer normalization function, and the specific calculation method is as follows: Where: μ and δ are the sample mean and standard deviation respectively, γ and β represent the trainable parameter vectors of scaling and translation respectively; (3.1f) An Encoder-Decoder Attention module is added between the multi-head attention module and the feedforward neural network module in the Decoder part. At the same time, the multi-head attention module uses the base vector with position information added as the input of the Encoder part, and linearly transforms the original base vector: Where: x is the original input sequence, d k is the dimension of matrix K; (3.1g) The Encoder-Decoder Attention module uses the output of the Encoder and the output of the previous multi-head attention module in the Decoder as the input matrix for self-attention calculation, and introduces a dynamic residual structure in the Decoder module. The output of each neural module is accumulated and constrained using the weight of the module as the final output of the Decoder part. The output of each neural module can be expressed as: out=LN(F(y)+y) Where F(·) is the feedforward neural network function, LN is the layer normalization function, and y is the output of the multi-head attention module; (3.1h) Combine the high-level vector representation with the low-level vector representation to prevent information loss and enhance the high-dimensional representation learned by the base vector in different spaces. The specific calculation process is as follows: Where: Block i is the output of each module, W i is the weight corresponding to the module; (3.1i) The output of the Decoder part is sent to a shallow fully connected neural network to obtain the output of the Transformer part.

2. The Transformer-based CRISPR / Cas9 single guide RNA targeting activity prediction method according to claim 1, characterized in that: The sgRNA sequence in step (1) consists of 23 bases, including a 20-base pre-spacer sequence and a 3-base PAM sequence.

3. The Transformer-based CRISPR / Cas9 single guide RNA targeting activity prediction method according to claim 1, characterized in that: The preprocessing in step (2) is used to normalize the indel frequency, limit it to the range of (0, 1), and use it as a label for the corresponding sequence.

4. The Transformer-based CRISPR / Cas9 single guide RNA targeting activity prediction method according to claim 1, characterized in that: The loss function is MSE, which continuously updates the model weights and continuously fits the true values ​​through loss calculation and gradient back propagation of the predicted value and the true value, thereby improving the generalization ability and robustness of the model. The calculation formula of MSE is as follows: Where: n is the size of the set; y i is the true value; is the test value.

Images