Chemically modified small nucleic acid drug zero sample prediction method and system based on large language model and multi-view deep learning

Abstract

The invention relates to the technical field of biology, and discloses a chemically modified small nucleic acid drug zero sample prediction method and system based on a large language model and multi-view deep learning. The method comprises the following steps: collecting chemically modified RNA data; performing Token conversion by using an SMILES token regular expression, and performing embedding by using a pre-trained large language model; chemical modification position information is obtained and combined with SMILES in an embedded mode; the RNA sequence is embedded; processing the comprehensive embedded vector by using a self-attention mechanism; fusing RNA sequence embedding and comprehensive embedding by adopting a cross attention mechanism; performing feature extraction and dimension reduction by using a three-dimensional convolutional neural network; and inputting the features into a full-connection layer to predict the drug efficiency. According to the method, high-precision zero-sample prediction of novel chemical modification is realized, the generalization ability and prediction precision of the model are remarkably improved, and a flexible, efficient and accurate method is provided for design and development of RNAi drugs.

Description

Technical Field

[0001] The present application relates to the fields of bioinformatics and drug design, and in particular to a deep learning-based RNA interference (RNAi) chemically modified small nucleic acid drug efficiency prediction technology. Background Art

[0002] As a revolutionary mechanism for regulating gene expression, RNA interference (RNAi) technology has shown great potential in the field of disease treatment. With the continuous development of RNAi technology, the design and development of chemically modified small nucleic acid drugs has become a hot topic in biomedical research. These chemical modifications play a key role in the stability, specificity and therapeutic effect of RNAi molecules. However, with the continuous emergence of new chemical modifications, researchers are faced with a series of challenges.

[0003] First, traditional RNAi efficiency prediction models mainly rely on existing chemical modification data for training, which leads to poor performance when faced with new chemical modifications. This limitation seriously hinders the innovation and application of RNAi technology because researchers cannot quickly and accurately evaluate the effects of new modifications.

[0004] Secondly, existing models often use simple encoding or feature extraction methods when processing chemical modification information, which makes it difficult to fully capture the complex chemical properties of molecular structures. This lack of representation ability is particularly evident when dealing with modifications with high structural diversity, affecting the accuracy and reliability of predictions.

[0005] Another key issue is the impact of the position of chemical modifications in the RNA sequence. Existing models often ignore or simplify this important factor, resulting in an inability to accurately understand the impact of each modification at a specific position on RNAi efficiency. This lack of precision directly affects the guiding value of drug design.

[0006] In addition, existing methods are insufficient in integrating multimodal information. There are complex interactions between RNA sequence, chemical modification structure, and modification location information, but existing models are difficult to fully capture these relationships, thus limiting the improvement of prediction performance.

[0007] The scalability and adaptability of the model is also an urgent problem to be solved. With the rapid development of RNAi technology, new chemical modifications continue to emerge, and existing models often need to be retrained or significantly adjusted in the face of these changes, lacking flexibility and continuous adaptability.

[0008] Finally, in the process of RNAi drug development, traditional experimental screening methods are time-consuming and costly, which seriously restricts the speed of innovative drug development. The lack of efficient and accurate prediction tools makes it difficult for researchers to quickly evaluate and screen potential candidate drugs, slowing down the process of new RNAi treatments entering clinical applications.

[0009] In the face of these challenges, an innovative technical solution is urgently needed to achieve high-precision zero-sample prediction of new chemical modifications, enhance the characterization ability of chemical modification information, optimize the integration of modification position information, achieve deep fusion of multimodal information, improve the scalability and adaptability of the model, and ultimately accelerate the development of RNAi drugs. Such a technological breakthrough will provide strong support for the design, development and screening of RNAi drugs and promote the widespread application of RNAi technology in the field of biomedicine. Summary of the invention

[0010] The purpose of this application is to provide a zero-sample prediction method and system for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning to solve the problems raised in the above background technology.

[0011] The present application discloses a zero-sample prediction method for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning, comprising the following steps:

[0012] Collect and organize chemically modified RNA data, including RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula; use SMILES token regular expression to tokenize SMILES molecular formula; use pre-trained large-scale language model to embed the tokenized SMILES molecular formula to generate SMILES embedding vector that characterizes the chemical modification structural characteristics; obtain the position information of the chemical modification in the RNA sequence, and embed the position information to generate a position embedding vector; combine the SMILES embedding vector with the position embedding vector to form a comprehensive embedding vector; embed the RNA sequence to generate an RNA sequence embedding vector;

[0013] The comprehensive embedding vector is used to capture the relationship between the chemical modification and its position; the RNA sequence embedding vector is fused with the comprehensive embedding vector by adopting a cross attention mechanism to obtain a multimodal fusion feature;

[0014] A three-dimensional convolutional neural network is used to extract and reduce the dimension of the multimodal fusion features; the features are input into a fully connected layer to predict the efficiency of chemically modified small nucleic acid drugs, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecules, including prediction of new chemical modifications that have never been seen.

[0015] In a preferred embodiment, the method can perform zero-sample prediction for novel chemical modifications that have never been seen before without retraining the model.

[0016] In a preferred embodiment, the pre-trained large language model is pre-trained specifically for chemical structures and is selected from the following group: ChemBERTa, Llama.

[0017] In a preferred example, the SMILES token regular expression can tokenize the SMILES molecular formula in units of atoms, such as decomposing "CC(=O)O" into ["C","C","(","=","O",")","O"], thereby improving the fine-grained understanding of the molecular structure.

[0018] In a preferred example, in the step of obtaining the position information of the chemical modification in the RNA sequence and embedding the position information to generate a position embedding vector, the comprehensive embedding vector formed by combining the SMILES embedding vector with the position embedding vector can simultaneously capture the structural characteristics of the chemical modification and the position information in the RNA sequence. The dimension of the comprehensive embedding vector is 818 dimensions, including 768 dimensions of SMILES embedding information and 50 dimensions of position embedding information.

[0019] In a preferred example, in the step of embedding the RNA sequence to generate the RNA sequence embedding vector, the RNA sequence embedding adopts one-hot encoding or a pre-trained RNA sequence embedding model to improve the model's understanding of the RNA sequence information.

[0020] In a preferred example, the method simultaneously processes chemical modification structure information, position information and RNA sequence information through multimodal information fusion to improve the model's ability to understand the complex interactions between different features.

[0021] In a preferred embodiment, the method processes chemical modifications with different structures but similar biological effects to reflect the high correlation between chemical structure and biological function.

[0022] In a preferred example, in the step of using the comprehensive embedding vector to capture the relationship between the chemical modification and its position, the self-attention mechanism adopts a multi-head self-attention mechanism, including 8 attention heads, and uses sinusoidal position encoding or learnable position embedding to further improve the ability to capture the chemical modification position.

[0023] In a preferred example, in the step of adopting a cross-attention mechanism to fuse the RNA sequence embedding vector with the comprehensive embedding vector to obtain a multimodal fusion feature, the cross-attention mechanism adopts a multi-head cross-attention mechanism including 4 attention heads.

[0024] In a preferred example, a three-dimensional convolutional neural network is used to perform feature extraction and dimensionality reduction on the multimodal fusion features, wherein the convolution kernel size of the three-dimensional convolutional neural network is 3×3×3, and includes a maximum pooling layer and a batch normalization layer with a pooling kernel size of 2×2×2.

[0025] In a preferred example, the features are input into a fully connected layer to predict the efficiency of chemically modified small nucleic acid drugs, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecules, including the step of predicting new chemical modifications that have not been seen. The fully connected layer uses a multi-layer perceptron with a discard method to prevent overfitting and improve the generalization ability of the model.

[0026] In a preferred embodiment, the method further comprises the step of performing an interpretability analysis on the prediction results to reveal the relationship between different chemically modified structures and their biological effects.

[0027] In a preferred embodiment, the method further includes a step of using principal component analysis (PCA) to perform visual analysis on the SMILES embedding results.

[0028] In a preferred example, the method further comprises a step of verifying the prediction result, including comparing the prediction result with experimental data.

[0029] In a preferred embodiment, the method further comprises the step of optimizing the design of RNAi drugs based on the prediction results.

[0030] In a preferred example, after step 100, a data enhancement step is also included to generate new samples by transforming, combining or simulating the collected data to expand the training data set.

[0031] In a preferred example, a transfer learning step is also included, in which network parameters are initialized using a model pre-trained on a related task to improve the performance of the model on the target task.

[0032] In a preferred embodiment, a multi-task learning strategy is adopted during the training process to simultaneously predict multiple relevant drug properties, such as siRNA efficiency, off-target effects and stability, to improve the generalization ability of the model.

[0033] In a preferred embodiment, the method further includes a step of post-processing the prediction results, including but not limited to threshold screening, probability calibration or confidence interval estimation, so as to improve the reliability of the prediction results.

[0034] In a preferred example, the method can also be applied to other similar molecular design tasks, such as small molecule drugs, protein engineering and other fields, and cross-domain knowledge transfer can be achieved through appropriate adjustments and transfer learning.

[0035] In a preferred embodiment, the calculation of the self-attention mechanism includes:

[0036] a) Calculate the Query (Q), Key (K) and Value (V) matrices:

[0037] Q=XW Q ,K=XW K ,V=XW V

[0038] Where W Q ,W K ,W V is a trainable weight matrix;

[0039] b) Calculate the attention score:

[0040]

[0041] where d k is the dimension of K, and softmax is used to normalize the weights.

[0042] In a preferred embodiment, the calculation of the cross attention mechanism includes:

[0043] a) For two input matrices X1 and X2, calculate the cross query, key and value matrices respectively;

[0044] b) Calculate the cross-attention score;

[0045] c) Concatenate the obtained Z_1 and Z_2 to get the final output

[0046] z = Concat(Z1, Z2)

[0047] In a preferred embodiment, the method further includes verifying the biological effects of the predicted results in a cardiovascular-related cell model, including:

[0048] a) Evaluate the inhibitory effect of siRNA on target gene expression by fluorescence quantitative PCR (qPCR) and Western blot methods;

[0049] b) Compare the differences in biological effects of siRNA molecules with different chemical modifications and screen out the optimal modification combination.

[0050] The present application also discloses a zero-sample prediction system for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning, including:

[0051] Data collection module, used to collect and organize chemically modified RNA data, including RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula;

[0052] The SMILES embedding module is used to tokenize the SMILES molecular formula using the SMILES token regular expression to improve the fine-grained understanding of the molecular structure; and to embed the tokenized SMILES molecular formula using a pre-trained large language model to generate a SMILES embedding vector that characterizes the chemically modified structural features;

[0053] A position embedding module is used to obtain the position information of the chemical modification in the RNA sequence, embed the position information, generate a position embedding vector, and combine the SMILES embedding vector with the position embedding vector to form a comprehensive embedding vector;

[0054] RNA sequence embedding module, used to embed RNA sequences and generate RNA sequence embedding vectors;

[0055] A self-attention processing module, which processes the comprehensive embedding vector using the self-attention mechanism to capture the relationship between chemical modifications and their positions;

[0056] The cross-attention fusion module is used to fuse the RNA sequence embedding vector with the comprehensive embedding vector after self-attention processing using the cross-attention mechanism to obtain multimodal fusion features;

[0057] A feature extraction module is used to extract and reduce the dimension of multimodal fusion features using a three-dimensional convolutional neural network;

[0058] The prediction module is used to input the features obtained by the feature extraction module into the fully connected layer to predict the efficiency of the chemically modified small nucleic acid drug, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecule, including the prediction of new chemical modifications that have not been seen.

[0059] The implementation methods of this application have the following technical effects:

[0060] Achieve high-precision zero-shot prediction of new chemical modifications: By introducing SMILES embedding technology based on a large language model, this application can effectively capture subtle differences in molecular structure and achieve accurate prediction of new chemical modifications that have never been seen before. This innovation greatly improves the generalization ability of the model, enabling it to efficiently predict RNAi molecules containing new modifications without prior data, providing a strong tool support for the innovative design of RNAi drugs.

[0061] Enhanced characterization of chemical modification information: By using SMILES token regular expressions to tokenize molecular formulas, this application achieves a more detailed understanding of molecular structures. This method significantly improves the model's ability to capture complex chemical properties, especially when dealing with modifications with high structural diversity, thereby greatly improving the accuracy and reliability of predictions.

[0062] Optimizing the integration of modification position information: By combining position embedding and self-attention mechanisms, this application successfully solves the problem that chemical modifications have different effects at different positions in the RNA sequence. This innovative design enables the model to accurately understand the role of each modification at a specific position, significantly improving the accuracy of RNAi efficiency prediction and providing more precise guidance for drug design.

[0063] Realize the deep fusion of multimodal information: This application realizes the multimodal deep fusion of RNA sequence, chemical modification information and their positional relationship through the cross-attention mechanism. This strategy enables the model to fully understand the complex interactions between different features, thereby achieving a significant improvement in prediction performance, providing a more comprehensive and accurate evaluation basis for the design and screening of RNAi small molecules.

[0064] Improve the scalability and adaptability of the model: The model structure of this application has good scalability and adaptability. It can not only predict currently known modifications, but also achieve seamless integration and prediction of new modifications through simple expansion. This feature ensures that the model can continue to maintain its leading position in predictive ability in the context of the rapid development of RNAi technology, providing stable and reliable support for long-term drug development.

[0065] Accelerate the development of RNAi drugs: By providing accurate predictions of new chemical modifications, this application significantly reduces the time and resource investment required for traditional experimental screening. This efficient prediction method greatly accelerates the development of RNAi drugs, helps to push innovative treatment options into clinical applications more quickly, and ultimately brings more treatment options to patients.

[0066] In summary, this application effectively solves the key challenges faced by existing RNAi efficiency prediction models through innovative technical solutions, and provides a powerful and flexible tool for the design, development and screening of chemically modified small nucleic acid drugs, which has significant scientific value and application prospects.

[0067] A large number of technical features are recorded in the specification of this application, which are distributed in various technical solutions. If all possible combinations of technical features of this application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of this application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions are all deemed to have been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be deemed to have been recorded because it is technically infeasible, and the solution of A+B+C+E should be deemed to have been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flowchart of a zero-sample prediction method for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning according to the first embodiment of the present application.

[0069] Figure 2 It is a structural schematic diagram of a zero-sample prediction system for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning according to the second embodiment of the present application.

[0070] Figure 3 The model architecture of the large language model based zero-shot prediction (LLM based zero-shot prediction) according to an embodiment of the present application is shown.

[0071] Figure 4 The specific steps of a zero-sample prediction method based on a large language model (LLM-zs pred) are shown in a specific example according to an embodiment of the present application.

[0072] Figure 5 In a specific example according to an embodiment of the present application, a SMILES embedding process based on a large language model is shown, and the conversion process from a SMILES molecular formula to an embedding vector is described in detail.

[0073] Figure 6 The principal component analysis (PCA) results after embedding of six common siRNA chemical modifications are shown.

[0074] Figure 7The large language model-based zero-shot prediction (LLM-zs pred) model architecture is shown, detailing the complete process from RNA sequence embedding to the final prediction result. DETAILED DESCRIPTION

[0075] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0076] Description of some concepts:

[0077] RNA interference (RNAi): A biological process that inhibits gene expression through small RNA molecules, widely used in gene function research and treatment.

[0078] Chemical modification: refers to the addition or replacement of chemical groups on RNA molecules to enhance their stability, specificity and therapeutic effect.

[0079] Zero-Shot Prediction: A machine learning technique that can make effective predictions without ever seeing a specific sample.

[0080] Large Language Model (LLM): Such as ChatGPT and Llama, which are pre-trained models using large amounts of text data and can understand and generate natural language. Language models fine-tuned with special training sets can be applied in specific areas, such as ChemBERTa, which is used to embed chemical molecular structures in this application, that is, it can understand information related to chemical molecular structures. This type of model is mainly used to embed chemical molecular structures in this application.

[0081] SMILES molecular formula: A string representation of chemical molecular structures used to encode chemical information.

[0082] SMILES Embedding: SMILES molecular formulas are converted into high-dimensional vector representations through a large language model to capture their complex chemical properties.

[0083] Tokenization: Decompose SMILES molecular formulas into basic units (such as atoms and chemical bonds) to facilitate model processing and understanding.

[0084] Self-Attention Mechanism: A neural network technique that can capture the relationships and dependencies between different parts of an input sequence.

[0085] Position embedding: The position information of chemical modifications in RNA sequences is encoded as a vector representation and combined with SMILES embedding to enhance the model’s understanding of the effects of modification positions.

[0086] Cross-Attention Mechanism: A neural network technique that combines information from different modalities (such as RNA sequence and chemical modifications) to capture the complex interactions between them.

[0087] 3D Convolutional Neural Network (3D-CNN): A neural network for feature extraction and dimensionality reduction that is capable of processing 3D data.

[0088] Fully Connected Layer: A layer in a neural network used to make final predictions and classifications of the extracted features.

[0089] Principal Component Analysis (PCA): A data dimensionality reduction and visualization technique used to reveal the main features of high-dimensional data.

[0090] Data augmentation: Increasing the diversity and quantity of training data by transforming, combining, or generating new samples of the original data.

[0091] Transfer learning: Initializing network parameters using a model pre-trained on a related task to improve performance on the target task.

[0092] Multi-task learning: A technique for training multiple related tasks simultaneously to improve the generalization and performance of the model.

[0093] Interpretable analysis: Explain the model predictions to reveal the impact of different features and modifications on the results.

[0094] Model generalization ability: The ability of a model to maintain good performance on unseen data.

[0095] Some innovative points of this application are explained below.

[0096] As the application of siRNA small nucleic acid drugs in the treatment of cardiovascular diseases becomes increasingly mature, indications such as hyperlipidemia, heart disease, and vascular-related diseases occupy an important position in the research and development of small nucleic acid drugs. In order to improve the stability, specificity and efficacy of small nucleic acid drugs, chemical modification has become a core link in the design, especially in siRNA drug design. However, the existing technology faces the challenge of being difficult to effectively process new chemical modification information, and the high cost of sequence synthesis also hinders the development process. The present invention aims to fill this technical gap, and through innovative methods, quickly screen the chemically modified small nucleic acid sequences with the greatest potential, significantly reducing the cost of new drug development. For example, in the verification process of new chemical modifications, this application can reduce the number of candidate sequences that need to be synthesized and experimentally verified from about 100 in traditional methods to about 20, which not only greatly saves development costs, but also significantly shortens the development cycle. The implementation of this application will accelerate the development of efficient siRNA therapeutic drugs, promote progress in the field of cardiovascular disease treatment, and bring significant health benefits and social benefits to the national public health field.

[0097] In order to achieve the corresponding research objectives, this application adopts a series of innovative research schemes and key technologies. Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 As described. First, this application will collect and organize data on chemically modified RNA sequences from public databases and published literature. These data include different types of chemical modifications, corresponding RNA sequences, and their known biological effects. The goal of data organization is to provide high-quality input data for subsequent model training and verification. This application will pre-process the collected data, including removing redundant sequences, standardizing modification representations, and dividing the data into training sets, verification sets, and test sets.

[0098] After the data is sorted, this application will use a large language model (such as ChemBERTa or other more generalized large models such as Llama) to embed chemical modifications. The large language model can automatically extract deep molecular structure features from the SMILES representation of chemical modifications, and these features will be embedded in a high-dimensional space, thereby providing a rich information basis for subsequent model training. In order to enhance the generalization ability of the model for new modifications, this application uses the "SMILES tokenregex" method to tokenize the SMILES molecular formula. This method is more fine-grained than the traditional Byte-PairEncoding (BPE) method and can more accurately understand the structural information of chemical modifications. In this study, this application uses an embedding dimension of 768 dimensions to fully capture the complex characteristics of chemical modifications.

[0099] After obtaining the embedded representation of the chemical modification, this application will train and evaluate the model. The goal of this step is to develop an LLM-zs pred model that can perform zero-shot predictions for novel chemically modified siRNAs. This application uses deep learning technology to integrate chemical modifications, RNA sequences, and position information through self-attention and cross-attention mechanisms to improve the prediction accuracy of the model. Model training and evaluation include building and optimizing the neural network architecture, training the model using the training data set, adjusting the model parameters on the validation set, evaluating the performance of the model, and avoiding overfitting, and finally performing a final evaluation on the test set to ensure the generalization ability of the model.

[0100] The architecture of the LLM-zs pred model consists of an embedding layer, a self-attention mechanism module, a cross-attention mechanism module, a convolutional layer (3D-CNN), and a fully connected layer, which are used to ultimately predict the efficiency of siRNA. Among them, a self-attention mechanism module and a cross-attention mechanism module are used to establish associations between RNA sequences and chemical modifications and their sites, further enhancing the model's ability to identify new modification sites. This innovative architectural design enables the model of this application to better understand and predict the effects of chemical modifications on siRNA efficiency, providing a powerful tool support for the development of new RNAi drugs.

[0101] In this study, the present application adopted a variety of advanced mechanisms to improve the performance of the model. The first is the self-attention mechanism, which is used to capture the dependencies between modifications at different positions in the sequence. In the model of the present application, the self-attention mechanism can capture the structural characteristics of the interdependent influence of chemical modifications at various positions in the RNA sequence. The calculation process of self-attention includes two main steps: first, the Query (Q), Key (K) and Value (V) matrices are calculated, and then the attention score is calculated. Specifically, given the input matrix X, the present application uses a trainable weight matrix W Q ,W K ,W V Calculate Q = XW Q ,K=XW K ,V=XW V , then, the attention score is calculated as:

[0102]

[0103] where d k is the dimension of K, and softmax is used to normalize the weights.

[0104] Secondly, this application introduces a cross-attention mechanism to capture the interaction between chemical modification information and RNA sequences. By crossing the embedding vector of the chemical modification with the embedding vector of the RNA sequence, the model can learn the impact of the modification position on the RNA interference effect. The calculation process of cross-attention is similar to self-attention, but involves two input matrices X1 and X2. First, the cross-Query, Key and Value matrices are calculated, and then the cross-attention score is calculated. Finally, the obtained Z1 and Z2 are spliced ​​to obtain the final output Z = Contact (Z1, Z2). Through this process, the model is able to simultaneously capture the complex interactions between chemical modifications and RNA sequences, and integrate this information to improve the accuracy of predictions.

[0105] In addition to the attention mechanism, the model architecture of the present application also includes convolutional layers (3D-CNN), pooling layers and fully connected layers for the final prediction of siRNA efficiency. By combining an 818-dimensional embedding vector, the model significantly improves the accuracy and generalization ability of the model in processing zero-sample predictions of novel chemical modifications, and can effectively cope with challenging tasks in RNAi technology. Specifically, the model of the present application adopts the following key parameters: the embedding dimension is 818 dimensions, including 768-dimensional chemical modification SMILES embedding and 50-dimensional position embedding; the self-attention mechanism uses 8 attention heads; the cross-attention mechanism uses 4 attention heads; the convolution kernel size of 3D-CNN is 3×3×3; the pooling layer uses maximum pooling, and the kernel size is 2×2×2. These carefully adjusted parameters enable the model of the present application to better understand and predict the impact of chemical modifications on siRNA efficiency, providing strong support for the development of novel RNAi drugs.

[0106] After the model training is completed, this application will use the developed LLM-zs pred method to predict new chemical modifications that have not been seen. The purpose of this step is to screen out the most potential chemically modified siRNA molecules through the model for subsequent biological verification. The key steps for new sample processing and zero sample prediction include SMILES embedding and modification position embedding. In the SMILES embedding process, the chemically modified SMILES string is first tokenized, the SMILES tokenregex method is used, and then a 768-dimensional embedding vector is generated using a large language model. The modification position embedding designs a 50-dimensional position embedding vector to represent the specific position of the modification in the RNA sequence, and is spliced ​​with the SMILES embedding vector to form a final 818-dimensional embedding vector. Through the effective integration of the modification position and sequence information, the 818-dimensional vector formed can comprehensively characterize the structural and positional characteristics of the chemical modification, thereby providing high-quality input features for the model. Finally, the model output results are analyzed and screened to determine the candidate siRNA molecules for subsequent experimental verification.

[0107] In the result prediction evaluation and experimental verification stage, this application will verify the biological effects of the selected siRNA molecules in cardiovascular-related cell models (such as HEK-293T and HepG2). The focus of the verification is to evaluate the inhibitory efficiency of these siRNAs on target genes, as well as to detect possible off-target effects and toxicity. The verification steps include evaluating the inhibitory effect of siRNA on target gene expression through methods such as fluorescent quantitative PCR (qPCR) and Western blot; comparing the differences in biological effects of siRNA molecules with different chemical modifications, and screening out the optimal modification combination; based on the experimental results, further evaluating the model prediction ability and optimizing the model parameters.

[0108] Through these steps, this application will complete the whole process from data collation, model development to experimental verification, ensuring that the developed LLM-zs pred method can provide effective support for the design of siRNA drugs related to complex diseases such as cardiovascular disease, and realize the efficient screening and verification of new chemically modified siRNA drugs. This comprehensive research method can not only accelerate the development process of siRNA drugs, but also improve the prediction accuracy of new chemical modifications, providing an important theoretical and practical basis for future RNA interference therapeutic strategies.

[0109] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0110] The first embodiment of the present application relates to a zero-sample prediction method for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning, the process of which is as follows: Figure 1 As shown, the method comprises the following steps:

[0111] Step 100: Data collection and organization:

[0112] Collect and organize chemically modified RNA data, including RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula.

[0113] Optionally, after step 100, a data enhancement step is also included to generate new samples by transforming, combining or simulating the collected data to expand the training data set.

[0114] Specifically, step 100 mainly involves the collection and organization of chemically modified RNA data. In this step, relevant data are first collected from multiple sources, including public databases (such as siRNAmod, miRBase, RNAcentral, etc.), published literature and experimental data, and possible proprietary data sets or laboratory internal data. The collected data needs to include RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula.

[0115] During the data collation process, the complete RNA nucleotide sequence needs to be recorded, usually represented by A, U, G, and C. For chemical modification types, all modifications involved are listed, such as 2'-O-methylation, phosphorothioation, etc. The modification position needs to accurately mark the position of each modification in the RNA sequence. At the same time, it is also necessary to generate a corresponding SMILES (Simplified Molecular Input Line Entry System) representation for each chemical modification. SMILES is a linear representation used to describe molecular structures. It can simplify complex chemical structures into a string of characters for easy computer processing and analysis.

[0116] For example, a typical data may be: RNA sequence is AUGCAUGCAUGC, chemical modification type is 2'-O-methylation, modification positions are 3 and 9, and the corresponding SMILES formula may be COC1C(OC(C1O)n1cnc2c1ncnc2N)CO. These data need to be organized into a structured format, such as CSV, JSON or a dedicated database format, for subsequent processing.

[0117] During the data collection process, data quality control is also required. This includes checking data integrity to ensure that each record contains all the required information; verifying the correctness of the SMILES molecular formula, which can be verified using chemical informatics tools such as RDKit; and dealing with possible missing values ​​and outliers.

[0118] As an optional step, data augmentation can be performed after step 100. The purpose of data augmentation is to expand the training data set and improve the generalization ability of the model. Specific methods include performing point mutations, insertions, or deletions on the RNA sequence; adding or replacing modifications based on the original modifications to generate new modification combinations; using a generative model (such as VAE or GAN) to generate new chemically modified SMILES; or making slight changes to the SMILES molecular formula to simulate modifications with similar structures.

[0119] For example, for the original data AUGCAUGCAUGC, 2'-O-methylation at positions 3 and 9, data augmentation can be used to generate new samples of AUGCAUGCAAUAC, 2'-O-methylation at positions 3, 9, and 11. This data augmentation technique can significantly expand the size and diversity of the training set, which helps improve the performance of the model in zero-shot learning scenarios.

[0120] Through systematic data collection, organization and optional data enhancement, a rich, diverse and high-quality dataset is provided for subsequent model training, which is of great significance for improving the generalization ability and prediction accuracy of the model, especially when facing new and unseen chemical modifications.

[0121] Step 200: SMILES formula embedding:

[0122] a) Use SMILES token regular expressions to tokenize SMILES molecular formulas to improve the fine-grained understanding of molecular structures;

[0123] b) Use a pre-trained large language model to embed the tokenized SMILES molecular formula and generate a SMILES embedding vector that represents the chemically modified structural features.

[0124] Optionally, the pre-trained large language model used in step 200 is pre-trained specifically for chemical structures, including but not limited to ChemBERTa, Llama or other language models applicable to the chemical field.

[0125] Optionally, the SMILES token regular expression in step 200 can tokenize the SMILES molecular formula in units of atoms, such as decomposing "CC(=O)O" into ["C","C","(","=","O",")","O"], thereby improving the fine-grained understanding of the molecular structure.

[0126] Specifically, step 200 involves embedding of SMILES molecular formula, which is a key step in converting chemical structure information into numerical representation understandable by machine learning models. This step is divided into two main parts: tokenization and embedding generation.

[0127] First, the SMILES token regular expression is used to tokenize the SMILES molecular formula. In a specific embodiment, the SMILES token regular expression can be in the following form:

[0128] token_regex = "(\[[^\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\(|\)|\.|=|#|\+|\\\\|\ / |:|@|\?|>|\\*|\$|\%[0-9]{2}|[0-9])" This regular expression can effectively decompose the SMILES string into individual chemical elements and structure tags.

[0129] SMILES (Simplified Molecular Input Line Entry System) is a widely used linear representation of chemical structures that can simplify complex molecular structures into strings. Tokenization is the process of further decomposing this string into the smallest meaningful units. Using regular expressions for tokenization can achieve a fine-grained understanding of molecular structures because it can accurately identify and separate the various components in the SMILES string, such as atoms, chemical bonds, branch structures, etc.

[0130] For example, for the SMILES string "CC(=O)O" (representing the acetic acid molecule), the possible results after tokenization are ["C","C","(","=","O",")","O"]. This decomposition method allows the model to better understand the structural characteristics of the molecule, including the atom types, chemical bond types, and the connection relationships between them.

[0131] Next, the tokenized SMILES molecular formula is embedded using a pre-trained large language model. This step utilizes the transfer learning technology in deep learning. By using a language model pre-trained on large-scale chemical data, chemical structure information can be converted into a high-dimensional vector representation. These vectors are called SMILES embedding vectors, which can capture complex chemical structure features and potential chemical property information.

[0132] The pre-trained large language model used can be a model pre-trained specifically for chemical structures, such as ChemBERTa or other language models applicable to the field of chemistry. These models are pre-trained on large-scale chemical datasets, so they can better understand and represent chemical structure information. For example, ChemBERTa is a model based on the BERT (Bidirectional Encoder Representations from Transformers) architecture, which is pre-trained specifically for chemical SMILES data and can generate high-quality chemical structure representations.

[0133] In addition, the design of SMILES token regular expressions can also be further optimized to achieve more sophisticated tokenization. For example, a regular expression can be designed to tokenize SMILES molecular formulas in units of atoms. This method can more accurately capture the structural information of the molecule, including the type of atoms, the type of chemical bonds, and their positions and connections in the molecule.

[0134] Through this fine-grained tokenization and professional language model embedding, this step can generate high-quality SMILES embedding vectors, providing rich and accurate molecular structure representation for subsequent machine learning tasks. This representation method can not only capture the local structural characteristics of the molecule, but also reflect the global chemical properties, thereby providing the model with more comprehensive and meaningful input information and improving the accuracy of the prediction of chemical modification effects.

[0135] Step 300: Position information embedding:

[0136] a) obtaining the position information of the chemical modification in the RNA sequence, and embedding the position information to generate a position embedding vector;

[0137] b) Combining the SMILES embedding vector obtained in step 200 with the position embedding vector to form a comprehensive embedding vector.

[0138] Optionally, the comprehensive embedding vector formed by combining the SMILES embedding vector and the position embedding vector in step 300 can simultaneously capture the structural characteristics of the chemical modification and the position information in the RNA sequence. The dimension of the comprehensive embedding vector is 818 dimensions, including 768-dimensional SMILES embedding information and 50-dimensional position embedding information.

[0139] Specifically, step 300 involves embedding processing of position information, which is an important step for converting the position information of chemical modification in RNA sequence into numerical representation. The purpose of this step is to combine the position information with the structural information of chemical modification to form a more comprehensive representation.

[0140] First, obtain the position information of chemical modification in RNA sequence. This information is represented by binary method. For a siRNA molecule, including forward strand and reverse strand, each position is represented by 0 or 1. For example, for a 25mer siRNA sequence, there are 50 positions in total (25 in forward strand and 25 in reverse strand). If there is chemical modification at a certain position, it is represented by 1; if there is no modification, it is represented by 0.

[0141] Specific example: Assume that in the forward strand of a 25mer siRNA, the 5th and 7th nucleotides are chemically modified, and their position information can be expressed as:

[0142] "000010100000000000000000000000000000000000000000000000000"

[0143] In this binary sequence, the first 25 bits represent the forward chain, and the last 25 bits represent the reverse chain. "1" at the 5th and 7th positions indicates that these two positions are chemically modified, and "0" at other positions indicates that there is no modification.

[0144] This representation method can accurately describe the distribution of chemical modifications in the siRNA double strands, which helps the model better understand and learn modification patterns.

[0145] Next, this position information is embedded to generate a position embedding vector. Position embedding is a technique that converts discrete position information into a continuous vector representation. Common position embedding methods include learned position embedding and fixed position embedding (such as sinusoidal position encoding). The output of this step is a vector representing the position information, whose dimension is usually smaller than that of the SMILES embedding vector.

[0146] Then, the SMILES embedding vector obtained in step 200 is combined with the position embedding vector to form a comprehensive embedding vector. This combination process can be a simple vector concatenation or a more complex fusion operation, such as weighted summation or fusion using a neural network layer. The purpose of this step is to integrate the structural information and position information of the chemical modification into a unified representation.

[0147] The dimension of the comprehensive embedding vector can be set to 818 dimensions, which includes 768-dimensional SMILES embedding information and 50-dimensional position embedding information. This design can capture both the structural features of chemical modifications and the position information in the RNA sequence. The 768-dimensional SMILES embedding information usually comes from the output of a large pre-trained language model (such as BERT or its variants), which is large enough to capture complex chemical structure information. The 50-dimensional position embedding information is sufficient to represent the position in the RNA sequence without making the total dimension too high.

[0148] For example, suppose there is an RNA sequence "AUGCAUGCAUGC" with 2'-O-methylation modifications at positions 3 and 9. For the modification at position 3, its SMILES embedding vector (768 dimensions) represents the chemical structure of the 2'-O-methylation, while the position embedding vector (50 dimensions) encodes the position information of "3". Combining these two vectors, a comprehensive embedding vector of 818 dimensions is formed, which fully represents the chemical modification information at this specific position.

[0149] The technical effect of this comprehensive embedding method is that it can not only represent the structure of the chemical modification, but also reflect the position of the modification in the RNA sequence, which is crucial for accurately predicting the effect of the modification on RNA function. Because the same chemical modification may have different biological effects at different positions in the RNA sequence. In this way, the model can learn the position-related chemical modification effects, thereby improving the accuracy of the prediction and the generalization ability of the model.

[0150] Step 400: RNA sequence embedding:

[0151] Embed the RNA sequence to generate an RNA sequence embedding vector.

[0152] Optionally, the RNA sequence embedding in step 400 uses one-hot encoding or a pre-trained RNA sequence embedding model to improve the model's understanding of RNA sequence information.

[0153] Specifically, step 400 involves the embedding process of RNA sequence, which is a key step to convert RNA sequence information into numerical representation. The purpose of RNA sequence embedding is to convert discrete nucleotide sequences into continuous vector space so that machine learning models can effectively process and learn these sequence information.

[0154] In this step, the input RNA sequence needs to be preprocessed first. RNA sequences are usually composed of four nucleotides (A, U, G, C). Preprocessing may include standardization of sequence length, processing of special characters, etc.

[0155] Next, the preprocessed RNA sequence is embedded to generate an RNA sequence embedding vector. This process can be done in a variety of ways, two of which are commonly used methods: one-hot encoding and pre-trained RNA sequence embedding models.

[0156] One-hot encoding is a simple and effective sequence representation method. For RNA sequences, each nucleotide can be represented by a 4-dimensional vector, for example, A can be represented as [1,0,0,0], U as [0,1,0,0], G as [0,0,1,0], and C as [0,0,0,1]. The entire RNA sequence can be represented as a series of such vectors. For example, the sequence "AUG" can be represented as [[1,0,0,0], [0,1,0,0], [0,0,1,0]]. The advantage of this method is that it is simple and intuitive, but the disadvantage is that it cannot capture the similarity and dependency between nucleotides.

[0157] Another more complex but more powerful approach is to use a pre-trained RNA sequence embedding model. This approach borrows word embedding techniques from natural language processing to map each nucleotide or k-mer (k consecutive nucleotides) to a high-dimensional vector space. This embedding can be obtained through unsupervised learning on a large amount of RNA sequence data, such as using Word2Vec, GloVe, or more modern neural network models such as Transformer. Pre-trained RNA sequence embedding models can capture complex relationships and sequence patterns between nucleotides, and therefore can generally provide richer sequence representations.

[0158] For example, suppose a pre-trained RNA sequence embedding model is used to map each nucleotide to a 100-dimensional vector space. Then for the sequence "AUG", three 100-dimensional vectors may be obtained, representing the representation of A, U, and G in this vector space. These three vectors can be further combined (for example, by concatenation or averaging) to obtain the representation of the entire sequence.

[0159] The technical effects of using this RNA sequence embedding method are mainly reflected in the following aspects: First, it can convert discrete sequence information into continuous vector representations, allowing deep learning models to better process this information. Second, especially when using pre-trained embedding models, local structural features and global dependencies in the sequence can be captured, which is crucial for understanding the function of RNA. Finally, this representation method can be combined with the chemical modification information obtained in the previous step to provide comprehensive and rich input features for subsequent prediction tasks.

[0160] In this way, the model is able to better understand and utilize RNA sequence information, thereby improving the accuracy of predictions about the effects of chemical modifications. This is important for the design and optimization of RNA interference (RNAi) drugs because it can help researchers more accurately predict the effects of specific sequences and modification combinations.

[0161] Step 500: Self-attention mechanism processing:

[0162] The comprehensive embedding vector obtained in step 300 is processed using a self-attention mechanism to capture the relationship between the chemical modification and its position.

[0163] Optionally, the self-attention mechanism in step 500 adopts a multi-head self-attention mechanism, including 8 attention heads, and uses sinusoidal position encoding or learnable position embedding to further improve the ability to capture chemical modification positions.

[0164] Optionally, in this step, the comprehensive embedding vector obtained in step 300 is processed using a self-attention mechanism to capture the relationship between the chemical modification and its position. The calculation of the self-attention mechanism includes the following specific steps:

[0165] a) Calculate the Query (Q), Key (K) and Value (V) matrices:

[0166] Q=XW Q ,K=XW K ,V=XW V

[0167] Where W Q ,W K ,W V is a trainable weight matrix.

[0168] b) Calculate the attention score:

[0169]

[0170] where d k is the dimension of K, and softmax is used to normalize the weights.

[0171] Through this self-attention mechanism, the relationship between chemical modifications and their positions in RNA sequences can be effectively captured, thereby improving the prediction accuracy of the model.

[0172] Specifically, step 500 introduces a self-attention mechanism to process the comprehensive embedding vector, which is a powerful technique that can effectively capture the complex relationship between chemical modifications and their positions in the RNA sequence. The core idea of ​​the self-attention mechanism is to allow each element in the input sequence to interact with all other elements in the sequence, thereby learning the dependencies between elements.

[0173] In this step, the calculation process of the self-attention mechanism first involves the generation of three key matrices: query matrix (Query, Q), key matrix (Key, K) and value matrix (Value, V). These matrices are obtained by combining the input vector X with the trainable weight matrix W Q ,W K ,W V Multiply them together to get. Specifically, Q = XW Q ,K=XW K ,V=XW V The purpose of this step is to transform the input vector into a representation suitable for attention calculation.

[0174] Next, the calculation of the attention score is the core of the self-attention mechanism. The attention score is calculated by the following formula:

[0175]

[0176] This formula can be broken down into several steps: First, the dot product of the transpose of Q and K is calculated, which reflects the similarity or correlation between different positions in the input sequence. Then, the result is divided by d k (where d k is the dimension of K), which is a scaling factor used to stabilize the gradient. Finally, a softmax function is applied to normalize the scores into a probability distribution. These probabilities are used as weights and multiplied with V to get the final attention output.

[0177] To further improve the expressiveness of the model, a multi-head self-attention mechanism can be used. For example, using 8 attention heads, each head independently learns a different attention pattern, and then merges these results. This approach allows the model to focus on different types of interactions and patterns at the same time, thereby enhancing the ability to model complex relationships.

[0178] In the self-attention mechanism, the encoding of position information is also crucial. This can be achieved using either sinusoidal position encoding or learnable position embedding. Sinusoidal position encoding uses sine and cosine functions of different frequencies to represent positions. The advantage of this approach is that it can handle sequences of arbitrary length without retraining. On the other hand, learnable position embedding allows the model to adaptively learn the position representation that best suits the task.

[0179] Through this self-attention mechanism, the model can effectively capture the complex relationship between chemical modifications and their positions in RNA sequences. For example, it can learn the importance of certain chemical modifications at specific positions, or the mutual influence between modifications at different positions. This mechanism is particularly suitable for handling long-range dependencies, which is particularly important in RNA sequence analysis because the function of RNA often depends on its long-range structural features.

[0180] In general, the self-attention mechanism in step 500 provides the model with a powerful ability to fully understand the structural features, positional information, and interactions between chemical modifications. This in-depth understanding is crucial for accurately predicting the effects of chemical modifications on RNA function, thereby providing strong technical support for the design and optimization of siRNA drugs.

[0181] Step 600: Cross-attention mechanism fusion:

[0182] The RNA sequence embedding vector obtained in step 400 is fused with the comprehensive embedding vector processed in step 500 by using the cross attention mechanism to obtain a multimodal fusion feature.

[0183] Optionally, the cross-attention mechanism in step 600 adopts a multi-head cross-attention mechanism including 4 attention heads.

[0184] Optionally, the method of this embodiment can simultaneously process chemical modification structure information, position information and RNA sequence information through multimodal information fusion, thereby improving the model's ability to understand the complex interactions between different features.

[0185] Optionally, the calculation of the cross attention mechanism includes the following steps: a) for the two input matrices X1 (RNA sequence embedding vector) and X2 (processed comprehensive embedding vector), respectively calculate the cross query, key and value matrices; b) calculate the cross attention score; c) concatenate the obtained Z1 and Z2 to obtain the final output

[0186] Z = Concat(Z1, Z2)

[0187] Where Z is the multimodal fusion feature.

[0188] This processing method can effectively capture the complex interactive relationship between RNA sequence information and chemical modification information, thereby improving the model's ability to understand the interactions between different features and further improving the accuracy of predictions.

[0189] Specifically, step 600 introduces a cross-attention mechanism, which is an efficient multimodal information fusion technique for fusing the RNA sequence embedding vector with the comprehensive embedding vector processed by self-attention. The core idea of ​​the cross-attention mechanism is to allow information from two different sources to pay attention to each other, thereby capturing the complex interactive relationship between them.

[0190] In this step, the calculation process of the cross-attention mechanism first involves two input matrices: X1 (RNA sequence embedding vector) and X2 (processed comprehensive embedding vector). For these two matrices, the cross-query, key, and value matrices are calculated respectively. Specifically, for X1, calculate For X2, calculate Here are all trainable weight matrices.

[0191] Next, the cross attention score is calculated. This process includes attention calculations in two directions: X1's attention to X2 and X2's attention to X1. The calculation formula is as follows:

[0192]

[0193]

[0194] d k is the dimension of K. This step allows the elements of each modality to be weighted according to their relevance to the other modality.

[0195] Finally, the obtained Z1 and Z2 are concatenated to obtain the final multimodal fusion feature Z = [Z1; Z2]. This fusion feature contains the mutual attention information between the two modalities and can effectively capture the complex interactive relationship between RNA sequence information and chemical modification information.

[0196] To further improve the expressiveness of the model, a multi-head cross-attention mechanism can be used, for example, using 4 attention heads. Each head independently learns different attention patterns, and then merges these results. This approach allows the model to focus on different types of cross-modal interactions at the same time, thereby enhancing the ability to model complex relationships.

[0197] An important advantage of the criss-cross attention mechanism is that it can handle inputs of different lengths and dimensions. For example, the embedding vector of an RNA sequence may be longer than the embedding vector of chemical modification information, but the criss-cross attention mechanism can effectively fuse these two different dimensions of information.

[0198] A specific application example of this multimodal information fusion method can be: when predicting the effect of a specific chemical modification on RNA function, the model not only considers the chemical structure of the modification itself, but also pays attention to its position in the RNA sequence and the surrounding nucleotide sequence. For example, if a chemical modification is located in an important functional region of RNA (such as a binding site), the model can learn the importance of this position through the cross-attention mechanism.

[0199] In general, the cross-attention mechanism in step 600 provides the model with a powerful ability to fully understand the complex interactions between RNA sequence information and chemical modification information. This in-depth understanding is crucial for accurately predicting the effects of chemical modifications on RNA function, thereby providing strong technical support for the design and optimization of siRNA drugs. In this way, the model can better capture the interactions between different features and improve the understanding and prediction capabilities of complex biological processes.

[0200] Step 700: Feature extraction and dimensionality reduction:

[0201] A three-dimensional convolutional neural network is used to perform feature extraction and dimension reduction on the multimodal fusion features obtained in step 600.

[0202] Optionally, the convolution kernel size of the three-dimensional convolutional neural network used in step 700 is 3×3×3, and includes a maximum pooling layer (whose pooling kernel size is 2×2×2) and a batch normalization layer.

[0203] Specifically, step 700 introduces a three-dimensional convolutional neural network (3D CNN) to process the multimodal fusion features obtained in step 600, with the main purpose of feature extraction and dimensionality reduction. The design of this step fully considers the complexity and high-dimensional characteristics of multimodal fusion features. Through the powerful feature learning ability of 3D CNN, key information can be effectively extracted and feature dimensions can be reduced.

[0204] The 3D convolutional neural network is an extension of the 2D convolutional neural network and is particularly suitable for processing data with spatial and temporal (or sequence) dimensions. In this step, the multimodal fusion feature can be regarded as a 3D data structure that contains RNA sequence information, chemical modification information, and the interaction between them. 3D CNN can capture the complex spatial relationship between features by sliding the convolution kernel simultaneously in these three dimensions.

[0205] The 3D CNN used in this step uses a convolution kernel size of 3×3×3. This size of convolution kernel can effectively capture local features while maintaining computational efficiency. For example, it can simultaneously consider several adjacent nucleotides in the RNA sequence, the chemical modification information associated with them, and the interaction between them. By stacking multiple such convolution layers, the network can gradually learn more abstract and advanced feature representations.

[0206] In order to further reduce the feature dimension and improve the robustness of the model, the 3D CNN also includes a maximum pooling layer with a pooling kernel size of 2×2×2. The maximum pooling operation can effectively reduce the spatial resolution of the feature map while retaining the most significant features by selecting the maximum value in each 2×2×2 area. This not only reduces the amount of computation in subsequent layers, but also improves the model's invariance to small translations and rotations.

[0207] In addition, the 3D CNN in this step also introduces a batch normalization layer. Batch normalization is an important technique that stabilizes the training process of deep networks by standardizing the input of each layer. Specifically, the batch normalization layer normalizes the data of each batch and then introduces learnable scaling and translation parameters. This processing can alleviate the problem of internal covariate shift, accelerate the training process of the network, and play a regularization role to some extent, which helps to improve the generalization ability of the model.

[0208] For example, suppose the initial dimension of the multimodal fusion feature is 64×100×100, where 64 represents the number of feature channels and 100×100 represents the spatial dimension. After a convolution layer using a 3×3×3 convolution kernel (assuming a stride of 1 and padding of 1), the spatial dimension of the feature map remains unchanged, but the number of channels may increase, such as to 128×100×100. Then, after a 2×2×2 maximum pooling layer, the spatial dimension of the feature map is halved to 128×50×50. Through multiple such operations, a feature representation with a greatly reduced dimension but more concentrated information can be obtained, such as 256×12×12.

[0209] In this way, step 700 can effectively extract key information from high-dimensional multimodal fusion features and compress them into a more compact representation. This not only reduces the computational complexity of subsequent processing, but also helps the model better focus on the most important patterns and features in the data, thereby improving the accuracy of RNA function prediction. This process of feature extraction and dimensionality reduction is crucial for understanding complex RNA-chemical modification interactions and lays a solid foundation for subsequent prediction tasks.

[0210] Step 800 Efficiency Prediction:

[0211] The features obtained in step 700 are input into the fully connected layer to predict the efficiency of the chemically modified small nucleic acid drug, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecule, including prediction of new chemical modifications that have never been seen.

[0212] Among them, the above steps achieve zero-sample prediction of new chemically modified small nucleic acid drugs, significantly improving the generalization ability and prediction accuracy of the model, providing a flexible, efficient and accurate method for the design and development of RNAi drugs.

[0213] Optionally, the fully connected layer in step 800 uses a multi-layer perceptron with a dropout method to prevent overfitting and improve the generalization ability of the model.

[0214] Specifically, step 800 is the last step of the entire prediction process and is also the core step, which aims to use the high-level features extracted in the previous steps to predict the efficiency of chemically modified small nucleic acid drugs. This step uses a fully connected layer, also known as a multi-layer perceptron (MLP), to process the compressed features obtained in step 700 and output the final efficiency prediction result.

[0215] The fully connected layer is a basic structure in the neural network, which is characterized by the fact that each neuron is connected to all neurons in the previous layer. In this step, the fully connected layer receives the features output by the three-dimensional convolutional neural network, flattens them into a one-dimensional vector, and then gradually maps the high-dimensional features to the final efficiency prediction value through a series of linear transformations and nonlinear activation functions.

[0216] In order to improve the generalization ability of the model and prevent overfitting, the fully connected layer in this step uses the dropout method. Dropout is a powerful regularization technique that works by randomly "dropping" some neurons during the training process, that is, temporarily setting the output of these neurons to 0. This method can effectively prevent the neural network from overfitting the training data and improve the performance of the model on unseen data.

[0217] For example, suppose the feature dimension of step 700 output is 256×12×12. First, this feature will be flattened into a 36864-dimensional vector. Then, this vector may pass through multiple fully connected layers, such as:

[0218] 36864 → 1024 (using ReLU activation function and applying 50% dropout rate)

[0219] 1024 → 256 (using ReLU activation function and applying 50% dropout rate)

[0220] 256 → 64 (using ReLU activation function and applying 50% dropout rate)

[0221] 64→1 (final output layer, possibly using Sigmoid activation function to output efficiency value between 0 and 1)

[0222] In this structure, the middle layer uses the ReLU (Rectified Linear Unit) activation function to introduce nonlinearity, which helps the model learn more complex patterns. The last layer may use the Sigmoid function to compress the output to between 0 and 1, which is suitable for regression tasks such as efficiency prediction.

[0223] A key advantage of this structural design is that it enables zero-shot prediction of novel chemically modified small nucleic acid drugs. Zero-shot prediction refers to the ability of the model to predict new categories or new features that have not been seen during training. In this method, through deep feature extraction and fusion of RNA sequences and chemical modifications, the model learns a universal feature representation and efficiency prediction mechanism, rather than simply memorizing the known chemical modification-efficiency correspondence.

[0224] For example, suppose the model has only seen two modifications, methylation and phosphorylation, during training, but by learning how these modifications affect RNA function, the model may be able to infer the effect of a new ethylation modification on RNA efficiency. This is because the model learns the general rules of how factors such as the chemical nature and position of the modification affect RNA function, rather than just the effect of a specific modification.

[0225] This method significantly improves the generalization ability and prediction accuracy of the model. It can not only accurately predict the effects of known chemical modifications, but also make reasonable predictions for new chemical modifications. This provides a flexible, efficient and accurate method for the design and development of RNAi drugs. Researchers can use this model to quickly evaluate various potential chemical modification schemes, even new modifications that have not yet been synthesized in the laboratory, thereby greatly accelerating the drug development process and reducing R&D costs.

[0226] In general, step 800 realizes the mapping from complex multimodal features to specific efficiency predictions by combining multiple advanced technologies in deep learning, providing a powerful computational tool for the development of chemically modified small nucleic acid drugs.

[0227] In general, step 800 realizes the mapping from complex multimodal features to specific efficiency predictions by combining multiple advanced technologies in deep learning, providing a powerful computational tool for the development of chemically modified small nucleic acid drugs.

[0228] After completing the design and initial implementation of the model, the next key step is to fully train, evaluate, and validate the model to ensure its effectiveness and reliability in practical applications. The following details the methods and strategies adopted by this application in model training and result evaluation.

[0229] Model training and result evaluation

[0230] After completing the model construction and preliminary training, comprehensive evaluation of the model performance is the key link to ensure its practical application effect. This application uses a variety of evaluation indicators and methods to verify the predictive ability of the model.

[0231] a) Determination of efficiency classification thresholds To meet the needs of different gene silencing efficiencies, this application sets three classification thresholds for the model: 60%, 70% and 80%. These thresholds are used to classify the prediction results as effective (1) or ineffective (0). In RNAi technology, gene silencing efficiency is an important indicator for evaluating the efficacy of siRNA drugs, usually expressed as a percentage reduction in the expression of the target gene. Selecting multiple thresholds can ensure the applicability and robustness of the model in different application scenarios.

[0232] b) Evaluation indicators Model performance is mainly evaluated through the following key indicators:

[0233] Area under the ROC curve (AUC): used to evaluate the classification ability of the model at different thresholds, reflecting the model's ability to balance correctly predicting positive and negative examples.

[0234] Area under the precision-recall curve (PRC) (AUPRC): It is particularly suitable for evaluating model performance on unbalanced datasets. It provides a more refined model evaluation by measuring the relationship between precision and recall.

[0235] c) Cross-validation This application uses the K-fold cross-validation method to verify the generalization ability of the model. In the five-fold cross-validation, the distribution of effective and ineffective siRNAs in different folds is ensured to be as uniform as possible to ensure the fairness of the evaluation. Cross-validation is not only used to evaluate the performance of the model on the training data set, but also for testing of the external validation data set to ensure the robustness of the model under different experimental conditions.

[0236] d) Model Validation After completing cross-validation, the model is further evaluated on an external validation dataset to confirm its generalization ability. The external validation dataset contains siRNA sequences that were not used in the training process, which helps to evaluate the performance of the model in practical applications. By comparing the model performance under different classification thresholds, the applicability of the model under various gene silencing efficiency standards is further verified.

[0237] e) Model optimization and adjustment Based on the evaluation results, the model may need to be optimized. For example, in the case of large data imbalance, it may be necessary to adjust the model weights or use other optimization techniques to improve the model's predictive performance.

[0238] Through this series of rigorous evaluation and validation processes, this application ensures that the developed model is highly accurate, reliable, and adaptable, providing strong support for the design and development of RNAi drugs.

[0239] After step 800, the following steps may be optionally further included:

[0240] Step 900: Biological Validation of Prediction Results

[0241] Optionally, after completing the efficiency prediction in step 800, the prediction result is verified for biological effect. This step can be performed in relevant cell models (such as HEK-293T and HepG2), and mainly includes:

[0242] a) Evaluate the inhibitory effect of siRNA on target gene expression by fluorescence quantitative PCR (qPCR) and Western blot methods;

[0243] b) Compare the differences in biological effects of siRNA molecules with different chemical modifications and screen out the optimal modification combination.

[0244] This validation step aims to confirm the actual biological effects of the model prediction results and further verify the accuracy and practicality of the model. Through this step, the correlation between the predicted results and the actual biological effects can be established, providing an important basis for subsequent drug development and optimization.

[0245] The method of the above embodiment achieves zero-sample prediction of new chemical modifications that have never been seen before, without the need to retrain the model. Through multimodal information fusion, the method can simultaneously process chemical modification structural information, positional information and RNA sequence information, improving the model's ability to understand the complex interactions between different features. The method can handle chemical modifications with different structures but similar biological effects, such as "2′-O-methyl (2′-OMe)" and "2′-deoxy-2′-fluoro (2′-F)", reflecting the high correlation between chemical structure and biological function.

[0246] In addition, the method also includes a step of interpreting the prediction results to reveal the relationship between different chemically modified structures and their biological effects. The SMILES embedding results are visualized using principal component analysis (PCA) to further verify and optimize the prediction results. RNAi drugs are optimized and designed based on the prediction results. After step 100, a data enhancement step is also included to generate new samples by transforming, combining or simulating the collected data to expand the training data set.

[0247] The method also includes a transfer learning step, which uses a model pre-trained on a related task to initialize network parameters and improve the performance of the model on the target task. During the training process, a multi-task learning strategy is used to simultaneously predict multiple relevant drug properties, such as siRNA efficiency, off-target effects, and stability, to improve the generalization ability of the model.

[0248] In order to improve the reliability of the prediction results, the method also includes the step of post-processing the prediction results, including but not limited to threshold screening, probability calibration or confidence interval estimation. In addition, the method can also be applied to other similar molecular design tasks, such as small molecule drugs, protein engineering and other fields, to achieve cross-domain knowledge transfer through appropriate adjustments and transfer learning.

[0249] The above-mentioned embodiments have the following technical effects:

[0250] Achieve high-precision zero-shot prediction of new chemical modifications: By introducing SMILES embedding technology based on a large language model, the above embodiment can effectively capture subtle differences in molecular structure and achieve accurate prediction of new chemical modifications that have never been seen. This innovation greatly improves the generalization ability of the model, enabling it to efficiently predict RNAi molecules containing new modifications without prior data, providing a strong tool support for the innovative design of RNAi drugs.

[0251] Enhanced characterization of chemical modification information: The above embodiment uses SMILES token regular expressions to tokenize molecular formulas, and achieves a more detailed understanding of molecular structures. This method significantly improves the model's ability to capture complex chemical properties, especially when dealing with modifications with high structural diversity, thereby greatly improving the accuracy and reliability of predictions.

[0252] Optimizing the integration of modification position information: By combining position embedding and self-attention mechanisms, the above embodiment successfully solves the problem that chemical modifications have different effects at different positions in the RNA sequence. This innovative design enables the model to accurately understand the role of each modification at a specific position, significantly improving the accuracy of RNAi efficiency prediction and providing more precise guidance for drug design.

[0253] Realize deep fusion of multimodal information: The above embodiment realizes deep multimodal fusion of RNA sequence, chemical modification information and their positional relationship through the cross-attention mechanism. This strategy enables the model to fully understand the complex interactions between different features, thereby achieving significant improvements in prediction performance and providing a more comprehensive and accurate evaluation basis for the design and screening of RNAi small molecules.

[0254] Improve the scalability and adaptability of the model: The model structure of the above embodiment has good scalability and adaptability. It can not only predict the currently known modifications, but also achieve seamless integration and prediction of new modifications through simple expansion. In addition, the model design allows flexible access to different large model embedding parameters, further enhancing its scalability and adaptability. This feature ensures that the model can continue to maintain its leading position in predictive ability in the context of the rapid development of RNAi technology, providing stable and reliable support for long-term drug development. The high flexibility and scalability of the model enables it to quickly adapt to new research progress and technological changes, and continuously optimize predictive performance.

[0255] Accelerate the RNAi drug development process: By providing accurate predictions of new chemical modifications, the above embodiments significantly reduce the time and resource investment required for traditional experimental screening. This efficient prediction method greatly accelerates the development of RNAi drugs, helps to push innovative treatment plans into clinical application more quickly, and ultimately brings more treatment options to patients.

[0256] In summary, the above embodiments effectively solve the key challenges faced by existing RNAi efficiency prediction models through innovative technical solutions, and provide a powerful and flexible tool for the design, development and screening of chemically modified small nucleic acid drugs, which has significant scientific value and application prospects.

[0257] In order to better understand the technical solution of this example, a specific example is used for illustration below. The details listed in this example are mainly for ease of understanding and are not intended to limit the scope of protection of this example.

[0258] This example proposes a method for predicting the efficiency of RNAi small molecules based on deep learning, especially zero-shot prediction for new chemical modifications. Through the fusion of multimodal information and innovative neural network architecture, this example can make high-precision efficiency predictions for new chemically modified siRNA molecules without prior data. The following will explain the various modules of this example in detail with the help of flow charts and block diagrams.

[0259] 1. Chemically modified RNA data collection

[0260] First, the chemical modification information of RNAi molecules is collected and organized through databases or experimental data ( Figure 4 Step ① in the above example). These data include RNA sequence, chemical modification type and location, modified SMILES molecular formula, etc. These data are converted into a standardized input format for subsequent model processing.

[0261] 2. Use large language models to embed chemical modification information

[0262] In step 2, a large language model (LLM) such as ChemBERTa (Llama and other public large models can also be used, and even have better generalization capabilities) is used to embed the SMILES molecular formulas of chemical modifications. These embedding vectors have 768 dimensions and can capture the complex chemical properties of the molecular structure. In order to enhance the generalization ability of the model for new modifications, this example uses the "SMILES token regex (Schwaller et al., 2018)" method to tokenize the SMILES molecular formula, which can understand the structural information of chemical modifications at a finer granularity than the traditional Byte-Pair Encoding (BPE) method.

[0263] 3. Model training and evaluation

[0264] like Figure 4 As shown in step 3, the processed RNA sequence and modification embedding vector are input into the model for training. The model architecture contains a self-attention mechanism and a cross-attention module to establish associations between RNA sequences and modification sites and further enhance the model's ability to identify new modification sites. During the training process, the parameters of the model are adjusted through optimization methods such as gradient descent. The division of training data sets, validation data sets, and test data sets ensures the generalization ability and performance evaluation of the model.

[0265] Parameters and formulas:

[0266] The calculation formula of the self-attention module is:

[0267] a) Query (Q), Key (K) and Value (V) matrix calculation:

[0268] Q=XW Q ,K=XW K ,V=XW V

[0269] Where W Q ,W K ,W V is a trainable weight matrix

[0270] b) Attention score calculation:

[0271]

[0272] where d k is the dimension of K, and softmax is used to normalize the weights.

[0273] The cross-attention module is used to capture the interaction between different modal information. The formula is as follows:

[0274] Given two input matrices X1 and X2:

[0275] a) Cross Query (Q), Key (K) and Value (V) matrix calculation:

[0276] a.

[0277] b.

[0278] b) Cross attention score calculation:

[0279]

[0280]

[0281] c) Splicing operation:

[0282] Finally, Z1 and Z2 are concatenated to get the final output:

[0283] Z=Contact(Z1,Z2)

[0284] in is the corresponding weight matrix.

[0285] 4. Special instructions for smiles and location embedding layers

[0286] The model can associate modification location information with RNA sequences, thereby accurately evaluating the prediction results of new samples.

[0287] Important parameters:

[0288] Embedding Dimension: 818 (768-dimensional chemical modification SMILES embedding + 50-dimensional position embedding)

[0289] Number of self-attention heads: 8

[0290] Number of cross attention heads: 4

[0291] 3D-CNN convolution kernel size: 3×3×33\times 3\times 33×3×3

[0292] Pooling layer: Maximum pooling, kernel size is 2×2×22\times 2\times

[0293] 22×2×2

[0294] By combining an 818-dimensional embedding vector, this example significantly improves the accuracy and generalization ability of the model in handling zero-shot predictions of novel chemical modifications, and can effectively cope with challenging tasks in RNAi technology.

[0295] Summary of module parameters and formulas

[0296] Embedding layer: RNA sequences are one-hot encoded and embedded into a high-dimensional space

[0297] (e.g. 818 dimensions), the vector dimension after chemical modification and position embedding is 818 dimensions;

[0298] Self-attention layer: used to capture long-range dependencies between chemical modifications and their positions;

[0299] Cross-attention layer: used to fuse chemical modifications (including position information) with RNA sequence information;

[0300] 3D-CNN convolutional layer: used to extract features and reduce dimensionality;

[0301] Fully connected layer: finally predicts siRNA efficiency.

[0302] Through the synergistic effect of these modules, this example achieves efficient prediction of RNAi small molecules, especially capable of handling zero-sample prediction problems of novel chemical modifications, filling the gap in existing technologies.

[0303] See also Figure 4 and Figure 5 .

[0304] The following example illustrates how to convert commonly used RNAi small molecule chemical modifications into numerical vector representations, which is crucial for subsequent machine learning model training.

[0305] 1. Select chemical modification

[0306] First, we selected six chemical modifications commonly used in RNAi small molecules and used SMILES strings to represent their chemical structures:

[0307] a."phosphorothioate(PS)Rp isomer":"S=P(O)(OC[C@H](O)COC1=CC=CC=C1)OC[C@H](O)COC1=CC=CC=C1",

[0308] b."phosphorothioate(PS)Sp isomer":"S=P(O)(OC[C@@H](O)COC1=CC=CC=C1)OC[C@@H](O)COC1=CC=CC=C1",

[0309] c."2′-O-methyl(2′-OMe)":"CC1=CN(C(=O)NC1=O)C2C(C(C(O2)COC)O)O",

[0310] d."glycol nucleic acid(GNA)":"OCC1OC(O)C(O)C1O",

[0311] e."5′-(E)-vinylphosphonate[5′-(E)-VP]":"O=C(C=C)P(O)(O)=O",

[0312] f."2′-deoxy-2′-fluoro(2′-F)":"CC1=CN(C(=O)NC1=O)C2C(C(C(O2)F)O)O",

[0313] }

[0314] 2. tokenize:

[0315] Next, use a regular expression to break the SMILES string into individual tokens:

[0316] import re

[0317] token_regex=r"(\[[^\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\(|\)|\.|=|#|\+|\\\\|\ / |:|@|\?|>|\\*|\$|\%[0-9]{2}|[0-9])"

[0318] def tokenize_smiles(smiles):

[0319] return re.findall(token_regex,smiles)

[0320] tokenized_smiles={name:tokenize_smiles(smiles)for name,smiles insmiles_strings.items()}

[0321] print(tokenized_smiles)

[0322] …(results omitted)

[0323] 3. Embedding

[0324] Finally, we use the pre-trained ChemBERTa model to convert the tokenized SMILES into embedding vectors:

[0325] from transformers import AutoTokenizer,AutoModel

[0326] import torch

[0327] #Load ChemBERTa model and tokenizer

[0328] tokenizer = AutoTokenizer.from_pretrained("DeepChem / ChemBERTa-77M-MLM")

[0329] model = AutoModel.from_pretrained("DeepChem / ChemBERTa-77M-MLM")

[0330] def get_embeddings(smiles):

[0331] inputs = tokenizer(smiles,return_tensors = "pt")

[0332] with torch.no_grad():

[0333] outputs = model(**inputs)

[0334] return outputs.last_hidden_state.mean(dim = 1).numpy()

[0335] #Get embeddings for all SMILES strings

[0336] embeddings = np.array([get_embeddings(smiles) for smiles in smiles_strings.values()])

[0337] Embedding results:

[0338] array([[[0.06836312,-0.15891491,-0.03059691,...,0.14920269,

[0340] -0.11842784,-0.08760355]],

[0341] [[0.06836312,-0.15891491,-0.03059691,...,0.14920269,

[0343] -0.11842784,-0.08760355]],

[0344] [[0.10209371,-0.02063187,-0.01797257,...,-0.03831936,

[0345] -0.24070218,-0.15923478]],

[0346] [[0.13249582,-0.2146938,0.07984648,...,0.24477713,

[0348] -0.3008799,-0.25935832]],

[0349] [[0.33947778,-0.1743333,0.08069328,...,0.15641546,

[0351] 0.0477776,-0.4914851]],

[0352] [[0.09244784,-0.0237443,-0.00976594,...,-0.01056609,

[0353] -0.21331328,-0.13001028]]],dtype=float32)

[0354] This step converts each chemical modification into a 768-dimensional vector representation.

[0355] in conclusion:

[0356] Through this process, we successfully converted the SMILES representation of chemical modifications into numerical vectors. These vectors capture the characteristics of chemical structures and can be directly input into machine learning models for training and prediction, laying the foundation for subsequent siRNA activity prediction.

[0357] 4. Use PCA to view the embedding results, see Figure 6 .

[0358] Summarize:

[0359] The embedding method in this example generates a high-dimensional representation for chemically modified RNA sequences by using the ChemBERTa model (more generalized large models such as LLaMA can also be used). This method first tokenizes the SMILES string and captures the complex chemical features with the powerful capabilities of the ChemBERTa pre-trained language model. To intuitively display these embedded representations, this example performs principal component analysis (PCA) on the results and visualizes them.

[0360] First, the results show that “phosphorothioate(PS)Rp isomer” and “phosphorothioate(PS)Spisomer” are almost identical in molecular formula and structure, so they almost completely overlap in PCA space. This shows that the embedding method in this example can accurately capture and reflect their chemical modifications.

[0361] On the other hand, although "2′-O-methyl (2′-OMe)" and "2′-deoxy-2′-fluoro (2′-F)" are obviously different in molecular structure, their biological effects on siRNA molecules are very similar among all chemical modifications. This is also reflected in the space after PCA dimensionality reduction - the two show close clustering in the space. This proximity not only verifies the ability of this example method to capture functional similarities, but also emphasizes the important role of this method in understanding the effects of different modifications on RNAi stability and efficiency. This result further verifies that the embedding representation method of this example can fully demonstrate the high correlation between chemical structure and biological function.

[0362] The results of this example indicate that this innovative embedding method provides a powerful tool for the design of chemically modified siRNA molecules, which can go beyond the limitations of traditional methods and deeply explore the potential laws and trends of the properties and biological functions of related chemically modified molecules.

[0363] Technical effects of the above examples:

[0364] Advantages of the above examples include:

[0365] Zero-Shot Prediction:

[0366] Limitations of existing technologies: Existing models rely on large amounts of historical data to predict chemical modifications, and therefore their predictive capabilities are severely limited when dealing with novel modifications that have never been seen before.

[0367] Advantages of the above example: By using SMILES embedding technology based on a large language model, the above example can achieve zero-shot prediction of new modifications. This means that even when faced with chemical modifications that have never appeared in the training set, the above example can still accurately predict their effects in RNAi molecules.

[0368] Enhanced integration of modification and location information:

[0369] Limitations of existing technologies: Existing models are relatively simple in integrating modification and RNA sequence position information and fail to fully capture the impact of modification position on RNAi efficiency.

[0370] Advantages of the above example: By combining position embedding and self-attention mechanism, the above example can finely capture the effect of each modification at a specific RNA sequence position. This design not only improves the accuracy of the prediction, but also enables the model to better understand the contextual information of the modification site.

[0371] Deep fusion of multimodal information:

[0372] Limitations of existing technologies: Existing models introduce cross-attention mechanisms, but they still have limitations in the fusion of multimodal information and cannot fully capture the complex interactions between sequence, modification, and position information.

[0373] Advantages of the above example: The above example deeply integrates RNA sequence and chemical modification-position information through the cross-attention mechanism of multimodal information, ensuring that the model can extract richer features from information of different modalities and improve the prediction performance.

[0374] Powerful scalability and adaptability:

[0375] Limitations of existing technologies: Existing models need to be retrained or adjusted when dealing with new modifications and lack adaptability to the ever-changing RNAi modification field.

[0376] Advantages of the above example: The model structure design of the above example has good scalability, can seamlessly integrate new chemical modifications, and predict newly modified RNAi molecules without large-scale retraining. This enables the above example to maintain its leading prediction capabilities in the face of the rapid development of RNAi technology.

[0377] In summary, the above examples not only overcome the limitations of existing technologies but also significantly improve the accuracy, generalization ability and adaptability of predictions by introducing zero-shot prediction capabilities, enhancing the integration of modification and position information, and deep fusion of multimodal information.

[0378] The second embodiment of the present application relates to a zero-sample prediction system for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning, and its structure is as follows: Figure 2 As shown in the figure, the zero-sample prediction system for chemically modified small nucleic acid drugs based on large language model and multi-perspective deep learning includes:

[0379] The data collection module is used to collect and organize chemically modified RNA data, including RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula.

[0380] The SMILES embedding module is used to tokenize the SMILES molecular formula using the SMILES token regular expression to improve the fine-grained understanding of the molecular structure. The tokenized SMILES molecular formula is embedded using a pre-trained large language model to generate a SMILES embedding vector that characterizes the chemically modified structural features.

[0381] The position embedding module is used to obtain the position information of the chemical modification in the RNA sequence, embed the position information, generate a position embedding vector, and combine the SMILES embedding vector with the position embedding vector to form a comprehensive embedding vector.

[0382] The RNA sequence embedding module is used to embed the RNA sequence and generate the RNA sequence embedding vector.

[0383] A self-attention processing module is used to process the comprehensive embedding vector using the self-attention mechanism to capture the relationship between chemical modifications and their positions.

[0384] The cross-attention fusion module is used to fuse the RNA sequence embedding vector with the comprehensive embedding vector after self-attention processing using the cross-attention mechanism to obtain multimodal fusion features.

[0385] The feature extraction module is used to extract and reduce the dimension of multimodal fusion features using a three-dimensional convolutional neural network.

[0386] The prediction module is used to input the features obtained by the feature extraction module into the fully connected layer to predict the efficiency of the chemically modified small nucleic acid drug, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecule, including the prediction of new chemical modifications that have not been seen.

[0387] The first implementation manner is a method implementation manner corresponding to the present implementation manner. The technical details in the first implementation manner can be applied to the present implementation manner, and the technical details in the present implementation manner can also be applied to the first implementation manner.

[0388] It should be noted that those skilled in the art should understand that the implementation functions of each module shown in the above-mentioned embodiment of the zero-sample prediction system for chemically modified small nucleic acid drugs based on large language models and multi-perspective deep learning can be understood by referring to the relevant description of the zero-sample prediction method for chemically modified small nucleic acid drugs based on large language models and multi-perspective deep learning. The functions of each module shown in the above-mentioned embodiment of the zero-sample prediction system for chemically modified small nucleic acid drugs based on large language models and multi-perspective deep learning can be implemented by a program (executable instruction) running on a processor, or by a specific logic circuit. If the zero-sample prediction system for chemically modified small nucleic acid drugs based on large language models and multi-perspective deep learning in the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0389] Accordingly, an embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are executed by a processor, the various method embodiments of the present application are implemented.

[0390] In addition, the embodiment of the present application also provides a zero-sample prediction system for chemically modified small nucleic acid drugs based on a large language model and multi-perspective deep learning, which includes a memory for storing computer executable instructions, and a processor; the processor is used to implement the steps in the above-mentioned method implementation when executing the computer executable instructions in the memory. Among them, the processor can be a central processing unit (Central Processing Unit, referred to as "CPU"), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as "DSP"), Application Specific Integrated Circuit (Application Specific Integrated Circuit, referred to as "ASIC"), etc. The aforementioned memory can be a read-only memory (read-only memory, referred to as "ROM"), a random access memory (random access memory, referred to as "RAM"), a flash memory (Flash), a hard disk or a solid-state hard disk, etc. The steps of the method disclosed in each embodiment of the present application can be directly embodied as a hardware processor execution, or a combination of hardware and software modules in the processor.

[0391] It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the application documents of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0392] All documents mentioned in this application are considered to be included in the disclosure of this application as a whole, so that they can be used as the basis for modification when necessary. In addition, it should be understood that after reading the above disclosure of this application, those skilled in the art can make various changes or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A zero-shot prediction method for chemically modified small nucleic acid drugs based on large language model and multi-perspective deep learning. It is characterized in that The following steps are involved: Collect and organize chemically modified RNA data, including RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula; use SMILES token regular expression to tokenize SMILES molecular formula; Use a pre-trained large language model to embed the tokenized SMILES molecular formula and generate a SMILES embedding vector that represents the chemically modified structural features. Obtaining the position information of the chemical modification in the RNA sequence, and embedding the position information to generate a position embedding vector; combining the SMILES embedding vector with the position embedding vector to form a comprehensive embedding vector; embedding the RNA sequence to generate an RNA sequence embedding vector; The comprehensive embedding vector is used to capture the relationship between the chemical modification and its position; the RNA sequence embedding vector is fused with the comprehensive embedding vector by adopting a cross attention mechanism to obtain a multimodal fusion feature; A three-dimensional convolutional neural network is used to extract and reduce the dimension of the multimodal fusion features; the features are input into a fully connected layer to predict the efficiency of chemically modified small nucleic acid drugs, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecules, including prediction of new chemical modifications that have never been seen.

2. The method according to claim 1, It is characterized in that The method is able to make zero-shot predictions for novel, unseen chemical modifications without the need to retrain the model.

3. The method according to claim 1, It is characterized in that The pre-trained large language model is specifically pre-trained on chemical structures and is selected from the following group: ChemBERTa, Llama.

4. The method according to claim 1, It is characterized in that The SMILES token regular expression can tokenize the SMILES molecular formula in units of atoms, such as decomposing "CC(=O)O" into ["C","C","(","=","O",")","O"], thereby improving the fine-grained understanding of the molecular structure.

5. The method according to claim 1, It is characterized in that In the step of obtaining the position information of the chemical modification in the RNA sequence, embedding the position information, and generating a position embedding vector, the comprehensive embedding vector formed by combining the SMILES embedding vector with the position embedding vector can simultaneously capture the structural characteristics of the chemical modification and the position information in the RNA sequence. The dimension of the comprehensive embedding vector is 818 dimensions, including 768-dimensional SMILES embedding information and 50-dimensional position embedding information.

6. The method according to claim 1, It is characterized in that The method simultaneously processes chemical modification structure information, position information and RNA sequence information through multimodal information fusion to improve the model's ability to understand the complex interactions between different features.

7. The method according to claim 1, It is characterized in that The method deals with chemical modifications with different structures but similar biological effects to reflect the high correlation between chemical structure and biological function.

8. The method according to claim 1, It is characterized in that In the step of using the comprehensive embedding vector to capture the relationship between the chemical modification and its position, the self-attention mechanism adopts a multi-head self-attention mechanism, including 8 attention heads, and uses sinusoidal position encoding or learnable position embedding to further improve the ability to capture the chemical modification position.

9. The method according to claim 1, It is characterized in that In the step of adopting a cross-attention mechanism to fuse the RNA sequence embedding vector with the comprehensive embedding vector to obtain a multimodal fusion feature, the cross-attention mechanism adopts a multi-head cross-attention mechanism including 4 attention heads.

10. The method according to claim 1, It is characterized in that In the step of inputting the features into the fully connected layer to predict the efficiency of the chemically modified small nucleic acid drug, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecule, including prediction of new chemical modifications that have not been seen, the fully connected layer uses a multi-layer perceptron with a discard method to prevent overfitting and improve the generalization ability of the model.

11. The method according to claim 1, It is characterized in that It also includes a step of interpretability analysis of the prediction results to reveal the relationship between different chemical modification structures and their biological effects.

12. The method of claim 1, It is characterized in that It also includes steps for visualizing the SMILES embedding results using principal component analysis (PCA).

13. The method of claim 1, It is characterized in that The method also includes a step of optimizing the design of RNAi drugs based on the prediction results.

14. The method of claim 1, It is characterized in that It also includes a transfer learning step, which uses a model pre-trained on a related task to initialize the network parameters to improve the performance of the model on the target task.

15. The method of claim 1, It is characterized in that A multi-task learning strategy is adopted during the training process to simultaneously predict multiple relevant drug properties, such as siRNA efficiency, off-target effects, and stability, to improve the generalization ability of the model.

16. The method of claim 1, It is characterized in that The method also includes a step of post-processing the prediction results, including but not limited to threshold screening, probability calibration or confidence interval estimation, so as to improve the reliability of the prediction results.

17. The method of claim 8, It is characterized in that The calculation of the self-attention mechanism includes: a) Calculate the Query (Q), Key (K) and Value (V) matrices: Q=XW Q ,K=XW K ,V=XW V Where W Q ,W K ,W V is a trainable weight matrix; b) Calculate the attention score: where d k is the dimension of K, and softmax is used to normalize the weights.

18. The method of claim 1, It is characterized in that The calculation of the cross-attention mechanism includes: a) For two input matrices X1 and X2, calculate the cross query, key and value matrices respectively; b) Calculate the cross-attention score; c) Concatenate the obtained Z_1 and Z_2 to get the final output Z=Concat(Z 1 ,Z 2 )。 19. A zero-shot prediction system for chemically modified small nucleic acid drugs based on large language model and multi-perspective deep learning. It is characterized in that include: Data collection module, used to collect and organize chemically modified RNA data, including RNA sequence, chemical modification type, modification position and its corresponding SMILES molecular formula; SMILES embedding module, which is used to tokenize SMILES molecular formulas using SMILES token regular expressions to improve the fine-grained understanding of molecular structures; The tokenized SMILES molecular formula is embedded using a pre-trained large language model to generate a SMILES embedding vector that represents the chemically modified structural features. A position embedding module is used to obtain the position information of the chemical modification in the RNA sequence, embed the position information, generate a position embedding vector, and combine the SMILES embedding vector with the position embedding vector to form a comprehensive embedding vector; RNA sequence embedding module, used to embed RNA sequences and generate RNA sequence embedding vectors; A self-attention processing module, which processes the comprehensive embedding vector using the self-attention mechanism to capture the relationship between chemical modifications and their positions; The cross-attention fusion module is used to fuse the RNA sequence embedding vector with the comprehensive embedding vector after self-attention processing using the cross-attention mechanism to obtain multimodal fusion features; A feature extraction module is used to extract and reduce the dimension of multimodal fusion features using a three-dimensional convolutional neural network; The prediction module is used to input the features obtained by the feature extraction module into the fully connected layer to predict the efficiency of the chemically modified small nucleic acid drug, thereby achieving zero-sample prediction of the chemically modified small nucleic acid molecule, including the prediction of new chemical modifications that have not been seen.

Images