Improved Transformer-based miRNA mutation harmfulness prediction system

Abstract

The present invention discloses a miRNA mutation harmfulness prediction system based on an improved Transformer, comprising: a miRNA dataset construction module, which integrates miRNA mutation information and classification labels from different sources to construct a miRNA mutation dataset; a miRNA feature extraction module, which extracts three types of features from the miRNA mutation dataset: energy features, interaction features, and miRNA mutation harmfulness scores from existing prediction methods; a training module, which captures mutation information and classification labels from the miRNA mutation dataset based on an improved Transformer model, learns the relationships between the three types of features, and ultimately obtains a trained improved Transformer model; and a harmfulness prediction module, which predicts the harmfulness of miRNA by introducing a GMM clustering model into the feature matrix obtained by the improved Transformer model. The present invention can effectively predict the harmfulness of miRNA mutations, providing a precise method for clinical diagnosis and precision medicine, with the advantages of high accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and biomedicine, and in particular to a miRNA mutation harmfulness prediction system based on an improved Transformer. Background Art

[0002] Currently, identifying clinically valuable mutations from massive amounts of genetic variation data has become a challenging task. Furthermore, the inherent diversity and complexity of this vast amount of genetic variation data present numerous challenges in studying the deleteriousness of genetic mutations, particularly miRNA mutations, creating a significant pain point and challenge in this field that needs to be addressed. Therefore, predicting the deleteriousness of miRNA mutations has become a pressing research topic.

[0003] As endogenous non-coding RNA, microRNA is crucial for the regulation of gene expression. In disease research, miRNA mutations can alter regulatory functions and affect cell physiology. miRNAs are sequence-conserved, tissue-specific, and temporally expressed, and can undergo a variety of mutations that may interfere with their functions. Mutations in seed regions and precursor structures are often harmful. However, miRNA mutation data are diverse and complex, scattered across different databases and literature. The information is large and fragmented, difficult to obtain, and requires integration and cleaning. In addition, datasets often have type imbalance problems, which affect model training and performance evaluation. Not all miRNA mutations are pathogenic. Many mutations have little or no functional impact, resulting in inaccurate mutation classification in clinical testing, affecting precision medicine. Accurately mining harmful miRNA mutations can provide clinical markers for precise diagnosis, guide the understanding of disease mechanisms, drug design, and personalized medication, and has important medical and social significance. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology, and proposes a miRNA mutation harmfulness prediction system based on an improved Transformer to solve the problem of insufficient feature extraction and model generalization capabilities of traditional machine learning in miRNA mutation prediction. By in-depth research on the advantages and disadvantages of various algorithm models, we finally explore a new model construction idea that can effectively solve the shortcomings of traditional machine learning algorithms. At the same time, in order to assist researchers in judging the harmfulness of miRNA mutations, realize artificial intelligence-assisted screening and diagnosis, deepen people's understanding of miRNA functions and disease mechanisms, and provide a new perspective for research in related fields, a miRNA mutation harmfulness prediction system based on the improved Transformer model will be implemented.

[0005] To achieve the above objectives, the present invention provides a technical solution: a miRNA mutation harmfulness prediction system based on an improved Transformer, comprising:

[0006] The miRNA dataset construction module is used to integrate miRNA mutation information from different sources and their classification labels of "harmful", "uncertain" and "harmless". After data cleaning, standardization and deduplication, the miRNA mutation dataset is obtained;

[0007] The miRNA feature extraction module uses various miRNA prediction tools to extract three types of features: miRNA energy features, miRNA interaction features, and miRNA mutation harmfulness scores from existing prediction methods.

[0008] The training module, based on an improved Transformer model, captures mutation information and classification labels from a miRNA mutation dataset, learns the relationships between the three types of features, and ultimately obtains a trained improved Transformer model. This improved Transformer model improves the feature processor and attention mechanism of the Transformer model. The feature processor is improved by introducing the three types of features at the input of the feature processor. The attention mechanism is improved by replacing the original single attention mechanism with a local attention mechanism and a dynamic sparse attention mechanism. The local attention mechanism captures features of adjacent positions within a local window by limiting the attention calculation, and the dynamic sparse attention mechanism performs calculations by dynamically selecting key positions.

[0009] The harmfulness prediction module predicts the harmfulness of miRNA by introducing the GMM clustering model into the feature matrix obtained by the improved Transformer model, and divides the mutations in the miRNA mutation dataset into three key categories: harmful, uncertain, and harmless.

[0010] Furthermore, the miRNA dataset construction module integrates four databases related to miRNA mutations, namely miRbase, ClinVar, dbSNP, and COSMIC, and then uses a union processing method to summarize miRNA mutation data, sequences before and after miRNA mutations, and classification labels with clinical significance to obtain a miRNA mutation dataset; the merged miRNA mutation dataset is then cleaned: first, miRNA mutation data that exceeds the normal range and has missing information are eliminated according to the statistical method Z-score, and then the classification labels are standardized and unified into "harmful", "uncertain" and "harmless", and then a deduplication operation is performed to ensure that each miRNA mutation data is unique in the miRNA mutation dataset; finally, a miRNA mutation dataset with miRNA mutation data, sequences before and after miRNA mutations and classification labels is obtained.

[0011] Furthermore, the miRNA feature extraction module performs the following operations:

[0012] Obtain miRNA energy characteristics: Use the RNAFold prediction tool to input miRNA mutation data in the miRNA mutation dataset to obtain the energy value before and after the miRNA mutation and the energy change value;

[0013] Obtain miRNA interaction features: Obtain FASTA files of the complete sequences of three target genes: lncRNA, 3'UTR, and circRNA from the LncBase database, TargetScan database, and CircBase database, respectively. Then, input these files and the sequences before and after the miRNA mutation into the miRanda prediction tool to obtain a list of lncRNA, 3'UTR, and circRNA target genes that interact with the miRNA mutation dataset. Count the number and change values ​​of the interactions between miRNA and the three target genes: lncRNA, 3'UTR, and circRNA before and after the mutation, and generate a one-hot encoding.

[0014] Obtaining the harmfulness scores of miRNA mutations from existing prediction methods: Input the miRNA mutation dataset into existing prediction tools related to miRNA mutation harmfulness prediction and extract the harmfulness scores generated by each method.

[0015] Furthermore, the training module performs the following operations:

[0016] a. Input the miRNA mutation dataset into the improved Transformer model, add position encoding to retain the position information of the sequences before and after the miRNA mutation in the miRNA mutation dataset; the miRNA mutation dataset is converted from discrete data to a continuous vector H through the embedding layer of the improved Transformer model. embed , and realize the mapping of each data through the embedding matrix of the embedding layer; then, the continuous vector H output by the embedding layer embed The features obtained by the miRNA feature extraction module are input into the feature processor to generate the token-level embedding H token ; Then, after the model layer normalization and linear transformation, the output is sent to the encoder of the improved Transformer model; Among them, the improvement of the feature processor is: introducing three types of features related to miRNA: miRNA energy feature F energy , miRNA interaction characteristics F interaction and the harmfulness score F of miRNA mutations using existing prediction methods score , which increases the diversity of input features and is conducive to generating richer token-level embeddings H token , improve the accuracy of the improved Transformer model, expressed as follows:

[0017] H token =Concat(H embed ,F energy ,F interaction ,F score )

[0018] b. The encoder receives the token-level embedding H output by the embedding layer token , and is processed by an improved attention mechanism; the encoder transforms and combines each token-level embedding multiple times to extract deeper features, and generates new feature representations through a feedforward neural network; finally, the encoder completes the processing and output of all token-level embeddings to obtain a feature matrix, in which each row of the matrix corresponds to the feature representation of each miRNA mutation in the miRNA mutation dataset; among them, the improvement of the attention mechanism is: changing the original single attention mechanism to a multi-head attention mechanism, which is conducive to utilizing the advantages of different attention mechanisms, improving the efficiency of the model when processing long sequence data, reducing computational complexity, and retaining important global information.

[0019] Furthermore, the multi-head attention mechanism consists of a local attention mechanism and a dynamic sparse attention mechanism;

[0020] The local attention mechanism defines a fixed-size window w, calculates the attention weight within the window, and after inputting the token-level embedding, only focuses on the current token i and its surrounding tokens to extract local context information; the query vector Q of the current token i i will be combined with the key vector K within the window [i-w,i+w] Calculate and generate attention weights for the surrounding tokens; the attention weights are then used to weight the value vector V within the window [i-w,i+w] , thereby obtaining the representation of the local context; the process of the local attention mechanism is expressed as follows:

[0021]

[0022] Where LocalAttention represents the local attention mechanism, d k is the dimension of the key vector;

[0023] The dynamic sparse attention mechanism uses the query matrix Q of all tokens embedded at the token level to calculate the attention weights associated with the selected key token set S; then, the corresponding key matrix K is selected based on the attention weights S Sum matrix V S , to reduce the amount of computation while retaining key information; the process of the dynamic sparse attention mechanism is expressed as follows:

[0024]

[0025] Where SparseAttention represents the dynamic sparse attention mechanism.

[0026] Furthermore, the harmfulness prediction module performs the following operations:

[0027] a. Input the feature matrix output by the encoder of the improved Transformer model into the GMM clustering model for clustering operation, wherein the parameters of the GMM clustering model are initialized, the covariance in the parameters is set to the unit matrix, and the initial value of the mixing coefficient is set to equal value;

[0028] b. Use the EM algorithm for iterative optimization: Step E calculates the eigenvector through the i′th Gaussian mixture component in the GMM clustering model to obtain the posterior probability of it being harmful or harmless; Step M calculates the GMM coefficients of X classification clusters; After completing one EM algorithm iteration, update the model parameters and enter the next iteration until the change in model parameters is less than 10 -4 ;

[0029] c. Use the optimized GMM clustering model to calculate the probability P of each miRNA mutation data in the miRNA mutation dataset belonging to the harmful class. If P>0.5, the miRNA mutation data is classified as harmful, otherwise it is classified as harmless. Traverse each miRNA mutation data in the harmful and harmless classes obtained by the GMM clustering model. If the posterior probability is greater than the preset threshold, the miRNA mutation data is considered to be acceptable in the category at the confidence level and is retained in the original category. Otherwise, it is considered that the confidence level is insufficient and is assigned to the uncertain class.

[0030] d. After completing GMM clustering and obtaining the probability P of each miRNA mutation data, calculate the miRNA mutation score S, S∈[0,1], where 0 is considered harmless and 1 is considered harmful. If the clustering category of a miRNA mutation data is harmful, then S=P; if the clustering category is harmless, then S=1-P.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] 1. This invention innovatively combines deep learning and machine learning. The Transformer model within the deep learning framework can mine connections between miRNA mutations. The introduction of a self-attention mechanism allows the model to delve deeper into connections between data layer by layer, while also improving the model's generalization capabilities.

[0033] 2. The introduction of local attention mechanism and dynamic sparse attention mechanism improves the efficiency and performance of the Transformer model when processing long sequence data.

[0034] 3. The machine learning framework uses an unsupervised clustering model (GMM clustering model) to classify miRNA mutations by receiving matrix information obtained by the encoder of the Transformer model. Unsupervised classification solves the problem of small number of sample labels.

[0035] 4. The combination of deep learning and machine learning is conducive to exploring the intrinsic connections between mutation information, alleviating the limitations brought by the small number of sample labels, and providing a new research solution for predicting the harmfulness of miRNA mutations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the relationship between the various modules of the system of the present invention.

[0037] Figure 2 Flowchart of training and prediction of the system of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to specific embodiments.

[0039] This embodiment discloses a miRNA mutation harmfulness prediction system based on an improved Transformer, which uses a large machine learning model as a basis for miRNA mutation harmfulness prediction. The relationship between the various modules of the system is as follows: Figure 1 As shown, the system training and prediction process is as follows Figure 2 It includes:

[0040] The miRNA dataset construction module is used to integrate miRNA mutation information from different sources and their classification labels of "harmful", "uncertain" and "harmless". After data cleaning, standardization and deduplication, the miRNA mutation dataset is obtained;

[0041] The miRNA feature extraction module uses various miRNA prediction tools to extract three types of features: miRNA energy features, miRNA interaction features, and miRNA mutation harmfulness scores from existing prediction methods.

[0042] The training module, based on an improved Transformer model, captures mutation information and classification labels from a miRNA mutation dataset, learns the relationships between the three types of features, and ultimately obtains a trained improved Transformer model. This improved Transformer model improves the feature processor and attention mechanism of the Transformer model. The feature processor is improved by introducing the three types of features at the input of the feature processor. The attention mechanism is improved by replacing the original single attention mechanism with a local attention mechanism and a dynamic sparse attention mechanism. The local attention mechanism captures features of adjacent positions within a local window by limiting the attention calculation, and the dynamic sparse attention mechanism performs calculations by dynamically selecting key positions.

[0043] The harmfulness prediction module predicts the harmfulness of miRNA by introducing the GMM clustering model into the feature matrix obtained by the improved Transformer model, and divides the mutations in the miRNA mutation dataset into three key categories: harmful, uncertain, and harmless.

[0044] Specifically, the miRNA dataset construction module integrates four databases related to miRNA mutations, namely miRbase, ClinVar, dbSNP, and COSMIC, and then uses a union processing method to summarize miRNA mutation data, sequences before and after miRNA mutations, and classification labels with clinical significance to obtain a miRNA mutation dataset; the merged miRNA mutation dataset is then cleaned: first, miRNA mutation data that exceeds the normal range and has missing information are eliminated according to the statistical method Z-score, and then the classification labels are standardized and unified into "harmful", "uncertain" and "harmless", and then a deduplication operation is performed to ensure that each miRNA mutation data is unique in the miRNA mutation dataset; finally, a miRNA mutation dataset with miRNA mutation data, sequences before and after miRNA mutations and classification labels is obtained.

[0045] Specifically, the miRNA feature extraction module performs the following operations:

[0046] Obtain miRNA energy characteristics: Use the RNAFold prediction tool to input miRNA mutation data in the miRNA mutation dataset to obtain the energy value before and after the miRNA mutation and the energy change value;

[0047] Obtain miRNA interaction features: Obtain FASTA files of the complete sequences of three target genes: lncRNA, 3'UTR, and circRNA from the LncBase database, TargetScan database, and CircBase database, respectively. Then, input these files and the sequences before and after the miRNA mutation into the miRanda prediction tool to obtain a list of lncRNA, 3'UTR, and circRNA target genes that interact with the miRNA mutation dataset. Count the number and change values ​​of the interactions between miRNA and the three target genes: lncRNA, 3'UTR, and circRNA before and after the mutation, and generate a one-hot encoding.

[0048] Obtaining the harmfulness scores of miRNA mutations from existing prediction methods: Input the miRNA mutation dataset into existing prediction tools related to miRNA mutation harmfulness prediction and extract the harmfulness scores generated by each method.

[0049] Specifically, the training module performs the following operations:

[0050] a. Input the miRNA mutation dataset into the improved Transformer model, add position encoding to retain the position information of the sequences before and after the miRNA mutation in the miRNA mutation dataset; the miRNA mutation dataset is converted from discrete data to a continuous vector H through the embedding layer of the improved Transformer model. embed , and realize the mapping of each data through the embedding matrix of the embedding layer; then, the continuous vector H output by the embedding layer embed The features obtained by the miRNA feature extraction module are input into the feature processor to generate the token-level embedding H token ; Then, after the model layer normalization and linear transformation, the output is sent to the encoder of the improved Transformer model; Among them, the improvement of the feature processor is: introducing three types of features related to miRNA: miRNA energy feature F energy , miRNA interaction characteristics F interaction and the harmfulness score F of miRNA mutations using existing prediction methods score , which increases the diversity of input features and is conducive to generating richer token-level embeddings H token , improve the accuracy of the improved Transformer model, expressed as follows:

[0051] H token =Concat(H embed ,F energy ,F interaction ,F score )

[0052] b. The encoder receives the token-level embedding H output by the embedding layertoken , and is processed by an improved attention mechanism; the encoder transforms and combines each token-level embedding multiple times to extract deeper features, and generates new feature representations through a feedforward neural network; finally, the encoder completes the processing and output of all token-level embeddings to obtain a feature matrix, in which each row of the matrix corresponds to the feature representation of each miRNA mutation in the miRNA mutation dataset; among them, the improvement of the attention mechanism is: changing the original single attention mechanism to a multi-head attention mechanism, which is conducive to utilizing the advantages of different attention mechanisms, improving the efficiency of the model when processing long sequence data, reducing computational complexity, and retaining important global information.

[0053] Specifically, the multi-head attention mechanism consists of a local attention mechanism and a dynamic sparse attention mechanism;

[0054] The local attention mechanism defines a fixed-size window w, calculates the attention weight within the window, and after inputting the token-level embedding, only focuses on the current token i and its surrounding tokens to extract local context information; the query vector Q of the current token i i will be combined with the key vector K within the window [i-w,i+w] Calculate and generate attention weights for the surrounding tokens; the attention weights are then used to weight the value vector V within the window [i-w,i+w] , thereby obtaining the representation of the local context; the process of the local attention mechanism is expressed as follows:

[0055]

[0056] Where LocalAttention represents the local attention mechanism, d k is the dimension of the key vector;

[0057] The dynamic sparse attention mechanism uses the query matrix Q of all tokens embedded at the token level to calculate the attention weights associated with the selected key token set S; then, the corresponding key matrix K is selected based on the attention weights S Sum matrix V S , to reduce the amount of computation while retaining key information; the process of the dynamic sparse attention mechanism is expressed as follows:

[0058]

[0059] Where SparseAttention represents the dynamic sparse attention mechanism.

[0060] Specifically, the harmfulness prediction module performs the following operations:

[0061] a. Input the feature matrix output by the encoder of the improved Transformer model into the GMM clustering model for clustering operation, wherein the parameters of the GMM clustering model are initialized, the covariance in the parameters is set to the unit matrix, and the initial value of the mixing coefficient is set to equal value;

[0062] b. Use the EM algorithm for iterative optimization: Step E calculates the eigenvector through the i′th Gaussian mixture component in the GMM clustering model to obtain the posterior probability of it being harmful or harmless; Step M calculates the GMM coefficients of X classification clusters; After completing one EM algorithm iteration, update the model parameters and enter the next iteration until the change in model parameters is less than 10 -4 ;

[0063] c. Use the optimized GMM clustering model to calculate the probability P of each miRNA mutation data in the miRNA mutation dataset belonging to the harmful class. If P>0.5, the miRNA mutation data is classified as harmful, otherwise it is classified as harmless. Traverse each miRNA mutation data in the harmful and harmless classes obtained by the GMM clustering model. If the posterior probability is greater than the preset threshold, the miRNA mutation data is considered to be acceptable in the category at the confidence level and is retained in the original category. Otherwise, it is considered that the confidence level is insufficient and is assigned to the uncertain class.

[0064] d. After completing GMM clustering and obtaining the probability P of each miRNA mutation data, calculate the miRNA mutation score S, S∈[0,1], where 0 is considered harmless and 1 is considered harmful. If the clustering category of a miRNA mutation data is harmful, then S=P; if the clustering category is harmless, then S=1-P.

[0065] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An improved Transformer-based miRNA mutation deleteriousness prediction system, characterized by: include: The miRNA dataset construction module is used to integrate miRNA mutation information from different sources and their classification labels of "harmful", "uncertain", and "harmless". After data cleaning, standardization, and deduplication, the miRNA mutation dataset is obtained; The miRNA feature extraction module uses miRNA prediction tools to extract three types of features: miRNA energy features, miRNA interaction features, and miRNA mutation harmfulness scores from the miRNA mutation dataset; The training module, based on an improved Transformer model, captures mutation information and classification labels from a miRNA mutation dataset, learns the relationships between the three types of features, and ultimately obtains a trained improved Transformer model. This improved Transformer model improves the feature processor and attention mechanism of the Transformer model. The improvement to the feature processor is to introduce the three types of features mentioned above into the input of the feature processor. The improvement to the attention mechanism is to change the original single attention mechanism into a local attention mechanism and a dynamic sparse attention mechanism. The local attention mechanism captures features of adjacent positions within a local window by limiting the attention calculation, and the dynamic sparse attention mechanism dynamically selects key positions for calculation. The harmfulness prediction module predicts the harmfulness of miRNA by introducing the GMM clustering model into the feature matrix obtained by the improved Transformer model, and divides the mutations in the miRNA mutation dataset into three key categories: harmful, uncertain, and harmless.

2. The improved Transformer-based miRNA mutation deleteriousness prediction system according to claim 1, characterized in that: The miRNA dataset construction module integrates four databases related to miRNA mutations, namely miRbase, ClinVar, dbSNP, and COSMIC, and then uses a union processing method to summarize miRNA mutation data, miRNA mutation sequences before and after miRNA mutations, and clinically significant classification labels to obtain a miRNA mutation dataset. The merged miRNA mutation dataset is then cleaned: first, miRNA mutation data that exceeds the normal range and has missing information are eliminated based on the statistical method Z-score, and then the classification labels are standardized and unified into "harmful", "uncertain" and "harmless". Then, a deduplication operation is performed to ensure that each miRNA mutation data is unique in the miRNA mutation dataset. Finally, a miRNA mutation dataset with miRNA mutation data, miRNA mutation sequences before and after miRNA mutations, and classification labels is obtained.

3. The improved Transformer-based miRNA mutation harmfulness prediction system according to claim 2, characterized in that: The miRNA feature extraction module performs the following operations: Obtain miRNA energy characteristics: Use the RNAFold prediction tool to input miRNA mutation data in the miRNA mutation dataset to obtain the energy value before and after the miRNA mutation and the energy change value; Obtain miRNA interaction features: Obtain FASTA files of the complete sequences of three target genes: lncRNA, 3'UTR, and circRNA from the LncBase database, TargetScan database, and CircBase database, respectively. Then, input these files and the sequences before and after the miRNA mutation into the miRanda prediction tool to obtain a list of lncRNA, 3'UTR, and circRNA target genes that interact with the miRNA mutation dataset. Count the number and change values ​​of the interactions between miRNA and the three target genes: lncRNA, 3'UTR, and circRNA before and after the mutation, and generate a one-hot encoding. Obtaining the harmfulness score of miRNA mutations: Input the miRNA mutation dataset into a prediction tool related to miRNA mutation harmfulness prediction to extract the harmfulness score.

4. The improved Transformer-based miRNA mutation harmfulness prediction system according to claim 3, characterized in that: The training module performs the following operations: a. Input the miRNA mutation dataset into the improved Transformer model, add position encoding to retain the position information of the sequences before and after the miRNA mutation in the miRNA mutation dataset; the miRNA mutation dataset is converted from discrete data to a continuous vector H through the embedding layer of the improved Transformer model. embed , and realize the mapping of each data through the embedding matrix of the embedding layer; then, the continuous vector H output by the embedding layer embed The features obtained by the miRNA feature extraction module are input into the feature processor to generate the token-level embedding H token ; Then, after the model layer normalization and linear transformation, the output is sent to the encoder of the improved Transformer model; Among them, the improvement of the feature processor is: introducing three types of features related to miRNA: miRNA energy feature F energy , miRNA interaction characteristics F interaction and miRNA mutation deleteriousness score F score , which increases the diversity of input features and is conducive to generating richer token-level embeddings H token , improve the accuracy of the improved Transformer model, expressed as follows: H token =Concat(H embed ,F energy ,F interaction ,F score ) b. The encoder receives the token-level embedding H output by the embedding layer token , and is processed by an improved attention mechanism; the encoder transforms and combines each token-level embedding multiple times to extract deeper features, and generates new feature representations through a feedforward neural network; finally, the encoder completes the processing and output of all token-level embeddings to obtain a feature matrix, in which each row of the matrix corresponds to the feature representation of each miRNA mutation in the miRNA mutation dataset; among them, the improvement of the attention mechanism is: changing the original single attention mechanism to a multi-head attention mechanism, which is conducive to utilizing the advantages of different attention mechanisms, improving the efficiency of the model when processing long sequence data, reducing computational complexity, and retaining important global information.

5. The improved Transformer-based miRNA mutation harmfulness prediction system according to claim 4, characterized in that: The multi-head attention mechanism consists of a local attention mechanism and a dynamic sparse attention mechanism; The local attention mechanism defines a fixed-size window w, calculates the attention weight within the window, and after inputting the token-level embedding, only focuses on the current token i and its surrounding tokens to extract local context information; the query vector Q of the current token i i will be combined with the key vector K within the window [i-w,i+w] Perform calculations to generate attention weights for surrounding tokens; The attention weights are then used to weight the value vector V within the window [i-w,i+w] , thereby obtaining the representation of the local context; the process of the local attention mechanism is expressed as follows: Where LocalAttention represents the local attention mechanism, d k is the dimension of the key vector; The dynamic sparse attention mechanism uses the query matrix Q of all tokens embedded at the token level to calculate the attention weights associated with the selected key token set S; then, the corresponding key matrix K is selected based on the attention weights S Sum matrix V S , to reduce the amount of computation while retaining key information; the process of the dynamic sparse attention mechanism is expressed as follows: Where SparseAttention represents the dynamic sparse attention mechanism.

6. The improved Transformer-based miRNA mutation harmfulness prediction system according to claim 5, characterized in that: The harmfulness prediction module performs the following operations: a. Input the feature matrix output by the encoder of the improved Transformer model into the GMM clustering model for clustering operation, wherein the parameters of the GMM clustering model are initialized, the covariance in the parameters is set to the unit matrix, and the initial value of the mixing coefficient is set to equal value; b. Use the EM algorithm for iterative optimization: Step E calculates the eigenvector through the i′th Gaussian mixture component in the GMM clustering model to obtain the posterior probability of it being harmful or harmless; Step M calculates the GMM coefficients of X classification clusters; After completing one EM algorithm iteration, update the model parameters and enter the next iteration until the change in model parameters is less than 10 -4 ; c. Use the optimized GMM clustering model to calculate the probability P of each miRNA mutation data in the miRNA mutation dataset belonging to the harmful class. If P>0.5, the miRNA mutation data is classified as harmful, otherwise it is classified as harmless. Traverse each miRNA mutation data in the harmful and harmless classes obtained by the GMM clustering model. If the posterior probability is greater than the preset threshold, the miRNA mutation data is considered to be acceptable in the category at the confidence level and is retained in the original category. Otherwise, it is considered that the confidence level is insufficient and is assigned to the uncertain class. d. After completing GMM clustering and obtaining the probability P of each miRNA mutation data, calculate the miRNA mutation score S, S∈[0,1], where 0 is considered harmless and 1 is considered harmful. If the clustering category of a miRNA mutation data is harmful, then S=P; if the clustering category is harmless, then S=1-P.

Images