A miRNA-mRNA target prediction method based on sequence statistical representation learning

Abstract

The present invention relates to a miRNA-mRNA target prediction method based on sequence statistical representation learning, and belongs to the field of bioinformatics. The structural features of miRNA and mRNA sequences are extracted in a multi-scale and multi-granular manner by combining a neural network and an attention mechanism, so that the features obtained contain both local and global multi-scale features of each base character of the sequence and the target region sequence, and fine-grained and coarse-grained multi-granular semantic information feature connections. A variational autoencoder structure is used as the overall framework, and the probability distribution of the input data is used to ensure the accuracy of the prediction, and the interpretability of the model is increased. The present invention uses a miRNA-mRNA target prediction method based on sequence statistical representation learning to effectively complete the target prediction of miRNA-mRNA.

Description

Technical Field

[0001] The invention belongs to the field of bioinformatics and relates to a miRNA-mRNA target prediction method based on sequence statistical representation learning. Background Art

[0002] The expression process of RNA has always been an important research direction in the field of bioinformatics. Among them, the prediction of miRNA target sites that play an important regulatory role in the gene expression process of mRNA is more challenging because the splicing or transcription that occurs during gene regulation is dynamic. In the past decade, RNA therapy has developed rapidly under the joint promotion of bioinformatics and deep learning. It can use functional nucleic acids to regulate the expression of pathogenic genes from the root to achieve the purpose of treating diseases. RNA therapy can be divided into three categories according to its mechanism of action: (1) mRNA therapy encoding therapeutic proteins or antigens; (2) small nucleic acid therapy targeting nucleic acids to inhibit pathogenic RNA activity or activate gene activity; (3) nucleic acid aptamer therapy targeting proteins to regulate protein activity. In particular, nucleic acid-targeted miRNA has a more significant effect than other RNA methods in the treatment of tumors, whether it is the inhibition of pathogenic genes or the expression of active therapeutic genes. A miRNA can simultaneously act on multiple target regions in the same cell signaling pathway, but the binding of miRNA to target sites in this process is dynamic and diverse. Usually, when miRNA interacts with pathogenic genes, the accuracy and efficiency of miRNA recognition of target sites and interactions on target sites are not ideal. Although the accuracy of target prediction can be improved through sophisticated instruments and manual inspection, such methods are often very costly and time-consuming. Even if the accuracy of target prediction is enhanced, the dynamic combination feature expression during miRNA targeting will greatly affect the accuracy of the experiment. If the number of candidate targeting sites is reduced, the prediction results will be incomplete and the model will have a low generalization ability. Adding a neural network structure can allow the model to improve the screening ability of dynamic combination features, but the algorithm of this type of model focuses on the classification of target features, and does not consider the mutual connection and common action mechanism between different features. And there is a lack of an interpretable learning path to guide the process of miRNA target prediction. Therefore, designing a good model to predict credible targets on miRNA-mRNA pairs is currently the focus of research in RNA therapy targeting nucleic acids. In order to alleviate the above problems without introducing new problems, the present invention proposes a prediction network of miRNA-mRNA sequence targets based on data driven and Bayesian estimation, which consists of three modules: multi-scale and multi-granularity representation learning of miRNA / mRNA sequences, miRNA-mRNA interactive learning and miRNA-mRNA target prediction.The structure of miRNA sequence statistical representation learning is the same as that of mRNA sequence statistical representation learning, which consists of convolutional neural networks and attention mechanisms. The purpose is to obtain multi-scale (local structural features and global structural features) and multi-granularity (base characters are fine-grained markers, and target sequences are coarse-grained markers) abstract features of miRNA and mRNA sequences, and to establish global and local connections between targets in each sequence. Specifically, convolutional neural networks of different depths are used to establish local connections between different targets in the sequence and extract abstract features of different target structures; attention mechanisms are used to establish global connections between different targets in the sequence and extract abstract features of different target structures. Interactive learning uses an improved Transformer network to establish interactive connections between the structures of each target in mRNA and miRNA sequences, and to enhance the features of targets with interactive relationships by combining the target affinity information of miRNA-mRNA pairs in the existing bioinformatics library. Finally, the fusion results of the miRNA / mRNA sequence statistical representation learning and miRNA-mRNA interactive learning modules are imported into the miRNA-mRNA target prediction module. Specifically, the miRNA-mRNA target prediction module is composed of the decoder module of Transformer, and its purpose is to process the fusion results to obtain the output of the affinity relationship value of each target. In particular, the entire model structure can be regarded as an improved variational autoencoder, in which the mRNA / miRNA sequence statistical representation learning and interactive learning modules constitute the encoder module of the variational autoencoder, and the miRNA-mRNA target prediction constitutes the decoder module of the variational autoencoder. Thanks to this structure, while completing the prediction of the miRNA-mRNA target interaction relationship, the model can also use the Bayesian distribution of the data to provide guidance for the prediction learning of the model, and has the probability of generating mRNA and miRNA sequences with similar structural functions to the original output. The present invention invents a biological miRNA sequence homology feature extraction and target prediction algorithm that combines data-driven and Bayesian estimation, which can reduce the prediction accuracy error caused by the dynamic changes and target diversity in the process of mRNA and miRNA action, and can extract and retain the structural information and semantic information of the interaction between mRNA and miRNA sequences, and has better accuracy and robustness for the target prediction of miRNA-mRNA pairs. Summary of the invention

[0003] In view of this, the object of the present invention is to provide a miRNA-mRNA target prediction method based on sequence statistical representation learning.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] The present invention comprises the following steps:

[0006] 1) Preprocessing of miRNA and mRNA sequence data: It is mainly composed of the classic sequence labeling algorithm in the NLP field, which is used to process and label the base positions of miRNA and miRNA sequences and the sequence regions of the miRNA-mRNA target action points. The CRF algorithm can label the single symbol position or region position specified in the sequence according to the observed sequence. In order to better explain the feasibility of the method of the present invention, the use of the CRF algorithm as a miRNA and mRNA sequence labeling algorithm is taken as an example. The specific steps of the miRNA and mRNA sequence labeling algorithm are as follows:

[0007] a) Given multiple groups of miRNA and mRNA sequences, denoted as and

[0008] b) Initialize the labeling algorithm for the sequence group, extract the position information of the bases and target regions in the miRNA and mRNA sequences, and record them as the data set and Then scan and record the number of times the target information appears repeatedly in the model input data set, which is Add a new importance tag c to the target region sequence tag in r and c tr , forming a new data tag set and The sequence labeling algorithm can construct a sequence labeling dictionary consisting of all bases and target regions in the sequence. The key of the dictionary is the sequence represented by the base or target region, and the value of the dictionary is a set of their positions in the sequence and importance labels.

[0009] c) The sequence regions represented by the importance marks greater than the threshold V are cropped and recorded as and

[0010] d) Mark the important * All target sequences in miRNA and mRNA are assembled into a matrix of dimension k×n and an o×m matrix Where n and m represent the number of miRNA and mRNA sequences, respectively; * represents miRNA or mRNA; represents a set of base markers, represents the target region sequence marker set; k and o represent the new marker data set W *p and c represent the position and importance of the marker point or region, respectively; h and g represent the number of sequences with importance marks greater than the threshold V; the threshold V is generally set to 2. In general, this algorithm can first split the sequence into smaller base characters and target region sequences, and give importance marks according to the number of repetitions of effective targets, so as to establish a more comprehensive sequence information connection.

[0011] 2) Statistical representation learning of miRNA / mRNA sequences: The structure of statistical representation learning of miRNA sequences is the same as that of mRNA sequences, which consists of a convolutional neural network extraction network and an attention mechanism structure. With the development of deep learning, there are many mature convolutional neural network models, such as LeNet5, CNN, AlexNet, etc., which are based on attention mechanism structures, such as Muti-head attention, Hierarchical attention, Attention over attention, etc. Here we take the example of statistical representation learning of miRNA sequences composed of the classic convolutional neural network CNN and the multi-head attention mechanism. The representation learning process can be summarized as the following steps:

[0012] a) Base tag set Convert to an n-dimensional matrix in one-hot encoding form Each row in the matrix represents the symbol of the base in the sequence, and each column represents the position of the 1 in the sequence.

[0013] b) and The concatenation is sent to the encoding network. When passing through the convolutional neural network, the sequence dependency is modeled using a full convolution operation. Each column of the two-dimensional matrix sequence information is associated with a local region in the original miRNA sequence from left to right. The two input matrices are convolved using a filter with a width of v to generate contextual information on each element represented by the input matrix. In order to better extract neighborhood features, the present invention also proposes a method of using stacked residual connection convolution layers to expand the receptive field size and maintain high efficiency for feature extraction. In particular, the spatial range of the established sequence region correlation can be adjusted by controlling the number of stacked convolutional layers and residual blocks. The output of this process is a feature map of size C×H×W. Where C, H, and W represent the input channel, height, and length, respectively.

[0014] c) Press F i The columns of feature channels C are trimmed, and all the same columns of channels C are merged into a vector of dimension C×W. This results in a sequence with W vectors, and the feature F i The context feature information expressed is recorded as

[0015] d) Reduce S to a 1D sequence and feed it into the multi-head attention mechanism, where each element represents a sequence token in the multi-head attention mechanism. In order to aggregate the local features of CNN and the global features of the self-attention mechanism, the present invention proposes miRNA\mRNA multi-scale representation learning. Regularization is added to the attention mechanism, and a token is randomly selected and regularized using context information S to mark all tokens. This allows the model to reduce the number of tokens at each stage while increasing its feature dimension. It helps to achieve multi-scale aggregation of different feature information extracted by the convolution mechanism and the multi-head attention mechanism. The process can be expressed by formula (1):

[0016]

[0017] Among them, F M (Q, K, V) represents the attention feature aggregation function composed of linear projection of S to subspaces Q, K and V respectively. They represent the corresponding linear projections in different dimensional spaces, h represents the number of attention modules, and i represents the number of heads in each attention module. Then the total output can be expressed as:

[0018]

[0019] Similarly, the output of this stage of miRNA representation learning is O r .

[0020] 3) Due to the dynamic characteristics of the miRNA-mRNA target action process, the target region changes in diversity. In order to obtain more target action information while maintaining prediction accuracy, the present invention proposes miRNA-mRNA interactive learning. The sequence of adjacent bases or target site regions will affect the gene expression process, and the global gene sequence expression will affect the gene regulation. In particular, target sites with similar structures or regions have a greater probability of exhibiting similar functions. In order to obtain more feature information with a small amount of bioinformatics data input, the feature information dimension is expanded by extracting sequence statistical representation information of different scales and different region sizes. The semantic information and structural feature information represented by the bases and target regions in the sequence are organically combined to enhance the interaction within and between sequences. By segmenting the sequence representation to establish a connection between different subsequences or base representations of miRNA and mRNA, the interaction mechanism of miRNA-mRNA can be better simulated. The context feature information S obtained by representation learning of miRNA and mRNA is combined with the context feature information S obtained by representation learning of miRNA and mRNA. r and S tr Dimensionality reduction to 1D sequence S r and S trThe multi-head attention mechanism is used as an example to illustrate the process. The purpose of this process is to establish a connection between the subspaces Q, K and V of miRNA and mRNA sequences, using W r and W tr Add position weights to token sequences and

[0021] This process can be expressed by formulas (2) and (3):

[0022]

[0023] Where {f(Q r ,K r ,V r ),f(Q tr ,K tr ,V tr )} are respectively represented as the attention functions of miRNA and mRNA, and the input {S mRNA ,S miRNA} linearly projected to subspace Q r , K r 、V r and Q tr , K tr 、V tr The weight coefficients are and Further aggregate all feature spaces to obtain:

[0024]

[0025] Among them, F avg is a 1×n dimensional aggregate feature matrix, i indicates the position of each variable in the total number of sequences n, d r and d tr Represent the input sequence S r and S tr Dimension.

[0026] 4) After the above steps, the results of miRNA / mRNA characterization learning can be obtained. r and O tr , the result of mRNA-miRNA interaction learning F avg The structure of mRNA-miRNA target prediction corresponds to the multi-head attention mechanism to set the decoding mechanism. Since Transformer uses the multi-head attention mechanism, the process is explained here using the Transformer decoder as an example. As mentioned above, miRNA / mRNA representation learning is an encoder structure. By learning the mean μ and variance σ of the input data, the output Or and O tr Convert to hidden variable z about miRNA / mRNA input x r and z tr .z r and z tr Merge to get the hidden representation z of miRNA-mRNA interactive learning s . Then it is sent to the decoder to obtain the target prediction probability value. This process can be represented by a vector as:

[0027] y=L(Wz s +b) (12)

[0028] Where y is the target prediction probability value, L is a linear function (such as sigmoid), W represents the weight coefficient, and b is the offset. This process obtains y by minimizing formula (6):

[0029]

[0030] in and are the recognition model and the generation model respectively; φ and θ are the parameters of the neural network model; * represents {r, tr}; {r, tr} represent miRNA and mRNA respectively; is the hidden feature variable z * Prior distribution of ; prediction model and They produce the corresponding input x * and {x r ,x tr The distribution of p(x * ) and p(y).

[0031] In particular, miRNA / mRNA representation learning can set the corresponding decoder according to its structure. Taking this structure as an example, by setting up the corresponding deconvolution layer and multi-head attention decoding structure, miRNA and mRNA sequences similar to the input structure can be obtained. Let z and x refer to the input and the hidden variables corresponding to the input, and the generated variables can be obtained by minimizing (7)

[0032]

[0033] Identification model q φ (z|x) and the generative model p θ (z|x), logp θ (x i ) is the marginal log-likelihood value. KL (q φ (z|x i )||p θ (z) is about qφ (z|x i ) and p θ The entire network is trained in an end-to-end manner, and the network weights are continuously updated by minimizing equations (6) and (7) until the model converges.

[0034] The beneficial effects of the present invention are:

[0035] The present invention is a miRNA-mRNA sequence homology feature extraction and target prediction method based on data-driven and Bayesian estimation. Because there is less information on the known miRNA / mRNA target interaction, and the combination of miRNA and target is dynamic and diverse, the traditional miRNA sequence target detection method is generally limited to the specified area of ​​the miRNA or mRNA sequence. Although the accuracy of target prediction can be improved by precise instruments and manual inspection, the time and economic costs of this type of method are very high. If the number of candidate targeting sites is reduced, the accuracy and generalization ability of the prediction results will be reduced. In addition, the existing prediction model uses a neural network as a backbone network, which has low interpretability for the prediction results. In order to solve these problems, the present invention proposes a miRNA-mRNA target prediction method based on sequence statistical representation learning, which uses a combination of a neural network and an attention mechanism to extract multi-scale and multi-granular structural features of miRNA and mRNA sequences, which can not only establish local and global feature connections between each base in the sequence and the existing target area, but also enhance the sequence feature extraction capability while keeping the parameter amount unchanged. At the same time, the miRNA / mRNA interactive learning method is adopted to establish an interactive connection between the bases and target regions of the two sequences, which can not only transfer the structural feature information of the miRNA and mRNA sequences between the two sequences, but also establish an interactive connection between the base characters, the minimum component units of the sequences, and the target region sequences between the two sequences. At the same time, the structure of the variational encoder is adopted to organically combine the miRNA / mRNA sequence statistical representation learning, miRNA-mRNA interactive learning and miRNA-mRNA target prediction module. In this way, the similarity in data distribution between the input miRNA / mRNA sequence data and the output target prediction data can be maintained. Similarly, the encoder reconstructed for the statistical representation learning of the miRNA / mRNA sequence is designed, and the generated new miRNA / mRNA data can have a similar data distribution to the input data. This process not only increases the interpretability of the prediction model, but also ensures that the model has a certain prediction accuracy while also having good generalization ability.

[0036] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 Learning structural graphs for multi-scale and multi-granular representation of miRNA / mRNA sequences;

[0039] Figure 2 It is a diagram of the interactive learning structure of miRNA-mRNA sequences;

[0040] Figure 3 This is the overall structure diagram of the network model. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0043] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0044] See also Figure 1 to Figure 3 , which is a miRNA-mRNA target prediction method based on sequence statistical representation learning.

[0045] 1. Utilize statistical characterization learning of miRNA / miNA sequences to extract local and global features of miRNA and mRNA bases and target regions in their respective sequences, and establish their local and global connections through their positional relationships;

[0046] 2. By using interactive learning of miRNA-mRNA sequences, the features of different scales and levels obtained by statistical representation learning of miRNA / mRNA sequences are integrated, and the semantic information represented by bases and target regions in the sequences is transmitted between the two sequences, from global to local, from overall structure to substructure, to further enrich the feature information of each sequence, and at the same time enhance the connection between these features and improve the accuracy of the prediction model;

[0047] 3. After obtaining multiple sets of feature expressions extracted by the above modules, the hidden vector of the model is obtained by combining the mean and variance of the miRNA and mRNA input sequence conversion parameters with the output of the miRNA-mRNA sequence interactive learning. Through the constraints of formulas (6) and (7), the prediction results of the target area can be obtained through the decoder structure of the Transformer.

[0048] 4. Combine the features learned from the statistical representation of miRNA / mRNA sequences with the mean and variance of the miRNA / mRNA input sequence after conversion parameters to obtain the hidden vectors of the respective input data. The hidden vectors are decoded and reconstructed with the opposite structure of the input module to obtain new miRNA and mRNA outputs. The entire network adopts an end-to-end training method, and the weight parameters are continuously updated until the network converges.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A miRNA-mRNA target prediction method based on sequence statistical representation learning, characterized in that: The method comprises the following steps: S1: The miRNA-mRNA target prediction method based on sequence statistical representation learning uses a combination of a convolutional neural network and a network structure based on an attention mechanism to extract multi-scale sequence structural feature information and multi-granularity sequence semantic feature information of miRNA and mRNA sequences. First, a basic convolutional neural network and a structure based on an attention mechanism are selected to construct a miRNA / mRNA representation learning network, and then the miRNA / mRNA sequence processed by the data mining algorithm is input into the network. S2: The miRNA / mRNA sequence processed by the data mining algorithm is input into the backbone network, which focuses on the local structural features and shallow semantic information of the sequence information; the structure network layer based on the attention mechanism is connected to the output of the backbone network, and the global and local multi-scale structural feature information of the sequence information, as well as the fine-grained and coarse-grained multi-granular sequence semantic information, are extracted by combining the base characters in the labeled sequence and the position information of the target sequence; the different features of the sequence information are organically merged through formula (4) to obtain the aggregate feature F avg ; S3: Interactive learning transfers the multi-scale structural information features and multi-depth semantic features extracted by miRNA and mRNA representation learning through a network based on the attention mechanism; through the positional relationship between different base characters and target region sequences in the miRNA / mRNA sequence, their feature connections are established; the depth and number of interactions are determined by the hyperparameters α and β; S4: After obtaining the extracted multi-scale structural information features and multi-depth semantic features, the hidden vector of the model is obtained by combining the mean and variance of the miRNA and mRNA input sequence conversion parameters with the output of the miRNA-mRNA sequence interactive learning; through the constraints of formula (6) and formula (7), through the decoder structure of Transformer, the prediction result of the target area is obtained; S5: The features learned by statistical representation of miRNA / mRNA sequences are combined with the mean and variance of the miRNA / mRNA input sequence after conversion parameters to obtain the hidden vectors of the respective input data; the hidden vectors are decoded and reconstructed with the opposite structure of the input module to obtain new miRNA and mRNA outputs; the entire network adopts an end-to-end training method, and the weight parameters are continuously updated until the network converges; In S1, the miRNA / mRNA sequence processed by the data mining algorithm is input into the backbone network, as follows: a) Given multiple groups of miRNA and mRNA sequences, denoted as and b) Initialize the labeling algorithm for the sequence group, extract the position information of the bases and target regions in the miRNA and mRNA sequences, and record them as the data set and Then scan and record the number of times the target information appears repeatedly in the model input data set, which is Add a new importance tag c to the target region sequence tag in r and c tr , forming a new data tag set and The sequence tagging algorithm constructs a sequence tagging dictionary consisting of all bases and target regions in the sequence. The key of the dictionary is the sequence represented by the base or target region, and the value of the dictionary is a set of their positions in the sequence and importance tags. c) The sequence regions represented by the importance marks greater than the threshold V are cropped and recorded as and d) Mark the important * All target sequences in miRNA and mRNA are assembled into a matrix of dimension k×n and an o×m matrix Where n and m represent the number of miRNA and mRNA sequences, respectively; * represents miRNA or mRNA; represents a set of base markers, represents the target region sequence marker set; k and o represent the new marker data set The number of miRNA and mRNA markers in the image; p and c represent the position and importance of the marker point or region respectively; h and g represent the importance of the marker greater than the threshold The number of sequences; threshold 2; first split the sequence into smaller base characters and target region sequences, and give importance marks according to the number of repetitions of effective targets to establish a more comprehensive sequence information connection; In S2, miRNA / mRNA sequence statistical representation learning: miRNA sequence statistical representation learning has the same structure as mRNA sequence statistical representation learning, which consists of a convolutional neural network extraction network and an attention mechanism structure, including LeNet5, CNN and AlexNet, and the structure based on the attention mechanism includes Muti-head attention, Hierarchical attention and Attention over attention; wherein, the miRNA sequence statistical representation learning process composed of the convolutional neural network CNN and the multi-head attention mechanism includes the following steps: a) Base tag set Convert to an n-dimensional matrix in one-hot encoding form Each row in the matrix represents the symbol of the base in the sequence, and each column represents the position of the base in the sequence; b) and The concatenation is sent to the encoding network; when passing through the convolutional neural network, the full convolution operation is used to model the sequence dependency; each column of the two-dimensional matrix sequence information is associated with a local area in the original miRNA sequence from left to right; the two input matrices are convolved with a filter of width v to generate contextual information on each element represented by the input matrix; in order to better extract neighborhood features, the stacked residual connection convolution layer method is used to expand the receptive field size and maintain high efficiency of feature extraction; the spatial range of the established sequence region correlation is adjusted by controlling the number of stacking convolution layers and residual blocks; the output of this process is a feature map of size C×H×W Where C, H and W represent the input channel, height and length respectively; c) Press F i The columns of the feature channel C are cut, and the same columns of all channels C are merged into a vector of dimension C×W; thereby a sequence with W vectors is obtained, and the feature F i The context feature information expressed is recorded as d) Reduce S to a 1D sequence and feed it into the multi-head attention mechanism, where each element represents a sequence token in the multi-head attention mechanism; to aggregate the local features of CNN and the global features of the self-attention mechanism, this Ayong miRNA\mRNA multi-scale representation learning; add regularization to the attention mechanism, randomly select a token and use the context information S for regularization to mark all tokens; this allows the model to reduce the number of tokens at each stage while increasing its feature dimension; help achieve multi-scale aggregation of different feature information extracted by the convolution mechanism and the multi-head attention mechanism; the process is expressed by formula (1): where F M (Q, K, V) represents the attention feature aggregation function composed of linear projection of S to subspaces Q, K and V respectively. They represent the corresponding linear projections in different dimensional spaces, h represents the number of attention modules, and i represents the number of heads in each attention module; the total output is expressed as: Similarly, the output of this stage of miRNA representation learning is O r ; In S3, different subsequences or bases of miRNA and mRNA are represented by segmenting the sequence representation to establish a connection, simulating the interaction mechanism of miRNA-mRNA; the context feature information S obtained by representation learning of miRNA and mRNA is r and S tr Dimensionality reduction to 1D sequence S r and S tr Feed into the attention mechanism; establish connections between the subspaces Q, K and V of miRNA and mRNA sequences, and use and Add position weight W to the token sequence t p and It is expressed by formulas (2) and (3): Where {f(Q r ,K r ,V r ),f(Q tr ,K tr ,V tr )} are respectively represented as the attention functions of miRNA and mRNA, and the input {S mRNA ,S miRNA } linearly projected to subspace Q r , K r 、V r and Q tr , K tr 、V tr The weight coefficients are and Aggregate all feature spaces to get: where F avg is a 1×n dimensional aggregate feature matrix, i indicates the position of each variable in the total number of sequences n, d r and d tr Represent the input sequence S r and S tr Dimensions; In S4, the result of miRNA / mRNA characterization learning is obtained. r and O tr , the result of mRNA-miRNA interaction learning F avg ; The structure of mRNA-miRNA target prediction corresponds to the multi-head attention mechanism to set the decoding mechanism; The process is explained by taking the Transformer decoder as an example; It is mentioned that miRNA / mRNA representation learning is an encoder structure that learns the mean μ and variance σ of the input data to output O r and O tr Convert to hidden variable z about miRNA / mRNA input x r and z tr ; z r and z tr Merge to get the hidden representation z of miRNA-mRNA interactive learning s ; Then it is sent to the decoder to obtain the target prediction probability value. This process is represented by the vector: y=L(Wz s +b) (5) Where y is the target prediction probability value, L is a linear function, sigmoid, W is the weight coefficient, and b is the offset; this process obtains y by minimizing formula (6): in and are the recognition model and the generation model respectively; φ and θ are the parameters of the neural network model; * represents {r, tr}; {r, tr} represent miRNA and mRNA respectively; is the hidden feature variable z * Prior distribution of ; prediction model and Generates the corresponding input x * and {x r ,x tr The distribution of p(x * ) and p(y); The miRNA / mRNA representation learning sets the corresponding decoder according to its structure; establishes the corresponding deconvolution layer and multi-head attention decoding structure to obtain miRNA and mRNA sequences similar to the input structure; z and x represent the input and the hidden variables corresponding to the input, and the generated variables are obtained by minimizing formula (7) Identification model q φ (z|x) and the generative model p θ (z|x), logp θ (x i ) is the marginal log-likelihood value; where D KL (q φ (z|x i )||p θ (z) is about q φ (z|x i ) and p θ (z); The entire network adopts an end-to-end training method, and the network weights are continuously updated by minimizing equations (6) and (7) until the model converges.

Images