Gene data mining method based on Transformer

Abstract

The present invention belongs to the field of biometric identification technology, and relates to a gene data mining method based on Transformer, which loads gene expression data from a file and performs preprocessing; then the data is input into a designed model for training and evaluation; the designed model includes a CNN convolutional neural network and an improved Transformer model; the improved Transformer model includes an encoder and a classifier; the encoder adopts a multi-head self-attention mechanism, a convolutional self-attention mechanism and a feedforward neural network; then the trained model is used to process the data, and in the process of data processing, the fused attention weight obtained by the encoding layer is stored; finally, the fused attention weights of all batches are averaged, and the comprehensive attention weight of each gene at different time points is calculated, and the most important genes are identified by calculating and analyzing the attention score of each gene. This method can solve the problems of scarcity and imbalance of gene data, reduce the risk of overfitting, and improve the accuracy of gene data analysis.

Description

Technical Field

[0001] The present invention belongs to the field of biometric identification technology, and relates to the fields of bioinformatics and artificial intelligence, and in particular to a gene data mining method based on deep learning. Background Art

[0002] Bioinformatics data mining technology is an interdisciplinary method that combines biology, computer science, statistics and information technology. It is used to extract valuable information from large amounts of biological data, such as identifying rice blast pathogenic genes and screening disease-resistant genes.

[0003] Existing deep learning-based genetic data mining methods mainly rely on several popular model architectures, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs) and their variants, long short-term memory networks (LSTMs) and gated recurrent units (GRUs), and the recently widely adopted Transformer model. These models have their own advantages and disadvantages when processing genomic data. CNNs are good at capturing local features and can effectively extract meaningful patterns from gene sequences, such as regulatory elements such as promoters and enhancers. However, CNNs may not handle long-range dependencies well, which is an important limitation for understanding complex gene regulatory networks. RNNs and their variants (such as LSTMs and GRUs) can handle temporal or sequential information in sequence data and are very suitable for modeling time series data of gene expression. However, RNNs may encounter problems with gradient vanishing or gradient exploding when processing very long sequences, and have high computational costs and long training times. Compared with traditional RNNs, Transformers can capture long-range dependencies more effectively through self-attention mechanisms, and have a higher degree of parallelization, which gives them significant advantages in processing large-scale genomic data. Nevertheless, the original Transformer model was designed for natural language processing tasks and still has some inadaptability to biological sequence data, such as the lack of integration of biological background knowledge; in addition, the encoder part of the traditional Transformer is mainly used for classification tasks because it can convert the input sequence into a fixed-length vector representation, but cannot realize the mining of key genes. Summary of the invention

[0004] In view of the above problems and defects, the present invention proposes a Transformer-based genetic data mining method for mining key pathogenic genes in the process of rice being infected by rice blast. The method uses an improved Transformer to classify genes into two categories, and at the same time mines key genes to obtain possible pathogenic genes. The improved Transformer optimizes the encoder part of the Transformer to better adapt to the characteristics of biological sequence data; in addition, in response to the high dimensionality and sparsity of genetic data, a special data preprocessing and dimensionality reduction technology is developed to solve the scarcity and imbalance of genetic data, ensure that the model can reduce the risk of overfitting while maintaining high performance, and improve the accuracy of genetic data analysis.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A gene data mining method based on Transformer, the method comprising the following steps:

[0007] Step 1. Data processing

[0008] Load gene expression data from files and perform preprocessing operations, including normalization, label generation, and dataset partitioning;

[0009] Step 2. Construct a data mining model for key pathogenic genes of rice blast

[0010] The data mining model of key pathogenic genes of rice blast is used to extract local features and capture sequence dependencies to achieve the binary classification task of rice gene expression data, including CNN convolutional neural network, embedding layer and improved Transformer model; CNN convolutional neural network contains two layers of one-dimensional convolutional layers, and the activation function used after each convolution operation is ReLU, which is used to extract local features of the differential expression matrix, while capturing the spatial patterns between adjacent time points and identifying the local correlation between gene expression values ​​at different time points; the embedding layer is used to map the CNN output features to the Transformer input dimension; the improved Transformer model includes an encoder and a classifier; the encoder adopts a multi-head self-attention mechanism, a convolutional self-attention mechanism and a feedforward neural network, which consists of 6 encoding layers, each of which is used to process Sequence data and capture long-distance dependencies, and output fused attention weights at the same time; the output of the previous encoding layer after residual connection and normalization is used as the input of the current encoding layer; the encoding layer includes a self-attention layer, a normalization layer, a convolutional self-attention layer, a normalization layer, a feedforward neural network and a normalization layer arranged in sequence, the self-attention layer is used to capture the long-distance dependencies in the gene expression data, obtain the self-attention weight of each gene, and analyze the importance of the gene; the normalization layer performs residual connection and normalization operations to ensure that the input of each layer has the same dimension; the convolutional self-attention layer is used to capture local patterns and short-distance dependencies, calculate the convolutional attention weights and fuse the attention mechanism, the feedforward neural network consists of two linear transformation layers and a ReLu activation function sandwiched between the two linear transformation layers, and the output of the feedforward neural network is subjected to residual connection and normalization operations;

[0011] The classifier uses a fully connected layer to perform classification, maps the encoder output to the target space of the classification task, and uses the Softmax function to calculate the probability distribution of each category and output the classification result;

[0012] Step 3. Initialize the data mining model of key pathogenic genes of rice blast, train and evaluate the model, and then use the trained data mining model of key pathogenic genes of rice blast to classify rice gene data. During the data processing process, the data mining model of key pathogenic genes of rice blast will generate a fused attention weight for each gene at each time step in each encoding layer, and store the obtained fused attention weight;

[0013] Step 4. Average the fused attention weights of all batches, calculate the comprehensive attention weight of each gene at different time points, and identify the most important genes by calculating and analyzing the attention score of each gene;

[0014]

[0015] in, attn_weights_avg The average attention weight of each gene at each time step ， For the i The layer encoding layer obtains the fused attention weight of each gene at each time step;

[0016]

[0017] in, is the comprehensive attention weight of each gene at n time steps, For the i Time step weight ， The larger the time step, the greater the weight;

[0018]

[0019] in, The final attention weight for each gene ， is a sample before or after infection, i=1 is a sample before infection, and i=2 is a sample after infection;

[0020] Then, the genes are sorted according to their final attention weights, and the top N most important genes are output as the key pathogenic genes in the rice blast infection process and their corresponding scores.

[0021] As a preferred embodiment of the present invention, the step 1 comprises the following steps:

[0022] Step 1.1 Data loading and preprocessing

[0023] Gene expression data were read from the file, converted into gene count data using the RNA-SEQ tool, and then the differentially expressed gene matrix was obtained through differential analysis. The gene expression values ​​of the differentially expressed gene matrix were then standardized using the StandardScaler software so that the mean of each feature was 0 and the standard deviation was 1.

[0024] Step 1.2 Create labels and adjust data format

[0025] The data is divided into an uninfected group and an infected group according to the sample content. The uninfected group is 0 and the infected group is 1. The corresponding binary classification label file is generated.

[0026] Step 1.3: Divide the training set and test set

[0027] The expression values ​​of each gene at different time points were reshaped into a feature matrix suitable for model input, and the data were divided into training and test sets.

[0028] As a preferred embodiment of the present invention, the convolution operation of the CNN convolutional neural network is expressed as: ;

[0029] in, is the input tensor, is the convolution kernel, is the bias term, * represents the convolution operation, is the activation function (such as ReLU); Represents the output after the convolution operation, the output data is (batch_size, 3, 64);

[0030] The data output by the CNN convolutional neural network is used as the input of the embedding layer, which is linearly mapped to a high-dimensional space and then submitted to the encoder layer. The embedding layer is represented as a linear transformation: ;

[0031] in, is the weight matrix of the embedding layer; is the bias term of the embedding layer; is the output after embedding; the output data of the embedding layer is (batch_size, 3, 256);

[0032] The multi-head self-attention mechanism is expressed as:

[0033] ;

[0034] ;

[0035] Among them, Q, K, V are query, key and value matrices respectively; It is the scaled dot product attention mechanism; is a linear matrix used to transform multiple self-attention head outputs into a projection matrix of the original shape. Representative i Head self-attention layer;

[0036] The formula for calculating the self-attention weight of each time step in the self-attention layer is:

[0037] ;

[0038] Among them, Q, K, V are query, key and value matrices respectively; is the dimension of the key; is the self-attention weight matrix; T represents the transpose of the matrix;

[0039] The calculation formula for fusion attention weight is:

[0040] A fused =w1×A csa +w2×A mu ;

[0041] Among them, A fused To integrate the attention weight, A csa is the convolution attention weight, A mu is the multi-head self-attention weight, w1 and w2 are weight coefficients, w1+w2=1.

[0042] As a preferred embodiment of the present invention, the expression of the classifier is:

[0043] ;

[0044] Among them, A fused is the output of the encoding layer; is the weight matrix of the classifier; is the bias term of the classifier; P is the probability distribution of the classification results.

[0045] As a preferred embodiment of the present invention, when training the model, a cross entropy loss function is used, and a category weight method is combined to deal with the category imbalance problem; an Adam optimizer is used, a learning rate is set, and a cyclic training is started; forward propagation: the training data is input into the model, the predicted value is calculated, and the loss is calculated based on the predicted value and the true label; back propagation is used to calculate the gradient and update the model parameters.

[0046] Advantages and beneficial effects of the present invention:

[0047] 1. Traditional deep learning models (such as pure CNN or RNN) have limitations when processing sequence data. CNN is good at extracting local features, but it is difficult to capture long-distance dependencies. Transformer can effectively capture global dependencies through the multi-head self-attention mechanism. By combining CNN and Transformer, local patterns and long-distance dependencies can be extracted from gene expression data at the same time, thereby improving classification accuracy and realizing the mining of key genes, improving the accuracy of gene data analysis.

[0048] 2. Gene expression data usually have a small sample size, especially for certain specific infection states; Transformer's multi-head self-attention mechanism can help the model better capture complex relationships in a small number of samples, solve the problems of scarcity and imbalance of genetic data, and avoid overfitting.

[0049] 3. Gene expression data is usually time series data, and expression values ​​at different time points may have different contributions to the classification task; the present invention extracts local features of each time step through CNN, and captures the dependencies between time steps through Transformer, so that the model can utilize time information more effectively.

[0050] 4. The improved Transformer model of the present invention integrates the convolutional self-attention mechanism (CSA) and the multi-head self-attention mechanism (MHSA). The convolutional self-attention mechanism (CSA) is good at capturing local patterns and short-distance dependencies; the multi-head self-attention mechanism (MHSA) can model long-distance dependencies and capture the interactions between any positions in the input sequence. After the fusion of the two, the model can process information at different scales, retaining local details and capturing global structures to obtain richer representations; this makes the model more flexible and powerful when dealing with complex, multi-scale tasks, and can better understand complex patterns in the data; in addition, by introducing a learnable weight vector, the model can dynamically adjust the contribution ratio of convolutional self-attention and multi-head self-attention according to the characteristics of the input data, which enables the model to automatically select the most suitable attention mechanism under different tasks or different input conditions, thereby enhancing its adaptability and generalization ability.

[0051] 5. The present invention uses compute_class_weight to calculate the class weights and applies them to the cross entropy loss function; this helps the model pay more attention to the minority class (infected group) during training, thereby improving the prediction performance of the minority class.

[0052] 6. The present invention uses a linear embedding layer to map the features extracted by CNN to the input dimension of Transformer. This combination enables the model to capture complex nonlinear relationships in low-dimensional space and enhances the model's representation learning ability.

[0053] 7. The present invention uses a multi-head self-attention mechanism, and the model can generate self-attention weights for each time step. These weights reflect the degree of attention of the model to gene expression at different time points. These attention weights can be used to analyze which time points or which genes contribute most to the classification results, thereby providing interpretability; in addition, the self-attention mechanism allows the model to dynamically adjust the degree of attention to different time points. Even if the data at some time points is noisy or incomplete, the model can still ignore these unimportant parts by adjusting the attention weights and focus on more meaningful time points.

[0054] 8. The present invention proposes to use CNN convolutional neural network to extract sample features with time series relationship, and the filters in the convolution operation share the same weight on the entire input data (all convolution kernels in the same layer use the same weight on the entire input data). This means that the same filter can detect the same type of features at different positions, reducing the number of parameters of the model, thereby improving the generalization ability of the model and reducing the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easy to understand. In the accompanying drawings:

[0056] Figure 1 This is a flow chart of the gene data mining method based on Transformer of the present invention;

[0057] Figure 2 The improved Transformer model architecture diagram provided by the present invention;

[0058] Figure 3 Evaluate the indicator change curve for the training process;

[0059] Figure 4 The fluctuation trend of the top ten selected genes in the source data. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the technical solution and advantages of the present invention, the present application is described in detail below in conjunction with the accompanying drawings, but it is not intended to limit the protection scope of the present invention.

[0061] like Figure 1 , Figure 2 As shown, this embodiment provides a gene data mining method based on Transformer, which includes the following steps:

[0062] Step 1. Data processing

[0063] Load gene expression data from files and perform preprocessing operations, including normalization, label generation, and dataset partitioning;

[0064] Specifically, in this embodiment, step 1 includes the following steps:

[0065] Step 1.1. Data loading and preprocessing

[0066] Read gene expression data from the file. The data is a two-dimensional array or tensor. Skip the first row and set the column names. Each row represents a sample (gene in this example), and each column represents a feature, that is, the gene expression value at different time points. Separate the gene name and expression value. The gene name is used as an identifier, and the expression value is used as the input feature of the model.

[0067] In this embodiment, the original SRA data is first obtained through the illumina sequencing platform, the data is converted into gene count data through the RNA-SEQ tool, and then the differential expression gene matrix is ​​obtained through differential analysis; then the gene expression values ​​of the differential expression gene matrix are standardized using the StandardScaler software so that the mean of each feature is 0 and the standard deviation is 1. This is a very important step because the expression levels of different genes may vary greatly. Standardization can ensure that the expression value of each gene is on the same scale, thereby avoiding excessive influence of certain features on the model, which helps to accelerate model convergence and improve performance, while improving the stability of model training; in addition, the gene data used are the infected group and the uninfected group data at 3 time steps, so the first 3 columns are set as the uninfected group and the last 3 columns are set as the infected group.

[0068] Specifically, the data used in this embodiment comes from the original SRA data obtained from the PRJNA1342239 biological sequencing project, which provides the original sequencing data of Japanese rice and Pi9 transgenic Japanese rice under fungal infection, cold and heat stress, and salt stress. The present invention mainly studies the infection process of rice blast fungus, so the original SRA data of rice infected with rice blast fungus for 24, 48, 72 hours and uninfected rice at the same time are selected. Then, the SRA data was converted into a fastq format by RNA-SEQ tools and quality control was performed. After the quality control was completed, the hisat2 software was used to establish an index and compare it with the index file. Finally, the featureCounts software was used for comparison, and the results were technically statistically analyzed to obtain an initial gene count file. After irrelevant entries were removed, the sequencing count data was obtained. The first column of the data was the gene name, and the following six columns represented six samples, which were rice gene data for 24, 48, and 72 hours without infection and rice gene data for 24, 48, and 72 hours after infection. Due to too much irrelevant data, gene difference analysis was performed, and the DESeq2 software was used for difference analysis. Differential genes were extracted for data with Log2FoldChange>2 (4 times the difference) and p.adj<0.01. Generally, log2FoldChange>4 and p.adj<0.05 were selected. However, due to too many genes, a more stringent standard was selected in the present invention.

[0069] Step 1.2. Create labels and adjust data format

[0070] According to the sample content, the data is divided into an uninfected group and an infected group (the first 3 columns are marked as uninfected, and the last 3 columns are marked as infected). The uninfected group is 0, and the infected group is 1, and the corresponding binary classification label file is generated;

[0071] In this example, compute_class_weight is used to calculate class weights to address possible class imbalance (for example, the number of samples in the uninfected group and the infected group are not equal), which is very important to ensure that the model does not favor the majority class during training;

[0072] Step 1.3. Divide the training set and test set

[0073] The expression values ​​of each gene at different time points were reshaped into a format suitable for model input (feature matrix); and the data were divided into training set and test set with a ratio of 4:1, that is, 80% training set and 20% test set.

[0074] The data in this example has N genes, and each gene has expression values ​​at 6 time points (3 uninfected and 3 infected), so the shape of the input data is (N, 6). However, the model expects the input shape to be (batch_size, seq_len, num_features); where batch_size represents the number of samples used in each gradient descent update, seq_len is the number of time steps (3 time points), also known as the length of the time series; num_features is the number of features at each time point (num_features is 2 in this example, because each gene has two expression values ​​at each time point, namely uninfected and infected); therefore, the input data needs to be reshaped into a feature matrix of (batch_size, 3, 2).

[0075] Step 2. Construct a data mining model for key pathogenic genes of rice blast

[0076] The data mining model for key pathogenic genes of rice blast is used to extract local features and capture sequence dependencies, and ultimately achieve the binary classification task of rice gene expression data (distinguishing whether a sample is infected or not, and predicting the state of each sample by analyzing the gene expression values ​​at different time points). It includes CNN convolutional neural network, embedding layer and improved Transformer model; among them, CNN convolutional neural network contains two layers of one-dimensional convolutional layers, and the activation function used after each convolution operation is ReLU, which is used to extract local features of the differential expression matrix, and at the same time capture the spatial patterns between adjacent time points, and identify the local correlation between gene expression values ​​at different time points, such as expression changes at adjacent time points. Trend; The embedding layer is used to map the CNN output features to the Transformer input dimension in order to better capture complex relationships; The improved Transformer model includes an encoder and a classifier; The encoder adopts a multi-head self-attention mechanism, a convolutional self-attention mechanism, and a feedforward neural network. It consists of 6 encoding layers, each of which is used to process sequence data and capture long-distance dependencies. At the same time, it outputs fused attention weights. The output of the previous encoding layer after residual connection and normalization is used as the input of the current encoding layer. Each position in the input sequence will be converted into a query, key, and value vector, which are used to calculate the attention weight;

[0077] Specifically, the encoding layer includes a self-attention layer (Multi-Head Self-Attention), a normalization layer (Add&Norm), a convolutional self-attention layer (Convolutional Attention), a normalization layer (Add&Norm), a feedforward neural network (Feed Forward) and a normalization layer (Add&Norm) arranged in sequence. The self-attention layer is used to capture the long-distance dependencies in the gene expression data, obtain the self-attention weight of each gene, and analyze the importance of the gene; the normalization layer performs residual connection and normalization operations to ensure that the input of each layer has the same dimension; the convolutional self-attention layer is used to capture local patterns and short-distance dependencies, calculate the convolutional attention weights and fuse the attention mechanism. The feedforward neural network consists of two linear transformation layers and a ReLu activation function sandwiched between the two linear transformation layers. The output of the feedforward neural network performs residual connection and normalization operations;

[0078] The classifier uses a fully connected layer (Linear) for classification, maps the encoder output to the target space of the classification task, and uses the Softmax function to calculate the probability distribution of each category and output the classification result;

[0079] Specifically, the convolution operation of the CNN convolutional neural network is expressed as: ;

[0080] in, is the input tensor, is the convolution kernel, is the bias term, * represents the convolution operation, is the activation function (ReLU is used in this embodiment); Represents the output after the convolution operation, the output data is (batch_size, 3, 64);

[0081] The data output by the CNN convolutional neural network is used as the input of the embedding layer, which is linearly mapped to a high-dimensional space and then submitted to the encoder layer. The embedding layer is represented as a linear transformation: ;

[0082] in, is the output of the CNN convolutional neural network (after convolution operation); is the weight matrix of the embedding layer; is the bias term of the embedding layer; is the output after embedding, and the output data of the embedding layer is (batch_size, 3, 256);

[0083] The multi-head self-attention mechanism is expressed as:

[0084] ;

[0085] ;

[0086] Among them, Q, K, V are query, key and value matrices respectively; It is the scaled dot product attention mechanism; is a linear matrix used to transform multiple self-attention head outputs into a projection matrix of the original shape. Representative i Head self-attention layer;

[0087] The formula for calculating the self-attention weight of each time step in the self-attention layer is:

[0088] ;

[0089] Among them, Q, K, V are query, key and value matrices respectively; is the dimension of the key; is the self-attention weight matrix; T represents the transpose of the matrix;

[0090] The calculation formula for fusion attention weight is:

[0091] A fused =w1×A csa +w2×Amu ;

[0092] Among them, A fused To integrate the attention weight, A csa is the convolution attention weight, A mu is the multi-head self-attention weight, w1 and w2 are weight coefficients, w1+w2=1.

[0093] The expression of the classifier is:

[0094] ;

[0095] Among them, A fused is the output of the encoding layer; is the weight matrix of the classifier; is the bias term of the classifier; P is the probability distribution of the classification results.

[0096] Step 3. Initialize the data mining model of key pathogenic genes of rice blast, train and evaluate the model, and then use the trained data mining model of key pathogenic genes of rice blast to classify rice gene data. During the data processing process, the data mining model of key pathogenic genes of rice blast will generate a fused attention weight for each gene at each time step in each encoding layer, and store the obtained fused attention weight;

[0097] Specifically, according to the input model instance, category weight, batch size, number of training rounds, early stopping method (patience = 5, that is, if the performance of the validation set does not improve for 5 consecutive rounds, stop training), Dropout (dropout = 0.5), batch normalization, initialize the loss function and optimizer; within the specified number of training rounds, perform the following steps:

[0098] Input the training data into the model, perform forward propagation, calculate the predicted value, and calculate the loss based on the predicted value and the true label. Backward propagation calculates the gradient and updates the model parameters. After each round of training, record the current training loss, and use the test set to evaluate the performance of the model, calculate the accuracy on the test set, and record it. After the training is completed, draw the curve of the change of training loss and test accuracy to help analyze the convergence and performance of the model.

[0099] Step 4. Average the fused attention weights of all batches, calculate the comprehensive attention weight of each gene at different time points, and identify the most important genes by calculating and analyzing the attention score of each gene;

[0100]

[0101] in , attn_weights_avgThe average attention weight of each gene at each time step ， For the i The layer encoding layer obtains the fused attention weight of each gene at each time step;

[0102]

[0103] in, is the comprehensive attention weight of each gene at n time steps, For the i Time step weight ， The larger the time step, the greater the weight;

[0104]

[0105] in, The final attention weight for each gene ， is a sample before or after infection, i=1 is a sample before infection, and i=2 is a sample after infection;

[0106] Then, the genes are sorted according to their final attention weights, and the top N most important genes are output as the key pathogenic genes in the rice blast infection process and their corresponding scores.

[0107] Furthermore, in this embodiment, the cross entropy loss function is used in combination with the category weight method to deal with the category imbalance problem; the Adam optimizer is used, an appropriate learning rate (eg, 1e-3) is set, and then the cyclic training is started.

[0108] );

[0109] in, is the predicted probability of the model, that is, the probability that the model predicts that the sample belongs to the positive sample (infected sample); y is the true label, which takes a value of 0 or 1.

[0110] When evaluating the model during training, this embodiment records the loss, precision, recall, and F1 score during the training process, and draws a training curve. These indicators can help us understand the performance of the model on unseen data. Figure 3 is the loss change curve of the training process, Figure 3 It can be seen that the training loss (Train Loss) and test loss (Test Loss) drop rapidly in the early stage, indicating that the model is learning quickly. As the training progresses, the loss gradually stabilizes (steadily decreases and tends to a lower value), which proves that the model remains at a high level, has high accuracy, and good generalization ability.

[0111] The present invention uses the above method to mine the top ten genes. Figure 4 The fluctuation trend of the top ten selected genes in the source data is shown. Figure 4 The trends of the top ten selected genes in the source data show the following observations:

[0112] 1. **Uninfected group (Moc24, Moc48, Moc72)**:

[0113] The expression levels of most genes in the uninfected group were relatively stable or slightly fluctuated. For example, the expression levels of GeneOs01g0546800 and Gene Os04g0664500 in the uninfected group did not change much, showing a relatively stable trend.

[0114] 2. **Infection group (Xoc24, Xoc48, Xoc72)**:

[0115] The expression levels of most genes in the infected group showed significant changes. Specifically:

[0116] The expression level of Gene Os07g0539100 was significantly increased in the infected group, but remained at a low level in the uninfected group.

[0117] The expression level of Gene Os01g0316600 in the infected group first increased and then decreased, showing obvious fluctuations.

[0118] The expression level of Gene Os06g0589600 also showed an increasing trend in the infected group, but declined in the later stage.

[0119] Specific analysis:

[0120] 1. **Gene Os01g0546800**:

[0121] Uninfected group: The expression level gradually increased, but the increase was small.

[0122] Infected group: The expression level also gradually increased, but the increase was significantly greater than that of the uninfected group.

[0123] 2. **Gene Os04g0664500**:

[0124] Uninfected group: The expression level remained basically unchanged, showing a stable trend.

[0125] Infected group: Expression levels increased significantly, especially at later time points (Xoc48 and Xoc72).

[0126] 3. **Gene Os06g0589600**:

[0127] Uninfected group: The expression level was relatively stable with slight fluctuations.

[0128] Infected group: Expression levels increased significantly, especially at the mid-term time point (Xoc48), and then decreased at the late stage.

[0129] 4. **Gene Os07g0539100**:

[0130] Uninfected group: The expression level was low and relatively stable.

[0131] Infected group: The expression level increased significantly, especially at the mid-term time point (Xoc48), showing a strong response.

[0132] 5. **Gene Os01g0316600**:

[0133] Uninfected group: The expression level was low and relatively stable.

[0134] Infection group: The expression level first increased and then decreased, showing an obvious fluctuation trend.

[0135] These graphs clearly show the dynamic changes of the selected genes during rice infection. The expression levels of most genes in the uninfected group were relatively stable, while they showed a significant change trend in the infected group. This fluctuation trend shows that the genes selected by the model do play a key role in the rice infection process. The expression changes of these genes may reflect the response mechanism of rice to pathogen infection, providing important clues for subsequent biological research.

[0136] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art of the present invention, according to the idea of ​​the present invention, some simple deductions, deformations or substitutions can be made. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A gene data mining method based on Transformer, characterized in that: The method comprises the following steps: Step 1. Data processing Load gene expression data from files and perform preprocessing operations, including normalization, label generation, and dataset partitioning; Step 2. Construct a data mining model for key pathogenic genes of rice blast The data mining model of key pathogenic genes of rice blast is used to extract local features and capture sequence dependencies to achieve the binary classification task of rice gene expression data, including CNN convolutional neural network, embedding layer and improved Transformer model; CNN convolutional neural network contains two layers of one-dimensional convolutional layers, and the activation function used after each convolution operation is ReLU, which is used to extract local features of the differential expression matrix, while capturing the spatial patterns between adjacent time points and identifying the local correlation between gene expression values ​​at different time points; the embedding layer is used to map the CNN output features to the Transformer input dimension; the improved Transformer model includes an encoder and a classifier; the encoder adopts a multi-head self-attention mechanism, a convolutional self-attention mechanism and a feedforward neural network, which consists of 6 encoding layers, each of which is used to process Sequence data and capture long-distance dependencies, and output fused attention weights at the same time; the output of the previous encoding layer after residual connection and normalization is used as the input of the current encoding layer; the encoding layer includes a self-attention layer, a normalization layer, a convolutional self-attention layer, a normalization layer, a feedforward neural network and a normalization layer arranged in sequence, the self-attention layer is used to capture the long-distance dependencies in the gene expression data, obtain the self-attention weight of each gene, and analyze the importance of the gene; the normalization layer performs residual connection and normalization operations to ensure that the input of each layer has the same dimension; the convolutional self-attention layer is used to capture local patterns and short-distance dependencies, calculate the convolutional attention weights and fuse the attention mechanism, the feedforward neural network consists of two linear transformation layers and a ReLu activation function sandwiched between the two linear transformation layers, and the output of the feedforward neural network is subjected to residual connection and normalization operations; The classifier uses a fully connected layer to perform classification, maps the output of the encoder to the target space of the classification task, and uses the Softmax function to calculate the probability distribution of each category and output the classification result; Step 3. Initialize the data mining model of key pathogenic genes of rice blast, train and evaluate the model, and then use the trained data mining model of key pathogenic genes of rice blast to classify rice gene data. During the data processing process, the data mining model of key pathogenic genes of rice blast will generate a fused attention weight for each gene at each time step in each encoding layer, and store the obtained fused attention weight; Step 4. Average the fused attention weights of all batches, calculate the comprehensive attention weight of each gene at different time points, and identify the most important genes by calculating and analyzing the attention score of each gene; ； in , attn_weights_avg The average attention weight of each gene at each time step ， For the i The layer encoding layer obtains the fused attention weight of each gene at each time step; ; in, is the comprehensive attention weight of each gene at n time steps, For the i Time step weight ， The larger the time step, the greater the weight; ； in, The final attention weight for each gene ， is a sample before or after infection, i=1 is a sample before infection, and i=2 is a sample after infection; Then, the genes are sorted according to their final attention weights, and the top N most important genes are output as the key pathogenic genes in the rice blast infection process and their corresponding scores.

2. The method for genetic data mining based on Transformer according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1 Data loading and preprocessing Gene expression data were read from the file, converted into gene count data using the RNA-SEQ tool, and then the differentially expressed gene matrix was obtained through differential analysis. The gene expression values ​​of the differentially expressed gene matrix were then standardized using the StandardScaler software so that the mean of each feature was 0 and the standard deviation was 1. Step 1.2 Create labels and adjust data format The data is divided into an uninfected group and an infected group according to the sample content. The uninfected group is 0 and the infected group is 1. The corresponding binary classification label file is generated. Step 1.3: Divide the training set and test set The expression values ​​of each gene at different time points were reshaped into a feature matrix suitable for model input, and the data were divided into training and test sets.

3. The method for genetic data mining based on Transformer according to claim 1, characterized in that: The convolution operation of the CNN convolutional neural network is expressed as: ; in, is the input tensor, is the convolution kernel, is the bias term, * represents the convolution operation, is the activation function; Represents the output after the convolution operation, the output data is (batch_size, 3, 64); The data output by the CNN convolutional neural network is used as the input of the embedding layer, which is linearly mapped to a high-dimensional space and then submitted to the encoder layer. The embedding layer is represented as a linear transformation: ; in, is the weight matrix of the embedding layer; is the bias term of the embedding layer; is the output after embedding; the output data of the embedding layer is (batch_size, 3, 256); The multi-head self-attention mechanism is expressed as: ； ； Among them, Q, K, V are query, key and value matrices respectively; It is the scaled dot product attention mechanism; is a linear matrix used to transform multiple self-attention head outputs into a projection matrix of the original shape. Representative i Head self-attention layer; The formula for calculating the self-attention weight of each time step in the self-attention layer is: ； Among them, Q, K, V are query, key and value matrices respectively; is the dimension of the key; is the self-attention weight matrix; T represents the transpose of the matrix; The calculation formula for fusion attention weight is: A fused =w1×A csa +w2×A mu ; Among them, A fused To integrate the attention weight, A csa is the convolution attention weight, A mu is the multi-head self-attention weight, w1 and w2 are weight coefficients, w1+w2=1.

4. The method for genetic data mining based on Transformer according to claim 1, characterized in that: The expression of the classifier is: ； Among them, A fused is the output of the encoding layer; is the weight matrix of the classifier; is the bias term of the classifier; P is the probability distribution of the classification results.

5. The method for genetic data mining based on Transformer according to claim 1, characterized in that: When training the model, the cross entropy loss function is used, combined with the category weight method to deal with the category imbalance problem; Use the Adam optimizer, set the learning rate, and start cyclic training; Forward propagation: input the training data into the model, calculate the predicted value, and calculate the loss based on the predicted value and the true label; Backpropagation calculates the gradient and updates the model parameters.

Images