Model training method and device and DNA methylation recognition method

Abstract

The invention discloses a model training method and device and a DNA methylation recognition method. The model training method comprises the following steps: performing digital coding processing on DNA methylation site data to obtain a first feature vector; capturing a dependency relationship inside the first feature vector by adopting a self-attention layer to obtain a second feature vector; performing parallel convolution processing on the second feature vector to obtain a plurality of convolution features of different scales; performing weighted aggregation on each convolution feature according to the importance weight to obtain a plurality of third feature vectors; performing fusion processing on the plurality of third feature vectors to obtain a classification result; and determining a loss function according to a classification result, and obtaining a trained deep learning model under the condition that the loss function meets a preset convergence condition. According to the method and the device, the technical problem that the generalization ability of the model is limited due to the fact that the dependency of the related model for identifying the DNA methylation site on feature coding is high is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology. Specifically, it relates to a model training method, apparatus, and DNA methylation recognition method. Background Art

[0002] In the fields of life sciences and genomics, the methylation status of DNA, especially the identification of N4-methylcytosine (m4C), is crucial for understanding gene expression regulation, biological development processes, and disease occurrence mechanisms. Traditional methods for detecting N4-methylcytosine sites, such as experimental sequencing techniques, although highly accurate, generally suffer from problems such as long time consumption, high cost, and difficulty in large-scale application. In recent years, with the rapid development of bioinformatics and artificial intelligence technologies, researchers have begun to explore the use of computer models to predict DNA N4-methylation sites to improve efficiency and reduce costs.

[0003] Multiple related machine learning-based prediction models have been proposed, such as 4mCpred-EL and i4mC-Mouse, which use different feature extraction methods (such as Kmer, EIIP, RFHC, DPCP, TPCP, BPF, and M6AMRFS, etc.) combined with ensemble learning strategies (such as extremely randomized trees, support vector machines, random forests, and gradient boosting) to identify N4-methylcytosine sites. These methods have improved the prediction accuracy and efficiency to a certain extent, but when dealing with complex or diverse DNA sequences, the limitations of feature selection and the insufficient generalization ability of the models are still a major challenge.

[0004] On the other hand, deep learning technologies, especially subclasses of recurrent neural networks such as long short-term memory networks and gated recurrent units, have been applied to the prediction of N4-methylcytosine sites due to their powerful ability to process sequence data. For example, the Mouse4mC-BGRU and Mus4mCPred methods use bidirectional GRU models to capture the context information of the sequence, improving the prediction accuracy. However, when dealing with large-scale data, these models have low computational efficiency and are difficult to meet the needs of real-time analysis. Especially when dealing with long sequence data, the computational complexity and running time of LSTM and GRU become bottlenecks.

[0005] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0006] The present application provides a model training method, apparatus, and DNA methylation recognition method to at least solve the technical problem of limited model generalization ability caused by the strong dependence of the related models for identifying DNA methylation sites on feature encoding.

[0007] According to one aspect of the present application, a model training method is provided, including: obtaining DNA methylation site data, and performing digital encoding processing on the DNA methylation site data to obtain a first feature vector; using a self-attention layer in a deep learning model to capture the dependencies within the first feature vector to obtain a second feature vector; using a multi-scale convolutional neural network layer in the deep learning model to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales; using a word attention layer in the deep learning model to calculate the importance weights of each convolutional feature, and performing weighted aggregation on each convolutional feature according to the importance weights to obtain multiple third feature vectors; using a fully connected neural network in the deep learning model to fuse the multiple third feature vectors to obtain a classification result, where the classification result includes: methylated sites, non-methylated sites; determining a loss function according to the classification result, and obtaining a trained deep learning model when the loss function meets a preset convergence condition.

[0008] Optionally, using a self-attention layer in a deep learning model to capture the dependencies within the first feature vector to obtain a second feature vector includes: multiplying the first feature vector by a preset query matrix weight, a preset key matrix weight, and a preset value matrix weight respectively to obtain a query matrix, a key matrix, and a value matrix; multiplying the query matrix by the transpose of the key matrix to obtain a first calculation result, and determining an attention score according to the first calculation result; multiplying the attention score by the value matrix to obtain a second feature vector.

[0009] Optionally, a self-attention layer in a deep learning model is used to capture dependencies within the first feature vector to obtain a second feature vector, including: using a first attention head in the self-attention layer to map each position of the first feature vector to a query vector, a key vector, and a value vector; determining a first bias term related to distance; for the query vector at each position, according to the first bias term, calculating a first attention score between the query vector and the key vectors of all other positions, multiplying the value vector at each position by its corresponding first attention score to obtain a first target calculation result for each position, and adding up the first target calculation results of all positions to obtain a first output result of the first attention head; using a second attention head in the self-attention layer to map each position of the first feature vector to a query vector, a key vector, and a value vector; determining a second bias term related to a motif region, where if any position belongs to the motif region, the second bias term is 1, and if any position does not belong to the motif region, the second bias term is 0; for the query vector at each position, according to the second bias term, calculating a second attention score between the query vector and the key vectors of all other positions, multiplying the value vector at each position by its corresponding second attention score to obtain a second target calculation result for each position, and adding up the second target calculation results of all positions to obtain a second output result of the second attention head; concatenating the first output result and the second output result to obtain a concatenated result, and performing a linear transformation on the concatenated result to obtain the second feature vector.

[0010] Optionally, a multi-scale convolutional neural network layer in a deep learning model is used to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales, including: using a first convolutional kernel, a second convolutional kernel, and a third convolutional kernel to perform parallel convolutional processing on the second feature vector to obtain a first output result, a second output result, and a third output result, where the first convolutional kernel is used to capture local features at the level of 3 consecutive bases, the second convolutional kernel is used to capture local features at the level of 5 consecutive bases, and the third convolutional kernel is used to capture local features of subsequences with an interval distance greater than a preset distance between any two, and the subsequences include at least one of the following: a promoter sequence, an enhancer sequence; respectively performing padding processing on the first output result, the second output result, and the third output result to obtain three sets of convolutional features with the same sequence length as the DNA methylation site data.

[0011] Optionally, the length of the first convolutional kernel is 3 and the number of channels is 32; the length of the second convolutional kernel is 5 and the number of channels is 32; the length of the third convolutional kernel is 7 and the number of channels is 32.

[0012] Optionally, the word attention layer in the deep learning model is used to calculate the importance weight of each convolutional feature, and each convolutional feature is weighted and aggregated according to the importance weight to obtain a plurality of third feature vectors, including: for each convolutional feature, each position in the convolutional feature is mapped to a query vector, a key vector, and a value vector using a linear transformation; for the query vector of each position, calculate the attention score between the query vector and the key vectors of all other positions; determine the attention score as the importance weight and perform normalization processing on the importance weight; multiply the value vector of each position by its corresponding importance weight to obtain the second calculation result of each position, and add up the second calculation results of all positions to obtain the third feature vector.

[0013] Optionally, the DNA methylation site data consists of bases A, T, C, and G, where each base is mapped to a unique binary vector; digital encoding processing is performed on the DNA methylation site data to obtain a first feature vector, including: replacing each base in the DNA methylation site data with its corresponding vector and arranging the vectors in a preset order to obtain the first feature vector.

[0014] According to another aspect of the present application, a DNA methylation recognition method is further provided, including: obtaining DNA sequence data and performing digital encoding processing on the DNA sequence data; using a deep learning model to determine the classification result of each position in the DNA sequence data after digital encoding processing, where the classification result includes: methylated site, non-methylated site, and the deep learning model is obtained by training through the above model training method.

[0015] According to another aspect of the present application, a model training device is further provided, including: an acquisition module for obtaining DNA methylation site data and performing digital encoding processing on the DNA methylation site data to obtain a first feature vector; a capture module for using the self-attention layer in the deep learning model to capture the dependencies inside the first feature vector to obtain a second feature vector; a convolution module for performing parallel convolution processing on the second feature vector using the multi-scale convolutional neural network layer in the deep learning model to obtain a plurality of convolutional features of different scales; a processing module for using the word attention layer in the deep learning model to calculate the importance weight of each convolutional feature and performing weighted aggregation on each convolutional feature according to the importance weight to obtain a plurality of third feature vectors; a fusion module for performing fusion processing on the plurality of third feature vectors using the fully connected neural network in the deep learning model to obtain a classification result, where the classification result includes: methylated site, non-methylated site; a training module for determining a loss function according to the classification result and obtaining a trained deep learning model when the loss function meets the preset convergence condition.

[0016] According to another aspect of the present application, a non-volatile storage medium is further provided. The storage medium includes a stored program. When the program runs, it controls the device where the storage medium is located to execute the above model training method.

[0017] According to another aspect of the present application, an electronic device is further provided, including: a memory and a processor. The processor is used to run a program stored in the memory. When the program runs, it executes the above model training method.

[0018] According to another aspect of the present application, a computer program is further provided. When the computer program is executed by a processor, it implements the above model training method.

[0019] According to another aspect of the present application, a computer program product is further provided. The computer program product includes a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the above model training method.

[0020] In the present application, DNA methylation site data is obtained and digitally encoded to obtain a first feature vector; the self-attention layer in the deep learning model is used to capture the dependencies within the first feature vector to obtain a second feature vector; the multi-scale convolutional neural network layer in the deep learning model is used to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales; the word attention layer in the deep learning model is used to calculate the importance weights of each convolutional feature, and each convolutional feature is weighted and aggregated according to the importance weights to obtain multiple third feature vectors; the fully connected neural network in the deep learning model is used to fuse the multiple third feature vectors to obtain a classification result, where the classification result includes: methylated sites, non-methylated sites; according to the classification result, a loss function is determined, and when the loss function meets the preset convergence condition, a deep learning model that has completed training is obtained. In this way, the purpose of reducing the dependence on the feature encoding of DNA sequence data is achieved, thereby realizing the technical effect of improving the generalization ability of the model, and further solving the technical problem that the generalization ability of the model is limited due to the strong dependence of the related model for identifying DNA methylation sites on feature encoding. Description of the Drawings

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0022] Figure 1 is a flowchart of a model training method according to an embodiment of the present application;

[0023] Figure 2 It is a schematic diagram of the architecture of a deep learning model according to an embodiment of the present application;

[0024] Figure 3 It is a structural diagram of a model training device according to an embodiment of the present application;

[0025] Figure 4 It is a hardware structure block diagram of a computer terminal for a model training method according to an embodiment of the present application. Specific embodiments

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to an embodiment of the present application, a method embodiment of a model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0029] Figure 1 It is a flowchart of a model training method according to an embodiment of the present application. As Figure 1 shown, the method includes the following steps:

[0030] Step S101, obtain DNA methylation site data, and perform digital coding processing on the DNA methylation site data to obtain a first feature vector.

[0031] Download the DNA methylation site data of mice from the database. The dataset includes 746 DNA sequences with known methylation sites and 746 DNA sequences without methylation sites. These sequences are composed of four bases: A, G, C, and T. Use one-hot encoding to convert the DNA sequences into digital matrices. For example, the base "A" is mapped to the vector [1, 0, 0, 0], "G" is mapped to [0, 1, 0, 0], "C" is mapped to [0, 0, 1, 0], and "T" is mapped to [0, 0, 0, 1]. After encoding, each DNA sequence is converted into a digital matrix encoded by bases, that is, the first feature vector is obtained.

[0032] Figure 2 It is a schematic diagram of the architecture of a deep learning model according to an embodiment of the present application. As Figure 2 shown, the deep learning model includes: a self-attention layer, a multi-scale convolutional layer, a word attention layer, and a fully connected layer.

[0033] Step S102, use the self-attention layer in the deep learning model to capture the dependencies within the first feature vector and obtain a second feature vector.

[0034] Send the first feature vector into the self-attention layer of the deep neural network model. The self-attention layer automatically identifies the dependencies between key bases in the input sequence by calculating the weights of the query matrix (Q), the key matrix (K), and the value matrix (V), and then generates a second feature vector. This process helps the model understand the context connection of the DNA sequence without relying on a specific feature encoding method.

[0035] Step S103, use the multi-scale convolutional neural network layer in the deep learning model to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales.

[0036] The second feature vector then passes through the multi-scale convolutional neural network layer. In this embodiment, three different scales of convolutional kernels are used to perform parallel convolutional processing on the second feature vector. Each scale of convolutional kernel will capture the features of the sequence in different length windows, and finally three different scales of convolutional features, that is, the multi-scale convolutional result, are obtained.

[0037] Step S104, use the word attention layer in the deep learning model to calculate the importance weights of each convolutional feature, and perform weighted aggregation on each convolutional feature according to the importance weights to obtain multiple third feature vectors.

[0038] The word attention layer receives the output of the multi-scale convolutional layer and calculates the importance weights of each convolutional feature. By weighted aggregation of the convolutional results at different scales, the word attention layer can highlight those convolutional features crucial for identifying methylation sites while reducing the impact of irrelevant features. After weighted aggregation, multiple third feature vectors are obtained, with each vector representing the DNA sequence features weighted by importance.

[0039] In step S105, a fully connected neural network in the deep learning model is used to fuse multiple third feature vectors to obtain a classification result, where the classification result includes: methylation sites and non-methylation sites.

[0040] Multiple third feature vectors are fed into the fully connected layer of the deep neural network model. The role of the fully connected layer is to further fuse the features from different scales and those processed by the word attention, and transform them into a comprehensive feature representation for classification. Through the activation function and the final classification layer, the model outputs the classification result, that is, predicts whether the sequence is a methylation site (output 1 for methylation site, output 0 for non-methylation site).

[0041] In step S106, according to the classification result, a loss function is determined, and when the loss function meets the preset convergence condition, a trained deep learning model is obtained.

[0042] Using the training data collected in step S101, the model is trained in a supervised manner. During the training process, the predicted results of the model are compared with the actual labels (methylation sites are labeled as 1, non-methylation sites are labeled as 0), and the loss function (such as cross-entropy loss) is calculated. Through backpropagation and optimization algorithms (such as gradient descent), the parameters of the model are continuously adjusted to minimize the loss function. When the loss function meets the preset convergence condition, that is, the loss value no longer decreases significantly, the training process ends, and a trained deep learning model is obtained.

[0043] The deep learning model combining multi-scale convolution and self-attention in this application has achieved significant technological breakthroughs in the field of DNA N4-methylcytosine site prediction, specifically reflected in the optimization of processing speed and resource consumption. 1. Significantly improved processing speed: The multi-scale convolution operation effectively focuses on the key features in the DNA sequence while eliminating unnecessary information, which significantly improves the efficiency of data processing. The introduced self-attention mechanism enables the model to capture long-range dependencies across multiple positions in the sequence, and this process can be processed in parallel, greatly shortening the time required for prediction. Experimental verification shows that the prediction accuracy of this model is as high as 81%, effectively reducing the processing time while ensuring high accuracy and improving the overall efficiency. 2. Greatly reduced resource consumption: The high-efficiency processing ability of the model means that it is more flexible in terms of hardware requirements. Even under limited computing resources, it can accurately predict N4-methylcytosine sites without relying on high-cost computing devices.

[0044] The significant advantages of this application in practical applications include: 1. In the medical scenario empowered by telecommunications services, this model can quickly provide the prediction results of DNA N4-methylation sites, supporting doctors in remote disease diagnosis and treatment plan planning, and optimizing the quality and response speed of telemedicine services. Combined with the cloud computing infrastructure of the operator, the model can efficiently process massive genomic data, providing a powerful data processing tool for medical research and clinical applications while avoiding excessive occupation of cloud resources. 2. The technological innovation of this application has attracted the attention of the medical industry to telecommunications services, promoting in-depth cooperation between the two parties in the fields of medical big data processing and gene detection. Through high-precision DNA N4-methylcytosine site prediction, it is possible to more deeply understand the molecular mechanism of diseases, laying a solid foundation for the development of personalized treatment plans, further promoting the development of precision medicine, and significantly expanding the application scope and influence of telecommunications services in the medical industry.

[0045] The following gives an exemplary illustration and explanation of Figure 1 the steps shown.

[0046] According to some optional embodiments of this application, the self-attention layer in the deep learning model is used to capture the dependencies within the first feature vector to obtain the second feature vector, which can be achieved by the following method: multiplying the first feature vector by the preset query matrix weight, preset key matrix weight, and preset value matrix weight respectively to obtain the query matrix, key matrix, and value matrix; multiplying the query matrix by the transpose of the key matrix to obtain the first calculation result, and determining the attention score based on the first calculation result; multiplying the attention score by the value matrix to obtain the second feature vector.

[0047] In the above embodiments, the main function of the self-attention layer is to automatically capture internal dependencies when processing sequence data, without explicitly specifying these relationships. The input of the self-attention layer is a set of sequence data, which is the preprocessed data in this embodiment. The input is multiplied by the query matrix weight, the key matrix weight, and the value matrix weight respectively to obtain the query matrix, the key matrix, and the value matrix. The calculation formula is:

[0048] Q = XW Q , K = XW K , V = XW V

[0049] where X is the input, and W Q , W K and W V are the query matrix weight, the key matrix weight, and the value matrix weight respectively.

[0050] Next, calculate the product of the query matrix Q and the transpose of the key matrix K to obtain the attention scores. Multiply the scores by the value matrix to obtain the final output, and its calculation formula is:

[0051]

[0052] where d k is the dimension of the key matrix K, and softmax is the activation function.

[0053] According to some other alternative embodiments of the present application, the self-attention layer in the deep learning model is used to capture the dependencies within the first feature vector to obtain a second feature vector, which can be achieved by the following method: The first attention head in the self-attention layer maps each position of the first feature vector to a query vector, a key vector, and a value vector; determine a first bias term related to distance; for the query vector at each position, according to the first bias term, calculate the first attention score between the query vector and the key vectors of all other positions, multiply the value vector at each position by its corresponding first attention score to obtain the first target calculation result at each position, and add up the first target calculation results of all positions to obtain the first output result of the first attention head; The second attention head in the self-attention layer maps each position of the first feature vector to a query vector, a key vector, and a value vector; determine a second bias term related to the motif region, where if any position belongs to the motif region, the second bias term is 1, and if any position does not belong to the motif region, the second bias term is 0; for the query vector at each position, according to the second bias term, calculate the second attention score between the query vector and the key vectors of all other positions, multiply the value vector at each position by its corresponding second attention score to obtain the second target calculation result at each position, and add up the second target calculation results of all positions to obtain the second output result of the second attention head; concatenate the first output result and the second output result to obtain a concatenated result, and perform a linear transformation on the concatenated result to obtain the second feature vector.

[0054] In the above embodiment, the design of the first bias term related to distance is to reflect the influence of the relative distance between different positions on the attention weight when calculating the attention score. For example, the farther the distance between two positions, the higher their attention score.

[0055] By considering the first bias term related to distance, the first attention head can effectively identify and process long-range dependencies in the sequence. This is crucial for understanding the bases or structures in the DNA sequence that may be far apart but have an important impact on the methylation site. In the DNA sequence, the methylation site may have a dependency on the base sequence far away from it. Traditional machine learning models or simple neural networks may have difficulty capturing such long-range interactions. The self-attention mechanism, by calculating the similarity score between the query vector and the key vector, can effectively identify the dependencies between any two positions in the sequence, regardless of how far apart they are. The addition of the first bias term further emphasizes the calculation of this dependency, enabling the model to pay more attention to the distal features that may be importantly related to the current position when processing long sequences.

[0056] The second attention head focuses on the motif regions, i.e., those combinations of bases that are known or hypothesized to be sensitive to N4 methylation. By setting the second bias term, the model can concentrate its attention on these regions, capturing local features that may have a direct indication for identifying methylation sites.

[0057] The two attention heads extract features from different perspectives (global and local) respectively, and these features are then concatenated together. This concatenation operation can fuse long-range dependence information and local motif patterns to form a more comprehensive and richer feature representation. Since different types of features may provide complementary information, this integration method helps the model to more accurately understand the complexity of DNA sequences. In addition, by simultaneously utilizing the long-range dependence relationship and the local features of the motif regions, the model can not only handle DNA sequences of various lengths but also maintain a high prediction accuracy in different types of sequences. This increases the robustness of the model when facing new data or unseen sequence variations and enhances its generalization ability.

[0058] Furthermore, the combination of the two attention heads not only helps the model learn the complex patterns in DNA sequences but also allows the model to deeply understand the recognition mechanism of methylation sites from a biological perspective, that is, how to combine long-range signals and local motif information to jointly determine whether a site is methylated.

[0059] Through the above self-attention processing method, the long-range dependence relationship inside the DNA sequence can be effectively captured, and at the same time, additional attention is given to the motif regions, thereby generating a more accurate feature representation, providing an optimized input for subsequent model training and prediction.

[0060] In some alternative embodiments of the present application, a multi-scale convolutional neural network layer in the deep learning model is used to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales, which can be achieved by the following method: the first convolutional kernel, the second convolutional kernel, and the third convolutional kernel are used to perform parallel convolutional processing on the second feature vector to obtain the first output result, the second output result, and the third output result, wherein the first convolutional kernel is used to capture local features at the level of 3 consecutive bases, the second convolutional kernel is used to capture local features at the level of 5 consecutive bases, and the third convolutional kernel is used to capture local features of subsequences with an interval distance greater than a preset distance between any two, and the subsequences include at least one of the following: promoter sequence, enhancer sequence; padding processing is respectively performed on the first output result, the second output result, and the third output result to obtain three groups of convolutional features that are consistent with the sequence length of the DNA methylation site data.

[0061] Preferably, the length of the first convolutional kernel is 3, and the number of channels is 32; the length of the second convolutional kernel is 5, and the number of channels is 32; the length of the third convolutional kernel is 7, and the number of channels is 32.

[0062] In the above embodiments, the first convolutional kernel, the second convolutional kernel, and the third convolutional kernel are respectively used to capture local features of different scales. The length of the first convolutional kernel is 3, which means it can capture patterns of 3 consecutive bases, and such patterns represent key biological motifs. The length of the second convolutional kernel is 5, which can identify a wider combination of 5 consecutive bases, covering more types of motif regions or local structures. The length of the third convolutional kernel is 7, which can not only capture features of longer consecutive base sequences, but also pay attention to features composed of subsequences with an interval greater than a preset distance, such as promoter sequences and enhancer sequences, which play important roles in regulating gene expression and methylation status.

[0063] The three convolutional kernels perform convolutional operations on the second feature vector simultaneously, which can be processed in parallel to speed up the calculation. The parallel design can significantly reduce the processing time and improve the overall efficiency, especially when dealing with large-scale data sets.

[0064] After the convolutional operation, in order to keep the output features consistent with the input sequence length, padding processing needs to be performed on the first output result, the second output result, and the third output result. This involves adding extra elements or values at the edges of the sequence to ensure that each convolutional feature can reflect the information of the entire sequence without losing important data due to boundary effects.

[0065] Through the above padding processing, the three sets of convolutional features obtained have the same sequence length, which facilitates subsequent feature fusion. By integrating these features through splicing or more complex aggregation operations, a unified feature representation containing multi-scale information of the DNA sequence can be formed, which can more comprehensively reflect the complex characteristics of the sequence and help improve the prediction performance of the model.

[0066] By adopting the above design of multi-scale convolutional kernels, the model can simultaneously pay attention to short, medium, and long-distance local features in the DNA sequence, including short fragments that have a direct impact on the methylation state, as well as promoter and enhancer sequences that may span a large distance but are crucial for gene regulation. This comprehensive feature extraction method not only improves the recognition accuracy of the model, but also enhances its adaptability to different types of DNA sequences, making the model have stronger generalization ability and be applicable to a wider range of bioinformatics analysis tasks. At the same time, the unified feature representation ensures that the model can fully consider the global information of the sequence during the feature fusion stage, providing a more reliable data basis for the final prediction.

[0067] As some alternative embodiments of the present application, the word attention layer in the deep learning model is used to calculate the importance weights of each convolutional feature, and each convolutional feature is weighted and aggregated according to the importance weights to obtain a plurality of third feature vectors, which can be implemented by the following method: for each convolutional feature, use a linear transformation to map each position in the convolutional feature to a query vector, a key vector, and a value vector; for the query vector of each position, calculate the attention score between the query vector and the key vectors of all other positions; determine the attention score as the importance weight and perform normalization processing on the importance weight; multiply the value vector of each position by its corresponding importance weight to obtain the second calculation result of each position, and add up the second calculation results of all positions to obtain the third feature vector.

[0068] In the above embodiment, each position in the convolutional feature is converted into a query vector, a key vector, and a value vector through a linear transformation. This process involves matrix multiplication of the weight matrix and the convolutional feature to ensure that each vector can capture specific information from the convolutional feature. The query vector and the key vector are used to calculate the attention score, while the value vector contains the information to be weighted. For the query vector of each position, calculate the attention score between it and the key vectors of all other positions. This step utilizes the similarity between the query and the key vectors and reflects the correlation degree of information at different positions.

[0069] The calculated attention score is used as the importance weight of the value vector at each position. Through normalization processing, such as using the softmax function, these weights are adjusted to between 0 and 1 to ensure that they can be interpreted as probabilities and the sum of the weights of all positions is 1. Multiply the value vector of each position by its corresponding importance weight to obtain the second calculation result of that position. In this way, the value vector of each position is assigned a weight calculated according to its importance in the sequence. Subsequently, add up the second calculation results of all positions to obtain the third feature vector, which is a feature representation that synthesizes all position information in the sequence and is obtained by weighted summation according to importance.

[0070] In some alternative embodiments of the present application, the DNA methylation site data consists of bases A, T, C, G, and each base is mapped to a unique binary vector.

[0071] Furthermore, digital coding processing is performed on the DNA methylation site data to obtain the first feature vector, which can be implemented by the following method: replace each base in the DNA methylation site data with its corresponding vector and arrange the vectors in a preset order to obtain the first feature vector.

[0072] DNA sequences are composed of four characters: A, G, C, and T. Before inputting into the model, they are digitized through one-hot encoding. The mapping relationships of each character are as follows:

[0073]

[0074] According to the above mapping relationships, each base symbol in DNA is converted one by one, enabling the digital processing of sample data. Suppose the input data is "ACG…TA", after preprocessing, the output will be a matrix data of [[1, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0]…[0, 0, 0, 1], [1, 0, 0, 0]].

[0075] Figure 3 It is a structural diagram of a model training device according to an embodiment of the present application. As Figure 3 shown, the device includes:

[0076] An acquisition module 31, configured to acquire DNA methylation site data and perform digital encoding processing on the DNA methylation site data to obtain a first feature vector.

[0077] A capture module 32, configured to capture the dependency relationships within the first feature vector by using a self-attention layer in a deep learning model to obtain a second feature vector.

[0078] A convolution module 33, configured to perform parallel convolution processing on the second feature vector by using a multi-scale convolutional neural network layer in a deep learning model to obtain multiple convolution features of different scales.

[0079] A processing module 34, configured to calculate the importance weights of each convolution feature by using a word attention layer in a deep learning model, and perform weighted aggregation on each convolution feature according to the importance weights to obtain multiple third feature vectors.

[0080] A fusion module 35, configured to perform fusion processing on multiple third feature vectors by using a fully connected neural network in a deep learning model to obtain a classification result, where the classification result includes: methylated sites, non-methylated sites.

[0081] A training module 36, configured to determine a loss function according to the classification result, and obtain a trained deep learning model when the loss function meets a preset convergence condition.

[0082] Optionally, the capture module 32 is further configured to perform the following steps: multiplying the first feature vector by a preset query matrix weight, a preset key matrix weight, and a preset value matrix weight respectively to obtain a query matrix, a key matrix, and a value matrix; multiplying the query matrix by the transpose of the key matrix to obtain a first calculation result, and determining an attention score according to the first calculation result; multiplying the attention score by the value matrix to obtain a second feature vector.

[0083] Optionally, the capture module 32 is further configured to perform the following steps: using the first attention head in the self-attention layer to map each position of the first feature vector to a query vector, a key vector, and a value vector; determining a first bias term related to distance; for the query vector at each position, calculating a first attention score between the query vector and the key vectors at all other positions according to the first bias term, multiplying the value vector at each position by its corresponding first attention score to obtain a first target calculation result at each position, and adding up the first target calculation results at all positions to obtain a first output result of the first attention head; using the second attention head in the self-attention layer to map each position of the first feature vector to a query vector, a key vector, and a value vector; determining a second bias term related to the motif region, where if any position belongs to the motif region, the second bias term is 1, and if any position does not belong to the motif region, the second bias term is 0; for the query vector at each position, calculating a second attention score between the query vector and the key vectors at all other positions according to the second bias term, multiplying the value vector at each position by its corresponding second attention score to obtain a second target calculation result at each position, and adding up the second target calculation results at all positions to obtain a second output result of the second attention head; concatenating the first output result and the second output result to obtain a concatenated result, and performing a linear transformation on the concatenated result to obtain a second feature vector.

[0084] Optionally, the convolution module 33 is further configured to perform the following steps: performing parallelized convolution processing on the second feature vector using a first convolution kernel, a second convolution kernel, and a third convolution kernel to obtain a first output result, a second output result, and a third output result, where the first convolution kernel is used to capture local features at the level of 3 consecutive bases, the second convolution kernel is used to capture local features at the level of 5 consecutive bases, and the third convolution kernel is used to capture local features of subsequences with an interval distance greater than a preset distance between any two, and the subsequences include at least one of the following: promoter sequence, enhancer sequence; respectively performing padding processing on the first output result, the second output result, and the third output result to obtain three sets of convolution features consistent with the sequence length of the DNA methylation site data.

[0085] Optionally, the length of the first convolution kernel is 3 and the number of channels is 32; the length of the second convolution kernel is 5 and the number of channels is 32; the length of the third convolution kernel is 7 and the number of channels is 32.

[0086] Optionally, the processing module 34 is further configured to perform the following steps: for each convolutional feature, use a linear transformation to map each position in the convolutional feature to a query vector, a key vector, and a value vector; for the query vector of each position, calculate the attention score between the query vector and the key vectors of all other positions; determine the attention score as the importance weight and perform normalization processing on the importance weight; multiply the value vector of each position by its corresponding importance weight to obtain a second calculation result for each position, and add up the second calculation results of all positions to obtain a third feature vector.

[0087] Optionally, the DNA methylation site data consists of bases A, T, C, and G, where each base is mapped to a unique binary vector; perform digital encoding processing on the DNA methylation site data to obtain a first feature vector, including: replacing each base in the DNA methylation site data with its corresponding vector, and arranging the vectors in a preset order to obtain the first feature vector.

[0088] It should be noted that the above Figure 3 Each of the modules can be a program module (for example, a set of program instructions that implement a specific function), or a hardware module. For the latter, it can be presented in the following forms, but not limited to: the manifestation form of each of the above modules is a processor, or the functions of each of the above modules are implemented by a processor.

[0089] It should be noted that Figure 3 For the preferred implementation manners of the illustrated embodiments, reference may be made to Figure 1 the relevant descriptions of the illustrated embodiments, which will not be elaborated here.

[0090] Figure 4 shows a hardware structure block diagram of a computer terminal for implementing a model training method. As Figure 4 shown, the computer terminal 40 may include one or more (shown as 402a, 402b,..., 402n in the figure) processors 402 (the processor 402 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 404 for storing data, and a transmission module 406 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 4 the structure shown is only schematic and does not limit the structure of the above electronic device. For example, the computer terminal 40 may further include more or fewer components than Figure 4 shown in the figure, or have the same asFigure 4 The different configurations shown.

[0091] It should be noted that one or more of the above-mentioned processors 402 and / or other data processing circuits can generally be referred to as "data processing circuits" herein. The data processing circuit can be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of other elements in the computer terminal 40. As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0092] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in the embodiments of the present application. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, that is, implements the above-mentioned model training method. The memory 404 can include high-speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 404 can further include a memory remotely disposed relative to the processor 402, and these remote memories can be connected to the computer terminal 40 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0093] The transmission module 406 is used to receive or send data via a network. Specific examples of the above-mentioned network can include the wireless network provided by the communication provider of the computer terminal 40. In one instance, the transmission module 406 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission module 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0094] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables a user to interact with the user interface of the computer terminal 40.

[0095] It should be noted here that in some alternative embodiments, the above Figure 4 shown computer terminal can include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 4This is just one example of a specific concrete instance and is intended to illustrate the types of components that may exist in the above computer terminal.

[0096] It should be noted that Figure 4 the computer terminal shown is used to execute Figure 1 the model training method shown. Therefore, the relevant explanations in the execution method of the above commands also apply to this electronic device and will not be elaborated here.

[0097] The embodiment of the present application also provides a non-volatile storage medium. The non-volatile storage medium includes a stored program. When the program runs, it controls the device where the storage medium is located to execute the above model training method.

[0098] A program for the non-volatile storage medium to perform the following functions: obtaining DNA methylation site data, and performing digital encoding processing on the DNA methylation site data to obtain a first feature vector; using the self-attention layer in the deep learning model to capture the dependencies within the first feature vector to obtain a second feature vector; using the multi-scale convolutional neural network layer in the deep learning model to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales; using the word attention layer in the deep learning model to calculate the importance weights of each convolutional feature, and performing weighted aggregation on each convolutional feature according to the importance weights to obtain multiple third feature vectors; using the fully connected neural network in the deep learning model to perform fusion processing on the multiple third feature vectors to obtain a classification result, where the classification result includes: methylated sites, non-methylated sites; determining a loss function according to the classification result, and obtaining a trained deep learning model when the loss function satisfies a preset convergence condition.

[0099] The embodiment of the present application also provides an electronic device, including: a memory and a processor. The processor is used to run the program stored in the memory. When the program runs, it executes the above model training method.

[0100] The processor is used to run a program that performs the following functions: obtaining DNA methylation site data, digitally encoding the DNA methylation site data to obtain a first feature vector; using the self-attention layer in the deep learning model to capture the dependencies within the first feature vector to obtain a second feature vector; using the multi-scale convolutional neural network layer in the deep learning model to perform parallel convolutional processing on the second feature vector to obtain multiple convolutional features of different scales; using the word attention layer in the deep learning model to calculate the importance weights of each convolutional feature, and performing weighted aggregation on each convolutional feature according to the importance weights to obtain multiple third feature vectors; using the fully connected neural network in the deep learning model to fuse the multiple third feature vectors to obtain a classification result, where the classification result includes: methylated sites, non-methylated sites; determining a loss function according to the classification result, and obtaining a trained deep learning model when the loss function meets the preset convergence condition.

[0101] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0102] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0103] In the above embodiments of the present application, the information collected is information and data authorized by the user or fully authorized by all parties, and the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure, and application, all comply with relevant laws, regulations, and standards, take necessary protection measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0104] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, may exist physically alone for each unit, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0107] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0108] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A model training method, characterized in that: include: Acquiring DNA methylation site data, and performing digital encoding processing on the DNA methylation site data to obtain a first feature vector; A self-attention layer in a deep learning model is used to capture the dependency relationship within the first feature vector to obtain a second feature vector; Performing parallel convolution processing on the second feature vector using a multi-scale convolutional neural network layer in the deep learning model to obtain multiple convolution features of different scales; Using the word attention layer in the deep learning model to calculate the importance weight of each of the convolutional features, and performing weighted aggregation on each of the convolutional features according to the importance weight to obtain a plurality of third feature vectors; Using a fully connected neural network in the deep learning model to fuse the multiple third feature vectors to obtain a classification result, wherein the classification result includes: methylation sites and non-methylation sites; A loss function is determined based on the classification result, and when the loss function meets a preset convergence condition, a deep learning model that has completed training is obtained.

2. The method according to claim 1, characterized in that The self-attention layer in the deep learning model is used to capture the dependencies within the first feature vector to obtain a second feature vector, including: Multiplying the first feature vector by a preset query matrix weight, a preset key matrix weight, and a preset value matrix weight respectively to obtain a query matrix, a key matrix, and a value matrix; Multiplying the query matrix and the transpose of the key matrix to obtain a first calculation result, and determining an attention score based on the first calculation result; The attention score is multiplied by the value matrix to obtain the second feature vector.

3. The method according to claim 1, characterized in that The self-attention layer in the deep learning model is used to capture the dependencies within the first feature vector to obtain a second feature vector, including: Using the first attention head in the self-attention layer to map each position of the first feature vector to a query vector, a key vector and a value vector; determining a first bias term related to the distance; for the query vector at each position, calculating a first attention score between the query vector and the key vectors of all other positions according to the first bias term, multiplying the value vector of each position by the first attention score corresponding thereto to obtain a first target calculation result for each position, and adding the first target calculation results of all positions to obtain a first output result of the first attention head; Using the second attention head in the self-attention layer to map each position of the first feature vector to a query vector, a key vector and a value vector; determining a second bias item related to the motif region, wherein if any position belongs to the motif region, the second bias item is 1, and if any position does not belong to the motif region, the second bias item is 0; for the query vector at each position, according to the second bias item, calculating the second attention score between the query vector and the key vectors of all other positions, multiplying the value vector of each position by the second attention score corresponding thereto, obtaining a second target calculation result for each position, and adding the second target calculation results of all positions to obtain a second output result of the second attention head; The first output result and the second output result are concatenated to obtain a concatenated result, and a linear transformation is performed on the concatenated result to obtain the second feature vector.

4. The method according to claim 1, characterized in that: The second feature vector is processed by parallel convolution using a multi-scale convolutional neural network layer in the deep learning model to obtain multiple convolutional features of different scales, including: The second feature vector is subjected to parallel convolution processing using a first convolution kernel, a second convolution kernel, and a third convolution kernel to obtain a first output result, a second output result, and a third output result, wherein the first convolution kernel is used to capture local features at three consecutive base levels, the second convolution kernel is used to capture local features at five consecutive base levels, and the third convolution kernel is used to capture local features of any two subsequences whose interval distance is greater than a preset distance, and the subsequence includes at least one of the following: a promoter sequence and an enhancer sequence; The first output result, the second output result and the third output result are filled respectively to obtain three groups of convolution features with the same sequence length as the DNA methylation site data.

5. The method according to claim 4, characterized in that The length of the first convolution kernel is 3, and the number of channels is 32; the length of the second convolution kernel is 5, and the number of channels is 32; the length of the third convolution kernel is 7, and the number of channels is 32.

6. The method according to claim 1, characterized in that The importance weight of each of the convolutional features is calculated using the word attention layer in the deep learning model, and each of the convolutional features is weightedly aggregated according to the importance weight to obtain multiple third feature vectors, including: For each of the convolutional features, use a linear transformation to map each position in the convolutional feature to a query vector, a key vector, and a value vector; For each query vector at each position, calculate the attention score between the query vector and the key vectors at all other positions; Determining the attention score as an importance weight, and normalizing the importance weight; The value vector of each position is multiplied by the importance weight corresponding thereto to obtain the second calculation result of each position, and the second calculation results of all positions are added together to obtain the third feature vector.

7. The method according to claim 1, characterized in that The DNA methylation site data consists of bases A, T, C, and G, wherein each base is mapped to a unique binary vector; The DNA methylation site data is digitally encoded to obtain a first feature vector, including: Each base in the DNA methylation site data is replaced with its corresponding vector, and the vectors are arranged in a preset order to obtain the first feature vector.

8. A method for identifying DNA methylation, characterized in that: include: Acquiring DNA sequence data, and performing digital encoding processing on the DNA sequence data; A deep learning model is used to determine the classification result of each position in the DNA sequence data after digital coding processing, wherein the classification result includes: methylation sites, non-methylation sites, and the deep learning model is obtained by training through the model training method described in any one of claims 1 to 7.

9. A model training device, characterized in that: include: An acquisition module, used to acquire DNA methylation site data, and perform digital encoding processing on the DNA methylation site data to obtain a first feature vector; A capture module, configured to capture the dependency relationship within the first feature vector using a self-attention layer in a deep learning model to obtain a second feature vector; A convolution module, configured to perform parallel convolution processing on the second feature vector using a multi-scale convolutional neural network layer in the deep learning model to obtain a plurality of convolution features of different scales; A processing module, configured to calculate the importance weight of each of the convolutional features by using the word attention layer in the deep learning model, and perform weighted aggregation on each of the convolutional features according to the importance weight to obtain a plurality of third feature vectors; A fusion module, used to use a fully connected neural network in the deep learning model to fuse the multiple third feature vectors to obtain a classification result, wherein the classification result includes: methylation sites and non-methylation sites; The training module is used to determine the loss function according to the classification result, and obtain a deep learning model that completes the training when the loss function meets the preset convergence condition.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the model training method described in any one of claims 1 to 7.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the model training method described in any one of claims 1 to 7 when running.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method described in any one of claims 1 to 7 is implemented.

Images