Model training method and device based on reinforcement learning, electronic equipment and medium

Abstract

The present application provides a model training method and device based on reinforcement learning, an electronic device and a medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining a sample DNA sequence fragment and a preset original reinforcement learning model; inputting the sample DNA sequence fragment into the original reinforcement learning model for state extraction to extract the sample base distribution state from the reinforcement learning environment; performing motif search processing on the sample base distribution state by an intelligent agent to obtain an initial DNA motif; performing reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score; updating the gradient parameters of the original reinforcement learning model according to the motif evaluation score to train the original reinforcement learning model to obtain a motif search model. The present application can improve the training effect of the model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method based on reinforcement learning, a motif search method and device, an electronic device and a medium. Background Art

[0002] At present, DNA motif search methods based on neural network models often need to rely on a large amount of labeled data to train the model. However, it is difficult to obtain labeled data, which affects the training effect of the model. Therefore, how to improve the training effect of the model has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method, a motif search method and device, an electronic device and a medium based on reinforcement learning, aiming to improve the training effect of the model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method based on reinforcement learning, the method comprising:

[0005] Obtaining a sample DNA sequence fragment and a preset original reinforcement learning model, wherein the original reinforcement learning model includes a reinforcement learning environment and an intelligent agent;

[0006] Inputting the sample DNA sequence fragment into the original reinforcement learning model for state extraction to extract the sample base distribution state from the reinforcement learning environment;

[0007] The intelligent agent performs motif search processing on the base distribution state of the sample to obtain an initial DNA motif; wherein the intelligent agent includes a convolutional network and an activation network;

[0008] Performing reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score;

[0009] The gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score to train the original reinforcement learning model to obtain a motif search model.

[0010] In some embodiments, the agent includes a convolutional network and an activation network, and the step of performing motif search processing on the sample base distribution state by the agent to obtain an initial DNA motif includes:

[0011] Performing convolution processing on the sample base distribution state through the convolution network to obtain a sample base distribution matrix;

[0012] Performing activation processing on the sample base distribution matrix through the activation network to obtain a base probability matrix;

[0013] The initial DNA motif is obtained according to the preset base type and the base probability matrix.

[0014] In some embodiments, the convolutional network includes a convolutional layer, a batch normalization layer, and a first activation layer, and the step of performing convolution processing on the sample base distribution state through the convolutional network to obtain a sample base distribution matrix includes:

[0015] Extracting features of the sample base distribution state through the convolution layer to obtain an initial base distribution feature map;

[0016] Normalizing the initial base distribution feature map through the batch normalization layer to obtain a target base distribution feature map;

[0017] The target base distribution characteristic map is activated by the first activation layer to obtain the sample base distribution matrix.

[0018] In some embodiments, the activation network includes a first fully connected layer, a second activation layer, and a prediction layer, and the step of performing activation processing on the sample base distribution matrix through the activation network to obtain a base probability matrix includes:

[0019] Mapping the sample base distribution matrix to a preset vector space through the first fully connected layer to obtain a sample base distribution vector;

[0020] Performing dimension-changing processing on the sample base distribution vector through the second activation layer to obtain a target base distribution vector;

[0021] The prediction layer calculates the distribution probability of the target base distribution vector to obtain the base probability matrix.

[0022] In some embodiments, the step of performing reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score includes:

[0023] Performing base statistics on the sample DNA sequence fragments in the reinforcement learning environment to obtain a base count matrix;

[0024] In the reinforcement learning environment, a divergence score calculation is performed on each preset site according to the base count matrix and the initial DNA motif to obtain a site score value;

[0025] The site scoring values ​​are summed to obtain the motif evaluation score.

[0026] To achieve the above object, a second aspect of an embodiment of the present application proposes a motif search method, the method comprising:

[0027] Obtaining a target DNA sequence fragment to be processed;

[0028] The target DNA sequence fragment is input into a motif search model for motif search processing to obtain a target DNA motif, wherein the motif search model is trained according to the model training method of the first aspect.

[0029] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a model training device based on reinforcement learning, the device comprising:

[0030] A first acquisition module is used to acquire a sample DNA sequence fragment and a preset original reinforcement learning model, wherein the original reinforcement learning model includes a reinforcement learning environment and an intelligent agent;

[0031] A state extraction module, used for inputting the sample DNA sequence fragment into the original reinforcement learning model for state extraction, so as to extract the sample base distribution state from the reinforcement learning environment;

[0032] A first motif search module, used to perform motif search processing on the base distribution state of the sample through the intelligent agent to obtain an initial DNA motif; wherein the intelligent agent includes a convolutional network and an activation network;

[0033] A reward calculation module, used to perform reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score;

[0034] A training module is used to update the gradient parameters of the original reinforcement learning model according to the motif evaluation score to train the original reinforcement learning model to obtain a motif search model.

[0035] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present application provides a motif search device, the device comprising:

[0036] The second acquisition module is used to acquire the target DNA sequence fragment to be processed;

[0037] The second motif search module is used to input the target DNA sequence fragment into a motif search model for motif search processing to obtain a target DNA motif, wherein the motif search model is trained according to the model training method described in the first aspect.

[0038] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect or the method described in the second aspect is realized.

[0039] To achieve the above-mentioned purpose, the sixth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect or the method described in the second aspect above.

[0040] The model training method, motif search method and device, electronic device and medium based on reinforcement learning proposed in this application obtain sample DNA sequence fragments and a preset original reinforcement learning model, and input the sample DNA sequence fragments into the original reinforcement learning model for state extraction, thereby extracting the sample base distribution state from the reinforcement learning environment. This method can easily obtain the specific arrangement of bases on the sample DNA sequence. Further, the intelligent agent performs motif search processing on the sample base distribution state to obtain the initial DNA motif, and calculates the reward for the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain the motif evaluation score. This method can search for the corresponding initial DNA motif according to the sample base distribution state, and perform score calculation on the initial DNA motif, and quantitatively evaluate the quality of the initial DNA motif. Finally, the gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score, thereby realizing the training of the original reinforcement learning model and obtaining the motif search model. This method can feed back the motif evaluation score to the original reinforcement learning model, so that the intelligent agent continuously optimizes the gradient parameters and learns according to the gradient parameters, thereby obtaining the motif search model. Compared with the traditional neural network model, the model of the embodiment of the present application can realize the training of the model without relying on label data, thereby improving the training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a model training method based on reinforcement learning provided in an embodiment of the present application;

[0042] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0043] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.

[0044] Figure 4 yes Figure 2 Flow chart of step S202 in FIG.

[0045] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0046] Figure 6 is a flow chart of a motif search method provided in an embodiment of the present application;

[0047] Figure 7 It is a structural diagram of a model training device based on reinforcement learning provided in an embodiment of the present application;

[0048] Figure 8 is a schematic diagram of the structure of a motif search device provided in an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] First, some nouns involved in this application are analyzed:

[0054] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0055] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0056] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0057] Reinforcement Learning (RL): Also known as reinforcement learning, evaluation learning or enhanced learning, it is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals through learning strategies during its interaction with the environment. The goal of reinforcement learning is to learn the mapping from environmental states to behaviors, so that the behaviors selected by the agent can obtain the maximum reward from the environment, and the external environment's evaluation of the learning system (or the operating performance of the entire system) is optimal in a certain sense.

[0058] Agent: The problem background of reinforcement learning is a binary structure, namely environment and agent. Agent mainly refers to the entity that can interact with the environment and has the ability to change the state of the environment. This change is mainly achieved by action. In short, the agent is a neural network model composed of several neural network layers that maps observation results to actions.

[0059] DNA motif: DNA motifs are short (usually 8-20 bp) fragments on the genome (DNA). These short sequences are enriched in DNA fragments with the same biological function (such as the promoter region of genes and around transcription factor binding sites).

[0060] DNA motif finding or DNA motif discovery: refers to the search for potential DNA motifs from a specific set of DNA fragments (usually fragments with the same biological function, such as the promoter region of a gene or transcription factor binding sites obtained by experimental means).

[0061] Conditional random field algorithm (CRF): is a mathematical algorithm; it combines the characteristics of maximum entropy model and hidden Markov model, and is an undirected graph model. In recent years, it has achieved good results in sequence labeling tasks such as word segmentation, part-of-speech tagging and named entity recognition. Conditional random field is a typical discriminant model, and its joint probability can be written as a combination of several potential functions, among which the most commonly used is linear chain conditional random field. If x = (x1, x2, ... xn) represents the observed input data sequence, y = (y1, y2, ... yn) represents a state sequence, given an input sequence, the linear chain CRF model defines the joint conditional probability of the state sequence as p(y|x) = exp{}(2-14); Z(x) = {}(2-15); where: Z is the probability normalization factor conditioned on the observed sequence x; fj(yi-1, yi, x, i) is an arbitrary feature function.

[0062] Bi-directional Long Short-Term Memory (Bi-LSTM): It is composed of forward LSTM and backward LSTM. It is often used to model contextual information in natural language processing tasks. Bi-LSTM combines the information of the input sequence in both the forward and backward directions based on LSTM. For the output at time t, the forward LSTM layer has the information at time t and before in the input sequence, while the backward LSTM layer has the information at time t and after in the input sequence. The output of the forward LSTM layer at time t is denoted as, and the output of the backward LSTM layer at time t is denoted as. The vectors output by the two LSTM layers can be processed by addition, average, or concatenation.

[0063] Activation: The activation operation on the input data is actually a function transformation, which is performed element by element. Commonly used activation functions include sigmoid function, tanh function, relu function, etc.

[0064] Normalization: There are two forms of normalization methods, one is to convert the number into a decimal between (0, 1), and the other is to convert the dimensional expression into a dimensionless expression. It is mainly proposed for the convenience of data processing. Mapping the data to the range of 0 to 1 for processing is more convenient and faster, and should be classified as part of digital signal processing.

[0065] Softmax function: The Softmax function is a normalized exponential function that can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector so that the range of each element is between (0,1) and the sum of all elements is 1. This function is often used in multi-classification problems.

[0066] In terms of DNA motif search, most of the methods use heuristic search to search for DNA motifs, but this method often requires repeated search process to make the final result more accurate. The heuristic search algorithm is time-consuming and cannot guarantee the optimal result. Therefore, the neural network model is introduced in the relevant technology to search for DNA motifs. The DNA motif search method based on the neural network model often needs to rely on a large amount of labeled data to train the model. However, it is difficult to obtain labeled data, which affects the training effect of the model. Therefore, how to improve the training effect of the model has become a technical problem that needs to be solved urgently.

[0067] Based on this, the embodiments of the present application provide a model training method, a motif search method and device, an electronic device and a medium based on reinforcement learning, aiming to improve the training effect of the model.

[0068] The model training method, motif search method and device, electronic device and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the motif search method in the embodiments of the present application is described.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] The motif search method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The motif search method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the motif search method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] Figure 1 is an optional flowchart of the model training method based on reinforcement learning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0074] Step S101, obtaining a sample DNA sequence fragment and a preset original reinforcement learning model, wherein the original reinforcement learning model includes a reinforcement learning environment and an intelligent agent;

[0075] Step S102, inputting the sample DNA sequence fragment into the original reinforcement learning model for state extraction, so as to extract the sample base distribution state from the reinforcement learning environment;

[0076] Step S103, performing motif search processing on the sample base distribution state through an intelligent agent to obtain an initial DNA motif; wherein the intelligent agent includes a convolutional network and an activation network;

[0077] Step S104, performing reward calculation on the initial DNA motif and the sample DNA sequence fragment in a reinforcement learning environment to obtain a motif evaluation score;

[0078] Step S105, updating the gradient parameters of the original reinforcement learning model according to the motif evaluation score to train the original reinforcement learning model to obtain a motif search model.

[0079] In the steps S101 to S105 shown in the embodiment of the present application, the sample DNA sequence fragment is input into the original reinforcement learning model for state extraction, thereby extracting the sample base distribution state from the reinforcement learning environment, and the specific arrangement of the bases on the sample DNA sequence can be easily obtained. The sample base distribution state is subjected to motif search processing by the agent to obtain the initial DNA motif, and the reward calculation is performed on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain the motif evaluation score, and the corresponding initial DNA motif can be searched according to the sample base distribution state, and the initial DNA motif is scored and calculated, and the quality of the initial DNA motif is quantitatively evaluated. Finally, the gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score, thereby realizing the training of the original reinforcement learning model, and obtaining the motif search model, which can feed back the motif evaluation score to the original reinforcement learning model, so that the agent continuously optimizes the gradient parameters and learns according to the gradient parameters, thereby obtaining the motif search model. Compared with traditional neural network models, the model of the embodiment of the present application can realize model training without relying on label data, and can obtain a better DNA motif by inputting a small amount of sample DNA fragments, thereby improving the training effect of the model.

[0080] In step S101 of some embodiments, a preset number of DNA sequence fragments can be extracted from an existing DNA database, and these DNA sequence fragments are used as sample DNA sequence fragments in this application. The original reinforcement learning model can be constructed based on reinforcement learning, and the original reinforcement learning model includes an intelligent agent and a reinforcement learning environment.

[0081] In an embodiment of the present application, the reinforcement learning environment (environment) is mainly a sample DNA sequence fragment, the agent (agent) is a neural network, the sample base distribution state is used as the state (state) of the reinforcement environment, the sample base distribution state is input into the agent for processing, the initial DNA motif output by the agent is used as the action (action) of the agent, and the evaluation criteria for whether the initial DNA motif can represent the sample DNA sequence fragment is used as the reward (reward) of reinforcement learning. Therefore, the present application mainly extracts the sample base distribution state from the reinforcement learning environment by taking the sample DNA sequence fragment as input, processes the sample base distribution state by the agent to generate an initial DNA motif, and scores the generated initial DNA motif by the reinforcement learning environment, and feeds back the scoring results to the agent, so that the agent continuously optimizes the gradient parameters according to the feedback to generate a better initial DNA motif.

[0082] In step S102 of some embodiments, the sample DNA sequence fragment is input into the original reinforcement learning model for state extraction, and the sample base distribution state (State, S) is extracted from the reinforcement learning environment. Specifically, the entity features of the sample DNA sequence fragment can be identified and extracted by a named entity extraction algorithm (e.g., a conditional random field algorithm (CRF), a bidirectional long short-term memory algorithm (Bi-LSTM), a maximum entropy algorithm, etc.). For example, the preset entity keywords are the number and length of the sample DNA sequence. In the model based on the Bi-LSTM algorithm, the embedding of the input sample DNA sequence fragment is generated at the position where the output is connected through the left-to-right long short-term memory and the right-to-left long short-term memory. Through this output layer, the input entity features can be directly transferred to the corresponding keyword label, thereby obtaining the sample base distribution state (state, s) of the sample DNA sequence fragment, wherein the sample base distribution state includes the type of base (generally 4), the length L of the sample DNA sequence fragment and the number N of the sample DNA sequence fragment, and the sample base distribution state can be expressed as s=N*L*4.

[0083] See also Figure 2 In some embodiments, the agent includes a convolutional network and an activation network, and step S103 may include but is not limited to steps S201 to S203:

[0084] Step S201, performing convolution processing on the sample base distribution state through a convolutional network to obtain a sample base distribution matrix;

[0085] Step S202, activating the sample base distribution matrix by activating the network to obtain a base probability matrix;

[0086] Step S203, obtaining an initial DNA motif according to a preset base type and base probability matrix.

[0087] In step S201 of some embodiments, the convolutional network includes at least one convolutional layer, at least one batch normalization layer, and at least one first activation layer. For example, in a specific embodiment, the convolutional network includes two convolutional layers, two batch normalization layers, and two first activation layers. Through two convolutional layers to batch normalization layers and batch normalization layers to first activation layers, feature extraction of sample base distribution states is achieved to obtain a sample base distribution matrix.

[0088] In step S202 of some embodiments, the activation network includes a first fully connected layer, a second activation layer, and a prediction layer. The activation network is used to activate the sample base distribution matrix, map the sample base distribution matrix to a vector space of a specific dimension, and perform distribution probability calculation on the sample base distribution matrix of a fixed dimension to obtain a base probability matrix.

[0089] In step S203 of some embodiments, the preset base types include four types, namely adenine (A); guanine (G); thymine (T); cytosine (C), and the element value of each point in the base probability matrix represents the probability of the four base types at the position. By converting the base probability matrix from a matrix representation to a base representation, the initial DNA motif can be obtained more conveniently.

[0090] See also Figure 3 In some embodiments, the convolutional network includes a convolutional layer, a batch normalization layer, and a first activation layer. Step S201 may include but is not limited to steps S301 to S303:

[0091] Step S301, extracting features of the sample base distribution state through a convolutional layer to obtain an initial base distribution feature map;

[0092] Step S302, normalizing the initial base distribution feature map through a batch normalization layer to obtain a target base distribution feature map;

[0093] Step S303, performing activation processing on the target base distribution feature map through the first activation layer to obtain a sample base distribution matrix.

[0094] In step S301 of some embodiments, feature extraction is performed on the sample base distribution state through a convolution layer to capture the spatial distribution state of the sample bases and obtain an initial base distribution feature map.

[0095] In step S302 of some embodiments, the initial base distribution feature map is normalized by a batch normalization layer, so that the initial base distribution feature map is converted from a dimensional form to a dimensionless form to obtain a target base distribution feature map.

[0096] In step S303 of some embodiments, the target base distribution feature map is activated by the activation function of the first activation layer, wherein the activation function may be a sigmoid function, a tan function, etc. The target base distribution feature map is transformed by the sigmoid function to obtain a sample base distribution matrix.

[0097] See also Figure 4 In some embodiments, the activation network includes a first fully connected layer, a second activation layer, and a prediction layer. Step S202 may include, but is not limited to, steps S401 to S403:

[0098] Step S401, mapping the sample base distribution matrix to a preset vector space through a first fully connected layer to obtain a sample base distribution vector;

[0099] Step S402, performing dimension-changing processing on the sample base distribution vector through the second activation layer to obtain a target base distribution vector;

[0100] Step S403, calculating the distribution probability of the target base distribution vector through the prediction layer to obtain a base probability matrix.

[0101] In step S401 of some embodiments, the vector dimension of the preset vector space can be set according to the application scenario and actual situation, and the sample base distribution matrix is ​​mapped to the preset vector space through the first fully connected layer, so that the distribution of the sample bases is converted from matrix form to vector form, and the sample base distribution vector is obtained.

[0102] In step S402 of some embodiments, the sample base distribution vector is subjected to dimension transformation processing through the second activation layer to achieve mapping of the sample base distribution vector from a low-dimensional space to a high-dimensional space, thereby converting the low-dimensional sample base distribution vector into a high-dimensional target base distribution vector.

[0103] In step S403 of some embodiments, the prediction layer includes a prediction function, which may be a softmax function. The softmax function can be used to calculate the distribution probability of the target base distribution vector, and calculate the probability of each type of base in the target base distribution vector at each preset position to obtain a base probability matrix, wherein the size of the base probability matrix is ​​M*4, where M is the base length. In the base probability matrix, the sum of the probabilities of each row is 1. The base probability matrix can represent the probabilities of the four bases that constitute DNA in these M positions. Therefore, the base probability matrix can be one of the representations of the initial DNA motif.

[0104] For example, assuming that an initial DNA motif includes three positions (M=3), the initial DNA motif is represented in matrix form, and the base probability matrix can be expressed as:

[0105]

[0106] See also Figure 5 In some embodiments, step S104 may include but is not limited to steps S501 to S503:

[0107] Step S501, performing base statistics on the sample DNA sequence fragment in a reinforcement learning environment to obtain a base count matrix;

[0108] Step S502, in a reinforcement learning environment, a divergence score calculation is performed on each preset site according to the base count matrix and the initial DNA motif to obtain a site score value;

[0109] Step S503: summing the site scores to obtain a motif evaluation score.

[0110] In step S501 of some embodiments, different types of bases at each position of the sample DNA sequence fragment are accumulated to obtain a base count matrix. For example, the input sample DNA fragment includes AGT, AGC, GAC, ACG, and the base length is M=3, and the base count matrix can be expressed as

[0111]

[0112] In step S502 of some embodiments, a KDIC scoring algorithm (discrete simulation algorithm of Kullback–Leibler divergence) is introduced as an evaluation method, and a divergence score calculation is performed on each preset site through the KDIC scoring algorithm, the base count matrix and the initial DNA motif to obtain a site score value KDIC. Specifically, the calculation process can be expressed as shown in formula (1):

[0113]

[0114] Where N is the number of sample DNA sequence fragments; k is the type of base (k=4); x i is the number of times the i-th base appears at the current site; q i It is the proportion of the i-th base in the entire input DNA set, which can be obtained by counting the total number of bases in the input sample DNA sequence fragments and the total number of each base.

[0115] In step S503 of some embodiments, since the base length of the initial DNA motif is M, the score of the entire initial DNA motif should be the sum of the scores of each site, and the site scores are summed to obtain a motif evaluation score, which is an evaluation of whether the initial DNA motif can represent the sample DNA sequence fragment. Therefore, the motif evaluation score is used as a reward or return G for reinforcement learning.

[0116] In step S105 of some embodiments, the gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score, so as to train the original reinforcement learning model and obtain the motif search model. Specifically, when updating the gradient parameters, the policy gradient principle can be used to update the parameters. The parameter update process can be expressed as shown in formula (2):

[0117]

[0118] Among them, π(θ) refers to the preset gradient strategy, S refers to the sample base distribution state, A is the base probability matrix (i.e., the initial DNA motif), θ is the gradient parameter, and G is the reward, i.e., the motif evaluation score.

[0119] The gradient parameter θ is continuously modified by the above formula (2) to make the entire function optimal, and the gradient parameter that makes the entire function optimal is used as the final gradient parameter, thereby realizing the training of the original reinforcement model and obtaining the motif search model.

[0120] The above-mentioned motif search model can be used to instruct the intelligent agent to respond to the target base distribution state output by the reinforcement learning environment, and output the target DNA motif according to the target base distribution state.

[0121] The model training method based on reinforcement learning in the embodiment of the present application obtains a sample DNA sequence fragment and a preset original reinforcement learning model, and inputs the sample DNA sequence fragment into the original reinforcement learning model for state extraction, thereby extracting the sample base distribution state from the reinforcement learning environment. This method can easily obtain the specific arrangement of bases on the sample DNA sequence. Further, the sample base distribution state is processed by the intelligent agent to obtain an initial DNA motif, and the initial DNA motif and the sample DNA sequence fragment are rewarded and calculated in the reinforcement learning environment to obtain a motif evaluation score. This method can search for the corresponding initial DNA motif according to the sample base distribution state, and score and calculate the initial DNA motif, and quantitatively evaluate the quality of the initial DNA motif. Finally, the gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score, thereby realizing the training of the original reinforcement learning model and obtaining a motif search model. This method can feed back the motif evaluation score to the original reinforcement learning model, so that the intelligent agent continuously optimizes the gradient parameters and learns according to the gradient parameters, thereby obtaining a motif search model. Compared with traditional neural network models, the model of the embodiment of the present application can realize model training without relying on label data, and can obtain a better DNA motif by inputting a small amount of sample DNA fragments, thereby improving the training effect of the model.

[0122] See also Figure 6 The present application also provides a motif search method, which includes but is not limited to steps S601 to S602:

[0123] Step S601, obtaining a target DNA sequence fragment to be processed;

[0124] Step S602: input the target DNA sequence fragment into a motif search model for motif search processing to obtain a target DNA motif, wherein the motif search model is trained according to the model training method of the first aspect embodiment.

[0125] In step S601 of some embodiments, a target DNA sequence fragment to be processed is obtained, where the target DNA sequence fragment is a DNA sequence for which a potential DNA motif needs to be searched.

[0126] In step S602 of some embodiments, the target DNA sequence fragment is input into the motif search model for motif search processing, the state of the target DNA sequence fragment is extracted through the reinforcement learning environment of the motif search model, and the base distribution state of the target DNA sequence fragment is convoluted and activated by the intelligent agent to determine the corresponding target DNA motif.

[0127] The motif search method of the embodiment of the present application performs motif search on the target DNA sequence fragment through a motif search model based on reinforcement learning, and introduces reinforcement learning and deep neural network at the same time, which can effectively improve the extraction capability of the input DNA sequence fragment. At the same time, compared with the heuristic search algorithm in the traditional technology, the use of neural network model for motif search can also greatly improve the efficiency of DNA motif search.

[0128] See also Figure 7 The embodiment of the present application also provides a model training device based on reinforcement learning, which can implement the above-mentioned model training method based on reinforcement learning, and the device includes:

[0129] A first acquisition module 701 is used to acquire a sample DNA sequence fragment and a preset original reinforcement learning model, wherein the original reinforcement learning model includes a reinforcement learning environment and an intelligent agent;

[0130] A state extraction module 702 is used to input the sample DNA sequence fragment into the original reinforcement learning model for state extraction, so as to extract the sample base distribution state from the reinforcement learning environment;

[0131] A first motif search module 703 is used to perform motif search processing on the sample base distribution state through an agent to obtain an initial DNA motif; wherein the agent includes a convolutional network and an activation network;

[0132] A reward calculation module 704 is used to calculate rewards for the initial DNA motif and the sample DNA sequence fragment in a reinforcement learning environment to obtain a motif evaluation score;

[0133] The training module 705 is used to update the gradient parameters of the original reinforcement learning model according to the motif evaluation score to train the original reinforcement learning model and obtain the motif search model.

[0134] In some embodiments, the agent includes a convolutional network and an activation network, and the first motif search module 703 includes:

[0135] A convolution unit, used to perform convolution processing on the sample base distribution state through a convolution network to obtain a sample base distribution matrix;

[0136] An activation unit, used to activate the sample base distribution matrix through an activation network to obtain a base probability matrix;

[0137] The initial DNA motif generating unit is used to obtain the initial DNA motif according to the preset base type and base probability matrix.

[0138] In some embodiments, the convolutional network includes a convolutional layer, a batch normalization layer, and a first activation layer, and the convolutional unit includes:

[0139] A feature extraction subunit is used to extract features of the sample base distribution state through a convolutional layer to obtain an initial base distribution feature map;

[0140] A normalization subunit, used to normalize the initial base distribution feature map through a batch normalization layer to obtain a target base distribution feature map;

[0141] The activation subunit is used to activate the target base distribution feature map through the first activation layer to obtain a sample base distribution matrix.

[0142] In some embodiments, the activation network includes a first fully connected layer, a second activation layer, and a prediction layer, and the activation unit includes:

[0143] A mapping subunit, used to map the sample base distribution matrix to a preset vector space through a first fully connected layer to obtain a sample base distribution vector;

[0144] A dimension-changing subunit, used for performing dimension-changing processing on the sample base distribution vector through the second activation layer to obtain a target base distribution vector;

[0145] The probability calculation subunit is used to calculate the distribution probability of the target base distribution vector through the prediction layer to obtain the base probability matrix.

[0146] In some embodiments, the reward calculation module 704 includes:

[0147] A statistical unit is used to perform base statistics on sample DNA sequence fragments in a reinforcement learning environment to obtain a base count matrix;

[0148] A scoring calculation unit, used to calculate the divergence score of each preset site according to the base count matrix and the initial DNA motif in the reinforcement learning environment to obtain a site score value;

[0149] The summing unit is used to sum the site score values ​​to obtain the motif evaluation score.

[0150] The specific implementation of the reinforcement learning-based model training device is basically the same as the specific implementation of the above-mentioned reinforcement learning-based model training method, and will not be repeated here.

[0151] See also Figure 8 The present application also provides a motif search device that can implement the above motif search method. The device includes:

[0152] The second acquisition module 801 is used to acquire the target DNA sequence fragment to be processed;

[0153] The second motif search module 802 is used to input the target DNA sequence fragment into the motif search model for motif search processing to obtain the target DNA motif, wherein the motif search model is trained according to the model training method of the first aspect embodiment.

[0154] The specific implementation of the motif search device is substantially the same as the specific implementation of the motif search method described above, and will not be described in detail herein.

[0155] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned motif search method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0156] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0157] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0158] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the model training method or motif search method based on reinforcement learning in the embodiments of this application;

[0159] Input / output interface 903, used to implement information input and output;

[0160] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0161] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0162] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0163] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned reinforcement learning-based model training method or motif search method.

[0164] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The reinforcement learning-based model training method, motif search method, reinforcement learning-based model training device, motif search device, electronic device and medium provided in the embodiments of the present application obtain sample DNA sequence fragments and a preset original reinforcement learning model, and input the sample DNA sequence fragments into the original reinforcement learning model for state extraction, thereby extracting the sample base distribution state from the reinforcement learning environment. This method can easily obtain the specific arrangement of bases on the sample DNA sequence. Further, the sample base distribution state is subjected to motif search processing by an intelligent agent to obtain an initial DNA motif, and the initial DNA motif and the sample DNA sequence fragment are rewarded and calculated in the reinforcement learning environment to obtain a motif evaluation score. This method can search for the corresponding initial DNA motif according to the sample base distribution state, and score and calculate the initial DNA motif, and quantitatively evaluate the quality of the initial DNA motif. Finally, the gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score, so as to realize the training of the original reinforcement learning model and obtain the motif search model. This method can feed back the motif evaluation score to the original reinforcement learning model, so that the intelligent agent continuously optimizes the gradient parameters and learns according to the gradient parameters, thereby obtaining the motif search model. Compared with the traditional neural network model, the model of the embodiment of the present application can realize the training of the model without relying on label data, and can obtain a better DNA motif when a small amount of sample DNA fragments are input, thereby improving the training effect of the model.

[0166] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0167] It can be understood by those skilled in the art that Figure 1-6 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0170] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0171] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0176] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model training method based on reinforcement learning, characterized in that: The method comprises: Obtaining a sample DNA sequence fragment and a preset original reinforcement learning model, wherein the original reinforcement learning model includes a reinforcement learning environment and an agent, and the reinforcement learning environment is a sample DNA sequence fragment; Inputting the sample DNA sequence fragment into the original reinforcement learning model for state extraction to extract the sample base distribution state from the reinforcement learning environment; The intelligent agent performs motif search processing on the base distribution state of the sample to obtain an initial DNA motif; wherein the intelligent agent includes a convolutional network and an activation network; Performing reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score; The gradient parameters of the original reinforcement learning model are updated according to the motif evaluation score to train the original reinforcement learning model to obtain a motif search model.

2. The model training method according to claim 1, characterized in that: The intelligent agent includes a convolutional network and an activation network. The step of performing motif search processing on the sample base distribution state by the intelligent agent to obtain an initial DNA motif includes: Performing convolution processing on the sample base distribution state through the convolution network to obtain a sample base distribution matrix; Performing activation processing on the sample base distribution matrix through the activation network to obtain a base probability matrix; The initial DNA motif is obtained according to the preset base type and the base probability matrix.

3. The model training method according to claim 2, characterized in that: The convolutional network includes a convolutional layer, a batch normalization layer, and a first activation layer. The step of performing convolution processing on the sample base distribution state through the convolutional network to obtain a sample base distribution matrix includes: Extracting features of the sample base distribution state through the convolution layer to obtain an initial base distribution feature map; Normalizing the initial base distribution feature map through the batch normalization layer to obtain a target base distribution feature map; The target base distribution characteristic map is activated by the first activation layer to obtain the sample base distribution matrix.

4. The model training method according to claim 2, characterized in that: The activation network includes a first fully connected layer, a second activation layer, and a prediction layer. The step of performing activation processing on the sample base distribution matrix through the activation network to obtain a base probability matrix includes: Mapping the sample base distribution matrix to a preset vector space through the first fully connected layer to obtain a sample base distribution vector; Performing dimension-changing processing on the sample base distribution vector through the second activation layer to obtain a target base distribution vector; The prediction layer calculates the distribution probability of the target base distribution vector to obtain the base probability matrix.

5. The model training method according to any one of claims 1 to 4, characterized in that: The step of performing reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score comprises: Performing base statistics on the sample DNA sequence fragments in the reinforcement learning environment to obtain a base count matrix; In the reinforcement learning environment, a divergence score calculation is performed on each preset site according to the base count matrix and the initial DNA motif to obtain a site score value; The site scoring values ​​are summed to obtain the motif evaluation score.

6. A motif search method, characterized in that: The method comprises: Obtaining a target DNA sequence fragment to be processed; The target DNA sequence fragment is input into a motif search model for motif search processing to obtain a target DNA motif, wherein the motif search model is trained according to the model training method according to any one of claims 1 to 5.

7. A model training device based on reinforcement learning, characterized in that: The device comprises: A first acquisition module is used to acquire a sample DNA sequence fragment and a preset original reinforcement learning model, wherein the original reinforcement learning model includes a reinforcement learning environment and an agent, and the reinforcement learning environment is a sample DNA sequence fragment; A state extraction module, used for inputting the sample DNA sequence fragment into the original reinforcement learning model for state extraction, so as to extract the sample base distribution state from the reinforcement learning environment; A first motif search module, used to perform motif search processing on the base distribution state of the sample through the intelligent agent to obtain an initial DNA motif; wherein the intelligent agent includes a convolutional network and an activation network; A reward calculation module, used to perform reward calculation on the initial DNA motif and the sample DNA sequence fragment in the reinforcement learning environment to obtain a motif evaluation score; A training module is used to update the gradient parameters of the original reinforcement learning model according to the motif evaluation score to train the original reinforcement learning model to obtain a motif search model.

8. A motif search device, characterized in that: The device comprises: The second acquisition module is used to acquire the target DNA sequence fragment to be processed; The second motif search module is used to input the target DNA sequence fragment into a motif search model for motif search processing to obtain a target DNA motif, wherein the motif search model is trained according to the model training method according to any one of claims 1 to 5.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the reinforcement learning-based model training method as described in any one of claims 1 to 5 or the motif search method as described in claim 6 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the reinforcement learning-based model training method as described in any one of claims 1 to 5 or the motif search method as described in claim 6.

Images