Model training method and device based on reinforcement learning, electronic equipment and medium

Through the reinforcement learning method, the network parameters of the DNA model search model are updated, and the problem of poor convergence in the training of DNA model search model in the prior art is solved, and efficient DNA model search is achieved.

CN114974423BActive Publication Date: 2024-11-19PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210600455.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-30
Publication Date
2024-11-19
Estimated Expiration
2042-05-30

AI Technical Summary

Technical Problem

The prior art is difficult to effectively train DNA model search models, especially in the absence of labeled data, the model training has poor convergence and it is difficult to accurately search DNA model.

Method used

Using reinforcement learning-based method, DNA sequence fragments are extracted and rewarded by initial action selection network and action evaluation network, network parameters are updated, and target reinforcement learning model is constructed to reduce sample random interference and improve the convergence of action selection network sex.

Benefits of technology

It realizes that without labeling data, the convergence and accuracy of the DNA model search model is improved, the random interference of model training is reduced, and the efficiency of DNA model search is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114974423B_ABST
    Figure CN114974423B_ABST
Patent Text Reader

Abstract

The present embodiment provides a model training method and device based on reinforcement learning, an electronic device and a medium, belonging to the field of artificial intelligence technology, the method comprising: obtaining and inputting a sample DNA sequence fragment into a preset initial action selection network and an initial action evaluation network; performing state extraction and base distribution selection processing on the sample DNA sequence fragment through the initial action selection network to obtain an initial DNA motif; performing reward calculation on the sample base distribution and the initial DNA motif through the initial action evaluation network to obtain multiple initial evaluation values; updating the evaluation network parameters of the initial action evaluation network according to the multiple initial evaluation values ​​to obtain a target action evaluation network; performing reward calculation on the sample base distribution and the initial DNA motif through the target action evaluation network to obtain a target evaluation value; updating the selection network parameters of the initial action selection network according to the target evaluation value to obtain a target reinforcement learning model. The target reinforcement learning model constructed by the present disclosed embodiment is more convergent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method based on reinforcement learning, a DNA motif search method and device, an electronic device and a medium. Background Art

[0002] DNA motifs are helpful in determining the potential mechanisms of biological functions. Therefore, in order to regulate the potential mechanisms of biological functions, it is necessary to search for potential DNA motifs in DNA fragments, and it is necessary to construct a motif search model for DNA motif search. Among them, the general search model requires a large amount of annotated data for model training, but the DNA motifs in DNA fragments are uncertain, and it is difficult to use annotated data for model training. Therefore, how to train the model to obtain a motif search model has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method based on reinforcement learning, a DNA motif search method and device, an electronic device and a medium, which can improve the convergence of the DNA motif search model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method based on reinforcement learning, the model training method comprising:

[0005] Obtaining a sample DNA sequence fragment; wherein the sample DNA sequence fragment is used as an environment for reinforcement learning;

[0006] Inputting the sample DNA sequence fragment into a preset original reinforcement learning model; wherein the original reinforcement learning model includes: an initial action selection network and an initial action evaluation network;

[0007] The sample DNA sequence fragment is subjected to state extraction by the initial action selection network to obtain a sample base distribution, and the sample base distribution is subjected to base distribution selection processing by the initial action selection network to obtain an initial DNA motif; wherein the sample base distribution is used as a state of reinforcement learning, and the initial DNA motif is used as an action of reinforcement learning;

[0008] Performing reward calculation on the sample base distribution and the initial DNA motif through the initial action evaluation network to obtain a plurality of initial evaluation values;

[0009] Update the evaluation network parameters of the initial action evaluation network according to the multiple initial evaluation values ​​to obtain a target action evaluation network;

[0010] Performing reward calculation on the sample base distribution and the initial DNA motif through the target action evaluation network to obtain a target evaluation value;

[0011] The selection network parameters of the initial action selection network are updated according to the target evaluation value to obtain a target action selection network and a target reinforcement learning model; the target reinforcement learning model includes the target action evaluation network and the target action selection network.

[0012] In some embodiments, updating the evaluation network parameters of the initial action evaluation network according to the plurality of initial evaluation values ​​to obtain the target action evaluation network includes:

[0013] Calculate the difference between any two of the initial evaluation values ​​to obtain an evaluation difference;

[0014] The evaluation network parameters of the initial action evaluation network are updated according to the evaluation difference to obtain the target action evaluation network.

[0015] In some embodiments, updating the selection network parameters of the initial action selection network according to the target evaluation value to obtain the target action selection network includes:

[0016] Performing gradient calculation on the target evaluation value according to a preset policy gradient algorithm to obtain a parameter gradient value;

[0017] The selection network parameters of the initial action selection network are updated according to the parameter gradient values ​​to obtain the target action selection network.

[0018] In some embodiments, after the initial action evaluation network performs reward calculation on the sample base distribution and the initial DNA motif to obtain an initial evaluation value, the model training method further includes:

[0019] Construct a network training data set, including:

[0020] Matching the initial evaluation value with the initial DNA motif to obtain an action-value pair;

[0021] The network training data set is constructed according to the action-value pairs and the sample DNA sequence fragments.

[0022] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a DNA motif search method, the DNA motif search method comprising:

[0023] Obtaining a target DNA sequence fragment to be processed;

[0024] Inputting the target DNA sequence fragment into the target reinforcement learning model; the target reinforcement learning model includes a target action selection network; the target reinforcement learning model is obtained according to the model training method described in the first aspect;

[0025] The target action selection network is used to perform motif search processing to obtain a target DNA motif.

[0026] In some embodiments, the step of performing motif search processing through the target action selection network to obtain a target DNA motif includes:

[0027] Extracting the state of the target DNA sequence fragment through the target action selection network to obtain a target base distribution;

[0028] Performing base distribution selection processing on the target base distribution through the target action selection network to obtain a probability matrix of the target base; wherein the target base is a base for constructing the target DNA sequence fragment;

[0029] Probability calculation is performed according to the probability matrix to obtain a probability value; the probability value is used to characterize the probability of the target base;

[0030] The target DNA motif is screened out from the target base according to the probability value.

[0031] To achieve the above purpose, a third aspect of an embodiment of the present application proposes a model training device based on reinforcement learning, the model training device comprising:

[0032] A sample acquisition module is used to acquire a sample DNA sequence fragment; wherein the sample DNA sequence fragment is used as an initial action selection network initial action evaluation network of the reinforcement learning environment;

[0033] A sample input module, used to input the sample DNA sequence fragment into a preset original reinforcement learning model; wherein the original reinforcement learning model includes: an initial action selection network and an initial action evaluation network;

[0034] The initial action selection network is used to extract the state of the sample DNA sequence fragment to obtain a sample base distribution, and perform base distribution selection processing on the sample base distribution to obtain an initial DNA motif; wherein the sample base distribution is used as the state of reinforcement learning, and the initial DNA motif is used as the action of reinforcement learning;

[0035] The initial action evaluation network is used to perform reward calculation on the sample base distribution and the initial DNA motif to obtain multiple initial evaluation values;

[0036] An evaluation updating module, used for updating evaluation network parameters of the initial action evaluation network according to the plurality of initial evaluation values ​​to obtain a target action evaluation network;

[0037] The target action evaluation network is used to perform reward calculation on the sample base distribution and the initial DNA motif to obtain a target evaluation value;

[0038] A selection update module is used to update the selection network parameters of the initial action selection network according to the target evaluation value, obtain a target action selection network, and obtain a target reinforcement learning model; the target reinforcement learning model includes the target action evaluation network and the target action selection network.

[0039] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present application provides a DNA motif search device, the DNA motif search device comprising:

[0040] A target acquisition module is used to acquire a target DNA sequence fragment to be processed;

[0041] A target input module, used to input the target DNA sequence fragment into the target reinforcement learning model; the target reinforcement learning model includes a target action selection network; the target reinforcement learning model is obtained according to the model training method as described in the first aspect;

[0042] The target action selection network is used to perform motif search processing to obtain a target DNA motif.

[0043] To achieve the above-mentioned purpose, a fifth aspect of an embodiment of the present application provides an electronic device, including:

[0044] at least one memory;

[0045] at least one processor;

[0046] at least one program;

[0047] The program is stored in the memory, and the processor executes at least one program to implement:

[0048] The model training method based on reinforcement learning as described in the first aspect, or the steps of the DNA motif searching method as described in the second aspect.

[0049] To achieve the above-mentioned purpose, a sixth aspect of the embodiments of the present application provides a storage medium, wherein the storage medium is a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and wherein the computer-executable instructions are used to enable a computer to execute:

[0050] The model training method based on reinforcement learning as described in the first aspect, or the steps of the DNA motif searching method as described in the second aspect.

[0051] The reinforcement learning-based model training method, DNA motif search method and device, electronic device and medium proposed in the present application update the selection network parameters of the initial action selection network through the target evaluation value output by the target action evaluation network to obtain the target action selection network, thereby reducing the interference of the randomness of a single sample DNA sequence fragment on the network training, reducing the jumpiness of the initial action selection network when adjusting the selection network parameters, and the obtained target action selection network is more convergent. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of a model training method based on reinforcement learning provided in an embodiment of the present application;

[0053] Figure 2 It is a structural diagram of an action selection network in a model training method based on reinforcement learning provided in an embodiment of the present application;

[0054] Figure 3 It is a structural diagram of an action evaluation network in a model training method based on reinforcement learning provided in an embodiment of the present application;

[0055] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0056] Figure 5 yes Figure 1 Flow chart of step S107 in FIG.

[0057] Figure 6 is a flowchart of a model training method based on reinforcement learning provided in another embodiment of the present application;

[0058] Figure 7 is a flow chart of the DNA motif search method provided in an embodiment of the present application;

[0059] Figure 8 yes Figure 7 Flow chart of step S703 in FIG.

[0060] Fig. 9 It is a module block diagram of a model training device based on reinforcement learning provided in an embodiment of the present application;

[0061] Fig.10 is a module block diagram of a DNA motif search device provided in an embodiment of the present application;

[0062] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, image data uploaded by the user, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with the relevant laws, regulations and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0067] First, some nouns involved in this application are analyzed:

[0068] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0070] Reinforcement Learning (RL): Also known as reinforcement learning, evaluation learning or enhanced learning, it is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals through learning strategies during its interaction with the environment. The goal of reinforcement learning is to learn the mapping from environmental states to behaviors, so that the behaviors selected by the agent can obtain the maximum reward from the environment, and the external environment's evaluation of the learning system (or the operating performance of the entire system) is optimal in a certain sense.

[0071] Actor-Critic with Experiment Replay (ACER): Actor-Critic with Experiment Replay is a reinforcement learning method that combines actor / evaluator, self-benefit, and different-policy learning. ACER is an off-policy Actor-Critic model with experience replay, which can use the trust region policy optimization method of off-policy training, greatly improving sample efficiency and reducing data correlation.

[0072] DNA motif: DNA motifs are short (usually 8-20 bp) fragments on the genome (DNA). These short sequences are enriched in DNA fragments with the same biological function (such as the promoter region of genes and around transcription factor binding sites).

[0073] DNA motif finding or DNA motif discovery: refers to the search for potential DNA motifs from a specific set of DNA fragments (usually fragments with the same biological function, such as the promoter region of a gene or transcription factor binding sites obtained by experimental means).

[0074] Activation: The activation operation on the input data is actually a function transformation, which is performed element by element. Commonly used activation functions include sigmoid function, tanh function, relu function, etc.

[0075] Normalization: There are two forms of normalization methods, one is to convert the number into a decimal between (0, 1), and the other is to convert the dimensional expression into a dimensionless expression. It is mainly proposed for the convenience of data processing. Mapping the data to the range of 0 to 1 for processing is more convenient and faster, and should be classified as part of digital signal processing.

[0076] Softmax function: The Softmax function is a normalized exponential function that can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector so that the range of each element is between (0, 1) and the sum of all elements is 1. This function is often used in multi-classification problems.

[0077] PWM matrix (position weight matrix): PWM matrix is ​​a way to represent DNA motifs. Its full name is position-specific weight matrix (PSWM) or position-specific scoring matrix (PSSM). To construct a PWM matrix, we must first obtain the position frequency matrix (PFM), which is the number of times the four nucleotides appear at each position.

[0078] Searching for potential DNA motifs in DNA fragments with the same biological function helps to determine the potential mechanism regulating such biological function. Therefore, searching for DNA motifs from DNA fragments is crucial for the study of transcriptional regulation. Since the DNA fragments input for DNA motif search are usually about several thousand DNA frequency bands, each DNA fragment is about tens to hundreds of bp long, and is characterized by no label marking where it contains DNA motifs. In the related art, heuristic search is mainly used for DNA motif search, and the entire search process needs to be repeated many times to ensure that the final result does not fall into the dilemma of local optimality, and the heuristic search algorithm is time-consuming and cannot guarantee the optimal result. In addition, supervised learning in machine learning has also been widely used in the field of information search, but in reality, the annotation of DNA sequences in DNA motif search requires the characteristics of DNA motifs to be known in advance. Since it is difficult to clearly know the characteristics of DNA motifs in DNA sequences, and the DNA motifs to be searched each time are different, supervised learning is difficult to apply to DNA motif search.

[0079] Based on this, the present application adopts reinforcement learning for model training, takes sample DNA sequence fragments as the environment of reinforcement learning, takes sample base distribution as the state of reinforcement learning, and takes initial DNA motif as the action of reinforcement learning, thereby training to obtain a target reinforcement learning model, so as to search for DNA motifs through the target reinforcement learning model, without setting labels for DNA sequence fragments in advance, and can also realize the search for DNA motifs. Among them, the constructed target reinforcement learning model first updates the evaluation network parameters of the initial action evaluation network to obtain the target action evaluation network, and then updates the initial action selection network with the target evaluation value obtained by the target action evaluation network to obtain the target action selection network, so as to construct a more convergent target reinforcement learning model.

[0080] The model training method, DNA motif search method and device, electronic device and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the DNA motif search method in the embodiments of the present application is described.

[0081] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0082] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0083] The DNA motif search method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The DNA motif search method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the motif search method, etc., but is not limited to the above forms.

[0084] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0085] Figure 1 is an optional flowchart of the model training method based on reinforcement learning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0086] Step S101: Obtain a sample DNA sequence fragment; wherein the sample DNA sequence fragment is used as an environment for reinforcement learning.

[0087] It should be noted that by applying reinforcement learning to the field of DNA motif search, this application uses the ACER algorithm to introduce it into DNA motif search, and it is necessary to map the framework involved in ACER and the parameters corresponding to DNA motif search. Among them, the framework involved in ACER mainly includes environment, state, action, reward and experience pool, and the environment of reinforcement learning corresponds to the sample DNA sequence fragment.

[0088] Step S102: inputting the sample DNA sequence fragment into a preset original reinforcement learning model; wherein the original reinforcement learning model includes: an initial action selection network and an initial action evaluation network.

[0089] It should be noted that the reinforcement learning model for DNA motif search is constructed through reinforcement learning. Therefore, a preset original reinforcement learning model is first constructed, and the original reinforcement learning model is updated according to the sample base distribution of the sample DNA sequence fragment to obtain a target reinforcement learning model that can be used for DNA motif search.

[0090] The original reinforcement learning model constructed includes: initial action selection network and initial action evaluation network. Please refer to Figure 2 and Figure 3 , Figure 2 The schematic diagram of the action selection network is shown in Figure 2. Figure 3 This is a schematic diagram of the structure of the action evaluation network, where the structure of the initial action selection network is as follows Figure 2 As shown in the figure, the initial action selection network takes the sample base distribution obtained from the sample DNA sequence fragment as input and the initial DNA motif as output. The initial action evaluation network takes the sample base obtained from the sample DNA sequence fragment and the initial DNA motif as input and the evaluation of the initial DNA motif as output. By constructing the original reinforcement learning model including the initial action selection network and the initial action evaluation network, the model training is made easier and the overfitting phenomenon in the model training is reduced.

[0091] Step S103: extract the state of the sample DNA sequence fragment through the initial action selection network to obtain the sample base distribution, and perform base distribution selection processing on the sample base distribution through the initial action selection network to obtain the initial DNA motif; wherein the sample base distribution is used as the state of reinforcement learning, and the initial DNA motif is used as the action of reinforcement learning.

[0092] It should be noted that, by inputting the sample DNA sequence fragment into the initial action selection network, the initial action selection network first extracts the state of the sample DNA sequence fragment to obtain the sample base distribution. The sample base distribution corresponds to the state of reinforcement learning, and the initial action selection network performs base distribution selection processing with the sample base distribution of the sample DNA sequence fragment to obtain the initial DNA motif. Among them, the present application adopts the ACER algorithm for model training, so it is necessary to pre-set the experience pool, then data can be extracted from the experience pool for experience playback to complete the training of the initial action selection network and the initial action evaluation network. After the initial action selection network outputs the initial DNA motif, the initial DNA motif is stored in the experience pool, and the initial action evaluation network can directly extract the initial DNA motif from the experience pool when it needs to be trained.

[0093] Specifically, the specific structure of the initial action selection network refers to Figure 2 , it can be seen that the initial action selection network includes: at least one first convolutional layer, at least one first normalization layer, at least one first activation layer, at least one first fully connected layer and at least one Softmax layer. For example, refer to Figure 2 , the initial action selection network of this embodiment is composed of two first convolutional layers, one first normalization layer, two first activation layers, two first fully connected layers and a Softmax layer. When the sample base distribution of the input sample DNA sequence fragment is processed by the structure composed of two first convolutional layers, the first normalization layer and the first activation layer, the sample base distribution is extracted to obtain a sample base distribution matrix, and then processed by the first fully connected layer, the first activation layer, the first fully connected layer and the Softmax layer to map the sample base distribution matrix to a vector space of a specific dimension, and calculate the distribution probability of the sample base distribution matrix in a fixed dimension to obtain a base probability matrix, and then determine the initial DNA motif according to the base probability matrix. Among them, there are four types of bases, namely adenine (A); guanine (G); thymine (T); cytosine (C), the columns in the base probability matrix represent the positions of the bases, and the rows in the base probability matrix represent the probability values ​​corresponding to the bases, so the initial DNA motif can be determined by the base probability matrix.

[0094] Step S104: The reward calculation is performed on the sample base distribution and the initial DNA motif through the initial action evaluation network to obtain multiple initial evaluation values.

[0095] It should be noted that, since the sample base distribution is a key factor in the evaluation result of the initial action evaluation network for the initial DNA motif, the input of the initial action evaluation network includes two parts, namely, the sample base distribution and the initial DNA motif. The initial action evaluation network uses the sample base distribution obtained from the sample DNA sequence fragment and the initial DNA motif as input to evaluate the initial DNA motif and obtain multiple initial evaluation values.

[0096] Specifically, in order to adjust the stability of the initial action evaluation network, the initial action evaluation network needs to give multiple initial evaluation values ​​for the same environment, that is, the initial action selection network uses the sample base distribution of the same sample DNA sequence fragment to perform base distribution selection processing to obtain the initial DNA motif, and the initial action evaluation network uses the same sample DNA sequence fragment to evaluate the initial DNA motif multiple times to generate multiple initial evaluation values. This indicates that the randomness of a single sample DNA sequence fragment exists during the model training process, which will affect the initial action evaluation network's inability to accurately evaluate the initial DNA motif, thereby reducing the convergence of the initial action selection network training.

[0097] Step S105: Update the evaluation network parameters of the initial action evaluation network according to multiple initial evaluation values ​​to obtain the target action evaluation network.

[0098] It should be noted that in order to improve the accuracy of the initial action evaluation network's evaluation of the initial DNA motif, it is necessary to update the evaluation network parameters of the initial action evaluation network according to multiple initial evaluation values ​​to obtain the target action evaluation network. The initial action selection network is trained according to the target evaluation values ​​output by the target action evaluation network. The obtained target initial action selection network has better convergence and can reduce the interference of the randomness of the samples on the training of the target initial action selection network.

[0099] Step S106: The target action evaluation network is used to calculate the reward for the sample base distribution and the initial DNA motif to obtain a target evaluation value.

[0100] It should be noted that the initial action evaluation network is learned according to the initial evaluation value generated by itself to obtain the target action evaluation network. The target action evaluation network is updated to calculate the reward for the sample base distribution and the initial DNA motif to obtain the target evaluation value, so as to update the initial evaluation value to the target evaluation value. The initial action selection network can be trained by the target evaluation value, thereby improving the convergence of the initial action selection network training.

[0101] Specifically, the specific structure of the target action evaluation network is consistent with the initial action evaluation network, that is, Figure 3The specific results of the target action evaluation network can be known. The target action evaluation network includes: at least one second convolutional layer, at least one second batch normalization layer, at least one second activation layer, and at least one second fully connected layer. In this embodiment, the target action evaluation network consists of two second convolutional layers, one second batch normalization layer, two second activation layers, and two second fully connected layers. When the input sample base distribution and the initial DNA motif pass through the structure composed of two second convolutional layers, the second batch normalization layer, and the second activation layer, and then pass through the second fully connected layer, the second activation layer, and the second fully connected layer, the sample base distribution is used as the state, and the initial DNA motif is evaluated to output a single value, that is, the target evaluation value is output.

[0102] Step S107: updating the selection network parameters of the initial action selection network according to the target evaluation value, obtaining the target action selection network, and obtaining the target reinforcement learning model; the target reinforcement learning model includes the target action evaluation network and the target action selection network.

[0103] It should be noted that the target action selection network is obtained by updating the selection network parameters of the initial action selection network through the target evaluation value output by the updated target action evaluation network, and the convergence of the target action selection network is better, and the target reinforcement learning model is constructed according to the target action selection network and the target action evaluation network. Therefore, the convergence of the model training for DNA motif search through the target reinforcement learning model is stronger.

[0104] In the steps S101 to S107 implemented in the embodiment of the present application, the sample DNA sequence fragment is subjected to state extraction by the initial action selection network to obtain the sample base distribution, and the sample base distribution is subjected to base distribution selection processing by the initial action selection network to obtain the initial DNA motif, and then the reward calculation is performed by the initial action evaluation network according to the sample base distribution and the initial DNA motif to evaluate the initial DNA motif to obtain multiple initial evaluation values. If there are discrete sample DNA sequence fragments, the evaluation of the initial DNA motif by the initial action evaluation network will be interfered, so the evaluation network parameters of the initial action evaluation network are updated by multiple initial evaluation values ​​to obtain the target action evaluation network, that is, the initial action evaluation network is learned. Finally, the sample base distribution and the initial DNA motif are rewarded and calculated by the target action evaluation network to obtain the target evaluation value, and the selection network parameters of the initial action selection network are updated according to the target evaluation value to obtain the target action selection network, so that the interference of the discrete sample DNA sequence fragments on the training of the target action selection network can be reduced, so that the target action selection network is more converged, and the jump of the selection network parameter adjustment of the initial action selection network is reduced.

[0105] Please refer to Figure 4In some embodiments, step S105 may include but is not limited to step S401 and step S402.

[0106] Step S401: Calculate the difference between any two initial evaluation values ​​to obtain an evaluation difference.

[0107] It should be noted that the initial action evaluation network gives multiple initial evaluation values ​​to the initial DNA motif based on the same sample DNA sequence fragment, and then obtains any two initial evaluation values, and the initial action evaluation network learns based on the evaluation difference between any two initial evaluation values. Therefore, the difference between any two initial evaluation values ​​is calculated to obtain the evaluation difference. Through the evaluation difference, it can be seen that the initial action evaluation network has different evaluations of the initial DNA motif in the same environment, indicating that the initial action evaluation network is affected by the randomness of the sample and it is difficult to stably make an accurate evaluation of the initial DNA motif. The initial action selection network learns based on the initial evaluation value output by the initial action evaluation network, and the parameter adjustment of the initial action selection network is jumpy. Therefore, the initial action evaluation network is updated through the evaluation difference to reduce the jumpiness of the parameter adjustment of the initial action selection network.

[0108] Step S402: updating the evaluation network parameters of the initial action evaluation network according to the evaluation difference to obtain the target action evaluation network.

[0109] It should be noted that the initial action evaluation network is learned according to the evaluation difference to obtain the target action evaluation network. By updating the evaluation network parameters of the initial action evaluation network to obtain a more stable target action evaluation network, the initial action selection network selects the target evaluation value output by the target action evaluation network, and the jump of the network parameter adjustment is reduced, so that the initial action selection network is trained more convergently.

[0110] Specifically, the initial action evaluation network uses the sample base distribution and the initial DNA motif as input, and the sample base distribution and the initial DNA motif are rewarded by the initial action evaluation network to evaluate the initial DNA motif to obtain multiple initial evaluation values. Among them, the initial evaluation value is a real value greater than 0. The initial action evaluation network updates its own evaluation network parameters based on its own multiple initial evaluation values ​​of the initial DNA motif. The evaluation difference is calculated by the following formula (1):

[0111]

[0112] In the formula, U is the initial evaluation value or target evaluation value, G is a constant, Evaluation difference. Therefore, the gradient Δw of the evaluation network parameters of the initial action evaluation network behind the initial evaluation value U is calculated through gradient propagation, and the initial action evaluation network is updated according to the learning rate.

[0113] Update the evaluation network parameters through formula (2):

[0114] w2=w1+lr*Δw (2)

[0115] In the formula, w2 is the updated evaluation network parameter, w1 is the evaluation network parameter before the update, lr is the preset learning rate, and Δw is the gradient of the evaluation network parameter. Therefore, the gradient of the evaluation network parameter is calculated according to the evaluation difference through gradient propagation, and then the evaluation network parameter of the initial action evaluation network is calculated by the gradient and the preset learning rate, so as to update the evaluation network parameter of the initial action evaluation network according to the calculated evaluation network parameter to obtain the target action evaluation network.

[0116] For example, in this application, the initial action evaluation network and the target action evaluation network use the KDIC (discrete analog of Kullback–Leibler divergence) evaluation algorithm to evaluate the initial DNA motif. The initial DNA motif is evaluated by the KDIC evaluation algorithm, and the specific calculation formula of the KDIC evaluation algorithm is shown in the following formula (3):

[0117]

[0118] Where N is the number of sample DNA sequence fragments; k is the type of base (k = 4); x i is the number of times the i-th base appears at the current site; q i It is the proportion of the i-th base in the entire input DNA set, which can be obtained by counting the total number of bases in the input sample DNA sequence fragment and the total number of each base. Therefore, the initial DNA motif is evaluated by the KDIC evaluation algorithm to obtain the initial evaluation value, making the calculation of the initial evaluation value easier.

[0119] Please refer to Figure 5 In some embodiments, step S107 may include but is not limited to step S501 and step S502.

[0120] Step S501: Perform gradient calculation on the target evaluation value according to the preset policy gradient algorithm to obtain the parameter gradient value.

[0121] It should be noted that the target action evaluation network evaluates the initial DNA motif to obtain the target evaluation value, and the initial action selection network learns according to the target evaluation value and the preset policy gradient algorithm. Therefore, the gradient of the selection network parameter is calculated by the preset policy gradient algorithm for the target evaluation value, that is, the parameter gradient value is obtained, and the selection network parameters of the initial action selection network can be updated according to the parameter gradient value, making it easier to adjust the selection network parameters of the action selection network.

[0122] Specifically, the preset policy gradient algorithm is shown in the following formula (4):

[0123]

[0124] In the formula, π(θ) refers to the preset gradient strategy, θ is the gradient parameter, G is the reward, that is, the motif evaluation score, U in formula (4) is a constant, S is the sample DNA sequence fragment, A is the initial DNA motif, and w is the selected network parameter. Therefore, the parameter gradient value can be obtained by inputting the target evaluation value into formula (4).

[0125] Step S502: Update the selection network parameters of the initial action selection network according to the parameter gradient value to obtain the target action selection network.

[0126] It should be noted that the parameter gradient value is the gradient value of the selection network parameter, so the selection network parameter of the initial action selection network is adjusted according to the parameter gradient value to obtain the target action selection network. By obtaining the target action selection network for DNA motif search, the efficiency of DNA motif search is accelerated and the complexity of DNA motif search is reduced.

[0127] In some embodiments, after step S107, the model training method further includes:

[0128] Construct a network training data set.

[0129] It should be noted that in order to facilitate the training of the initial action evaluation network and the initial action selection network to obtain the target action evaluation network and the target action selection network, a network training data set needs to be constructed. Therefore, when training the initial action evaluation network and the initial action selection network, it is only necessary to extract the corresponding sample DNA sequence fragments, initial DNA motifs and sample base distributions from the network training data set.

[0130] Please refer to Figure 6 , construct a network training data set, specifically including but not limited to step S601 and step S602.

[0131] Step S601: Match the initial evaluation value and the initial DNA motif to obtain an action-value pair.

[0132] It should be noted that when the initial action selection network performs base distribution processing on the sample base distribution of the sample DNA sequence fragment to obtain the initial DNA motif, and the initial action evaluation network performs reward calculation on the initial DNA motif and the sample base distribution, that is, the initial DNA motif is evaluated to obtain the initial evaluation value. Among them, the sample base distribution of the same sample DNA sequence fragment is subjected to multiple base distribution processing by the initial action selection network to obtain multiple initial DNA motifs, and the initial action evaluation network evaluates the multiple initial DNA motifs to obtain multiple initial evaluation values, then the initial DNA motifs and initial evaluation values ​​obtained at the same time are matched to obtain an action value pair.

[0133] Step S602: construct a network training data set based on action-value pairs and sample DNA sequence fragments.

[0134] It should be noted that after the action-value pairs are obtained, the action-value pairs and sample DNA sequence fragments are combined to generate a network training data set.

[0135] Specifically, multiple initial evaluation values, multiple initial DNA motifs, and sample DNA sequence fragments are generated according to the pre-set model training parameters to obtain a network training data set. The model training parameters include: motif length I, learning rate lr, discount factor γ, number of learning rounds, and number of steps in each learning round T. The initial DNA motif and initial evaluation value obtained in the i-th training are combined to form an action value pair (Ai, Ri+1), and the network training data set (S, A0, R1, S, ..., S, A) is repeatedly generated within the number of learning rounds according to the set model training parameters. T-1 , R T , S), and then store the network training data set in a preset experience pool. Where S is a sample DNA sequence fragment, A is an initial DNA motif, and R is an initial evaluation value. Therefore, by first constructing a network training data set and storing the network training data set in a preset experience pool, it is convenient for the initial action evaluation model to extract its own initial evaluation value from the experience pool for learning, so as to facilitate the evaluation network parameters of the initial action evaluation network to be updated.

[0136] Please refer to Figure 7 The embodiment of the present application also provides a DNA motif search method, which includes but is not limited to step S701, step S702 and step S703.

[0137] Step S701: Obtain a target DNA sequence fragment to be processed.

[0138] It should be noted that if the target DNA sequence fragment is a DNA sequence fragment for which a DNA motif search is required, wherein the target DNA sequence fragment is input without any annotation, it is not clear whether there is a DNA motif in the target DNA sequence fragment and the position of the DNA motif.

[0139] Step S702: Input the target DNA sequence fragment into the target reinforcement learning model; the target reinforcement learning model includes a target action selection network; the target reinforcement learning model is obtained according to the model training method as described above.

[0140] It should be noted that the target reinforcement learning model obtained by the above-mentioned model training method realizes DNA motif search by inputting the target DNA sequence fragment into the target reinforcement learning model, searching for DNA motifs from the target DNA sequence fragment and determining the position of the DNA motif through the target reinforcement learning model, and the search time is better than that of the heuristic search algorithm.

[0141] Step S703: Perform motif search processing through the target action selection network to obtain the target DNA motif.

[0142] It should be noted that the target reinforcement learning model includes a target action selection network and a target action evaluation network. The target action selection network performs a motif search process on the target DNA sequence fragment to determine whether the target DNA motif exists in the target DNA sequence fragment and determines the position of the target DNA motif in the target DNA sequence fragment to achieve rapid search of the target DNA motif.

[0143] Please refer to Figure 8 In some embodiments, step S703 may include but is not limited to step S801, step S802, step S803 and step S804.

[0144] Step S801: extract the state of the target DNA sequence fragment through the target action selection network to obtain the target base distribution.

[0145] It should be noted that before the target action selection network performs motif search processing on the target DNA sequence fragment, it is necessary to perform state extraction on the target DNA sequence fragment through the target action selection network to extract the base arrangement in the target DNA sequence fragment to obtain the target base distribution. Among them, the target base distribution is used as the state in the environment of the target action selection network.

[0146] Step S802: performing base distribution selection processing on the target base distribution through the target action selection network to obtain a probability matrix of the target base; wherein the target base is the base for constructing the target DNA sequence fragment.

[0147] It should be noted that the target base distribution is input into the target action selection network, and the target action selection network performs base distribution processing on the target base distribution to calculate the position weight of each base to obtain a probability matrix. Among them, the target base is the base that constitutes the target DNA sequence fragment, and the target base in the target DNA sequence fragment may be a random base sequence. The target base is mainly composed of four bases, namely adenine (A); guanine (G); thymine (T); cytosine (C). The position weight of each base is calculated through the target action selection network to obtain the probability matrix PWM, that is, the probability matrix of the target base is obtained.

[0148] Step S803: Probability calculation is performed according to the probability matrix to obtain a probability value; the probability value is used to characterize the probability of the target base.

[0149] It should be noted that since the columns of the probability matrix represent the positions of the bases and the rows represent the probabilities of the bases, the probability values ​​are obtained by performing probability calculation on the probability matrix. The probability values ​​of each target base can be determined through the probability values ​​to determine the probability values ​​of the target bases in the target DNA sequence fragments.

[0150] Step S804: Filter out target DNA motifs from target bases according to the probability values.

[0151] It should be noted that if the probability values ​​of all target bases are less than 0, it means that the target DNA motif does not exist in the target DNA sequence fragment. If the probability value of the target base is greater than or equal to 0, it means that the target base may be a potential DNA motif. Therefore, after calculating the probability value of the target base, the target DNA motif is screened out from the target base by the probability value.

[0152] For example, if the target DNA sequence fragment is as follows:

[0153] ①、ACGAGACTGACGTACTAGGACTACG

[0154] ②、AACTGTAACGTGGCGTTAGTAGATATG

[0155] ③、TGGACTGCAGGTAAAGGCTGATCGATACG

[0156] ④、GAAAAATGTCCTGAAGTAAAGGTC

[0157] ⑤、TAAGAAAAAATGTAAGAAGTAAAGGTC

[0158] Among them, it is impossible to know whether there is a DNA motif based on the target DNA sequence fragment, and it is impossible to know the specific location of the DNA motif. The target base distribution is processed by the target action selection network to obtain the probability matrix of the target base as follows:

[0159]

[0160] The columns represent the positions of the bases, the first row corresponds to the A base, the second row corresponds to the C base, the third row corresponds to the G base, the fourth row corresponds to the T base, and the rows represent the probability of the ACGT base. Through the probability matrix, we can know that the target bases that can be used as target DNA motifs are: TGACGT, TGGCGT, TGTCCT, AGAAGT, TCACGG, where the probability values ​​of each target base can be calculated according to the probability matrix:

[0161] P(TGACGT)=0.857+0.857+0.714+0.857+0.857+0.857=4.999

[0162] P(TGGCGT)=0.857+0.857+0.143+0.857+0.857+0.857=4.428

[0163] P(TGTCCT)=0.857+0.857+0.143+0.857+0.143+0.857=3.714

[0164] P(AGAAGT)=0.143+0.857+0.714+0.143+0.857+0.857=3.571

[0165] P(TCACGG)=0.857+0.143+0.714+0.857+0.857+0.143=3.57

[0166] Therefore, the target base that can be used as the target DNA motif can be determined through the probability matrix. If the target DNA sequence fragment has multiple target bases with probability values, the target base with the highest probability value is determined as the target DNA motif. Therefore, after the target action selection network outputs the probability matrix, it can be determined whether the target DNA sequence fragment has the target DNA motif, and then the target DNA motif is screened out according to the probability value of the target base, thereby realizing the search for the target DNA motif.

[0167] The DNA motif search method of the present application obtains a target reinforcement learning model through model training based on reinforcement learning, and performs motif search processing on a target DNA sequence fragment through a target action selection network to obtain a target DNA motif, so as to realize the target DNA motif search. Therefore, the target action selection network obtained through reinforcement learning training searches for the target DNA motif, which improves the efficiency of the target DNA motif search compared to the heuristic search algorithm.

[0168] Please refer to Fig. 9 The embodiment of the present application further provides a model training device based on reinforcement learning, which can implement the above-mentioned model training method based on reinforcement learning. The model training device includes:

[0169] The sample acquisition module 901 is used to acquire a sample DNA sequence fragment; wherein the sample DNA sequence fragment is used as an initial action selection network initial action evaluation network of the reinforcement learning environment;

[0170] The sample input module 902 is used to input the sample DNA sequence fragment into a preset original reinforcement learning model; wherein the original reinforcement learning model includes: an initial action selection network and an initial action evaluation network;

[0171] The initial action selection network 903 is used to extract the state of the sample DNA sequence fragment to obtain the sample base distribution, and perform base distribution selection processing on the sample base distribution to obtain the initial DNA motif; wherein the sample base distribution is used as the state of reinforcement learning, and the initial DNA motif is used as the action of reinforcement learning;

[0172] An initial action evaluation network 904 is used to calculate rewards for sample base distribution and initial DNA motifs to obtain multiple initial evaluation values;

[0173] An evaluation updating module 905 is used to update the evaluation network parameters of the initial action evaluation network according to the multiple initial evaluation values ​​to obtain a target action evaluation network;

[0174] The target action evaluation network 906 is used to calculate the reward for the sample base distribution and the initial DNA motif to obtain the target evaluation value;

[0175] The selection update module 907 is used to update the selection network parameters of the initial action selection network according to the target evaluation value, obtain the target action selection network, and obtain the target reinforcement learning model; the target reinforcement learning model includes the target action evaluation network and the target action selection network.

[0176] The specific implementation of the reinforcement learning-based model training device is basically the same as the specific implementation of the above-mentioned reinforcement learning-based model training method, and will not be repeated here.

[0177] The model training device based on reinforcement learning of the embodiment of the present application extracts the state of the sample DNA sequence fragment through the initial action selection network to obtain the sample base distribution, and the initial action selection network performs base distribution selection processing on the sample base distribution to obtain the initial DNA motif, and the sample base distribution and the initial DNA motif are rewarded and calculated through the initial action evaluation network, that is, the initial DNA motif is evaluated by the initial action evaluation network to obtain multiple initial evaluation values. The evaluation network parameters of the initial action evaluation network are updated according to the initial evaluation values ​​to obtain the target action evaluation network, and the selection network parameters of the initial action selection network are updated by the target evaluation values ​​output by the target action evaluation network to obtain the target action selection network, thereby reducing the interference of the randomness of a single sample DNA sequence fragment on the training, so that the jumpiness of the initial action selection network when adjusting the selection network parameters is reduced, and the obtained target action selection network is more convergent.

[0178] Please refer to Fig.10 The present application also provides a DNA motif search device that can implement the above-mentioned DNA motif search method. The DNA motif search device includes:

[0179] A target acquisition module 101 is used to acquire a target DNA sequence fragment to be processed;

[0180] A target input module 102 is used to input the target DNA sequence fragment into a target reinforcement learning model; the target reinforcement learning model includes a target action selection network; the target reinforcement learning model is obtained according to the model training method as described above;

[0181] The target DNA motif is obtained by performing motif search processing through the target action selection network.

[0182] The specific implementation of the DNA motif searching device is substantially the same as the specific implementation of the above-mentioned DNA motif searching method, and will not be described in detail herein.

[0183] The DNA motif search device of the embodiment of the present application inputs the target DNA sequence fragment into the target reinforcement learning model obtained by the above model training method, and performs motif search on the target DNA sequence fragment through the target action selection network in the target reinforcement learning model to obtain the target DNA motif. Therefore, the use of reinforcement learning to search for target DNA motifs is more efficient than the heuristic search algorithm.

[0184] In addition, another embodiment of the present application discloses an electronic device, at least one memory;

[0185] at least one processor;

[0186] at least one program;

[0187] The program is stored in the memory, and the processor executes the at least one program to implement the model training method based on reinforcement learning or the DNA motif search method implemented in the present disclosure. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a desktop computer, a personal digital assistant (PDA), a car computer, etc.

[0188] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0189] The processor 111 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure;

[0190] The memory 112 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 112 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 112, and the processor 111 calls and executes the model training method based on reinforcement learning as described above, or the steps of the DNA motif search method as described above in the embodiments of this disclosure;

[0191] Input / output interface 113, used to implement information input and output;

[0192] The communication interface 114 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0193] A bus 115 that transmits information between various components of the device (e.g., processor 111, memory 112, input / output interface 113, and communication interface 114);

[0194] The processor 111 , the memory 112 , the input / output interface 113 and the communication interface 114 are connected to each other in communication within the device via a bus 115 .

[0195] The embodiments of the present disclosure also provide a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions, which are used to enable a computer to execute the above-mentioned reinforcement learning-based model training method, or the steps of the above-mentioned DNA motif search method.

[0196] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0197] The model training method based on reinforcement learning, the DNA motif search method and device, the electronic device and the medium provided by the embodiments of the present disclosure obtain a target action evaluation network by updating the evaluation network parameters of the initial action evaluation network according to multiple initial evaluation values, and then the target action evaluation network calculates the reward according to the sample base distribution and the initial DNA motif to obtain the target evaluation value, and then updates the selection network parameters of the initial action selection network according to the target evaluation value to obtain the target action selection network. Therefore, by updating the evaluation network parameters of the initial action evaluation network to obtain the target action evaluation network, the interference caused by the randomness of a single sample DNA sequence fragment to the network training can be reduced, and the jumpiness of the selection network parameter update of the initial action selection network is reduced, so that the target action selection network is more convergent.

[0198] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0199] It can be understood by those skilled in the art that Figures 1 to 9 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0200] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0202] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0203] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0204] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0207] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0208] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but the scope of the rights of the present disclosure is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present disclosure should be within the scope of the rights of the present disclosure.

Claims

1. A model training method based on reinforcement learning, It is characterized in that The model training method comprises: Obtaining a sample DNA sequence fragment; wherein the sample DNA sequence fragment is used as an environment for reinforcement learning; Inputting the sample DNA sequence fragment into a preset original reinforcement learning model; wherein the original reinforcement learning model includes: an initial action selection network and an initial action evaluation network; The sample DNA sequence fragment is subjected to state extraction by the initial action selection network to obtain a sample base distribution, and the sample base distribution is subjected to base distribution selection processing by the initial action selection network to obtain an initial DNA motif; wherein the sample base distribution is used as a state of reinforcement learning, and the initial DNA motif is used as an action of reinforcement learning; Performing reward calculation on the sample base distribution and the initial DNA motif through the initial action evaluation network to obtain a plurality of initial evaluation values; Update the evaluation network parameters of the initial action evaluation network according to the multiple initial evaluation values ​​to obtain a target action evaluation network; Performing reward calculation on the sample base distribution and the initial DNA motif through the target action evaluation network to obtain a target evaluation value; The selection network parameters of the initial action selection network are updated according to the target evaluation value to obtain a target action selection network and a target reinforcement learning model; the target reinforcement learning model includes the target action evaluation network and the target action selection network.

2. The model training method according to claim 1, It is characterized in that The step of updating the evaluation network parameters of the initial action evaluation network according to the plurality of initial evaluation values ​​to obtain the target action evaluation network comprises: Calculate the difference between any two of the initial evaluation values ​​to obtain an evaluation difference; The evaluation network parameters of the initial action evaluation network are updated according to the evaluation difference to obtain the target action evaluation network.

3. The model training method according to claim 1, It is characterized in that The updating of the selection network parameters of the initial action selection network according to the target evaluation value to obtain the target action selection network includes: Performing gradient calculation on the target evaluation value according to a preset policy gradient algorithm to obtain a parameter gradient value; The selection network parameters of the initial action selection network are updated according to the parameter gradient values ​​to obtain the target action selection network.

4. The model training method according to any one of claims 1 to 3, It is characterized in that After the initial action evaluation network is used to calculate the reward for the sample base distribution and the initial DNA motif to obtain an initial evaluation value, the model training method further includes: Construct a network training data set, including: Matching the initial evaluation value with the initial DNA motif to obtain an action-value pair; The network training data set is constructed according to the action-value pairs and the sample DNA sequence fragments.

5. A DNA motif search method, It is characterized in that The DNA motif searching method comprises: Obtaining a target DNA sequence fragment to be processed; Inputting the target DNA sequence fragment into the target reinforcement learning model; the target reinforcement learning model includes a target action selection network; the target reinforcement learning model is obtained according to the model training method according to any one of claims 1 to 4; The target action selection network is used to perform motif search processing to obtain a target DNA motif.

6. The DNA motif searching method according to claim 5, It is characterized in that The step of performing motif search processing through the target action selection network to obtain a target DNA motif includes: Extracting the state of the target DNA sequence fragment through the target action selection network to obtain a target base distribution; Performing base distribution selection processing on the target base distribution through the target action selection network to obtain a probability matrix of the target base; wherein the target base is a base for constructing the target DNA sequence fragment; Probability calculation is performed according to the probability matrix to obtain a probability value; the probability value is used to characterize the probability of the target base; The target DNA motif is screened out from the target base according to the probability value.

7. A model training device based on reinforcement learning, It is characterized in that The model training device comprises: A sample acquisition module is used to acquire a sample DNA sequence fragment; wherein the sample DNA sequence fragment is used as an initial action selection network initial action evaluation network of the reinforcement learning environment; A sample input module, used to input the sample DNA sequence fragment into a preset original reinforcement learning model; wherein the original reinforcement learning model includes: an initial action selection network and an initial action evaluation network; The initial action selection network is used to extract the state of the sample DNA sequence fragment to obtain a sample base distribution, and perform base distribution selection processing on the sample base distribution to obtain an initial DNA motif; wherein the sample base distribution is used as the state of reinforcement learning, and the initial DNA motif is used as the action of reinforcement learning; The initial action evaluation network is used to perform reward calculation on the sample base distribution and the initial DNA motif to obtain multiple initial evaluation values; An evaluation updating module, used for updating evaluation network parameters of the initial action evaluation network according to the plurality of initial evaluation values ​​to obtain a target action evaluation network; The target action evaluation network is used to perform reward calculation on the sample base distribution and the initial DNA motif to obtain a target evaluation value; A selection update module is used to update the selection network parameters of the initial action selection network according to the target evaluation value, obtain a target action selection network, and obtain a target reinforcement learning model; the target reinforcement learning model includes the target action evaluation network and the target action selection network.

8. A DNA motif search device, It is characterized in that The DNA motif searching device comprises: A target acquisition module is used to acquire a target DNA sequence fragment to be processed; A target input module, used for inputting the target DNA sequence fragment into the target reinforcement learning model; the target reinforcement learning model includes a target action selection network; the target reinforcement learning model is obtained according to the model training method according to any one of claims 1 to 4; The target action selection network is used to perform motif search processing to obtain a target DNA motif.

9. An electronic device, It is characterized in that include: at least one memory; at least one processor; at least one program; The program is stored in the memory, and the processor executes at least one program to implement: The model training method based on reinforcement learning as described in any one of claims 1 to 4, or the steps of the DNA motif searching method as described in any one of claims 5 to 6.

10. A storage medium, wherein the storage medium is a computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute: The model training method based on reinforcement learning as described in any one of claims 1 to 4, or the steps of the DNA motif searching method as described in any one of claims 5 to 6.