Causal network discovery system based on reinforcement learning

Abstract

The application provides a causal network discovery system based on reinforcement learning, takes biomolecules as variables, obtains observation data of the biomolecules, and filters possible parent variables of the variables according to the data to obtain a sparse graph after initial neighborhood selection; uses a graph attention network to extract embedded features of each biomolecule from the observation data of the biomolecules and the sparse graph after the initial neighborhood selection; calculates a weight vector of all biomolecules according to the embedded features of each biomolecule, samples on the weight vector in a loop, and sequentially sorts the extracted variables from back to front; calculates a reward for the sorting according to a scoring function, updates parameters of each network under a reinforcement learning framework according to the reward, and guides generation of variable sorting with a higher reward according to the updated parameters; selects variables for the sorting with the highest reward to obtain a final biomolecular causal regulation network; and the application improves the explainability of a causal graph search process and thus improves the accuracy of causal structure discovery.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bioengineering and artificial intelligence, and particularly relates to a causal network discovery system based on reinforcement learning. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present application and do not necessarily constitute prior art.

[0003] Under the background of the combination of big data and artificial intelligence with various professional disciplines, especially in bioinformatics, the causal discovery task begins to discover the causal interaction relationship behind these variables from the observed data of genes or proteins, etc. How to discover the real and correct variable interaction causal network more quickly and accurately is the main difficulty of current causal discovery in this field.

[0004] The current causal discovery methods are mainly divided into three categories: constraint-based methods, under the assumption of faithfulness, identify the Markov equivalence class of the true causal graph by testing the conditional independence between variables; model-based methods impose additional assumptions on the distribution and function class of data based on a correctly defined causal model, thereby identifying the direction of all edges in the causal graph; score-based methods use scoring functions and search strategies to find the causal graph with the optimal score. Due to the difficulty of searching for a real causal graph, which is a combinatorial optimization problem, recently, the use of continuous functions to represent acyclic constraints of causal graphs has greatly improved the efficiency of causal discovery; considering the powerful search capability of reinforcement learning and the degree of fit between the scoring function and the reward, using reinforcement learning as a search strategy to find the real causal graph is a new emerging method of causal discovery.

[0005] Due to the complexity of high-dimensional causal graphs, the lack of prior knowledge of the causal action relationship of biological molecules, and the existence of unobserved variables, the accuracy of the biological molecule causal regulation network structure discovery technology based on reinforcement learning is not very high. SUMMARY

[0006] In order to solve the problems of the prior art, the present application provides a causal network discovery system based on reinforcement learning, which regards each biological molecule as a variable in the causal relationship, and first selects the initial neighborhood of the variable according to the observed data of the variable to screen the parent-child relationship between the variables, so as to make the search space of the biological molecule causal regulation network more sparse, and thus reduce the calculation difficulty of the causal graph score; the graph attention-based encoding network can capture the structural knowledge on the sparse graph after the initial neighborhood selection, thereby improving the search capability of the causal graph in the sparse graph space; in addition, the ordering of the variables from back to front is very consistent with the generation process of the causal graph, which improves the explainability of the causal graph search process and thus improves the accuracy of the causal structure discovery.

[0007] To achieve the above object, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a causal network discovery system based on reinforcement learning.

[0009] A causal network discovery system based on reinforcement learning comprises:

[0010] The initial neighborhood selection module is configured to: take biomolecules as variables, obtain observation data of the biomolecules, and filter possible parent variables of the variables according to the data to obtain a sparse graph after initial neighborhood selection;

[0011] The feature encoding module is configured to: extract embedded features of each biomolecule from the observation data of the biomolecules and the sparse graph after initial neighborhood selection by using a graph attention network;

[0012] The variable ordering module is configured to: calculate a weight vector of all biomolecules according to the embedded features of each biomolecule, sample on the weight vector in a loop, and sequentially order the extracted variables from back to front;

[0013] The ordering scoring and searching module is configured to: calculate a reward for the ordering generated by the variable ordering module according to a scoring function, update parameters of the feature encoding module and the variable ordering module under a reinforcement learning framework according to the reward, and guide the feature encoding module and the variable ordering module to generate a variable ordering with a higher reward after updating the parameters;

[0014] The variable selection module is configured to: select variables for the ordering with the highest reward to obtain a final biomolecular causal regulation network.

[0015] The second aspect of the present application provides a computer readable storage medium having a program stored thereon, the program being executed by a processor to implement the following steps:

[0016] Take biomolecules as variables, obtain observation data of the biomolecules, and filter possible parent variables of the variables according to the data to obtain a sparse graph after initial neighborhood selection;

[0017] Extract embedded features of each biomolecule from the observation data of the biomolecules and the sparse graph after initial neighborhood selection by using a graph attention network;

[0018] Calculate a weight vector of all biomolecules according to the embedded features of each biomolecule, sample on the weight vector in a loop, and sequentially order the extracted variables from back to front;

[0019] Calculate a reward for the ordering according to a scoring function, update parameters of each network under a reinforcement learning framework according to the reward, and guide to generate a variable ordering with a higher reward according to the updated parameters.

[0020] selecting variables according to the ranking with the highest reward to obtain the final biomolecule causal regulation network.

[0021] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the program:

[0022] The biomolecules are regarded as variables, observation data of the biomolecules are obtained, and possible parent variables of the variables are screened according to the data to obtain an initial neighborhood-selected sparse graph;

[0023] The embedded features of each biomolecule are extracted from the observation data of the biomolecules and the initial neighborhood-selected sparse graph by using a graph attention network;

[0024] The weight vectors of all biomolecules are calculated according to the embedded features of each biomolecule, sampling is performed on the weight vectors in a loop, and the extracted variables are sequentially ranked from back to front;

[0025] The reward is calculated for the ranking according to a scoring function, the parameters of each network are updated under a reinforcement learning framework according to the reward, and the updated parameters are used to guide the generation of a variable ranking with a higher reward;

[0026] selecting variables according to the ranking with the highest reward to obtain the final biomolecule causal regulation network.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] 1、The present application screens possible parent variables of the variables through initial neighborhood selection, regards each biomolecule as a variable in a causal relationship, and screens the parent-child relationship between the variables according to the observation data of the variables to perform initial neighborhood selection on the variables, so that the search space of the biomolecule causal regulation network is more sparse, and the calculation difficulty of the score of the causal graph is reduced.

[0029] 2、The encoding network based on the graph attention of the present application can capture the structural knowledge on the sparse graph after initial neighborhood selection, thereby improving the search ability of the causal graph in the sparse graph space.

[0030] 3、The present application sequentially ranks the variables from back to front, which is very consistent with the generation process of the causal graph, improves the explainability of the search process of the causal graph, and thereby improves the accuracy of the causal structure discovery. BRIEF DESCRIPTION OF DRAWINGS

[0031] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the exemplary embodiments of the present application and their description, serve to explain the present application, and do not constitute improper limitations on the present application.

[0032] Figure 1 This is a structural diagram of the biomolecular causal regulation network discovery system based on reinforcement learning provided in Embodiment 1 of the present invention.

[0033] Figure 2 This is a schematic diagram of the workflow of the reinforcement learning-based biomolecular causal regulation network discovery system provided in Embodiment 1 of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0037] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0038] Example 1:

[0039] like Figure 1 As shown, Embodiment 1 of the present invention provides a causal network discovery system based on reinforcement learning, comprising:

[0040] The initial neighborhood selection module is configured to: treat biomolecules as variables, acquire observation data of biomolecules, and filter possible parent variables of the variables based on the data to obtain a sparse graph after initial neighborhood selection;

[0041] The feature encoding module is configured to extract embedded features of each biomolecule from the observation data of the biomolecules and the sparse graph after the initial neighborhood selection using a graph attention network.

[0042] The variable sorting module is configured to: calculate the weight vector of all biomolecules based on the embedded features of each biomolecule, cyclically sample the weight vector, and sort the extracted variables sequentially from back to front.

[0043] The ranking and scoring and searching module is configured to: calculate a reward for the ranking generated by the variable ranking module according to a scoring function, update parameters of the feature encoding module and the variable ranking module under a reinforcement learning framework according to the reward, and guide the feature encoding module and the variable ranking module to generate a variable ranking with higher reward after updating the parameters;

[0044] The variable selection module is configured to: select variables for the ranking with the highest reward to obtain a final biological molecule causal regulation network.

[0045] Specifically, the working method of the system includes the following processes:

[0046] S101, performing initial neighborhood selection on data.

[0047] Specifically, the biomolecules are regarded as variables, the observation data of the variables are obtained, and the possible parent variables of the variables are screened according to the data.

[0048] Specifically, the specific implementation mode of the step 101 is as follows:

[0049] S1011, obtaining simulation data and real data for causal discovery;

[0050] The simulation data refers to that a causal model is defined in advance, and a simulation causal graph and corresponding observation data are randomly generated under the model.

[0051] The real data refers to observation data corresponding to a protein signal network based on protein and phospholipid expression levels. The corresponding real causal graph has 11 variables and 17 edges, and the corresponding observation data has 853 samples.

[0052] S1012, performing initial neighborhood selection to screen possible parent variables of the variables, specifically including:

[0053] A bidirectional fully connected graph consistent with the dimension of the observation data is constructed;

[0054] Linear regression is performed on linear data to obtain the weight between each pair of variables, and edges with a weight greater than a threshold are retained;

[0055] For nonlinear data and real data, an additive model is first fitted for each variable, and then a boosting algorithm is used to select 10 parent variables for each variable.

[0056] S102, encoding variable features.

[0057] Specifically, a graph attention network is used to extract an embedded feature vector of each variable from passive observation data and a sparse graph after initial neighborhood selection.

[0058] Specifically, the specific implementation mode of the step 102 is as follows:

[0059] S1021, randomly sample m samples from the observation data, each sample has d variables, concatenate the values of the variables corresponding to these samples to form the initial features of the variables;

[0060] Repeat the construction process of the initial features several times to form the batch samples to be input into the encoding network;

[0061] The batch samples and the sparse graph structure information after initial neighborhood selection are input into the graph attention-based encoding network. Figure 1 The network first calculates the attention coefficient e ij between each pair of variables i and j according to the initial features of the samples, which is defined as follows:

[0062]

[0063] Where T and represent the transpose and concatenation operations, and are vectors formed by concatenating the variable values corresponding to variables i and j in each sample in the batch sample, and the variables in and represent the shared linear transformation parameterized by the weight matrix and the single-layer feedforward neural network parameterized by the weight vector.

[0064] S1022, inject the graph structure information of the initial neighborhood selection into the network to calculate the embedded features enc i of each variable, which is defined as follows:

[0065]

[0066] Where Pa(i) represents the parent variable set of variable i in the sparse graph after initial neighborhood selection, and the dimension of enc i is d e .

[0067] The graph attention-based encoding network considers both the observation data and the prior structural knowledge to encode the feature vector for each variable.

[0068] S103, generate a variable order.

[0069] Specifically, the weight vector of all variables is calculated according to the embedded features of each variable; the variables extracted by sampling on the weight vector are arranged from back to front in turn.

[0070] Specifically, the specific implementation of the step 103 is as follows:

[0071] S1031, combine the feature vectors of all variables to form a feature matrix

[0072] S1032 decode all the feature matrices of variables into a weight vector h using a single-layer feedforward neural network as the decoder, which is defined as follows:

[0073]

[0074] wherein, is the decoding network parameter, and flatten(·) represents the operation of flattening a matrix.

[0075] Then, each dimension of the weight vector h is associated with each variable, and the variable corresponding to the dimension with the maximum value is selected, and then the features corresponding to the variable are masked in the feature matrix of all variables.

[0076] S1033, repeat the decoding process d times, and arrange the variables selected each time in turn from back to front to obtain the variable ordering generated by the decoding network.

[0077] The entire encoding-decoding network works as an action network of reinforcement learning. For each step of the variable ordering, the input state s t represents the feature matrix input into the action network, and the output action a t represents the variable selected according to the weight vector h.

[0078] S104, calculate the reward and search for the optimal ordering.

[0079] Specifically, the reward is calculated according to the ordering generated by the variable ordering module according to the scoring function; the parameters of the feature encoding module and the variable ordering module are updated under the reinforcement learning framework according to the reward; after updating the parameters, the feature encoding module and the variable ordering module are guided to generate a variable ordering with higher reward.

[0080] Specifically, the specific implementation of the step 104 is as follows:

[0081] S1041, based on the variable ordering process, calculate the immediate reward r i for each step of the ordering, which is defined as follows:

[0082]

[0083] wherein, is the corresponding estimated value, which is the value of the i-th variable of the k-th sample.

[0084] S1042, use a simple two-layer feedforward neural network as the evaluation network to estimate the expected reward of each step in the ordering process, and input the feature matrix (state s t) is flattened into a long vector and input into the evaluation network to estimate the corresponding expected reward

[0085] The mean squared error between the expected reward and the actual reward r i is calculated, and the evaluation network is updated using a stochastic gradient optimization method.

[0086] Then, the update gradient of the encoding-decoding network is calculated under the reinforcement learning framework and is defined as follows:

[0087]

[0088] where π φ (a t |s t ) and φ represent the policy and network parameters of the reinforcement learning, respectively, r represents the actual reward at each step of the ranking process, and γ l is the discount rate.

[0089] Then, the encoding-decoding network is updated using a stochastic gradient optimization method and the calculated update gradient .

[0090] S1043, the encoding-decoding network continuously generates new variable rankings, calculates rewards for the rankings, updates the encoding-decoding network based on the rewards, and stops until a certain number of iterations is reached. The ranking with the highest reward is selected from all the searched rankings

[0091] S105, variable selection is performed on the optimal ranking.

[0092] Specifically, the ranking is scored, and the directed acyclic graph corresponding to the ranking with the highest reward in the search module is pruned to obtain the final causal graph.

[0093] Specifically, the specific implementation of step 105 is as follows:

[0094] S1051, the ranking with the highest reward is mapped into a fully connected directed acyclic graph;

[0095] S1052, for linear data, the weights of the edges are calculated using linear regression, and then the edges are pruned by a threshold value, which is greater than the threshold value used in the initial neighborhood selection;

[0096] For nonlinear data and real data, a generalized additive model is fitted for each variable with its current parent variables, and a significance test is performed on the covariates. Variables with a significance less than a threshold value are removed from the parent variable set.

[0097] After variable selection, the final output causal graph is obtained.

[0098] In summary, the present application regards each biomolecule as a variable, first performs initial neighborhood selection on the variable, then encodes an embedded feature for each biomolecule using a graph attention-based encoding network, then uses a decoding network to generate a backward ordering of the biomolecules, then calculates a reward for the ordering according to a scoring function and updates the encoding-decoding network under a reinforcement learning framework according to the reward, and finally selects the ordering with the highest reward to perform variable selection to obtain the final biomolecule causal action network.

[0099] The present application considers selecting the initial neighbors of each biomolecule to construct a sparse graph structure, then encodes an embedded feature for each biomolecule on the sparse graph using a graph attention network, then generates a backward ordering of the biomolecules through a recurrent classification manner, then calculates a reward for the generated ordering according to a scoring function and constantly searches for an ordering with a higher reward through reinforcement learning, and finally performs variable selection on the optimal ordering to filter the causal action relationships between biomolecules to obtain the final output causal regulation network.

[0100] The present application can search for the causal action network of biomolecules in a more sparse space by selecting the possible parent variables of the variable through initial neighborhood selection, greatly improves the search efficiency of the causal graph and effectively reduces the calculation difficulty of the reward of the causal graph, and the embedded feature of the variable encoded based on the graph attention can effectively capture the causal structure knowledge in the sparse graph obtained through initial neighborhood selection, so that the decoding ordering process can generate a variable ordering that is more consistent with the prior knowledge. In addition, the generation mechanism based on the backward ordering is consistent with the generation process of the causal graph, thus providing a certain interpretability for the generation process of the variable ordering, thereby effectively improving the accuracy of the final result.

[0101] Embodiment 2

[0102] The present application embodiment 2 provides a computer readable storage medium, which stores a program, the program is executed by a processor to realize the following steps:

[0103] The biomolecules are regarded as variables, the observation data of the biomolecules are obtained, and the possible parent variables of the variables are filtered according to the data to obtain a sparse graph after initial neighborhood selection;

[0104] The embedded feature of each biomolecule is extracted from the observation data of the biomolecules and the sparse graph after initial neighborhood selection using a graph attention network;

[0105] The weight vectors of all biomolecules are calculated according to the embedded feature of each biomolecule, and the variables are sequentially ordered from back to front by sampling on the weight vectors;

[0106] The reward is calculated for the ranking according to a scoring function, the parameters of each network are updated under the reinforcement learning framework according to the reward, and the variable ranking with higher reward is generated according to the updated parameters.

[0107] The variable selection is performed on the ranking with the highest reward, and the final biomolecule causal regulation network is obtained.

[0108] The detailed steps are the same as the working method of the system in Embodiment 1, and will not be repeated here.

[0109] Embodiment 3

[0110] Embodiment 3 of the present application provides an electronic device, comprising a memory, a processor and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program:

[0111] The biomolecules are regarded as variables, observation data of the biomolecules are acquired, and possible parent variables of the variables are screened according to the data to obtain an initial sparse graph after neighborhood selection;

[0112] The embedded features of each biomolecule are extracted from the observation data of the biomolecules and the initial sparse graph after neighborhood selection by using a graph attention network;

[0113] The weight vectors of all biomolecules are calculated according to the embedded features of each biomolecule, sampling is performed on the weight vectors in a loop, and the extracted variables are sequentially ranked from back to front;

[0114] The reward is calculated for the ranking according to a scoring function, the parameters of each network are updated under the reinforcement learning framework according to the reward, and the variable ranking with higher reward is generated according to the updated parameters;

[0115] The variable selection is performed on the ranking with the highest reward, and the final biomolecule causal regulation network is obtained.

[0116] The detailed steps are the same as the working method of the system in Embodiment 1, and will not be repeated here.

[0117] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer usable program code.

[0118] The embodiments of methods, apparatuses (systems) and computer program products according to the present application can be described in the general context of method steps and processes, which can be implemented in one embodiment by a program of instructions on a computer-readable storage medium executed by a computer or other programmable apparatus. The apparatuses can be specially constructed for executing the embodiments of methods, apparatuses (systems) and computer program products according to the present application or can include a computer or other programmable apparatus. Figure 1 The flow and / or block diagram in the flow and / or block diagram Figure 1 The flow and / or block diagram in the flow and / or block diagram

[0119] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow and / or block diagram in the flow and / or block diagram Figure 1 The flow and / or block diagram in the flow and / or block diagram

[0120] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow and / or block diagram in the flow and / or block diagram Figure 1 The flow and / or block diagram in the flow and / or block diagram

[0121] Those skilled in the art can understand that all or part of the flow of the above-mentioned embodiment method can be completed by computer program instructions instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the flow of the above-mentioned embodiment of each method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0122] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A system for discovering causal networks based on reinforcement learning, comprising: an initial neighborhood selection module configured to: treat biomolecules as variables, obtain observation data of the biomolecules, and filter possible parent variables of the variables according to the data to obtain a sparse graph after initial neighborhood selection; a feature encoding module configured to: extract embedded features of each biomolecule from the observation data of the biomolecules and the sparse graph after initial neighborhood selection using a graph attention network; a variable ordering module configured to: calculate a weight vector of all biomolecules according to the embedded features of each biomolecule, sample on the weight vector in a loop, and sequentially order the variables extracted from back to front; an ordering scoring and searching module configured to: calculate a reward for the ordering generated by the variable ordering module according to a scoring function, update parameters of the feature encoding module and the variable ordering module under a reinforcement learning framework according to the reward, and guide the feature encoding module and the variable ordering module to generate a variable ordering with a higher reward after updating the parameters; and a variable selection module configured to: select variables for the ordering with the highest reward to obtain a final biomolecular causal regulatory network. 2.The system for discovering causal networks based on reinforcement learning of claim 1, wherein: the observation data comprises simulated data and real data, the simulated data comprises: a predefined causal model, and simulated causal graphs and corresponding observation data randomly generated under the model; and the real data comprises: observation data corresponding to a protein signaling network based on protein and phospholipid expression levels; filtering possible parent variables of the variables according to the data comprises: constructing a bidirectional fully connected graph consistent with the dimension of the observation data; performing linear regression on linear data to obtain the weight between each pair of variables, and retaining edges with a weight greater than a preset threshold; and for nonlinear data and real data, fitting an additive model for each variable, and then selecting multiple parent variables for each variable using a boosting algorithm. 3.The system for discovering causal networks based on reinforcement learning of claim 1, wherein: extracting embedded features of each biomolecule from the observation data of the biomolecules and the sparse graph after initial neighborhood selection using a graph attention network comprises: randomly extracting m samples from the observation data, each sample having d variables, concatenating the values of the variables corresponding to the samples to form initial features of the variables; repeating the construction process of the initial features several times to form a batch sample to be input into an encoding network; inputting the batch sample and the sparse graph after initial neighborhood selection into a graph attention-based encoding network, the network first calculating an attention coefficient between each pair of variables i and j according to the initial features of the sample; and injecting the graph structure information of the initial neighborhood selection into the network, combining the attention coefficient between each pair of variables i and j, and calculating the embedded features of each variable. 4.The system for discovering causal networks based on reinforcement learning of claim 3, wherein: the graph attention-based encoding network comprises: a multi-layer perceptron (MLP) network, and a graph attention network. 5.The system for discovering causal networks based on reinforcement learning of claim 1, wherein: the variable ordering module comprises: combining the feature vectors of all variables together to form a feature matrix. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ Embedded feature enc i , comprising: where e ij is the attention coefficient between variables i and j, Pa(i) is the parent variable set of variable i in the sparse graph after the initial neighborhood selection, e ik is the attention coefficient between variables i and k. ​ ​ ​ The single-layer feedforward neural network is used as a decoder to decode the feature matrix of all variables into a weight vector; Each dimension of the weight vector is matched with each variable, and the variable corresponding to the dimension with the maximum value is selected, and then the feature corresponding to the variable is masked in the feature matrix of all variables; The decoding process is repeated d times, and the selected variables are arranged from back to front to obtain the variable ordering generated by the decoding network.

6. The causal network discovery system based on reinforcement learning according to claim 1, wherein: The ordering scoring and searching module comprises: The variable ordering process is used to calculate the immediate reward for each step of the ordering; A double-layer feedforward neural network is used as an evaluation network to estimate the expected reward of each step in the ordering process, and the feature matrix input in each step of the decoding process is flattened into a long vector and input into the evaluation network to estimate the corresponding expected reward; The mean square error between the expected reward and the immediate reward of each step in the ordering process is calculated, and the evaluation network is updated by using the method of stochastic gradient optimization; The update gradient of the encoding-decoding network is calculated by using the reward in the reinforcement learning framework; The encoding-decoding network is updated by using the method of stochastic gradient optimization and the calculated update gradient; New variable ordering is continuously generated by the encoding-decoding network, the reward is calculated for the ordering, the encoding-decoding network is updated according to the reward, and the ordering with the highest reward is selected from all searched orderings until a certain number of iterations is met.

7. The causal network discovery system based on reinforcement learning according to claim 1, wherein: The variable selection module comprises: pruning the directed acyclic graph corresponding to the ordering with the highest reward in the ordering scoring and searching module to obtain the final causal graph.

8. The causal network discovery system based on reinforcement learning according to claim 7, wherein: The ordering with the highest reward is mapped into a fully connected directed acyclic graph; For linear data, the weight of the edge is calculated by using linear regression, and then the edge is pruned by using a threshold value, which is greater than the threshold value in the initial neighborhood selection; For nonlinear data and real data, a generalized additive model is fitted for each variable with its current parent variable, and a significance test is performed on the covariates, and the variable with a significance less than a threshold value is removed from the parent variable set; After the variable selection, the final output causal graph is obtained.

9. A computer-readable storage medium having stored thereon a program, characterized in that, When the program is executed by the processor, the following steps are implemented: The biomolecules are regarded as variables, the observation data of the biomolecules are obtained, and the possible parent variables of the variables are screened according to the data to obtain a sparse graph after initial neighborhood selection; The embedded features of each biomolecule are extracted from the observation data of the biomolecules and the sparse graph after the initial neighborhood selection by using a graph attention network; The weight vector of all biomolecules is calculated according to the embedded features of each biomolecule, and the variables are sampled on the weight vector in a loop to arrange the variables from back to front; The reward is calculated for the ordering according to a scoring function, the parameters of the networks are updated in the reinforcement learning framework according to the reward, and the variable ordering with a higher reward is generated according to the updated parameters; The variable selection is performed on the ordering with the highest reward to obtain the final biomolecule causal regulation network.

10. An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, The processor implements the following steps when executing the program: Regarding the biomolecules as variables, obtaining observation data of the biomolecules, and screening possible parent variables of the variables according to the data to obtain an initial sparse graph after neighborhood selection; Extracting embedded features of each biomolecule from the observation data of the biomolecules and the initial sparse graph after neighborhood selection by using a graph attention network; Calculating a weight vector of all the biomolecules according to the embedded features of each biomolecule, sampling on the weight vector in a loop, and sequentially ranking the extracted variables from back to front; Calculating a reward for the ranking according to a scoring function, updating parameters of each network in a reinforcement learning framework according to the reward, and guiding generation of a variable ranking with a higher reward according to the updated parameters; Selecting variables according to the ranking with the highest reward to obtain a final biomolecular causal regulatory network.

Images