Reinforcement learning drug recommendation method based on hierarchical perception graph neural network
By using a reinforcement learning method based on hierarchical perceptual graph neural networks, the problem of neglecting hierarchical information association in drug recommendation is solved, which improves the accuracy and efficiency of drug recommendation, reduces computational noise, and enhances the expressive power of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-03-10
AI Technical Summary
Existing drug recommendation methods neglect the hierarchical information relationships between patients, leading to information distortion and excessive computational load, which reduces the accuracy of drug recommendations.
We employ a reinforcement learning approach based on hierarchical perceptual graph neural networks. By encoding patient information into a hyperbolic space, we optimize the drug recommendation model using an inverse variance reinforcement learning algorithm. Combined with the evaluation and updating of the physician knowledge base, we dynamically select a set of relevant drugs.
It improves the accuracy and efficiency of drug recommendations, reduces computational noise, and enhances the expressive power of the model.
Smart Images
Figure CN121641328A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of natural language processing, and particularly relates to a reinforcement learning drug recommendation method based on a hierarchical perception graph neural network. BACKGROUND
[0002] Electronic medical records are the main data carriers for patient medical research in the medical field, and drive researchers and doctors to build various decision support systems. Among these systems, drug recommendation is an important research direction. Drug recommendation aims to predict a set of drugs according to the symptoms of a patient, and only a set of drugs that are harmless to each other can assist doctors in making medical decisions.
[0003] Traditional drug recommendation methods learn various medical entity representations from electronic medical records, and then use the learned entity representations to predict drugs, but these methods ignore the hierarchical information association between some entities. This hierarchical information is the association between entities of the same type and entities of different types. For example, patients 1 and 2 have the same symptoms, and when recommending drugs, this relationship is not considered, but only the historical diagnosis information of the patient is used to recommend drugs. Since knowledge graph technology can closely link entities, integrating knowledge graph into the recommendation model can enable the model to take into account the aforementioned hierarchical information, ultimately improving the model's ability to recommend accuracy. Currently, there have been some works based on knowledge graph to solve the above problems.
[0004] Although the drug recommendation method based on the knowledge graph has good performance, there are still two problems.
[0005] First, the method based on the knowledge graph models the patient and the drug in the Euclidean space, while the hierarchical information is ubiquitous in the knowledge graph. Therefore, when extracting hierarchical structure data, the method based on the Euclidean distance will have information distortion phenomenon. In addition, the fine-grained hierarchical relationship of the knowledge graph is also ignored. The hierarchical relationship provides the attributes of the patient, and the non-hierarchical relationship only indicates the association between entities. If this feature is ignored when encoding data, the model's expression ability will often be weakened.
[0006] Second, the existing method encodes electronic medical records, DDI (Drug-Drug Interaction) drug graphs and long-range patient records together to predict drugs. This method of using all data has serious problems. For predicting drugs for a patient, only part of the data needs to be considered because much of the knowledge in the electronic medical record and DDI data is irrelevant. For example, a patient's medical history mentions an experience of joint pain once, but the current patient's main symptoms are other health problems unrelated to joint pain, such as respiratory distress. When the actual knowledge graph is large, using all knowledge to recommend drugs will bring additional computational complexity and data noise, which is not conducive to the representation learning of key drug relationships, thereby resulting in poor accuracy of drug recommendation. SUMMARY
[0007] In order to solve the technical problem of poor accuracy of drug recommendation, the present application provides a reinforcement learning drug recommendation method based on a hierarchical perception graph neural network.
[0008] The present application provides a reinforcement learning drug recommendation method based on a hierarchical perception graph neural network, which comprises: Collecting diagnosis, surgery and prescription information of multiple different patients at different times to form a patient information data set; Inputting the patient information in the patient information data set into an initial drug recommendation model, and encoding the patient information by the drug recommendation model to obtain an initialization state; The drug recommendation model gives a recommended drug according to the initialization state, the doctor knowledge base judges whether the recommended drug is reliable and gives a reward evaluation value; The old drug graph determines whether the drug is used for updating according to the evaluation result of the doctor knowledge base.
[0009] Optionally, The diagnosis information of the first patient is represented as patient information The surgery information of the first patient is represented as patient information The surgery information of the first patient is represented as patient information The surgery information of the first patient is represented as patient information The surgery information of the first patient is represented as patient information The surgery information of the first patient is represented as patient information .
[0010] Optionally, the method for obtaining the drug graph comprises: The drug graph at time t is represented as The patient information of the first patient at time t is represented as The relationship set between nodes at time t is represented as The patient information of the first patient at time t is represented as The relationship set between nodes at time t is represented as The patient information of the first patient at time t is represented as The relationship set between nodes at time t is represented as The relationship set between nodes at time t is represented as The relationship set between nodes at time t is represented as Patient's medication prescription at all times Includes old drug maps With newly added nodes, the hierarchy of the knowledge graph adopts constant curvature. of The Viponcaré sphere model represents this.
[0011] Optionally, the drug recommendation model encodes patient information, including: Using multi-heat vector pairs of nodes and The encoding process employs three convolutional layers to further encode each node, resulting in an encoding with shallow semantics. and The calculation process is as follows: in, This represents a three-layer convolutional neural network; It is the first Patient information for each patient; For the first The patient's medication prescription; and Mapping to hyperbolic space; encoding node information in hyperbolic space through logarithmic mapping; node z is mapped to y in hyperbolic space, and the logarithmic mapping process is represented as: in, yes and The norm of the Möbius strip result, It is Möbiuska, and it is represented as: Where x and y are two distinct elements of hyperbolic space. Dimensional nodes; yes and Lorentz inner product, It is the curvature of hyperbolic space. yes Its dot product with itself; according to hyperbolic space theory, the distribution of the encoding nodes in hyperbolic space is as follows: in, It is a hyperbolic space Dimensional nodes; and Represent hyperbolic space and Euclidean space respectively; the distance between nodes is: Cluster nodes in Euclidean space; map the nodes back to hyperbolic space, obtaining a hierarchical node representation through exponential mapping; the Euclidean propagation-based method uses mean clustering to average the neighborhood information of the nodes, and uses mean clustering to cluster information in Euclidean space. The clustering process is as follows: Where 0 and l refer to the 0th and 1st layers of the network, respectively. and These are logarithmic and exponential mappings, respectively; when aggregating node information, points represented in hyperbolic space are mapped back to Euclidean space; for node z, Represented as: After obtaining the modules representing all nodes, the model architecture can be technically represented as follows: in, It is the first graph neural network. The node at the th Layer representation, This represents the number of model layers.
[0012] Optionally, the drug recommendation process includes: An inverse variance reinforcement learning algorithm is used to update the drug recommendation model, dynamically selecting the relevant drug set through a deep network; the policy function is... The value function is Intensive learning Moment State This is now represented as a combination of patient and medication. for Drugs recommended by the time-matter model; state of time Recorded as , Recommended medication at any time Recorded as , state of time Recorded as , Recommended medication at any time Recorded as ; The target network T and the current network Q are the same multilayer perceptron network; at time step t, the playback memory unit provides the current network Q with a representation of the patient and drug code combination. The network selects drugs based on the Q value. The information is passed to the physician knowledge base and drug atlas, and the physician knowledge base evaluates the drugs. And the updated Together they are sent to the playback memory unit; at time t+1, the playback memory unit will store the data at this time. The data is fed into the target network T; at this point, the playback memory units have a certain amount of data. and Effective coding combinations are extracted using inverse variance and fed to the target network T. The parameters of the target network T are updated by the current network Q. The reinforcement learning process terminates when all combinations in the replay memory unit have been extracted. The current network Q is at time step t, based on the input. get ;according to Choose the drug that yields the maximum Q value. ;use The layer perceptron is used to approximate the Q function, and the output of the Q network is the Q value, expressed as: in, It is the Q network number The output of the layer, It is a parameter matrix. It is the bias matrix; It is the output vector of the last layer of the network; After the current network Q updates for round C, the parameters are copied to the target network T, and then output by the target network T. According to the Bellman formula, the target value used for the current network Q update is... The calculation is as follows: in, It is the attenuation factor; After obtaining the target value, the Q network parameters The update process is minimized and The L2 loss function between them is used to complete the process: For a batch size of D and a total sample size of K, assume the noise is... , , The L2 loss function of the neural network after applying inverse variance is: in, It's a hyperparameter; For a given input The predicted mean was obtained. and variance ; a loss attenuation mechanism is adopted to capture the change of noise on the label; for K samples , a loss attenuation mechanism is adopted to optimize network parameters The process is as follows: Wherein, the neural network optimizes the uncertainty according to the error between And The weight of high uncertainty label is reduced by variance ; For samples And drugs , the final optimization process is as follows: Wherein, And Are the expectation and variance of Q network respectively; Is a balance factor, used to unify the magnitude of the two optimization functions; Is the variance of the target network.
[0013] The present application has the following beneficial effects: The present application improves the efficiency and accuracy of drug recommendation. BRIEF DESCRIPTION OF DRAWINGS
[0014] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.
[0015] Figure 1 It is a flow chart of a reinforcement learning drug recommendation method based on hierarchical perception graph neural network of the present application; Figure 2 It is a framework diagram of a reinforcement learning drug recommendation method based on hierarchical perception graph neural network of the present application; Figure 3 It is a patient information encoding process schematic diagram of the present application; Figure 4 It is a drug recommendation method flow chart based on inverse variance reinforcement learning of the present application. DETAILED DESCRIPTION
[0016] To further clarify the technical means and effects taken by the present application to achieve the predetermined inventive objectives, the specific implementation, structure, features and effects of the technical solutions proposed according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.
[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.
[0018] Reference is made to Figure 1 , which shows the flow of some embodiments of a reinforcement learning drug recommendation method based on a hierarchical perception graph neural network of the present application. Reference is made to Figure 2 , which shows a framework diagram of a reinforcement learning drug recommendation method based on a hierarchical perception graph neural network of the present application.
[0019] Specifically, Figure 1 The reinforcement learning drug recommendation method based on a hierarchical perception graph neural network shown includes the following steps: Step S1, collect the diagnosis, surgery, prescription information of multiple different patients at different times, and form a patient information data set.
[0020] In some embodiments, the diagnosis, surgery, prescription information of multiple different patients at different times can be collected to form a patient information data set.
[0021] Specifically, after obtaining the patient information data set, the data set is randomly divided into a training set and a test set.
[0022] As an example, the diagnosis information of the first patient, the surgery information of the first patient, the diagnosis information of the second patient, the surgery information of the second patient, and are combined to represent the patient information .
[0023] Step S2, input the patient information in the patient information data set into an initial drug recommendation model, and the drug recommendation model encodes the patient information to obtain an initial state.
[0024] In some embodiments, the patient information and can be input into the initial drug recommendation model, and the model encodes the patient information.
[0025] As an example, as shown inFigure 3 As shown, the drug recommendation model encodes patient information can include the following steps: Multi-hot vector to node And Encode, then use three layers of convolution to further encode each node, get the shallow semantic And , the calculation process is as follows: Wherein, Indicates a three-layer convolutional neural network. Is the patient information of the first Patient. Is the medication prescription of the first Patient. Then, And Map to hyperbolic space that can fully represent the hierarchical relationship of nodes. The node information in hyperbolic space is encoded by logarithmic mapping. The node z is mapped to hyperbolic space y, and the logarithmic mapping process is represented as: Wherein, Is the norm of the Mobius addition result of And , Is the Mobius addition, which is expressed as: Wherein, And Respectively, two different Dimensional nodes in hyperbolic space, Is the Lorentz inner product of And , Is the curvature of hyperbolic space, Is the dot product of And itself. According to the theory of hyperbolic space, the distribution of encoded nodes in hyperbolic space is: Wherein, And Respectively represent hyperbolic space and Euclidean space. The distance between nodes is: Then, nodes are clustered in Euclidean space. Finally, the nodes are mapped back to hyperbolic space, and a hierarchical node representation is obtained through exponential mapping. Since the Euclidean propagation-based method mainly uses mean clustering to average the neighborhood information of nodes, mean clustering is used to cluster information in Euclidean space. The clustering process is as follows: Where 0 and l refer to the 0th and 1st layers of the network, respectively. and These are logarithmic and exponential mappings, respectively. Since neural network operations are performed in Euclidean space, aggregating node information requires mapping the points represented in hyperbolic space back to Euclidean space. For node z, Represented as: This aggregation method integrates relational information into the target representation in hyperbolic space, avoiding complex hyperbolic average pooling through mapping. Unlike the information propagation method of classic graph neural networks, after obtaining the modules representing all nodes, the model architecture is technically uniformly represented as follows: in, It is the first Graph Neural Network (GNN) The node at the th Layer representation, This refers to the number of model layers. It should be noted that the reason for stacking is... Layered networks are used because they can represent high-level relationships between patients and medications.
[0026] In step S3, the drug recommendation model provides recommended drugs based on the initial state, and the doctor knowledge base determines whether the recommended drugs are reliable and provides a reward evaluation value.
[0027] In some embodiments, the model is based on the initialization state. Provide recommended medications Then Hand over to the doctor's knowledge base and old drug atlas The doctor's knowledge base first determines the recommended medication. Is it reliable? Provide a reward assessment value. Based on the recommended medication Should a reward assessment value be given in the doctor's knowledge base? If the recommended medication is in the doctor's knowledge base, a reward value of 1 is given; otherwise, -1 is given.
[0028] As an example, methods for obtaining drug spectra include: using... express a drug profile at time t, denotes a patient profile at time t, a patient profile at time t, is a set of relations between nodes at time t, is a drug prescription for a patient at time t. For convenience of description, only two types of relations, drug-related and harmful, are included in. includes the old drug profile and the newly added nodes, the hierarchy of the knowledge graph is represented by a constant curvature of a d-dimensional Poincare ball model.
[0029] As another example, the method for obtaining the drug recommendation process can include the following steps: An inverse variance reinforcement learning algorithm is used to update the drug recommendation model, aiming to dynamically select those relevant drug sets through a deep network, as shown in the framework Figure 4 . The policy function is , the value function is , and the state at time t is represented by the combination of the patient and their medications, is the drug recommended by the model at time t. For convenience of writing, the state at time t is denoted as , the recommended drug at time t is denoted as , the state at time t is denoted as , the recommended drug at time t is denoted as . The target network T and the current network Q are the same multi-layer perceptron network. At time step t, the replay memory unit provides the current network Q with the representation of the combination of patient and drug encoding
[0030] . Then, the network Q selects the drug according to the Q value, which is passed to the doctor's knowledge base and the drug profile. The doctor's knowledge base evaluates the drug and the updated together with the are fed into the replay memory unit. At time t+1, the replay memory unit feeds the at this time into the target network T. Since the replay memory unit at this time has a certain amount of and , the target network T is updated with the effective combination extracted from the replay memory, and the parameter updating of the target network T is completed by the current network Q. When all the combinations in the replay memory are extracted, the reinforcement learning process is terminated.
[0031] At time step t, the current network Q is updated according to the input obtained . Then, the drug with the maximum Q value is selected according to . A layer perceptron is used to approximate the Q function, and the output of the Q network is the Q value, which is represented as: wherein is the output of the Q network at the th layer, is a parameter matrix, is a bias matrix. is the output vector of the last layer of the network. The perceptron is a known technology, and will not be described in detail.
[0032] After the current network Q is updated for C rounds, the parameters are copied to the target network T, and the output of the target network T is obtained. According to the Bellman formula, the target value used for updating the current network Q is calculated as follows: wherein is a decay factor, which determines the importance of the doctor knowledge base evaluation in the reinforcement learning. If γ = 0, the model will not learn any future reward information, and will become short-sighted, only focusing on the current interests; if γ >= 1, the expected value is continuously accumulated and has no decay, so the expected value can diverge.
[0033] After the target value is obtained, the Q network parameter updating process can be completed by minimizing the L2 loss function between and : J = (R - Q)2 There is inevitably noise in the reinforcement learning, and part of the noise comes from , because itself is the output from the neural network, and the output of the neural network always has uncertainty, which leads to being noisy. Therefore, the effective Instead of taking all the values out, the sample is taken. The inverse variance is a weighting scheme that is robust to poor variance estimates and can improve the performance of neural networks. For a sample of size K with batch size D, assume the noise , , The L2 loss function of the neural network after using the inverse variance is: where is a hyperparameter that adjusts the experimental effect and optimizes the stability of the network.
[0034] Although using the inverse variance to optimize the neural network can improve the performance, the uncertainty of the network itself still exists. When sampling, it is necessary to consider reducing this uncertainty to achieve the effect of suppressing noise. Since the normalization layer of the neural network uses the mean and variance of the hidden layer output to standardize the distribution of each layer feature, for a given input , the predicted mean and variance are obtained. Then, the loss attenuation mechanism is used to capture the change of noise on the label. For K samples , the process of optimizing the network parameters using the loss attenuation mechanism is as follows: where the neural network optimizes the uncertainty according to the error between and , and the variance reduces the weight of the label with high uncertainty. Therefore, the noise on the label can be captured by this loss attenuation.
[0035] Since the neural network has a certain uncertainty, using the L2 loss function alone to optimize the neural network may lead to too high uncertainty, which affects the optimization of the entire network. Therefore, considering the performance and uncertainty of network optimization, the advantages of L2 loss function and loss attenuation mechanism are comprehensively utilized. For samples and drugs , the final optimization process is as follows: where and are the expectation and variance of the Q network respectively. is a balance factor to unify the magnitude of the two optimization functions. is the variance of the target network.
[0036] Step S4: The old drug profile determines whether the drug is used for updating according to the evaluation result of the doctor knowledge base.
[0037] In some embodiments, the old drug profile determines whether the drug can be used for updating according to the evaluation result of the doctor knowledge base.
[0038] In summary, the present application improves the drug recommendation efficiency and accuracy.
[0039] The above examples are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A reinforcement learning drug recommendation method based on a hierarchical perception graph neural network, characterized in that, The method comprises the following steps: Collecting diagnosis, surgery, and prescription information of a plurality of different patients at different times to form a patient information dataset; Inputting patient information in the patient information dataset into an initial drug recommendation model, encoding the patient information by the drug recommendation model to obtain an initialization state; The drug recommendation model gives a recommended drug according to the initialization state, a doctor knowledge base judges whether the recommended drug is reliable, and gives a reward evaluation value; An old drug atlas determines whether the drug is used for updating according to the evaluation result of the doctor knowledge base.
2. The reinforcement learning drug recommendation method based on hierarchical perception graph neural network according to claim 1, characterized in that, diagnostic information for the first patient, surgical information for the first patient, and in combination represent patient information .
3. The reinforcement learning drug recommendation method based on hierarchical perception graph neural network according to claim 1, characterized in that, The method for obtaining the drug atlas comprises: Adopting representing the drug profile at the moment, representing patient information of the patient at the moment, is the relationship set between nodes at the moment, is the medication prescription of the patient at the moment, including the old drug profile and the newly added node, the hierarchy of the knowledge graph adopts the dimensional Poincare ball model representation. 4. The reinforcement learning drug recommendation method based on hierarchical perception graph neural network according to claim 1, characterized in that, The drug recommendation model encodes the patient information, which comprises: With multiple heat vectors to nodes And Encoding, using three layers of convolution to further encode each node, to obtain a shallow semantic And The calculation process is: wherein, represents a three-layer convolutional neural network; is patient information of the th patient; is a medication prescription of the th patient; and and are mapped to hyperbolic space; node information in the hyperbolic space is encoded through a logarithmic mapping; after the node z is mapped to the hyperbolic space, it becomes y, and the logarithmic mapping process is represented as: where is with a norm of the Möbius sum of is the Möbius sum, which is expressed as: where x and y are two different points in the hyperbolic space dimensional nodes; is and the Lorentz inner product of x and y, is the curvature of the hyperbolic space, is the inner product of x with itself; according to the hyperbolic space theory, the distribution of the coding nodes in the hyperbolic space is: wherein, is a hyperbolic space dimensional node; and denote a hyperbolic space and a Euclidean space, respectively; the distance between nodes is: The nodes are aggregated in the Euclidean space; the nodes are mapped back to the hyperbolic space to obtain the node representation with a hierarchical structure through exponential mapping; the neighborhood information of the nodes is averaged based on the method of Euclidean propagation, and the information is aggregated in the Euclidean space by using mean aggregation, and the aggregation process is: where 0 and 1 denote the 0th and 1st layers of the network, respectively, and are the log and exponential mappings, respectively; mapping points represented in hyperbolic space back to Euclidean space when aggregating node information; for a node z, is represented as: After obtaining all the node representations, the model architecture is technically uniformly represented as: in, It is the first graph neural network. The node at the th Layer representation, This represents the number of model layers.
5. The reinforcement learning drug recommendation method based on hierarchical perception graph neural network according to claim 1, characterized in that, The drug recommendation process comprises: The inverse reinforcement learning algorithm is used to update the drug recommendation model, and a deep network is used to dynamically select a relevant drug set; a policy function is , a value function is , and a state in the reinforcement learning is presented as a combination of a patient and a drug taken by the patient, is a drug recommended by the model at time , the state at time is recorded as , , the recommended drug at time is recorded as , , the state at time is recorded as , and the recommended drug at time is recorded as . Target network T and current network Q are the same multi-layer perceptron network; at time step t, the replay memory unit provides the current network Q with the representation of the patient and drug code combination ; network Q selects the drug according to the Q value Transferred to the physician knowledge base and drug atlas, the physician's evaluation of the drug and the updated are sent to the replay memory unit together; at t+1, the replay memory unit sends the at this time to the target network T; the replay memory unit at this time has a certain amount of and , the effective code combination is extracted using inverse variance, and the parameter update of the target network T is completed by the current network Q; when all the combinations in the replay memory unit are extracted, the reinforcement learning process is terminated; The current network Q at time step t, according to the input Obtained ; according to Select the drug with the maximum Q value ; using Layer perceptron to approximate the Q function, the output of the Q network is the Q value, denoted as: wherein, is the output of the Qth layer of the network, is the output of the Qth layer of the network, is a parameter matrix, is a bias matrix; is the output vector of the last layer of the network; After the current network Q update C rounds, the parameters are copied to the target network T, and output by the target network T ; according to Bellman formula, the calculation of target value used by the current network Q update is: wherein is an attenuation factor; After obtaining the target value, the Q-network parameters The update process is done by minimizing and the L2 loss function between For a batch size of D, a total sample size of K, and assuming noise , , , the L2 loss function of the inverse variance posterior neural network is: wherein is a hyperparameter; For a given input , the predicted mean and variance are obtained; a loss decay mechanism is employed to capture the variation of noise on the labels; for K samples , the network parameters are optimized with the loss decay mechanism. wherein the neural network optimizes the uncertainty based on an error between and reduces the weight of high-uncertainty labels by variance For the samples and drugs the final optimization process is: where, and are the expectation and variance of the Q network, respectively; is a balancing factor to unify the magnitude of the two optimization functions; is the variance of the target network.