Variable component reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescription

Abstract

The invention discloses a traditional Chinese medicine prescription intelligent generation-oriented variable component reinforcement learning algorithm, which comprises the steps of S1, constructing a traditional Chinese medicine prescription intelligent generation-oriented variable component reinforcement learning model so as to realize variable component reinforcement learning modeling traditional Chinese medicine prescription generation; s2, designing a variable component sub-circuit oriented to an action value function; and S3, designing a training architecture and a training algorithm of the variable component sub reinforcement learning model, completing optimization of trainable parameters of the variable component sub line, realizing approximation of an optimal action value function Q * (s, a), and using the trained model to generate a traditional Chinese medicine prescription meeting the patient's condition needs. According to the variable component reinforcement learning algorithm for intelligent generation of the traditional Chinese medicine prescription, intelligent generation of the traditional Chinese medicine prescription is realized by adopting a variable component reinforcement learning technology, so that the interpretability and efficiency of intelligent generation of the traditional Chinese medicine prescription are improved.

Description

Technical Field

[0001] The present invention belongs to the field of variational quantum reinforcement learning and intelligent traditional Chinese medicine diagnosis and treatment technology, and specifically relates to a variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions. Background Art

[0002] Intelligent generation of Chinese medicine prescriptions is the core technical direction of intelligent Chinese medicine diagnosis and treatment. Its goal is to enable computers to intelligently prescribe Chinese medicine prescriptions that meet the patient's condition based on the patient's disease diagnosis information, just like professional Chinese medicine doctors. As artificial intelligence technology continues to enhance its capabilities in knowledge acquisition and integration, data processing and analysis, Chinese medicine prescription intelligent generation technology will be further developed and will play an increasingly important role in integrating Chinese medicine clinical medication experience and assisting doctors in clinical prescriptions.

[0003] Existing research on intelligent generation of Chinese medicine prescriptions focuses on extracting abstract features of the relationship between symptoms, syndromes, Chinese medicines and other elements by establishing a deep neural network model, and then fitting the nonlinear mapping relationship from symptoms to Chinese medicines to achieve computer automatic generation of Chinese medicine prescriptions. Since the meaning of the abstract features extracted by the neural network in Chinese medicine is difficult to clarify, the use of deep learning technology to complete prescription generation has the problem of poor interpretability of medication. Modeling Chinese medicine prescription generation through reinforcement learning technology and introducing the Chinese medicine drug screening mechanism into the reward function design can improve the interpretability of artificial intelligence prescriptions and further improve the accuracy of medication. Specifically, compared with viewing prescription generation as a nonlinear mapping from symptoms to Chinese medicines, viewing prescription generation as an optimized screening of Chinese medicine combinations is more in line with the process of Chinese medicine clinical prescriptions. In fact, the process of Chinese medicine clinical prescriptions can be regarded as a process in which a doctor selects the optimal combination of Chinese medicines that can fully match the patient's symptoms based on the patient's disease diagnosis information and the efficacy of Chinese medicines, prescription rules, etc. Since the optimization screening problem of Chinese medicine combinations can be modeled and solved through reinforcement learning technology, if the mechanism of Chinese medicine drug screening can be introduced into the design of the reward function, the generated Chinese medicine prescriptions can be made more rational in terms of medication, thereby improving the interpretability of artificial intelligence prescriptions and enhancing the performance of prescription generation.

[0004] At present, deep reinforcement learning technology occupies a core position in the field of reinforcement learning, and the use of deep reinforcement learning technology to realize the intelligent generation of Chinese medicine prescriptions is the first choice. However, it is difficult to efficiently complete the intelligent generation of Chinese medicine prescriptions using deep reinforcement learning technology in a classic environment. This is because, in traditional Chinese medicine, for a disease, there are a large number of possible symptoms and candidate Chinese medicines. The state space and action space of deep reinforcement learning need to be built on the two, which will cause the deep reinforcement learning model to face problems such as high parameter space complexity and long training time.

[0005] Variational quantum reinforcement learning is the use of variational quantum circuits to approximate the policy function or optimal action value function of the reinforcement learning agent. With the help of quantum mechanical properties such as quantum superposition and quantum entanglement, it is expected to reduce the parameter space complexity of the deep reinforcement learning model and shorten the model training time. Therefore, further improvements are made to the above problems. Summary of the invention

[0006] The main purpose of the present invention is to provide a variational quantum reinforcement learning algorithm for the intelligent generation of traditional Chinese medicine prescriptions, and to use variational quantum reinforcement learning technology to realize the intelligent generation of traditional Chinese medicine prescriptions, so as to improve the interpretability and efficiency of the intelligent generation of traditional Chinese medicine prescriptions.

[0007] To achieve the above objectives, the present invention provides a variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions, comprising the following steps:

[0008] Step S1: construct a variational quantum reinforcement learning model for intelligent generation of traditional Chinese medicine prescriptions, thereby realizing variational quantum reinforcement learning modeling for traditional Chinese medicine prescription generation;

[0009] Step S2: Design a variational quantum circuit for the action-value function, targeting the optimal action-value function Q of the agent * (s, a) needs to be approximated, and the corresponding variational quantum circuit is designed to obtain the action-value function approximator, which is used to approximate the optimal action-value function Q * (s,a);

[0010] Step S3: Design the training architecture and training algorithm of the variational quantum reinforcement learning model, optimize the trainable parameters of the variational quantum circuit, and achieve the optimal action value function Q * After training, the model will be used to generate Chinese medicine prescriptions that meet the patient's condition.

[0011] As a further preferred technical solution of the above technical solution, step S1 is specifically implemented as follows:

[0012] Step S1.1: Suppose patient p, record p's symptom set ST p ={st p,1 ,st p,2 ,...,st p,n}, the syndrome diagnosis result set obtained by syndrome differentiation is recorded as SD p ={sd p,1 ,sd p,2 ,...,sd p,m}, ST p and SD p Together they constitute the TCM diagnosis result Diag of patient p p =(ST p ,SDp ), the goal of intelligent generation of Chinese medicine prescriptions is to generate a Chinese medicine prescription based on the patient’s TCM diagnosis results. p , calculate the optimal Chinese medicine prescription Pres * ={ph1,ph2,...,ph l},in H={h1,h2,...,h k} is a set of candidate Chinese medicines;

[0013] Step S1.2: Use variational quantum reinforcement learning to model the problem of Chinese medicine prescription generation, and design a variational quantum reinforcement learning framework for intelligent generation of Chinese medicine prescriptions. The environment faced by the intelligent agent is the disease syndrome; the disease syndrome environment state is jointly determined by symptoms, syndrome diagnosis results and Chinese medicine;

[0014] In the specific learning process, the agent interacts with the disease environment through drug selection actions, observes the state changes of the disease environment after drug administration, and continuously optimizes the VQC-based strategy function according to the reward value returned by the disease medication evaluation, and finally obtains the optimal drug selection strategy function based on VQC, which maximizes the cumulative reward return obtained in the process of selecting drugs for diseases.

[0015] Formally, the Markov decision process corresponding to this process is defined as in, It is the disease-symptom-environment state space; Action space for drug selection for the agent; is the disease-syndrome-environment state transfer function; is the reward function; γ is the discount factor, γ∈[0,1]; define the drug selection strategy π: The goal of the agent is to learn the optimal drug selection strategy π * , so that the cumulative reward for selecting medicines for diseases is maximized, and r is defined t is the immediate reward value obtained at time t, then the learning objective can be defined as follows:

[0016]

[0017] Among them, T is the termination time.

[0018] As a further preferred technical solution of the above technical solution, under the variational quantum reinforcement learning framework given in step S1.2, the variational quantum reinforcement learning model has a disease-symptom environment state space, an agent drug selection action space, a disease-symptom environment state transfer function, a reward function, a termination condition, and an optimal strategy approximation, which are as follows:

[0019] Disease-Syndrome-Environment State Space Construction

[0020] Disease-Syndrome-Environment State Space It is determined by the symptom set ST, syndrome diagnosis result set SD and candidate Chinese medicine set H;

[0021] First, consider the set U = ST∪SD∪H, which contains all the symptoms, syndrome differentiation results, and Chinese medicine terms involved. Based on the set U, define the disease-syndrome environment state space for:

[0022]

[0023] Any state s in is a 0-1 vector of dimension |U|. In the order of symptoms first, then syndrome differentiation results, and finally Chinese medicine, the first |ST| position of the 0-1 state vector corresponds to the symptom words. If a symptom is in the given Chinese medicine diagnosis result, the corresponding position value of the state vector is 1, and the corresponding position value of the symptom that does not appear is 0; similarly, the syndrome differentiation result corresponds to |ST|+1 to |ST|+|SD|, and the relevant values ​​are the same as above; in addition to symptoms and syndrome differentiation results, the state vector also reserves the corresponding position of each candidate Chinese medicine. If the agent selects a certain medicine when taking an action, the corresponding position value of the medicine in the state vector is 1. For the initial state, the default value of all drug positions is 0;

[0024] Agent drug selection action space Construction

[0025] Agent drug selection action space It consists of drug selection actions based on the candidate Chinese medicine set H. Any action Represents a Chinese medicine selected by the agent in the current state;

[0026] Disease-Syndrome-Environment State Transfer Function Design

[0027] The state transfer function of the disease-symptom environment is deterministic, that is, the Chinese medicine corresponding to the drug selection action taken by the agent will directly change the value of the state vector of the current agent at the corresponding position to 1, which will be used as the new state of the disease-symptom environment feedback after taking the action;

[0028] Reward Function Design

[0029] The reward function is defined by offline simulation of TCM doctors judging whether the selected medicine is reasonable. For the TCM diagnosis result part of the disease state, if the medicine predicted by the agent appears in the list of medicines actually prescribed by the doctor for the diagnosis result, the agent will receive a positive reward r t = 1, otherwise a negative reward r is given t = -1;

[0030] Termination Condition

[0031] The collected TCM prescription data set is used as the learning object of the intelligent agent. In each round of learning, the TCM prescription data set is traversed, and each traversed prescription record is used as the minimum learning unit. The intelligent agent takes the action of selecting medicine according to the recorded TCM diagnosis results of the patient. When the number of selected medicines is equal to the number of medicinal ingredients in the recorded TCM prescription, the intelligent agent terminates the learning of the current prescription record. For each round, it terminates after the TCM prescription data set is traversed.

[0032] Optimal strategy approximation

[0033] Using an action-value function-based approach, Q is approximated through a variational quantum circuit * (s,a), using the quantum mechanical properties of quantum superposition and quantum entanglement, the number of training parameters required for approximation is reduced and the calculation of the action-value function is accelerated, shortening the training completion time or iteration cycle.

[0034] As a further preferred technical solution of the above technical solution, in step S2, a variational quantum circuit design for the action-value function is designed, and a variational quantum circuit including a state encoding layer, a trainable parameter layer, and a measurement operation is designed, wherein:

[0035] State coding layer: using amplitude coding circuit, the classical disease and environmental state is encoded into the amplitude of the quantum bit superposition state, realizing quantum coding of the disease and environmental state;

[0036] Trainable parameter layer: A trainable parameter layer based on the R (α, β, γ) three-parameter rotation gate and CNOT gate is used to make the entire circuit a strongly entangled variational quantum circuit;

[0037] Measurement operation: Use Pauli-Z measurement to measure the quantum state of the quantum bit corresponding to each drug selection action after being processed by the variational quantum circuit. Expand the expected value of the Pauli-Z observable obtained according to the value range of the action value function, and use the final result as an approximate estimate of the action value function value of the variational quantum circuit for each drug selection action.

[0038] As a further preferred technical solution of the above technical solution, for step S3, a model training architecture based on the target network and the experience replay mechanism and a model training algorithm based on Deep Q-Learning are designed. Through model training, the intelligent agent completes the optimization adjustment of the trainable parameters of the variational quantum circuit to achieve the optimal action value function Q * (s,a) approximation, estimating Q through variational quantum circuit approximation * (s,a), obtain the optimal drug selection strategy π *According to this strategy, a Chinese medicine prescription that meets the patient's condition is generated. The training algorithm is specifically implemented in the following steps:

[0039] Step S3.1: Given a Chinese medicine prescription dataset P;

[0040] Step S3.2: Initialize the playback memory D to capacity N and use random parameters to initialize the main VQC-Q * Function approximator VQC_Q(s,a;θ), and main VQC-Q * The parameters θ of the function approximator initialize the target VQC-Q * Function approximator VQC_Q(s,a;θ - ), where θ - =θ;

[0041] Step S3.3: Start M training rounds, and in each round, learn each prescription record in the data set one by one under the given traditional Chinese medicine prescription data set {;

[0042] Step S3.3: Start learning the i-th prescription record, and initialize the disease state s1 with the patient's TCM diagnosis result in the current prescription record P[i];

[0043] Step S3.4: According to the number of Chinese medicines in the prescription record P[i], len(P[i].herbs), start the tth iteration, and each iteration uses the ε-greedy strategy based on the Q function to select action a t ;

[0044] Step S3.5: Take action a t And observe the reward r t and the next state s t+1 , will learn experience (s t ,a t ,r t ,s t+1 ) is stored in the playback memory;

[0045] Step S3.6: Randomly sample a batch of learning experiences from the replay memory D, calculate the loss value and the gradient of the parameter θ according to the defined loss function, and use the parameter optimizer to optimize and adjust the parameter θ;

[0046] Step S3.7: Update the target VQC-Q every C steps * The parameters of the function approximator θ - =θ;

[0047] Step S3.8: When M training rounds are completed, the algorithm terminates. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1It is a schematic diagram of the variational quantum reinforcement learning framework for intelligent generation of traditional Chinese medicine prescriptions of the present invention.

[0049] Figure 2 It is a schematic diagram of the variational quantum circuit structure for action-value function designed by the present invention.

[0050] Figure 3 This is a diagram of the variational quantum reinforcement learning model training architecture of the present invention.

[0051] Figure 4 It is a flow chart of the variational quantum reinforcement learning model training algorithm of the present invention. DETAILED DESCRIPTION

[0052] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0053] In the preferred embodiments of the present invention, those skilled in the art should note that the Chinese medicine prescriptions and the like involved in the present invention may be regarded as prior art.

[0054] Preferred embodiments.

[0055] The present invention discloses a variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions, comprising the following steps:

[0056] Step S1: construct a variational quantum reinforcement learning model for intelligent generation of traditional Chinese medicine prescriptions, thereby realizing variational quantum reinforcement learning modeling for traditional Chinese medicine prescription generation;

[0057] Step S2: Design a variational quantum circuit for the action-value function, targeting the optimal action-value function Q of the agent * (s, a) needs to be approximated, and the corresponding variational quantum circuit is designed to obtain the action-value function approximator, which is used to approximate the optimal action-value function Q * (s,a);

[0058] Step S3: Design the training architecture and training algorithm of the variational quantum reinforcement learning model, optimize the trainable parameters of the variational quantum circuit, and achieve the optimal action value function Q * (s, a) is an approximation. After training, the model will be able to generate Chinese medicine prescriptions that meet the patient's condition (from the optimal action value function Q * The approximate estimate of (s,a) obtains the following optimal drug selection strategy π *The focus of this invention is to realize the variational quantum circuit for Q through reinforcement learning. * (s,a) approximation).

[0059] Specifically, step S1 is implemented as follows:

[0060] Step S1.1: First, the problem of generating a Chinese medicine prescription targeted by the present invention is described. Suppose a patient p is included, and the symptom set ST of p is recorded. p ={st p,1 ,st p,2 ,...,st p,n}, the elements in the symptom set are TCM symptom words, such as "fatigue", "tongue with teeth marks", "weak pulse", etc.

[0061] The set of syndrome diagnosis results obtained by syndrome differentiation (abbreviated as syndrome differentiation results) is recorded as SD p ={sd p,1 ,sd p,2 ,...,sd p,m In traditional Chinese medicine, common syndrome differentiation results include syndrome types, syndrome elements, etc. Syndrome types include "Qi deficiency and blood stasis syndrome" and "Yang deficiency and cold stagnation syndrome"; syndrome elements include "blood stasis" and "phlegm turbidity". Syndrome differentiation result set SD p The elements in are the names of syndrome types, syndrome elements and other results.

[0062] ST p and SD p Together they constitute the TCM diagnosis result Diag of patient p p =(ST p ,SD p The goal of intelligent generation of Chinese medicine prescriptions is to generate a Chinese medicine prescription based on the patient’s TCM diagnosis results. p , calculate the optimal Chinese medicine prescription Pres * ={ph1,ph2,...,ph l},in H={h1,h2,...,h k} is a set of candidate Chinese medicines (the elements in the set of candidate Chinese medicines are the names of Chinese medicines, such as "ginseng", "astragalus", etc.).

[0063] Step S1.2: The present invention uses variational quantum reinforcement learning to model the problem of Chinese medicine prescription generation and designs a variational quantum reinforcement learning framework for intelligent generation of Chinese medicine prescriptions, such as Figure 1 shown.

[0064] The environment faced by the intelligent agent is the disease; the state of the disease environment is determined by symptoms, syndrome diagnosis results and traditional Chinese medicine to reflect the treatment state of the disease under the action of traditional Chinese medicine; the action taken by the intelligent agent is the action of selecting a single traditional Chinese medicine; the strategy function of the intelligent agent is designed based on the variational quantum circuit (VQC), and the choice of action will be determined by the output of the VQC.

[0065] In the specific learning process, the intelligent agent interacts with the disease environment through drug selection actions, observes the state changes of the disease environment after drug administration, and continuously optimizes the VQC-based strategy function according to the reward value returned by the disease medication evaluation. Finally, the optimal drug selection strategy function based on VQC is obtained, which maximizes the cumulative reward returns obtained from the process of selecting drugs for diseases.

[0066] Formally, the Markov decision process (MDP) corresponding to this process is defined as in, It is the disease-symptom-environment state space; Action space for drug selection for the agent; is the disease-syndrome-environment state transfer function; is the reward function; γ is the discount factor, γ∈[0,1]; define the drug selection strategy π: The goal of the agent is to learn the optimal drug selection strategy π * , so that the cumulative reward for selecting medicines for diseases is maximized, and r is defined t is the immediate reward value obtained at time t, then the learning objective can be defined as follows:

[0067]

[0068] Among them, T is the termination time.

[0069] Variational quantum reinforcement learning will optimize the variational quantum circuits it uses according to the above learning objectives, and achieve approximation of the optimal drug selection strategy function. After that, it can complete the intelligent generation of prescriptions in the form of a drug selection action sequence based on the initial given disease state (such as the patient's traditional Chinese medicine diagnosis results).

[0070] More specifically, under the variational quantum reinforcement learning framework given in step S1.2, the variational quantum reinforcement learning model has the disease-symptom environment state space, agent drug selection action space, disease-symptom environment state transfer function, reward function, termination condition, and optimal strategy approximation, which are as follows:

[0071] Disease-Syndrome-Environment State Space Construction

[0072] Disease-Syndrome-Environment State Space It is determined by the symptom set ST (the elements of the symptom set are TCM symptom words, such as "fatigue", "tongue with teeth marks", "weak pulse", etc.), the syndrome diagnosis result set SD (the elements in the syndrome differentiation result set are the names of syndrome types, syndrome elements and other results, such as "yang deficiency and cold stagnation syndrome", "blood stasis", etc.) and the candidate Chinese medicine set H (the elements in the candidate Chinese medicine set are the names of Chinese medicines, such as "ginseng", "astragalus", etc.).

[0073] First, consider the set U = ST∪SD∪H, which contains all the symptoms, syndrome differentiation results, and Chinese medicine terms involved. Based on the set U, define the state space for:

[0074]

[0075] You can see Any state s in is a 0-1 vector of dimension |U|. According to the order of symptoms first, then syndrome differentiation results, and finally Chinese medicine, the first |ST| bits of the 0-1 state vector correspond to symptom words (the words can be sorted by the first spelling). If a symptom is in a given Chinese medicine diagnosis result, the corresponding position value in the state vector is 1, and the corresponding position value of the symptom that does not appear is 0. Similarly, the syndrome differentiation results correspond to |ST|+1 to |ST|+|SD|, and the relevant values ​​are the same as above. In addition to symptoms and syndrome differentiation results, the state vector also reserves the corresponding position of each candidate Chinese medicine. If the agent selects a medicine when taking an action, the corresponding position value of the medicine in the state vector is 1. For the initial state, the default value of all drug positions is 0.

[0076] Agent drug selection action space Construction

[0077] Agent drug selection action space It consists of drug selection actions based on the candidate Chinese medicine set H. Any action Represents a Chinese medicine selected by the agent in the current state.

[0078] Disease-Syndrome-Environment State Transfer Function Design

[0079] The state transfer function of the disease-symptom environment is deterministic, that is, the Chinese medicine corresponding to the medicine selection action taken by the intelligent agent will directly change the value of the state vector of the current intelligent agent at the corresponding position to 1, which will be used as the new state of the disease-symptom environment feedback after the action is taken.

[0080] Reward Function Design

[0081] The reward function is the key to solving the bottleneck problem of poor interpretability of medication in the sense of traditional Chinese medicine in current intelligent prescription generation. By introducing the traditional Chinese medicine drug screening mechanism into the design of the reward function, the interpretability of intelligent prescriptions can be improved. In the present invention, the reward function is defined by offline simulation of traditional Chinese medicine doctors judging whether the selected drugs are reasonable. For the part of the traditional Chinese medicine diagnosis result in the disease state, if the drug predicted by the agent appears in the list of drugs actually prescribed by the doctor for the diagnosis result, the agent will receive a positive reward r t = 1, otherwise a negative reward r is given t =-1.

[0082] In the present invention, the designed reward function has not yet involved a more practical screening mechanism for traditional Chinese medicine. In the future, the quantitative evaluation of the efficacy of traditional Chinese medicine prescriptions can be introduced into the design of the reward function, so that the prescribed prescriptions can generate explanations for the maximum efficacy of the disease in the sense of traditional Chinese medicine.

[0083] Termination Condition

[0084] The present invention uses the collected Chinese medicine prescription data set (including Chinese medicine prescription records, each prescription record at least including the patient's TCM diagnosis result and the Chinese medicine prescription issued by the doctor) as the learning object of the variational quantum reinforcement learning agent. In each round of learning, the Chinese medicine prescription data set is traversed, and each traversed prescription record is used as the minimum learning unit. The agent takes the action of selecting medicine according to the recorded patient's TCM diagnosis result. When the number of selected medicines is equal to the number of medicinal flavors in the recorded Chinese medicine prescription, the agent terminates the learning of the current prescription record. For each round, it terminates after the Chinese medicine prescription data set is traversed.

[0085] Optimal strategy approximation

[0086] In deep reinforcement learning, there are two ways to obtain the optimal strategy: directly based on the strategy and based on the action value function. * The classic deep reinforcement learning algorithm DQN adopts the action-value function-based approach and designs an action-value function approximator based on a deep neural network to approximate the optimal action-value function Q * (s,a), and from Q * In (s,a), the action corresponding to the maximum action value is taken as the result of strategy selection.

[0087] The present invention adopts an action-value function-based approach to approximate Q through a variational quantum circuit (VQC, VariationalQuantum Circuit) *(s,a), using quantum superposition and quantum entanglement (etc.) quantum mechanical properties, reducing the number of training parameters required for approximation and is expected to accelerate the calculation of action value functions, shortening the training completion time or iteration cycle (compared to traditional neural network-based approximators, VQC can achieve higher computing efficiency with fewer computing resources, can use the superposition state of quantum bits, reduce the dimension of input data, flexibly adapt to task requirements, and make VQC-based Q * The (s,a) approximator has less resource consumption and stronger processing capabilities when dealing with complex and changeable environments. The purpose here is to point out that the present invention designs an action-value function approximator through a variational quantum circuit, and specifically mentions how to perform circuit design in step S2).

[0088] Furthermore, in step S2, a variational quantum circuit design for the action-value function is designed, which includes a state encoding layer, a trainable parameter layer, and a measurement operation. The circuit structure is as follows: Figure 2 As shown, where:

[0089] State coding layer: Amplitude coding circuit is used to encode the classical disease-symptom environment state into the amplitude of the quantum bit superposition state, realizing quantum coding of the disease-symptom environment state. Specifically, in the present invention, according to the above definition, the classical disease-symptom environment state s is a 0-1 vector (x1, x2, ..., x |U| ); Calculate the value according to the dimension of s n is the number of qubits required by the amplitude coding circuit. Furthermore, if n>log2|U|, then fill s to obtain the vector Among them, the filling component The value range is (0,1); if n = log2|U|, let the vector s′ = s. Further, construct an amplitude coding circuit to enable it to prepare a superposition state of n quantum bits based on the normalization of the s′ component. (|i> is the calculation basis vector). The state coding layer of the present invention adopts the aforementioned amplitude coding circuit, and on the basis of filling and normalizing the classical disease-symptom environment state vector, encodes the classical disease-symptom environment state into the amplitude of the quantum bit superposition state, thereby realizing quantum coding of the disease-symptom environment state.

[0090] Trainable parameter layer: A trainable parameter layer based on the R(α,β,γ) three-parameter rotation gate and CNOT gate is used to make the entire circuit a strongly entangled variational quantum circuit. Specifically, on a circuit with m≥2 qubits, for each qubit q j ∈{1,2,...,m}, respectively set a three-parameter revolving door R(α j ,β j ,γ j).

[0091] Based on the setting of the three-parameter revolving door mentioned above, the CNOT gate is set again. A total of m CNOT gates need to be set. Specifically, for the first CNOT gate The output of q1 after R(α1, β1, γ1) operation is used as the control bit input (the solid area in the CNOT gate schematic diagram), and the output of q2 after R(α2, β2, γ2) operation is used as the target bit input (the hollow area in the CNOT gate schematic diagram).

[0092] For all CNOT gates except the last one Its q j After R(α j ,β j ,γ j )and The output after operation is used as the control bit input, with q j+1 After R(α j+1 ,β j+1 ,γ j+1 ) operation is used as the target bit input.

[0093] For the final CNOT gate Its q m After R(α m ,β m ,γ m )and The output after the operation is used as the control bit input, with q1 passing through R(α1, β1, γ1) and The output obtained after the operation is used as the target bit input.

[0094] After clarifying the setting process of the R (α, β, γ) three-parameter rotation gate and the CNOT gate, it is necessary to further clarify how many quantum bits the trainable parameter layer should be based on in the variational quantum reinforcement learning model for intelligent generation of traditional Chinese medicine prescriptions constructed by the present invention. Specifically, the present invention sets each drug selection action a i Corresponding to a quantum bit q i , a i The action value function value Q(s,α i ) is built on q i The drug selection action space proposed by the present invention is based on the measurement results obtained after the variational quantum circuit processing and measurement. Number of actions This means that the need qubits to obtain the action value function value of the drug selection action. And because the amplitude coding circuit used in the aforementioned state coding layer requires qubits, so the variational quantum circuit designed by the present invention needs to be established on The trainable parameter layer is located here. It is built on the basis of qubits.

[0095] Related research shows that compared with variational quantum circuits with weak ability to generate high entanglement quantum states, strongly entangled variational quantum circuits can more effectively capture the complex relationships between elements in the data and improve the circuit's ability to learn and express complex functions. By constructing the aforementioned trainable parameter layer, the variational quantum circuit designed by the present invention can effectively achieve an approximation of the optimal action-value function.

[0096] Measurement operation: Use Pauli-Z measurement to measure the quantum state of the quantum bit corresponding to each drug selection action after being processed by the variational quantum circuit. Expand the expected value of the Pauli-Z observable obtained according to the value range of the action value function, and use the final result as an approximate estimate of the action value function value of the variational quantum circuit for each drug selection action.

[0097] Furthermore, for step S3, a model training architecture based on the target network and experience replay mechanism and a model training algorithm based on Deep Q-Learning are designed. Through model training, the intelligent agent completes the optimization adjustment of the trainable parameters of the variational quantum circuit and realizes the optimal action value function Q * (s,a). Estimate Q through variational quantum circuit approximation * (s,a), obtain the optimal drug selection strategy π * , based on this strategy, a traditional Chinese medicine prescription that meets the patient's condition is generated.

[0098] Furthermore, for the model training architecture designed in step S3 based on the target network and the experience replay mechanism, such as Figure 3 shown.

[0099] The agent has the subject VQC-Q * Function approximator VQC_Q(s,a;θ) and target VQC-Q * Function approximator VQC_Q(s,a;θ - ). At time t, the agent observes the disease environment state s t , by calculating Get the drug selection action a taken t ; The disease environment is based on the received drug selection action a t , change its state to s t+1 And feedback reward value r t ; The learning experience generated at time t (s t ,at ,r t ,s t+1 ) is stored in the replay memory; the agent learns based on the learning experience (s j ,a j ,r j ,s j+1 ), for the subject VQC-Q * The function approximator VQC_Q(s,a;θ) performs parameter optimization based on the loss function,

[0100]

[0101] The optimization algorithm selected for parameter optimization is the AdamW algorithm. Each optimization will make the subject VQC-Q * The parameter θ of the function approximator VQC_Q(s,a;θ) is adjusted; every C iterations, the agent applies the principal VQC-Q * The parameter θ of the function approximator VQC_Q(s,a;θ) is copied to the target VQC-Q * Function approximator VQC_Q(s,a;θ - ) parameter θ - Among.

[0102] Furthermore, for the model training algorithm designed based on Deep Q-Learning in step S3, the process is as follows Figure 4 As shown, the specific implementation is as follows:

[0103] Step S3.1: Given a Chinese medicine prescription dataset P;

[0104] Step S3.2: Initialize the playback memory D to capacity N and use random parameters to initialize the main VQC-Q * Function approximator VQC_Q(s,a;θ), and main VQC-Q * The parameters θ of the function approximator initialize the target VQC-Q * Function approximator VQC_Q(s,a;θ - ), where θ - =θ (the variational quantum circuit corresponding to the initial VQC_Q(s,a;θ) has not been trained and optimized yet);

[0105] Step S3.3: Start M training rounds, and in each round, under a given Chinese medicine prescription data set P, learn each prescription record in the data set one by one;

[0106] Step S3.3: Start learning the i-th prescription record, and initialize the disease state s1 with the patient's TCM diagnosis result in the current prescription record P[i];

[0107] Step S3.4: According to the number of Chinese medicines in the prescription record P[i], len(P[i].herbs), start the tth iteration, and each iteration uses the ε-greedy strategy based on the Q function to select action a t ;

[0108] Step S3.5: Take action a t And observe the reward r t and the next state s t+1 , will learn experience (s t ,a t ,r t ,s t+1 ) is stored in the playback memory;

[0109] Step S3.6: Randomly sample a batch of learning experiences from the replay memory D, calculate the loss value and the gradient of the parameter θ according to the defined loss function, and use the parameter optimizer to optimize and adjust the parameter θ;

[0110] Step S3.7: Update the target VQC-Q every C steps * The parameters of the function approximator θ - =θ;

[0111] Step S3.8: When M training rounds are completed, the algorithm terminates.

[0112] In terms of constructing a variational quantum reinforcement learning model for intelligent generation of TCM prescriptions, it is necessary to design a variational quantum reinforcement learning framework for the scenario of intelligent generation of TCM prescriptions, clarify the basic process of using variational quantum reinforcement learning to achieve prescription generation, and define the disease-symptom environment state space and agent drug selection action space under this framework, design the environment state transfer function and reward function, set the termination condition, and determine the optimal strategy approximation method;

[0113] In the design of variational quantum circuits, for the optimal action value function Q * To approximate (s, a), we need to design the corresponding variational quantum circuit, including designing the state encoding layer, the trainable parameter layer, and the measurement layer.

[0114] In the training of the variational quantum reinforcement learning model for Chinese medicine prescription generation, in order to optimize the trainable parameters of the variational quantum circuit and achieve the optimal action value function Q * To approximate (s,a), it is necessary to design a training architecture and training algorithm.

[0115] In summary, the variational quantum reinforcement learning agent plays the role of a Chinese medicine doctor. By interacting with the disease environment, it optimizes the optimal action-value function approximator based on VQC and realizes the optimal action-value function Q * (s,a) by approximating Q *(s,a), obtain the optimal drug selection strategy π * , and complete the intelligent generation of Chinese medicine prescriptions based on the optimal strategy.

[0116] It is worth mentioning that the technical features such as the traditional Chinese medicine prescription involved in the patent application of this invention should be regarded as the prior art. The specific structure, working principle and possible control method and spatial layout method of these technical features can be selected by conventional methods in the field, and should not be regarded as the inventive point of this patent. This patent will not be further elaborated.

[0117] For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions, characterized in that: The following steps are involved: Step S1: construct a variational quantum reinforcement learning model for intelligent generation of traditional Chinese medicine prescriptions, thereby realizing variational quantum reinforcement learning modeling for traditional Chinese medicine prescription generation; Step S2: Design a variational quantum circuit for the action-value function. Aiming at the need to approximate the optimal action-value function Q*(s, a) possessed by the agent, design a corresponding variational quantum circuit, thereby obtaining an action-value function approximator for approximating the optimal action-value function Q*(s, a); Step S3: Design the training architecture and training algorithm of the variational quantum reinforcement learning model, complete the optimization of the trainable parameters of the variational quantum circuit, and achieve approximation of the optimal action value function Q*(s, a). After the training is completed, the model will be used to generate a traditional Chinese medicine prescription that meets the patient's condition.

2. A variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions according to claim 1, characterized in that: Step S1 is specifically implemented as follows: Step S1.1: Suppose patient p, record p's symptom set ST p ={st p，1 , st p，2 ,...,st p，n }, the syndrome diagnosis result set obtained by syndrome differentiation is recorded as SD p ={sd p，1 , s.d. p，2 , ..., sd p，m }, ST p and SD p Together they constitute the TCM diagnosis result Diag of patient p p =(ST p , S.D. p ), the goal of intelligent generation of Chinese medicine prescriptions is to generate a Chinese medicine prescription based on the patient’s TCM diagnosis results. p , calculate the optimal Chinese medicine prescription Pres * = {ph1, ph2, ..., ph l },in H = {h1, h2, ..., h k } is a set of candidate Chinese medicines; Step S1.2: Use variational quantum reinforcement learning to model the problem of Chinese medicine prescription generation, and design a variational quantum reinforcement learning framework for intelligent generation of Chinese medicine prescriptions. The environment faced by the intelligent agent is the disease syndrome; the disease syndrome environment state is jointly determined by symptoms, syndrome diagnosis results and Chinese medicine; In the specific learning process, the agent interacts with the disease environment through drug selection actions, observes the state changes of the disease environment after drug administration, and continuously optimizes the VQC-based strategy function according to the reward value returned by the disease medication evaluation, and finally obtains the optimal drug selection strategy function based on VQC, which maximizes the cumulative reward return obtained in the process of selecting drugs for diseases. Formally, the Markov decision process corresponding to this process is defined as in, It is the disease-symptom-environment state space; Action space for drug selection for the agent; is the disease-syndrome-environment state transfer function; is the reward function; γ is the discount factor, γ∈[0,1]; define the drug selection strategy The goal of the agent is to learn the optimal drug selection strategy π*, so that the cumulative reward for drug selection for the disease is maximized. Define r t is the immediate reward value obtained at time t, then the learning objective can be defined as follows: Among them, T is the termination time.

3. A variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions according to claim 2, characterized in that: Under the variational quantum reinforcement learning framework given in step S1.2, the variational quantum reinforcement learning model has the disease-symptom environment state space, the agent drug selection action space, the disease-symptom environment state transfer function, the reward function, the termination condition and the optimal strategy approximation, which are as follows: Disease-Syndrome-Environment State Space Construction Disease-Syndrome-Environment State Space It is determined by the symptom set ST, syndrome diagnosis result set SD and candidate Chinese medicine set H; First, consider the set U = ST∪SD∪H, which contains all the symptoms, syndrome differentiation results, and Chinese medicine terms involved. Based on the set U, the disease environment state space S is defined as: Any state s in is a 0-1 vector with dimension |U|. In the order of symptoms first, then syndrome differentiation results, and finally Chinese medicine, the first |ST| bits of the 0-1 state vector correspond to symptom words. If a symptom appears in a given Chinese medicine diagnosis result, the corresponding position value in the state vector is 1, and the corresponding position value of the symptom that does not appear is 0; similarly, the syndrome differentiation results correspond to |ST|+1 to |ST|+|SD| bits, and the relevant values ​​are the same as above; In addition to symptoms and syndrome differentiation results, the state vector also contains the corresponding position of each candidate Chinese medicine. If the agent selects a medicine when taking an action, the corresponding position value of the medicine in the state vector is 1. For the initial state, the default value of all medicine positions is 0. Agent drug selection action space Construction Agent drug selection action space It consists of drug selection actions based on the candidate Chinese medicine set H. Any action Represents a Chinese medicine selected by the agent in the current state; Disease-Syndrome-Environment State Transfer Function Design The state transfer function of the disease-symptom environment is deterministic, that is, the Chinese medicine corresponding to the drug selection action taken by the agent will directly change the value of the state vector of the current agent at the corresponding position to 1, which will be used as the new state of the disease-symptom environment feedback after taking the action; Reward Function Design The reward function is defined by offline simulation of TCM doctors judging whether the selected medicine is reasonable. For the TCM diagnosis result part of the disease state, if the medicine predicted by the agent appears in the list of medicines actually prescribed by the doctor for the diagnosis result, the agent will receive a positive reward r t = 1, otherwise a negative reward r is given t = -1; Termination Condition The collected TCM prescription data set is used as the learning object of the intelligent agent. In each round of learning, the TCM prescription data set is traversed, and each traversed prescription record is used as the minimum learning unit. The intelligent agent takes the action of selecting medicine according to the recorded TCM diagnosis results of the patient. When the number of selected medicines is equal to the number of medicinal ingredients in the recorded TCM prescription, the intelligent agent terminates the learning of the current prescription record. For each round, it terminates after the TCM prescription data set is traversed. Optimal strategy approximation Based on the action-value function approach, Q*(s, a) is approximated by variational quantum circuits. The quantum mechanical properties of quantum superposition and quantum entanglement are utilized to reduce the number of training parameters required for approximation and accelerate the calculation of the action-value function, thereby shortening the training completion time or iteration cycle.

4. A variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions according to claim 3, characterized in that: In step S2, a variational quantum circuit for the action-value function is designed, which includes a state encoding layer, a trainable parameter layer, and a measurement operation, wherein: State coding layer: using amplitude coding circuit, the classical disease and environmental state is encoded into the amplitude of the quantum bit superposition state, thus realizing quantum coding of the disease and environmental state; Trainable parameter layer: A trainable parameter layer based on the R (α, β, γ) three-parameter rotation gate and CNOT gate is used to make the entire circuit a strongly entangled variational quantum circuit; Measurement operation: Use Pauli-Z measurement to measure the quantum state of the quantum bit corresponding to each drug selection action after being processed by the variational quantum circuit. Expand the expected value of the Pauli-Z observable obtained according to the value range of the action value function, and use the final result as an approximate estimate of the action value function value of the variational quantum circuit for each drug selection action.

5. A variational quantum reinforcement learning algorithm for intelligent generation of traditional Chinese medicine prescriptions according to claim 4, characterized in that: For step S3, a model training architecture based on the target network and experience replay mechanism and a model training algorithm based on Deep Q-Learning are designed. Through model training, the intelligent agent completes the optimization adjustment of the trainable parameters of the variational quantum circuit, realizes the approximation of the optimal action value function Q*(s, a), and estimates Q*(s, a) through the variational quantum circuit to obtain the optimal drug selection strategy π*. According to the strategy, a Chinese medicine prescription that meets the patient's condition is generated. The training algorithm is specifically implemented in the following steps: Step S3.1: Given a Chinese medicine prescription dataset P; Step S3.2: Initialize the playback memory D to capacity N and use random parameters to initialize the main VQC-Q * function approximator VQC_Q(s, a; θ), and initialize the target VQC-Q* function approximator VQC_Q(s, a; θ-) with the parameters θ of the subject VQC-Q* function approximator, where θ-=θ; Step v3.3: Start M training rounds. In each round, each prescription record in the given TCM prescription dataset P is learned one by one. Step S3.3: Start learning the i-th prescription record, and initialize the disease state s1 with the patient's TCM diagnosis result in the current prescription record P[i]; Step S3.4: According to the number of Chinese medicines in the prescription record P[i], len(P[i].herbs), start the tth iteration, and each iteration uses the ε-greedy strategy based on the Q function to select action a t ; Step S3.5: Take action a t And observe the reward r t and the next state s t+1 , will learn experience (s t , a t , r t ,s t+1 ) is stored in the playback memory; Step S3.6: Randomly sample a batch of learning experiences from the replay memory D, calculate the loss value and the gradient of the parameter θ according to the defined loss function, and use the parameter optimizer to optimize and adjust the parameter θ; Step S3.7: Update the parameter θ-=θ of the target VQC-Q* function approximator every C steps; Step S3.8: When M training rounds are completed, the algorithm terminates.

Images