Reinforcement learning method based on quantum classical hybrid architecture

Abstract

The invention provides a reinforcement learning method based on a quantum classical hybrid architecture, electronic equipment and a non-instantaneous computer readable storage medium. The method comprises the following steps: determining a reinforcement learning environment; according to the environment, constructing a quantum network and a classic network to form an artificial intelligence model with a quantum and classic hybrid architecture; and training the artificial intelligence model to obtain a model after reinforcement learning.

Description

Technical Field

[0001] The present application relates to the fields of quantum computing and quantum artificial intelligence, and specifically to a reinforcement learning method, electronic device, and non-transient computer-readable storage medium based on a quantum-classical hybrid architecture. Background Art

[0002] In recent years, with the development of quantum computing technology, quantum hardware has made rapid progress. The combination of quantum hardware and classical machine learning, especially in the field of quantum reinforcement learning, has shown great potential in recent years. Under this development trend, exploring how to use quantum hardware to enhance the solution of traditional computing problems and how to migrate traditional reinforcement learning algorithms to quantum hardware platforms have become the focus of research.

[0003] On the one hand, quantum computing cloud platforms such as IBMQ, Amazon Braket, and Quafu play an important role in the NISQ era, bridging the gap between quantum software application developers and cutting-edge quantum hardware. On the other hand, both policy-based and value-based quantum reinforcement learning methods have shown great potential in solving standard tasks and theoretically surpassing traditional algorithms.

[0004] As a result, the rapid development of quantum computing has made researchers pay more attention to how to use quantum hardware to accelerate the solution of traditional computing problems. In this context, the research on quantum reinforcement learning algorithms is particularly important. However, current noisy intermediate-scale quantum (NISQ) technology is constrained by factors such as the number of quantum bits, quantum state decoherence, and imprecise quantum operations, which limit the realization of the so-called quantum advantage. Therefore, in quantum reinforcement learning, designing algorithms that can migrate classical algorithms to quantum computers and reduce the impact of device noise has become an extremely challenging task. Summary of the invention

[0005] In order to solve the above problems in the prior art, the present application provides a reinforcement learning method, an electronic device and a non-transitory computer-readable storage medium based on a quantum classical hybrid architecture.

[0006] According to one aspect of the present application, a reinforcement learning method based on a quantum classical hybrid architecture is provided, comprising:

[0007] Determine the environment for reinforcement learning;

[0008] According to the environment, constructing a quantum network and a classical network to form an artificial intelligence model with a quantum-classical hybrid architecture; and

[0009] The artificial intelligence model is trained to obtain a model trained by reinforcement learning.

[0010] According to another aspect of the present application, there is provided an electronic device, including:

[0011] Processor; and

[0012] The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the method as described above.

[0013] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor is caused to execute the method as described above.

[0014] Therefore, the strategy of a classical network plus a quantum network constructed by this application can significantly improve the performance of quantum reinforcement learning algorithms on noisy quantum devices. By combining quantum networks with classical networks, this application combines the advantages of traditional and quantum computing when dealing with specific reinforcement learning problems, thereby achieving more effective quantum reinforcement learning on current quantum hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a reinforcement learning method based on a quantum-classical hybrid architecture according to one embodiment of the present application is shown.

[0017] Figure 2 A flowchart for constructing a quantum network according to one embodiment of the present application is shown.

[0018] Figure 3 A schematic diagram of a quantum network constructed according to an embodiment of the present application is shown.

[0019] Figure 4 A schematic diagram of a classical neural network constructed according to one embodiment of the present application is shown.

[0020] Figure 5 A flowchart for training an artificial intelligence model according to one embodiment of the present application is shown.

[0021] Figure 6 A flowchart of a reinforcement learning method based on a quantum-classical hybrid architecture according to another embodiment of the present application is shown.

[0022] Figure 7 A schematic diagram of a frame of an animation of executing step S140 on quantum hardware according to one embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to better understand the technical solutions and advantages of the present application, the contents of the present application are further described in detail below in conjunction with the accompanying drawings and specific embodiments. However, the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be used in combination except for situations where they conflict with each other, thereby constituting other embodiments within the scope of the present application.

[0024] The content described below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numerals and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, and itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0025] In the methods described below, except for the steps with a clear sequence, the execution order of the remaining steps can be changed to form a new technical solution, which also falls within the scope of protection of this application. In other words, whether it is the sequence shown in the drawings or the sequence described in the specification, it does not mean that the specific execution order between the steps is so fixed, but is only for illustration.

[0026] Figure 1 FIG. 1 is a flow chart of a reinforcement learning method based on a quantum classical hybrid architecture according to an embodiment of the present application. Figure 1 As shown, the method 100 may include steps S110 to S130.

[0027] In step S110, the environment of reinforcement learning is determined. In this step, the environment of reinforcement learning is determined, that is, the specific reinforcement learning problem to be solved is determined. In the present application, the environment of reinforcement learning can be any one of a plurality of preset benchmark test environments, for example, it can be a benchmark test environment provided in the OpenAI gym open source library. In addition, the user can also customize the reinforcement learning environment.

[0028] According to one embodiment of the present application, the environment of reinforcement learning can be based on a CartPole environment. The CartPole environment in the OpenAI Gym open source library is a classic problem for testing the performance of reinforcement learning algorithms. The environment simulates a vertical pole on a cart, and the goal is to keep the pole balanced in a vertical position by moving the cart left and right. The CartPole environment is a discrete action space problem, and its actions can be moving the cart to the left and moving the cart to the right so that the cart does not fall on the balance pole.

[0029] In step S120, according to the determined environment, a quantum network and a classical network are constructed to form an artificial intelligence model with a quantum-classical hybrid architecture. According to one embodiment of the present application, the quantum-classical hybrid architecture constructed in this step can be an actor-critic network architecture, wherein the actor network (strategy network) can be a quantum network, and the critic network (value network) can be a classical neural network. In another embodiment, the actor network (strategy network) can be a classical neural network, and the critic network (value network) can be a quantum network.

[0030] In this application, quantum networks refer to parameterized quantum circuits (PQCs). Parameterized quantum circuits can play a role in quantum reinforcement learning algorithms similar to that of classical neural networks in deep reinforcement learning.

[0031] In step S130, the artificial intelligence model is trained to obtain a model trained by reinforcement learning. In this step, based on the network constructed in step S120, a quantum PPO (Proximal Policy Optimization) algorithm can be executed based on quantum hardware to obtain training model parameters within a limited interaction process and achieve the preset goals of the environment. The specific training process will be described in detail below.

[0032] The current quantum reinforcement learning algorithms are basically implemented on quantum simulators on classical computers to solve several standard benchmark environments, such as the CartPole problem and the MountainCar problem, which are simple reinforcement learning models from the OpenAI Gym open source library. However, most of the current hardware devices are noisy devices. If the algorithm on the simulator is used directly on these hardware devices, the algorithm's ability to complete the task will be greatly reduced.

[0033] This application takes into account the characteristics of real quantum hardware and provides a quantum-classical hybrid version of reinforcement learning algorithm, such as the proximal policy optimization (PPO) algorithm, which is specially designed for quantum reinforcement learning on current noisy quantum computing hardware. Different from the existing quantum reinforcement learning algorithm scheme, this application takes into account that the reinforcement learning algorithm of the actor-critic architecture includes a value network and a policy network, so the proposed strategy is to build one network as a quantum network and the other network as a classical network.

[0034] Since current quantum devices generally have noise, the constructed classical network can be regarded as an error correction network, which can play a role in noise reduction and error correction. This strategy of a classical network plus a quantum network can significantly improve the performance of quantum reinforcement learning algorithms on noisy quantum devices. Through the combined use of quantum networks and classical networks, this application combines the advantages of traditional and quantum computing when dealing with specific reinforcement learning problems (for example, CartPole environment), thereby achieving more effective quantum reinforcement learning on current quantum hardware.

[0035] Figure 2 A flowchart for constructing a quantum network according to one embodiment of the present application is shown. Figure 3 FIG. 1 shows a schematic diagram of a quantum network constructed according to an embodiment of the present application. Figure 3 As shown, the quantum network acts as an actor network, that is, a strategy network.

[0036] like Figure 2 and Figure 3 As shown, step S120 may include sub-step S121. In sub-step S121, a plurality of quantum bits are arranged in the quantum network, wherein the number of the quantum bits corresponds to the state space dimension of the environment.

[0037] Specifically, Figure 2 and Figure 3 As shown in the figure, still taking the CartPole environment as an example, in the process of constructing an actor-critic network for the problem, the quantum network is used as the actor network and the classical neural network is used as the critic network. Since the state space of the CartPole environment is four-dimensional, four quantum bits |0>0, |0>1, |0>2, |0>3 are set in the quantum network. Figure 3 As shown, in a quantum network, each quantum bit can be connected to an H gate, which can make the quantum bit enter an entangled state, thereby making the algorithm more stable.

[0038] Furthermore, if Figure 2 and Figure 3 As shown, in addition to sub-step S121, step S120 may further include sub-steps S122 to S125.

[0039] In sub-step S122, a variational quantum circuit layer x1(θ) is set in the quantum network. The variational quantum circuit layer x1(θ) is connected through a plurality of single-bit rotation gates R x , R y , R z The circuits acting on multiple quantum bits are used for parameter training. In the entire quantum network, the variational quantum circuit layer x1(θ) is the parameter-containing main training part, where θ is a trainable parameter.

[0040] In sub-step S123, a state coding layer x2(d,λ) is set in the quantum network, which is used to encode the state of the environment (by applying R to each quantum bit). x ), thereby encoding the classical state into a quantum state, where d represents the environment state and λ is a trainable scale parameter of the state.

[0041] In sub-step S124, a quantum entanglement layer x3 is set in the quantum network, and the quantum entanglement layer x3 is used to enable the features encoded as quantum states to interact and entangle. Figure 3 As shown, the quantum entanglement layer x3 can use multiple controlled Z gates to achieve circular entanglement, with a controlled Z gate set between every two adjacent quantum bits and a controlled Z gate set between the first quantum bit and the last quantum bit.

[0042] In sub-step S125, a measurement layer x0(Ψ) is set in the quantum network, and the measurement layer x0(Ψ) is used to measure the circuits of multiple quantum bits, so as to measure the quantum state data as classical state data output. Figure 3 As shown, the measurement layer x0(Ψ) includes x1(Ψ) and four Pauli Z gates, where Ψ is a trainable parameter, and each Pauli Z gate is located at the end of the line of each quantum bit as an observable.

[0043] Figure 4 A schematic diagram of a classical neural network constructed according to an embodiment of the present application is shown. In step S120, it may also include: setting an input layer, a hidden layer and an output layer in the classical network to form a classical neural network.

[0044] like Figure 4 As shown, the classic neural network is used as a critic network, that is, a value network, which may include an input layer, two hidden layers, and an output layer, wherein the two hidden layers may have 64 neurons, the activation functions of the input layer and the hidden layer are hyperbolic tangent functions, and the output layer has only one neuron without an activation function. Figure 4In the equation, x1, x2, x3, and x4 represent the input variables of the classical neural network, and the actual input is the environmental state d0, d1, d2, and d3; b (0) 、b (1) 、b (3) represents the bias of the classic neural network; w (0) 、w (1) 、w (3) represents the weight value, which is the parameter that needs to be learned in the classical neural network; z1 represents the output variable, which is actually the value output by the critic. Those skilled in the art can understand that any appropriate classical neural network in the prior art can be selected according to the environment (i.e., problem) of reinforcement learning.

[0045] Figure 5 FIG. 1 shows a flow chart of training an artificial intelligence model according to an embodiment of the present application, which represents the operation process of the quantum PPO algorithm. Figure 5 As shown, step S130 may include sub-steps S131 to S134.

[0046] In sub-step S131, the parameters of the environment and the parameters of the quantum network and the classical network are initialized. For example, the various parameters described above can be initialized in this sub-step.

[0047] In sub-step S132, it is determined whether the preset training completion condition is met. Generally, the preset training completion condition can be, for example, the average cumulative reward reaches a preset value, or reaches a preset maximum number of iterations N. If the preset training completion condition is met, the training process is terminated; if the preset training completion condition is not met, the next interaction between the quantum PPO algorithm and the environment is started, that is, sub-steps S133 and S134 are continued.

[0048] In sub-step S133, when the training completion condition is not met, trajectory information is collected and a generalized advantage estimation function is calculated based on the current parameters of the environment and the current parameters of the quantum network and the classical network.

[0049] In this substep, the current strategy (i.e., according to the current parameters of the environment and the current parameters of the quantum network and the classical network) is used to collect the trajectory {s1, a1, r1, s2, a2, r2, ...}, where s, a, and r represent the state of the environment, the action, and the reward, respectively. Specifically, the collected agent-related information s i Input into the policy neural network (in this embodiment, a quantum network), we can get action a i , and then let the agent take action a i , the environment will change from s i becomes i+1 , then the environment returns a reward r i. This allows the collection of trajectory data.

[0050] Then, the generalized advantage estimation function is calculated based on the collected trajectory and the value network (a classical neural network in this embodiment). The specific calculation process is as follows:

[0051] First, calculate the timing difference error δ t =r t +γV(s t+1 )-V(s t ), where r t Is to complete action a t The reward obtained, V represents the value function, specifically the critic in this case; γ represents the discount used to measure future rewards.

[0052] Then, the generalized advantage estimation function is calculated Where λ is a hyperparameter introduced for the generalized advantage estimation function, and T represents the total time step of an interaction.

[0053] In sub-step S134, the policy gradient function is calculated according to the generalized advantage estimation function, and the parameters of the environment and the parameters of the quantum network and the classical network are updated.

[0054] In this sub-step, we first need to calculate the probability π(a t |s t’ θ), where θ is a trainable parameter in the policy network (i.e., quantum network), and calculates the probability π of taking each action under the old policy corresponding to the policy used during sampling old (a t |s t’ θ old Then, the cutoff ratio π(a t |s t’ θ) / π old (a t |s t’ θ old ), and then calculate the truncated policy gradient function where r(θ) is the cutoff ratio and ε is the cutoff parameter, which is a small constant. It is the generalized advantage estimation function calculated in step sub-step S133, and clip represents the clipping function, which limits r(θ) to a preset range. Afterwards, the parameters of the policy network (i.e., quantum network) are updated according to the objective function maximized according to the truncated policy gradient function, where the truncated policy gradient function L_clip, truncation means: it is expected to be around 1, so when the calculated value is outside [1-epision, 1+epision], the boundary value is taken, where epision is a preset threshold. Then, using the above-collected trajectory data, the parameters of the value network (i.e., classical network) are updated by minimizing the mean square error of the value function, and the environmental parameters are updated. Here, the purpose of minimizing the mean square error of the value function (i.e., the value function loss) is to optimize the state value function V so that it can more accurately estimate the expected return of a given state. Specifically, the value function loss measures the error between the return predicted by the value function and the actual return.

[0055] After sub-step S134 is completed, return to sub-step S132 to determine again whether the preset training completion condition is met, and repeat the cycle until the training completion condition is met, for example, the average cumulative reward reaches a preset value, or reaches a preset maximum number of iterations N.

[0056] Figure 6 FIG. 2 shows a flow chart of a reinforcement learning method based on a quantum classical hybrid architecture according to another embodiment of the present application. Figure 6 As shown, in addition to steps S110 to S130, the method 100' may further include step S140. For the sake of brevity, only the following will describe Figure 6 The embodiment shown is Figure 1 The differences between the implementation modes will not be described in detail.

[0057] In step S140, the model trained by reinforcement learning interacts with the environment to verify the training effect. In this step, the quantum circuit parameters of the model trained by reinforcement learning obtained in step S130 are loaded, the circuit is executed on the quantum hardware, the strategy is output, the action is selected, and the environment is interacted until the round ends, thereby verifying the training effect of step S130. Figure 7 A schematic diagram of a frame of an animation of executing step S140 on quantum hardware according to one embodiment of the present application is shown.

[0058] This application provides a complete hardware-based quantum reinforcement learning method. The above only takes the cart pole problem (CartPole) as an example, which can also be used to solve other reinforcement learning environments and application scenarios.

[0059] According to another aspect of the present application, an electronic device is provided, which may include a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method as described above.

[0060] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor executes the method as described above.

[0061] Those skilled in the art will appreciate that the technical solution of the present application may be implemented as a system, method or computer program product. Therefore, the present application may be in the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware, which may generally be referred to as a "circuit", "module" or "system". In addition, the present application may be in the form of a computer program product, which is embedded in any tangible expression medium, and the tangible expression medium has a computer-usable program code embedded in the medium.

[0062] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It is understood that each box in the flowchart and / or block diagram, as well as the combination of multiple boxes in the flowchart and / or block diagram, can be executed by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0063] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer or other programmable data processing device to implement functions in a specific manner, so that the instructions stored in the computer-readable medium produce an instruction device including an instruction device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0064] Computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0065] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each frame in the flow chart or block diagram may represent a part of a module, section or code, which includes one or more executable instructions for realizing a specific logical function. It should also be noted that in some alternative implementations, the function marked in the frame may not occur in the order marked in the accompanying drawings. For example, depending on the functionality involved, the two frames shown in succession can actually be executed roughly at the same time, or these frames are sometimes executed in the opposite order. It can also be noted that each frame in the block diagram and / or the flow chart diagram, and the combination of multiple frames in the block diagram and / or the flow chart diagram can be realized by a hardware-based system or a combination of special-purpose hardware and computer instructions for the special purpose of performing a specific function or action.

[0066] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. A reinforcement learning method based on a quantum-classical hybrid architecture, comprising: Determine the environment for reinforcement learning; According to the environment, a quantum network and a classical network are constructed to form an artificial intelligence model with a quantum-classical hybrid architecture; as well as The artificial intelligence model is trained to obtain a model trained by reinforcement learning.

2. The method of claim 1, wherein the quantum-classical hybrid architecture is an actor-critic network architecture, and the quantum network and the classical network are respectively one of a value network and a policy network in the actor-critic network architecture.

3. The method of claim 1, wherein the environment is one of a plurality of preset benchmark test environments, or a user-defined reinforcement learning environment.

4. The method of claim 1, further comprising: The model trained by reinforcement learning interacts with the environment to verify the training effect.

5. The method of claim 1, wherein according to the environment, a quantum network and a classical network are constructed to form an artificial intelligence model with a quantum-classical hybrid architecture, comprising: A plurality of quantum bits are arranged in the quantum network, wherein the number of the quantum bits corresponds to the dimension of the state space of the environment.

6. The method of claim 5, wherein according to the environment, a quantum network and a classical network are constructed to form an artificial intelligence model with a quantum-classical hybrid architecture, further comprising: The following layers are also provided in the quantum network: A variational quantum circuit layer, which acts on the circuits of the plurality of quantum bits respectively through a plurality of single-bit rotation gates for parameter training; A state encoding layer for encoding the state of the environment, thereby encoding the classical state into a quantum state; A quantum entanglement layer, used to enable the interaction and entanglement between features encoded into quantum states; as well as The measurement layer is used to measure the circuits of the plurality of quantum bits, thereby measuring the quantum state data as classical state data output.

7. The method of claim 1, wherein according to the environment, a quantum network and a classical network are constructed to form an artificial intelligence model with a quantum-classical hybrid architecture, comprising: An input layer, a hidden layer, and an output layer are set in the classical network.

8. The method of claim 1, wherein training the artificial intelligence model comprises: i. Initialize the parameters of the environment and the parameters of the quantum network and the classical network; ii. Determine whether the preset training completion conditions are met; iii. If the training completion condition is not met, collecting trajectory information and calculating a generalized advantage estimation function according to current parameters of the environment and current parameters of the quantum network and the classical network; iv. calculating a policy gradient function according to the generalized advantage estimation function, and updating the parameters of the environment and the parameters of the quantum network and the classical network; v. Return to step ii until the training completion condition is met.

9. An electronic device, comprising: processor; as well as A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

Images