Quantum Computing through Reinforcement Learning

Abstract

To provide quantum calculation via a reinforcement learning.SOLUTION: The present invention provides a method, a system, and a device for designing a quantum control locus for implementing a quantum gate by using quantum hardware. According to an embodiment, the method includes: an action to express a quantum gate as a sequence of control actions; and an action to repeatedly adjust each control action in the sequence of control actions, implementing the quantum gate, and apply a reinforcement learning model for determining a quantum control locus that reduces leakage of the quantum gate, a non-fidelity, and a total execution time in order to improve its robustness of performance against control noise during repetitive adjustments.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This specification relates to quantum computing. [Background technology]

[0002] Large-scale quantum computers have the potential to provide fast solutions to several classes of difficult problems. In gate-model quantum computers, computational tasks are performed by implementing a sequence of universal quantum gates, each of which specifies a trajectory for the evolution of quantum computers. The faster the quantum gates execute, the more computational power a given quantum device possesses. Summary of the Invention [Means for solving the problem]

[0003] This specification describes methods and systems for designing quantum control trajectories using reinforcement learning.

[0004] In general, one inventive aspect of the subject matter described herein may be implemented in a method for designing a quantum control trajectory for implementing a quantum gate using quantum hardware, the method comprising the actions of: representing the quantum gate as a sequence of control actions; iteratively adjusting each control action in the sequence of control actions to implement the quantum gate, and applying a reinforcement learning model to determine a quantum control trajectory that reduces leakage, non-fidelity, and total execution time of the quantum gate during the iterative adjustments, wherein for each iteration, by an agent, a control action for the iteration is determined based on a current state of a quantum system contained in the quantum hardware; by a training environment, updating the current state of the quantum system to a subsequent state of the quantum system using the determined control action and a sample control noise; and by the agent, calculating a discounted future reward using i) a universal control cost function that penalizes leakage, non-fidelity, and total gate execution time as a reinforcement learning discounted future reward function, and ii) a discounted future reward function that penalizes leakage, non-fidelity, and total gate execution time as a reinforcement learning discounted future reward function. and adjusting the values ​​of one or more control trajectory parameters for the iteration by the agent and based on the determined discounted future reward.

[0005] Other implementations of this aspect include corresponding classical or quantum computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. One or more computer systems may be configured to perform particular operations or actions by having installed thereon software, firmware, hardware, or combinations thereof that, when operated, cause the system to perform the actions. One or more computer programs may be configured to perform particular operations or actions by including instructions that, when executed by a data processing device, cause the apparatus to perform the actions.

[0006] The above and other implementations may each optionally include one or more of the following features, alone or in combination: In some implementations, the agent comprises a first neural network, and determining a control action for an iteration based on a current state of a quantum system included in the quantum hardware includes providing as an input to the first neural network a vector of parameter values ​​representing the current state of the quantum system, and obtaining as an output from the first neural network a vector of parameter values ​​representing the control action.

[0007] In some implementations, the parameter values ​​representing the control action comprise values ​​representing respective probability distributions for each component of the control action that, when sampled by the agent, define the control action.

[0008] In some implementations, adjusting the values ​​of one or more control trajectory parameters by the agent and based on the determined discounted future rewards includes adjusting a first neural network, the step including adjusting values ​​of the first neural network parameters based on the determined discounted future rewards; determining adjusted control actions for the iteration using the adjusted first neural network; and adjusting values ​​of the control trajectory parameters for the iteration using the adjusted control actions for the iteration.

[0009] In some implementations, adjusting the value of the first neural network parameter includes applying a gradient descent method with a learning rate determined by the determined discounted future reward.

[0010] In some implementations, the agent comprises a second neural network configured to determine the discounted future rewards.

[0011] In some implementations, determining the discounted future reward using i) a universal control cost function that penalizes leakage, non-fidelity, and total gate execution time as the reinforcement learning discounted future reward function, and ii) the updated state of the quantum system includes evaluating a sum of weighted universal control cost functions for future positions in the sequence of control actions based on the updated universal control cost function.

[0012] In some implementations, the step of applying the reinforcement learning model further includes training the first neural network and the second neural network at each iteration by sampling from the agent and from different possible control trajectories: i) a complete sequence of quantum states under the different control trajectories, and ii) a complete sequence of universal control cost function values ​​for the different control trajectories; determining sampled discounted future rewards corresponding to the sampled quantum states and the universal control cost function values; fitting the second neural network to the sampled discounted future rewards; and fitting the first neural network according to a gradient estimated from the sampling together with a learning rate determined by the second neural network that encodes the discounted future rewards.

[0013] In some implementations, applying the reinforcement learning model to adjust each control action in the sequence of control actions to determine a quantum control trajectory that implements the quantum gate includes performing a policy gradient method.

[0014] In some implementations, each iteration is repeated multiple times until the tuned values ​​of the agent parameters converge to within predefined limits.

[0015] In some implementations, updating a current state of the quantum system to a subsequent state of the quantum system using the determined control actions and sampled control noise by the training environment includes adjusting a vector of parameter values ​​representing the control actions based on the randomly sampled quantum hardware noise, solving a time dependent Hamiltonian evolution using the adjusted vector of parameter values ​​that achieves the current control actions, and updating the state of the quantum system using the solved time dependent Hamiltonian evolution.

[0016] In some implementations, adjusting the vector of parameter values ​​representing the control action based on the randomly sampled quantum hardware noise includes randomly sampling the control noise and adding the randomly sampled noise to each entry of the vector of parameter values.

[0017] In some implementations, randomly sampling the control noise includes sampling the amplitude variation for different control amplitudes according to a zero mean Gaussian distribution with a predetermined variance.

[0018] In some implementations, the quantum hardware comprises one or more qubits, and the control noise comprises random quantum hardware noise arising from one or more of i) qubit anharmonicity, ii) qubit detuning amplitude, iii) microwave control amplitude, and iv) two-qubit coupling pulse amplitude.

[0019] In some implementations, solving the time-dependent Hamiltonian evolution that realizes the current control action using the vector of adjusted parameter values ​​includes evaluating a Schrödinger equation using the vector of adjusted parameter values.

[0020] In some implementations, the first neural network comprises multiple fully connected neural network layers.

[0021] In some implementations, the second neural network comprises multiple fully connected neural network layers.

[0022] In some implementations, the method further includes implementing a quantum gate using the designed quantum control trajectory.

[0023] The subject matter described herein can be implemented in a particular manner so as to realize one or more of the following advantages.

[0024] Systems implementing quantum computation through reinforcement learning as described herein may improve the performance and computational efficiency of quantum computing devices, or hybrid classical-quantum computing devices. For example, quantum computing devices performing the techniques described herein may implement quantum gates with reduced errors and execution times, increased quantum gate fidelity, and improved robustness against unavoidable quantum hardware controlled noise that causes unknown variations in the quantum dynamics of the computational process.

[0025] As another example, a balance between quantum gate execution time and fidelity can be determined. In addition, by implementing high-fidelity quantum gates at reduced gate execution times, near-term quantum computing devices can be used to solve difficult problems that are beyond the power of classical computers. Furthermore, the generality of the techniques described herein provides for improved controllability of quantum devices.

[0026] A system implementing quantum computation through reinforcement learning as described herein can suppress any kind of leakage error across different frequency domains during a generic time-dependent Hamiltonian evolution and is not limited to suppressing leakage errors from a single source. Moreover, any kind of leakage error is suppressed without requiring difficult constraints on the allowable form of Hamiltonian modulation, which would undermine the generality of quantum control.

[0027] Systems implementing quantum computing through reinforcement learning as described herein are not limited to settings where complete knowledge of the physical model of the environment is available.

[0028] Systems implementing quantum computing through reinforcement learning as described herein may implement arbitrary unitary single and multi-qubit gates.

[0029] For convenience, the techniques described herein are described as implementing a single quantum gate on one or more qubits. However, the applicability of the described systems and techniques is fully scalable and can be extended to the implementation of a sequence of quantum gates, where each control used to implement a sequence of gates can be collapsed into a single control, resulting in increased gate sequence fidelity while accelerating computation time.

[0030] The details of one or more implementations of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 illustrates an exemplary system for designing and implementing quantum control trajectories. [Figure 2A] FIG. 1 is a diagram of a conventional reinforcement learning model. [Figure 2B] FIG. 1 is a diagram of a reinforcement learning model for designing quantum gate control schemes. [Diagram 3] 1 is a flow diagram of an exemplary process for designing quantum control trajectories for implementing quantum gates using quantum hardware. [Figure 4] 1 is a flow diagram of an example iteration of applying a reinforcement learning model to determine a quantum control trajectory. [Diagram 5] 13 is a flow diagram of an exemplary process for updating a current state of a quantum system using determined control actions and sample controlled noise. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Like reference numbers and designations in the various drawings indicate like elements.

[0033] In gate model quantum computing, a computational task is performed by a sequence of universal quantum gates, each of which specifies a trajectory for the evolution of a quantum computer. In general, the faster a quantum gate or sequence of quantum gates executes, the more computational power the quantum computer will possess.

[0034] A major obstacle to the realization of fast high-fidelity quantum gates is leakage error. Leakage error can be defined as the leakage of quantum information encoded in the state of a qubit from a predefined computational subspace to a non-computational subspace. There are two separate sources of leakage error: coherent leakage error and incoherent leakage error. Coherent leakage error arises from a direct coupling between the computational and non-computational subspaces of a qubit. Incoherent leakage error arises from modulation of the system Hamiltonian at a time interval shorter than allowed by the condition for adiabaticity. Existing approaches for designing control trajectories to realize fast high-fidelity quantum gates typically do not consider both sources of leakage but rather consider different types of leakage separately.

[0035] Another major obstacle to the realization of fast high-fidelity quantum gates is the unavoidable quantum hardware control noise, which causes unknown fluctuations in the quantum dynamics of the computation process. Quantum hardware control noise can be defined as noise arising from one or more of qubit anharmonicity, qubit detuning amplitude, microwave control amplitude, and / or qubit coupling pulse amplitude. Existing approaches for designing control sequences for realizing fast high-fidelity quantum gates typically do not consider such random control noise and therefore cannot be directly applied to realistic experimental settings. Instead, efforts toward improving the robustness of quantum control sequences against random noise have focused on closed-loop feedback control optimization. These approaches require frequent measurements of the quantum system, which can be expensive to realize in existing quantum computing architectures. Alternatively, existing open-loop optimization methods address control robustness through analysis of control curvature, which requires the calculation of the control Hessian, making them intractable for solving multi-qubit control problems.

[0036] This specification describes methods and systems for applying reinforcement learning techniques to design quantum gate control schemes for near-term quantum computers. To minimize leakage errors, the reinforcement learning model applies a universal quantum control cost function as a reward function that penalizes perfect leakage errors, non-fidelity, and realistic control constraints. To provide overall fidelity robustness to noise, the reinforcement learning model includes a stochastic training environment that incorporates random noise in the control amplitudes. These methods and systems may be applied to any quantum gate and multi-qubit system in general.

[0037] Example Operating Environment 1 illustrates an exemplary system 100 for designing and implementing quantum control trajectories. The exemplary system 100 is an example of a system in which the systems, components, and techniques described below may be implemented as classical or quantum computer programs on one or more classical computers or quantum computing devices at one or more locations.

[0038] System 100 includes quantum hardware 102 in data communication with a classical processor 104. System 100 is configured to use the classical processor 104 and the quantum hardware 102 to perform classical computations in combination with quantum computations.

[0039] The quantum hardware 102 includes one or more qubits 106. The qubits 106 may include qubits that can be efficiently prepared in an initial state and operated upon via the application of quantum gates. The type of physical realization of the qubits included in the quantum hardware 102 may vary. For example, in some implementations, the quantum hardware 102 may include superconducting qubits, e.g., superconducting charge qubits, superconducting flux qubits, or superconducting phase qubits. In other implementations, the quantum hardware 102 may include qubits realized by spin, e.g., electron spin, nuclear spin, or atomic spin. In general, the qubits 106 may be frequency tunable.

[0040] Quantum hardware 102 may include a set of qubit frequency control lines, e.g., where each qubit frequency control line corresponds to an individual qubit. The qubit frequency control lines control the frequencies of qubits 106, e.g., where each qubit frequency control line controls the frequency of its corresponding qubit.

[0041] The quantum hardware 102 may include one or more excitation drive lines. For convenience, one drive line, e.g., drive line 108, is shown in FIG. 1, but in some implementations the quantum hardware may include multiple drive lines, e.g., one drive line corresponding to each of the qubits 106. The one or more excitation drive lines provide excitation control of the qubits 106. The one or more excitation drive lines may be configured to run excitation pulses (also referred to herein as control pulses), e.g., control pulses 108, with different quantum gates at different frequencies. Each qubit may be tuned toward or away from these frequencies on the one or more excitation drive lines.

[0042] The quantum hardware 102 may include a set of couplers. Each coupler in the set couples a corresponding qubit to an excitation drive line. The couplers may be any type of coupler, for example, a capacitive coupler. To achieve capacitive coupling, a microwave line may be run adjacent to the qubit capacitor.

[0043] The quantum hardware 102 includes a qubit control device 110. The control device 110 includes a device configured to operate on one or more qubits 106. For example, the control device 110 may include hardware for implementing quantum logic gates, e.g., a control pulse generator 112 that generates excitation pulses to be run on one or more excitation drive lines. In some implementations, the control pulse generator 112 may be a microwave control device. In addition, the control device 110 may include hardware for performing measurements on one or more qubits 106, e.g., an oscillator.

[0044] Classical processor 104 is configured to execute a quantum control optimization procedure. Specifically, classical processor 104 is configured to design a control trajectory for a sequence of control pulses to implement each quantum gate. For example, classical processor 104 may receive data, e.g., input data 114, that specifies a particular unitary quantum gate, or a sequence of multiple unitary quantum gates. Classical processor 104 may then design a control trajectory that may be generated by qubit control device 110, e.g., control pulse generator 112, and applied to one or more of qubits 106.

[0045] The control trajectories designed by the classical processor 104 can be used to implement any unitary quantum gate with reduced leakage error, gate infidelity, and total gate execution time, while being robust to hardware control noise.

[0046] To design such control trajectories, the classical processor 104 represents the quantum gate as a sequence of control actions. The classical processor 104 includes a reinforcement learning model 118 that iteratively adjusts each control action in the sequence of control actions to implement the quantum gate and determines a quantum control trajectory that reduces the quantum gate's leakage, infidelity, and total execution time during the iterative adjustments. Data, e.g., output data 116, representing the quantum control trajectory determined by the reinforcement learning model 118 may be transmitted from the classical processor 104 to the quantum hardware 102. An exemplary reinforcement learning model is described in more detail below with reference to FIGS. 2A and 2B. An exemplary process for designing quantum control trajectories for implementing quantum gates using quantum hardware is described in more detail below with reference to FIGS. 3-5.

[0047] 2A is a diagram of an exemplary conventional reinforcement learning model 200. The exemplary conventional reinforcement learning model 200 includes an agent 202 and a training environment 204 that interacts with the agent 202. The training environment is a fully observable environment. At each step in a sequence of steps, the agent 202 receives an observation 206 and a reward 208 from the training environment 204. The agent 202 then selects an appropriate action 210 from a set of available actions to take based on the received observations 206 and rewards 208.

[0048] The agent 202 provides the selected action 210 to a training environment 204. The training environment 204 updates its state and determines a reward 212 associated with the state update. The training environment provides the determined reward 212 to the agent 202 along with a subsequent observation 214 for the next step in the sequence of steps that uses the action 210.

[0049] The exemplary conventional reinforcement learning model 200 implements reinforcement learning techniques to teach an agent 202 to take actions, e.g., action 210, with the goal of maximizing the overall reward at the end of a sequence of steps - agent 202 may receive no reward or a maximum reward at each step. To operate near optimally, agent 202 must determine the long-term consequences of its actions, even if the immediate reward for this may be negative. Agent 202 learns to take appropriate actions based on the rewards it receives - there is no supervisor.

[0050] 2B is a diagram of an exemplary reinforcement learning model 250 for designing a quantum gate control scheme. The exemplary reinforcement learning model 250 includes an agent 252 in data communication with a training environment 254. The agent 252 includes a policy neural network 253 and a value function neural network 258. The training environment 254 includes a control noise integrator 256 and a time-dependent Hamiltonian evolution solver 270. For each control action in a sequence of control actions, where the sequence of control actions represents a corresponding quantum gate, the agent 252 receives data representing a quantum state. The data may include data representing values ​​of state variables that define the state. The agent 252 processes the received data using the policy neural network 253 and the value function neural network 258.

[0051] The policy neural network 253 is a deep neural network, e.g., with one or more fully connected layers. The policy neural network 253 is configured to process received inputs representing quantum states and to generate respective outputs representing control actions of the Hamiltonian control. That is, the policy neural network 253 encodes the quantum control trajectories and captures regularities in optimal control actions under random control noise that are undetectable by traditional control optimization techniques. Such regularities facilitate more efficient optimization and often also enable transfer learning from old targets to new targets.

[0052] Agent 252 provides training environment 254 with data representing generated control actions 262 and received data representing quantum state 260. Training environment 254 provides data representing generated control actions 262 to control noise integrator 256. Control noise integrator 256 randomly samples noise and provides the randomly sampled noise to a time-dependent Hamiltonian evolution solver that generates data representing updated quantum state 260. Training environment 254 uses the data representing updated quantum state 260 to update a universal quantum control cost function. The data representing updated quantum state 260 and updated control cost function 264 are provided to agent 252 to update both value function neural network 258 and policy neural network 253.

[0053] Value function neural network 258 is configured to process the received inputs to generate respective outputs representing discounted future rewards, e.g., discounted future rewards 266. That is, value function neural network 258 encodes predicted future interactions with the stochastic environment to facilitate sampling on future trajectories to avoid overfitting of policy neural network 253 and to perform optimization on policy neural network 253.

[0054] Hardware Programming 3 is a flow diagram of an exemplary process 300 for designing quantum control trajectories for implementing quantum gates using quantum hardware. For convenience, process 300 is described as being performed by a system of one or more classical or quantum computing devices located at one or more locations. For example, system 100 of FIG. 1 , suitably programmed in accordance with this specification, may perform process 300.

[0055] The system represents a quantum gate as a sequence of control actions (step 302). A quantum gate can be a single qubit gate operating on one qubit, or a multi-qubit gate operating on multiple qubits. Representing a quantum gate as a sequence of control actions includes representing the quantum gate as a sequence of unitary transformations, where each element in the sequence of unitary transformations is determined by a respective control action. Exemplary control actions include microwave control pulses that can be applied to quantum hardware to implement the corresponding unitary transformation.

[0056] The system applies a reinforcement learning model to determine a quantum control trajectory that iteratively adjusts the control actions in the sequence of control actions to implement the quantum gate and reduces the leakage, infidelity, and total execution time of the quantum gate during the iterative adjustments (step 304). Applying the reinforcement learning model may include applying a policy gradient method. An exemplary reinforcement learning model is described above with reference to FIG. 2B. An exemplary iteration of applying the reinforcement learning model to determine a quantum control trajectory is described in more detail below with reference to FIGS. 4 and 5.

[0057] The system implements quantum gates using the designed quantum control trajectories.

[0058] 4 is a flow diagram of an example iteration 400 of applying a reinforcement learning model to determine a quantum control trajectory. For convenience, process 400 is described as being performed by a system of one or more classical or quantum computing devices located at one or more locations. For example, system 100 of FIG. 1, suitably programmed in accordance with this specification, may perform process 400.

[0059] The system determines, by a reinforcement learning agent, a control action for an iteration based on a current state of a quantum system included in quantum hardware (step 402). As described above with reference to Figures 1 and 2, in some implementations, the agent may include a policy neural network (first neural network) configured to process inputs representing the quantum state to generate outputs representing control actions that may be used to update the quantum state, as described below with reference to step 404. In these implementations, the system may determine the control action for an iteration by providing as input to the first neural network, parameter values ​​representing the current state of the quantum system, e.g., a vector of state variables, and obtaining as output from the first neural network, a vector of parameter values ​​representing the control action.

[0060] In some implementations, the output generated by the first neural network may include a vector of values ​​representing probability distributions for each component of the control action that, when sampled by the agent, defines the control action. Exemplary components of the control action include system Hamiltonian anharmonicity, mode coupling strength, detuning, or microwave pulse strength.

[0061] The system updates the current state of the quantum system to a subsequent state of the quantum system using the determined control actions and sample controlled noise by the training environment (step 404). Updating the current state of the quantum system is described in more detail below with reference to FIG.

[0062] The system uses the updated state of the quantum system to update a universal quantum control cost function used by the training environment as a reinforcement learning discounted future reward function. The universal control cost function includes a penalty term in the form of a unitary transformation determined by the control action for the iteration and depends on the state of the quantum system. Such a penalty term provides the system with improved controllability of the quantum system and implementation of quantum gates.

[0063] To constrain the total leakage error, which causes quantum information to be lost to the environment, the universal quantum control cost function includes a qubit leakage penalty term L, which represents both coherent and incoherent qubit leakage during the time-dependent Hamiltonian evolution. TSWTLB Includes.

[0064] At the beginning and end of each Hamiltonian evolution, in order to conveniently prepare and measure the qubits in the computational basis, the terms representing the time-dependent Hamiltonian couplings within the qubit computational subspace and the terms representing the control pulse couplings of the qubit computational subspace with the higher energy subspace are required to vanish at both boundaries. Such control constraints can be implemented by adding a boundary control constraint penalty term to the total cost function. For example, in the case of the gmon Hamiltonian, the system can implement a universal quantum control cost function by adding a boundary control constraint penalty term [δ(t) 2 | t=0.T +f(t) 2 | t=0.T ] may be defined as including

[0065] To reduce the total unitary quantum gate execution time T—a desirable property for near-term quantum devices—the universal quantum control cost function may further include a total execution-time penalty term.

[0066] To reduce the infidelity of the unitary transformation U(T)—the difference between the fidelity of an actual unitary transformation from its maximum value 1—the universal quantum control cost function may further include a fidelity penalty term 1-F(U(T)).

[0067] An example universal quantum cost function for the gmon Hamiltonian is given below in equation (1). C(α,β,γ,κ)=α[1-F(U(T))]+βL TSMTLB +γ[δ(t) 2 | t=0.T +f(t) 2 | t=0.T ]+κT (1)

[0068] In equation (1), 1-F(U(T)) is

[0069]

number

[0070] where U(T) denotes a unitary transformation and U target For example, represents the intended action of the unitary transformation in the absence of leakage errors or control noise.

[0071] In equation (1), L TSMTLB represents the qubit leakage penalty term,

[0072]

number

[0073] where T represents the total gate execution time, Δ represents the energy gap between the two lowest energy eigenstates,

[0074]

number

[0075] represents the block-off-diagonal components of the effective Hamiltonian for a quantum system with direct-coupling leakage errors constrained to a given order.

[0076] The leakage penalty term is formulated through the development of a generalized time-dependent Schrieffer-Wolff transformation (TSWT). The leakage bound exploits a beneficial virtual transition between the computational subspace and the undesired higher energy subspace, while placing an upper bound on both the direct-coupling (coherent) and non-adiabatic (incoherent) leakage errors caused by both on-resonant and off-resonant couplings throughout the time-dependent Hamiltonian evolution.

[0077] To derive the leakage bound, a general time-dependent Schrieffer-Wolff transform (TSWT) is formulated, whose solutions are given up to any given order, e.g., second order. The generalized TSWT provides a rotated basis supported in a higher dimensional subspace in which the direct coupling leakage error is constrained to a given order. This rotated basis coincides with the qubit basis at the beginning and end of the quantum control procedure to allow for convenient state initialization and readout. A first leakage bound corresponding to the direct coupling leakage is formulated using the generalized TSWT.

[0078] The second leakage bound is formulated through a generalization of the adiabatic theorem from pure states to the energy subspace, which allows the time-dependent Hamiltonian evolution to occur both within and between different subspaces. The generalized adiabatic theorem places an upper bound on the nonadiabatic (incoherent) leakage error in the TSWT basis during the generalized time-dependent Hamiltonian evolution.

[0079] Because the direct coupling leakage error is dominated by the off-resonance frequency component and the non-adiabatic leakage error is dominated by the on-resonance frequency component, the first leakage limit and the second leakage limit may be combined in a universal cost function leakage penalty term to provide an upper limit on all sources of leakage error caused by both off-resonance and on-resonance leakage contributions.

[0080] In equation (1), [δ(t) 2 | t=0.T +f(t) 2 | t=0.T ] represents the control constraint penalty term, δ represents the detuning, and f represents the microwave pulse intensity.

[0081] In equation (1), T represents the total gate execution time, α penalizes gate non-fidelity, and β penalizes all sources of leakage error from the leakage limit L TSMTLB , γ penalizes violation of zero-valued bound constraints, and κ penalizes gate times.

[0082] The system determines a discounted future reward by the reinforcement learning agent using i) the updated universal control cost function and ii) the updated state of the quantum system as described with reference to step 404 (step 406). Determining the discounted future reward includes evaluating a sum of weighted universal control cost functions for future positions in the sequence of control actions based on the updated universal control cost function. For example, for an n-th iteration of a sequence of N iterations, the system determines the discounted future reward by evaluating a sum of weighted universal control cost functions for positions n+1, n+2, ..., N in the sequence of control actions. The discounted future reward provides a predicted total control cost for the control actions determined by the reinforcement learning agent, i.e., a measure indicative of the fidelity of the partial sequence of unitary transformations up to the current step n, and its future predictions.

[0083] The system adjusts the values ​​of one or more control trajectory parameters for the iteration based on the determined discounted future reward by the agent (step 408). If the agent includes a policy neural network, adjusting the values ​​of the one or more control trajectory parameters for the iteration includes adjusting the policy neural network by adjusting parameters of the policy neural network based on the determined discounted future reward. This may include applying a policy gradient method. For example, this may include applying a gradient descent method with a learning rate defined by the determined discounted future reward. An adjusted control action for the iteration may then be determined by the adjusted policy neural network. The system then adjusts the control trajectory parameters based on the adjusted control action for the iteration.

[0084] The system may also adjust the value function neural network (second neural network) at each iteration, i.e., applying the reinforcement learning model may include training both the policy neural network and the value function neural network at each iteration. This may be achieved by sampling from the agent and from different possible control trajectories: i) a complete sequence of quantum states under different control trajectories, and ii) a complete sequence of universal control cost function values ​​for the different control trajectories. The system may then determine sampled discounted future rewards corresponding to the sampled quantum states and universal control cost function values. The system may then fit the second neural network to the sampled discounted future rewards, and fit the first neural network according to the gradient estimated from the sampling, together with the learning rate determined by the second neural network, which encodes the discounted future rewards.

[0085] By training a reinforcement learning agent using the determined discounted future reward, the agent can be rewarded for outputting control actions that result in quantum control trajectories, i.e., optimal quantum control trajectories, that can be used to implement quantum gates with reduced leakage, infidelity, and total execution time. By repeating process 400 multiple times, the reinforcement learning agent can improve the control actions it outputs, i.e., generate control actions that increase the discounted future reward.

[0086] In some implementations, the process 400 may be repeated until the adjusted values ​​of the agent parameters converge to within predefined limits. That is, each iteration may be repeated to adjust the enhanced agent parameters from initial values, e.g., randomly initialized values, to trained values. Determining whether the agent parameters converge to within predefined limits may be performed by the training environment based on satisfaction conditions associated with the design of the quantum trajectory, e.g., when the fidelity of the gates reaches a threshold and the bound constraints described above with reference to the universal control cost function are within a predefined accuracy.

[0087] 5 is a flow diagram of an exemplary process 500 for updating a current state of a quantum system using determined control actions and sample control noise. For convenience, process 500 is described as being performed by a system of one or more classical or quantum computing devices located at one or more locations. For example, system 100 of FIG. 1 , suitably programmed in accordance with this specification, may perform process 500.

[0088] The system calculates a vector of parameter values ​​that represent control actions based on the randomly sampled quantum hardware noise δ.

[0089]

number

[0090] (step 502). For example, the system may randomly sample the control noise by sampling the amplitude variations for different control amplitudes according to a zero-mean Gaussian distribution with a predetermined variance. The system may then add the randomly sampled control noise to each entry of the vector of parameter values.

[0091] For example, the following Hamiltonian in the rotating wave approximation

[0092]

number

[0093] In a quantum system containing two interacting gmon circuits, the system operates at some discretized time step t k For η→η+δη, g(t k ) →g(t k )+δg(t k ), δ j (t k ) → δ j (t k )+δδ j (t k ), f j (t k ) → f j (t k ) + δf j (t k ), where η represents the anharmonicity, g represents the two-mode coupling, and δ j represents the detuning, and f j represents the microwave pulse intensity.

[0094] The system uses the vector of adjusted parameter values ​​to solve the time-dependent Hamiltonian evolution that realizes the current control action (step 504). This involves using the vector of adjusted parameter values ​​to evaluate the Schrödinger equation, e.g.,

[0095]

number

[0096] where U n represents the current state of the quantum system.

[0097] The system uses the solved time-dependent Hamiltonian evolution to update the state of the quantum system (step 506). That is, the system:

[0098]

number

[0099] Then, the updated quantum state U n+1 may be provided to a value function neural network (second neural network) contained in the agent for processing, as described above with reference to FIG.

[0100] The digital and / or quantum subject matter described herein, and implementations of digital functional operations and quantum operations, including the structures disclosed herein and their structural equivalents, may be implemented in digital electronic circuitry, suitable quantum circuitry, or more generally in a quantum computing system, in tangibly embodied digital and / or quantum computer software or firmware, in digital and / or quantum computer hardware, or in a combination of one or more of these. The term "quantum computing system" may include, but is not limited to, a quantum computer, a quantum information processing system, a quantum cryptography system, or a quantum simulator.

[0101] Implementations of the digital and / or quantum subject matter described herein may be implemented as one or more digital and / or quantum computer programs, i.e., one or more modules of digital and / or quantum computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The digital and / or quantum computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory device, one or more quantum bits, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal capable of encoding digital and / or quantum information, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode digital and / or quantum information for transmission to a receiver device suitable for execution by a data processing apparatus.

[0102] The terms quantum information and quantum data refer to information or data carried by or held or stored in a quantum system, where the smallest non-trivial system is a qubit, i.e., a system that defines a unit of quantum information. It will be understood that the term "qubit" encompasses all quantum systems that can be suitably approximated as a two-level system in the corresponding context. Such quantum systems may include multi-level systems, for example, with more than two levels. By way of example, such systems may include atoms, electrons, photons, ions, or superconducting qubits. In many implementations, the computational basis state is identified using a ground state and a first excited state, but it will be understood that other setups are possible in which the computational state is identified using a higher level excited state. The term "data processing apparatus" refers to digital and / or quantum data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing digital and / or quantum data, including, by way of example, programmable digital processors, programmable quantum processors, digital computers, quantum computers, multiple digital and quantum processors or computers, and combinations thereof. The device may also be or further include special-purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a quantum simulator, i.e., a quantum data processing device designed to simulate or create information about a particular quantum system. In particular, a quantum simulator is a special-purpose quantum computer that does not have the capability to perform universal quantum computation. In addition to hardware, the device may optionally include code that creates an execution environment for digital and / or quantum computer programs, e.g., code constituting a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0103] A digital computer program, which may be referred to or described as a program, software, software application, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a digital computing environment. A quantum computer program, which may be referred to or described as a program, software, software application, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be converted to a suitable quantum programming language or written in a quantum programming language, e.g., QCL or Quipper.

[0104] A digital and / or quantum computer program may correspond to a file in a file system, but need not. A program may be stored in a portion of a file holding other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., a file storing one or more modules, subprograms, or portions of code. A digital and / or quantum computer program may be deployed to run on one digital or one quantum computer, or on multiple digital and / or quantum computers located at one site or distributed across multiple sites and interconnected by a digital and / or quantum data communication network. A quantum data communication network is understood to be a network that may transmit quantum data using quantum systems, e.g., qubits. In general, a digital data communication network cannot transmit quantum data, but a quantum data communication network may transmit both quantum data and digital data.

[0105] The processes and logic flows described herein may be performed by one or more programmable digital and / or quantum computers operating in conjunction with one or more digital and / or quantum processors executing one or more digital and / or quantum computer programs to perform functions by operating on input digital and quantum data and generating output, as appropriate. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA or ASIC, or a quantum simulator, or a combination of special purpose logic circuitry or a quantum simulator with one or more programmed digital and / or quantum computers.

[0106] One or more digital and / or quantum computer systems "configured to" perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform the operation or action. One or more digital and / or quantum computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a digital and / or quantum data processing device, cause the device to perform the operation or action. A quantum computer may receive instructions from a digital computer that, when executed by a quantum computing device, causes the device to perform an operation or action.

[0107] A digital and / or quantum computer suitable for executing a digital and / or quantum computer program may be based on a general purpose or dedicated digital and / or quantum processor, or both, or any other kind of central digital and / or quantum processing unit. In general, the central digital and / or quantum processing unit will receive instructions and digital and / or quantum data from a read-only memory, a random access memory, or a quantum system suitable for transmitting quantum data, e.g., photons, or a combination thereof.

[0108] The essential elements of a digital and / or quantum computer are a central processing unit for carrying out or executing instructions and one or more memory devices for storing instructions and digital and / or quantum data. The central processing unit and memory can be supplemented by or incorporated in a dedicated logic circuit or quantum simulator. In general, a digital and / or quantum computer will also include or be operatively coupled to one or more mass storage devices for storing digital and / or quantum data, such as magnetic disks, magneto-optical disks, optical disks, or quantum systems suitable for storing quantum information, for receiving digital and / or quantum data therefrom or transferring digital and / or quantum data thereto, or both. However, a digital and / or quantum computer need not have such devices.

[0109] Digital and / or quantum computer readable media suitable for storing digital and / or quantum computer program instructions, as well as digital and / or quantum data, include, by way of example, all forms of non-volatile digital and / or quantum memories, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, CD-ROM and DVD-ROM disks, and quantum systems, e.g., trapped atoms or electrons. It will be understood that a quantum memory is a device capable of storing quantum data with high fidelity and efficiency for long periods of time, e.g., a light-matter interface where light is used for transmission and matter is used to store and preserve the quantum characteristics of the quantum data, such as superposition or quantum coherence.

[0110] Control of the various systems described herein, or portions thereof, may be implemented in a digital and / or quantum computer program product that includes instructions stored on one or more non-transitory machine-readable storage media and executable on one or more digital and / or quantum processing devices. The systems described herein, or portions thereof, may be implemented as an apparatus, method, or system, each of which may include one or more digital and / or quantum processing devices and a memory for storing executable instructions for performing the operations described herein.

[0111] Although this specification includes many specific implementation details, these should not be interpreted as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular implementation. Some features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any suitable subcombination. Moreover, although features may be described above as acting in some combinations and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of a subcombination.

[0112] Similarly, although operations are illustrated in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence illustrated, or that all of the illustrated operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0113] Particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes illustrated in the accompanying figures do not necessarily require the particular order illustrated, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0114] 100 Exemplary System, System 102 Quantum Hardware 104 Classical Processors 106 qubits 108 Drive line, control pulse 110 Quantum bit control device, control device 112 Control Pulse Generator 114 Input Data 116 Output Data 118 Reinforcement Learning Model 200 Exemplary conventional reinforcement learning models 202, 252 Agent 204, 254 Training Environment 206, 214 Observations 208, 212 Rewards 210 Action 250 Exemplary Reinforcement Learning Models 253 Policy Neural Network (First Neural Network) 256 Controlled Noise Integrator 258 Value Function Neural Network (Second Neural Network) 260 quantum state, updated quantum state 262 Generated Control Actions 264 Updated control cost function 266 Discounted Future Rewards 270 Time-Dependent Hamiltonian Evolution Solver

Claims

1. A method for performing a gate model quantum computation comprising a sequence of quantum gates, comprising: Regarding the quantum gates included in the gate model quantum computation, implementing the quantum gate and identifying a predetermined quantum control trajectory that reduces one or more of leakage, infidelity, or total execution time of the quantum gate; and implementing the quantum gate using the predetermined quantum control trajectory; The quantum control trajectory is determined using a reinforcement learning model that iteratively adjusts each control action in a sequence of control actions representing the quantum gate, the iterative adjustment comprising, for each iteration: determining, by the agent, i) a universal control cost function that penalizes one or more of leakage, non-fidelity, or total gate execution time as a reinforcement learning discounted future reward function, and ii) a discounted future reward using the state of the quantum system for said iteration received from a training environment; adjusting, by the agent and based on the discounted future rewards, values ​​of one or more control trajectory parameters for the iteration; and providing the values ​​of one or more control trajectory parameters for the iteration to the training environment.

2. Iteratively adjusting each control action in the sequence of control actions includes, for each iteration: determining, by an agent, a control action for said iteration based on a current state of said quantum system; and updating, via a training environment, the current state of the quantum system to a subsequent state of the quantum system using the control action and sample controlled noise. The method of claim 1.

3. the agent comprises a first neural network; determining a control action for the iteration based on a current state of a quantum system contained in quantum hardware; providing as an input to the first neural network a vector of parameter values ​​representing the current state of the quantum system; obtaining, as an output from the first neural network, a vector of parameter values ​​representative of the control action. The method of claim 2.

4. The method of claim 3, wherein the first neural network has multiple fully connected neural network layers.

5. the parameter values ​​representing the control action comprise values ​​representing respective probability distributions for each component of the control action which, when sampled by the agent, define the control action; The method of claim 3.

6. adjusting, by the agent and based on the determined discounted future rewards, values ​​of one or more control trajectory parameters; tuning the first neural network, the tuning comprising adjusting values ​​of first neural network parameters based on the determined discounted future rewards; determining an adjusted control action for the iteration using the adjusted first neural network; and adjusting the values ​​of the control trajectory parameters for the iteration using the adjusted control actions for the iteration.

6. The method of claim 3, 4, or 5.

7. adjusting the values ​​of the first neural network parameters includes applying a gradient descent method with a learning rate determined by the determined discounted future rewards. The method of claim 6.

8. the agent comprises a second neural network configured to determine the discounted future rewards; 8. The method according to any one of claims 3 to 7.

9. The method of claim 8, wherein the second neural network comprises multiple fully connected neural network layers.

10. determining said discounted future rewards using i) a universal control cost function that penalizes one or more of leakage, non-fidelity, or total gate execution time as a reinforcement learning discounted future reward function; and ii) said state of said quantum system; evaluating a sum of weighted universal control cost functions for future positions in the sequence of control actions based on the updated universal control cost function; 10. The method according to any one of claims 1 to 9.

11. applying the reinforcement learning model, Sampling from said agent and from different possible control trajectories: i) a complete sequence of quantum states under the different control trajectories, and ii) a complete sequence of universal control cost function values ​​for the different control trajectories; determining sampled discounted future rewards corresponding to the sampled quantum states and universal control cost function values; fitting the second neural network to the sampled discounted future rewards; and adapting the first neural network according to the gradient estimated from the sampling together with a learning rate determined by the second neural network encoding the discounted future rewards. and training the first neural network and the second neural network at each iteration. The method of claim 8.

12. applying a reinforcement learning model to adjust each control action in the sequence of control actions to determine a quantum control trajectory that implements the quantum gate; performing a policy gradient method; 12. The method according to any one of claims 1 to 11.

13. Each iteration is repeated multiple times until the tuned values ​​of one or more agent parameters converge to within predefined limits.

13. The method according to any one of claims 1 to 12.

14. updating, by the training environment, the current state of the quantum system to a subsequent state of the quantum system using the determined control action and sample controlled noise; adjusting a vector of parameter values ​​representing the control action based on randomly sampled quantum hardware noise; using the adjusted vector of parameter values ​​to solve a time-dependent Hamiltonian evolution that implements the current control action; and updating the state of the quantum system using the solved time-dependent Hamiltonian evolution. The method of claim 2.

15. adjusting the vector of parameter values ​​representing the control action based on randomly sampled quantum hardware noise; randomly sampling the sampled control noise; and adding said randomly sampled noise to each entry of said vector of parameter values. The method of claim 14.

16. the step of randomly sampling the sample control noise comprises the step of sampling amplitude variations for different control amplitudes according to a zero-mean Gaussian distribution with a predetermined variance. The method of claim 15.

17. the quantum hardware comprises one or more qubits; the control noise comprises random quantum hardware noise arising from one or more of: i) qubit anharmonicity, ii) qubit detuning amplitude, iii) microwave control amplitude, and iv) two-qubit coupling pulse amplitude; 17. The method according to any one of claims 1 to 16.

18. using the vector of adjusted parameter values ​​to solve a time-dependent Hamiltonian evolution that implements the current control action; evaluating the Schrödinger equation using the vector of adjusted parameter values. The method of claim 14.

19. A system comprising one or more computers and one or more storage devices storing instructions, the instructions causing the one or more computers to perform operations including the method of any one of claims 1 to 18. system.

20. A computer-readable storage medium storing instructions executable by a processing device, the instructions causing the processing device to perform operations including the method of any one of claims 1 to 18. A computer-readable storage medium.

Images