Quantum reinforcement learning based method for generating key scenarios of autonomous vehicle

Abstract

The application belongs to the technical field of intelligent transportation, and particularly relates to a key scene generation method for an automatic driving vehicle based on quantum reinforcement learning. The method comprises the following steps: step one, constructing a simulation environment; step two, designing a quantum policy network for continuous control; step three, constructing a multi-dimensional sparse reward function; step four, performing interactive training and optimization on the quantum policy network; and step five, outputting a structured test case and performing visual analysis. The application models the generation of key scenes as a dynamic interaction process between an agent and an environment, efficiently explores sparse dangerous behaviors in a high-dimensional state space through a learnable quantum policy network, significantly improves the search efficiency of sparse key scenes in a high-dimensional scene space by introducing a quantum superposition state and a quantum amplitude amplification mechanism, and solves the combination explosion and sparsity problems faced by traditional methods.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically a method for generating key scenarios for autonomous vehicles based on quantum reinforcement learning. Background Technology

[0002] In the development of autonomous driving systems, test scenario generation is a crucial step in verifying their safety and robustness. Existing methods have significant limitations when dealing with high-dimensional, continuous, and sparse dangerous scenario search tasks: rule-based or random sampling strategies lack intelligent exploration capabilities, while classical deep reinforcement learning algorithms, although capable of adaptively optimizing adversarial behavior, face the curse of dimensionality in complex scenarios. Furthermore, because critical failure scenarios account for a very small percentage of the entire space, reward signals are extremely sparse, resulting in slow policy training convergence and low sample efficiency. In recent years, quantum reinforcement learning has been proposed as a potential breakthrough. Its core idea is to utilize quantum superposition and interference mechanisms to achieve more efficient policy exploration than classical methods in an exponential state space. However, applying quantum reinforcement learning to generate autonomous driving test scenarios still faces several unresolved scientific challenges: First, how to construct a quantum encoding mechanism that matches the high-dimensional continuous traffic state space, enabling heterogeneous information such as vehicle dynamics and environmental semantics to be effectively mapped to quantum states, avoiding information loss or dimensional collapse; second, how to design quantum policy gradient update rules suitable for sparse reward environments, achieving exploration efficiency superior to classical neural networks through parameterized quantum circuits, even when current noisy, medium-scale quantum devices cannot achieve complete quantum parallel search; third, how to establish an interpretable feedback loop between the quantum policy and the failure modes of the autonomous driving system, ensuring that the generated adversarial scenarios not only "trigger failures" but also reveal the causes of failures, thereby supporting iterative system improvement. Currently, no research systematically addresses these issues. Existing quantum reinforcement learning work is mostly focused on discrete, low-dimensional environments, lacking the ability to model high-dimensional continuous states, sparse reward adaptation mechanisms, and engineering-deployable architectures for real-world autonomous driving tests. Therefore, developing a quantum-inspired reinforcement learning generation framework for high-dimensional sparse hazardous scenarios is not only a key path to improve testing efficiency but also a core scientific challenge for promoting the application of quantum machine learning in safety-critical systems. Summary of the Invention

[0003] This invention provides a method for generating key scenarios for autonomous vehicles based on quantum reinforcement learning. The method models the generation of key scenarios as a dynamic interaction process between an agent and the environment. It efficiently explores sparse dangerous behaviors in a high-dimensional state space through a learnable quantum policy network. By introducing quantum superposition and quantum amplitude amplification mechanisms, it significantly improves the search efficiency for sparse key scenarios in a high-dimensional scene space, solving the combinatorial explosion and sparsity problems faced by traditional methods.

[0004] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0005] A method for generating key scenes for autonomous vehicles based on quantum reinforcement learning includes the following steps:

[0006] Step 1: Build the simulation environment;

[0007] 11) Define the scene state space;

[0008] 12) Each time a scene is generated, the speed of the main vehicle, the position and speed of the interfering vehicle, and the lane are independently and randomly initialized;

[0009] Step 2: Design a quantum policy network for continuous control;

[0010] 21) Establish alignment between the number of qubits and the state dimension;

[0011] 22) Mapping is performed using a normalized rotation angle;

[0012] 23) A two-layer parameterized quantum circuit structure is adopted, each layer sequentially including: applying an adjustable rotation operation around the Y-axis to each qubit, and constructing chain entanglement through controlled NOT gates between adjacent qubits;

[0013] 24) Design a deterministic output mechanism for continuous actions;

[0014] Step 3: Construct a multidimensional sparse reward function;

[0015] 31) Define the total reward as the sum of five factors: TTC risk, main vehicle deceleration, sudden acceleration change of interfering vehicle, collision, and entry reward;

[0016] 32) Define each of the five items separately;

[0017] Step 4: Interactively train and optimize the quantum policy network;

[0018] 41) Obtain the objective of the strategy;

[0019] 42) Update parameters using a synchronous perturbation random approximation algorithm;

[0020] 43) In each iteration, reset the environment, execute the episode, update parameters, and record key scenarios;

[0021] Step 5: Output structured test cases and perform visual analysis;

[0022] 51) Determine whether it is a critical scenario based on preset criteria, and extract the scenario fingerprint;

[0023] 52) Save key scenarios as JSON files and generate visualization analysis results such as trajectory diagrams and acceleration curves.

[0024] Furthermore, the specific method for step 11) is as follows:

[0025] Define the scene state space as ;

[0026] In the formula, The relative distance between the main vehicle and the interfering vehicle; It is the relative velocity; To interfere with lane markings;

[0027] The primary vehicle's behavior is driven by an Intelligent Driver Model (IDM), which simulates its longitudinal acceleration response to the influence of a vehicle ahead. This model dynamically calculates the acceleration commands the primary vehicle should take based on the relative distance and relative speed between the two vehicles to achieve safe following. The specific calculation formula for the IDM model is as follows:

[0028] ;

[0029] In the formula, This is the current acceleration; This is the maximum acceleration; Main vehicle speed; Free speed; The speed response index; The desired spacing; Actual following distance; expected spacing The calculation method is as follows:

[0030] ;

[0031] In the formula, This is a static safety distance; For safe head spacing; It is the relative velocity; For comfortable braking deceleration;

[0032] The specific method for step 12) is as follows:

[0033] Each time a scene is generated, the speed of the main vehicle is independently and randomly initialized to 20–30 m / s; the initial position of the interference vehicle is set to 30–50 meters in front of the main vehicle, with an initial speed of 15–30 m / s, and the vehicle randomly selects its lane, either the main lane or an adjacent lane.

[0034] Furthermore, the specific method for step 21) is as follows:

[0035] Three qubits are allocated, corresponding to the relative distance and relative speed between the main vehicle and the jamming vehicle, and the lane marking of the jamming vehicle, respectively.

[0036] The specific method for step 22) is as follows:

[0037] The relative distance between the main vehicle and the interfering vehicle Define the normalization factor The rotation angle is set to For relative velocity Define the normalization factor The rotation angle is set to For lane markings Mapped to rotation angle The state-encoded quantum state is constructed as follows:

[0038] ;

[0039] State-encoded quantum state representation is a quantum state encoding process that encodes the states of a classical scenario. Mapped to a quantum state ;left It is a composite quantum state, consisting of three subsystems, corresponding to the relative distance and relative speed between the main vehicle and the interfering vehicle, and the lane marking information of the interfering vehicle; each of the subsystems on the right... It is a rotation gate around the Y-axis, acting on a single qubit; the initial state is... That is, the ground state of a quantum bit;

[0040] pass The initial quantum state The calculation method for rotating to a superposition state is as follows:

[0041] ;

[0042] In the formula, The control parameters are obtained after normalization based on the current scene state, and their values ​​range from [0, 2π]. Their magnitude determines the quantum state. and The superposition ratio between them.

[0043] The specific method for step 23) is as follows:

[0044] Employing a two-layer RY-CNOT hardware-efficient ansatz; each layer contains: 3 single-bit parameterized rotating gates. Two CNOT entanglement gates: sequentially connecting bits 0→1 and 1→2 to form a chain-like entanglement structure; the adjustable parameter vector of the entire circuit is: ; The Y-axis rotation angle for the corresponding relative distance encoded qubit in the first layer, i.e., bit 0; The Y-axis rotation angle of the corresponding relative velocity-encoded qubit in the first layer, i.e., bit 1; The Y-axis rotation angle of the qubit corresponding to the lane identifier in the first layer, i.e., bit 2; This represents the Y-axis rotation angle corresponding to bit 0 in the second layer; This is the Y-axis rotation angle corresponding to bit 1 in the second layer; This represents the Y-axis rotation angle corresponding to bit 2 in the second layer; A differentiable mapping function from input state to desired output measurement value implemented for parameterized quantum circuits;

[0045] The specific method for step 24) is as follows:

[0046] The Pauli-Z expectation value of the quantum circuit output was measured, specifically as follows:

[0047] ;

[0048] ;

[0049] In the formula, and Acting on the first two qubits respectively; For a quantum-encoded state with input state s as classical state; This is the inverse operation of parameterized quantum circuits; For adjustable parameters, output This is then mapped to physical actions using a linear decoding function:

[0050] Longitudinal acceleration: lateral lane change intention .

[0051] Furthermore, the specific method for step 31) is as follows:

[0052] The total reward is defined as the sum of five items, specifically:

[0053] ;

[0054] In the formula, Total reward; Rewards are based on continuous risk measurement using TTC. The linear reward of the absolute value of the vehicle's deceleration; A reward is given for indicating sudden changes in the acceleration of the interfering vehicle; For collision rewards; To introduce rewards;

[0055] The specific method for step 32) is as follows:

[0056] 321) Calculate TTC when the interfering vehicle is in front of the main lane:

[0057] ;

[0058] In the formula, The longitudinal relative distance between the main vehicle and the interfering vehicle at time t; Let be the longitudinal relative velocity between the main vehicle and the interfering vehicle at time t;

[0059] like Continuous rewards will be given:

[0060] ;

[0061] 322) Obtain the deceleration from the main vehicle dynamics model. If the absolute value exceeds the human comfort threshold of 3.0 m / s², then grant a linear bonus:

[0062] ;

[0063] In the formula, Let t be the longitudinal acceleration of the main vehicle;

[0064] 323) Calculate the rate of change of longitudinal acceleration of the interfering vehicle. If it exceeds 6.0 m / s³, a fixed bonus of 2.0 will be awarded.

[0065] 324) When the relative distance is less than 1.5m and they are in the same lane, it is considered a collision, and the reward is 20;

[0066] 325) When the interfering vehicle successfully cuts into the main lane from the right lane and intends to change lanes. If the value is greater than 0.7, a cut-in bonus of 10 will be given.

[0067] Furthermore, the specific method for step 41) is as follows:

[0068] The goal of the strategy is to maximize the expected cumulative reward, specifically:

[0069] ;

[0070] In the formula, The objective function is... These are the parameters of the policy network; For a trajectory; Quantum policy represents a policy network based on variable quantum circuits. The parameters correspond to the adjustable rotation angles of quantum gates. Classical states are encoded into quantum states, and the action output is obtained by measuring the evolution through quantum circuits, thereby realizing the probabilistic mapping from state to action. In the state Next action The instant reward received; The length of the trajectory; In accordance with the strategy Distributed sampling;

[0071] The specific method for step 42) is as follows:

[0072] The SPSA algorithm is used to update the parameters, specifically as follows:

[0073] ;

[0074] In the formula, For the first The strategy parameters after the next iteration; For the first The strategy parameters at the next iteration; The learning rate, or step size, decreases with the number of iterations; gradient estimation Obtained through two perturbation samplings:

[0075] ;

[0076] In the formula, The amplitude of the disturbance; For random perturbation direction; , To set the learning rate, a typical value is used: ;

[0077] The specific method for step 43) is as follows:

[0078] In each training iteration k:

[0079] ① Create a completely new environment instance to ensure that the initial state is independent;

[0080] ② Execute the rollout policy: Use the current policy Run the complete episode and record the total reward;

[0081] ③ Update parameters: Calculate according to the SPSA formula ;

[0082] ④ Assessment and Recording: Using Generate a new trajectory, determine if it is a critical scenario, and update the fingerprint set.

[0083] The beneficial effects of this invention are as follows:

[0084] 1) This invention, for the first time, constructs a quantum reinforcement learning closed-loop framework for continuous control tasks in autonomous driving, solving the passivity and static nature of traditional scene generation methods. By modeling key scene generation as a dynamic interaction process between an agent and a high-fidelity simulation environment, and employing a variable quantum circuit with adaptive physical dimensions as the policy network, this invention can efficiently explore sparse dangerous behaviors in a high-dimensional state space. Compared to methods based on rule enumeration or random sampling, this framework possesses adaptive learning capabilities; compared to classical deep reinforcement learning, its quantum policy theoretically has more compact representation capabilities and higher sample efficiency, making it particularly suitable for testing and verification scenarios where combinatorial explosion and reward sparsity coexist.

[0085] 2) The multidimensional sparse reward function and scene fingerprint deduplication mechanism proposed in this invention can improve the diversity and engineering practicality of key scenarios. The reward function is closely aligned with the typical risk dimensions in the ISO 21448 SOTIF standard, guiding the agent to discover diverse edge cases covering the system performance boundary, rather than focusing solely on the single outcome of collision. Simultaneously, the scene fingerprint mechanism based on physical indicators achieves automatic semantic-level deduplication in a continuous space, effectively avoiding redundant consumption of test resources and providing technical support for building a high-quality, non-redundant test case library.

[0086] 3) The training and output process designed in this invention has good engineering feasibility and can be integrated into existing autonomous driving verification systems. By employing gradient-free optimization algorithms such as SPSA, this method does not require a differentiable environment and can directly interface with industrial-grade simulation platforms such as CARLA and LGSVL. The generated scenarios are output in a structured format, supplemented by visualization analysis and key scenario discovery curves, which not only support automated regression testing but also provide quantitative evidence for test adequacy assessment. This solution closely integrates cutting-edge exploration of quantum artificial intelligence with the practical needs of autonomous driving functional safety verification, providing an innovative technical path for the reliability and safety verification of high-level autonomous driving systems. Attached Figure Description

[0087] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is an overall flowchart of the present invention;

[0089] Figure 2 This is a schematic diagram of the reward curve during the training process;

[0090] Figure 3 A schematic diagram of the generation process curves for key test scenarios;

[0091] Figure 4 This is a schematic diagram showing the distribution of collision events;

[0092] Figure 5 This is a minimum distance distribution histogram;

[0093] Figure 6 This is a histogram of the maximum deceleration distribution.

[0094] Figure 7 To accumulate key scenario discovery curves;

[0095] Figure 8 To smooth the learning curve;

[0096] Figure 9 Discovery rate curve for key scenarios;

[0097] Figure 10 The collision rate (moving average) curve;

[0098] Figure 11 Scatter plot of key scenarios;

[0099] Figure 12 Discover time distribution histograms for key scenarios;

[0100] Figure 13 Box plot for key scenarios;

[0101] Figure 14 For feature correlation heatmap;

[0102] Figure 15 A schematic diagram of the longitudinal movement trajectories of two vehicles in a single key scenario;

[0103] Figure 16 A schematic diagram of the acceleration of a combat vehicle in a single key scenario;

[0104] Figure 17 This is a schematic diagram illustrating the lane-changing intent for a single key scenario. Detailed Implementation

[0105] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0106] Example 1:

[0107] See Figures 1-17This invention provides a method for generating key scenes for autonomous vehicles based on quantum reinforcement learning, comprising the following steps:

[0108] Step 1: Build the simulation environment;

[0109] Traditional scene generators treat the environment as a static snapshot, failing to reflect the true response of the autonomous driving system to disruptive behaviors. This results in generated dangerous scenarios that are not actually dangerous in the real system. Therefore, this invention constructs a dynamic simulation environment with master vehicle response capabilities, enabling the behavior of disruptive vehicles to elicit observable responses from the master vehicle, thereby providing meaningful reward signals for reinforcement learning, as detailed below:

[0110] 11) Define the scene state space as ;

[0111] In the formula, The relative distance between the main vehicle and the interfering vehicle; It is the relative velocity; To avoid interfering with lane markings; the main vehicle's behavior is driven by the IDM car-following model, which automatically adjusts acceleration based on the status of the vehicle ahead; the IDM calculation formula is as follows:

[0112] ;

[0113] In the formula, This is the current acceleration; This is the maximum acceleration; Main vehicle speed; Free speed; The speed response index; The desired spacing; This is the actual following distance; the expected spacing is calculated as follows:

[0114] ;

[0115] In the formula, This is a static safety distance; For safe head spacing; It is the relative velocity; For comfortable braking deceleration;

[0116] 12) Each time a scene is generated, the following parameters are initialized independently and randomly to ensure the diversity and unbiasedness of interactions:

[0117] Main vehicle initial speed Random sampling is performed from a uniform distribution U(20,30), with units of m / s;

[0118] Initial position of jamming vehicle Random samples are taken from the interval [-50, 30]m, with units of m;

[0119] Initial speed of the jamming vehicle Random sampling is performed from a uniform distribution U(15,30), with units of m / s;

[0120] The lane where the interfering vehicle is located Randomly selected from the set of drivable lanes, ensuring that there is a potential interaction between them and the main vehicle lane, including adjacent lanes, same lanes, and lanes one lane apart.

[0121] The above parameters are sampled independently each time a scene is generated, avoiding fixation and thus improving the generalization ability of the simulation environment.

[0122] Step 2: Design a quantum policy network for continuous control;

[0123] In autonomous driving scenario generation, the control commands of interfering vehicles are continuous variables, and the state space has explicit physical dimensions (e.g., distance is measured in meters, speed in m / s). Existing quantum reinforcement learning methods are mostly designed for discrete action spaces. Directly discretizing continuous actions can lead to action space explosion or insufficient control precision. On the other hand, if a quantum sampling strategy is adopted, it is prone to getting trapped in local optima due to exploration noise under sparse rewards. Furthermore, if the dimensional differences of the state components are not addressed, it can lead to an imbalance in the quantum gate rotation angle, weakening the policy's expressive power.

[0124] Therefore, this invention proposes a physical dimension-adaptive continuous quantum policy network, whose core design principles are: ① state encoding is aligned with physical units to avoid dimensional imbalance; ② policy output is a deterministic continuous action to avoid discrete sampling noise; ③ the circuit structure is lightweight to adapt to the coherence time and gate fidelity limitations of current noisy medium-scale quantum devices; specifically as follows:

[0125] 21) Establish alignment between the number of qubits and the state dimension;

[0126] Alignment is established between the number of qubits and the state dimension; the environmental state of this invention is a triplet. , representing the relative distance, relative speed, and lane marking of the interfering vehicle, respectively. To accurately encode this three-dimensional state, the system allocates 3 qubits, each corresponding to a state component, achieving a one-to-one mapping. This design avoids information loss caused by high-dimensional state compression, while controlling quantum resource consumption within the acceptable range of the NISQ device;

[0127] 22) Mapping is performed using a normalized rotation angle, specifically:

[0128] To address the differences in the physical range of each state component, this invention employs a normalized rotation angle mapping; for the relative distance between the main vehicle and the interfering vehicle... Define the normalization factor The rotation angle is set to For relative velocity Define the normalization factor The rotation angle is set to For lane markings Because it is a binary variable, it is directly mapped to the rotation angle. The state-encoded quantum state is constructed as follows:

[0129] ;

[0130] This formula represents a quantum state encoding process, which encodes the classical scene state. Mapped to a quantum state ;left This is a composite quantum state consisting of three subsystems (qubits), corresponding to the relative distance and speed between the host vehicle and the interfering vehicle, and the lane marking information of the interfering vehicle, respectively; each of the ones on the right... It is a rotation gate around the Y-axis, acting on a single qubit; the initial state is... That is, the ground state of a quantum bit;

[0131] pass Will The calculation method for rotating to a certain superposition state is as follows:

[0132] ;

[0133] In the formula, The control parameters are obtained after normalization based on the current scene state, and their values ​​range from [0, 2π]. Their magnitude determines the quantum state. and The overlap ratio between them;

[0134] 23) To achieve efficient policy expression with limited quantum resources, this invention employs a two-layer RY-CNOT hardware-efficient ansatz. Each layer contains: 3 single-bit parameterized rotation gates. Two CNOT entanglement gates: sequentially connecting bits 0→1 and 1→2 to form a chain-like entangled structure. Therefore, the adjustable parameter vector of the entire circuit is: .in, The Y-axis rotation angle for the corresponding relative distance encoded qubit in the first layer, i.e., bit 0; The Y-axis rotation angle of the corresponding relative velocity-encoded qubit in the first layer, i.e., bit 1; The Y-axis rotation angle of the qubit corresponding to the lane identifier in the first layer, i.e., bit 2; This represents the Y-axis rotation angle corresponding to bit 0 in the second layer; This is the Y-axis rotation angle corresponding to bit 1 in the second layer; This represents the Y-axis rotation angle corresponding to bit 2 in the second layer; A differentiable mapping function from input state to desired output measurement value for parameterized quantum circuits; this structure, while ensuring quantum entanglement capability, keeps the circuit depth within 10 quantum gates, significantly reducing the impact of decoherence and gate error;

[0135] 24) Deterministic output mechanism for continuous actions. To generate continuous control commands, this invention performs Pauli-Z expectation measurement on the quantum circuit output, as shown below:

[0136] ;

[0137] ;

[0138] In the formula, and Acting on the first two qubits respectively; For a quantum-encoded state with input state s as classical state; This is the inverse operation of parameterized quantum circuits; For adjustable parameters, output Then, it is mapped to physical actions through a linear decoding function;

[0139] Longitudinal acceleration: lateral lane change intention This mechanism avoids discrete action selection based on probability amplitude sampling, providing stable and differentiable policy outputs in sparse reward environments, thereby significantly improving learning efficiency.

[0140] Step 3, Refer to Figure 2 Construct a multidimensional sparse reward function;

[0141] In autonomous driving testing, the real risks are not limited to collisions, but also include the system operating at its performance limits. If only binary collision signals are used as rewards, the agent will only learn how to crash, ignoring more subtle but equally dangerous edge cases such as how to induce the vehicle to respond at its limits. Furthermore, collision events have extremely low probabilities in high-dimensional spaces (sparseness), making it difficult to effectively update the strategy.

[0142] Therefore, based on the definition of "Known-Unsafe Scenarios" in the ISO 21448 SOTIF standard, this invention constructs a multi-dimensional, hierarchical, and physically interpretable reward function. This function decomposes sparse collision signals into multiple intermediate risk indicators that can be propagated through gradients, thereby guiding the agent to gradually learn the complete behavioral chain from "approaching danger" to "triggering danger," as detailed below:

[0143] 31) Define the total reward as the sum of the five items;

[0144] ;

[0145] In the formula, Total reward; Rewards are based on continuous risk measurement using TTC. The linear reward of the absolute value of the vehicle's deceleration; A reward is given for indicating sudden changes in the acceleration of the interfering vehicle; For collision rewards; To introduce rewards;

[0146] 32) Define the five items separately, as follows:

[0147] 321) : TTC-based continuous risk measurement rewards;

[0148] Calculate TTC when the interfering vehicle is in front of the main lane:

[0149] ;

[0150] In the formula, The longitudinal relative distance between the main vehicle and the interfering vehicle at time t; Let be the longitudinal relative velocity between the main vehicle and the interfering vehicle at time t;

[0151] like Continuous rewards will be given:

[0152] ;

[0153] 322) Linear bonus to the absolute value of the vehicle's deceleration;

[0154] The deceleration is obtained from the vehicle's dynamics model. If the absolute value exceeds the human comfort threshold of 3.0 m / s², a linear bonus is awarded.

[0155] ;

[0156] In the formula, Let t be the longitudinal acceleration of the main vehicle;

[0157] 323) : Indication reward for sudden acceleration changes in the interfering vehicle;

[0158] Calculate the rate of change of longitudinal acceleration of the interfering vehicle. If it exceeds 6.0 m / s³, a fixed bonus of 2.0 will be awarded.

[0159] 324) Collision rewards;

[0160] When the relative distance is less than 1.5m and they are in the same lane, it is considered a collision, and the reward is 20.

[0161] 325) : Cut-in reward;

[0162] When the interfering vehicle successfully cuts into the main lane from the right lane and intends to change lanes... If the value is greater than 0.7, a cut-in bonus of 10 will be given.

[0163] Step 4: Interactively train and optimize the quantum policy network;

[0164] The core of reinforcement learning lies in the continuous optimization of the policy through interaction with the environment. However, autonomous driving simulation environments are classical programs that cannot provide gradient signals; simultaneously, the parameter updates of quantum circuits need to be adapted to the noise characteristics of NISQ devices. Existing methods that directly use gradient-based optimizers (such as Adam) will cause parameters to freeze due to the non-differentiability of the environment, preventing the policy from learning. Therefore, this invention employs a gradient-free stochastic optimization algorithm (SPSA) to estimate the policy gradient direction through multiple independent interactions with the environment, achieving efficient updates of quantum parameters without requiring differentiability of the environment, as detailed below:

[0165] 41) The goal of the strategy is to maximize the expected cumulative reward:

[0166] ;

[0167] In the formula, The objective function is... These are the parameters of the policy network; For a trajectory; Quantum policy represents a policy network based on variable quantum circuits. The parameters correspond to the adjustable rotation angles of quantum gates. Classical states are encoded into quantum states, and the action output is obtained by measuring the evolution through quantum circuits, thereby realizing the probabilistic mapping from state to action. In the state Next action The instant reward received; The length of the trajectory; In accordance with the strategy Distributed sampling;

[0168] 42) The parameters are updated using the Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm, specifically as follows:

[0169] ;

[0170] In the formula, For the first The strategy parameters after the next iteration; For the first The strategy parameters at the next iteration; The learning rate, or step size, decreases with the number of iterations; gradient estimation Obtained through two perturbation samplings:

[0171] ;

[0172] In the formula, The amplitude of the disturbance; For random perturbation direction; , To set the learning rate, a typical value is used: ;

[0173] 43) Training the interaction protocol;

[0174] In each training iteration k:

[0175] ① Create a completely new environment instance to ensure that the initial state is independent;

[0176] ② Execute the rollout policy: Use the current policy Run the complete episode and record the total reward;

[0177] ③ Update parameters: Calculate according to the SPSA formula ;

[0178] ④ Assessment and Recording: Using Generate a new trajectory, determine if it is a critical scenario, and update the fingerprint set.

[0179] Step 5, Refer to Figure 3 Output structured test cases and perform visual analysis;

[0180] If the generated scenarios cannot be directly used by the testing platform or lack interpretability, they will be difficult to integrate into the actual verification process. Therefore, this invention provides structured output and visualization assistance, as follows:

[0181] First, the initial state of the scene, action sequence, and fingerprint are saved to a JSON file, compatible with platforms such as CARLA; trajectory diagrams, acceleration curves, and lane change intention timing diagrams are automatically generated; and a "key scene discovery quantity - training rounds" curve is output to assist in testing decisions.

[0182] Example 2:

[0183] See Figures 1-17The objective of this embodiment is to train a quantum agent to control interfering vehicles in a highway following scenario, and automatically generate key test scenarios that can expose the defects of the master vehicle's autonomous driving system.

[0184] Simulation environment: The main vehicle adopts a simplified IDM car-following model with a maximum speed of 25 m / s; the interference vehicle is initially positioned 30–50 meters behind the main vehicle with an initial speed of 15–30 m / s and an initial lane that is random (main lane or right lane); the simulation step size is 0.1 seconds, and the maximum duration of a single episode is 6 seconds.

[0185] Quantum policy network: Number of qubits: 3; The variational circuit structure adopts two layers of parameterized quantum circuits consisting of rotating gates around the Y-axis and controlled NOT gates, containing a total of 6 adjustable parameters; Action output: longitudinal acceleration alon∈[−5,5]m / s2, lateral lane change intention alat∈[0,1].

[0186] Training configuration: The Adam optimizer (learning rate 0.08) was used to update the policy parameters; a total of 120 training rounds were conducted; the key scenario judgment criteria were that any of the following conditions were met: minimum relative distance less than 5.0 meters, maximum deceleration of the main vehicle exceeding 3.0 m / s², or a collision occurred. The entire training process was completed on a regular laptop (Intel i7 CPU, 16 GB RAM), without relying on quantum hardware.

[0187] Conclusion: By introducing a parameterized quantum policy network, this invention significantly improves the discovery efficiency and diversity of key scenarios in a high-dimensional sparse reward environment, effectively alleviating the bottlenecks of traditional methods in terms of combinatorial explosion and sample waste, and verifying its practical value in autonomous driving safety testing.

[0188] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating key scenes for autonomous vehicles based on quantum reinforcement learning, characterized in that, Includes the following steps: Step 1: Build the simulation environment; 11) Define the scene state space; 12) Each time a scene is generated, the speed of the main vehicle, the position and speed of the interfering vehicle, and the lane are independently and randomly initialized; Step 2: Design a quantum policy network for continuous control; 21) Establish alignment between the number of qubits and the state dimension; 22) Mapping is performed using a normalized rotation angle; 23) A two-layer parameterized quantum circuit structure is adopted, each layer sequentially including: applying an adjustable rotation operation around the Y-axis to each qubit, and constructing chain entanglement through controlled NOT gates between adjacent qubits; 24) Design a deterministic output mechanism for continuous actions; Step 3: Construct a multidimensional sparse reward function; 31) Define the total reward as the sum of five factors: TTC risk, main vehicle deceleration, sudden acceleration change of interfering vehicle, collision, and entry reward; 32) Define each of the five items separately; Step 4: Interactively train and optimize the quantum policy network; 41) Obtain the objective of the strategy; 42) Update parameters using a synchronous perturbation random approximation algorithm; 43) In each iteration, reset the environment, execute the episode, update parameters, and record key scenarios; Step 5: Output structured test cases and perform visual analysis; 51) Determine whether it is a critical scenario based on preset criteria, and extract the scenario fingerprint; 52) Save key scenarios as JSON files and generate visualization analysis results such as trajectory diagrams and acceleration curves.

2. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 11) is as follows: Define the scene state space as ; In the formula, The relative distance between the main vehicle and the interfering vehicle; It is the relative velocity; To interfere with lane markings; The driver's behavior is driven by an intelligent driver model (IDM) to simulate longitudinal acceleration response under the influence of a vehicle ahead. The model dynamically calculates the acceleration commands the driver should take based on the relative distance and speed between the driver and the vehicle ahead to achieve safe following. The specific calculation formula for the IDM model is as follows: ； In the formula, This is the current acceleration; This is the maximum acceleration; Main vehicle speed; Free speed; The speed response index; The desired spacing; This is the actual following distance; the expected spacing is calculated as follows: ； In the formula, This is a static safety distance; For safe head spacing; It is the relative velocity; For comfortable braking deceleration; The specific method for step 12) is as follows: Each time a scene is generated, the speed of the main vehicle is independently and randomly initialized to 20–30 m / s; the initial position of the interference vehicle is set to 30–50 meters in front of the main vehicle, with an initial speed of 15–30 m / s, and the vehicle randomly selects its lane, either the main lane or an adjacent lane.

3. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 21) is as follows: Three qubits are allocated, corresponding to the relative distance and relative speed between the main vehicle and the jamming vehicle, and the lane marking of the jamming vehicle, respectively. The specific method for step 22) is as follows: The relative distance between the main vehicle and the interfering vehicle Define the normalization factor The rotation angle is set to For relative velocity Define the normalization factor The rotation angle is set to For lane markings Mapped to rotation angle The state-encoded quantum state is constructed as follows: ； State-encoded quantum state representation is a quantum state encoding process that encodes the states of a classical scenario. Mapped to a quantum state ;left It is a composite quantum state, consisting of three subsystems, corresponding to the relative distance and relative speed between the main vehicle and the interfering vehicle, and the lane marking information of the interfering vehicle; each of the subsystems on the right... It is a rotation gate around the Y-axis, acting on a single qubit; the initial state is... That is, the ground state of a quantum bit; pass The initial quantum state The calculation method for rotating to a superposition state is as follows: ； In the formula, The control parameters are obtained after normalization based on the current scene state, with values ​​ranging from [0, 2π]. Their magnitude determines the quantum state. and The overlap ratio between them; The specific method for step 23) is as follows: Employing a two-layer RY-CNOT hardware for high-efficiency ansatz; Each layer contains: 3 single-bit parameterized rotating gates Two CNOT entanglement gates: sequentially connecting bits 0→1 and 1→2 to form a chain-like entanglement structure; the adjustable parameter vector of the entire circuit is: ;in, The Y-axis rotation angle for the corresponding relative distance encoded qubit in the first layer, i.e., bit 0; The Y-axis rotation angle of the corresponding relative velocity-encoded qubit in the first layer, i.e., bit 1; The Y-axis rotation angle of the qubit corresponding to the lane identifier in the first layer, i.e., bit 2; This represents the Y-axis rotation angle corresponding to bit 0 in the second layer; This is the Y-axis rotation angle corresponding to bit 1 in the second layer; This represents the Y-axis rotation angle corresponding to bit 2 in the second layer; A differentiable mapping function from input state to desired output measurement value implemented for parameterized quantum circuits; The specific method for step 24) is as follows: The Pauli-Z expectation value of the quantum circuit output was measured, specifically as follows: ； ； In the formula, and Acting on the first two qubits respectively; For a quantum-encoded state with input state s as classical state; This is the inverse operation of parameterized quantum circuits; For adjustable parameters, output This is then mapped to physical actions using a linear decoding function: Longitudinal acceleration: lateral lane change intention .

4. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 31) is as follows: The total reward is defined as the sum of five items, specifically: ； In the formula, Total reward; Rewards are based on continuous risk measurement using TTC. The linear reward of the absolute value of the vehicle's deceleration; A reward is given for indicating sudden changes in the acceleration of the interfering vehicle; For collision rewards; To introduce rewards; The specific method for step 32) is as follows: 321) Calculate TTC when the interfering vehicle is in front of the main lane: ； In the formula, The longitudinal relative distance between the main vehicle and the interfering vehicle at time t; Let be the longitudinal relative velocity between the main vehicle and the interfering vehicle at time t; like Continuous rewards will be given: ； 322) Obtain the deceleration from the main vehicle dynamics model. If the absolute value exceeds the human comfort threshold of 3.0 m / s², then grant a linear bonus: ； In the formula, Let t be the longitudinal acceleration of the main vehicle; 323) Calculate the rate of change of longitudinal acceleration of the interfering vehicle. If it exceeds 6.0 m / s³, a fixed bonus of 2.0 will be awarded. 324) When the relative distance is less than 1.5m and they are in the same lane, it is considered a collision, and the reward is 20; 325) When the interfering vehicle successfully cuts into the main lane from the right lane and intends to change lanes. If the value is greater than 0.7, a cut-in bonus of 10 will be given.

5. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 41) is as follows: The goal of the strategy is to maximize the expected cumulative reward, specifically: ； In the formula, The objective function is... These are the parameters of the policy network; For a trajectory; Quantum policy represents a policy network based on variable quantum circuits. The parameters correspond to the adjustable rotation angles of quantum gates. Classical states are encoded into quantum states, and the action output is obtained by measuring the evolution through quantum circuits, thereby realizing the probabilistic mapping from state to action. In the state Next action The instant reward received; The length of the trajectory; In accordance with the strategy Distributed sampling; The specific method for step 42) is as follows: The SPSA algorithm is used to update the parameters, specifically as follows: ； In the formula, For the first The strategy parameters after the next iteration; For the first The strategy parameters at the next iteration; The learning rate, or step size, decreases with the number of iterations; gradient estimation Obtained through two perturbation samplings: ； In the formula, The amplitude of the disturbance; For random perturbation direction; , To set the learning rate, a typical value is used: ; The specific method for step 43) is as follows: In each training iteration k: ① Create a completely new environment instance to ensure that the initial state is independent; ② Execute the rollout policy: Use the current policy Run the complete episode and record the total reward; ③ Update parameters: Calculate according to the SPSA formula ; ④ Assessment and Recording: Using Generate a new trajectory, determine if it is a critical scenario, and update the fingerprint set.

Images