Automatic driving automobile key scene generation method based on quantum reinforcement learning

Abstract

The invention belongs to the technical field of intelligent transportation, and particularly relates to an automatic driving automobile key scene generation method based on quantum reinforcement learning. Comprising the following steps: step 1, constructing a simulation environment; 2, designing a continuous control-oriented quantum strategy network; 3, constructing a multi-dimensional sparse reward function; 4, performing interactive training and optimization on the quantum strategy network; step 5, outputting a structured test case and performing visual analysis; according to the method, a key scene is generated and modeled as a dynamic interaction process of an intelligent agent and an environment, sparse dangerous behaviors are efficiently explored in a high-dimensional state space through a learnable quantum strategy network, the search efficiency of the sparse key scene in the high-dimensional scene space is remarkably improved by introducing a quantum superposition state and a quantum amplitude amplification mechanism, and the search efficiency of the sparse key scene in the high-dimensional scene space is improved. And the problems of combination explosion and sparsity faced by the traditional method are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent transportation, and in particular to a key scene generation method for an autonomous vehicle based on quantum reinforcement learning. BACKGROUND

[0002] In the development of an autonomous driving system, test scene generation is a key link for verifying the safety and robustness thereof. Existing methods have significant limitations in dealing with high-dimensional, continuous, and sparse dangerous scene search tasks. The strategies based on rules or random sampling lack intelligent exploration capabilities, while classical deep reinforcement learning algorithms can adaptively optimize adversarial behaviors but face dimension disaster in complex scenes. Moreover, because key failure scenes account for a very low proportion in the entire space, the reward signal is extremely sparse, which leads to slow convergence of strategy training and low sample efficiency. In recent years, quantum reinforcement learning has been proposed as a potential breakthrough. The core idea is to use quantum superposition and interference mechanisms to achieve more efficient strategy exploration in an exponential state space than classical methods. However, applying quantum reinforcement learning to autonomous driving test scene generation still faces some unresolved scientific problems. First, how to construct a quantum coding mechanism that matches the high-dimensional continuous traffic state space so that heterogeneous information such as vehicle dynamics and environmental semantics can be effectively mapped to quantum states, avoiding information loss or dimension collapse. Second, how to design a quantum policy gradient update rule suitable for a sparse reward environment so that, under the premise that current noisy medium-scale quantum devices cannot implement complete quantum parallel search, the exploration efficiency of a parameterized quantum circuit can still be better than that of a classical neural network. Third, how to establish an interpretable feedback loop between the quantum policy and the failure mode of the autonomous driving system so that the generated adversarial scenes not only trigger failures but also reveal the causes of the failures, thereby supporting system iteration and improvement. Currently, there is no research that systematically solves the above problems. Existing quantum reinforcement learning works mostly focus on discrete and low-dimensional environments and lack high-dimensional continuous state modeling capabilities, sparse reward adaptation mechanisms, and engineering deployable architectures for real autonomous driving tests. Therefore, developing a quantum heuristic reinforcement learning generation framework for high-dimensional sparse dangerous scenes is not only a key path to improving test efficiency but also a core scientific challenge in promoting quantum machine learning in safety-critical systems. SUMMARY

[0003] The application provides a key scene generation method for an autonomous vehicle based on quantum reinforcement learning. The key scene generation is modeled as a dynamic interaction process between an agent and an environment, and a learnable quantum policy network is used to efficiently explore sparse dangerous behaviors in a high-dimensional state space. By introducing quantum superposition states and quantum amplitude amplification mechanisms, the search efficiency for sparse key scenes in a high-dimensional scene space is significantly improved, solving the combinatorial explosion and sparsity problems faced by traditional methods.

[0004] The technical scheme of the present application is explained in combination with the drawings as follows:

[0005] A key scene generation method for an autonomous vehicle based on quantum reinforcement learning, comprising the following steps:

[0006] Step one, build a simulation environment;

[0007] 11) Define the scene state space;

[0008] 12) Independently and randomly initialize the host vehicle speed, disturbance vehicle position, speed and lane at each scene generation;

[0009] Step two, design a quantum policy network for continuous control;

[0010] 21) Align the number of qubits with the state dimension;

[0011] 22) Use normalized rotation angles for mapping;

[0012] 23) Use a two-layer parameterized quantum circuit structure, each layer containing: applying adjustable rotation operation around Y axis to each qubit, and building chain entanglement through controlled non-gate between adjacent qubits;

[0013] 24) Design a deterministic output mechanism for continuous action;

[0014] Step three, build a multi-dimensional sparse reward function;

[0015] 31) Define the total reward as the sum of five items: TTC risk, host vehicle deceleration, disturbance vehicle acceleration mutation, collision, and cut-in reward;

[0016] 32) Define the five items respectively;

[0017] Step four, interactive training and optimization of the quantum policy network;

[0018] 41) Obtain the target of the policy;

[0019] 42) Update the parameters using the synchronous disturbance stochastic approximation algorithm;

[0020] 43) Reset the environment, execute the episode, update the parameters and record the key scene in each iteration;

[0021] Step five, output structured test cases and perform visual analysis;

[0022] 51) Determine whether it is a key scene according to the preset standard, and extract the scene fingerprint;

[0023] 52) Save the key scene as a JSON file, and generate trajectory graph, acceleration curve and other visual analysis results.

[0024] Further, the specific method of step 11) is as follows:

[0025] The scene state space is defined as ;

[0026] In the formula, is the relative distance between the host vehicle and the interfering vehicle; is the relative speed; is the lane identifier of the interfering vehicle;

[0027] The host vehicle behavior is driven by the Intelligent Driver Model (IDM), which is used to simulate its longitudinal acceleration response under the influence of the front vehicle. According to the relative distance and relative speed between the host vehicle and the front vehicle, the IDM model dynamically calculates the acceleration command that the host vehicle should take to achieve safe following. The specific calculation formula of the IDM model is as follows:

[0028] ;

[0029] In the formula, is the current acceleration; is the maximum acceleration; is the host vehicle speed; is the free running speed; is the speed response index; is the desired spacing; is the actual following distance; the desired spacing The calculation method is as follows:

[0030] ;

[0031] In the formula, is the static safety distance; is the safety time headway; is the relative speed; is the comfortable braking deceleration;

[0032] The specific method of step 12) is as follows:

[0033] Each time the scene is generated, the host vehicle speed is randomly initialized to 20-30 m / s; the initial position of the interfering vehicle is set to 30-50 meters in front of the host vehicle, the initial speed is 15-30 m / s, and the lane is randomly selected, i.e. the host lane or the adjacent lane.

[0034] Further, the specific method of step 21) is as follows:

[0035] Three qubits are allocated, corresponding to the relative distance between the host vehicle and the interfering vehicle, the relative speed, and the lane identifier of the interfering vehicle;

[0036] The specific method of the step 22) is as follows:

[0037] For the relative distance of the host vehicle and the interference vehicle , a normalization factor is defined as , and the rotation angle is set as ; for the relative speed , a normalization factor is defined as , and the rotation angle is set as ; for the lane mark , the mapping is a rotation angle ; the state encoding quantum state is constructed as:

[0038] ;

[0039] The state encoding quantum state represents a quantum state encoding process, which maps the classical scene state to a quantum state ; the left side is a composite quantum state composed of three subsystems corresponding to the relative distance of the host vehicle and the interference vehicle, the relative speed, and the lane mark information of the interference vehicle; each in the right side is a rotation gate around the Y axis acting on a single quantum bit; the initial state is , that is, the ground state of the quantum bit;

[0040] The initial quantum state is rotated to a superposition state by , and the calculation method is as follows:

[0041] ;

[0042] In the formula, is the control parameter obtained after normalization according to the current scene state, and the value range is [0, 2π], and the size determines the superposition ratio of the quantum state between and .

[0043] The specific method of the step 23) is as follows:

[0044] A two-layer RY-CNOT hardware efficient ansatz is adopted; each layer includes: 3 single-bit parameterized rotation gates ; 2 CNOT entanglement gates: connecting bits 0→1 and 1→2 in turn to form a chain entanglement structure; the adjustable parameter vector of the entire circuit is: ; is the Y-axis rotation angle of the corresponding relative distance encoding quantum bit, that is, bit 0, in the first layer; Y-axis rotation angle of the quantum bit 1 corresponding to the relative speed in the first layer; Y-axis rotation angle of the quantum bit 2 corresponding to the lane mark in the first layer; Y-axis rotation angle of the quantum bit 0 in the second layer; Y-axis rotation angle of the quantum bit 1 in the second layer; Y-axis rotation angle of the quantum bit 2 in the second layer; The differentiable mapping function from the input state to the output measurement expectation value realized by the parameterized quantum circuit;

[0045] The specific method of the step 24) is as follows:

[0046] The Pauli-Z expectation value measurement is performed on the quantum circuit output, specifically:

[0047] ;

[0048] ;

[0049] wherein, and act on the first two quantum bits, respectively; is the quantum encoding state with the input state being the classical state s; is the inverse operation of the parameterized quantum circuit; is the adjustable parameter, and the output is then mapped into the physical action through a linear decoding function:

[0050] Longitudinal acceleration: Lateral lane-changing intention .

[0051] Further, the specific method of the step 31) is as follows:

[0052] The total reward is defined as the sum of the five items, specifically:

[0053] ;

[0054] wherein, is the total reward; is the TTC-based continuous risk measurement reward; is the linear reward of the absolute value of the host vehicle deceleration; is the indication reward of the acceleration mutation of the interference vehicle; is the collision reward; is the cut-in reward;

[0055] The specific method of the step 32) is as follows:

[0056] 321) When the interfering vehicle is in front of the host vehicle, calculate TTC:

[0057] ;

[0058] where, is the longitudinal relative distance between the host vehicle and the interfering vehicle at time t; is the longitudinal relative speed between the host vehicle and the interfering vehicle at time t;

[0059] If , give continuous reward:

[0060] ;

[0061] 322) Get the deceleration from the host vehicle dynamics model, if the absolute value exceeds the human comfort threshold of 3.0 m / s², give a linear reward:

[0062] ;

[0063] where, is the longitudinal acceleration of the host vehicle at time t;

[0064] 323) Calculate the rate of change of the longitudinal acceleration of the interfering vehicle, if it exceeds 6.0 m / s³, give a fixed reward of 2.0;

[0065] 324) When the relative distance is less than 1.5 m and in the same lane, it is considered as a collision, the reward is 20;

[0066] 325) When the interfering vehicle successfully cuts into the host lane from the right lane and the lane change intention is greater than 0.7, give a cut-in reward of 10.

[0067] Further, the specific method of step 41) is as follows:

[0068] The goal of the strategy is to maximize the expected cumulative reward, specifically:

[0069] ;

[0070] where, is the objective function; is the parameter of the policy network; is a trajectory; is a quantum policy, representing a policy network implemented based on a variational quantum circuit, the parameters correspond to the adjustable rotation angles of the quantum gates, the classical state is encoded into a quantum state, and after evolution through the quantum circuit, the action output is measured, thereby realizing the probability mapping from state to action; is the state ; is the actionThe instant reward received; The length of the trajectory; In accordance with the strategy Distributed sampling;

[0071] The specific method for step 42) is as follows:

[0072] The SPSA algorithm is used to update the parameters, specifically as follows:

[0073] ;

[0074] In the formula, For the first The strategy parameters after the next iteration; For the first Strategy parameters at the next iteration; The learning rate, or step size, decreases with the number of iterations; gradient estimation Obtained through two perturbation samplings:

[0075] ;

[0076] In the formula, The amplitude of the disturbance; For random perturbation direction; , To set the learning rate, a typical value is used: ;

[0077] The specific method for step 43) is as follows:

[0078] In each training iteration k:

[0079] ① Create a completely new environment instance to ensure that the initial state is independent;

[0080] ② Execute the rollout policy: Use the current policy Run the complete episode and record the total reward;

[0081] ③ Update parameters: Calculate according to the SPSA formula ;

[0082] ④ Assessment and Recording: Using Generate a new trajectory, determine if it is a critical scenario, and update the fingerprint set.

[0083] The beneficial effects of this invention are as follows:

[0084] 1) The application first constructs a quantum reinforcement learning closed-loop framework for automatic driving continuous control tasks, solving the passivity and staticity problems of traditional scene generation methods. By modeling the generation of key scenes as a dynamic interaction process between the agent and the high-fidelity simulation environment, and using a physical dimension-adaptive variational quantum circuit as the policy network, the application can efficiently explore sparse dangerous behaviors in high-dimensional state space. Compared with methods based on rule enumeration or random sampling, this framework has adaptive learning ability; compared with classical deep reinforcement learning, the quantum policy has a more compact representation and higher sample efficiency in theory, especially suitable for test verification scenarios with combinatorial explosion and sparse rewards;

[0085] 2) The multi-dimensional sparse reward function and scene fingerprint deduplication mechanism proposed in the application can improve the diversity and engineering practicability of key scenes. The reward function closely aligns with the typical risk dimensions in ISO 21448 SOTIF standard, guiding the agent to discover diverse edge cases covering the system performance boundary, rather than focusing on only one single result of collision. At the same time, the scene fingerprint mechanism based on physical indicators realizes automatic deduplication at the semantic level in continuous space, effectively avoiding redundant consumption of test resources, providing technical support for building a high-quality, non-redundant test case library;

[0086] 3) The training and output process designed in the application has good engineering landing performance and can be integrated into existing automatic driving verification systems. By using SPSA and other gradient-free optimization algorithms, this method does not require environment differentiability and can be directly connected to industrial-level simulation platforms such as CARLA and LGSVL; the generated scenes are output in a structured format, supplemented by visual analysis and key scene discovery quantity curves, supporting not only automated regression testing but also providing quantitative basis for test adequacy evaluation. This solution closely combines the cutting-edge exploration of quantum artificial intelligence with the actual needs of automatic driving functional safety verification, providing an innovative technical path for the reliability and safety verification of high-order automatic driving systems. BRIEF DESCRIPTION OF DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0088] Figure 1 is the overall flowchart of the application;

[0089] Figure 2 is the reward curve diagram of the training process;

[0090] Figure 3 Process curve for critical test scenario generation;

[0091] Figure 4 Distribution plot for collision events;

[0092] Figure 5 Histogram for minimum distance distribution;

[0093] Figure 6 Histogram for maximum deceleration distribution;

[0094] Figure 7 Cumulative critical scenario discovery curve;

[0095] Figure 8 Smoothed learning curve;

[0096] Figure 9 Critical scenario discovery rate curve;

[0097] Figure 10 Collision rate (moving average) curve;

[0098] Figure 11 Critical scenario scatter plot;

[0099] Figure 12 Critical scenario discovery time distribution histogram;

[0100] Figure 13 Critical scenario box plot;

[0101] Figure 14 Feature correlation heat map;

[0102] Figure 15 Single critical scenario two vehicle longitudinal motion trajectory plot;

[0103] Figure 16 Single critical scenario opposing vehicle acceleration plot;

[0104] Figure 17 Single critical scenario lane change intent plot. DETAILED DESCRIPTION

[0105] The application will be further described below in conjunction with the drawings and examples. It can be understood that the specific examples described herein are only used to explain the application, but not to limit the application. In addition, it should be noted that, for ease of description, only the parts related to the application are shown in the drawings, not all structures.

[0106] Example One:

[0107] Reference Figures 1-17The application provides a kind of automatic driving car key scene generation method based on quantum reinforcement learning, comprising the following steps:

[0108] Step one, build simulation environment;

[0109] Traditional scene generator regards environment as static snapshot, cannot reflect the real response of automatic driving system to interference behavior, lead to the dangerous scene generated in actual system is not dangerous. Therefore, the dynamic simulation environment with the response ability of host vehicle constructed by the application makes the behavior of interference vehicle can cause the observable reaction of host vehicle, thereby providing meaningful reward signal for reinforcement learning, as follows:

[0110] 11) define scene state space as ;

[0111] In the formula, is the relative distance between host vehicle and interference vehicle; is relative speed; is lane mark of interference vehicle;Host vehicle behavior is driven by IDM follow-up model, and acceleration is automatically adjusted according to the state of front vehicle;IDM calculation formula is as follows:

[0112] ;

[0113] In the formula, is current acceleration; is maximum acceleration; is host vehicle speed; is free running speed; is speed response index; is expected distance; is actual follow-up distance;Expected distance calculation method is as follows:

[0114] ;

[0115] In the formula, is static safety distance; is safety time headway; is relative speed; is comfortable braking deceleration;

[0116] 12) the following parameters are independently randomly initialized each time scene generation is generated, to ensure the diversity and unbiasedness of interaction:

[0117] Host vehicle initial speed , randomly sampled from uniform distribution U(20,30), unit: m / s;

[0118] Interference vehicle initial position , randomly sampled from interval [-50, 30]m, unit: m;

[0119] Initial speed of interference vehicle , randomly sampled from a uniform distribution U(15, 30) with units of m / s;

[0120] Lane where the interference vehicle is located , randomly selected from the set of drivable lanes, ensuring potential interaction with the host vehicle lane, including adjacent lanes, the same lane, and one lane apart.

[0121] The above parameters are independently sampled each time the scene is generated to avoid fixation and improve the generalization ability of the simulation environment.

[0122] Step 2, design a quantum policy network for continuous control;

[0123] In the automatic driving scene generation, the control instructions of the interference vehicle are continuous variables, and the state space has a clear physical dimension (such as distance in meters and speed in m / s). Existing quantum reinforcement learning methods are mostly designed for discrete action spaces. If the continuous action is discretized directly, it will lead to action space explosion or insufficient control precision; if a quantum sampling strategy is used, it is easy to fall into a local optimum due to exploration noise under sparse rewards. In addition, if the dimension difference of the state components is not handled, it will lead to unbalanced quantum gate rotation angles, weakening the policy expression ability.

[0124] Therefore, the present application proposes a continuous quantum policy network with physical dimension self-adaptation, whose core design principles are: ① state encoding is aligned with physical units to avoid dimension imbalance; ② policy output is a deterministic continuous action to avoid discrete sampling noise; ③ circuit structure is lightweight to adapt to the coherence time and gate fidelity limitations of current noisy intermediate-scale quantum (NISQ) devices; the specific design is as follows:

[0125] 21) Establish the alignment of the number of qubits and the state dimension;

[0126] Establish the alignment of the number of qubits and the state dimension; the environment state of the present application is a three-tuple , representing the relative distance between the host vehicle and the interference vehicle, the relative speed, and the lane identifier of the interference vehicle. To accurately encode the three-dimensional state, the system allocates 3 qubits, each corresponding to a state component, achieving one-to-one mapping. This design avoids information loss caused by high-dimensional state compression, while controlling quantum resource consumption within the range that NISQ devices can bear;

[0127] 22) Use normalized rotation angles for mapping, specifically:

[0128] For the physical range differences of each state component, the present application uses normalized rotation angle mapping; for the relative distance between the host vehicle and the interference vehicle , the normalized factor , the rotation angle is set as ; for the relative speed , the normalization factor is defined as , the rotation angle is set as ; for the lane mark , since it is a binary variable, it is directly mapped to the rotation angle ; the state encoding quantum state is constructed as:

[0129] ;

[0130] The formula represents a quantum state encoding process, which maps the classical scene state to a quantum state ; the left side is a composite quantum state composed of three subsystems (qubits), corresponding to the relative distance between the main vehicle and the interfering vehicle, the relative speed, and the lane mark information of the interfering vehicle; each on the right side is a rotation gate around the Y axis acting on a single quantum bit; the initial state is , that is, the ground state of the quantum bit;

[0131] By , the is rotated to a certain superposition state, and the calculation method is as follows:

[0132] ;

[0133] In the formula, is the control parameter obtained after normalization according to the current scene state, and the value range is [0, 2π], and the size determines the superposition ratio of the quantum state between and ;

[0134] 23) To realize effective strategy expression under limited quantum resources, the present application adopts a two-layer RY-CNOT hardware efficient ansatz. Each layer contains: 3 single-bit parameterized rotation gates ; 2 CNOT entanglement gates: connecting bits 0→1 and 1→2 in turn, forming a chain entanglement structure. Thus, the adjustable parameter vector of the entire circuit is: . Among them, is the Y-axis rotation angle of the relative distance encoding quantum bit, that is, bit 0, in the first layer; is the Y-axis rotation angle of the relative speed encoding quantum bit, that is, bit 1, in the first layer; is the Y-axis rotation angle of the lane mark encoding quantum bit, that is, bit 2, in the first layer; is the Y-axis rotation angle of bit 0 in the second layer; Y-axis rotation angle for bit 1 in the second layer; Y-axis rotation angle for bit 2 in the second layer; A differentiable mapping function from input state to output measurement expectation value realized by the parameterized quantum circuit; this structure controls the circuit depth within 10 quantum gates while ensuring quantum entanglement capability, significantly reducing the influence of decoherence and gate errors;

[0135] 24) Deterministic output mechanism for continuous actions. To generate continuous control instructions, the present invention performs Pauli-Z expectation value measurement on the quantum circuit output, as follows:

[0136] ;

[0137] ;

[0138] wherein, and act on the first two qubits, respectively; is the quantum encoding state with input state being the classical state s; is the inverse operation of the parameterized quantum circuit; is the adjustable parameter, and the output is then mapped to physical actions through a linear decoding function;

[0139] Longitudinal acceleration: Lateral lane-changing intention Through this mechanism, the selection of discrete actions based on probability amplitude sampling is avoided, providing stable and differentiable policy output in a sparse reward environment, thereby significantly improving learning efficiency.

[0140] Step three, refer to Figure 2 , construct a multi-dimensional sparse reward function;

[0141] In autonomous driving testing, the real risk is not limited to collision, but also includes the state where the system is at the performance boundary. If only a binary collision signal is used as a reward, the agent will only learn how to crash, ignoring how to induce the host vehicle to respond to the limit, and other more hidden but equally dangerous edge cases. In addition, collision events have extremely low probability in high-dimensional space (sparsity), making it difficult for the strategy to update effectively;

[0142] Therefore, according to the definition of "Known-Unsafe Scenarios" in the ISO 21448 SOTIF standard, the present invention constructs a multi-dimensional, hierarchical, and physically interpretable reward function, decomposes the sparse collision signal into multiple gradient-propagating intermediate risk indicators, thereby guiding the agent to gradually learn the complete behavior chain from "approaching danger" to "triggering danger", as follows:

[0143] 31) Define the total reward as the sum of the five terms;

[0144] ;

[0145] where, is the total reward; is the TTC-based continuous risk metric reward; is the linear reward of the absolute value of host vehicle deceleration; is the indicator reward of the jerk of the interfering vehicle; is the collision reward; is the cut-in reward;

[0146] 32) Define the five terms respectively, as follows:

[0147] 321) TTC-based continuous risk metric reward;

[0148] When the interfering vehicle is in front of the host lane, calculate the TTC:

[0149] ;

[0150] where, is the longitudinal relative distance between the host vehicle and the interfering vehicle at time t; is the longitudinal relative speed between the host vehicle and the interfering vehicle at time t;

[0151] If , give the continuous reward:

[0152] ;

[0153] 322) Linear reward of the absolute value of host vehicle deceleration;

[0154] Get the deceleration from the host vehicle dynamics model, if the absolute value exceeds the human comfort threshold, i.e. 3.0 m / s², give the linear reward:

[0155] ;

[0156] where, is the longitudinal acceleration of the host vehicle at time t;

[0157] 323) Indicator reward of the jerk of the interfering vehicle;

[0158] Calculate the longitudinal acceleration rate of change of the interfering vehicle, if it exceeds 6.0 m / s³, give a fixed reward of 2.0;

[0159] 324) : collision reward;

[0160] When the relative distance is less than 1.5 m and in the same lane, it is considered as a collision, and the reward is 20;

[0161] 325) : cut-in reward;

[0162] When the interfering vehicle successfully cuts into the main lane from the right lane and the lane change intention is greater than 0.7, a cut-in reward of 10 is given.

[0163] Step four, interactive training and optimization of the quantum policy network;

[0164] The core of reinforcement learning is that the policy is continuously optimized through interaction with the environment. However, the autonomous driving simulation environment is a classical program that cannot provide gradient signals; at the same time, the parameter update of the quantum circuit needs to adapt to the noise characteristics of the NISQ device. If the existing method directly uses a gradient-based optimizer (such as Adam), the parameters will be frozen due to the non-differentiable environment, making the policy unable to learn. Therefore, the present invention uses a gradient-free stochastic optimization algorithm (SPSA) to estimate the policy gradient direction through multiple independent environment interactions, and realizes the effective update of quantum parameters without the need for environment differentiability, as follows:

[0165] 41) The goal of the policy is to maximize the expected cumulative reward:

[0166]

[0167] where, is the objective function; is the parameter of the policy network; is a trajectory; is a quantum policy, representing a policy network implemented based on a variational quantum circuit, the parameter corresponds to the adjustable rotation angle of the quantum gate, the classical state is encoded into a quantum state, and the action output is obtained by measuring after quantum circuit evolution, thereby realizing the probability mapping from state to action; is the immediate reward obtained by executing action in state ; is the trajectory length; is sampled according to the policy distribution;

[0168] 42) Update the parameters using the Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm, specifically:

[0169] ​;

[0170] where, is the policy parameter after the th iteration; is the policy parameter at the th iteration; is the learning rate, i.e., step size, which decays with the iteration number; the gradient estimate is obtained by twice perturbation sampling:

[0171] ;

[0172] where, is the perturbation amplitude; is the random perturbation direction; , is the schedule learning rate, which takes a typical value: ;

[0173] 43) Training interaction protocol;

[0174] In each round of training iteration k:

[0175] ① Create a brand new environment instance to ensure the initial state is independent;

[0176] ② Execute the policy rollout: use the current policy Run a complete episode and record the total reward;

[0177] ③ Update the parameters: calculate ;

[0178] ④ Evaluation and record: use Generate a new trajectory, determine whether it is a key scene and update the fingerprint set.

[0179] Step five, refer to Figure 3 , output structured test cases and perform visual analysis;

[0180] If the generated scene cannot be directly used by the test platform or lacks interpretability, it will be difficult to integrate into the actual verification process. Therefore, the present application provides structured output and visual assistance, as follows:

[0181] First, save the initial state of the scene, action sequence, and fingerprint to a JSON file, compatible with CARLA and other platforms; automatically generate trajectory graphs, acceleration curves, and lane change intention timing diagrams; and output a "number of key scene discoveries - training rounds" curve to assist in testing decisions.

[0182] Embodiment two:

[0183] Refer to Figures 1-17The task goal of the embodiment is to train a quantum agent to control the disturbance vehicle in a highway following scenario, and automatically generate key test scenarios that can expose the defects of the host vehicle's automatic driving system.

[0184] Simulation environment: the host vehicle adopts a simplified IDM following model with a maximum speed of 25 m / s; the initial position of the disturbance vehicle is 30-50 meters behind the host vehicle, the initial speed is 15-30 m / s, and the initial lane is random (host lane or right lane); the simulation step is 0.1 seconds, and the longest single episode is 6 seconds.

[0185] Quantum policy network: number of qubits: 3; the variational circuit structure adopts a two-layer parameterized quantum circuit composed of Y-axis rotation gates and controlled NOT gates, which contains 6 adjustable parameters; action output: longitudinal acceleration alon∈[−5,5]m / s2, lateral lane-changing intention alat∈[0,1].

[0186] Training configuration: the Adam optimizer (learning rate 0.08) is used to update the policy parameters; a total of 120 rounds of training are performed; the key scene judgment standard is to meet any of the following conditions: the minimum relative distance is less than 5.0 meters, the maximum deceleration of the host vehicle exceeds 3.0 m / s², or a collision occurs. The entire training process is completed on a common notebook computer (Intel i7 CPU, 16 GB memory) without relying on quantum hardware.

[0187] Conclusion: the invention introduces a parameterized quantum policy network, significantly improves the efficiency and diversity of key scene discovery in a high-dimensional sparse reward environment, effectively alleviates the bottleneck of traditional methods in terms of combinatorial explosion and sample waste, and verifies its practical value in automatic driving safety testing.

[0188] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating key scenes for autonomous vehicles based on quantum reinforcement learning, characterized in that, Includes the following steps: Step 1: Build the simulation environment; 11) Define the scene state space; 12) Each time a scene is generated, the speed of the main vehicle, the position and speed of the interfering vehicle, and the lane are independently and randomly initialized; Step 2: Design a quantum policy network for continuous control; 21) Establish alignment between the number of qubits and the state dimension; 22) Mapping is performed using a normalized rotation angle; 23) A two-layer parameterized quantum circuit structure is adopted, each layer sequentially including: applying an adjustable rotation operation around the Y-axis to each qubit, and constructing chain entanglement through controlled NOT gates between adjacent qubits; 24) Design a deterministic output mechanism for continuous actions; Step 3: Construct a multidimensional sparse reward function; 31) Define the total reward as the sum of five factors: TTC risk, main vehicle deceleration, sudden acceleration change of interfering vehicle, collision, and entry reward; 32) Define each of the five items separately; Step 4: Interactively train and optimize the quantum policy network; 41) Obtain the objective of the strategy; 42) Update parameters using a synchronous perturbation random approximation algorithm; 43) In each iteration, reset the environment, execute the episode, update parameters, and record key scenarios; Step 5: Output structured test cases and perform visual analysis; 51) Determine whether it is a critical scenario based on preset criteria, and extract the scenario fingerprint; 52) Save key scenarios as JSON files and generate visualization analysis results such as trajectory diagrams and acceleration curves.

2. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 11) is as follows: Define the scene state space as ; In the formula, The relative distance between the main vehicle and the interfering vehicle; It is the relative velocity; To interfere with lane markings; The driver's behavior is driven by an intelligent driver model (IDM) to simulate longitudinal acceleration response under the influence of a vehicle ahead. The model dynamically calculates the acceleration commands the driver should take based on the relative distance and speed between the driver and the vehicle ahead to achieve safe following. The specific calculation formula for the IDM model is as follows: ； In the formula, This is the current acceleration; This is the maximum acceleration; Main vehicle speed; Free speed; The speed response index; The desired spacing; This is the actual following distance; the expected spacing is calculated as follows: ； In the formula, This is a static safety distance; For safe head spacing; It is the relative velocity; For comfortable braking deceleration; The specific method for step 12) is as follows: Each time a scene is generated, the speed of the main vehicle is independently and randomly initialized to 20–30 m / s; the initial position of the interference vehicle is set to 30–50 meters in front of the main vehicle, with an initial speed of 15–30 m / s, and the vehicle randomly selects its lane, either the main lane or an adjacent lane.

3. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 21) is as follows: Three qubits are allocated, corresponding to the relative distance and relative speed between the main vehicle and the jamming vehicle, and the lane marking of the jamming vehicle, respectively. The specific method for step 22) is as follows: The relative distance between the main vehicle and the interfering vehicle Define the normalization factor The rotation angle is set to For relative velocity Define the normalization factor The rotation angle is set to For lane markings Mapped to rotation angle The state-encoded quantum state is constructed as follows: ； State-encoded quantum state representation is a quantum state encoding process that encodes the states of a classical scenario. Mapped to a quantum state ;left It is a composite quantum state, consisting of three subsystems, corresponding to the relative distance and relative speed between the main vehicle and the interfering vehicle, and the lane marking information of the interfering vehicle; each of the subsystems on the right... It is a rotation gate around the Y-axis, acting on a single qubit; the initial state is... That is, the ground state of a quantum bit; pass The initial quantum state The calculation method for rotating to a superposition state is as follows: ； In the formula, The control parameters are obtained after normalization based on the current scene state, with values ​​ranging from [0, 2π]. Their magnitude determines the quantum state. and The overlap ratio between them; The specific method for step 23) is as follows: Employing a two-layer RY-CNOT hardware for high-efficiency ansatz; Each layer contains: 3 single-bit parameterized rotating gates Two CNOT entanglement gates: sequentially connecting bits 0→1 and 1→2 to form a chain-like entanglement structure; the adjustable parameter vector of the entire circuit is: ;in, The Y-axis rotation angle for the corresponding relative distance encoded qubit in the first layer, i.e., bit 0; The Y-axis rotation angle of the corresponding relative velocity-encoded qubit in the first layer, i.e., bit 1; The Y-axis rotation angle of the qubit corresponding to the lane identifier in the first layer, i.e., bit 2; This represents the Y-axis rotation angle corresponding to bit 0 in the second layer; This is the Y-axis rotation angle corresponding to bit 1 in the second layer; This represents the Y-axis rotation angle corresponding to bit 2 in the second layer; A differentiable mapping function from input state to desired output measurement value implemented for parameterized quantum circuits; The specific method for step 24) is as follows: The Pauli-Z expectation value of the quantum circuit output was measured, specifically as follows: ； ； In the formula, and Acting on the first two qubits respectively; For a quantum-encoded state with input state s as classical state; This is the inverse operation of parameterized quantum circuits; For adjustable parameters, output This is then mapped to physical actions using a linear decoding function: Longitudinal acceleration: lateral lane change intention .

4. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 31) is as follows: The total reward is defined as the sum of five items, specifically: ； In the formula, Total reward; Rewards are based on continuous risk measurement using TTC. The linear reward of the absolute value of the vehicle's deceleration; A reward is given for indicating sudden changes in the acceleration of the interfering vehicle; For collision rewards; To introduce rewards; The specific method for step 32) is as follows: 321) Calculate TTC when the interfering vehicle is in front of the main lane: ； In the formula, The longitudinal relative distance between the main vehicle and the interfering vehicle at time t; Let be the longitudinal relative velocity between the main vehicle and the interfering vehicle at time t; like Continuous rewards will be given: ； 322) Obtain the deceleration from the main vehicle dynamics model. If the absolute value exceeds the human comfort threshold of 3.0 m / s², then grant a linear bonus: ； In the formula, Let t be the longitudinal acceleration of the main vehicle; 323) Calculate the rate of change of longitudinal acceleration of the interfering vehicle. If it exceeds 6.0 m / s³, a fixed bonus of 2.0 will be awarded. 324) When the relative distance is less than 1.5m and they are in the same lane, it is considered a collision, and the reward is 20; 325) When the interfering vehicle successfully cuts into the main lane from the right lane and intends to change lanes. If the value is greater than 0.7, a cut-in bonus of 10 will be given.

5. The method for generating key scenes for autonomous vehicles based on quantum reinforcement learning according to claim 1, characterized in that, The specific method for step 41) is as follows: The goal of the strategy is to maximize the expected cumulative reward, specifically: ； In the formula, The objective function is... These are the parameters of the policy network; For a trajectory; Quantum policy represents a policy network based on variable quantum circuits. The parameters correspond to the adjustable rotation angles of quantum gates. Classical states are encoded into quantum states, and the action output is obtained by measuring the evolution through quantum circuits, thereby realizing the probabilistic mapping from state to action. In the state Next action The instant reward received; The length of the trajectory; In accordance with the strategy Distributed sampling; The specific method for step 42) is as follows: The SPSA algorithm is used to update the parameters, specifically as follows: ； In the formula, For the first The strategy parameters after the next iteration; For the first Strategy parameters at the next iteration; The learning rate, or step size, decreases with the number of iterations; gradient estimation Obtained through two perturbation samplings: ； In the formula, The amplitude of the disturbance; For random perturbation direction; , To set the learning rate, a typical value is used: ; The specific method for step 43) is as follows: In each training iteration k: ① Create a completely new environment instance to ensure that the initial state is independent; ② Execute the rollout policy: Use the current policy Run the complete episode and record the total reward; ③ Update parameters: Calculate according to the SPSA formula ; ④ Assessment and Recording: Using Generate a new trajectory, determine if it is a critical scenario, and update the fingerprint set.

Images