Vehicle trajectory planning method based on behavioral anchoring double-time scale parameterized action

Abstract

The application discloses a vehicle trajectory planning method based on behavior anchoring double-time scale parameterized action, comprising the following steps: constructing a vehicle mixed decision state space and a parameterized action space based on the interaction between a vehicle and a traffic environment; performing high-layer discrete behavior latching and target lane keeping; performing low-layer continuous trajectory parameter and longitudinal acceleration frame-by-frame refreshing; performing rolling trajectory re-planning and new and old trajectory fusion; constructing a behavior experience pool and a micro experience pool and performing hierarchical sample storage; performing hierarchical value function training based on a behavior Critic and a micro Critic; and performing parameter updating and closed-loop iteration. The application can balance the lateral behavior stability and the longitudinal control real-time performance, and can maintain reliability in a long-time domain continuous driving task.

Description

Technical Field

[0001] This invention relates to the field of automotive technology, and more specifically to a vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions. Background Technology

[0002] In the field of autonomous driving, Parameterized Deep Q-Network (PDQN) is a typical hybrid reinforcement learning algorithm that can simultaneously output discrete actions and corresponding continuous control parameters, making it suitable for handling autonomous driving tasks involving discrete lane-changing decisions and continuous trajectory acceleration control. Existing single-timescale PDQN baseline schemes have the policy decision cycle and action execution cycle perfectly aligned. When a low-frequency policy (e.g., 0.5Hz) is used to reduce training overhead, the discrete lane-changing behavior, trajectory parameters, and acceleration parameters output in a single cycle remain unchanged throughout the entire decision cycle, leading to several drawbacks: First, discrete behavior and continuous parameters are strongly coupled and locked. When lane-changing intentions are locked, acceleration and trajectory parameters are also locked. When surrounding vehicles undergo dynamic changes such as acceleration, deceleration, and lane-changing, the vehicle cannot adjust longitudinal control in real time, resulting in a lagging longitudinal response and significantly increasing the risk of collision.

[0003] Second, the open-loop trajectory execution cannot keep up with the vehicle's real-time position and posture correction. Directly replacing the trajectory with a new one will cause trajectory jumps, resulting in steering vibration and poor ride comfort.

[0004] Third, the training samples are poorly processed, storing only the cumulative reward of the complete decision window while losing frame-by-frame control feedback information, resulting in a large bias in the value function estimation.

[0005] If the frequency of the strategy is directly increased to solve the response lag problem, it will lead to frequent switching of discrete lane-changing behavior, repeated jumps in lateral intent, and frequent back-and-forth lane corrections by the vehicle, which will also deteriorate safety and comfort. Other baseline algorithms, such as PPO and SAC, can output continuous actions at high frequency, but they lack explicit discrete-continuous hierarchical modeling for lane-changing behavior, resulting in high collision risk and low passage completion rate in high-density mixed traffic flow. In addition, the performance evaluation of existing deep reinforcement learning-based autonomous driving decision-making methods is mostly carried out in short time-domain simulations of 20 to 40 seconds, and the test duration only covers a single or a few driving decision-making processes. The above-mentioned short time-domain evaluation paradigm cannot reflect the safety risks that gradually accumulate in long-term continuous driving tasks caused by the coupling and locking of discrete behavior and continuous parameters under the low-frequency decision framework. When the test time domain is extended to 200 seconds, the collision rate rises sharply and the passage completion rate drops significantly, posing a serious challenge to the reliability of existing methods.

[0006] In summary, existing technologies face a dilemma: low-frequency strategies exhibit lag in longitudinal response, while high-frequency strategies suffer from lateral intent jitter. How to decouple the discrete behavior selection frequency from the continuous control parameter update frequency, balancing lateral behavior stability and longitudinal control real-time performance, and maintaining reliability in long-term continuous driving tasks, is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0007] In view of this, the present invention provides a vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions, so as to take into account both lateral behavior stability and longitudinal control real-time performance, and maintain reliability in long-term continuous driving tasks.

[0008] A vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions includes: S1, based on the interaction between the vehicle and the traffic environment, obtains the current environmental observation data, and then constructs the vehicle's hybrid decision-making state space and parameterized action space; S2, based on the vehicle hybrid decision state space, the high-level behavior network outputs high-level discrete behavior at the starting boundary of the behavior window according to the first decision frequency, and determines the target lane according to the current lane and the high-level discrete behavior. The high-level discrete behavior is latched within the behavior window and the target lane remains unchanged. S3, within the behavior window, based on the latched high-level discrete behavior and the vehicle observation state of each low-level simulation frame, the micro-policy network outputs low-level continuous control parameters in each low-level simulation frame. The low-level continuous control parameters include trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control parameters. The low-level simulation frame operates at a second frequency higher than the first decision frequency. S4. In each underlying simulation frame, the current state of the vehicle is used as the starting point for new trajectory planning. A new trajectory is generated while keeping the target lane unchanged. The new trajectory is then weighted and fused with the remaining segments of the old trajectory that have not yet been executed in the reference trajectory of the previous moment to obtain the fused reference trajectory. The fused reference trajectory is then input into the trajectory tracking controller to output the vehicle's lateral steering control quantity. S5. Construct a behavior experience pool and a micro experience pool respectively. Store the state of the starting boundary of the behavior window, the latched high-level discrete behavior, the accumulated outer layer reward within the behavior window, the state of the starting boundary of the next behavior window, and the termination mark as behavior window-level samples in the behavior experience pool. Store the vehicle observation state, latched high-level discrete behavior, trajectory longitudinal displacement adjustment parameters, longitudinal acceleration control quantity, instantaneous reward, vehicle observation state of the next bottom-level simulation frame, and micro sample boundary mark as bottom-level simulation frame-level samples in the micro experience pool. S6 trains a high-level behavior Critic based on samples in the behavior experience pool. The high-level behavior Critic only updates the high-level discrete behavior value at the behavior window boundary. It trains micro-Critics and micro-Actors based on samples in the micro-experience pool. The micro-Critic learns the value function of the low-level continuous control parameters under the condition of latching high-level discrete behavior and updates the value using a multi-step temporal difference objective. The micro-Actor takes maximizing the value of the micro-Critic as its training objective. Then, it updates the network parameters of the high-level behavior network and the micro-policy network according to the training results for the next round of vehicle trajectory decision and low-level control.

[0009] The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions provided by the present invention has the following beneficial effects: 1. This invention employs a dual-timescale decoupling design, separating high-level discrete behavioral decisions from low-level continuous parameter updates. A behavior window latches high-level discrete behaviors, preventing frequent lane-changing intention jumps and resolving the lateral intention jitter problem caused by high-frequency strategies. Simultaneously, continuous trajectory parameters and acceleration are refreshed frame-by-frame, overcoming the longitudinal response lag problem caused by low-frequency strategies. This enables the vehicle to respond in real-time to dynamic changes such as acceleration, deceleration, and lane-changing movements of surrounding vehicles, significantly reducing the collision risk in dynamic traffic scenarios.

[0010] 2. This invention proposes a rolling trajectory replanning and fusion mechanism, which retains the unexecuted part of the old trajectory, eliminates the starting point deviation between the old and new trajectories by translation alignment, and then performs weighted fusion with the new trajectory. This avoids lateral abrupt changes caused by directly replacing the trajectory, reduces steering variance, and improves ride comfort.

[0011] 3. This invention employs a hierarchical experience pool and a hierarchically connected Bellman objective to distinguish between high-level lane-changing behavior samples and low-level frame-by-frame control samples. This ensures that the behavioral decision boundary, trajectory execution boundary, and value function training boundary remain consistent across the time scale. It fully utilizes real-time frame-by-frame reward feedback to improve the accuracy of value function estimation, enhance sample utilization efficiency, and accelerate algorithm convergence. The training convergence curves show that the method of this invention exhibits faster convergence speed and higher reward levels even in the early stages of training.

[0012] 4. This invention demonstrates superior stability improvement at a traffic flow density of V / C=0.3 and maintains its performance advantage in a 200-second long-term continuous driving test, overcoming the shortcomings of existing methods that experience a sharp performance degradation with increasing test duration. According to unified simulation tests, compared to the baseline method, this invention offers higher average round rewards, higher simulation execution completion rate, and lower collision rate, reducing invalid lane changes and frequent acceleration / deceleration, and improving the traffic efficiency of mixed traffic flows. Therefore, this invention can balance lateral behavioral stability and longitudinal control real-time performance while maintaining reliability in long-term continuous driving tasks. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions. Figure 2 This is a schematic diagram of the rolling trajectory replanning and fusion mechanism of the present invention; Figure 3 A comparison chart of typical core indicators for each method under a traffic flow density of V / C=0.3; Figure 4 The graph shows a comparison of the training convergence curves of each method at a traffic flow density of V / C=0.3. Detailed Implementation

[0014] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain embodiments of the present invention, and should not be construed as limiting the present invention.

[0015] Please see Figure 1 The present invention provides a vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions, the method comprising steps S1 to S6: S1 obtains current environmental observation data based on the interaction between the vehicle and the traffic environment, and then constructs the vehicle's hybrid decision-making state space and parameterized action space.

[0016] The vehicle's hybrid decision-making state space includes the vehicle's motion state, lane state, and interaction state with surrounding vehicles. The vehicle's motion state includes at least its speed, heading angle, and lateral offset. The interaction state with surrounding vehicles includes at least the relative distance and relative speed between the vehicle in front and vehicles in adjacent lanes. The parameterized action space consists of high-level discrete behaviors and low-level continuous control parameters. The high-level discrete behaviors include at least lane keeping, left lane changing, and right lane changing. The low-level continuous control parameters include at least trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control parameters. Represented as:

[0017] in, For high-level discrete behavior, Indicates lane keeping. This indicates a lane change to the left. This indicates a lane change to the right. For adjusting the longitudinal displacement of the trajectory, For longitudinal acceleration control, and These are the minimum and maximum values ​​of the longitudinal displacement adjustment parameter for the trajectory, respectively. and These are the minimum and maximum values ​​of the longitudinal acceleration control quantity, respectively.

[0018] S2, based on the vehicle hybrid decision state space, the high-level behavior network outputs high-level discrete behavior at the starting boundary of the behavior window according to the first decision frequency, and determines the target lane according to the current lane and the high-level discrete behavior. The high-level discrete behavior is latched within the behavior window and the target lane remains unchanged.

[0019] It should be noted that the high-level behavioral network is a behavioral value network during decision-making. During training, it is a high-level behavioral criterion.

[0020] The number of underlying simulation frames contained in the behavior window is determined by the following formula:

[0021]

[0022]

[0023] in, For the underlying simulation of the second frequency, The highest frequency of decision-making for senior management. This is the time interval for the underlying simulation. The number of underlying simulation frames contained in a behavior window. The total duration of the behavior window. This represents the rounding function; High-level discrete behaviors are selected only by the behavior value network at the initial boundary of the behavior window, and the calculation formula is as follows:

[0024] in, For the first The vehicle state at the starting boundary of each behavior window. For a set of high-level discrete behaviors, For behavioral value networks, For the first High-level discrete behavior for selecting the initial boundary of a behavior window; Within the behavior window, the high-level discrete behavior remains latched unchanged, represented as:

[0025]

[0026] in, For the first Within the first behavior window High-level discrete behavior executed by a low-level simulation frame.

[0027] After the initial boundary of the behavior window is determined, the target lane is determined based on the current lane and the high-level discrete behavior, and the target lane remains unchanged within the behavior window. The formula for determining the target lane is:

[0028] in, For the first The current lane at the starting boundary of each behavior window. The function is the mapping from high-level discrete behavior to the target lane. For the first The target lane corresponding to each behavior window.

[0029] Inside the behavior window In the lowest-level simulation frame, the target lane remains unchanged, represented as:

[0030] in, For the first Within the first behavior window The target lane corresponding to each underlying simulation frame.

[0031] S3, within the behavior window, based on the latched high-level discrete behavior and the vehicle observation state of each low-level simulation frame, the micro-policy network outputs low-level continuous control parameters in each low-level simulation frame. The low-level continuous control parameters include trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control quantities. The low-level simulation frames operate at a second frequency higher than the first decision frequency.

[0032] The micro-policy network takes the vehicle observation state of the current low-level simulation frame and the latched high-level discrete behavior as input, and outputs the trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control parameters, calculated as follows:

[0033] in, For the first Within the first behavior window The vehicle observation state of each low-level simulation frame For the first The high-level discrete behaviors latched within the behavior window (i.e., the aforementioned first behavior window) (High-level discrete behavior of selecting the starting boundary of each behavior window) For micro-policy networks, For the first Within the first behavior window The longitudinal displacement adjustment parameters of the trajectory output from each low-level simulation frame. For the first Within the first behavior window The longitudinal acceleration control quantity output by each low-level simulation frame; Thus, the high-level discrete behavior is updated only at the beginning boundary of the behavior window and latched within the window, while the low-level continuous control parameters are updated in each low-level simulation frame, achieving dual-timescale decoupling between discrete behavior decision-making and continuous control parameter updates.

[0034] S4. In each underlying simulation frame, the current state of the vehicle is used as the starting point for new trajectory planning. A new trajectory is generated while keeping the target lane unchanged. The new trajectory is then weighted and fused with the remaining segments of the old trajectory that have not yet been executed in the reference trajectory of the previous moment to obtain the fused reference trajectory. The fused reference trajectory is then input into the trajectory tracking controller to output the vehicle's lateral steering control quantity.

[0035] The process involves translating and aligning the remaining segment of the old trajectory to the starting point of the new trajectory, resulting in the aligned remaining segment of the old trajectory. The calculation formula is as follows:

[0036] in, The first segment of the remaining segment of the old trajectory A trajectory point, This is the starting point of the remaining segment of the old trajectory. The starting point of the new trajectory. For the first line after translation and alignment An old trajectory point; The remaining segments of the aligned old trajectory are then weighted and fused with the new trajectory to obtain the fused reference trajectory. The calculation formula is as follows:

[0037] in, For the first in the new trajectory A trajectory point, To fuse the first reference trajectory A trajectory point, For trajectory fusion weight coefficients, and The fused reference trajectory is input into the trajectory tracking controller, which then outputs the vehicle's lateral steering control quantity.

[0038] The new trajectory is generated using a quintic polynomial, and the quintic polynomial trajectory is represented as:

[0039] in, These are the local longitudinal coordinates in the road coordinate system. The horizontal position , , , , , The coefficients are those of a fifth-degree polynomial. The coefficients of the fifth-order polynomial are determined based on the current vehicle state and the target lane constraints. The boundary constraints include:

[0040]

[0041]

[0042]

[0043]

[0044]

[0045] in, Represents the local vertical coordinate The horizontal position of the location (i.e., the starting point of the new trajectory). The first derivative of the lateral position with respect to the longitudinal coordinate is... The value at that point (i.e., the horizontal slope at the starting point of the new trajectory). The second derivative of the lateral position with respect to the axial coordinate is... The value at that point (i.e., the lateral curvature correlation quantity at the starting point of the new trajectory). The vehicle's current heading angle. This is a quantity related to the rate of change of the vehicle's current heading. The vehicle's current speed. This represents the lateral offset of the target lane relative to the current lane. The longitudinal distance to the endpoint of the trajectory. This indicates the horizontal position at the vertical endpoint. The first derivative of the lateral position with respect to the longitudinal coordinate is... The value at that location, The second derivative of the lateral position with respect to the axial coordinate is... The value at that location.

[0046] S5. Construct a behavioral experience pool and a micro-experience pool respectively. Store the state of the starting boundary of the behavioral window, the latched high-level discrete behavior, the accumulated outer layer reward within the behavioral window, the state of the starting boundary of the next behavioral window, and the termination mark as behavioral window-level samples in the behavioral experience pool. Store the vehicle observation state, latched high-level discrete behavior, trajectory longitudinal displacement adjustment parameters, longitudinal acceleration control quantity, instantaneous reward, vehicle observation state of the next bottom-level simulation frame, and micro-sample boundary marker as bottom-level simulation frame-level samples in the micro-experience pool.

[0047] Among them, the sample set of the behavioral experience pool Represented as:

[0048] in, For the first The state of the starting boundary of each behavior window For the first The outer reward accumulated within each behavior window For the first The state of the starting boundary of each behavior window This is a termination marker; The sample set of the micro-experience pool is represented as:

[0049] in, For the first The vehicle observation state of each low-level simulation frame For the first The latching of high-level discrete behavior corresponding to each low-level simulation frame. For the first The longitudinal displacement adjustment parameters of the trajectory output from each low-level simulation frame. For the first The longitudinal acceleration control value output by each low-level simulation frame For the first Instant reward for each underlying simulation frame For the first The vehicle observation status of each low-level simulation frame; Microscopic sample boundary markers are used to identify whether a microscopic sample has reached the boundary of the behavior window or is in a termination state; By separating and storing the behavioral experience pool and the micro-experience pool, the high-level behavioral value learning corresponds to the behavioral window boundary, and the micro-continuous control value learning corresponds to the low-level simulation frame boundary.

[0050] S6 trains a high-level behavior Critic based on samples in the behavior experience pool. The high-level behavior Critic only updates the high-level discrete behavior value at the behavior window boundary. It trains micro-Critics and micro-Actors based on samples in the micro-experience pool. The micro-Critic learns the value function of the low-level continuous control parameters under the condition of latching high-level discrete behavior and updates the value using a multi-step temporal difference objective. The micro-Actor takes maximizing the value of the micro-Critic as its training objective. Then, it updates the network parameters of the high-level behavior network and the micro-policy network according to the training results for the next round of vehicle trajectory decision and low-level control.

[0051] In this process, a high-level behavior Critic is trained based on a behavior experience pool. The high-level behavior Critic only updates the high-level discrete behavior value at the boundaries of the behavior window. The Bellman objective is expressed as follows:

[0052] in, The target value for high-level behavior Critic. For high-level discount factors, The next row represents the set of legal high-level discrete behaviors in the initial state of the window. For high-level behavioral goals and value networks The next line represents the state of the window's starting boundary. The next line represents the high-level discrete behavior to be evaluated at the starting boundary of the window. Loss function of high-level behavior Critic Represented as:

[0053] in, For high-level behavioral value networks.

[0054] In this embodiment, micro-critic and micro-actor are trained based on a micro-experience pool. The number of multi-step temporal difference steps is 5, and the cumulative micro-reward for 5 steps is represented as:

[0055] in, For from the first The cumulative reward over 5 steps starting from the lowest simulation frame. For micro-level discount factor, It is the summation index in the multi-step time difference summation formula. For the first Instant rewards for each underlying simulation frame; The target value of the micro-critic is expressed as:

[0056] in, The target value for microscopic Critic, For the first The termination marker corresponding to each underlying simulation frame. For micro-level target value networks, For the first The vehicle observation state of each low-level simulation frame The micro-policy network is represented in the first... The continuous control parameters output by each low-level simulation frame are equivalent to , For the first The longitudinal displacement adjustment parameters of the trajectory output from each low-level simulation frame. For the first The longitudinal acceleration control quantity output by each low-level simulation frame; When the 5-step temporal difference objective reaches the boundary of the behavior window, the terminal value of the micro-critic is linked with the value estimate of the high-level behavioral critic. Loss function of micro-critic Represented as:

[0057] in, For micro-value networks; The micro-actuator is trained to maximize the value of the micro-critic, and the objective function and loss function are expressed as follows:

[0058]

[0059] in, Let the objective function of the microscopic actor be... The loss function for micro-actors. Indicates the vehicle observation status The mathematical expectation, The micro-policy network is represented in the first... Continuous control parameters output from each low-level simulation frame.

[0060] After completing the training of the hierarchical value function, the network parameters of the dual-timescale parameterized decision model are updated based on the training results of the behavioral Critic, micro-Critic, and micro-Actor. The dual-timescale parameterized decision model includes a high-level behavioral network and a micro-policy network. The updated high-level behavioral evaluation network is used to select high-level discrete behaviors at the starting boundary of the next behavioral window, while the updated micro-evaluation network and continuous parameter policy network are used to output trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control quantities frame-by-frame in the next vehicle state. By continuously using behavioral window-level experience and low-level simulation frame-level experience collected during vehicle execution for model training and updates, the dual-timescale parameterized decision model is continuously optimized during the continuous interaction between the vehicle and the traffic environment. The updated model is then used for the next round of vehicle trajectory decision-making, trajectory planning, and low-level control.

[0061] The following simulation test demonstrates the invention, specifically using a multi-vehicle mixed traffic simulation scenario on a highway as an example, employing an autonomous driving simulation environment based on highway-env. The underlying simulation frequency is 5Hz, and the first decision frequency for higher-level behaviors is 0.5Hz; therefore, one behavior window contains 10 underlying simulation frames; trajectory fusion weight coefficients... Set the value to 0.35; set the number of multi-step temporal difference steps for micro-critic to 5; set the capacity of both the behavioral experience pool and the micro-experience pool to 100,000; set the training batch size to 256; and set the network learning rate to 0.0001.

[0062] like Figure 2 As shown, during vehicle operation, the current vehicle pose is used as the starting point for new trajectory planning. The remaining segments of the old trajectory that were not executed are extracted, and these remaining segments are translated and aligned to the starting point of the new trajectory. Then, a weighted fusion is performed with the newly generated trajectory to obtain a continuous and smooth reference trajectory, avoiding lateral abrupt changes caused by directly replacing the trajectory. For example... Figure 3 As shown, under traffic flow conditions of V / C=0.3, the method of this invention is compared with traditional PDQN (parametric deep Q-network algorithm, as the baseline method), SAC (soft actor-critic algorithm), and PPO (proximate policy optimization algorithm). Each method is executed for 200 test rounds under this traffic flow density. The evaluation random seed is uniformly set from 1 to 200. The simulation time of each round is set to 200 seconds. The key evaluation indicators include average round reward, collision rate, round completion rate, steering variance, and acceleration variance.

[0063] Experimental results show that the method of this invention significantly outperforms the baseline method in three core metrics: average reward, completion rate, and collision rate. The average reward reaches 308.22, a 44.4% improvement compared to the optimal baseline PDQN-base (213.38); the completion rate reaches 92.94%, a 22.0% improvement compared to the optimal baseline PDQN-base (76.17%); and the collision rate is as low as 0.014, a 73.1% reduction compared to the optimal baseline PDQN-base (0.052). Regarding comfort metrics, the steering variance of the method of this invention is only 0.0007, far lower than SAC-con (0.0192) and PDQN-base (0.0061), verifying the effectiveness of the rolling trajectory replanning mechanism in reducing lateral jitter; the acceleration variance is 1.38, a 56.2% reduction compared to PDQN-base (3.15), demonstrating the advantage of the low-level continuous parameter frame-by-frame refresh mechanism in improving longitudinal smoothness. The above results verify the stability, safety, and ride comfort of the present invention in long-term continuous driving tasks.

[0064] like Figure 4As shown, the training convergence curves illustrate the reward trends of the four methods during 180,000 frames of environmental interaction. The results indicate that the method described in this invention exhibits faster convergence speed and higher reward levels in the early stages of training. As training progresses, the reward curve continues to rise steadily and maintains its leading position. In contrast, PPO and SAC have slower convergence speeds and lower reward levels. While PDQN-base shows some convergence effect, its overall performance is significantly weaker than that of the method described in this invention. This invention stabilizes lateral intent through a high-level behavior latching mechanism, achieves real-time longitudinal response through frame-by-frame refresh of low-level continuous parameters, ensures trajectory smoothness through rolling trajectory replanning, and improves sample efficiency and value function estimation accuracy through hierarchical experience pools and hierarchical Bellman objectives. This ensures that the behavioral decision boundary, trajectory execution boundary, and value function training boundary remain consistent across the time scale, thereby significantly improving training convergence stability and final decision performance.

[0065] In summary, the vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to the above embodiments has the following beneficial effects: 1. This invention employs a dual-timescale decoupling design, separating high-level discrete behavioral decisions from low-level continuous parameter updates. A behavior window latches high-level discrete behaviors, preventing frequent lane-changing intention jumps and resolving the lateral intention jitter problem caused by high-frequency strategies. Simultaneously, continuous trajectory parameters and acceleration are refreshed frame-by-frame, overcoming the longitudinal response lag problem caused by low-frequency strategies. This enables the vehicle to respond in real-time to dynamic changes such as acceleration, deceleration, and lane-changing movements of surrounding vehicles, significantly reducing the collision risk in dynamic traffic scenarios.

[0066] 2. This invention proposes a rolling trajectory replanning and fusion mechanism, which retains the unexecuted part of the old trajectory, eliminates the starting point deviation between the old and new trajectories by translation alignment, and then performs weighted fusion with the new trajectory. This avoids lateral abrupt changes caused by directly replacing the trajectory, reduces steering variance, and improves ride comfort.

[0067] 3. This invention employs a hierarchical experience pool and a hierarchically connected Bellman objective to distinguish between high-level lane-changing behavior samples and low-level frame-by-frame control samples. This ensures that the behavioral decision boundary, trajectory execution boundary, and value function training boundary remain consistent across the time scale. It fully utilizes real-time frame-by-frame reward feedback to improve the accuracy of value function estimation, enhance sample utilization efficiency, and accelerate algorithm convergence. The training convergence curves show that the method of this invention exhibits faster convergence speed and higher reward levels even in the early stages of training.

[0068] 4. This invention demonstrates superior stability improvement at a traffic flow density of V / C=0.3 and maintains its performance advantage in a 200-second long-term continuous driving test, overcoming the shortcomings of existing methods that experience a sharp performance degradation with increasing test duration. According to unified simulation tests, compared to the baseline method, this invention offers higher average round rewards, higher simulation execution completion rate, and lower collision rate, reducing invalid lane changes and frequent acceleration / deceleration, and improving the traffic efficiency of mixed traffic flows. Therefore, this invention can balance lateral behavioral stability and longitudinal control real-time performance while maintaining reliability in long-term continuous driving tasks.

[0069] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A vehicle trajectory planning method based on behavior-anchored double-time-scale parameterized motion, characterized in that, include: S1, based on the interaction between the vehicle and the traffic environment, obtains the current environmental observation data, and then constructs the vehicle's hybrid decision-making state space and parameterized action space; S2, based on the vehicle hybrid decision state space, the high-level behavior network outputs high-level discrete behavior at the starting boundary of the behavior window according to the first decision frequency, and determines the target lane according to the current lane and the high-level discrete behavior. The high-level discrete behavior is latched within the behavior window and the target lane remains unchanged. S3, within the behavior window, based on the latched high-level discrete behavior and the vehicle observation state of each low-level simulation frame, the micro-policy network outputs low-level continuous control parameters in each low-level simulation frame. The low-level continuous control parameters include trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control parameters. The low-level simulation frame operates at a second frequency higher than the first decision frequency. S4. In each underlying simulation frame, the current state of the vehicle is used as the starting point for new trajectory planning. A new trajectory is generated while keeping the target lane unchanged. The new trajectory is then weighted and fused with the remaining segments of the old trajectory that have not yet been executed in the reference trajectory of the previous moment to obtain the fused reference trajectory. The fused reference trajectory is then input into the trajectory tracking controller to output the vehicle's lateral steering control quantity. S5. Construct a behavior experience pool and a micro experience pool respectively. Store the state of the starting boundary of the behavior window, the latched high-level discrete behavior, the accumulated outer layer reward within the behavior window, the state of the starting boundary of the next behavior window, and the termination mark as behavior window-level samples in the behavior experience pool. Store the vehicle observation state, latched high-level discrete behavior, trajectory longitudinal displacement adjustment parameters, longitudinal acceleration control quantity, instantaneous reward, vehicle observation state of the next bottom-level simulation frame, and micro sample boundary mark as bottom-level simulation frame-level samples in the micro experience pool. S6 trains a high-level behavior Critic based on samples in the behavior experience pool. The high-level behavior Critic only updates the high-level discrete behavior value at the behavior window boundary. It trains micro-Critics and micro-Actors based on samples in the micro-experience pool. The micro-Critic learns the value function of the low-level continuous control parameters under the condition of latching high-level discrete behavior and updates the value using a multi-step temporal difference objective. The micro-Actor takes maximizing the value of the micro-Critic as its training objective. Then, it updates the network parameters of the high-level behavior network and the micro-policy network according to the training results for the next round of vehicle trajectory decision and low-level control.

2. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 1, characterized in that, In step S1, the vehicle hybrid decision state space includes the ego vehicle motion state, the lane state and the surrounding vehicle interaction state, the ego vehicle motion state at least includes the ego vehicle speed, the heading angle and the lateral offset, the surrounding vehicle interaction state at least includes the relative distance and the relative speed of the front vehicle and the adjacent lane vehicle; the parameterized action space is jointly constituted by the high-level discrete behaviors and the low-level continuous control parameters, wherein the high-level discrete behaviors at least include the lane keeping, the left lane changing and the right lane changing, the low-level continuous control parameters at least include the trajectory longitudinal displacement adjustment parameter and the longitudinal acceleration control amount, the parameterized action space is represented as: in, For high-level discrete behavior, Indicates lane keeping. This indicates a lane change to the left. This indicates a lane change to the right. For adjusting the longitudinal displacement of the trajectory, For longitudinal acceleration control, and These are the minimum and maximum values ​​of the longitudinal displacement adjustment parameter for the trajectory, respectively. and These are the minimum and maximum values ​​of the longitudinal acceleration control quantity, respectively.

3. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 2, characterized in that, In step S2, the number of underlying simulation frames contained in the behavior window is determined by the following formula: in, For the underlying simulation of the second frequency, The highest frequency of decision-making for senior management. This is the time interval for the underlying simulation. The number of underlying simulation frames contained in a behavior window. The total duration of the behavior window. This represents the rounding function; High-level discrete behaviors are selected only by the behavior value network at the initial boundary of the behavior window, and the calculation formula is as follows: in, For the first The vehicle state at the starting boundary of each behavior window. For a set of high-level discrete behaviors, For behavioral value networks, For the first High-level discrete behavior for selecting the initial boundary of a behavior window; Within the behavior window, the high-level discrete behavior remains latched unchanged, represented as: in, For the first Within the first behavior window High-level discrete behavior executed by a low-level simulation frame; The formula for determining the target lane is: in, For the first The current lane at the starting boundary of each behavior window. The function is the mapping from high-level discrete behavior to the target lane. For the first The target lane corresponding to each behavior window; Inside the behavior window In the lowest-level simulation frame, the target lane remains unchanged, represented as: in, For the first Within the first behavior window The target lane corresponding to each underlying simulation frame.

4. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 3, characterized in that, In step S3, the micro-policy network takes the vehicle observation state of the current low-level simulation frame and the latched high-level discrete behavior as input, and outputs the trajectory longitudinal displacement adjustment parameters and longitudinal acceleration control quantity, calculated as follows: in, For the first Within the first behavior window The vehicle observation state of each low-level simulation frame For micro-policy networks, For the first Within the first behavior window The longitudinal displacement adjustment parameters of the trajectory output from each low-level simulation frame. For the first Within the first behavior window The longitudinal acceleration control quantity output by each low-level simulation frame; Thus, the high-level discrete behavior is updated only at the beginning boundary of the behavior window and latched within the window, while the low-level continuous control parameters are updated in each low-level simulation frame, achieving dual-timescale decoupling between discrete behavior decision-making and continuous control parameter updates.

5. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 4, characterized in that, In step S4, the remaining segment of the old trajectory is translated and aligned to the starting point of the new trajectory to obtain the aligned remaining segment of the old trajectory. The calculation formula is as follows: in, The first segment of the remaining segment of the old trajectory A trajectory point, This is the starting point of the remaining segment of the old trajectory. The starting point of the new trajectory. For the first line after translation and alignment An old trajectory point; The remaining segments of the aligned old trajectory are then weighted and fused with the new trajectory to obtain the fused reference trajectory. The calculation formula is as follows: in, For the first in the new trajectory A trajectory point, To fuse the first reference trajectory A trajectory point, For trajectory fusion weight coefficients, and ; The new trajectory is generated using a quintic polynomial, and the quintic polynomial trajectory is represented as: in, These are the local longitudinal coordinates in the road coordinate system. The horizontal position , , , , , The coefficients are those of a fifth-degree polynomial. The coefficients of the fifth-order polynomial are determined based on the current vehicle state and the target lane constraints. The boundary constraints include: in, Represents the local vertical coordinate The horizontal position at that location The first derivative of the lateral position with respect to the longitudinal coordinate is... The value at that location, The second derivative of the lateral position with respect to the axial coordinate is... The value at that location, The vehicle's current heading angle. This is a quantity related to the rate of change of the vehicle's current heading. The vehicle's current speed. This represents the lateral offset of the target lane relative to the current lane. The longitudinal distance to the endpoint of the trajectory. This indicates the horizontal position at the vertical endpoint. The first derivative of the lateral position with respect to the longitudinal coordinate is... The value at that location, The second derivative of the lateral position with respect to the axial coordinate is... The value at that location.

6. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 5, characterized in that, In step S5, the sample set of the behavioral experience pool Represented as: in, For the first The state of the starting boundary of each behavior window For the first The outer reward accumulated within each behavior window For the first The state of the starting boundary of each behavior window This is a termination marker; The sample set of the micro-experience pool is represented as: in, For the first The vehicle observation state of each low-level simulation frame For the first The latching of high-level discrete behavior corresponding to each low-level simulation frame. For the first The longitudinal displacement adjustment parameters of the trajectory output from each low-level simulation frame. For the first The longitudinal acceleration control value output by each low-level simulation frame For the first Instant reward for each underlying simulation frame For the first The vehicle observation status of each low-level simulation frame; Microscopic sample boundary markers are used to identify whether a microscopic sample has reached the boundary of the behavior window or is in a termination state; By separating and storing the behavioral experience pool and the micro-experience pool, the high-level behavioral value learning corresponds to the behavioral window boundary, and the micro-continuous control value learning corresponds to the low-level simulation frame boundary.

7. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 6, characterized in that, In step S6, a high-level behavior Critic is trained based on the behavior experience pool. The high-level behavior Critic only updates the high-level discrete behavior value at the boundaries of the behavior window. The Bellman objective is expressed as: in, The target value for high-level behavior Critic. For high-level discount factors, The next row represents the set of legal high-level discrete behaviors in the initial state of the window. For high-level behavioral goals and value networks The next line represents the state of the window's starting boundary. The next line represents the high-level discrete behavior to be evaluated at the starting boundary of the window. Loss function of high-level behavior Critic Represented as: in, For high-level behavioral value networks.

8. The vehicle trajectory planning method based on behavior-anchored dual-time-scale parameterized actions according to claim 7, characterized in that, In step S6, micro-critic and micro-actor are trained based on the micro-experience pool. The number of multi-step time-series difference steps is 5, and the cumulative micro-reward for 5 steps is represented as: in, For from the first The cumulative reward over 5 steps starting from the lowest simulation frame. For micro-level discount factor, It is the summation index in the multi-step time difference summation formula. For the first Instant rewards for each underlying simulation frame; The target value of the micro-critic is expressed as: in, The target value for microscopic Critic, For the first The termination marker corresponding to each underlying simulation frame. For micro-level target value networks, For the first The vehicle observation state of each low-level simulation frame The micro-policy network is represented in the first... Continuous control parameters output from each low-level simulation frame; When the 5-step temporal difference objective reaches the boundary of the behavior window, the terminal value of the micro-critic is linked with the value estimate of the high-level behavioral critic. Loss function of micro-critic Represented as: in, For micro-value networks; The micro-actuator is trained to maximize the value of the micro-critic, and the objective function and loss function are expressed as follows: in, Let the objective function of the microscopic actor be... The loss function for micro-actors. Indicates the vehicle observation status The mathematical expectation, The micro-policy network is represented in the first... Continuous control parameters output from each low-level simulation frame.

Images