On-Ramp scene trajectory planning method based on deep inverse reinforcement learning

Abstract

The invention relates to the technical field of intelligent traffic, in particular to an On-Ramp scene trajectory planning method based on deep inverse reinforcement learning, and the method comprises the following steps: S100, candidate trajectory generation: generating a plurality of candidate trajectories based on vehicle state information and traffic environment information; s200, track authenticity filtering, wherein the generated candidate tracks are evaluated and screened through a track authenticity filtering framework, so that abnormal tracks are eliminated, and credible tracks are reserved; and S300, reward function training: utilizing reward function optimization of deep inverse reinforcement learning to realize continuous adjustment and improvement of a trajectory evaluation standard so as to carry out quantitative analysis on rationality and optimality of the trajectory. By constructing a trajectory authenticity filtering framework, optimizing a reward function design and training strategy, and introducing a multi-dimensional scoring mechanism, the trajectory authenticity filtering framework is optimized, so that the accuracy of the trajectory authenticity filtering framework is improved. According to the invention, efficient, safe and dynamically adaptive trajectory planning is realized, and reliable support is provided for real-time decision making of an automatic driving vehicle in a complex scene.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and specifically, to an On-Ramp scenario trajectory planning method based on deep inverse reinforcement learning. Background Art

[0002] Trajectory planning aims to generate safe, smooth, and efficient driving paths for autonomous vehicles and is a core technology for realizing intelligent transportation and autonomous driving. However, existing trajectory planning methods face many technical challenges in complex dynamic traffic scenarios. Especially in the highway ramp merging (On-Ramp) scenario, the limitations of traditional methods are particularly obvious.

[0003] In recent years, deep learning and reinforcement learning technologies have been introduced into the field of trajectory planning. For example, the trajectory planning method based on DIRL learns driving behavior patterns in a data-driven manner, and such methods overcome some limitations of traditional methods in the highway ramp merging scenario.

[0004] However, in the existing technology, the existing DIRL-based trajectory planning methods still have problems such as trajectory data quality issues, high computational cost of the partition function, and incomplete trajectory scoring criteria. Noise or outliers in the trajectory data may affect the accuracy of model training, and the high computational complexity of the partition function limits the training efficiency of the model. In addition, existing methods often only focus on a single optimization objective and it is difficult to comprehensively consider multi-dimensional scoring criteria. Summary of the Invention

[0005] The purpose of the present invention is to provide an On-Ramp scenario trajectory planning method based on deep inverse reinforcement learning to solve the problems mentioned in the above background art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] An On-Ramp scenario trajectory planning method based on deep inverse reinforcement learning includes the following steps:

[0008] S100. Candidate trajectory generation: Generate multiple candidate trajectories based on vehicle state information and traffic environment information;

[0009] S200. Trajectory authenticity filtering: Evaluate and screen the generated candidate trajectories through a trajectory authenticity filtering framework to eliminate abnormal trajectories and retain credible trajectories;

[0010] S300. Reward function training: Use the optimization of the reward function of deep inverse reinforcement learning to continuously adjust and improve the trajectory evaluation criteria, so as to quantitatively analyze the rationality and optimality of the trajectories;

[0011] S400, Trajectory Scoring and Selection: Score candidate trajectories based on a reward function and select the optimal trajectory for vehicle control.

[0012] Preferably, in the step S100, candidate trajectory generation includes the following steps:

[0013] S101, Input vehicle state and traffic environment information, as well as traffic environment information. The collected information is directly uploaded by the edge device to the trajectory generation module and used as the initial input conditions for trajectory calculation;

[0014] S102, Dynamically set motion constraints and traffic rules. According to the vehicle's dynamic characteristics and the current traffic environment, the system sets the constraint conditions for trajectory generation. At the same time, in combination with traffic rules such as road width and obstacle positions, set the feasible region of the trajectory to ensure the safety and feasibility of the trajectory;

[0015] S103, Generate candidate trajectories. The system uses an improved dynamic window method algorithm to generate candidate trajectories. Based on considering dynamic constraints and traffic restrictions, this algorithm generates multiple trajectories that meet driving logic;

[0016] S104, Output the candidate trajectory set. The generated candidate trajectory set is stored and transferred to the subsequent authenticity filtering module for processing.

[0017] Preferably, in the step S104, the generated candidate trajectory set is accompanied by detailed dynamic and environmental parameters to ensure that subsequent processing can comprehensively evaluate the trajectories.

[0018] Preferably, in the step S200, the steps for the trajectory authenticity filtering framework to evaluate and screen the generated candidate trajectories are as follows;

[0019] S201, Preprocess the candidate trajectory set output in step S100 to ensure the integrity and consistency of the input data;

[0020] S202, Calculate the geometric characteristics of each candidate trajectory;

[0021] S203, After the geometric characteristic analysis is completed, calculate the dynamic characteristics of the trajectory;

[0022] S204, After extracting the geometric and dynamic characteristics, input these characteristics into a meta-learning-based scoring model. Through training with a large amount of real driving data, this model can quantitatively score the authenticity of the trajectories;

[0023] S205, Compare the authenticity score of each trajectory with a set threshold and eliminate abnormal trajectories below the threshold;

[0024] S206, Output the filtered set of credible trajectories.

[0025] Preferably, in step S206, each output trajectory is accompanied by detailed geometric and dynamic characteristic data and marked with a authenticity score.

[0026] Preferably, in step S300, the optimization of the reward function includes the following steps:

[0027] S301. Input expert trajectory data and candidate trajectory data;

[0028] S302. Initialize the reward function. Based on the three core objectives of safety, efficiency, and comfort, initialize the basic structure of the reward function, and construct an initial reward function framework by quantitatively analyzing these three characteristics of each trajectory.

[0029] S303. Generate a trajectory strategy and compare it with the expert trajectory. Based on the current reward function, generate a trajectory strategy corresponding to the candidate trajectory.

[0030] Step S304. Optimize the reward function. The system uses sampling and Softmax approximation methods to optimize the reward function. In the sampling stage, randomly select a subset from the candidate trajectory set and calculate its behavior difference distribution.

[0031] S305. Check the convergence of the optimized reward function. Determine whether the current function can stably and effectively evaluate the rationality of the trajectory. The convergence criteria include the stability of the reward function after multiple iterations and the consistency of the evaluation results. If the reward function does not meet the convergence criteria, return to step S304 to continue optimization; if it meets the criteria, proceed to the next step.

[0032] S306. Output the optimized reward function.

[0033] Preferably, in step 304, in the Softmax approximation, the system improves the numerical stability and calculation efficiency of the reward function by optimizing the calculation of the partition function. At the same time, dynamically adjust the weight parameters of the reward function according to the evaluation results to enable it to better fit the characteristics of the expert trajectory.

[0034] Preferably, in step S400, the trajectory scoring and selection include the following steps:

[0035] S401. Comprehensive score calculation. Use the trained reward function or scoring model to calculate the comprehensive score for each candidate trajectory.

[0036] S402. Output the trajectory score distribution. Statistically analyze the comprehensive scoring results of the candidate trajectories and output the score distribution and reward value of each trajectory.

[0037] S403. Select the optimal trajectory. Compare the comprehensive scores of all candidate trajectories and select the trajectory with the highest score as the optimal trajectory.

[0038] S404. Output the optimal trajectory. Take the trajectory with the highest score as the final output and provide it to the vehicle decision-making module for generating control instructions.

[0039] A trajectory planning method for On-Ramp scenarios based on deep inverse reinforcement learning further includes the following steps:

[0040] S500. Monitor and optimize the execution effect of the optimal trajectory through a dynamic feedback mechanism to ensure the efficiency and adaptability of the trajectory in practical applications and achieve the overall closed-loop optimization.

[0041] Preferably, in the step S500, the dynamic feedback and closed-loop optimization include the following steps:

[0042] S501. Collect traffic environment and vehicle state information in real time. Collect the current traffic environment information and vehicle state information in real time.

[0043] S502. Dynamically adjust motion constraints and traffic rules. Dynamically adjust the original motion constraint conditions and traffic rules according to the traffic data collected in real time.

[0044] S503. Dynamically adjust weights. Dynamically adjust the weights of safety, efficiency, and comfort in trajectory evaluation according to scenario requirements.

[0045] S504. Generate new candidate trajectories, perform scoring and selection. Under the dynamically adjusted constraint conditions and weights, regenerate candidate trajectories and conduct comprehensive scoring and screening on them.

[0046] S505. Output the optimal trajectory and feedback the execution effect. Output the optimally optimized optimal trajectory for the vehicle to execute, and at the same time monitor the execution result in real time. Through the monitoring data, evaluate whether the trajectory needs further optimization and adjust the original algorithm and model parameters to form a closed-loop optimization mechanism.

[0047] Advantages of the present invention:

[0048] First, by constructing a trajectory authenticity filtering framework, optimizing the reward function design and training strategy, and introducing a multi-dimensional scoring mechanism, the present invention realizes efficient, safe, and dynamically adaptable trajectory planning, providing reliable support for real-time decision-making of autonomous vehicles in complex scenarios.

[0049] II. The present invention covers the entire process from data collection to trajectory generation and optimization. First, traffic environment information and vehicle status information are collected in real time through multi-source sensor devices (such as cameras, millimeter-wave radars, and vehicle-mounted sensors). After preprocessing, this information provides accurate input support for the trajectory generation module. The trajectory generation module uses the vehicle's dynamic model and the constraints of the traffic environment to generate a set of candidate trajectories containing time-series dynamic parameters. The generation of candidate trajectories comprehensively considers complex factors in the dynamic traffic environment, including road geometry, lane boundaries, signal light states, and the behavioral characteristics of traffic participants, ensuring the diversity and applicability of the generated trajectories.

[0050] III. The present invention designs a trajectory authenticity filtering framework to comprehensively evaluate and screen the set of candidate trajectories to improve the reliability of trajectory data. The authenticity filtering process includes a comprehensive analysis of geometric features and dynamic features. Among them, the geometric feature extraction step analyzes the length, curvature distribution, and two-dimensional coverage area of the trajectory to measure the physical rationality of the trajectory; the dynamic feature analysis step calculates the rate of change of speed, acceleration smoothness, and direction adjustment frequency of the trajectory to ensure that the trajectory meets the requirements of driving smoothness. Through a weighted scoring model, the filtering framework scores the authenticity of each candidate trajectory and eliminates abnormal trajectories with scores lower than the threshold. This design significantly reduces the negative impact of abnormal data on subsequent model training and trajectory scoring, improving the quality and reliability of trajectory planning from the source.

[0051] IV. The present invention makes several improvements to the design and training strategy of the reward function. First, the design of the reward function comprehensively considers three dimensions: safety, efficiency, and comfort. The safety index quantifies the minimum distance between the trajectory and obstacles or other vehicles; the efficiency index measures the driving time and energy consumption; the comfort index reflects the smoothness of the trajectory and the degree of acceleration change. In addition, the present invention jointly trains the reward function through expert trajectories and unfiltered candidate trajectories, enabling the model to simultaneously learn ideal driving behaviors and diverse features in complex scenarios. To further reduce the training complexity, a Softmax approximation method based on importance sampling is introduced to optimize the calculation of the partition function, significantly improving the training efficiency and stability of the model to enhance the training efficiency and generalization ability of the model.

[0052] V. Based on trajectory authenticity filtering and reward function optimization, the present invention comprehensively evaluates the credible trajectory set through a multi-dimensional scoring system and selects the trajectory with the highest score as the output. The scoring system includes three major dimensions: safety, efficiency, and comfort. The safety score is quantitatively evaluated based on the safety distance between the trajectory and the surrounding environment and the collision risk; the efficiency score analyzes the traffic efficiency of the trajectory, including average speed, travel time, and fuel economy; the comfort score is evaluated through the smoothness and dynamic change characteristics of the trajectory. The scoring results are comprehensively calculated through a weighted summation model, and the weights are dynamically adjusted according to the actual scenario requirements to meet the requirements of different planning objectives.

[0053] VI. The present invention designs a dynamic feedback mechanism to optimize the trajectory generation rules and scoring criteria in real time. Traffic environment data is collected in real time through multi-source sensors, and the trajectory generation constraint conditions and scoring criteria are adjusted according to the latest information to achieve closed-loop optimization. The dynamic feedback mechanism ensures that the system can quickly respond to emergencies (such as traffic jams, accidents, etc.) and correct the output trajectory in real time, maintaining the efficiency and robustness of trajectory planning to enhance the system's adaptability to the dynamic traffic environment.

[0054] VII. The present invention improves the reliability of trajectory data through the trajectory authenticity filtering framework, realizes accurate modeling of complex dynamic environments by combining deep inverse reinforcement learning technology, and optimizes the trajectory selection strategy through a multi-dimensional scoring system. In addition, the dynamic feedback mechanism significantly enhances the system's adaptability to environmental changes, enabling trajectory planning to maintain stable and efficient performance in complex scenarios.

[0055] VIII. The present invention is applicable to the on-ramp merging of highways, the traffic optimization of complex urban intersections, and the path planning module of autonomous driving systems. Through real-time and efficient trajectory planning methods, the present invention provides strong technical support for the efficient operation of intelligent transportation systems and has broad application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts;

[0057] Figure 1 is a flowchart of the On-Ramp scenario trajectory planning method based on deep inverse reinforcement learning provided in Embodiment 1 of the present invention;

[0058] Figure 2 is a schematic flow diagram of candidate trajectory generation in Embodiment 1 of the present invention;

[0059] Figure 3 It is a schematic flowchart of trajectory authenticity filtering in the first embodiment of the present invention;

[0060] Figure 4 It is a flowchart of training the reward function based on deep inverse reinforcement learning in the first embodiment of the present invention;

[0061] Figure 5 It is a flowchart of trajectory scoring and optimal trajectory selection in the first embodiment of the present invention;

[0062] Figure 6 It is a flowchart of the On-Ramp scenario trajectory planning method based on deep inverse reinforcement learning provided in the second embodiment of the present invention;

[0063] Figure 7 It is a flowchart of the operation of the dynamic feedback and closed-loop optimization module in the second embodiment of the present invention. Detailed implementation manners

[0064] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0065] Embodiment 1

[0066] The On-Ramp scenario is a typical dynamic traffic scenario, characterized by the need for main road and ramp vehicles to complete complex merging operations within a limited space. The technical challenges of this scenario are mainly reflected in the following aspects: First, it is highly dynamically complex, with the speeds, accelerations, and positions of main road and ramp vehicles constantly changing, and trajectory planning needs to be updated in real time to adapt to the dynamic changes of the environment. Second, it is a multi-agent interaction problem. During the merging process, the interaction between main road vehicles and ramp vehicles is significant, and the behaviors and intentions of all vehicles need to be comprehensively considered. Finally, it is a trade-off between safety and efficiency. Trajectory planning not only needs to ensure the safety of vehicles but also minimize the impact on traffic flow and improve the overall traffic efficiency.

[0067] As Figure 1 shown, the embodiments of the present invention provide a trajectory planning implementation framework based on deep inverse reinforcement learning in the On-Ramp scenario. The framework includes steps S100 - S400, which are introduced in detail as follows:

[0068] This method starts with step S100, where the end layer ((camera, lidar, dual-frequency RTK-GNSS, in-vehicle IMU, etc.) is responsible for generating multiple candidate trajectories according to the traffic environment and vehicle state.

[0069] In some embodiments, Figure 2 As shown, the terminal layer is responsible for real-time collection of vehicle status and traffic environment information and trajectory generation, specifically including:

[0070] Step S101, input vehicle status and traffic environment information, collect vehicle status information (such as position, speed, acceleration) and traffic environment information (such as road width, traffic signs, obstacle positions) from multiple data sources (camera, lidar, dual-frequency RTK-GNSS, automotive-grade IMU). The collected data is directly uploaded to the trajectory generation module by the end-layer device (end-layer devices include cameras, lidar, dual-frequency RTK-GNSS, automotive-grade IMU) and used as the initial input conditions for trajectory calculation.

[0071] Step S102, dynamically set motion constraints and traffic rules, and set the constraints for trajectory generation according to the vehicle's dynamic characteristics and the current traffic environment, including maximum acceleration, minimum turning radius and speed range. At the same time, set the feasible area of the trajectory in combination with traffic rules such as road width and obstacle location to ensure the safety and feasibility of the trajectory.

[0072] Step S103, generating candidate trajectories, using an improved dynamic window algorithm (DWA) to generate candidate trajectories. The algorithm generates multiple trajectories that meet driving logic based on dynamic constraints and traffic restrictions. Each trajectory includes detailed time series data (such as position information, speed and direction at a time point) to cover a variety of driving strategies (such as avoidance, acceleration, etc.).

[0073] Step S104, outputting a candidate trajectory set, the generated candidate trajectory set is stored and passed to the subsequent authenticity filtering module for processing. The generated candidate trajectory set is accompanied by detailed dynamic and environmental parameters (such as vehicle speed, acceleration and minimum distance to obstacles) to ensure that subsequent processing can fully evaluate the trajectory.

[0074] Step S200 , evaluating and screening the generated candidate trajectories through a trajectory authenticity filtering framework to eliminate abnormal trajectories and retain credible trajectories.

[0075] The trajectory authenticity filtering framework dynamically screens out trajectories that meet both safety constraints and anthropomorphic driving behaviors by performing spatiotemporal segmentation and hybrid feature extraction on the candidate trajectory set, combined with a dual-channel meta-learning scorer based on TCN+Transformer.

[0076] In some embodiments, Figure 3 As shown in FIG. 1 , the trajectory authenticity filtering framework comprehensively analyzes the geometric and dynamic characteristics of the trajectory, specifically including:

[0077] Step S201, preprocess the candidate trajectory set output in step S100 (including time normalization, coordinate transformation, outlier removal, and feature standardization). The output candidate trajectory set includes detailed time series data (such as position, speed, direction) and dynamic parameters (such as acceleration, steering angle) to ensure the integrity and consistency of the input data for the subsequent analysis to proceed smoothly.

[0078] Step S202, calculate the geometric properties for each candidate trajectory. The geometric properties to be calculated include: (1) Trajectory length: Calculate the overall path length of the trajectory through integration to ensure that the trajectory does not exceed a reasonable range. (2) Curvature distribution: Analyze the curvature change of the trajectory to identify whether there are abnormal sharp turns or overly straight paths. (3) Coverage area: Evaluate the coverage range of the trajectory on the two-dimensional plane to ensure that the trajectory is generated within a reasonable area. The geometric properties provide an important quantitative basis for the preliminary screening of the trajectory.

[0079] After the geometric property analysis in step S203, further calculate the dynamic properties of the trajectory to evaluate its physical rationality. The calculated dynamic properties of the trajectory include: (1) Rate of change of speed: Analyze whether the speed change of the trajectory is stable and detect abnormal behaviors such as sudden acceleration or deceleration. (2) Smoothness of acceleration: Calculate the continuity and volatility of the acceleration in the trajectory to ensure compliance with physical laws. (3) Frequency of direction adjustment: Count the frequency of direction changes in the trajectory to identify possible unreasonable steering behaviors. The dynamic property analysis can effectively eliminate abnormal trajectories that do not conform to the actual driving logic.

[0080] Step S204, calculate the authenticity score using a scoring model. After extracting the geometric and dynamic properties, input these properties into a meta-learning-based scoring model. This model can quantitatively score the authenticity of the trajectory through training on a large amount of real driving data. The higher the score of the trajectory, the stronger its rationality; conversely, trajectories with lower scores may be abnormal.

[0081] Step S205, eliminate abnormal trajectories. Compare the authenticity score of each trajectory with a set threshold and eliminate abnormal trajectories with scores lower than the threshold. The eliminated trajectories may exhibit unreasonable geometric or dynamic properties, such as rapidly changing speed, excessive steering frequency, or abnormal curvature distribution. This step ensures that the selected trajectories have a high level of credibility.

[0082] Step S206, output the set of credible trajectories. After filtering, output the filtered set of credible trajectories. Each trajectory is accompanied by detailed geometric and dynamic property data and is labeled with an authenticity score. These high-quality credible trajectories will be used as the input for the subsequent trajectory scoring and selection module to further improve the effect of trajectory optimization.

[0083] Step S300: By optimizing the reward function using deep inverse reinforcement learning, the trajectory evaluation criteria are continuously adjusted and improved, thereby quantitatively analyzing the rationality and optimality of the trajectory. Deep Inverse Reinforcement Learning (DIRL) is a method that simulates the human decision-making process by learning the potential reward function in expert behavior and has unique advantages in the field of trajectory planning. DIRL trains the reward function through expert trajectory data, can generate trajectories that better conform to human driving habits, and has strong adaptability to dynamic environments, and can dynamically adjust the trajectory planning strategy according to the real-time environment. In addition, the DIRL method also has the ability to optimize multiple objectives and can optimize the safety, efficiency, and comfort of the trajectory simultaneously, thus achieving a multi-dimensional balance.

[0084] In some embodiments, as Figure 4 shown, the reward function optimization includes the following steps:

[0085] Step S301: Input expert trajectory data and candidate trajectory data. Expert trajectories reflect the ideal behavior of human driving, while candidate trajectories are generated by the trajectory generation module. The input data includes detailed time series information (such as position, speed, acceleration) and environmental constraint conditions (such as obstacles, traffic rules). Preprocess the input expert trajectory data and candidate trajectory data (including time normalization, coordinate transformation, outlier removal, feature standardization) to ensure the consistency of format and accuracy.

[0086] Step S302: Initialize the reward function. Based on the three core objectives of safety, efficiency, and comfort, initialize the basic structure of the reward function. The safety metric includes the minimum distance from other vehicles and obstacles; the efficiency metric reflects the rationality of travel time and speed; and comfort is related to the smoothness of acceleration and the frequency of direction adjustment. By quantitatively analyzing these three characteristics of each trajectory, construct an initial reward function framework.

[0087] Step S303: Generate a trajectory strategy and compare it with the expert trajectory. Based on the current reward function, generate a trajectory strategy corresponding to the candidate trajectory. These strategies describe the specific movement behaviors of the vehicle at different time points. By comparing the generated trajectory strategy with the expert trajectory point by point, calculate the behavior differences between the two, and identify the deficiencies in the candidate trajectory. These differences will provide a basis for subsequent reward function optimization.

[0088] Step S304: Optimize the reward function. Use sampling and Softmax approximation methods to optimize the reward function. In the sampling stage, randomly select subsets from the candidate trajectory set and calculate their behavior difference distributions. In Softmax approximation, improve the numerical stability and computational efficiency of the reward function through the optimization calculation of the partition function. At the same time, dynamically adjust the weight parameters of the reward function according to the evaluation results so that it can better fit the characteristics of the expert trajectory.

[0089] Step S305: Check the convergence of the reward function. Check the convergence of the optimized reward function to determine whether the current function can stably and effectively evaluate the rationality of the trajectory. The convergence criteria include the stability of the reward function after multiple iterations and the consistency of the evaluation results. If the reward function does not meet the convergence criteria, return to Step S304 and continue to optimize; if it meets the criteria, proceed to the next step.

[0090] Step S306: Output the optimized reward function. Finally, output the optimized reward function, which can comprehensively evaluate the safety, efficiency, and comfort of the trajectory. The optimized reward function is passed to the trajectory scoring and selection module for subsequent comprehensive trajectory scoring and optimal trajectory selection. The optimization process of the reward function effectively improves the accuracy of trajectory evaluation and provides strong support for achieving high-quality driving behavior decisions.

[0091] Step S400: Select the optimal trajectory from the candidate trajectories generated through comprehensive evaluation and output it to the vehicle decision-making module for final driving control. In some embodiments, as Figure 5 shown, the trajectory scoring and selection include the following specific steps:

[0092] Step S401: Calculate the comprehensive score. Use the trained reward function or scoring model to calculate the comprehensive score for each candidate trajectory. The scoring takes into account multi-dimensional weights, such as safety (whether the distance from obstacles is sufficient), efficiency (the time consumed to pass the current section), and comfort (whether the acceleration change is smooth). The model optimizes the weight for each dimension based on historical data to ensure that the score can comprehensively reflect the quality of the trajectory.

[0093] Step S402: Output the trajectory score distribution. Statistically analyze the comprehensive scoring results of the candidate trajectories and output the score distribution and reward value of each trajectory. The score distribution provides the relative comparison results between the candidate trajectories, while the reward value quantitatively reflects the specific performance of the trajectory in each evaluation dimension. The output results will be used as the direct basis for trajectory selection.

[0094] Step S403: Select the optimal trajectory. Compare the comprehensive scores of all candidate trajectories and select the trajectory with the highest score as the optimal trajectory. The selection of the trajectory is not only based on a single total score value, but also comprehensively considers the scoring balance of each dimension to ensure that the selected trajectory can perform well in actual driving. For example, it can ensure safety while also taking into account efficiency and comfort.

[0095] Step S404: Output the optimal trajectory. Take the trajectory with the highest score as the final output and provide it to the vehicle decision-making module for generating control instructions. The output trajectory includes detailed time series information (such as acceleration, direction adjustment, and position change) so that the vehicle can execute in real time. In addition, the selection result of the optimal trajectory will be stored for subsequent optimization of the model, providing data support for continuous improvement.

[0096] In summary, the advantages of the present invention are reflected in the following aspects:

[0097] 1. Efficient trajectory generation and optimization. The present invention collects traffic environment and vehicle state information in real time through end-layer devices, generates candidate trajectories by combining the improved Dynamic Window Approach (DWA) algorithm and multiple constraint conditions, and eliminates abnormal trajectories in the authenticity filtering module to ensure the accuracy and reliability of trajectory generation, meeting the requirements of real-time planning for autonomous driving.

[0098] 2. Intelligent evaluation and learning ability. The present invention uses a reward function based on deep inverse reinforcement learning to train the model, comprehensively scores candidate trajectories in multiple dimensions, covering safety, efficiency, and comfort, improving the intelligent level of trajectory selection, and significantly optimizing driving decisions.

[0099] 3. Strong multi-scenario adaptability. By iteratively training the reward function by combining expert trajectories and candidate trajectories, the model of the present invention can be flexibly applied in complex scenarios, covering various traffic situations such as highways and ramp merges, improving the wide applicability of the model.

[0100] 4. Wide application in traffic optimization. The present invention can be applied not only to the trajectory planning of autonomous vehicles, but also to intelligent transportation fields such as traffic signal control, path optimization, and emergency response, providing efficient technical support and decision-making basis for smart cities. The present invention significantly reduces the complexity and cost of autonomous driving trajectory planning, improves the intelligent and collaborative level, and has broad promotion and application potential.

[0101] Embodiment 2

[0102] A trajectory planning method for On-Ramp scenarios based on deep inverse reinforcement learning further includes the following steps:

[0103] Step S500, monitor and optimize the execution effect of the optimal trajectory through a dynamic feedback mechanism to ensure the efficiency and adaptability of the trajectory in actual applications, and at the same time achieve the overall closed-loop optimization. In some embodiments, such as Figure 6 shown, the dynamic feedback and closed-loop optimization include the following specific steps:

[0104] Step S501, collect traffic environment and vehicle state information in real time, collect current traffic environment information (such as road conditions, surrounding vehicle states, signal light states) and vehicle state information (such as vehicle position, speed, acceleration) in real time to ensure that the latest traffic conditions and vehicle operation parameters can be quickly captured. The data is uploaded to the cloud or edge computing module through sensors, monitoring cameras and in-vehicle devices, providing basic information for subsequent feedback.

[0105] Step S502, dynamically adjust motion constraints and traffic rules. According to the traffic data collected in real time, dynamically adjust the original motion constraint conditions and traffic rules. For example, when the road is congested, reduce the maximum speed of the vehicle; in open areas, expand the range of drivable paths. Through flexible constraint adjustment, ensure that the trajectory planning result can be highly adapted to the current traffic scenario.

[0106] Step S503, dynamic weight adjustment. According to the scenario requirements, dynamically adjust the weights of safety, efficiency and comfort in trajectory evaluation. For example, in highway scenarios, the efficiency weight may be given priority; at complex intersections, more emphasis is placed on the safety weight. Dynamic weight adjustment can flexibly respond to different scenarios and improve the actual effect of trajectory planning.

[0107] Step S504, generate new candidate trajectories, score and select. Under the dynamically adjusted constraint conditions and weights, regenerate candidate trajectories and comprehensively score and screen them. The generated candidate trajectories are calculated by an optimization algorithm (such as the improved Dynamic Window Approach DWA) to ensure that they meet the dynamic requirements of the current scenario, and at the same time output the best candidate trajectory for vehicle execution.

[0108] Step S505, output the optimal trajectory and feedback the execution effect. Output the optimally optimized optimal trajectory for vehicle execution, and at the same time monitor the execution result in real time. The monitoring includes the actual deviation of the trajectory, the vehicle's response speed to the trajectory, and the record of abnormal behaviors during the execution process. Through the monitoring data, evaluate whether the trajectory needs further optimization, and adjust the original algorithm and model parameters (when the L2 deviation, average speed, and longitudinal acceleration variance between the actual trajectory and the planned trajectory exceed any threshold, optimize; online update the safety and efficiency weight coefficients through the Lyapunov optimizer), forming a closed-loop optimization mechanism.

[0109] In summary, the advantages of the second embodiment of the present invention are:

[0110] The present invention has an innovative feedback closed-loop mechanism. Through the dynamic feedback and closed-loop optimization module, the present invention can adjust the trajectory planning parameters in a real-time traffic environment, dynamically optimize the safety and efficiency weights, form an adaptive optimization mechanism, and enhance the robustness and reliability of the system.

[0111] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed.

Claims

1. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning, characterized in that, It includes the following steps: S100. Candidate trajectory generation: Generate multiple candidate trajectories based on vehicle state information and traffic environment information; S200. Trajectory authenticity filtering: Evaluate and screen the generated candidate trajectories through a trajectory authenticity filtering framework to eliminate abnormal trajectories and retain credible trajectories; S300. Reward function training: Utilize the optimization of the reward function of deep inverse reinforcement learning to continuously adjust and improve the trajectory evaluation criteria, thereby quantitatively analyzing the rationality and optimality of the trajectories; S400. Trajectory scoring and selection: Score the candidate trajectories based on the reward function and select the optimal trajectory for vehicle control.

2. The trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 1, wherein, In the step S100, the candidate trajectory generation includes the following steps: S101. Input vehicle state and traffic environment information, as well as traffic environment information. The collected information is directly uploaded by the end-layer device to the trajectory generation module and used as the initial input conditions for trajectory calculation; S102. Dynamically set motion constraints and traffic rules. According to the dynamic characteristics of the vehicle and the current traffic environment, the system sets the constraint conditions for trajectory generation. At the same time, in combination with traffic rules such as road width and obstacle positions, the feasible region of the trajectory is set to ensure the safety and feasibility of the trajectory; S103. Generate candidate trajectories. The system uses an improved dynamic window method algorithm to generate candidate trajectories. This algorithm generates multiple trajectories that meet the driving logic while considering dynamic constraints and traffic restrictions; S104. Output the candidate trajectory set. The generated candidate trajectory set is stored and transmitted to the subsequent authenticity filtering module for processing.

3. The on-ramp scenario trajectory planning method based on deep inverse reinforcement learning according to claim 2, wherein In the step S104, the generated candidate trajectory set is accompanied by detailed dynamic and environmental parameters to ensure that subsequent processing can comprehensively evaluate the trajectories.

4. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 1, wherein In the step S200, the steps for the trajectory authenticity filtering framework to evaluate and screen the generated candidate trajectories are as follows; S201. Preprocess the candidate trajectory set output in the step S100 to ensure the integrity and consistency of the input data; S202. Calculate the geometric characteristics of each candidate trajectory; S203. After the geometric characteristic analysis is completed, calculate the dynamic characteristics of the trajectory; S204. After extracting the geometric and dynamic characteristics, input these characteristics into a meta-learning-based scoring model. This model can quantitatively score the authenticity of the trajectories through training with a large amount of real driving data; S205. Compare the authenticity score of each trajectory with the set threshold, and eliminate abnormal trajectories below the threshold; S206. Output the screened credible trajectory set.

5. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 4, characterized in that, In the step S206, each output trajectory is accompanied by detailed geometric and dynamic characteristic data and is marked with the authenticity score.

6. The on-ramp scenario trajectory planning method based on deep inverse reinforcement learning according to claim 1, characterized in that, In the step S300, the reward function optimization includes the following steps: S301. Input expert trajectory data and candidate trajectory data; S302. Initialize the reward function. Based on the three core objectives of safety, efficiency, and comfort, initialize the basic structure of the reward function. By quantitatively analyzing these three characteristics of each trajectory, construct the initial reward function framework; S303. Generate a trajectory strategy and compare it with the expert trajectory. Based on the current reward function, generate a trajectory strategy corresponding to the candidate trajectory. Step S304. Optimize the reward function. The system optimizes the reward function using sampling and Softmax approximation methods. In the sampling stage, randomly select a subset from the candidate trajectory set and calculate its behavior difference distribution. S305. Check the convergence of the optimized reward function. Determine whether the current function can stably and effectively evaluate the rationality of the trajectory. The convergence criteria include the stability of the reward function after multiple iterations and the consistency of the evaluation results. If the reward function does not meet the convergence criteria, return to step S304 and continue to optimize. If it meets the criteria, proceed to the next step. S306. Output the optimized reward function.

7. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 6, characterized in that In step 304, in the Softmax approximation, the system improves the numerical stability and computational efficiency of the reward function through the optimized calculation of the partition function. At the same time, dynamically adjust the weight parameters of the reward function according to the evaluation results to make it better fit the characteristics of the expert trajectory.

8. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 1, characterized in that, In step S400, trajectory scoring and selection include the following steps: S401. Comprehensive score calculation. Use the trained reward function or scoring model to calculate the comprehensive score for each candidate trajectory. S402. Output the trajectory score distribution. Conduct statistical analysis on the comprehensive scoring results of the candidate trajectories and output the score distribution and reward value of each trajectory. S403. Select the optimal trajectory. Compare the comprehensive scores of all candidate trajectories and select the trajectory with the highest score as the optimal trajectory. S404. Output the optimal trajectory. Take the trajectory with the highest score as the final output and provide it to the vehicle decision-making module for generating control instructions.

9. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 1, characterized in that It also includes the following steps: S500. Monitor and optimize the execution effect of the optimal trajectory through a dynamic feedback mechanism to ensure the efficiency and adaptability of the trajectory in actual applications and achieve the overall closed-loop optimization.

10. A trajectory planning method for the On-Ramp scenario based on deep inverse reinforcement learning according to claim 9, characterized in that In step S500, dynamic feedback and closed-loop optimization include the following steps: S501. Real-time collect traffic environment and vehicle state information. Real-time collect the current traffic environment information and vehicle state information. S502. Dynamically adjust motion constraints and traffic rules. Dynamically adjust the original motion constraint conditions and traffic rules according to the real-time collected traffic data. S503. Dynamic weight adjustment. Dynamically adjust the weights of safety, efficiency, and comfort in trajectory evaluation according to the scenario requirements. S504. Generate new candidate trajectories, conduct scoring and selection. Under the dynamically adjusted constraint conditions and weights, regenerate candidate trajectories and conduct comprehensive scoring and screening on them. S505. Output the optimal trajectory and feedback the execution effect. Output the dynamically optimized optimal trajectory for the vehicle to execute. At the same time, conduct real-time monitoring on the execution results. Through the monitoring data, evaluate whether the trajectory needs further optimization and adjust the original algorithm and model parameters to form a closed-loop optimization mechanism.

Images