A ramp merging decision method based on knowledge data driving

Abstract

This invention discloses a knowledge-data-driven highway ramp merging decision-making method. After establishing a highway ramp merging environment and a reinforcement learning decision-making model environment, a proximal policy optimization algorithm is used to train and iterate the decision-making model. During training, a multi-objective reward function is used to calculate the immediate reward, and a total loss function including a physical information loss function and a main loss function is used to optimize the model parameters. After training, the current state information is input into the decision-making model, and this information, along with a structured scene semantic description and candidate actions output by the decision-making model, forms input prompts. A large language model is used to perform safety checks on the candidate actions to obtain a safety judgment result, and then either a candidate action or a conservative action is executed. The trained decision-making model and the large language model are deployed in the highway ramp merging scenario to achieve merging decision control. This invention can effectively reduce collision risk, improve merging success rate, decision stability, and driving comfort.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control, and more specifically to a knowledge-data-driven method for highway ramp merging decision-making. Background Technology

[0002] Highway entrance ramps are important transition areas for vehicles from side roads to the main road traffic flow. As main road vehicles and ramp vehicles interweave in a limited space, problems such as speed fluctuations, traffic conflicts, and congestion spread can easily occur, seriously affecting road traffic efficiency and traffic safety.

[0003] Existing ramp merging control methods mainly include model-driven methods based on traffic flow theory and rule-based control methods. Model-driven methods are typically based on traffic flow dynamics models and achieve traffic flow regulation through parameter calibration and feedback control, exhibiting good interpretability under certain conditions. However, these methods often rely on idealized model assumptions and have limited adaptability to changes in the traffic environment and uncertainties in driving behavior, making it difficult to maintain stable performance in complex dynamic traffic scenarios.

[0004] With the development of intelligent connected vehicle technology, reinforcement learning methods are increasingly being applied to the field of autonomous driving decision-making. Reinforcement learning learns optimal policies through interaction with the environment, enabling adaptive decision-making in complex interactive scenarios, and is particularly suitable for driving tasks involving continuous states and action spaces. However, in highway ramp merging scenarios, reinforcement learning methods still have certain limitations. On the one hand, their decision-making process often lacks explicit physical constraints, easily leading to behaviors that do not conform to vehicle kinematics or pose safety hazards. On the other hand, reinforcement learning strategies heavily rely on reward function design, resulting in problems such as unstable training, poor policy convergence, and insufficient interpretability, making it difficult to meet the application requirements of high-safety scenarios.

[0005] In recent years, large language models have demonstrated good performance in complex semantic understanding, knowledge reasoning, and long-tail scenario processing. Some studies have attempted to introduce them into autonomous driving decision-making for risk assessment or high-level decision support. However, large language models are usually difficult to directly handle low-level continuous control problems, and their outputs are also difficult to directly meet real-time control requirements. Therefore, they are difficult to apply alone to high-dynamic, high-risk scenarios such as ramp merging.

[0006] Furthermore, existing technologies also include solutions for applying large language models to highway ramp traffic control. For example, some solutions use drones to collect traffic flow data in ramp areas, utilize large language models for multimodal data parsing, traffic report generation, fuzzy control rule adjustment, and control strategy interpretation, and combine this with a type-two fuzzy controller to output ramp signal control or green light delay control strategies. These methods are primarily geared towards roadside traffic management and macro-level traffic flow control, and their control objects are typically ramp signals, capacity, or traffic flow status, making them suitable for improving overall traffic efficiency.

[0007] However, the above methods do not model the vehicle-level real-time decision-making problem of autonomous vehicles merging into ramps, and cannot directly output specific driving actions. At the same time, their large language model is mainly used for traffic data parsing, rule generation, policy description or simulation configuration, and does not form an action-level safety review mechanism with the reinforcement learning policy output. Furthermore, it does not simultaneously integrate vehicle kinematic constraints, collision time constraints, merging gap constraints and comfort constraints into the training and inference phases.

[0008] Therefore, there is an urgent need for a ramp merging decision-making method that can combine data-driven reinforcement learning with knowledge-driven reasoning capabilities, thereby improving safety and interpretability while ensuring decision-making efficiency. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide a knowledge-data-driven highway ramp merging decision-making method that addresses the above-mentioned problems in the prior art. This method combines the flexibility of data-driven methods with the interpretability and safety of knowledge-driven methods, effectively reducing collision risks and improving ramp merging success rate, decision stability, and vehicle driving comfort.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: Establish a highway ramp merging environment and a reinforcement learning decision model environment. Extract state information representing the interaction between merging vehicles and main road vehicles in the ramp merging scenario to establish the state space of the reinforcement learning decision model environment. This ensures that the state space of the reinforcement learning decision model environment includes state information representing the interaction between the vehicle and surrounding vehicles when merging on the ramp. Construct a set of candidate actions to establish the action space of the reinforcement learning decision model environment. This ensures that the action space of the reinforcement learning decision model environment includes different candidate actions when merging on the ramp. Based on the state information, a Proximal Policy Optimization (PPO) decision model is constructed. This model uses an Actor network to calculate the probability distribution of each candidate action based on the normalized state information, and a Critic network to estimate the state value of the current state. It also obtains the state information of the highway ramp merging into the environment. The PPO decision model is trained and iterated using the Proximal Policy Optimization (PPO) algorithm. During the training process, a multi-objective reward function is used to calculate the immediate reward, and a physical information loss function is introduced as an auxiliary loss term to jointly establish the parameters of the total loss function optimization model with the main loss function, thus integrating physical constraint information into the PPO policy optimization process. During the inference phase, after training iterations, the current state information of the highway ramp merging environment is acquired and input into the decision model. Simultaneously, the current state information is converted into a structured scene semantic description. This structured scene semantic description, along with the candidate actions output by the decision model, is used to construct the input prompt for the large language model. Through structured scene semantic encoding, parameterized security threshold injection, and a multi-dimensional security inference and consensus fusion mechanism, the large language model performs security checks on the candidate actions output by the PPO decision model, obtaining corresponding security judgment results. Based on these security judgment results, either the candidate action or a preset conservative action is executed as the final merging decision action. The trained PPO decision model and the large language model are deployed together as a complete decision strategy in the highway ramp merging scenario to achieve merging decision control.

[0011] Furthermore, the highway ramp merging environment includes a two-lane main road and parallel ramps connecting to the main road, and divides the road into the main road initiation section, transition section, merging conflict section and termination section. The state information characterizing the interaction between the vehicle and surrounding vehicles when merging onto a ramp includes the vehicle's state information, the relative position and relative speed of surrounding vehicles, and safety indicator information. The safety indicator information includes at least the collision time and the distance between vehicles in front and behind.

[0012] in, This represents the longitudinal distance between the vehicle and the vehicle in front. The relative speed between the vehicle and the vehicle in front; Different candidate actions for merging onto a ramp include changing lanes to the left, maintaining the current position, changing lanes to the right, accelerating, and decelerating. The set of candidate actions... Defined as a discrete action space, the set includes left lane change, maintaining the current state, right lane change, acceleration, and deceleration:

[0013] in, Indicates a left lane change maneuver. This indicates an action that maintains the current state. Indicates a right lane change maneuver. Indicates an acceleration action. This indicates a deceleration action.

[0014] Furthermore, the state information is constructed using a kinematic feature matrix to create an observation space, which includes the state information of the vehicle itself and the N nearest surrounding vehicles; the mathematical representation of the observation space is as follows:

[0015] Where N is the number of surrounding vehicles observed, and M is the feature dimension of each vehicle; M feature dimensions are extracted for each vehicle, and the M feature dimensions include:

[0016] in, Vertical position The horizontal position For longitudinal velocity, For lateral velocity, As a marker of existence; The observation space, after being normalized, is input into the PPO decision model. The normalization formula is:

[0017] in, Let be the normalization constant for road length. Let be the normalization constant for road width. This is the normalization constant for maximum velocity; A relative coordinate system is used to characterize the relative motion relationships between vehicles in order to improve the generalization ability of the merged decision strategy.

[0018] Furthermore, the PPO decision model employs a multi-objective reward function. This multi-objective reward function is pruned before output to ensure numerical stability during training. The multi-objective reward function includes at least a successful merging reward, a collision penalty, a speed maintenance reward, a comfort penalty, and a driving progress reward. These reward components are weighted and summed to form the total reward function, which guides the PPO decision model in learning a ramp merging strategy that balances safety, traffic efficiency, and driving comfort. Its mathematical expression is as follows:

[0019] in, For collision penalties, To successfully import the reward item, To maintain the speed bonus, For comfort penalty items, This is a reward item for driving progress. , , , , These are the weighting coefficients for each item; The collision penalty item The calculation formula is:

[0020] The reward for successful import The calculation formula is:

[0021] in, To add a successful import reward coefficient; The speed maintenance reward item The calculation formula is:

[0022] in, Current vehicle speed For the target driving speed, This refers to the speed reward weighting coefficient. The comfort penalty item The calculation formula is:

[0023] in, The weight of the penalty for lane changing actions. Change the penalty weight for general actions; The driving progress reward item The calculation formula is:

[0024] in, This represents the longitudinal displacement increment per unit time. As a reference distance constant, This refers to the progress reward weighting coefficient. The total reward function The output is after cropping, and the cropping range is... This is to ensure numerical stability during the training process.

[0025] Furthermore, during training, based on trajectory data obtained from environmental interaction sampling, the policy network is updated using a pruning objective function, and the advantage function is calculated using a generalized advantage estimation method to reduce policy update oscillations and improve model convergence stability. Let the policy (Actor) network be The parameters are ; The Critic network is The parameters are The Actor and Critic networks employ a shared feature extraction layer architecture to improve training efficiency. The Rollout trajectory buffer is... The discount factor is The generalized dominance estimation parameters are: The cutting factor is The value loss coefficient is The entropy regularization coefficient is .

[0026] The pruning objective function The formula for limiting the policy update magnitude is as follows:

[0027] in, The strategy probability ratio This is the estimated value of the dominance function; The formula for calculating the strategy probability ratio is:

[0028] in, As the current strategy, The strategy before the update; the clip function constrains the strategy ratio to... Within a certain range, to prevent excessive changes in strategy.

[0029] The advantage function The Generalized Advantage Estimation (GAE) method is used for calculation, and its formula is as follows:

[0030] in, The timing difference error is calculated using the following formula:

[0031] in, For instant rewards, For state value estimation.

[0032] The value function loss The formula used to train the Critic network is as follows:

[0033] in, The target value is calculated using Monte Carlo returns or time-series difference objectives.

[0034] The entropy regularization term It is used to encourage policy exploration and prevent premature policy convergence; its calculation formula is as follows:

[0035] The total loss function of the PPO algorithm The weighted combination of each loss component is calculated using the following formula:

[0036] in, This is the value loss weighting coefficient. The entropy regularization weight coefficients are used to minimize the weight coefficients. For Actor network parameters and Critic network parameters A joint update will be performed.

[0037] Furthermore, the physical information loss function includes a kinematic constraint loss term, a safety constraint loss term, and a comfort constraint loss term, the mathematical expression of which is as follows:

[0038] in, For kinematic constraint loss terms, For safety constraint loss items, For comfort constraint loss term, , , These are the weighting coefficients for each loss term; The kinematic constraint loss term Used to constrain the consistency between the predicted and actual states of a vehicle, and to reflect the physical rationality of the vehicle's motion, its calculation formula is as follows:

[0039] in, The current position is predicted based on the state at the previous moment. This is the current actual location. Let be the normalization constant for road length. The predicted speed is inferred from the action. This is the current actual speed. The maximum velocity normalization constant, and These are the corresponding weighting coefficients.

[0040] The security constraint loss item The formula used to constrain vehicles to maintain a safe driving state is as follows:

[0041] Where TTC is the collision time. min The minimum safe collision time threshold; d This represents the longitudinal distance between the vehicle and the vehicle in front. d min The minimum safe distance threshold; gap is the merging gap. min Minimum inflow gap threshold; w ttc , w d , w gap These are the corresponding penalty weight coefficients.

[0042] The comfort constraint loss item The formula used to constrain a vehicle to maintain stable driving is as follows:

[0043] in, The vehicle's current acceleration. The maximum permissible acceleration threshold, This refers to the impact intensity (rate of change of acceleration). The maximum permissible impact threshold, This is a lane change indicator function used to characterize the frequency or number of lane change actions performed by the vehicle, in order to suppress the discomfort caused by frequent lane changes. The value is 1 when the vehicle is currently performing a lane change action, and 0 otherwise. , , These are the weighting coefficients for the acceleration penalty, impact penalty, and lane change penalty, respectively.

[0044] The impact It is calculated from the ratio of the acceleration difference between adjacent moments to the time step size, and the calculation formula is as follows:

[0045] in, The vehicle's current acceleration; The decision-making time step.

[0046] Furthermore, the total loss function of the PPO decision model is a weighted combination of the PPO main loss function and the physical information loss function. By introducing the physical information loss function during the strategy optimization process, a physical consistency constraint is imposed on the ramp merging decision strategy, thereby improving the model training stability and the safety, rationality, and comfort of the decision results. The mathematical expression of the total loss function is as follows:

[0047]

[0048] in, This is the physical loss weighting coefficient.

[0049] Furthermore, the structured scene semantic description includes three semantic levels: current driving scene information, surrounding vehicle situation information, and safety indicator information. The current driving scenario information includes the lane type identifier, longitudinal position coordinates, and current driving speed of the vehicle:

[0050] in, Lane type identifier, used to distinguish whether the vehicle is currently in a main road lane or a ramp lane; The coordinates are the longitudinal position coordinates of the vehicle. This is the vehicle's current speed; The surrounding vehicle situation information uses relative coordinates to describe the position and motion status of each neighboring vehicle:

[0051] in, For the first The longitudinal distance of the vehicle relative to itself. The horizontal distance. For relative velocity, Vehicle type identifier; The safety metrics information includes Time to Collision (TTC), risk level assessment, and available merge gap analysis:

[0052] Where TTC is the estimated time to the nearest vehicle. This is a discrete risk level derived from a comprehensive assessment based on collision time and vehicle spacing, with values ​​ranging from low risk, medium risk, or high risk. The size of the gap between vehicles on the main road that can merge.

[0053] Furthermore, when constructing the input prompts for the large language model by combining the structured scene semantic description with the candidate actions output by the decision model, the specific steps include: The information from the above three semantic levels is combined with the candidate actions output by the PPO decision model. Structured text integrated into natural language form As input to a large language model:

[0054] in, The integration function is the candidate action output by the PPO decision model. The numerical state information and candidate actions are converted into natural language descriptions containing domain semantics, enabling the large language model to understand both the current traffic scenario and the specific actions to be evaluated. Physical constraint parameters are injected into the system instructions of the large language model in a parameterized form. A hierarchical modular prompt is constructed, which includes system instructions, safety examples, danger examples, current scene description and judgment instructions. The input prompt of the large language model adopts a hierarchical modular structure, and the domain safety threshold is injected into the system instructions in a parameterized manner, so that the safety reasoning of the large language model is guided by physical constraints. The hierarchical modular hint structure of input for large language models is defined as follows:

[0055] in, For system instructions that include parameterized valid security thresholds; This is an example of a security scenario; Example of a dangerous scenario; This describes the current structured scene. These are candidate actions to be evaluated; To determine the instruction; Security Scenario Examples Examples of dangerous scenarios These are pre-set few-shot learning examples used to guide large language models to understand the standards and output formats for safety judgments. The examples contain several typical safety scenarios and several typical dangerous scenarios corresponding to the set of safety thresholds.

[0056] Furthermore, the safety judgment criteria of the large language model are adaptively adjusted based on the agent's current driving stage and candidate action type to avoid overly conservative safety checks that could hinder the completion of the integration task. Before using the large language model to perform safety checks on the candidate actions output by the decision model, a step of determining a safety threshold is also included, including: Based on the lane type identifier in the current driving scenario information The driving phase of the intelligent agent is divided into the ramp merging phase and the main road driving phase:

[0057] in, Lane type identification, This indicates that the vehicle is currently in the ramp lane. This indicates that the vehicle is currently in the main lane. This is the merging phase of the ramp. During the main road driving phase; The set of valid security thresholds used in the security judgment By threshold mapping function Determined based on the current driving stage and candidate action type:

[0058] When the driving stage is And when the candidate action is an inflow direction action, the mapping function Output import priority effective threshold set This will be incorporated into the priority effective threshold. As an effective safety threshold Compared to the standard threshold set Appropriately relax collision time and vehicle distance constraints; When the driving stage is Or when the candidate action is a non-merging direction action, the mapping function Output a set of standard valid thresholds, and use these standard valid thresholds as the effective safety thresholds. ; The set of security thresholds Defined as:

[0059] in, The minimum safe collision time threshold; This is the minimum safe vehicle spacing threshold; Minimum safe inbound gap threshold; The maximum permissible speed threshold is defined as follows: the parameters in the safety threshold set are consistent with the physical constraint parameters used in the physical information loss function, so that the physical constraint knowledge in the training phase is transmitted to the large language model in the inference phase through prompt injection. The set of effective thresholds for priority inflow Defined as:

[0060] because , , This is the lower threshold for safety judgment; the smaller the threshold, the more lenient the allowable safety margin. All thresholds are less than The corresponding threshold reflects the tolerance strategy for actions in the merging direction during the merging phase.

[0061] The design basis of the merging priority mechanism is that the lane change action to the main road during the ramp merging stage is a necessary action to complete the task objective. If the standard safety threshold of driving on the main road is used for evaluation, it will cause the large language model to over-intercept the merging action, making it impossible for the agent to complete the merging task. By perceiving the agent's current task objective, the safety judgment standard is matched with the task stage, so as to maintain the feasibility of the merging task while ensuring safety.

[0062] System commands The effective safety threshold is embedded as a parameterized rule, including: when the collision time is less than In (or A collision risk is identified when the distance to the current vehicle is less than [a certain distance]. In (or When the available merging gap is less than 1, it is determined that the distance is too close; when the available merging gap is less than 1, it is determined that the distance is too close. In (or When the gap is insufficient, the parameterization rule enables the large language model to have a quantitative basis for domain safety judgment during the reasoning process, rather than relying on general language understanding for fuzzy judgment.

[0063] Furthermore, when using a large language model to perform safety checks on the candidate actions output by the decision model and obtain the corresponding safety judgment results, the large language model adopts a multi-dimensional step-by-step safety reasoning method for ramp merging scenarios to judge the safety of candidate actions. It decomposes the safety analysis into three orthogonal safety dimensions—kinematic, spatial, and task—for layer-by-layer reasoning, and constrains the output to a closed-form parsable format, including the following analysis steps: Step 1: Collision Risk Analysis from a Kinematic Perspective: Based on the Time to Collision (TTC) and Distance to the Leading Vehicle information from the aforementioned safety indicators, combined with the effective safety threshold... The minimum safe collision time threshold and minimum safe vehicle distance threshold in the set are used to assess the immediate collision risk of candidate actions in the current motion state. When the collision time is lower than the minimum safe collision time threshold or the distance to the vehicle in front is less than the minimum safe vehicle distance threshold, the candidate action is judged as dangerous. Otherwise, proceed to Step 2. Step 2: Spatial lane change feasibility analysis: Based on the available merging gaps in the aforementioned safety indicator information. The relative longitudinal distance and relative speed between the vehicle in the target lane and the adjacent vehicles in front and behind, based on the surrounding vehicle situation information, are used to evaluate whether the target lane space meets the safety constraints according to a preset rule set. When the available merging gap is less than the minimum safe merging gap threshold, or when the estimated collision time between the vehicle and the adjacent vehicles in front and behind, calculated based on the relative longitudinal distance and relative speed, is less than the minimum safe collision time threshold, the candidate action is judged as dangerous. Otherwise, proceed to Step 3. Step 3: Action rationality analysis at the task level: Based on the vehicle's current speed, lane type, and candidate actions, evaluate the physical rationality of the candidate actions according to the specified task matching rule set. If the candidate action is a deceleration action and the vehicle's current speed is greater than the product of the maximum permissible speed threshold and the high-speed rapid deceleration judgment coefficient, or if the candidate action is an acceleration action and the vehicle's current speed is greater than the maximum permissible speed threshold, or if the candidate action is a lane change action and the vehicle's lane type is identified as a main road, the candidate action will be judged as unreasonable. Based on the above layer-by-layer reasoning results, the large language model comprehensively outputs a security judgment conclusion constrained to a closed structured format, defined as:

[0064] The judgment result is a binary classification of safe or dangerous. If the candidate action is judged as dangerous or unreasonable, the judgment result is dangerous; otherwise, the judgment result is safe. The reasoning is based on the judgment reasons derived from the above three dimensions. The suggested action is to adopt the preset conservative action recommended by the large language model when the judgment result is dangerous. The closed output format ensures that the output of the large language model can be directly parsed and executed by the downstream consensus fusion module, avoiding the uncertainty brought about by free text output.

[0065] Furthermore, when executing a candidate action or a preset conservative action based on the security assessment result, the candidate action output by the PPO decision model is... The security judgment result output by the large language model The final execution action is generated through a consensus fusion mechanism, and the rules for generating the final execution action are as follows:

[0066] The final execution action is as follows: The candidate actions output by the PPO decision model are: When the large language model outputs a safety judgment result, the final action to be executed is the candidate action; when the large language model outputs a danger judgment result, the final action to be executed is a preset conservative action. The alternative action selected from the preset set of conservative actions.

[0067] The preset conservative action is defined as:

[0068] in, This indicates an action that maintains the current state. This indicates a deceleration action; all actions in the conservative action set are passive safety actions that will not increase the risk of collision in the current scenario. When the large language model determines that the candidate action is dangerous, from Select the alternative action that best matches the current scenario; when there is a clear risk of collision in the current scenario, prioritize the deceleration action. Otherwise, choose the action of maintaining the current state. ; The consensus fusion mechanism is used to achieve synergy between data-driven decision-making capabilities and knowledge-driven security review capabilities: Under normal driving conditions, the final execution action directly adopts the output of the PPO decision model, retaining the decision-making flexibility and efficiency of the reinforcement learning strategy; only when the large language model determines that the candidate action has a security risk based on the domain security threshold and multi-dimensional reasoning, the output of the PPO decision model is intercepted and corrected through alternative actions to improve the security and interpretability of the ramp merging decision.

[0069] Compared with the prior art, the advantages of the present invention are as follows: This invention constructs a state space including the vehicle's state, the relative positions and speeds of surrounding vehicles, and safety indicators, and designs an action space including lane changing, holding, acceleration, and deceleration. A proximal policy optimization algorithm is used to train the ramp merging decision model, incorporating a physical information loss function during training to integrate vehicle kinematic constraints, safety constraints, and comfort constraints into the policy optimization process. During the inference phase, domain semantic encoding couples the current traffic state with candidate actions into the input of a large language model. The physical constraint parameters from the training phase are injected into the prompts in parameterized form, guiding the large language model to perform step-by-step safety inference of candidate actions according to kinematic, spatial, and task dimensions, and adaptively adjusting safety judgment criteria based on the current driving stage. A consensus fusion mechanism determines the final execution action, effectively avoiding potentially dangerous decisions that might arise from relying solely on reinforcement learning, thus achieving safe, efficient, and interpretable decision-making in highway ramp merging scenarios. Attached Figure Description

[0070] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0071] Figure 2 This is a method framework diagram of an embodiment of the present invention.

[0072] Figure 3A comparison chart of import success rates in test scenarios.

[0073] Figure 4 A comparison chart of average speeds under test scenarios. Detailed Implementation

[0074] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0075] This embodiment proposes a knowledge-data-driven highway ramp merging decision-making method. This method addresses the challenges of complex interactions among traffic participants, high safety risks, and stringent real-time decision-making requirements in highway ramp merging scenarios. It establishes a state space and action space based on the highway ramp merging environment and uses the Proximal Policy Optimization (PPO) algorithm to learn ramp merging strategies. Then, a physical information loss function is introduced during training to ensure the decision model meets physical consistency and safety requirements during learning. Finally, in the inference phase, domain semantic encoding couples the current traffic state and candidate actions into a large language model input. Physical constraints from the training phase are injected into prompts in a parameterized manner. The large language model performs step-by-step safe inference of candidate actions according to kinematic, spatial, and task dimensions. A consensus fusion mechanism outputs the final execution action based on the review results, thereby achieving safe, efficient, and interpretable decision control in highway ramp merging scenarios.

[0076] like Figure 1 As shown, the method in this embodiment specifically includes the following steps: S101) Merging Environment Construction Phase: Establish the highway ramp merging environment and reinforcement learning decision model environment. The highway ramp merging environment includes a two-lane main road and parallel ramps connecting to the main road. The road is divided into the main road starting section, transition section, merging conflict section, and termination section, with the merging conflict section being the core decision area. The state space of the reinforcement learning decision model environment includes state information representing the interaction between the vehicle and surrounding vehicles during ramp merging. The action space of the reinforcement learning decision model environment includes different candidate actions during ramp merging. Extract state information representing the interaction between the vehicle and surrounding vehicles in the ramp merging scenario. The state information includes at least the vehicle's state information, the relative position and relative speed information of surrounding vehicles, and safety indicator information. The safety indicator information includes at least collision time and distance between vehicles. Construct a discrete action space, which includes at least left lane change, maintaining the current state, right lane change, acceleration, and deceleration. The observation space is constructed using a kinematic feature matrix. The state information of the vehicle and the N nearest surrounding vehicles is uniformly represented and normalized before being used as input to the PPO decision model to improve the stability and generalization ability of policy training.

[0077] S102) Model Training and Optimization Phase: An Actor-Critic architecture is used to construct the PPO decision model. The Actor network is used to output the probability distribution of each candidate action based on the normalized input state information, and the Critic network is used to estimate the state value of the current state. During the training iteration, the agent interacts with the environment in the highway ramp merging environment according to the current strategy, obtains state transition samples and immediate rewards, and stores the interaction trajectory in the Rollout trajectory buffer.

[0078] In this embodiment, the PPO decision model uses a multi-objective reward function to calculate the immediate reward. The multi-objective reward function includes at least a collision penalty, a successful merging reward, a speed maintenance reward, a comfort penalty, and a driving progress reward, which is used to guide the model to learn a ramp merging strategy that takes into account safety, traffic efficiency, and driving comfort.

[0079] A physical information loss function is introduced during training, which serves as an auxiliary loss term and is jointly optimized with the PPO main loss function. The physical information loss function includes at least kinematic constraint loss terms, safety constraint loss terms, and comfort constraint loss terms. The physical information loss function is weighted and combined with the PPO main loss function's PPO pruning objective function, value function loss, and entropy regularization term to form the total loss function. Gradient descent is then used to jointly update the Actor network parameters and Critic network parameters, ensuring that the trained decision policy simultaneously satisfies physical rationality, safety, and comfort requirements.

[0080] The specific steps of the training iteration process include: S1) The state information of vehicles merging into the scene from the highway ramp and surrounding vehicles is used as input to the agent. After processing, it forms a complete observation vector. The Actor policy network outputs the probability distribution of each candidate action according to the observation vector. The execution action is obtained by sampling according to the probability distribution and the execution action is executed in the simulation environment to obtain the next state and immediate reward. The state, action, reward and next state are combined to obtain the state transition trajectory and stored as state transition experience in the Rollout trajectory buffer. S2) Calculate the temporal difference error and the estimated value of the dominance function based on the collected trajectory data, and update the magnitude using the pruning objective function constraint strategy; at the same time, calculate the physical information loss function based on vehicle kinematics, safety threshold conditions and comfort index.

[0081] S3) The PPO main loss function and the physical information loss function are weighted and fused to form the total loss function. The parameters of the Actor-Critic decision model are updated, and a stable ramp merging decision model is obtained after multiple rounds of environmental interaction and training iteration.

[0082] S103) Semantic Reasoning Decision Stage: After completing the training iterations of the PPO decision model, the reasoning stage begins. First, the current traffic state information of the highway ramp merging environment is acquired. This current state information is input into the decision model and converted into a structured scene semantic description. This structured scene semantic description includes at least the current driving scenario information, surrounding vehicle situation information, and safety indicator information. Specifically, the current driving scenario information includes at least the vehicle's lane type, longitudinal position coordinates, and current speed; the surrounding vehicle situation information includes at least the longitudinal distance, lateral distance, and relative speed of neighboring vehicles relative to the vehicle; and the safety indicator information includes at least the collision time, risk level, and available merging gap. Subsequently, the structured scene semantic description and the candidate actions output by the PPO decision model are combined to construct a large language model input prompt. This prompt is input into the large language model to perform a safety check on the candidate actions output by the decision model, obtaining the corresponding safety judgment result. Based on the safety judgment result, either the candidate action or a preset conservative action is executed.

[0083] The input prompts for the large language model employ a hierarchical modular structure. The safety threshold parameters used in the physical information loss function during training are embedded into system instructions as parameterized rules. This enables the large language model to have quantifiable domain-specific safety judgment criteria during inference, rather than relying on fuzzy judgments based on general language understanding. The large language model uses a combination of few-shot learning and step-by-step inference for safety review. Several safe and dangerous scenario examples are pre-set in the input prompts to guide the large language model in understanding the safety judgment criteria for ramp merging scenarios. Simultaneously, the large language model is guided to sequentially complete collision risk analysis, lane change feasibility analysis, and action rationality analysis according to preset inference steps. The output is constrained to a closed-loop structured format containing judgment results, reasoning basis, and suggested actions, allowing downstream modules to directly parse and execute it.

[0084] The safety judgment criteria of the large language model for safety checks are adaptively adjusted based on the vehicle's current lane type and candidate action type: when the vehicle is on a ramp and the candidate action is a merging direction action, the safety threshold constraint is appropriately relaxed to avoid overly conservative safety checks that could hinder the completion of the merging task. The safety judgment result determines the final action to be executed through a consensus fusion mechanism: when the large language model determines that a candidate action is safe, the candidate action is executed directly; when it is determined to be dangerous, an alternative action is selected from a preset set of conservative actions, which includes two passive safety actions: maintaining the current state and deceleration.

[0085] S104) Strategy Application Stage: The trained decision model and the large language model are deployed together as a complete decision strategy in the highway ramp merging scenario to achieve safe, efficient and stable merging decision control.

[0086] The following provides a detailed explanation of each step.

[0087] This embodiment constructs the import environment through step S101, which specifically includes the following steps: A simulation environment for highway ramp merging was constructed, including a two-lane main road and parallel ramps connecting to the main road. The simulation environment was divided into the main road starting section, transition section, merging conflict section and termination section, with the merging conflict section being the core decision-making area.

[0088] A reinforcement learning decision-making model environment is established. The state space of the reinforcement learning decision-making model environment consists of state information representing the interaction between the vehicle and surrounding vehicles when merging into the ramp. The state information includes the vehicle's own state information, the relative position and relative speed information of surrounding vehicles, and safety index information. The safety index information includes at least the time of collision (TTC) and the distance between the front and rear vehicles.

[0089] in, This represents the longitudinal distance between the vehicle and the vehicle in front. The relative speed between the vehicle and the vehicle in front; The action space of the reinforcement learning decision model environment consists of different candidate actions when merging at a ramp, and the set of candidate actions... Defined as a discrete action space, the set includes left lane change, maintaining the current state, right lane change, acceleration, and deceleration:

[0090] in, Indicates a left lane change maneuver. This indicates an action that maintains the current state. Indicates a right lane change maneuver. Indicates an acceleration action. This indicates a deceleration action.

[0091] The state information is constructed using a kinematic feature matrix to create an observation space, which includes the state information of the vehicle itself and the N nearest surrounding vehicles; the mathematical representation of the observation space is as follows:

[0092] Where N is the number of surrounding vehicles observed, and M is the feature dimension of each vehicle; M feature dimensions are extracted for each vehicle, and the M feature dimensions include:

[0093] in, Vertical position The horizontal position For longitudinal velocity, For lateral velocity, As a marker of existence; The observation space, after being normalized, is input into the PPO decision model. The normalization formula is:

[0094] in, Let be the normalization constant for road length. Let be the normalization constant for road width. This is the normalization constant for maximum velocity; A relative coordinate system is used to characterize the relative motion relationships between vehicles in order to improve the generalization ability of the merged decision strategy.

[0095] The PPO decision model employs a multi-objective reward function, which includes at least a successful merging reward, a collision penalty, a speed maintenance reward, a comfort penalty, and a driving progress reward. These reward components are weighted and summed to form the overall reward function, which guides the PPO decision model in learning a ramp merging strategy that balances safety, traffic efficiency, and driving comfort. Its mathematical expression is as follows:

[0096] in, For collision penalties, To successfully import the reward item, To maintain the speed bonus, For comfort penalty items, This is a reward item for driving progress. , , , , These are the weighting coefficients for each item; Collision Penalty The calculation formula is:

[0097] Successful import reward The calculation formula is:

[0098] in, To add a successful import reward coefficient; Speed ​​maintenance bonus The calculation formula is:

[0099] in, Current vehicle speed For the target driving speed, This refers to the speed reward weighting coefficient. Comfort penalty items The calculation formula is:

[0100] in, The weight of the penalty for lane changing actions. Change the penalty weight for general actions; Driving progress reward items The calculation formula is:

[0101] in, This represents the longitudinal displacement increment per unit time. As a reference distance constant, This refers to the progress reward weighting coefficient. The total reward function The output is after cropping, and the cropping range is... This is to ensure numerical stability during the training process.

[0102] In this embodiment, the model is trained iteratively using the Proximal Policy Optimization (PPO) algorithm in step S102. During training, a physical information loss function is introduced to jointly optimize the PPO main loss function. The PPO algorithm is a reinforcement learning method based on policy gradients, suitable for decision control scenarios with continuous states, complex interactions, and high requirements for policy update stability. It can improve the stability of the training process by limiting the policy update magnitude. The PPO decision model adopts an Actor-Critic architecture, where the Actor network is used to output the probability distribution of each candidate action based on the input state, and the Critic network is used to estimate the state value of the current state. During training, the agent continuously interacts with the environment as it merges into the highway ramp, collecting trajectory data including state, action, reward, and next state from the Rollout trajectory buffer. Based on the trajectory data, the generalized dominance estimation method is used to calculate the dominance function, and the policy update magnitude is limited by pruning the objective function using PPO. At the same time, a physical information loss term is calculated based on physical information to constrain the model output to meet the requirements of physical rationality, safety, and driving comfort. The PPO main loss function and the physical information loss function are weighted and fused to form the total loss function. This is used to jointly update the parameters of the Actor and Critic networks, enabling the model to maintain data-driven adaptive learning capabilities while possessing stronger physical consistency, training stability, and decision controllability. After each round of updates, the Rollout trajectory cache is cleared, and a new round of environmental interaction and experience collection begins. Specific steps include: S201) Build a PPO decision model, adopting an Actor-Critic architecture. The policy network (Actor) outputs the probability distribution of each candidate action based on the current state, and the value network (Critic) estimates the state value of the current state to achieve coordinated optimization of policy update and value assessment. Let the policy (Actor) network be The parameters are ; The Critic network is The parameters are The Actor and Critic networks employ a shared feature extraction layer architecture to improve training efficiency. The Rollout buffer is... The discount factor is The generalized dominance estimation parameters are: The cutting factor is The value loss coefficient is The entropy regularization coefficient is .

[0103] The pruning objective function The formula for limiting the policy update magnitude is as follows:

[0104] in, The strategy probability ratio This is the estimated value of the dominance function; The formula for calculating the strategy probability ratio is:

[0105] in, As the current strategy, The strategy before the update; the clip function constrains the strategy ratio to... Within a certain range, to prevent excessive changes in strategy.

[0106] The advantage function The Generalized Advantage Estimation (GAE) method is used for calculation, and its formula is as follows:

[0107] in, The timing difference error is calculated using the following formula:

[0108] in, For instant rewards, For state value estimation.

[0109] The value function loss The formula used to train the Critic network is as follows:

[0110] in, The target value is calculated using Monte Carlo returns or time-series difference objectives.

[0111] The entropy regularization term It is used to encourage policy exploration and prevent premature policy convergence; its calculation formula is as follows:

[0112] The total loss function of the PPO algorithm The weighted combination of each loss component is calculated using the following formula:

[0113] in, This is the value loss weighting coefficient. The entropy regularization weight coefficients are used to minimize the weight coefficients. For Actor network parameters and Critic network parameters A joint update will be performed.

[0114] (S202) During the training of the PPO decision model, a physical information loss function is introduced. This physical information loss function serves as an auxiliary loss term and is jointly optimized with the main loss function of the PPO decision model. The physical information loss function includes kinematic constraint loss terms, safety constraint loss terms, and comfort constraint loss terms, and its mathematical expression is as follows:

[0115] in, For kinematic constraint loss terms, For safety constraint loss items, For comfort constraint loss term, , , These are the weighting coefficients for each loss term.

[0116] The kinematic constraint loss term Used to constrain the consistency between the predicted and actual states of a vehicle, and to reflect the physical rationality of the vehicle's motion, its calculation formula is as follows:

[0117] in, The current position is predicted based on the state at the previous moment. This is the current actual location. Let be the normalization constant for road length. The predicted speed is inferred from the action. This is the current actual speed. The maximum velocity normalization constant, and These are the corresponding weighting coefficients.

[0118] The security constraint loss item The formula used to constrain vehicles to maintain a safe driving state is as follows:

[0119] Where TTC is the collision time. min The minimum safe collision time threshold; d This represents the longitudinal distance between the vehicle and the vehicle in front. d min The minimum safe distance threshold; gap is the merging gap. min Minimum inflow gap threshold; w ttc , w d , w gap These are the corresponding penalty weight coefficients.

[0120] The comfort constraint loss item The formula used to constrain a vehicle to maintain stable driving is as follows:

[0121] in, The vehicle's current acceleration. The maximum permissible acceleration threshold, This refers to the impact intensity (rate of change of acceleration). The maximum permissible impact threshold, This is a lane change indicator function used to characterize the frequency or number of lane change actions performed by the vehicle, in order to suppress the discomfort caused by frequent lane changes. The value is 1 when the vehicle is currently performing a lane change action, and 0 otherwise. , , These are the weighting coefficients for the acceleration penalty, impact penalty, and lane change penalty, respectively.

[0122] The impact It is calculated from the ratio of the acceleration difference between adjacent moments to the time step size, and the calculation formula is as follows:

[0123] in, The vehicle's current acceleration; The decision-making time step.

[0124] (S203) The total loss function of the PPO decision model is a weighted combination of the PPO main loss function and the physical information loss function. By introducing the physical information loss function during the strategy optimization process, a physical consistency constraint is imposed on the ramp merging decision strategy to improve the model training stability and the safety, rationality, and comfort of the decision results. The mathematical expression of the total loss function is as follows:

[0125]

[0126] in, This is the physical loss weighting coefficient.

[0127] This embodiment achieves the semantic reasoning decision-making stage through step S103, converting the current traffic state into a structured scene semantic description that includes the driving scenario, the surrounding vehicle situation, and safety indicators. This description is then coupled with the candidate actions output by the PPO and incorporated into the large language model as input prompts. The physical constraint parameters from the training phase are injected into the prompt instructions in a parameterized form. The large language model completes the safety review through few-shot learning and step-by-step reasoning, outputting the safety judgment results and explanations of the candidate actions. Finally, a consensus fusion mechanism determines the final action to be executed. Specific steps include: S301) During the inference phase, the current traffic state information is converted into a structured scene semantic description through domain semantic encoding, and the candidate actions output by the PPO decision model are also incorporated into it, forming a large language model input for action-scene coupling; the structured scene semantic description includes three semantic levels: current driving scene information, surrounding vehicle situation information, and safety indicator information. The current driving scenario information includes the lane type identifier, longitudinal position coordinates, and current driving speed of the vehicle:

[0128] in, Lane type identifier, used to distinguish whether the vehicle is currently in a main road lane or a ramp lane; The coordinates are the longitudinal position coordinates of the vehicle. This is the vehicle's current speed; The surrounding vehicle situation information uses relative coordinates to describe the position and motion status of each neighboring vehicle:

[0129] in, For the first The longitudinal distance of the vehicle relative to itself. The horizontal distance. For relative velocity, Vehicle type identifier; The safety metrics information includes Time to Collision (TTC), risk level assessment, and available merge gap analysis:

[0130] Where TTC is the estimated time to the nearest vehicle. This is a discrete risk level derived from a comprehensive assessment based on collision time and vehicle spacing, with values ​​ranging from low risk, medium risk, or high risk. The size of the gap between vehicles merging on the main road; The information from the above three semantic levels is combined with the candidate actions output by the PPO decision model. Structured text integrated into natural language form As input to a large language model:

[0131] in, The integration function is the candidate action output by the PPO decision model. The numerical state information and candidate actions are converted into natural language descriptions containing domain semantics, enabling the large language model to understand both the current traffic scenario and the specific actions to be evaluated.

[0132] S302) The input prompts of the large language model adopt a hierarchical modular structure, and the domain security threshold is injected into the system instructions in a parameterized manner, so that the secure reasoning of the large language model is guided by physical constraints; In this embodiment, the current driving scenario information, surrounding vehicle situation information, and safety indicator information are integrated with the candidate actions into structured text in natural language form as input to the large language model. Physical constraint parameters are injected into the large language model system instructions in parameterized form. A hierarchical modular prompt structure is constructed, including system instructions, safety examples, danger examples, current scenario descriptions, and judgment instructions. The hierarchical modular prompt structure of the large language model input is defined as follows:

[0133] in, For system instructions that include parameterized valid security thresholds; This is an example of a security scenario; Example of a dangerous scenario; This describes the current structured scene. These are candidate actions to be evaluated; To determine the instruction; The safety judgment criteria of the large language model are adaptively adjusted according to the current driving stage of the agent and the candidate action type to avoid overly conservative safety checks that could hinder the completion of the integration task. Based on the lane type identifier in the current driving scenario information The driving phase of the intelligent agent is divided into the ramp merging phase and the main road driving phase:

[0134] in, Lane type identification, This indicates that the vehicle is currently in the ramp lane. This indicates that the vehicle is currently in the main lane. This is the merging phase of the ramp. During the main road driving phase; The set of effective security thresholds used in the security assessment By threshold mapping function Determined based on the current driving stage and candidate action type:

[0135] When the driving stage is And when the candidate action is an inflow direction action, the mapping function Output import priority effective threshold set ,Right now Compared to the standard threshold set Appropriately relax collision time and vehicle distance constraints; When the driving stage is Or when the candidate action is a non-merging direction action, the mapping function Output the set of standard valid thresholds, i.e. ; The set of security thresholds Defined as:

[0136] in, The minimum safe collision time threshold; This is the minimum safe vehicle spacing threshold; Minimum safe inbound gap threshold; The maximum permissible speed threshold is defined as follows: the parameters in the safety threshold set are consistent with the physical constraint parameters used in the physical information loss function in step S102, so that the physical constraint knowledge in the training phase is transmitted to the large language model in the inference phase through prompt injection. The set of effective thresholds for priority inflow Defined as:

[0137] because , , This is the lower threshold for safety judgment; the smaller the threshold, the more lenient the allowable safety margin. All thresholds are less than The corresponding threshold reflects the tolerance strategy for actions in the merging direction during the merging phase.

[0138] The lane change action to the main road during the ramp merging phase is a necessary action to complete the task objective. If the standard safety threshold of driving on the main road is used for evaluation, it will cause the large language model to over-intercept the merging action, making it impossible for the agent to complete the merging task. By perceiving the agent's current task objective, the safety judgment standard is matched with the task stage, so as to maintain the feasibility of the merging task while ensuring safety.

[0139] The system instructions The effective safety threshold is embedded in the form of parameterized rules, including: when the collision time is less than In (or A collision risk is identified when the distance to the current vehicle is less than [a certain distance]. In (or When the available merging gap is less than 1, it is determined that the distance is too close; when the available merging gap is less than 1, it is determined that the distance is too close. In (or When the gap is insufficient, the parameterization rule enables the large language model to have a quantitative basis for domain safety judgment during the reasoning process, rather than relying on general language understanding for fuzzy judgment.

[0140] Example of the security scenario Examples of dangerous scenarios These are pre-set few-shot learning examples used to guide large language models to understand the standards and output formats for safety judgments. The examples contain several typical safety scenarios and several typical dangerous scenarios corresponding to the set of effective safety thresholds.

[0141] When using a large language model to perform safety checks on the candidate actions output by the decision model and obtain the corresponding safety judgment results, the large language model adopts a multi-dimensional step-by-step safety reasoning method for ramp merging scenarios to judge the safety of candidate actions. It decomposes the safety analysis into three complementary safety dimensions: kinematic dimension, spatial dimension, and task dimension, and reasons layer by layer, and constrains the output to be a closed parsable format. The multi-dimensional step-by-step security reasoning includes the following analysis steps: Step 1: Collision Risk Analysis in Kinematic Dimensions: Based on the collision time and distance to the preceding vehicle in the aforementioned safety indicator information, combined with the effective safety threshold... Minimum safe vehicle spacing threshold in the set ( or ) and minimum safe vehicle spacing threshold ( or ), for candidate actions The instantaneous collision risk under the current motion state is determined by a piecewise function. A three-tier rating system was implemented. For example, the function expression is as follows:

[0142] in This is the warning amplification factor, used to issue a warning status in the critical region to trigger a degradation decision. When When the collision risk is detected, this dimension directly outputs a "collision risk" label and initially classifies the candidate action as dangerous; when... If it is safe, proceed to Step 2.

[0143] Step 2: Spatial Dimension Lane Change Feasibility Analysis: Only when Take a lane change action or This is executed when the action is in the direction of merging, based on the available merging gap in the aforementioned safety indicator information. The relative longitudinal distance between the vehicle in the target lane and the adjacent vehicle in front, as well as the surrounding vehicle situation information. The relative longitudinal distance between the vehicle and the adjacent following vehicle With relative velocity , According to the following rule set Assess whether the target lane space meets safety constraints. For example, the expression is as follows:

[0144]

[0145]

[0146] in To prevent division by zero of extremely small positive numbers; when Output a "Insufficient gap" label and classify candidate actions as dangerous; when For non-lane-changing maneuvers (hold / accelerate / decelerate), this step is directly set. And skip spatial assessment.

[0147] Step 3: Action Reasonableness Analysis at the Task Dimension: Based on the vehicle's current speed, lane type, and candidate actions, the following task matching rule set is applied. Evaluate the physical plausibility of executing candidate actions:

[0148] in The high-speed rapid deceleration judgment coefficient is used to characterize unreasonable situations of "high-speed rapid deceleration under non-emergency conditions". This step covers two typical unreasonable actions: unnecessary rapid deceleration risk when driving at high speed and unreasonable acceleration exceeding the speed limit when driving at low speed.

[0149] The large language model aggregates the above three-dimensional judgment results using "negation logic" to obtain the final judgment result. :

[0150] That is, any determination of unsafety in any dimension triggers the conclusion of overall unsafety, reflecting the orthogonal independent veto mechanism of the three dimensions of "kinematics-space-task"; when At that time, the large language model selects suggested actions from the preset set of conservative actions; the reasoning basis is composed of the judgment labels of each dimension in sequence.

[0151] The output of the large language model is constrained to a closed-structure format, defined as:

[0152] The judgment result is a binary classification of safe or dangerous; the reasoning basis is the judgment reason derived from the analysis of the above three dimensions; the suggested action is to adopt the preset conservative action recommended by the large language model when the judgment result is dangerous; the closed output format ensures that the output of the large language model can be directly parsed and executed by the downstream consensus fusion module, avoiding the uncertainty brought about by free text output.

[0153] S303) When executing a candidate action or a preset conservative action based on the security assessment result, the candidate action output by the PPO decision model. The security judgment result output by the large language model The final execution action is generated through a consensus fusion mechanism, and the rules for generating the final execution action are as follows:

[0154] The final execution action is as follows: The candidate actions output by the PPO decision model are: When the large language model outputs a safety judgment result, the final action to be executed is the candidate action; when the large language model outputs a danger judgment result, the final action to be executed is a preset conservative action. The alternative action selected from the preset set of conservative actions.

[0155] The preset conservative action is defined as:

[0156] in, This indicates an action that maintains the current state. This indicates a deceleration action; the actions in the conservative action set are all passive safety actions that will not increase the risk of collision in the current scenario; when the large language model determines that a candidate action is dangerous, it selects the appropriate action from the list of actions. Select the alternative action that best matches the current scene as When there is a clear risk of collision in the current scenario, deceleration should be the priority action. Otherwise, choose the action of maintaining the current state. ; The consensus fusion mechanism is used to achieve synergy between data-driven decision-making capabilities and knowledge-driven security review capabilities: Under normal driving conditions, the final execution action directly adopts the output of the PPO decision model, retaining the decision-making flexibility and efficiency of the reinforcement learning strategy; only when the large language model determines that the candidate action has a security risk based on the domain security threshold and multi-dimensional reasoning, the output of the PPO decision model is intercepted and corrected through alternative actions to improve the security and interpretability of the ramp merging decision.

[0157] This embodiment implements a safe, efficient, and stable strategy application in the scenario of merging onto highway ramps through step S104, specifically including the following steps: (S104) The trained PPO decision model and the large language model are deployed together as a complete decision strategy in the highway ramp merging simulation environment: the PPO decision model outputs candidate actions based on real-time observation vectors, and the large language model performs multi-dimensional step-by-step reasoning on the candidate actions according to the effective safety threshold set in S302 and three complementary safety dimensions: kinematic dimension, spatial dimension, and task dimension, outputting a closed-loop structured judgment result; finally, the final execution action is generated according to the consensus fusion mechanism described in S303. Through the collaborative deployment of the PPO decision model and the large language model, the data-driven decision-making capability formed by physical information reinforcement learning in the training phase and the knowledge-driven safety review capability formed by the multi-dimensional safety review of the large language model in the reasoning phase work synergistically in the same environment, thereby achieving safe, efficient, and stable real-time decision control during ramp merging.

[0158] like Figure 2 As shown, the method in this embodiment consists of two collaborative parts: a training closed loop and an inference decision-making process. During the training phase (shown on the left side of the figure), the Actor-Critic network outputs actions based on the current state and executes them in the environment, obtaining state transition samples stored in a trajectory buffer. The advantage function is calculated using generalized advantage estimation (GAE), and the network parameters are updated with gradients after loss function calculation, forming a policy iterative optimization closed loop. During the inference phase, the reinforcement learning decision-making agent outputs candidate actions based on the trained Actor network. Simultaneously, the current traffic state is input into the large language model safety review agent, which converts it into a structured scene semantic description containing three levels: the current driving scenario, the surrounding vehicle situation, and safety indicators, through domain semantic encoding. The candidate actions are also incorporated into this description, forming the large language model input prompt. Based on pre-set safe and dangerous scenario examples, the large language model performs a safety review of the candidate actions and outputs structured judgment results according to a step-by-step inference process involving collision risk analysis, lane change feasibility analysis, and action rationality analysis. The final action is determined through a consensus fusion mechanism: when a candidate action is deemed safe, it is executed directly; when deemed dangerous, an alternative action is selected from a pre-set set of conservative actions, enabling real-time interception and correction of dangerous actions. This includes the following steps: As in step S102 above, firstly, the state information of the vehicles merging into the scene on the highway ramp and the surrounding vehicles are used as input to the agent. After processing, a complete observation vector is formed. The Actor policy network outputs the probability distribution of each candidate action according to the observation vector. The execution action is obtained by sampling according to the probability distribution and the execution action is executed in the simulation environment to obtain the next state and immediate reward. The state transition experience is stored in the Rollout trajectory buffer. Then, after the empirical data acquisition is completed, the trajectory data in the buffer is processed to calculate the temporal difference error at each time step. The advantage function value is calculated using the generalized advantage estimation method. Calculate the probability ratio between the current strategy and the old strategy. And using the pruning objective function Constrain the policy update magnitude; simultaneously, calculate the physical information loss function based on the current state. The physical information loss function includes kinematic constraint loss, safety constraint loss, and comfort constraint loss, which are incorporated into the total loss function as auxiliary loss terms. Next, the gradient descent method is used to minimize the total loss function. For Actor network parameters and Critic network parameters Joint updates are performed, wherein the total loss function is composed of a weighted combination of the pruning objective function, the value function loss, the entropy regularization term, and the physical information loss function; after each round of updates, the Rollout trajectory buffer is cleared and a new round of environmental interaction and experience collection begins. Then, as in step S103 above, after training the PPO decision model through multiple rounds of environmental interaction and training iterations, it enters the inference phase, where the trained Actor network outputs candidate actions based on the current observations. The current traffic status information is converted into a structured scene semantic description through domain semantic encoding. The candidate actions are then incorporated into the input of the large language model that constitutes the action-scene coupling; the physical constraint parameters used in the training phase are injected into the large language model system instructions in a parameterized form, and a hierarchical modular prompt containing system instructions, safety examples, danger examples, current scene descriptions and judgment instructions is constructed. Next, the current driving stage is determined based on the current lane type identifier, and a threshold mapping function is used. Determine the set of effective safety thresholds The large language model performs step-by-step reasoning on the candidate actions according to three orthogonal safety dimensions: kinematic dimension, spatial dimension, and task dimension, and outputs a closed-loop structured judgment result. It includes the judgment result, the reasoning basis, and the suggested actions; Finally, the final execution action is determined based on the consensus fusion mechanism. If the large language model determines that the candidate action is safe, then the candidate action is executed; if it determines that the candidate action is dangerous, then a preset set of conservative actions is selected. Select alternative actions as Execution is required to ensure the safety of the ramp merging process; Finally, by combining physical information reinforcement learning in the training phase with large language model safety checks in the inference phase, a safe, efficient and interpretable ramp merging decision strategy is obtained that can output under different traffic conditions, as described in step S104 above. This decision strategy is then deployed in the highway ramp merging scenario to achieve safe, efficient and stable merging decision control.

[0159] The effectiveness of the method in this embodiment will be verified through specific experiments below.

[0160] This experiment considers a highway ramp merging scenario consisting of a 550m two-lane main road and parallel ramps. Eight NPC vehicles are randomly distributed on the main road, and each vehicle is randomly initialized at a position 20-100m longitudinally on the ramp, with an initial speed range of 15-25m / s. The experiment is conducted using Python based on the highway-env simulation platform.

[0161] To verify the effectiveness of this method, the proposed PPO+Physics+LLM collaborative decision-making method (i.e., a knowledge-data-driven highway ramp merging decision-making method) was evaluated against two typical comparative methods. The first was the IDM+MOBIL traditional rule-based driving method, serving as the rule-driven baseline; the second was the pure PPO method, i.e., the standard proximal policy optimization method without incorporating a physical information loss function and a large language model security review mechanism, serving as the pure data-driven reinforcement learning baseline. Both reinforcement learning methods underwent policy training for 1500 episodes in the same experimental scenario. Subsequently, the optimal policy model was loaded and its network parameters were frozen. Merging success rate and average speed were used as the core evaluation metrics.

[0162] Depend on Figure 3 As can be seen, the IDM+MOBIL method achieves an merging success rate of 61%, the pure PPO method 71%, and the method in this embodiment reaches 78%. This method improves upon the rule-driven baseline by approximately 17 percentage points and the pure data-driven baseline by approximately 7 percentage points. These results demonstrate that the introduced physical information loss function constrains the agent to learn decision-making strategies consistent with vehicle kinematics during the training phase, while the large language model security review mechanism corrects potentially high-risk decisions in real time during the inference phase. The synergistic effect of these two mechanisms significantly improves the safety and completion rate of the ramp merging task.

[0163] Depend on Figure 4 As can be seen, the average speed of the IDM+MOBIL method is only 9.43 m / s. This is because this rule-based method lacks the ability to predict the timing of traffic flow on the main road during the merging process, and can only passively wait for a safe gap by continuously decelerating, thus significantly sacrificing traffic efficiency and failing to meet the actual operational needs of highways. The average speed of the pure PPO method is 19.75 m / s, and the average speed of the method of this invention is 19.83 m / s. Both can maintain a normal highway traffic speed that matches the traffic flow on the main road, and the average speed of the method of this invention is slightly higher than the baseline of the pure PPO method without sacrificing traffic efficiency. The above results show that the method of this embodiment significantly improves merging safety without sacrificing traffic efficiency.

[0164] In summary, this invention proposes a knowledge-data-driven highway ramp merging decision-making method that combines the adaptive capability of data-driven methods with the safe reasoning capability of knowledge-driven methods. This method can effectively reduce collision risks and improve merging success rate, decision stability, and driving comfort. This method first establishes a simulation environment and a reinforcement learning decision-making environment for highway ramp merging. A state space is constructed, including the vehicle's state, the relative positions and speeds of surrounding vehicles, and safety indicators. An action space is designed, encompassing lane changing, holding, acceleration, and deceleration. Then, a proximal policy optimization algorithm is used to train the ramp merging decision model. During training, a physical information loss function is introduced, integrating vehicle kinematic constraints, safety constraints, and comfort constraints into the policy optimization process to improve the physical consistency, training stability, and safety controllability of the decision model. Finally, in the inference phase, the current traffic state is converted into a structured scene description through domain semantic encoding, and candidate actions are coupled and incorporated into the large language model input. By injecting the physical constraints from the training phase into prompts in a parameterized form, the large language model performs step-by-step safety inference on candidate actions according to kinematic, spatial, and task dimensions, adaptively adjusting the safety judgment criteria based on the current task stage. Finally, a consensus fusion mechanism determines the execution action. When a candidate action is deemed safe, the original action is executed; when a candidate action is deemed dangerous, an alternative action is selected from a pre-set set of conservative actions. This achieves safe, efficient, and interpretable decision-making in highway ramp merging scenarios. Compared with existing technologies, this invention, through the collaborative mechanism of physical information reinforcement learning and large language model, outperforms traditional rule-based methods and pure data-driven methods in both core indicators of merging success rate and average speed, verifying the effectiveness, safety and efficiency of the method in highway ramp merging scenarios.

[0165] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A knowledge-data-driven highway ramp merging decision-making method, characterized in that, Includes the following steps: Establish a highway ramp merging environment and a reinforcement learning decision model environment. The state space of the reinforcement learning decision model environment includes state information representing the interaction between the vehicle and surrounding vehicles when merging on the ramp. The action space of the reinforcement learning decision model environment includes different candidate actions when merging on the ramp. The state information of the highway ramp merging environment is obtained, and the near-end policy optimization algorithm (PPO) is used to train and iterate the decision model. The decision model uses an Actor network to calculate the probability distribution of each candidate action based on the normalized state information, and uses a Critic network to estimate the state value of the current state. During the training and iteration process, a multi-objective reward function is used to calculate the immediate reward, and a physical information loss function is introduced as an auxiliary loss term to jointly establish the total loss function with the main loss function to optimize the parameters of the decision model. After the training iteration is completed, the current state information of the highway ramp merging environment is obtained, the current state information is input into the decision model, and the current state information is converted into a structured scene semantic description. The structured scene semantic description and the candidate actions output by the decision model are used to construct the input prompt of the large language model. The large language model is used to perform a safety check on the candidate actions output by the decision model to obtain the corresponding safety judgment result. The candidate action or the preset conservative action is executed according to the safety judgment result. The trained decision-making model and the large language model are deployed together as a complete decision-making strategy in the highway ramp merging scenario to achieve merging decision control.

2. The knowledge-data-driven highway ramp merging decision method according to claim 1, characterized in that, The highway ramp merging environment includes a two-lane main road and parallel ramps connecting to the main road, and the road is divided into the main road initiation section, transition section, merging conflict section and termination section; the state information characterizing the interaction between the vehicle and surrounding vehicles during ramp merging includes the vehicle's state information, the relative position and relative speed information of surrounding vehicles, and safety index information, the safety index information including at least collision time and distance between vehicles in front and behind; different candidate actions during ramp merging include changing lanes to the left, maintaining the current state, changing lanes to the right, accelerating and decelerating.

3. The knowledge-data-driven highway ramp merging decision method according to claim 1, characterized in that, The multi-objective reward function is output after pruning to ensure numerical stability during training. The multi-objective reward function includes a successful merging reward, a collision penalty, a speed maintenance reward, a comfort penalty, and a driving progress reward to guide the decision model in learning a ramp merging strategy that balances safety, traffic efficiency, and driving comfort. The mathematical expression of the multi-objective reward function is as follows: in, For collision penalties, To successfully import the reward item, To maintain the speed bonus, As a comfort penalty item, This is a reward item for driving progress. , , , , These are the weighting coefficients for each item; The collision penalty item The calculation formula is: The reward for successful import The calculation formula is: in, To add a successful import reward coefficient; The speed maintenance reward item The calculation formula is: in, Current vehicle speed For the target driving speed, This refers to the speed reward weighting coefficient. The comfort penalty item The calculation formula is: in, The weight of the penalty for lane changing actions. Change the penalty weight for general actions; The driving progress reward item The calculation formula is: in, This represents the longitudinal displacement increment per unit time. As a reference distance constant, This is the progress reward weighting coefficient.

4. The knowledge-data-driven highway ramp merging decision method according to claim 1, characterized in that, When the physical information loss function is introduced as an auxiliary loss term and jointly used with the main loss function to establish the parameters of the total loss function optimization decision model, the mathematical expression of the total loss function is as follows: in, This is the physical loss weighting coefficient. Main loss function The physical information loss function; The mathematical expression for the main loss function is as follows: in, This is the value loss weighting coefficient. The entropy regularization weighting coefficient; To prune the objective function, For the value function loss, This is the entropy regularization term; The mathematical expression for the physical information loss function is as follows: in, For kinematic constraint loss terms, For safety constraint loss items, For comfort constraint loss term, , , These are the weighting coefficients for each loss term.

5. The knowledge-data-driven highway ramp merging decision method according to claim 4, characterized in that, The kinematic constraint loss term Used to constrain the consistency between the predicted and actual states of a vehicle, and to reflect the physical rationality of the vehicle's motion, its calculation formula is as follows: in, The current position is predicted based on the state at the previous moment. This is the current actual location. Let be the normalization constant for road length. The predicted speed is inferred from the action. This is the current actual speed. The maximum velocity normalization constant, and These are the corresponding weight coefficients; The security constraint loss item The formula used to constrain vehicles to maintain a safe driving state is as follows: Where TTC is the collision time. min The minimum safe collision time threshold; d This represents the longitudinal distance between the vehicle and the vehicle in front. d min The minimum safe distance threshold; gap is the merging gap. min Minimum inflow gap threshold; w ttc , w d , w gap These are the corresponding penalty weight coefficients; The comfort constraint loss item The formula used to constrain a vehicle to maintain stable driving is as follows: in, The vehicle's current acceleration. The maximum permissible acceleration threshold, The impact intensity is calculated as the ratio of the acceleration difference between adjacent moments to the time step size. The maximum permissible impact threshold, This is a lane change indicator function used to characterize the frequency or number of lane change actions performed by the vehicle, in order to suppress the discomfort caused by frequent lane changes. The value is 1 when the vehicle is currently performing a lane change action, and 0 otherwise. , , These are the weighting coefficients for the acceleration penalty, impact penalty, and lane change penalty, respectively.

6. The knowledge-data-driven highway ramp merging decision method according to claim 1, characterized in that, The structured scene semantic description includes current driving scene information, surrounding vehicle situation information, and safety indicator information. The current driving scene information includes the vehicle's lane type identifier, longitudinal position coordinates, and current driving speed. The surrounding vehicle situation information is the position and motion state of each neighboring vehicle described using relative coordinates. The safety indicator information includes collision time, risk level, and available merging gap. The risk level is a discrete risk level obtained by comprehensively evaluating collision time and vehicle spacing. The available merging gap is the size of the gap between vehicles on the main road that can merge.

7. The knowledge-data-driven highway ramp merging decision method according to claim 6, characterized in that, When constructing the input prompt of the large language model by combining the structured scene semantic description and the candidate actions output by the decision model, the current driving scene information, surrounding vehicle situation information, and safety indicator information are integrated with the candidate actions into structured text in natural language form as input to the large language model. Physical constraint parameters are injected into the large language model system instructions in parameterized form to construct a hierarchical modular prompt containing system instructions, safety examples, danger examples, current scene description, and judgment instructions. The system instructions are system instructions containing parameterized effective safety thresholds, including minimum safe collision time threshold, minimum safe vehicle spacing threshold, minimum safe merging gap threshold, and maximum permissible speed threshold.

8. The knowledge-data-driven highway ramp merging decision method according to claim 7, characterized in that, Before performing security checks on candidate actions output by the decision model using a large language model, the process also includes determining an effective security threshold, including: Based on the lane type identifier in the current driving scenario information, the corresponding driving stage is determined as follows: in, Lane type identification, This indicates that the vehicle is currently in the ramp lane. This indicates that the vehicle is currently in the main lane. This is the merging phase of the ramp. During the main road driving phase; When the driving stage is Furthermore, when the candidate action is a merging direction action, the merging priority effective threshold is used as the effective safety threshold, and when the driving stage is... When the candidate action is a non-merging direction action, the standard effective threshold is used as the effective safety threshold. The minimum safe collision time threshold, minimum safe vehicle spacing threshold, minimum safe merging gap threshold, and maximum allowable speed threshold of the standard effective threshold are all consistent with the corresponding physical constraint parameters used in the physical information loss function during the training iteration. The minimum safe collision time threshold, minimum safe vehicle spacing threshold, minimum safe merging gap threshold, and maximum allowable speed threshold of the merging priority effective threshold are all less than the corresponding thresholds in the standard effective threshold.

9. The knowledge-data-driven highway ramp merging decision method according to claim 7, characterized in that, When using a large language model to perform a security check on the candidate actions output by the decision model and obtain the corresponding security judgment result, specifically, the large language model performs step-by-step reasoning on the candidate actions according to three orthogonal security dimensions: kinematic dimension, spatial dimension, and task dimension, and outputs a closed-structured judgment result, which specifically includes: Based on the collision time and distance to the vehicle in front in the safety indicator information, and combined with the minimum safe collision time threshold and the minimum safe vehicle spacing threshold in the safety threshold, the immediate collision risk of the candidate action in the current motion state is evaluated. When the collision time is lower than the minimum safe collision time threshold, or the distance to the vehicle in front is less than the minimum safe vehicle spacing threshold, the candidate action is judged as dangerous. Otherwise, proceed to the next step. Based on the available merging gap in the safety indicator information and the relative longitudinal distance and relative speed between the vehicle in the target lane and the adjacent vehicles in front and behind in the surrounding vehicle situation information, the safety constraints of the target lane space are evaluated. When the available merging gap is less than the minimum safe merging gap threshold, or when the estimated collision time between the vehicle and the adjacent vehicles in front and behind calculated based on the relative longitudinal distance and relative speed is less than the minimum safe collision time threshold, the candidate action is judged as dangerous. Otherwise, proceed to the next step. Based on the vehicle's current speed, lane type, and candidate actions, the physical rationality of the candidate actions is evaluated. If the candidate action is a deceleration action and the vehicle's current speed is greater than the product of the maximum permissible speed threshold and the high-speed rapid deceleration judgment coefficient, or if the candidate action is an acceleration action and the vehicle's current speed is greater than the maximum permissible speed threshold, or if the candidate action is a lane change action and the vehicle's lane type is identified as a main road, the candidate action is judged as unreasonable. Based on the results of layer-by-layer reasoning, the large language model comprehensively outputs a security judgment conclusion constrained to a closed-structure format, defined as: The judgment result is a binary classification of safe or dangerous. If the candidate action is judged as dangerous or unreasonable, the judgment result is dangerous; otherwise, the judgment result is safe. The reasoning is based on the judgment reasoning derived from the above steps. The suggested action is to adopt the preset conservative action recommended by the large language model when the judgment result is dangerous.

10. The knowledge-data-driven highway ramp merging decision method according to claim 9, characterized in that, When executing a candidate action or a preset conservative action based on the safety judgment result, the following steps are taken: if the judgment result of the large language model is that the candidate action is safe, then the candidate action is executed; if the judgment result is that the candidate action is dangerous, then an alternative action is selected from the preset conservative action set and executed to ensure the safety of the ramp merging process.

Images