A hierarchical autonomous behavior decision and joint control method for highway and ramp scenarios

Abstract

The application provides a layered autonomous behavior decision and joint control method for highway and ramp scenes, and belongs to the technical field of auxiliary driving. The method comprises the following steps: S1, a training stage; S1.1, collecting expert data and training an IL decision maker; S1.2, training an RL control module; S1.3, joint overall verification and fine tuning; S2, an execution stage; S2.1, environment perception and state construction; S2.2, upper layer imitation learning decision; S2.3, calling a lower layer reinforcement learning control module; S2.4, calling a lower layer MPC control module; S2.5, vehicle execution and environment update. The application makes the control mode switching more adaptive and learnable, reduces the dependence on artificial experience, considers the strategy flexibility and the control feasibility or safety, can process complex tasks such as highway main road lane changing, ramp merging and ramp diverging in the same framework, and can generate smooth and comfortable control behavior under the premise of ensuring safety margin.

Description

TECHNICAL FIELD

[0001] The application provides a hierarchical autonomous behavior decision and joint control method for highway and ramp scenes, belonging to the technical field of assisted driving. BACKGROUND

[0002] With the development of intelligent networked vehicles and advanced driving assistance systems, automatic decision and control of vehicles in highway multi-lane lane changing, ramp merging and ramp exiting scenes have become a key problem. Decision control needs to balance between the following goals: ensure driving safety: avoid rear-end, side-scratching, ramp out, etc.; improve traffic efficiency: try to follow the speed limit, reasonable overtaking, quickly complete merging and splitting; improve ride comfort: avoid frequent and abrupt acceleration and deceleration and severe lateral operation; adapt to complex environment: including high traffic flow, non-cooperative driving vehicles, multi-ramp topology, etc.

[0003] An autonomous lane changing method and system integrating deep reinforcement learning (CN202111114372.3), the method uses deep reinforcement learning (DRL) to realize autonomous lane changing decision, trains a deep reinforcement learning model in a training environment, and makes the target vehicle perform lane keeping and autonomous obstacle avoidance driving. The training of the model uses a convolutional neural network (CNN) to extract features in the training environment, and a policy model obtained by training is used to control the target vehicle to make lane changing decisions. The method mainly consists of the following modules: deep reinforcement learning training module: through the training of the deep reinforcement learning model, the strategy parameters that can make the target vehicle perform lane keeping and autonomous obstacle avoidance driving are learned, and these parameters are saved. Rule-based autonomous driving strategy module: in the training environment, information of a target vehicle that follows a rule-based autonomous driving strategy is added, and simulation tests are performed based on this strategy.

[0004] The disadvantages of this technology are:

[0005] (1) Poor scene scalability; mainly models simple highway lane changing decisions, without considering complex traffic scenes such as multi-lane and ramp merging.

[0006] (2) Single decision, unable to integrate multiple information; this method fails to achieve collaborative optimization of decision and control, which may lead to unstable decision results in complex environments.

[0007] (3) Lower layer mostly uses single RL or rule control, and coupling with fine trajectory control is insufficient; in highway and ramp scenes, vehicle dynamics constraints (lateral or longitudinal acceleration, tire adhesion limit, etc.) are very important, existing RL control often uses simplified dynamics models, lacks collaboration with optimization control methods such as MPC that can explicitly consider constraints, resulting in insufficient strategy landing and safety margin.

[0008] (3) Security is difficult to guarantee in an all-round way; deep reinforcement learning relies on a large number of samples for training, and the training time is relatively long, which is difficult to make instant decisions, and cannot effectively respond to high-risk scenarios. SUMMARY

[0009] The application provides a layered autonomous behavior decision and joint control method for highway and ramp scenes, aiming to:

[0010] (1) Construct a unified multi-scene decision framework: for typical scenes such as highway main road driving, highway main road lane changing, ramp merging, and ramp merging, establish a unified "upper intelligent decision + lower joint control" system, rather than only for single lane changing scenes.

[0011] (2) Introduce a learnable upper decision maker to replace artificial rule arbitration: use imitation learning to train the upper intelligent decision maker to learn the strategy of "when to use RL control and when to use MPC control" from expert data, and realize intelligent switching of control modes in complex scenes.

[0012] (3) Collaborate reinforcement learning and MPC control at the bottom level: learn high-level behavior strategies (such as expected longitudinal speed, target lane, or target merging time, etc.) through the lower reinforcement learning control module, and then perform fine trajectory optimization by the MPC control module under the premise of considering vehicle dynamics constraints and collision constraints, realizing strategy-control integration.

[0013] (4) Improve safety, comfort, and efficiency in multiple scenes: under the premise of ensuring controllable collision risk, improve traffic efficiency (such as successful merging rate and average speed), and improve ride comfort (smooth acceleration and deceleration, smooth lane changing or merging).

[0014] The complete technical solution provided by the application is as follows:

[0015] A layered autonomous behavior decision and joint control method for highway and ramp scenes, comprising the following steps:

[0016] S1, training phase;

[0017] S1.1: Collect expert data and train IL decision maker;

[0018] S1.1: Collect expert data and train imitation learning (Imitation Learning, IL) decision maker;

[0019] The expert collects the mode labels of reinforcement learning (RL) or model prediction control (MPC) or different control strategies selected by the expert in the multiple scenes of "highway main road cruising, highway main road lane changing, ramp merging or merging out", and trains ;

[0020] S1.2: training the RL control module;

[0021] In the simulation environment, the IL decision maker is fixed, and the SAC algorithm is used to train the RL controller;

[0022] S1.3: joint overall verification and fine-tuning;

[0023] IL+RL+MPC is integrated, and closed-loop simulation is carried out under various traffic densities, speed distributions and ramp merging forms, and the reward weight, Q or R weight and mode switching strategy are fine-tuned;

[0024] S2, execution phase;

[0025] S2.1: environment perception and state construction;

[0026] The state vector is constructed by collecting real-time information of the ego vehicle and surrounding vehicles, road topology and ramp position , and the current scene label is determined ;

[0027] S2.2: upper-level imitation learning decision;

[0028] The is obtained by encoding function , input into IL decision maker , output control mode recommendation and determine the current mode ;

[0029] S2.3: call lower-level reinforcement learning control module;

[0030] The RL controller outputs the reference behavior according to ;

[0031] RL reference behavior is converted into MPC reference trajectory or constraint;

[0032] S2.4: call lower-level MPC control module;

[0033] According to the current state and reference information, the optimization objective and constraint condition are constructed;

[0034] Solving for the optimal control sequence in the finite time domain and output the current control action. ;

[0035] S2.5: Vehicle Execution and Environmental Updates;

[0036] Low-level execution control actions When the vehicle's status changes, the environmental perception module updates. The process of executing S2.1–S2.5 in a loop continues.

[0037] This invention employs a system composed of the following modules:

[0038] The system comprises: an environmental perception and state construction module; a scene recognition and feature encoding module; an upper-level intelligent decision-maker based on imitation learning; a lower-level reinforcement learning control module, i.e., an RL controller; a lower-level model prediction control module, i.e., an MPC controller; and a decision coordination and execution module.

[0039] Specifically, the environmental perception and state construction module and the scene recognition and feature encoding module acquire the following information through vehicle sensors and high-precision maps: vehicle status, surrounding vehicle status, and road topology.

[0040] Encode the perceived information into a state vector Typical forms include:

[0041]

[0042] in Indicates time The state vector; Indicates a discrete-time index; superscript Indicates a vehicle; superscript Indicates the first A week's worth of cars; This indicates the number of weekly vehicles included in the status code. ; , This indicates the longitudinal and lateral positions of the vehicle in the road coordinate system, in units of... ; Indicates the vehicle's speed, in units of ; Indicates the vehicle's heading angle, in units of ; , Indicates the first The longitudinal and lateral positions of the vehicle, in units ; Indicates the first Vehicle speed, unit ; Indicates the lane number where the vehicle is located; ramp topology flag, indicating the road type and partition where the ego vehicle is located.

[0043] The scene recognition sub-module determines the current scene as: highway main road cruising, highway main road lane changing candidate, ramp merging area, ramp merging area, and generates a scene label .

[0044] The input and output of the upper intelligent decision maker based on imitation learning are:

[0045] The decision maker input is the current state and the combined feature vector of the scene label :

[0046]

[0047] wherein represents the discrete decision time index; represents the environmental state vector at time ; represents the scene label at time (e.g. lane changing scene, ramp merging scene, etc.); represents the feature encoding function (such as multilayer perception, attention network, etc.); represents the feature vector encoded by for upper-level imitation learning decision.

[0048] The decision maker output is the lower control mode selection vector:

[0049]

[0050] wherein represents the upper-level imitation learning decision maker (classifier) mapping function; represents the mode selection vector (i.e. the adoption probability of each mode) output by the upper layer at time ; represents the probability of recommending the adoption of control; represents the probability of recommending the adoption of control; represents the upper-level input feature vector. The upper layer can use the "maximum value" or set a threshold to determine the specific mode:

[0051]

[0052] The imitation learning training target is to collect examples of expert control mode selection in different scenes through manual driving or offline expert strategy, get supervised labels , and encode them into one-hot vectors ; wherein denotes the time instant the true mode label demonstrated by the expert; denotes the corresponding one-hot vector label; denotes if then otherwise ; denotes if then otherwise .

[0053] Training the imitation learning decision maker Output probability distribution with cross-entropy loss:

[0054]

[0055] where are the decision maker parameters; denotes the predicted output (probability or confidence) of the classifier for time instant ; , denote the predicted and probabilities, respectively; denotes the upper-level imitation learning training objective; denotes the parameters of the upper-level decision maker ; denotes the expectation over the training samples ; denotes the natural logarithm. The upper-level intelligent decision maker achieves data-driven control mode switching by imitating the expert's selection policy for RL or MPC in different scenarios, replacing the artificial rule arbitration.

[0056] Specifically, the lower-level reinforcement learning control module, i.e., the RL controller, is responsible for outputting high-level behavior instructions in complex decision-making scenarios, such as the expected longitudinal speed , target lane, or merging time window;

[0057] State and action definitions:

[0058] The RL state selects similar environmental features as , and adds "history information" or "remaining merging distance" if necessary;

[0059] The RL action vector can be defined as:

[0060]

[0061] where representing time instants lower layer outputted high-level behavior instructions; representing desired longitudinal speed reference values; representing desired lateral position reference values; representing incremental decisions for longitudinal speed reference; representing incremental decisions for lateral position reference;

[0062] reward function and objectives:

[0063] reward function integrates safety, efficiency and comfort:

[0064]

[0065] where:

[0066] representing safety term reward, based on large negative reward set for car-following distance, collision or invasion event;

[0067] representing efficiency term reward, encouraging close to speed limit, smooth completion of merging or lane changing positive reward;

[0068] representing comfort term reward, giving punishment for too large acceleration or yaw rate;

[0069] representing weight coefficients of the above three terms;

[0070] adopting entropy regularized reinforcement learning algorithm, maximizing entropy regularized cumulative return:

[0071]

[0072] where representing optimization objective; representing expectation on trajectory distribution; representing episode length or cumulative return truncation horizon; representing discount factor; representing entropy regularization weight; representing conditional distribution policy of action at state lower layer representing policy entropy, used to encourage exploration, improve robustness;

[0073] interface with MPC:

[0074] RL output as trajectory reference or constraint target of MPC, MPC makes fine trajectory optimization and control instruction generation on this basis.

[0075] Specifically, the lower-layer model predictive control module MPC controller solves the optimal control sequence in a short time domain for vehicle dynamics and constraints, to generate feasible and smooth control actions:

[0076] Vehicle model and state vector:

[0077] Let the vehicle state be represented as:

[0078]

[0079] where is the MPC state vector at time ; is the vehicle position component in the global coordinate system or road coordinate system; is the vehicle velocity; is the vehicle heading angle; is the vector transpose;

[0080] The discrete-time dynamics are:

[0081]

[0082] is the discrete step index within the MPC prediction horizon; is the discrete-time vehicle dynamics update function; , is the system matrix and input matrix linearized at the current operating point;

[0083] or linearized as , and the control quantity is:

[0084]

[0085] is the control input vector of the MPC at the prediction step ; is the longitudinal acceleration control quantity; is the steering control quantity;

[0086] MPC optimization objective:

[0087] Given the RL output or preset reference trajectory , let the prediction horizon length be , and the MPC objective function is:

[0088]

[0089] where , is the weight matrix, respectively weighing the trajectory tracking error and control smoothness; denotes the MPC optimization objective function value; denotes the MPC prediction horizon length; denotes the prediction step of the reference trajectory state; , denotes the weighted two-norm square;

[0090] constraint:

[0091] vehicle dynamics constraint: ;

[0092] road geometry constraint: distance limit between vehicle center and lane boundary or ramp boundary;

[0093] collision safety constraint: collision avoidance distance to other vehicles ;

[0094] where denotes the longitudinal acceleration amplitude upper limit; denotes the steering angle amplitude upper limit; denotes the prediction step distance metric between ego vehicle and surrounding vehicle; denotes the surrounding vehicle index; denotes the minimum safety distance threshold; denotes the surrounding vehicle index;

[0095] By solving the above optimization problem, the control sequence is obtained, only the first control action is executed online , and the optimization is rolled over at the next time instant.

[0096] Specifically, the decision coordination and execution module receives the mode selection result output by the upper IL decision maker ;

[0097] If RL, the RL controller outputs the reference behavior , which is then handed over to the MPC for constrained trajectory optimization and control;

[0098] If MPC, the MPC directly constructs the reference trajectory and solves the control sequence according to the current state and scenario;

[0099] Finally, the optimized control quantity is issued to the vehicle bottom layer execution system. Here denotes the actual control input corresponding to the current time instant issued to the vehicle execution system; , denotes the longitudinal acceleration and steering angle control quantities executed at time instant , respectively.

[0100] The present application has the following remarkable beneficial effects:

[0101] (1) The upper decision is upgraded from "rule arbitration" to "imitation learning intelligent decision"; through the selection logic of IL learning experts in different scenarios for RL or MPC, the artificial threshold and rule design are eliminated, so that the control mode switching is more adaptive, learnable, and the dependence on artificial experience is reduced.

[0102] (2) The lower layer adopts "RL+MPC" collaborative control, rather than single RL or rule control; RL is responsible for high-level behavior and strategy learning, and MPC is responsible for fine trajectory control within the constraints, and the two are clearly divided and tightly coupled, balancing the flexibility of strategy and the feasibility or safety of control.

[0103] (3) Unified processing of multiple scenarios of highway trunk road and ramp merging or merging out; the state design, scene label and upper decision logic of the present application are all modeled for multiple scenarios, which can process complex tasks such as highway trunk road lane changing, ramp merging and ramp splitting in the same framework.

[0104] (4) Safety and comfort are considered; by explicitly introducing safety distance, acceleration and yaw rate constraints in RL reward and MPC target, the present application can generate smooth and comfortable control behavior under the premise of ensuring safety margin.

[0105] (5) The engineering can be more practical; since the bottom layer control is handled by MPC, it is easier to meet the vehicle dynamics and actuator constraints, and it is convenient to deploy on actual vehicle controllers, while the upper IL+RL can be iteratively optimized according to data. BRIEF DESCRIPTION OF DRAWINGS

[0106] Figure 1 is the flowchart of the present application. DETAILED DESCRIPTION

[0107] The present application provides a hierarchical autonomous behavior decision and joint control method for highway and ramp scenarios, which adopts a system composed of the following modules:

[0108] Environment perception and state construction module; scene recognition and feature coding module; upper intelligent decision maker based on imitation learning (IL-Decision); lower layer reinforcement learning control module (RL controller); lower layer model predictive control module (MPC controller); decision coordination and execution module. The overall operation process of the system is as shown in Figure 1 .

[0109] Each module is specifically:

[0110] 1. Environment perception and state construction module and scene recognition and feature coding module

[0111] Obtain by vehicle-mounted sensors and high-precision map: ego vehicle state (position, speed, heading, lane ID), surrounding vehicle state (relative distance, relative speed, lane), road topology (highway main road or ramp, merging point position, speed limit information), etc.

[0112] Encode perception information into state vector , typical forms include:

[0113]

[0114] where represents the state vector at time ; represents the discrete time index; superscript represents the ego vehicle; superscript represents the th surrounding vehicle; represents the number of surrounding vehicles included in the state encoding, ; , represents the longitudinal position and lateral position of the ego vehicle in the road coordinate system, with units of ; represents the ego vehicle speed, with units of ; represents the ego vehicle heading angle, with units of ; , represents the longitudinal position and lateral position of the th surrounding vehicle, with units of ; represents the speed of the th surrounding vehicle, with units of ; represents the lane number in which the ego vehicle is located; represents the ramp topology flag, indicating the type and partition of the road where the ego vehicle is located.

[0115] The scene recognition sub-module determines the current scenario according to the road topology and position: highway main road cruising, highway main road lane changing candidate, ramp merging area, ramp merging area, etc., and generates a scene label .

[0116] 2. Upper intelligent decision maker based on imitation learning

[0117] Input and output:

[0118] Input: current state and scene label combined feature vector:

[0119]

[0120] where denotes the discrete decision time index; denotes the time environment state vector; denotes the time scenario label; denotes the feature encoding function; denotes the feature vector encoded by for the upper-level imitation learning decision;

[0121] Output: lower-level control mode selection vector, for example:

[0122]

[0123] where denotes the upper-level imitation learning decision maker mapping function; denotes the time upper-level output mode selection vector (i.e., the adoption probability of each mode); denotes the probability of recommending the adoption of control; denotes the probability of recommending the adoption of control; the upper level can adopt the "maximum value" or set a threshold to determine the specific mode:

[0124]

[0125] The imitation learning training target is to collect expert examples of selecting control modes in different scenarios through manual driving or offline expert strategies, obtain supervised labels , and encode them into one-hot vectors ; where denotes the time real mode label given by the expert demonstration; denotes the corresponding one-hot vector label; denotes if otherwise ; denotes if otherwise ;

[0126] Imitation learning training target:

[0127] Collect expert examples of selecting control modes in different scenarios through manual driving or offline expert strategies, obtain supervised labels , and encode them into one-hot vectors ;

[0128] training the decision maker output probability distribution , using cross-entropy loss:

[0129]

[0130] where are decision maker parameters; denotes the predicted output of the classifier at time ; , denote the predicted and probabilities, respectively; denotes the upper-level imitation learning training objective; denotes the parameters of the upper-level decision maker ; denotes the expectation over the training samples ; denotes the natural logarithm;

[0131] The upper-level intelligent decision maker achieves data-driven control mode switching by imitating the expert's selection strategy for RL or MPC in different scenarios, replacing artificial rule arbitration.

[0132] 3. Lower-level reinforcement learning control module (RL controller)

[0133] The RL module is mainly responsible for outputting high-level behavior instructions, such as expected longitudinal speed , target lane, or merging time window, in complex decision-making scenarios (such as high-density traffic, complex interaction merging).

[0134] State and action definitions:

[0135] The RL state can use similar environmental features as , and if necessary, add "history information" or "remaining merging distance", etc.

[0136] The RL action vector can be defined as:

[0137]

[0138] denotes the incremental decision for the target longitudinal speed and target lateral displacement (or virtual reference lane center);

[0139] Reward function and objectives:

[0140] The reward function integrates safety, efficiency, and comfort:

[0141]

[0142] where:

[0143] This indicates a safety-related bonus, or a large negative bonus set based on distance from surrounding vehicles, collision, or intrusion events.

[0144] This indicates an efficiency-related reward, which encourages positive rewards for approaching the speed limit and successfully merging or changing lanes.

[0145] This indicates a comfort bonus and penalizes excessive acceleration or yaw rate.

[0146] This represents the weighting coefficients of the three items mentioned above;

[0147] Employ entropy-regularized reinforcement learning algorithms such as Soft Actor-Critic (SAC) to maximize the cumulative reward of entropy regularization:

[0148]

[0149] in express Optimization objectives; This represents the expectation of the trajectory distribution; Indicates the round length or cumulative reward truncation time domain; Indicates the discount factor; Indicates the entropy regularization weight; Indicates the state The next step is to develop a conditional distribution strategy for actions; Represents policy entropy, used to encourage exploration and improve robustness;

[0150] Interface with MPC:

[0151] RL output As a trajectory reference or constraint target for MPC, MPC performs fine trajectory optimization and generates control commands based on this.

[0152] 4. Lower-level Model Predictive Control Module (MPC Controller)

[0153] The MPC module solves for the optimal control sequence in the short time domain based on vehicle dynamics and constraints, and is used to generate feasible and smooth control actions.

[0154] Vehicle model and state vector:

[0155] Using a simplified dynamic monorail model or other applicable vehicle models, the vehicle state is represented as follows:

[0156]

[0157] in Indicates time MPC state vector; represents the position component of the vehicle in global or road coordinate system; represents the vehicle velocity; represents the vehicle heading angle; represents the vector transpose;

[0158] The discrete-time dynamics are:

[0159]

[0160] represents the discrete step index within the MPC prediction horizon; represents the discrete-time vehicle dynamics update function; , represents the system matrix and input matrix linearized or discretized at the current operating point;

[0161] or linearized as , the control quantity is:

[0162]

[0163] represents the control input vector of the MPC at the prediction step ; represents the longitudinal acceleration control quantity; represents the steering control quantity;

[0164] MPC optimization objective:

[0165] Given the RL output or preset reference trajectory , let the prediction horizon length be , the MPC objective function is:

[0166]

[0167] where , is the weight matrix, respectively weighing the trajectory tracking error and the control smoothness; represents the MPC optimization objective function value; represents the MPC prediction horizon length; represents the reference trajectory state at the prediction step ; , represents the weighted two-norm square;

[0168] Constraint conditions:

[0169] including but not limited to:

[0170] Vehicle dynamics constraints: ;

[0171] Road geometry constraints: distance limit between vehicle center and lane boundary or ramp boundary;

[0172] Collision safety constraints: collision avoidance distance to other vehicles .

[0173] where denotes the longitudinal acceleration amplitude upper limit; denotes the steering angle amplitude upper limit; denotes the prediction step distance measure between ego vehicle and surrounding vehicles; denotes the minimum safety distance threshold; denotes the surrounding vehicle index;

[0174] By solving the above optimization problem, the control sequence is obtained, and only the first control action is executed online , and the optimization is rolled over at the next time instant.

[0175] 5. Decision coordination and execution module

[0176] Receive the mode selection result output by the upper IL decision maker ;

[0177] If RL, call the RL controller to output the reference behavior , and then hand it over to the MPC for constrained trajectory optimization and control;

[0178] If MPC, the MPC can directly construct the reference trajectory according to the current state and the scene (such as "the ramp merging must be completed within a certain distance") and solve the control sequence;

[0179] Finally, the optimized control quantity is issued to the vehicle bottom layer execution system. Here denotes the actual control input issued to the vehicle execution system at the current time instant; 、 denote the longitudinal acceleration and steering angle control quantities executed at time , respectively.

[0180] A hierarchical autonomous behavior decision and joint control method for highway and ramp scenes according to the present application is implemented in the following steps:

[0181] I. Training phase

[0182] Step S1.1: Collect expert data and train the IL decision maker; ​

[0183] Collect the mode labels selected by experts in multiple scenarios such as "highway main road cruising, highway main road lane changing, ramp merging or merging out", train .

[0184] Step S1.2: Train the RL control module;

[0185] In the simulation environment, fix the IL decision maker (or only enable RL in specific scenarios), and train the RL controller using algorithms such as SAC.

[0186] Step S1.3: Joint overall verification and fine-tuning;

[0187] Integrate IL+RL+MPC, perform closed-loop simulation under various traffic density, speed distribution, and ramp merging patterns, and fine-tune the reward weight, Q or R weight, and mode switching strategy.

[0188] II. Execution phase

[0189] Step S2.1: Environment perception and state construction;

[0190] Collect real-time information of the ego vehicle and surrounding vehicles, road topology, and ramp position, construct state vector , and determine the current scene label .

[0191] Step S2.2: Upper-level imitation learning decision-making;

[0192] Encode the state vector through the encoding function to obtain the feature , input it into the IL decision maker , and output the control mode recommendation and determine the current mode .

[0193] Step S2.3: Call the lower-level reinforcement learning control module (if RL mode is selected);

[0194] The RL controller outputs the reference behavior such as target speed or target lane or merging time window according to ;

[0195] Convert the RL reference behavior into an MPC reference trajectory or constraint.

[0196] Step S2.4: Call the lower-level MPC control module;

[0197] According to the current state and reference information (derived from RL or directly constructed), construct the optimization objective and constraint conditions;

[0198] Solving optimal control sequence in finite time domain , and output the current control action .

[0199] Step S2.5: the vehicle performs environment update;

[0200] The bottom layer executes the control action , the vehicle state changes, the environment perception module updates , and steps S2.1-S2.5 are executed in a loop.

Claims

1. A hierarchical autonomous behavior decision and joint control method for freeway-ramp scenarios, characterized in that, Comprising the following steps: S1, training phase; S1.1: Collect expert data and train imitation learning IL decision maker; Collect the mode labels of experts in the scenes of "highway main road cruising, highway main road lane changing, ramp merging or merging out" to select reinforcement learning RL or model predictive control MPC or different control strategies, train the decision maker ; S1.2: Train RL control module; Fix IL decision maker in simulation environment, train RL controller with SAC algorithm; S1.3: Joint overall verification and fine-tuning; Integrate IL+RL+MPC, perform closed-loop simulation under various traffic density, speed distribution, ramp merging patterns, fine-tune reward weight, Q or R weight and mode switching strategy; S2, execution phase; S2.1: Environment perception and state construction; Real-time acquisition of vehicle and surrounding vehicle, road topology and ramp position information, construct state vector And determine the current scene label ; S2.2: Upper layer imitation learning decision; Will Encoded function Get features , input IL decider , output control mode recommendation And determine the current mode ; S2.3: Call lower layer reinforcement learning control module; The RL controller is configured to determine the control action based on outputting the reference behavior ; Convert RL reference behavior into MPC reference trajectory or constraint; S2.4: Call lower layer MPC control module; According to the current state With reference information construction optimization target With constraint conditions; Solving optimal control sequence in finite time domain and outputting current control action ; S2.5: Vehicle execution and environment update; The bottom layer executes the control action , the vehicle state changes, the environment perception module updates the next time state , S2.1 - S2.5 are executed in a loop.

2. The hierarchical autonomous behavior decision and joint control method for freeway-ramp scenario according to claim 1, characterized in that, The system composed of the following modules: Environment perception and state construction module; Scene recognition and feature coding module; Upper layer intelligent decision maker based on imitation learning; Lower layer reinforcement learning control module, i.e. RL controller; Lower layer model predictive control module, i.e. MPC controller; Decision coordination and execution module.

3. The hierarchical autonomous behavior decision and joint control method for freeway-ramp scenario according to claim 2, characterized in that, The environment perception and state construction module and the scene recognition and feature coding module obtain the following information through vehicle-mounted sensors and high-precision maps: ego vehicle state, surrounding vehicle state, road topology; Encoding perception information as a state vector Typically, this comprises: ， wherein denotes the state vector at time ; denotes the discrete time index; superscript denotes the ego vehicle; superscript denotes the th surrounding vehicle; denotes the number of surrounding vehicles incorporated into the state encoding, ; , denotes the longitudinal and lateral positions of the ego vehicle in the road coordinate system, in units of ; denotes the ego vehicle speed, in units of ; denotes the ego vehicle heading angle, in units of ; , denotes the longitudinal and lateral positions of the th surrounding vehicle, in units of ; denotes the speed of the th surrounding vehicle, in units of ; denotes the lane number in which the ego vehicle is located; denotes the ramp topology flag, used to indicate the road type and partition in which the ego vehicle is located; The scene recognition sub-module determines the current scene according to the road topology and the position, and generates a scene label .

4. The hierarchical autonomous behavior decision and joint control method for freeway-ramp scenario according to claim 2, wherein, Input and output of the upper layer intelligent decision maker based on imitation learning: The decision maker input is the current state The combination feature vector of the scene label and the scene label: ， wherein denotes the discrete decision time index; denotes the environment state vector at time denotes the scene label at time denotes the feature encoding function; denotes the feature vector encoded by for the upper-level imitation learning decision;​​ The decision maker output is a lower layer control mode selection vector: ， wherein represents the upper layer imitation learning decision maker mapping function; represents the time instant the mode selection vector of the upper layer output, i.e. the adoption probability of each mode; represents the recommended adoption probability of control; represents the recommended adoption probability of control; the upper layer decides the specific mode by "taking the maximum value" or setting a threshold value: ， The imitation learning training target is to collect examples of the expert selecting a control mode in different scenes by manual driving or an offline expert strategy to obtain a supervised label , and is encoded into a one-hot vector ; wherein denotes a moment , and a real mode label given by the expert demonstration; denotes , and a corresponding one-hot vector label; denotes if , then , otherwise ; denotes if , then , otherwise ; Training an imitation learning decision maker Output probability distribution with a cross-entropy loss: ， wherein is a decision maker parameter; denotes the predicted output of the classifier at time ; , denotes the predicted and probability, respectively; denotes the upper-level imitation learning training objective; denotes the parameter of the upper-level decision maker ; denotes the expectation over the training samples ; denotes the natural logarithm; The upper layer intelligent decision maker realizes data-driven control mode switching by imitating the expert's selection strategy for RL or MPC in different scenarios, replacing artificial rule arbitration.

5. The hierarchical autonomous behavior decision and joint control method for freeway-ramp scenario according to claim 2, wherein, The lower reinforcement learning control module, i.e., the RL controller, is responsible for outputting high-level behavior instructions in a complex decision-making scenario, including expected longitudinal speed , target lane or merging time window; State and action definition of RL: RL state selection with similar environmental features; RL The action vector can be defined as: ， wherein represents the time instant lower layer outputted high-level behavior instructions; represents the desired longitudinal speed reference value; represents the desired lateral position reference value; represents the incremental decision of the longitudinal speed reference; represents the incremental decision of the lateral position reference; Reward function and target: The reward function integrates safety, efficiency and comfort: ， Wherein: represents the safety item reward, based on a large negative reward set for a proximity to a vehicle, a collision, or an intrusion event; represents efficiency term reward, encouraging close to speed limit, smooth completion of merging or lane changing positive reward; represents a comfort term reward, which penalizes excessive acceleration or yaw rate; a weight coefficient representing the above three items; An entropy regular reinforcement learning algorithm is used to maximize the entropy regular cumulative return: ， in express Optimization objectives; This represents the expectation of the trajectory distribution; Indicates the round length or cumulative reward truncation time domain; Indicates the discount factor; Indicates the entropy regularization weight; Indicates the state The next step is to develop a conditional distribution strategy for actions; Represents policy entropy, used to encourage exploration and improve robustness; Interface with MPC: RL output As the trajectory reference or constraint target of MPC, MPC makes fine trajectory optimization and control command generation on this basis.

6. The hierarchical autonomous behavior decision and joint control method for freeway-ramp scenario according to claim 2, wherein, The lower layer model predictive control module, i.e. MPC controller, solves the optimal control sequence in a short time domain for vehicle dynamics and constraints, and is used to generate feasible and smooth control actions: Vehicle model and state vector: The vehicle state is represented as: ， wherein denotes the time instant MPC state vector; denotes the position component of the vehicle in a global coordinate system or road coordinate system; denotes the vehicle velocity; denotes the vehicle heading angle; denotes the vector transpose; Discrete-time dynamics: ， denotes a discrete step index within the MPC prediction horizon; denotes a discrete-time vehicle dynamics update function; , denotes the system matrix and input matrix resulting from linearization or discretization of the system at the current operating point; or linearized as , the control quantity is: ， denotes the MPC in the prediction step the control input vector; denotes the longitudinal acceleration control amount; denotes the steering control amount; MPC optimization target: Given RL output or preset reference trajectory , let the prediction horizon length be , the MPC objective function is: ， wherein , is a weight matrix, respectively weighing trajectory tracking error and control smoothness; denotes the MPC optimization objective function value; denotes the MPC prediction horizon length; denotes the predicted state of the reference trajectory at step ; , denotes the weighted two-norm square; Constraint conditions: Vehicle dynamics constraints: ; Road geometry constraint: distance limit between vehicle center and lane boundary or ramp boundary; Collision safety constraints: collision avoidance distance to other vehicles ; wherein represents a longitudinal acceleration amplitude upper limit; represents a steering angle amplitude upper limit; represents a prediction step represents a distance measure of the ego vehicle to the first surrounding vehicle; represents a minimum safety distance threshold value; represents a surrounding vehicle index; By solving the above optimization problem, the control sequence The first control action only is executed online And the optimization is rolled over at the next time instant.

7. The hierarchical autonomous behavior decision and joint control method for freeway-ramp scenario according to claim 2, wherein, The decision cooperation and execution module receives the mode selection result output by the upper-layer IL decision maker ; If RL, the RL controller outputs a reference action which is then passed to the MPC for constrained trajectory optimization and control. If MPC, the MPC directly constructs a reference trajectory from the current state and the scenario and solves for the control sequence. MPC, the MPC directly constructs a reference trajectory from the current state and the scenario and solves for the control sequence. Finally, the optimized control quantity The data is sent to the vehicle's underlying execution system. This indicates the actual time corresponding to the current moment. Control inputs sent to the vehicle's actuator system; , Indicates time respectively The longitudinal acceleration and steering angle control values ​​are executed.

Images