A Multi-Expert Autonomous Behavior Decision-Making Method for Highways Based on Scene-Aware Constraints

Abstract

This invention provides a multi-expert autonomous behavior decision-making method for highways based on scene-aware constraints, belonging to the field of autonomous driving technology. It includes: constructing a one-way three-lane straight highway environment; using a bicycle model to describe vehicle kinematics; employing an IDM model to model the longitudinal following behavior of surrounding vehicles, forming a traffic flow with a certain degree of randomness and uncertainty; dividing the driving scenario into multiple sub-scenarios; constructing a unified state space and action space; designing a unified scene constraint cost for each sub-scenarios; introducing scene-aware constraints, improving the DRL algorithm used for each sub-scenarios, and training a corresponding expert model based on the scene-aware constraint SAC algorithm; constructing training and testing sets; and conducting actual operation or simulation testing.

Description

Technical Field

[0001] This invention provides a multi-expert autonomous behavior decision-making method for highways based on scene perception constraints, belonging to the field of autonomous driving technology. Background Technology

[0002] Autonomous vehicle operation generally involves three stages: perception, decision-making and planning, and motion control. This invention focuses on behavioral decision-making and planning in a highway environment, primarily considering ramp merging and exiting behavior decisions and lane-changing behavior decisions, which profoundly affect vehicle driving safety and traffic efficiency. Existing technologies for highway autonomous driving behavioral decision-making mainly fall into two categories: First, rule-based decision-making methods, which manually design rules such as lane priorities, distance thresholds, and safety boundaries to make decisions on ramp merging, lane-changing overtaking, lane-keeping, or ramp exiting behavior based on the relative position and speed of the vehicle and surrounding vehicles. These methods are structurally clear and easy to interpret, but cannot fully adapt to complex and random traffic flows; furthermore, rule design and parameter adjustment rely on experience, making it difficult to balance efficiency and safety under various traffic densities and scenarios. Second, deep reinforcement learning-based decision-making methods, which model the highway behavioral decision-making problem as a reinforcement learning task, using DRL algorithms (such as DQN, DDPG, PPO, SAC, etc.) to directly output the vehicle's longitudinal acceleration and lateral steering angle, achieving near-end-to-end behavioral decision-making. These methods exhibit good adaptability and a certain level of intelligence in simulation environments, but they still have significant shortcomings in training efficiency, security, and generalization ability.

[0003] The existing technology, Chinese invention patent publication CN118238847A, entitled "An Autonomous Lane Changing Decision-Making Method and System Adaptive to Different Driving Styles and Road Surface Environments," is designed for highways or similar multi-lane road environments. Given the road surface adhesion coefficient and historical driving trajectories, it achieves automatic lane changing decisions and trajectory planning considering driving style and road surface adhesion conditions. It is mainly used for obstacle avoidance lane changing and personalized lane changing on straight sections of multi-lane roads in the same direction. This technology has the following drawbacks:

[0004] (1) The applicable scenarios are relatively limited and do not systematically cover decision-making needs in multiple scenarios such as ramp merging / merging.

[0005] (2) The decision-making model mainly imitates humans through supervised learning and lacks the ability to learn and explore in multiple scenarios;

[0006] (3) The strategy structure is in the form of "single model + style tag", lacking specialized strategy experts for different scenarios;

[0007] (4) The road surface adhesion coefficient only appears in the form of embedded analytical constraints, lacking closed-loop adaptive interaction with the upper-level decision-making strategy;

[0008] (5) The interaction modeling mainly focuses on the vehicle in front and the vehicle in the adjacent lane, without fully considering complex traffic interactions and collaborative behaviors;

[0009] (6) The system as a whole still mainly follows the serial structure of “decision + analytical trajectory planning”, with limited synergistic optimization of decision and planning.

[0010] In summary, the existing technology has provided a relatively complete personalized lane-changing solution in terms of driving style recognition, road surface adhesion adaptive constraints, and seventh-order polynomial trajectory. However, it still has shortcomings in multi-scenario unified modeling (especially ramp merging / exiting), multi-expert policy structure, reinforcement learning adaptive capability, and complex multi-vehicle interaction modeling. Summary of the Invention

[0011] This invention provides a multi-expert autonomous behavior decision-making method for highways based on scene-aware constraints, mainly addressing the following technical problems:

[0012] 1. Construct a unified multi-scenario autonomous driving decision-making framework that can output control actions (acceleration, steering wheel angle) in real time and continuously, while handling the following on highways: multi-lane lane change decisions; ramp merging decisions; ramp merging decisions.

[0013] 2. Improve driving efficiency while ensuring safety: Strictly control collision risk and intrusion depth in high-risk scenarios such as merging / exiting ramps; maintain a high average speed and a low lane change frequency in normal cruising and low-risk scenarios.

[0014] 3. Reduce the difficulty and time of reinforcement learning training, and improve the stability and interpretability of the strategy: By using a multi-expert structure and sub-scenario division, each expert focuses on a specific scenario; scene-aware constraint (SAC) is used to explicitly model and optimize safety constraints. This enables the invention to achieve high safety and high traffic efficiency in complex traffic scenarios, maximizing average driving speed while ensuring zero vehicle collision rate and boundary violation rate. Furthermore, it improves the interpretability and controllability of the decision-making process to a certain extent, making the decision logic for each scenario clearer and more verifiable.

[0015] The complete technical solution provided by this invention is as follows:

[0016] A multi-expert autonomous behavior decision-making method for highways based on scene-aware constraints includes the following steps:

[0017] Step S1: Construct a 1000-meter-long highway simulation environment using simulation software to build a one-way three-lane straight highway environment; use a bicycle model to describe vehicle kinematics, with inputs including longitudinal acceleration and front wheel steering angle, and outputs including vehicle longitudinal position, lateral position, speed, and heading angle; use the IDM model to model the longitudinal following behavior of surrounding vehicles, and use the MOBIL model to model the lane-changing behavior of surrounding vehicles, forming a traffic flow with a certain degree of randomness and uncertainty.

[0018] Step S2: Divide driving sub-scenes based on lane occupancy features. Within a preset distance in front of the vehicle, the driving scenario is divided into multiple sub-scenes based on whether there are surrounding vehicles in each lane and the merging / merging scenarios of ramps.

[0019] The vehicle may be in any lane or ramp during its interaction with the environment. Sub-scenario 1: There are no surrounding vehicles in the area ahead, and the vehicle is suitable to drive at high speed in the middle lane; Sub-scenario 2: There are surrounding vehicles in only one lane in the area ahead, and the vehicle should prioritize choosing the empty lane to overtake; Sub-scenario 3: There are surrounding vehicles in two lanes in the area ahead, and the vehicle should choose the only empty lane to overtake; Sub-scenario 4: There are surrounding vehicles in all three lanes, and the vehicle should follow at a lower speed and maintain a safe distance; Sub-scenario 5: Ramps entering the ramp, the vehicle is on the merging ramp and needs to find the right time to merge into the traffic flow; Sub-scenario 6: Ramps exiting the ramp, the vehicle is on or near the exiting ramp and needs to find the right time to exit the traffic flow and drive onto the ramp.

[0020] The above division can be expanded based on parameters such as the number of lanes and detection distance.

[0021] Each time step ( Based on the vehicle's position, its relationship to the ramp position, and the lane occupancy status ahead, sub-scene labels are generated. During the training phase, the data is stored in the replay buffer along with the state.

[0022] Step S3: Construct a unified state space and action space.

[0023] Let the state space be denoted as ( ), and with ( ) indicates a time step The state vector includes, but is not limited to:

[0024] The vehicle's longitudinal position, lateral position, current lane number, longitudinal speed, and lateral speed;

[0025] Within a certain perception range, the presence markers, relative longitudinal positions, relative lateral positions, and relative speeds of several surrounding vehicles sorted by distance;

[0026] Road segment location and road topology coding;

[0027] Current sub-scene tag ( This is used as an additional input during training and is provided by the scene classifier during online deployment.

[0028] Let the action space be denoted as ( ), and with ( ) indicates a time step Control actions;

[0029] Action space ( )include:

[0030] Continuous motion, caused by the longitudinal acceleration of the vehicle ( ) and front wheel steering angle ( Composition;

[0031] The range of values ​​satisfies the physical constraints: ( ), ( ), as a continuous action output.

[0032] Step S4: Design a unified scenario constraint cost for each sub-scenario. By setting different coefficients, the task objective preferences of each sub-scenario can be precisely controlled, and a constraint cost including merging / exit timing, lane centering, driving safety, and driving efficiency can be constructed. The constraint cost at each moment is then... The unified representation is:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] in, , These represent the maximum and minimum values ​​of the vehicle's desired speed range, respectively. Indicates a reward for merging or exiting at ramps; Indicates a lane centering bonus; and These represent collision rewards and lane departure rewards, respectively, and the criteria for determining them are as follows: First, regarding the distance to other vehicles: if the longitudinal distance between the vehicle and other vehicles is less than 20 meters and the lateral distance is less than 1.5 meters, then a near collision is considered to have occurred; Second, regarding the distance to the edge of the road: when the vehicle is traveling in the leftmost or rightmost lane and its lateral deviation from the edge of the road is less than 1.5 meters, then a risk of lane departure is considered to have occurred, and therefore it is considered a near departure. The above distance standards can be precisely adjusted according to actual needs.

[0039] Step S5: Introduce scene-aware constraints, improve the DRL algorithm (taking SAC as an example) used for each sub-scene, and train the corresponding expert model based on the scene-aware constraint SAC algorithm.

[0040] For each sub-scene, an independent Scene-Aware Constraint (SAC) algorithm is constructed, which specifically includes: two Q-value networks, including the current network and the target network; a policy network that outputs the mean and standard deviation of the action distribution; and a replay buffer to store the state, action, reward, and state samples.

[0041] The training process is as follows: In the simulation environment, based on the current sub-scene, only data corresponding to that scene is collected, and interaction with experts in that scene is performed; the expert network parameters are updated using the entropy regularization objective function and the Q-value loss function; expert training is completed according to the preset number of iterations or convergence index. Through the above method, multiple expert models that perform well in their respective sub-scenes are obtained, with each expert only responsible for focusing on solving the task in their corresponding sub-scene.

[0042] In this invention, for the scene-aware constrained SAC algorithm, the constrained Markov decision process (CMDP) framework is introduced to jointly model the reward and constraint costs.

[0043] Note: Rewards and benefits:

[0044]

[0045] in: Representation strategy, in Indicates time step Instant rewards, Indicates the discount factor; constrains cost-return, categorized by scenario:

[0046]

[0047] Among them: Indicates time step The immediate constraint cost, This can be achieved through step S4. The calculation yielded the following result. The goal is:

[0048]

[0049] Where d 𝑧 Cost budget for sub-scene z; Construct scene-aware Lagrange objective function:

[0050]

[0051] Among them: Let Lagrange multipliers be denoted as and the Lagrange objective function be denoted as . And the Lagrange multipliers are designed as state-scene dependent functions:

[0052]

[0053] Wherein: to achieve adaptive constraint weights related to state and scene, Indicates by parameters The output of the multiplier network; NN 𝜓 For a small neural network with parameter φ, [⋅] + This represents the ReLU or softplus function, used to ensure the output is non-negative; thus, in high-risk scenarios, the network will automatically learn larger safety weights 𝜆 𝜓 This allows for an adaptive increase in the penalty of constraint costs;

[0054] The "Lagrange correction reward" is defined as:

[0055]

[0056] In the scene-aware constraint SAC algorithm, using Replace the original SAC Perform Bellman backup and policy update of the soft Q function, representing a Lagrange correction reward; the update target of the soft Q function:

[0057]

[0058] in: Indicates by parameters The stochastic strategy of representation; The critics are soft function, Its parameters, For the target network parameters. Commentator Q: Network loss:

[0059]

[0060] The policy network loss is the objective of maximizing entropy under scenario conditions.

[0061]

[0062] in: As an experience replay buffer; Indicates soft Backup target; with Describe the critic loss function, in Let represent the maximum entropy objective of the policy. The entropy coefficient λ employs an adaptive adjustment mechanism to ensure the policy maintains an appropriate level of exploration in different scenarios. The update objective of the Lagrange multiplier network is:

[0063]

[0064] Among them: Denotes the target of the multiplier network update, where This represents an estimate of cost return or batch cost statistics, and can be compared with the current scenario budget. Compare to update By performing gradient ascent on , when the constraint cost exceeds the budget in a certain scenario, the corresponding ... 𝜓 (𝑠 𝑡 , 𝑡 The penalty for dangerous behavior will increase when the constraint is met for a long period of time; conversely, when the constraint is met for a long period of time, the penalty for dangerous behavior will be appropriately reduced. 𝜓 This allows strategies to pursue higher efficiency within safe limits;

[0065] The specific training process of the Scene-Aware Constraint SAC algorithm includes: interactive sampling in a highway simulation environment containing multiple lanes, ramp merging and ramp exiting; recording (X) at each step. 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑐 𝑡 , 𝑧 𝑡 , 𝑠 𝑡+1 To the experience replay buffer; periodically sample small batches of samples from the buffer and perform the following updates: update the Q-network parameters of each expert using the objective of the scene-aware constrained SAC algorithm; update the policy network parameters of each expert using the policy loss of the scene-aware constrained SAC algorithm; update the objective λ using Lagrange multiplication. 𝜆 (f) Update the scene-aware safety weight network; if adaptive entropy coefficients are used, update f simultaneously.

[0066] Step S6: Construct a scene classifier by using the state data from the multi-expert training process and its corresponding sub-scene labels to construct training and test sets;

[0067] A deep neural network is used as the classifier: the input layer dimension is consistent with the state space; multiple fully connected hidden layers are activated using the ReLU function; the output layer is the number of sub-scenes, and the Softmax function is used to output the probability of each scene; the classifier is trained using supervised learning so that it can accurately output the current sub-scene category given a state input.

[0068] Step S7: Online decision control During the actual operation or simulation test of the autonomous vehicle, the current state is collected at each time step; the current state is input into the scene classifier to obtain the current sub-scene category; according to the classification result, the corresponding expert model is selected; the state is input into the policy network of the expert model to output the action; the action is sent to the vehicle control module to update the vehicle state; the above process is repeated until the task endpoint is reached or the termination condition is triggered.

[0069] Compared with the prior art, the present invention has at least the following beneficial effects:

[0070] (1) Unify the decision-making process for lane changing and ramp merging / merging;

[0071] It handles multi-lane lane changes, ramp merging, and ramp exiting simultaneously within the same multi-expert framework, avoiding the conflicts and inconsistencies caused by the separate design of lane change and ramp behavior in traditional methods.

[0072] (2) The training difficulty and time are significantly reduced through a multi-expert structure;

[0073] Complex tasks are divided into multiple typical sub-scenarios, and each expert only needs to learn the strategy within the corresponding scenario; the state-action distribution is more concentrated, the training is more stable, and the number of training steps required is significantly less than that of a single unified strategy; it is expected that the training time can be reduced by about 50%–60%.

[0074] (3) Introduce Scene Awareness Constraints (SAC) to explicitly ensure safety and comfort;

[0075] Safety and comfort are modeled together as a constraint cost. It controls the risk budget by using Lagrange multipliers; it automatically strengthens safety constraints in high-risk scenarios such as ramp merging / exiting, and appropriately relaxes constraints in ordinary scenarios to achieve a scenario-adaptive safety-efficiency trade-off; compared with the approach of adjusting reward items only through experience weights, this invention has a clearer theoretical basis and stronger interpretability.

[0076] (4) Improve overall driving efficiency and safety;

[0077] In multi-scenario mixed testing, the method of the present invention can achieve a high average speed and a low ramp miss rate while maintaining an extremely low collision rate and a near-zero out-of-bounds rate; the ramp merging process has less interference with mainline traffic and reduces sudden braking and frequent deceleration events.

[0078] (5) The project is simple to implement and has low reasoning overhead;

[0079] Compared to the original DRL algorithm framework, this invention only adds constraint cost calculation and a lightweight Lagrange multiplier network, which has a small impact on inference latency; it is suitable for deployment on actual vehicle computing platforms. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of a highway driving scenario according to the present invention;

[0081] Figure 2 This is a schematic diagram of the system structure for an embodiment;

[0082] Figure 3 This is a schematic diagram of a bicycle model used in an embodiment. Detailed Implementation

[0083] This embodiment provides a multi-expert autonomous behavior decision-making method for highways based on scene-aware constraints, including the following steps:

[0084] Step S1: Using simulation software (taking Highway-env as an example), construct a 1000-meter-long highway simulation environment. Build a one-way three-lane straight highway environment (arranged from left to right), such as... Figure 1 As shown, the lane width is preset to 4 meters. A 100-meter-long single-lane ramp merging entrance is set 100 meters from the starting point (leftmost side), and a 100-meter-long single-lane ramp merging exit is set 100 meters from the ending point (rightmost side). A 5-meter-long and 2-meter-wide autonomous vehicle (autonomous driving vehicle) and several surrounding vehicles are set up. The autonomous vehicle starts at the starting point of the merging ramp, and 100 surrounding vehicles are evenly distributed along the 1000-meter three-lane highway. The system uses... Figure 3 The bicycle model shown describes vehicle kinematics. Figure 3 In bicycle models, with and These represent the vehicle's longitudinal and lateral positions in the global coordinate system, respectively. Indicates the vehicle's heading angle, in Indicates the longitudinal speed of the vehicle; Indicates longitudinal acceleration control input, to Indicates the front wheel steering angle control input; Indicates the vehicle's wheelbase, in and These represent the distances from the center of mass to the front and rear axles, respectively; The above symbols represent the discrete control cycle. They characterize the kinematic update relationship of the vehicle at discrete time steps and provide physical meaning for the definition of state and motion variables in step S3. Inputs include longitudinal acceleration and front wheel steering angle, while outputs include the vehicle's longitudinal position, lateral position, velocity, and heading angle. An IDM model is used to model the longitudinal following behavior of surrounding vehicles, and a MOBIL model is used to model the lane-changing behavior of surrounding vehicles, forming a traffic flow with a certain degree of randomness and uncertainty.

[0085] Step S2: Divide driving sub-scenarios based on lane occupancy characteristics. Within a preset distance (e.g., 100 meters) ahead of the vehicle, the driving scenario is divided into multiple sub-scenarios based on the presence of surrounding vehicles in each lane and merging / exiting scenarios. It's important to note that the vehicle may be in any lane or on any ramp during its interaction with the environment. Sub-scenario 1: No surrounding vehicles ahead; the vehicle is suitable for high-speed driving in the middle lane. Sub-scenario 2: Only one lane ahead has surrounding vehicles; the vehicle prioritizes overtaking in an empty lane. Sub-scenario 3: Two lanes ahead have surrounding vehicles; the vehicle chooses the only empty lane to overtake. Sub-scenario 4: All three lanes have surrounding vehicles; the vehicle follows at a lower speed, maintaining a safe distance. Sub-scenario 5: Ramp merging sub-scenario; the vehicle is on the merging ramp and needs to find the right moment to merge into the traffic flow. Sub-scenario 6: Ramp exit sub-scenario; the vehicle is on or near the exit ramp and needs to find the right moment to exit the traffic flow and enter the ramp. The above divisions can be expanded based on parameters such as the number of lanes and detection distance. Each time step ( Based on the vehicle's position, its relationship to the ramp position, and the lane occupancy status ahead, sub-scene labels are generated. During the training phase, the data is stored in the replay buffer along with the state.

[0086] Step S3: Construct a unified state space and action space.

[0087] state space ( Including but not limited to:

[0088] The vehicle's longitudinal position, lateral position, current lane number, longitudinal speed, and lateral speed;

[0089] Within a certain perception range, the presence markers, relative longitudinal positions, relative lateral positions, and relative speeds of several surrounding vehicles sorted by distance;

[0090] The location of the road segment (such as its distance from ramp entrances and exits) and the road topology code;

[0091] Current sub-scene tag ( This is used as an additional input during training and is provided by the scene classifier during online deployment.

[0092] The dimension of the state vector can be fixed as ( For example, around 25 dimensions.

[0093] Action space ( )include:

[0094] Continuous motion, caused by the longitudinal acceleration of the vehicle ( ) and front wheel steering angle ( Composition;

[0095] The range of values ​​satisfies the physical constraints: ( ), ( ), as a continuous action output.

[0096] Step S4: Design a unified scenario constraint cost for each sub-scenario. By setting different coefficients, the task objective preferences of each sub-scenario can be precisely controlled, and a constraint cost including merging / exit timing, lane centering, driving safety, and driving efficiency can be constructed. The constraint cost at each moment is then... The unified representation is:

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] in, For total constraint cost, For speed efficiency, For inflow and outflow bank items, For road alignment items, For safety items; For the vehicle's speed, The angle between the vehicle's heading and the tangential direction of the road or the centerline of the lane. The longitudinal velocity component along the road direction. and These are the minimum and maximum values ​​of the desired speed range, respectively. This refers to the weighting coefficient for the speed term; This indicates a reward for merging or merging at ramps; a reward is given when a merging or merging operation is successfully completed. Otherwise take , These are the corresponding weighting coefficients; The lane centering bonus is defined as follows: That is, the difference between the lateral coordinates of the vehicle and the center line of the lane. These are the corresponding weighting coefficients; This represents the collision reward sub-item, which is taken when a collision occurs. Otherwise take , For the event of deviating from the road or crossing the boundary, take the following when a deviation from the road or crossing the boundary occurs. Otherwise take The criteria for determining a near-collision are as follows: First, regarding the distance to other vehicles: if the longitudinal distance between the vehicle and other vehicles is less than 20 meters and the lateral distance is less than 1.5 meters, it is considered a near-collision. Second, regarding the distance to the road edge: when the vehicle is traveling in the leftmost or rightmost lane and its lateral deviation from the road edge is less than 1.5 meters, it is considered a potential risk of deviating from the road, and therefore considered a near-deviation. These distance standards can be precisely adjusted according to actual needs. and These are the corresponding penalty weight coefficients. See Table 1 for details on all the parameter settings mentioned above.

[0103] Table 1 Reward Function Settings

[0104]

[0105] Step S5: Introduce scene-aware constraints, improve the DRL algorithm (taking SAC as an example) used for each sub-scene, and train the corresponding expert model based on the scene-aware constraint SAC algorithm.

[0106] For each sub-scene, an independent Scene-Aware Constraint (SAC) algorithm is constructed, which includes: two Q-value networks (one for the current sub-scene and one for the target sub-scene); a policy network (outputting the mean and standard deviation of the action distribution); and a replay buffer to store state, action, reward, and state samples. The training process is as follows: in the simulation environment, based on the current sub-scene, only data corresponding to that sub-scene is collected, and interaction with experts in that sub-scene is performed; the expert network parameters are updated using an entropy regularization objective function and a Q-value loss function; and expert training is completed according to a preset number of iterations or convergence metric. Through this method, multiple expert models that perform well in their respective sub-scenes are obtained, with each expert focusing solely on solving the task within its corresponding sub-scene.

[0107] In this invention, for the scene-aware constrained SAC algorithm, a constrained Markov decision process (CMDP) framework is introduced to jointly model the reward and constraint costs.

[0108] Note: Rewards and benefits:

[0109]

[0110] Constrained cost-return (by scenario):

[0111]

[0112] The goal is:

[0113]

[0114] Where d 𝑧 Let z be the cost budget for the sub-scene. Therefore, this invention constructs a scene-aware Lagrange objective function:

[0115]

[0116] And the Lagrange multipliers are designed as state-scene dependent functions:

[0117]

[0118] Where NN 𝜓 For a small neural network with parameter φ, [⋅] + This represents the ReLU or softplus function, used to ensure the output is non-negative. This allows the network to automatically learn a larger safety weight, 𝜆, in high-risk scenarios such as ramp merging and merging. 𝜓 This allows for an adaptive increase in the penalty of constraint costs.

[0119] In its implementation, the "Lagrange correction reward" is defined as:

[0120]

[0121] In the scene-aware constraint SAC algorithm, using Replace the original SAC Perform Bellman backup and policy update for the soft Q function. The update target for the soft Q function is:

[0122]

[0123] Commentator Q: Network Loss

[0124]

[0125] Policy network loss (maximum entropy objective under scenario conditions):

[0126]

[0127] The entropy coefficient φ can be adaptively adjusted to maintain an appropriate level of exploration in different scenarios. The update objective of the Lagrange multiplier network is:

[0128]

[0129] By performing gradient ascent on , when the constraint cost exceeds the budget in a certain scenario, the corresponding ... 𝜓 (𝑠 𝑡 , 𝑡 The penalty for dangerous behavior will increase, thus strengthening the punishment; conversely, when the constraint is met for a long period of time, the penalty can be appropriately reduced. 𝜓 This allows strategies to pursue higher efficiency within a safe range.

[0130] The specific training process of the Scene-Aware Constraint SAC algorithm includes: interactive sampling in a highway simulation environment containing multiple lanes, ramp merging and ramp exiting; recording (X) at each step. 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑐 𝑡 , 𝑧 𝑡 , 𝑠 𝑡+1 To the experience replay buffer; periodically sample small batches of samples from the buffer and perform the following updates: update the Q-network parameters of each expert using the objective of the scene-aware constrained SAC algorithm; update the policy network parameters of each expert using the policy loss of the scene-aware constrained SAC algorithm; update the objective λ using Lagrange multiplication. 𝜆 (f) Update the scene-aware safety weight network; if adaptive entropy coefficients are used, update f simultaneously.

[0131] Step S6: Construct a scene classifier. Utilize the state data from the multi-expert training process and its corresponding sub-scene labels to construct training and testing sets. Use a deep neural network as the classifier: the input layer dimension is consistent with the state space (e.g., 25 dimensions); multiple fully connected hidden layers are activated using the ReLU function; the output layer is the number of sub-scenes (e.g., 4 neurons), and the Softmax function is used to output the probability of each scene. Train the classifier using supervised learning so that it can accurately output the current sub-scene category given a state input.

[0132] Step S7: Online Decision Control (Multiple Expert Integration) During the actual operation or simulation testing phase of the autonomous vehicle, the current state is collected at each time step; the current state is input into the scene classifier to obtain the current sub-scene category; based on the classification result, the corresponding expert model is selected; the state is input into the policy network of the expert model, and the action (longitudinal acceleration, steering angle) is output; the action is sent to the vehicle control module to update the vehicle state; the above process is repeated until the task endpoint is reached or the termination condition (collision, out of bounds, etc.) is triggered.

[0133] Through the aforementioned multi-expert integration approach, complex highway decision-making tasks are broken down into multiple sub-tasks, with each expert working in their area of ​​expertise, thereby improving overall learning efficiency and decision-making performance.

[0134] The present invention also provides a highway behavior decision-making system for implementing the above method, such as... Figure 2 As shown, it may include, but is not limited to, the following modules (corresponding to) Figure 2 System block diagram):

[0135] The environment modeling module is used to construct a three-lane highway simulation environment; it is used to update the status of surrounding vehicles and the driver vehicle based on vehicle dynamics models, IDM, MOBIL and other models.

[0136] The sub-scene segmentation module determines the current scene category based on the vehicle occupancy information of each lane within a certain distance in front of the vehicle; during the training phase, sub-scene labels are labeled for each sampled state.

[0137] The expert training module includes multiple DRL-based expert training sub-modules; each sub-module samples and updates only in a specified sub-scenario; and provides its own independent policy network and value network.

[0138] The scene classifier training module uses labeled state-scene data to train a scene classification network, outputs a scene recognition model for online decision-making, and calls the corresponding expert model to achieve real-time output of control commands.

[0139] The above modules can be implemented by the same computing platform (such as an on-board computing unit, server or simulation workstation), or by multiple computing units working together.

Claims

1. A multi-expert autonomous behavior decision-making method for highways based on scene-aware constraints, characterized in that, Includes the following steps: Step S1: Construct a 1000-meter-long highway simulation environment using simulation software to build a one-way three-lane straight highway environment; use a bicycle model to describe vehicle kinematics, with inputs including longitudinal acceleration and front wheel steering angle, and outputs including vehicle longitudinal position, lateral position, velocity, and heading angle; use the IDM model to model the longitudinal following behavior of surrounding vehicles, and use the MOBIL model to model the lane-changing behavior of surrounding vehicles, forming a traffic flow with a certain degree of randomness and uncertainty; Step S2: Divide driving sub-scenes based on lane occupancy features. Within a preset distance in front of the vehicle, the driving scene is divided into multiple sub-scenes based on whether there are surrounding vehicles in each lane and the merging and merging scenarios of ramps. Step S3: Construct a unified state space and action space. Step S4: Design a unified scenario constraint cost for each sub-scenario. By setting different coefficients, the task objective preferences of each sub-scenario can be precisely controlled, and a constraint cost including merging and merging timing, lane centering, driving safety and driving efficiency can be constructed. Step S5: Introduce scene-aware constraints, improve the DRL algorithm used for each sub-scene, and train the corresponding expert model based on the scene-aware constraint SAC algorithm. The specific method for step S5 is as follows: For each sub-scene, an independent Scene-Aware Constraint (SAC) algorithm is constructed, which specifically includes: two Q-value networks, including the current network and the target network; a policy network that outputs the mean and standard deviation of the action distribution; and a replay buffer to store the state, action, reward, and state samples. The training process is as follows: In the simulation environment, based on the current sub-scene, only the data corresponding to the scene is collected, and the expert of the scene is interacted with; the expert network parameters are updated using the entropy regularization objective function and the Q-value loss function; the expert training is completed according to the preset number of iterations or convergence index; through the above method, multiple expert models that perform well in their respective sub-scenes are obtained, and each expert is only responsible for focusing on solving the corresponding sub-scene task. For the scene-aware constrained SAC algorithm, the constrained Markov decision process (CMDP) framework is introduced to jointly model the reward and constraint costs. Note: Rewards and benefits: ， in: Representation strategy, in Indicates time step Instant rewards, Indicates the discount factor; constrains cost-return, categorized by scenario: ， Among them: Indicates time step The immediate constraint cost, This can be achieved through step S4. The calculation yielded the following result; the objective is: ， Where d 𝑧 Cost budget for sub-scene z; Construct scene-aware Lagrange objective function: ， Among them: Let Lagrange multipliers be denoted as and the Lagrange objective function be denoted as . The Lagrange multipliers are designed as state-scene dependent functions. ， Wherein: to achieve adaptive constraint weights related to state and scene, Indicates by parameters The output of the multiplier network; NN 𝜓 For a small neural network with parameter φ, [⋅] + This represents the ReLU or softplus function, used to ensure the output is non-negative; thus, in high-risk scenarios, the network will automatically learn a large safety weight 𝜆. 𝜓 This allows for an adaptive increase in the penalty of constraint costs; The "Lagrange correction reward" is defined as: ， In the scene-aware constraint SAC algorithm, using Replace the original SAC Perform Bellman backup and policy update of the soft Q function, representing a Lagrange correction reward; the update target of the soft Q function: ， in: Indicates by parameters The stochastic strategy of representation; The critics are soft function, Its parameters, For the target network parameters; commentator Q network loss: ， The policy network loss is the objective of maximizing entropy under scenario conditions. ， in: As an experience replay buffer; Indicates soft Backup target; with Describe the critic loss function, in Let represent the maximum entropy objective of the policy. The entropy coefficient λ employs an adaptive adjustment mechanism to ensure the policy maintains an appropriate level of exploration in different scenarios. The update objective of the Lagrange multiplier network is: ， Among them: Denotes the target of the multiplier network update, where This represents an estimate of cost return or batch cost statistics, and can be compared with the current scenario budget. Compare to update By performing gradient ascent on , when the constraint cost exceeds the budget in a certain scenario, the corresponding ... 𝜓 (𝑠 𝑡 , 𝑡 The penalty for dangerous behavior will increase when the constraint is met for a long period of time; conversely, when the constraint is met for a long period of time, the penalty for dangerous behavior will be appropriately reduced. 𝜓 This allows strategies to pursue higher efficiency within safe limits; The specific training process of the Scene-Aware Constraint SAC algorithm includes: interactive sampling in a highway simulation environment containing multiple lanes, ramp merging and ramp exiting; recording (X) at each step. 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑐 𝑡 , 𝑧 𝑡 , 𝑠 𝑡+1 To the experience replay buffer; periodically sample small batches of samples from the buffer and perform the following updates: update the Q-network parameters of each expert using the objective of the scene-aware constrained SAC algorithm; update the policy network parameters of each expert using the policy loss of the scene-aware constrained SAC algorithm; update the objective λ using Lagrange multiplication. 𝜆 (f) Update the scene-aware safety weight network; if adaptive entropy coefficients are used, update f simultaneously. Step S6: Construct a scene classifier by using the state data from the multi-expert training process and its corresponding sub-scene labels to construct training and test sets; Step S7: Online decision control During the actual operation or simulation test of the autonomous vehicle, the current state is collected at each time step; the current state is input into the scene classifier to obtain the current sub-scene category; according to the classification result, the corresponding expert model is selected; the state is input into the policy network of the expert model to output the action; the action is sent to the vehicle control module to update the vehicle state; the above process is repeated until the task endpoint is reached or the termination condition is triggered.

2. The method for multi-expert autonomous behavior decision-making on highways based on scene-aware constraints according to claim 1, characterized in that, In step S2, the driving scenario is divided into multiple sub-scenarios, specifically: Sub-scenario 1: There are no surrounding vehicles in the area ahead, and it is suitable for the vehicle to drive at high speed in the middle lane; Sub-scenario 2: There is only one lane in the area ahead with surrounding vehicles, and the vehicle prioritizes choosing the empty lane to overtake; Sub-scenario 3: There are two lanes ahead with surrounding vehicles, and the vehicle chooses the only empty lane to overtake; Sub-scenario 4: There are surrounding vehicles in all three lanes. The vehicle follows at a low speed to maintain a safe distance. Sub-scenario 5: Ramp entry sub-scenario, the vehicle is on the merging ramp and needs to find the right time to merge into the traffic flow; Sub-scenario 6: Ramp exit sub-scenario, the vehicle is on or near the exit ramp and needs to find the right time to exit the traffic flow and drive onto the ramp. Each time step Sub-scene labels are generated based on the vehicle's position, its relationship to the ramp position, and the lane occupancy status ahead. During the training phase, the data is stored in the replay buffer along with the state.

3. The method for multi-expert autonomous behavior decision-making on highways based on scene-aware constraints according to claim 1, characterized in that, Step S3: Construct a unified state space and action space. The state space is specifically as follows: Let the state space be denoted as and with Indicates time step The state vector includes: the vehicle's longitudinal position, lateral position, current lane number, longitudinal speed, and lateral speed; Within a certain perception range, the presence markers, relative longitudinal positions, relative lateral positions, and relative speeds of several surrounding vehicles sorted by distance; Road segment location and road topology coding; Current sub-scene tag It is used as an additional input during training and is provided by the scene classifier during online deployment; Let the action space be denoted as and with Indicates time step The control actions, with a motion space including: continuous actions, determined by the vehicle's longitudinal acceleration. With front wheel steering angle Composition; Value range satisfies physical constraints: , As a continuous action output.

4. The method for multi-expert autonomous behavior decision-making on highways based on scene-aware constraints according to claim 1, characterized in that, Constraint cost at each time step in step S4 The unified representation is: ， ， ， ， ， in, For total constraint cost, For speed efficiency, For the inflow and outflow bank items, For road alignment items, For safety items; For the vehicle's speed, The angle between the vehicle's heading and the tangential direction of the road or the centerline of the lane. The longitudinal velocity component along the road direction. and These are the minimum and maximum values ​​of the desired speed range, respectively. This refers to the weighting coefficient for the speed term; This indicates a reward for merging or merging at ramps; a reward is given when a merging or merging operation is successfully completed. Otherwise take , These are the corresponding weighting coefficients; The lane centering bonus is defined as follows: That is, the difference between the lateral coordinates of the vehicle and the center line of the lane. These are the corresponding weighting coefficients; This represents the collision reward sub-item, which is taken when a collision occurs. Otherwise take , For the event of deviating from the road or crossing the boundary, take the following when a deviation from the road or crossing the boundary occurs. Otherwise take The criteria for determining a near-collision are as follows: First, regarding the distance to other vehicles: if the longitudinal distance between the vehicle and other vehicles is less than 20 meters and the lateral distance is less than 1.5 meters, it is considered a near-collision. Second, regarding the distance to the road edge: when the vehicle is traveling in the leftmost or rightmost lane and its lateral deviation from the road edge is less than 1.5 meters, it is considered a potential risk of deviating from the road and is thus considered a near-deviance. and This represents the corresponding penalty weighting coefficient.

5. The method for multi-expert autonomous behavior decision-making on highways based on scene-aware constraints according to claim 1, characterized in that, The specific method for step S6 is to use a deep neural network as a classifier: the dimension of the input layer is consistent with that of the state space; The system consists of multiple fully connected hidden layers activated by the ReLU function; the output layer represents the number of sub-scenes, and the Softmax function is used to output the probability of each scene; a supervised learning method is used to train the classifier so that it can accurately output the current sub-scene category given a state input.

Images