A lane keeping assistance method based on risk situation awareness constraint policy optimization

Abstract

The present application is suitable for the field of automatic driving assistance technology, and provides a lane keeping assistance method based on risk situation awareness constraint strategy optimization. In the process of strategy reinforcement learning, by implementing quantitative risk assessment and safety constraint strategy optimization method for different levels of risk, the learned strategy can always meet the safety constraint overhead below the pre-set threshold, realizing robust safety control. The method overcomes the shortcomings of high computing resource and time requirement of MPC algorithm; at the same time, it overcomes the problem of poor safety of traditional deep reinforcement learning method used in safety-critical fields. Test results show that the lane keeping assistance method based on risk situation awareness constraint strategy optimization can continuously and stably control the vehicle to travel smoothly and safely in the expected lane, and also takes into account high comfort.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving assistance technology, and particularly relates to a lane keeping assistance method based on risk situation perception constraint strategy optimization. Background Technology

[0002] Lane Keeping Assist (LKA) is a driver assistance technology designed to help drivers keep the vehicle stable within its lane. It uses sensors such as cameras and radar to perceive the vehicle's surroundings, monitor the vehicle's movement in real time, and automatically adjust the vehicle's direction, change steering force or braking force to correct the vehicle's trajectory when it deviates from the lane, ensuring that the vehicle always stays in the correct lane.

[0003] Traditional lane-keeping assist technologies primarily utilize optimization control methods, such as Model Predictive Control (MPC). This method predicts the vehicle's trajectory over a future period using a vehicle dynamics model and formulates an optimal control strategy based on the prediction, enabling the vehicle to accurately track the designated lane. MPC recalculates the trajectory prediction and control strategy at each time step to adapt to real-time environmental changes and vehicle dynamics. This requires building an accurate model based on the vehicle's dynamics, considering factors such as vehicle mass, inertia, and friction. Simultaneously, it needs to acquire road information, such as lane markings and obstacles, to predict the vehicle's position and trajectory on the road. MPC offers several advantages: it accurately predicts and controls the vehicle's trajectory, improving stability and safety, and it adapts well to different road conditions and environmental changes. However, MPC requires real-time calculation and optimization of the control strategy, placing high demands on computational resources and time. Furthermore, it necessitates consideration of the vehicle's actual hardware and sensor performance to ensure the real-time nature and feasibility of the control strategy, resulting in high costs.

[0004] Besides traditional control methods, Deep Reinforcement Learning (DRL) is increasingly gaining attention in lane-keeping assist technology. Compared to traditional MPC methods, DRL is an artificial intelligence-based learning approach that simulates human learning and decision-making processes by building neural network models. It optimizes the accuracy and effectiveness of decisions through trial and error, enabling machines to make correct decisions autonomously. In lane-keeping assist, DRL can train neural network models to learn lane characteristics and patterns, accurately determining lane position and shape to ensure the vehicle stays in the correct lane. DRL can also optimize decision-making through reward mechanisms. Through continuous trial and error and adjustments, DRL can improve the vehicle's autonomous driving capabilities, ultimately achieving precise lane keeping. However, existing reinforcement learning-based lane-keeping assist methods generally lack safety considerations, leading to potential safety hazards for autonomous vehicles in real-world traffic scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a Lane Keeping Assist (LKA) method based on Hazardous Situation-Aware Constrained Policy Optimization (HSCPO), aiming to solve the problems mentioned in the background art.

[0006] The present invention is implemented as follows: a lane keeping assist method based on risk situation awareness constraint strategy optimization includes the following steps:

[0007] Step 1, Environmental Interaction Sampling: The operation of the lane keeping assist system is modeled as a constrained Markov decision process, including a state space, behavior space, performance reward function, and safety overhead function. The vehicle interacts with the environment and observes the current environment in real time, collecting the current state space from the environment. The current policy neural network receives the state and maps it to a behavior, which is then executed by the vehicle. After executing the behavior, the vehicle continues to interact with the environment, thereby obtaining a new state space and receiving reward and overhead values, thus forming a closed loop of vehicle-environment interaction.

[0008] Step 2, Risk Perception Constraint Strategy Optimization: Using trajectory data collected through environmental interaction sampling, the HSCPO method is used to quantify and estimate the level at which policy updates lead to dangerous situations for the vehicle. The safety constraint strategy neural network is updated for different levels of dangerous situations, which can maximize the expected reward while keeping the safety cost function below a specified threshold.

[0009] Step 3: Continue to work together with Step 1 and Step 2 until the trained policy achieves the desired LKA performance.

[0010] In a further technical solution, step 1 includes the following specific steps:

[0011] Step 1.1, State Space and Behavior Space Modeling: State Space Defined as the vehicle's lateral deviation u1 and relative yaw angle u2, i.e. The lateral deviation is the distance between the vehicle and the lane centerline, and the relative yaw angle is the angle between the vehicle's instantaneous speed and the lane tangent. Behavior space definition. Let ψ be the vehicle's heading angle, which is the angle between the vehicle's front wheels and its longitudinal axis (i.e., the direction of travel). The input to the lane-keeping strategy is the current vehicle's lateral deviation u1 and relative yaw angle u2. When the vehicle is about to deviate from its lane, it can accurately output the heading angle ψ to guide the vehicle back into the designated lane.

[0012] Step 1.2, Design of Reward Function and Cost Function: The performance reward function of this invention The reward function, related to the target vehicle's travel distance, lateral deviation, and relative yaw angle, guides the reinforcement learning policy towards updates prioritizing safety and comfort; the safety overhead function... Depending on whether a collision occurs and whether the vehicle deviates from its lane by a critical distance, the overhead function can guide the vehicle to avoid dangerous lane departures.

[0013] In a further technical solution, in step 2, the strategy optimization is constrained by the trust domain formed by the KL (Kullback-Leibler) divergence and the constraint domain formed by the safety overhead. Based on the relative positional relationship between the trust domain and the constraint domain, the probability of the strategy update causing the vehicle to be in a dangerous situation is divided into three levels:

[0014] Risk-free: The update strategy guided by the reward function lies entirely within the intersection of the trust region and the constraint region, meaning the overall safety level of the current strategy is high. In this case, even without imposing safety overhead constraints, updating the strategy only within the trust region defined by the KL divergence will not cause the vehicle to enter a dangerous situation. Therefore, this invention employs the TrustRegion Policy Optimization (TRPO) algorithm to update the strategy.

[0015] Medium risk: The update strategy guided by the reward function lies within the intersection of the trust domain and the constraint domain, while the strategy that maximizes reward updates is outside the constraint domain, meaning the current strategy is at a medium risk level. In this case, updating the strategy without imposing safety constraints would cause the vehicle to enter a dangerous situation; a safe strategy update requires sacrificing some rewards. Therefore, this invention employs a constrained strategy update method.

[0016] High Risk: The update policy guided by the reward function is completely outside the constraint region, and the trust region does not overlap with the constraint region, meaning the current policy is at a high risk level. In this case, policy updates cannot bring it back into the constraint region. Therefore, a backtracking search method is used to find the parameters of an appropriate update policy.

[0017] This invention provides a lane-keeping assist method based on risk situation awareness constraint policy optimization. During policy reinforcement learning, it implements quantitative risk assessment and safety constraint policy optimization methods for different risk levels, ensuring that the learned policy always maintains a safety constraint cost below a pre-set threshold, achieving robust safety control. This method overcomes the drawbacks of the MPC algorithm, which requires high computational resources and time; it also overcomes the problem of poor safety performance of traditional deep reinforcement learning methods in safety-critical fields. Test results show that the lane-keeping assist method based on risk situation awareness constraint policy optimization can continuously and stably control the vehicle to drive smoothly and safely within the expected lane, while also ensuring high comfort. Attached Figure Description

[0018] Figure 1 This is an architecture diagram of a lane keeping assist method based on risk situation awareness constraint strategy optimization provided in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the vehicle's lateral deviation and relative yaw angle.

[0020] Figure 3 A performance analysis diagram of the training process of a lane keeping assist method based on risk situation awareness constraint strategy optimization provided in an embodiment of the present invention;

[0021] Figure 4 This is a simulation verification diagram for a straight-track test scenario;

[0022] Figure 5 This is a diagram showing the lateral deviation during testing on a straight track.

[0023] Figure 6 The diagram shows the relative yaw angle results for the straight-line test;

[0024] Figure 7 The diagram shows the heading angle results for the straight-line test;

[0025] Figure 8 This is a simulation verification diagram for a curve test scenario;

[0026] Figure 9 This is a graph showing the lateral deviation results of the curve test;

[0027] Figure 10 The diagram shows the relative yaw angle results from the curve test.

[0028] Figure 11 This is a diagram showing the heading angle results from the curve test. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0030] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0031] like Figure 1 As shown, a lane keeping assist method based on risk situation awareness constraint strategy optimization is provided in one embodiment of the present invention, including the following steps:

[0032] Step 1, Environmental Interaction Sampling: The operation of the lane keeping assist system is modeled as a constrained Markov decision process, including a state space, behavior space, performance reward function, and safety overhead function. The vehicle interacts with the environment and observes the current environment in real time, collecting the current state space from the environment. The current policy neural network receives the state and maps it to a behavior, which is then executed by the vehicle. After executing the behavior, the vehicle continues to interact with the environment, thereby obtaining a new state space and receiving reward and overhead values, thus forming a closed loop of vehicle-environment interaction.

[0033] Step 2, Risk Perception Constraint Strategy Optimization: Using trajectory data collected through environmental interaction sampling, the HSCPO method is used to quantify and estimate the level at which policy updates lead to dangerous situations for the vehicle. The safety constraint strategy neural network is then updated for different levels of dangerous situations, which can maximize the expected reward while keeping the safety cost function below a specified threshold.

[0034] Step 3: Continue to work together with Step 1 and Step 2 until the trained policy achieves the desired LKA performance.

[0035] In a preferred embodiment of the present invention, step 1 includes the following specific steps:

[0036] Step 1.1, State Space and Behavior Space Modeling: Define the state space as... Where u1 represents the lateral deviation of the vehicle, i.e., the distance between the vehicle and the center line of the lane; u2 represents the relative yaw angle of the vehicle, i.e., the angle between the instantaneous speed of the vehicle and the tangent of the lane line, as shown in the specific diagram. Figure 2 As shown. Both of these data points were obtained by writing functions after collecting Carla simulation data, and were used as inputs to the policy neural network.

[0037] Define behavior space ψ represents the vehicle's heading angle, which is the angle between the vehicle's front wheels and its longitudinal axis (i.e., the direction of travel). The heading angle is the output of the policy neural network. In practice, it represents the heading angle that the model outputs when the vehicle has a certain lateral deviation and relative yaw angle, thereby guiding the vehicle back to the designated lane.

[0038] Step 1.2, Reward and Cost Function Design: The function of the lane keeping assist system is lane departure suppression, that is, controlling the lateral movement of the vehicle when it is about to deviate from its lane, and keeping the vehicle within its original lane. Since the lane keeping assist system only controls the vehicle when it is about to deviate from its lane, and does not intervene in the vehicle's movement at other times, the policy neural network in this method should only take over when the vehicle is about to deviate from its lane. To meet this requirement, a logistic coefficient is set in the reward function. The reward function determined by the lane keeping assist system is:

[0039]

[0040] The standard for determining whether a vehicle is about to deviate from its lane is whether the vehicle is at least 0.2 meters away from the lane line. If it is less than 0.2 meters, it indicates a risk of the vehicle deviating from its lane. When the outer edge of the wheel is less than 0.2 meters from the lane line... The value is 1, otherwise Set the value to 0. Through the above design, when the vehicle has no risk of deviating from its lane, the reward value network does not output a reward value. distance With r deviation These are all logical values; r changes every time the vehicle travels 200m. distance Take 40, otherwise take 0; r is set whenever the outer edge of the wheel exceeds the lane boundary by more than 0.4m. deviation Take -10, otherwise take 0. distance This allows for increased rewards after a vehicle successfully travels a certain distance along the lane. These rewards guide the vehicle to continue along the lane lines to maximize the overall reward. Logical value r deviation The purpose of this is to reduce the reward value when the vehicle deviates from the lane line. In order to maximize the reward, the model will guide the vehicle to avoid deviating from the lane, thus achieving the purpose of the lane keeping assist system. collision The formula for the reward value related to collision is shown below:

[0041]

[0042] When a collision occurs Take -50, when no collision occurs. Take 0.

[0043] Furthermore, in the reward function, r lateral It refers to the reward value related to lateral bias, r. angle This is the reward value for the relative yaw angle. By collecting simulation data and using a function to calculate the offset distance between the current position and the lane centerline, as well as the angle between the current speed and the lane tangent, the reward can be obtained. When the vehicle's relative yaw angle is not equal to 0, it will deviate from the lane and cause an increase in lateral deviation. However, it is generally desirable for the vehicle's lateral deviation to be kept near zero; that is, the greater the lateral deviation, the less reward should be received. Therefore, in the reward function, r... lateral With r angle These represent the absolute values ​​of lateral deviation and relative yaw angle multiplied by a coefficient, respectively. and As shown in formulas (3) and (4), this achieves the goal of reducing the reward function as the vehicle deviates from its intended path.

[0044]

[0045]

[0046] The design of the safety overhead function is related to whether the vehicle deviates from the lane to a critical value, and is expressed as follows:

[0047]

[0048] When the outer edge of the wheel exceeds the lane boundary by more than 0.4m, c deviationChoose 1, otherwise c deviation Take 0; c collision This represents the safety overhead value associated with a collision.

[0049]

[0050] Similar to the reward function, when a collision occurs, Take -50, when no collision occurs. Take 0.

[0051] In a preferred embodiment of the present invention, step 2 includes the following specific steps:

[0052] First, we introduce the optimization objective of the strategy update. Based on the trajectory data τ=(s) collected in the previous step of environmental interaction sampling... t a t r t c t s t+1 If the current strategy is π k The feasible strategy set is Π c The expected discounted return of the reward function is The objective of strategy optimization is:

[0053]

[0054] While achieving the above optimization goals, it is also necessary to ensure that the updated policy meets the safety threshold constraint, using d i To represent this security threshold, the set of feasible strategies is represented as follows:

[0055]

[0056] in Let π represent the expected discounted return of the strategy with respect to the safety overhead function. Then, formula (8) can also be expressed as:

[0057]

[0058] With reward function R(s) t a t s t+1 For example, it is usually used To represent the action-value function, that is, in state s t Below, when taking action a t The corresponding value at that time; use To represent the state-value function, that is, in state s t Below, the expected value of A for all possible actions a; π (s, a) = Q π (s, a)-V π(s, a) represents the dominance function, reflecting the dominance in state s. t The value corresponding to a specific action is the difference between the value of that action and the expected value for all possible actions.

[0059] When solving for the advantage function, it can be approximated by the generalized advantage estimation (GAE):

[0060]

[0061] Where, δ t Temporal difference error can be represented by the state-value function V. π (s) is obtained:

[0062] δ t =r t +γV(s t+1 )-V(s t (11)

[0063] Based on formulas (10) to (11), the incremental formula for the reward function discount advantage report can be derived:

[0064]

[0065] Therefore, the objective of strategy optimization can be written as:

[0066]

[0067] Next, we consider the strategy update method. Strategy optimization is constrained by the trust domain formed by KL degree and the constraint domain formed by security overhead. Based on the relative positional relationship between the trust domain and the constraint domain, three judgment indicators g are set respectively. c , and L, of which The gradient of the safety overhead advantage function is represented. This indicates how close the expected discounted return of the current policy's safety overhead function is to the safety threshold. This represents the intersection of the trust domain and the constraint domain. The probability that a policy update will cause the vehicle to be in a dangerous situation is divided into three levels:

[0068] Risk-free: when g c When L < 1e-8, it indicates that the current policy and its surrounding neighborhood are in a state of low danger potential, and updating the policy in any direction will not be dangerous; when L < 0 and At this point, the reward function-guided update strategy lies entirely within the intersection of the trust region and the constraint region, indicating a high overall safety level for the current strategy. In this scenario, even without imposing safety overhead constraints, updating the strategy only within the trust region bounded by the KL divergence will not lead the vehicle into a dangerous situation. Therefore, the Trust Region Policy Optimization (TRPO) algorithm is used to update the strategy at this safety level. TRPO limits the maximum change in the strategy during each update by using the most recently iterated policy π... k Iteratively update by maximizing the reward advantage function over a local neighborhood:

[0069]

[0070]

[0071] Combine the objective function in formula (14) with After performing second-order Taylor expansions and approximations, the objective function and constraints are obtained as follows:

[0072]

[0073]

[0074] in H represents the gradient of the reward advantage function. The policy parameter update formula is obtained by solving the Hessian matrix after the second-order Taylor expansion according to the KKT (Karush-Kuhn-Tucker) conditions of formula (15):

[0075]

[0076] Medium risk: When L > 0, the update strategy guided by the reward function lies within the intersection of the trust and constraint domains, while the strategy that maximizes reward updates falls outside the constraint domain, meaning the current strategy is at a medium risk level. In this case, updating the strategy without imposing safety constraints would lead the vehicle into a dangerous situation; a safe update requires sacrificing some rewards. Therefore, a constrained policy optimization method is used to update the strategy:

[0077]

[0078]

[0079]

[0080] Similarly, by performing a second-order Taylor expansion and approximating the objective function and constraints in formula (17), we define... The above formula can be approximated as:

[0081]

[0082]

[0083]

[0084] in This represents the gradient of the reward advantage function. H represents the gradient of the safety overhead advantage function. The policy parameter update formula is obtained by solving the Hessian matrix after the second-order Taylor expansion according to the KKT (Karush-Kuhn-Tucker) conditions of formula (18):

[0085]

[0086] Where α is obtained by linear search, and the Lagrange multiplier v in the KKT conditions. * With λ * The calculation is as follows:

[0087]

[0088]

[0089] High risk: If L < and At this point, the update policy guided by the reward function is completely outside the constraint region, and the trust region does not overlap with the constraint region, meaning the current policy is at a high-risk level. In this case, policy updates cannot bring it back into the constraint region. Therefore, a backtracking search method is used to find the parameters of the appropriate update policy:

[0090]

[0091] Based on the above analysis and calculation process, the algorithm flow for optimizing risk situation awareness constraint strategies can be written as shown in Algorithm 1 in Table 1 below. First, initialize as follows... Figure 1 The weight parameters of the policy neural network and the reward and cost value neural network are shown, and the hyperparameters set in the model are also determined. Among them, c d This indicates the initially set indicators used to calculate the security level. The safety threshold, γ represents the discount factor for multi-step reward or cost decay, and δ represents the maximum threshold of KL divergence in the TRPO algorithm, used in solving the judgment condition for the optimal policy. GAE This indicates solving for the coefficients in GAE, lrr and lr c Let n represent the learning rates of the reward value network and the cost value network, respectively. epoch n represents the number of training rounds. step n represents the number of time steps in a sampling process. batch n represents the sample size. total This represents the total number of time steps performed. ξ represents the coefficient in the calculation of the dynamic decay standard deviation. In the early stages of training, the policy requires greater exploratory power, while as the number of training iterations increases, the policy requires greater stability. By setting the dynamic decay standard deviation, this can be achieved, resulting in better training performance and faster convergence, thus enhancing the exploratory nature and stability of the model. The standard deviation is calculated as shown in formula (23):

[0092]

[0093] Table 1 Optimization of Risk Situation Perception Constraint Strategies

[0094]

[0095]

[0096] The second line begins the main loop of the algorithm. First, it collects the trajectory composed of the agent's state, action, reward, and cost (lines 2-7). Then, it calculates the index used to update the network parameters using the trajectory data (lines 8-9). Next, it determines different security levels based on the index calculated in the previous step and updates the network parameters according to different policies based on different security levels (lines 10-17). Finally, it updates the reward value network and cost value network by calculating the loss function and exits the main loop (lines 18-19).

[0097] As a preferred embodiment of the present invention, the above method was verified through simulation. A reinforcement learning model was built based on the PyTorch framework, using Carla version 0.9.11 and an NVIDIA GeForce RTX 3080 Ti GPU. According to the ISO 11270:2014 test standard "Intelligent transport systems—Lane keeping assist systems (LKAS) performance requirements and test procedures", the present invention determined the vehicle speed to be 20 m / s during simulation, and the test was divided into two parts: straight-line test and curve test. In the straight-line deviation test, the vehicle was driven along a straight road at a speed of 20 m / s, and within a random time interval, the vehicle was deviated to the left or right four times at a deviation speed of 0.4 ± 0.2 m / s. If the reinforcement learning strategy could control the vehicle to not exceed 0.4 m from the lane line edge, it was considered successful. In the curve deviation test, the test road was selected as a straight road connected to a curve, with the curve length being 200 m, ensuring the vehicle could travel for more than 5 seconds. The curve was divided into a constant curvature section and a variable curvature section: the curvature of the constant curvature section was 2 × 10⁻⁶. -3 m -1 The variable curvature section is the connecting segment between the straight section and the constant curvature section of the curve, and its curvature increases linearly from 0 to 2×10 with the length of the curve. -3 m -1 The rate of change of curvature dc / ds does not exceed 4×10 -5 m -2 The time step is set to t = 0.1s, and n = 2048 samples are collected in each policy iteration. The training algorithm is stopped after 2048 iterations. The Adam optimizer is used to optimize the reward and cost functions, and the learning rate linearly decays from 2e-4 to 0.

[0098] Training process analysis: Figure 3 The training curves for this method (HSCPO-LKA) are shown in (a) and (b), representing the reward score curve and cost score curve during training, respectively. The graphs show that the reward fluctuates in the first 200 rounds, caused by random lateral disturbances applied during vehicle movement, but the overall reward trend is upward. After 300 rounds, the reward gradually reaches its maximum value, with smaller fluctuations and a gradual stabilization. Conversely, the cost value decreases in the first 200 rounds, while also exhibiting significant fluctuations, again due to random lateral disturbances applied during vehicle movement. The cost value gradually converges after 300 rounds, with smaller fluctuations and a gradual stabilization. These results demonstrate that the vehicle gradually learns a more stable strategy after 300 rounds, enabling safe, comfortable, and efficient lane-keeping assist, proving the practicality and stability of this method.

[0099] Simulation Verification of Test Scenario: The simulation verification first involved a straight-line scenario. The vehicle entered the lane at a speed of 20 m / s. During the journey, at random moments, deviation speeds within the range of 0.4 ± 0.2 m / s were applied to the left and right sides four times, causing the vehicle to deviate from the lane. The vehicle's ability to maintain a distance of 0.4 m from the lane edge was observed. The straight-line test scenario for the vehicle is as follows: Figure 4 As shown in the figure. The test results show that, in straight-line driving conditions, the trained model can accurately output the correct heading angle to the vehicle when it is about to deviate from the lane, thus guiding the vehicle to stay within the lane. 4. Lateral deviation tested in straight-line driving conditions is shown in the figure. Figure 5 As shown, the test lane width was 3.75m, and the selected vehicle model was a Tesla Model 3 with a width of 1.849m. Since the lane departure mitigation function requires ensuring lane departure does not exceed 0.4m beyond the lane edge, the safe lateral deviation should not exceed 1.3325m. The test results show that the lateral deviation in all eight straight-line tests was below 1.3325m, indicating that this method can guide the vehicle within a safe lateral deviation range. The relative yaw angle results for the straight-line tests are shown below. Figure 6 As shown in the figure, the analysis reveals that the lane keeping assist system optimized based on the risk perception constraint strategy can ensure that the relative yaw angle of the vehicle remains within a small range and can adjust the relative yaw angle to close to 0 in a shorter time. This indicates that the present invention can control the vehicle to drive smoothly with a low risk of deviating from the lane. The heading angle results of the straight-line test are as follows: Figure 7 As shown in the image comparison, it can be concluded that the vehicle heading angle controlled by the lane keeping assist system optimized based on the risk perception constraint strategy is always kept at a low level. This indicates that the method can not only control the vehicle to drive smoothly and improve the comfort and smoothness of the vehicle ride, but also control the vehicle to drive within the lane more accurately with a smaller heading angle.

[0100] Then, simulation verification of the cornering scenario was conducted. Based on the ISO testing standards for lane keeping assist systems, two 400m long left-turn and right-turn cornering test scenarios were selected in the simulation environment, ensuring the vehicle could maintain its position for more than 5 seconds. The vehicle's cornering test scenario is as follows: Figure 8 As shown. The vehicle was driven into the lane at a speed of 20 m / s, and it was observed whether the vehicle could maintain its position within the lane. The lateral deviation results of the curve test are as follows. Figure 9 As shown in the figure. It can be seen that when there is a risk of lane departure due to lateral disturbances, this method can guide the vehicle back to its lane in a relatively short time. The relative yaw angle results of the curve test are shown in the figure. Figure 10As shown in the figure, the analysis reveals that the relative yaw angle of the lane keeping assist system optimized based on the risk perception constraint strategy decreases to near 0 within a short time after guiding the vehicle back into the lane, and the relative yaw angle is maintained within a small range. The heading angle results from the curve test are shown below. Figure 11 As shown, the heading angle remained at a low level throughout the cornering test. This indicates that, as with driving on straightaways, this method can consistently and stably control the vehicle to drive smoothly and safely within the intended lane, while maintaining high comfort.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lane keeping assist method based on risk situation awareness constraint strategy optimization, characterized in that, Includes the following steps: Step 1, Environmental Interaction Sampling: The operation of the lane keeping assist system is modeled as a constrained Markov decision process, including state space, behavior space, performance reward function and safety overhead function; the vehicle interacts with the environment and observes the current environment in real time, collects the current state space from the environment, receives the state by the current policy neural network and maps it to behavior, and then the vehicle executes the behavior; After performing this action, the vehicle continues to interact with the environment to obtain a new state space, and at the same time obtains reward value and cost value, thus forming a closed loop of vehicle-environment interaction; Step 2, Risk Perception Constraint Strategy Optimization: Using trajectory data collected by environmental interaction sampling, the HSCPO method is used to quantify and estimate the level at which the policy update leads the vehicle into a dangerous situation. The safety constraint strategy neural network is then updated for different levels of dangerous situation, which can maximize the expected reward while keeping the safety cost function below a specified threshold. Step 3: Continue to work together with Step 1 and Step 2 until the trained policy has the required LKA performance.

2. The lane keeping assist method based on risk situation awareness constraint strategy optimization according to claim 1, characterized in that, Step 1 includes the following specific steps: Step 1.1, State Space and Behavior Space Modeling: Define the state space as... Where u1 represents the lateral deviation of the vehicle; u2 represents the relative yaw angle of the vehicle, that is, the angle between the instantaneous speed of the vehicle and the lane tangent. Both of these data are calculated by writing functions after collecting Carla simulation data and are used as inputs to the policy neural network. Define the behavior space ψ represents the vehicle's heading angle, which is the output of the policy neural network. In practice, it means that when the vehicle has lateral deviation and relative yaw angle, the model will output the corresponding heading angle to guide the vehicle back to the designated lane. Step 1.2, Reward Function and Cost Function Design: A logical value coefficient is set in the reward function. The reward function determined by the lane keeping assist system is: The standard for determining whether a vehicle is about to deviate from its lane is whether the vehicle is at least 0.2 meters away from the lane line. When the outer edge of the wheels is less than 0.2 meters from the lane line... The value is 1, otherwise Set to 0; Through the above design, when the vehicle has no risk of deviating from its lane, the reward value network does not output a reward value for it; r distance With r deviation These are all logical values; r changes every time the vehicle travels 200m. distance Take 40, otherwise take 0; r is set whenever the outer edge of the wheel exceeds the lane boundary by more than 0.4m. deviation Take -10, otherwise take 0; r distance This is used to increase the reward after a vehicle successfully travels a certain distance along the lane. This reward guides the vehicle to move forward along the lane line to maximize the reward; logical value r deviation Its function is to reduce the reward value when a vehicle deviates from the lane line; collision The formula for the reward value related to collision is shown below: When a collision occurs Take -50, when no collision occurs. Set to 0; Furthermore, in the reward function, r lateral It refers to the reward value related to lateral bias, r. angle It is the reward value relative to the yaw angle; in the reward function, r lateral With r angle These represent the absolute values ​​of lateral deviation and relative yaw angle multiplied by a coefficient, respectively. and As shown in formulas (3) and (4): This achieves the goal of reducing the reward function as the vehicle deviates from its intended path. The design of the safety overhead function is related to whether the vehicle deviates from the lane to a critical value, and is expressed as follows: When the outer edge of the wheel exceeds the lane boundary by more than 0.4m, c deviation Choose 1, otherwise c deviation Take 0; c collision Indicates the safety overhead value related to the collision: Similar to the reward function, when a collision occurs, Take -50, when no collision occurs. Take 0.

3. The lane keeping assist method based on risk situation awareness constraint strategy optimization according to claim 2, characterized in that, Step 2 includes the following specific steps: Step 2.1, Optimization objective of strategy update: Based on the trajectory data τ = (s) collected in the environmental interaction sampling in the previous step... t a t r t c t s t+1 If the current strategy is π k The feasible strategy set is Π c The expected discounted return of the reward function is The objective of strategy optimization is: While achieving the above optimization objectives, it is also ensured that the updated policy satisfies the safety threshold constraint, using d i To represent this security threshold, the set of feasible strategies is represented as follows: in Let π represent the expected discounted return of policy as a function of security overhead. Then, Equation 8 can be expressed as: For the reward function R(s) t a t s t+1 ),use This represents the action-value function, i.e., in state s. t Below, when taking action a t The corresponding value at that time; use Represents the state-value function, that is, in state s t Below, the expected value of all possible actions a; A π (s, a) = Q π (s, a)-V π (s, a) represents the dominance function, reflecting the dominance in state s. t The value corresponding to a certain action is selected, and the difference between the expected values ​​of all possible actions is calculated. The advantage function is approximated by the generalized advantage estimation: Where, δ t The timing difference error is represented by the state-value function V. π (s) is obtained: δ t =r t +γV(s t+1 )-V(s t ) (11) Based on formula 10-11, the incremental formula for the reward function discount advantage report is derived as follows: The objective of strategy optimization can then be written as: Step 2.2: Selecting the strategy update method: Strategy optimization is constrained by the trust domain formed by KL degree and the constraint domain formed by security overhead. Based on the relative positional relationship between the trust domain and the constraint domain, three judgment indicators g are set simultaneously. c , and L, of which The gradient of the safety overhead advantage function is represented. This indicates how close the expected discounted return of the current policy's safety overhead function is to the safety threshold. The intersection of the trust domain and the constraint domain is represented, and the probability that a policy update will cause the vehicle to be in a dangerous situation is divided into three levels: Risk-free: when g c When <1e-8, and when L<0 and At that time, the Trust Domain Policy Optimization (TRPO) algorithm is used to update the policy. TRPO limits the maximum change in the policy during each update by using the policy π from the most recent iteration. k Iteratively update by maximizing the reward advantage function over a local neighborhood: Combine the objective function in formula 14 with... After performing second-order Taylor expansions and approximations, the objective function and constraints are obtained as follows: in H represents the gradient of the reward advantage function. The policy parameter update formula is obtained by solving the Hessian matrix after the second-order Taylor expansion according to the KKT conditions in Formula 15: Medium risk: When L > 0, a constrained policy optimization method is used to update the policy. Similarly, by performing a second-order Taylor expansion and approximating the objective function and constraints in Equation 17, we define... The above formula is then approximated as: in This represents the gradient of the reward advantage function. H represents the gradient of the safety overhead advantage function. The policy parameter update formula is obtained by solving the KKT conditions according to Equation 18 after the second-order Taylor expansion of the Hessian matrix: Where α is obtained by linear search, and the Lagrange multiplier v in the KKT conditions. * With λ * The calculation is as follows: High risk: If L < 0 and At this time, the backtracking search method is used to find the parameters of the update strategy:

4. The lane keeping assist method based on risk situation awareness constraint strategy optimization according to claim 3, characterized in that, The process of the risk situation awareness constraint strategy optimization algorithm is as follows: Step 1: First, initialize the weight parameters of the policy neural network and the reward and cost value neural network. and Simultaneously determine the hyperparameter c set in the model. d , γ, δ, λ GAE ,lr r ,lr c n epoch n step n batch n total , and ξ; where c d This indicates the initially set indicators used to calculate the security level. The safety threshold, γ represents the discount factor for multi-step reward or cost decay, δ represents the maximum threshold of KL divergence in the TRPO algorithm, and λ is used in the judgment condition for solving the optimal policy. GAE This indicates the solution for the coefficients in GAE, lr r and lr c Let n represent the learning rates of the reward value network and the cost value network, respectively. epoch n represents the number of training rounds. step n represents the number of time steps in a sampling process. batch n represents the sample size. total This represents the total number of time steps performed. ξ represents the coefficient in the calculation of the dynamic decay standard deviation; Process 2: Start the main algorithm loop: First, collect the trajectory composed of the agent's state, action, reward and cost. Then, calculate the index used to update the network parameters through the trajectory data. Next, determine different security levels based on the calculated index, and perform different strategy network parameter update methods according to different security levels. Step 3: Update the reward value network and cost value network by calculating the loss function and exit the main loop.

5. The lane keeping assist method based on risk situation awareness constraint strategy optimization according to claim 4, characterized in that, In process 1, to improve training effectiveness and convergence speed, a dynamic decay standard deviation is set, where the standard deviation is calculated as follows:

Images