Intelligent vehicle path control method based on improved reinforcement learning and adaptive fusion

Abstract

The invention discloses an intelligent vehicle path control method based on improved reinforcement learning adaptive fusion, and the method comprises the following steps: 1, building a cooperative steering model for an intelligent vehicle, and achieving the cooperative control through rapid convergence natural logarithmic sliding mode control and nonlinear model prediction control; 2, introducing an attention mechanism, using path error information and curvature as state input, and designing an improved control strategy based on near-end strategy optimization reinforcement learning, and 3, using physical information to optimize an objective function, and rapidly obtaining an optimal solution of near-end strategy optimization; and 4, obtaining the control quantity of the intelligent vehicle at the current moment according to the improved near-end strategy optimization control scheme and the control quantity of the coordination controller, and controlling the intelligent vehicle to run. According to the method, by integrating natural logarithm and nonlinear model predictive control, the path tracking precision is improved, the adaptive capacity of the intelligent vehicle in a dynamic scene is enhanced, the influence of tracking errors is reduced, the convergence speed of reinforcement learning is improved, and the real-time performance and stability of intelligent vehicle tracking are improved.

Description

Technical Field

[0001] The invention belongs to the field of robotics technology, and in particular relates to an intelligent vehicle path control method with improved reinforcement learning adaptive fusion. Background Art

[0002] Autonomous vehicles (AVs) have received great attention in recent years, offering a promising solution to improve road safety, traffic efficiency, and fuel economy. As a result, autonomous driving technologies that integrate positioning, sensing, path planning, and path tracking have been widely used in AVs. As a key technology that can independently complete tasks without human intervention, path tracking plays an important role in autonomous vehicles. This means that achieving high-precision path tracking is essential to ensure the safety of intelligent transportation systems. However, the real world is often complex, dynamic, and unpredictable, which poses a challenge to existing methods to achieve robustness and accurate tracking.

[0003] As a classic nonlinear system, path following techniques to solve the under-actuation problem associated with AVs have been extensively explored in related research. To improve the path following performance, various control methods have been developed. For example, PP is popular in autonomous steering control due to its simplicity. However, PP has difficulty in finding the optimal forward distance. To address this challenge, other model-based controllers such as LQR, SMC, and MPC methods have been developed to achieve accurate tracking. However, LQR does not consider physical constraints, which limits its practical application. To address this issue, linear MPC is used for AV tracking through online optimization. However, linear models rely heavily on accurate modeling, which is difficult to achieve in practical applications. Fortunately, due to its input-output nonlinear characteristics, nonlinear MPC is widely used in autonomous vehicle path tracking in large curvature scenarios. A comprehensive analysis of MPC and NMPC shows that NMPC shows advantages when the positioning error reaches the centimeter level. Although NMPC shows stronger adaptability to nonlinear systems, its real-time performance is still poor. To address this issue, researchers used genetic algorithms to optimize NMPC in real time to enhance its solution time capability. The researchers used the ACADO toolkit to solve the nonlinear optimal control problem and improved the convergence speed of NMPC. Due to the limitation of the solution rate, NMPC has insufficient stability and mediocre tracking performance, especially in cornering scenarios where understeer is prone to occur. Therefore, in order to achieve fast convergence of path tracking, sliding mode control (SMC) provides an effective solution. Sliding mode control is widely used in autonomous vehicle systems due to its stable performance and ease of practical application. However, in linear SMC (LSM), the tracking error can only converge to zero asymptotically in infinite time. In order to improve the convergence speed, terminal sliding mode (TSM) control is introduced, which uses a nonlinear sliding surface. However, a major limitation of TSM is the singularity problem. In addition, chattering remains a major challenge in TSM / NTSM control. Fortunately, the researchers proposed the global natural logarithm function (LnSM) to improve the system chattering. However, due to the characteristics of the natural function, the convergence speed of LnSM is significantly slower than that of LSM. Due to the limitations in convergence speed and chatter, reinforcement learning (RL) is widely used in path tracking control due to its ability to interact with the environment and adjust future behavior based on action feedback. Although RL has shown great potential in adapting to dynamic environments, its application is limited by some challenges, such as dependence on large amounts of training data and instability of significant tracking errors. Based on the above analysis, the tracking performance of different algorithms has its own characteristics. It is very challenging for a single controller to effectively track various scenarios and states of autonomous driving.

[0004] In order to achieve accurate tracking in different scenarios and states, a combination method is an effective solution. For example, researchers combined PP and PID in a complementary way. Researchers proposed a controller integrating MPC with hybrid PID and LQR for multi-mode tracking control. However, it only focused on the path curvature without considering the impact of different tracking errors. To address this problem, researchers integrated the PP and MPC methods and tested them under dynamic error conditions. However, it is still a challenge to properly adjust the weights of different algorithms to adapt to dynamic scenarios. This means that the tracking method should be optimized according to the scenario and state for effective execution. Therefore, some researchers have studied RL-based combination methods to improve flexibility in dynamic scenarios and states while maintaining tracking performance. For example, researchers proposed a method that combines baseline control with RL, but it does not consider different weights. Subsequently, researchers proposed a human-machine cooperative control system that combines TD3 with an optimal preview drive model and MPC. However, the paper partially relies on steering angles obtained from humans, so a large number of empirical data sets are required. In addition, the TD3 algorithm used in the paper is very sensitive to the choice of hyperparameters. Therefore, the researchers proposed an adaptive weight adjustment method based on PPO to balance smoothness and accuracy by adjusting the weight between PID and PP. However, as a model-free algorithm, PPO makes decisions based on single state information under model uncertainty, which may increase tracking risks in complex scenarios.

[0005] Although these studies have been applied to a certain extent, there are still certain challenges for proximal policy optimization control, namely, slow convergence speed and more data required. Especially for intelligent vehicle automatic control, its path tracking performance mainly depends on the real-time control of steering. The traditional proximal policy optimization path tracking method has poor dynamics in actual engineering, which in turn affects the tracking accuracy. Therefore, it is necessary to improve the real-time performance of path tracking by improving the proximal policy optimization algorithm. Summary of the invention

[0006] To solve the above problems, the present invention discloses an intelligent vehicle path control method with improved reinforcement learning adaptive fusion, which improves the path tracking accuracy and enhances the adaptability of intelligent vehicles in dynamic scenes by integrating natural logarithm and nonlinear model predictive control, reduces the impact of tracking errors, increases the convergence speed of reinforcement learning, and improves the real-time and stability of intelligent vehicle tracking.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] An intelligent vehicle path control method based on improved reinforcement learning adaptive fusion, the specific steps are as follows:

[0009] Step 1: Establish the error model of intelligent vehicle:

[0010]

[0011] Where X e Describes the longitudinal error, Y e represents the lateral error, ψ e represents the vehicle heading angle error, L represents the vehicle wheelbase, v represents the vehicle speed, and δ represents the vehicle front wheel steering angle;

[0012] Step 2: Design a fast natural logarithmic sliding mode controller for the intelligent vehicle, including the following steps:

[0013] (2.1) Establish the fast natural logarithmic sliding surface of the vehicle:

[0014]

[0015] Where ζ0, ζ1 and ζ2 are all positive numbers, and ln(·) represents the natural logarithm function.

[0016] (2.2) Establish the fast-converging sliding surface approach law:

[0017]

[0018] in C1 , C2 are all positive numbers and satisfy 0≤ C1 ≤1, C2 ≥1.

[0019] (2.3) According to the sliding surface and reaching law, the steering angle of the vehicle sliding mode controller is calculated:

[0020]

[0021]

[0022] Step 3: Design a nonlinear model predictive controller for an intelligent vehicle, including the following steps:

[0023] (3.1) Establish a prediction model for nonlinear model prediction:

[0024]

[0025] Where Yte represents the lateral error of the fitting curve, b is the quadratic term coefficient of the polynomial curve, z(t) = k / (1+k 2 ), k is the slope of the reference point.

[0026] (3.2) Establish the steering constraint predicted by the nonlinear model:

[0027] δ min ≤δ(k+n|k)≤δ max n=1,2…N (7)

[0028] Δδ min ≤Δδ(k+n|k)≤Δδ max (8)

[0029] (3.3) Establishing the objective function of nonlinear model prediction

[0030]

[0031] (3.4) Establish the steering angle predicted by the nonlinear model:

[0032] δ2(t)=δ2(t-1)+Δδ2(t) (10)

[0033] Step 4: Establish the proximal policy optimization fusion controller of the intelligent vehicle:

[0034]

[0035] Where δ1 represents the output steering angle of fast sliding mode control, and δ2 represents the output steering angle of nonlinear model predictive control.

[0036] Step 5: Use the proximal strategy optimization algorithm to quickly solve the problem in step 4 and The optimal solution is to predict the optimal state of the intelligent vehicle at the future moment and calculate the optimal parameters at the current moment.

[0037] Step 6: Set the current time Substitute the optimal gain parameters into the proximal strategy optimization fusion control model of the intelligent vehicle to obtain the optimal expected steering angle δ of the intelligent vehicle opt , control the operation of the intelligent vehicle; determine whether to switch the reference point, jump to step 2 to continue the control at the next moment, until the intelligent vehicle reaches the end of the reference path.

[0038] As a further improvement of the present invention, the intelligent vehicle is an unmanned vehicle.

[0039] As a further improvement of the present invention, the step 5 comprises:

[0040] (5.1) Establish the cumulative return factor:

[0041]

[0042] Where γ represents the discount factor, and 0≤γ≤1, R t represents the reward at time t.

[0043] (5.2) The state value function and action value function are used to evaluate the expected returns of all possible decision sequences after the intelligent vehicle starts to take action:

[0044]

[0045]

[0046] Where S t represents the state at time t, A t Represents the action at time t.

[0047] (5.3) Establish the advantage function under the selected action:

[0048] A π (s t ,a t )=Q π (s t ,a t )-V π (s t ) (15)

[0049] (5.4) Establish the estimation function of the advantage function:

[0050]

[0051] (5.5) Establish the policy gradient loss function:

[0052]

[0053]

[0054] Where r represents the probability ratio of the new and old strategies, and ε represents the clipping factor.

[0055] (5.5) Establish a state-decomposed attention mechanism to better focus on the impact of curvature and error on action selection:

[0056]

[0057] Where Q Ni , K Ni represents Q and K in the attention mechanism, and d represents the corresponding dimension.

[0058] (5.6) Construct the loss function based on the physical information of the intelligent vehicle:

[0059]

[0060] in represents the error between the predicted model and the actual model, represents the data observation error, γpinn and γ data Indicates the corresponding coefficient.

[0061] Finally get and The optimal parameters

[0062] As a further improvement of the present invention, step 6 includes: the optimal parameters obtained by the near-end strategy optimization algorithm Bring in the fusion control algorithm to calculate the control amount of the intelligent vehicle at the current moment:

[0063]

[0064] The optimal steering angle δ of the intelligent vehicle at the current moment can be obtained opt , to control the operation of smart vehicles.

[0065] The beneficial effects of the present invention are:

[0066] The intelligent vehicle path control method based on improved reinforcement learning adaptive fusion described in the present invention uses the error information and path curvature as the state input for generating action output through the attention mechanism, which greatly improves the adaptability of the proximal policy optimization algorithm to dynamic scenes and states. The proposed improved proximal policy optimization algorithm combines the neural network with the vehicle model, extracts the intrinsic principles and physical constraints, and achieves more reliable and stable tracking in dynamic scenes and states. This capability enhances the generalization performance of the intelligent vehicle model and significantly improves the stability and adaptability of the proximal policy optimization in the path tracking system in dynamic scenes and states, and can be effectively applied to the adaptive path tracking control of intelligent vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention discloses a flow chart of an intelligent vehicle path control method based on improved reinforcement learning adaptive fusion. DETAILED DESCRIPTION

[0068] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0069] The present invention discloses an intelligent vehicle path control method with improved reinforcement learning adaptive fusion, which can adaptively adjust the improved reinforcement learning control model according to the operation state of the intelligent vehicle, obtain the optimal intelligent vehicle steering angle, ensure the intelligent vehicle path tracking effect, and realize the precise operation of the intelligent vehicle.

[0070] As an embodiment of the present invention, the present invention discloses an intelligent vehicle path control method for improving reinforcement learning adaptive fusion, wherein the intelligent vehicle is an unmanned vehicle, and the flow chart is as follows: Figure 1 As shown, including:

[0071] Step 1: Establish the error model of intelligent vehicle:

[0072]

[0073] Where X e Describes the longitudinal error, Y e represents the lateral error, ψ e represents the vehicle heading angle error, L represents the vehicle wheelbase, v represents the vehicle speed, and δ represents the vehicle front wheel steering angle;

[0074] Step 2: Design a fast natural logarithmic sliding mode controller for the intelligent vehicle, including the following steps:

[0075] (2.1) Establish the fast natural logarithmic sliding surface of the vehicle:

[0076]

[0077] Where ζ0, ζ1 and ζ2 are all positive numbers, and ln(·) represents the natural logarithm function.

[0078] (2.2) Establish the fast-converging sliding surface approach law:

[0079]

[0080] in C1 , C2 are all positive numbers and satisfy 0≤ C1 ≤1, C2 ≥1.

[0081] (2.3) According to the sliding surface and reaching law, the steering angle of the vehicle sliding mode controller is calculated:

[0082]

[0083]

[0084] Step 3: Design a nonlinear model predictive controller for an intelligent vehicle, including the following steps:

[0085] (3.1) Establish a prediction model for nonlinear model prediction:

[0086]

[0087] Where Yte represents the lateral error of the fitting curve, b is the quadratic term coefficient of the polynomial curve, z(t) = k / (1+k 2 ), k is the slope of the reference point.

[0088] (3.2) Establish the steering constraint predicted by the nonlinear model:

[0089] δ min ≤δ(k+n|k)≤δ max n=1,2…N (7)

[0090] Δδ min ≤Δδ(k+n|k)≤Δδ max (8)

[0091] (3.3) Establishing the objective function of nonlinear model prediction

[0092]

[0093] (3.4) Establish the steering angle predicted by the nonlinear model:

[0094] δ2(t)=δ2(t-1)+Δδ2(t) (10)

[0095] Step 4: Establish the proximal policy optimization fusion controller of the intelligent vehicle:

[0096]

[0097] Where δ1 represents the output steering angle of fast sliding mode control, and δ2 represents the output steering angle of nonlinear model predictive control.

[0098] Step 5: Use the proximal strategy optimization algorithm to quickly solve the problem in step 4 and The optimal solution is to predict the optimal state of the intelligent vehicle at the future moment and calculate the optimal parameters at the current moment.

[0099] Step 6: Set the current time Substitute the optimal gain parameters into the proximal strategy optimization fusion control model of the intelligent vehicle to obtain the optimal expected steering angle δ of the intelligent vehicle opt , control the operation of the intelligent vehicle; determine whether to switch the reference point, jump to step 2 to continue the control at the next moment, until the intelligent vehicle reaches the end of the reference path.

[0100] As a further improvement of the present invention, the intelligent vehicle is an unmanned vehicle.

[0101] As a further improvement of the present invention, the step 5 comprises:

[0102] (5.1) Establish the cumulative return factor:

[0103]

[0104] Where γ represents the discount factor, and 0≤γ≤1, R t represents the reward at time t.

[0105] (5.2) The state value function and action value function are used to evaluate the expected returns of all possible decision sequences after the intelligent vehicle starts to take action:

[0106]

[0107]

[0108] Where S t represents the state at time t, A t Represents the action at time t.

[0109] (5.3) Establish the advantage function under the selected action:

[0110] A π (s t ,a t )=Q π (s t ,a t )-V π (s t ) (15)

[0111] (5.4) Establish the estimation function of the advantage function:

[0112]

[0113] (5.5) Establish the policy gradient loss function:

[0114]

[0115]

[0116] Where r represents the probability ratio of the new and old strategies, and ε represents the clipping factor.

[0117] (5.5) Establish a state-decomposed attention mechanism to better focus on the impact of curvature and error on action selection:

[0118]

[0119] Where Q Ni , K Ni represents Q and K in the attention mechanism, and d represents the corresponding dimension.

[0120] (5.6) Construct the loss function based on the physical information of the intelligent vehicle:

[0121]

[0122] in represents the error between the predicted model and the actual model, represents the data observation error, γ pinn and γ data Indicates the corresponding coefficient.

[0123] Finally get and The optimal parameters

[0124] As a further improvement of the present invention, step 6 includes: the optimal parameters obtained by the near-end strategy optimization algorithm Bring in the fusion control algorithm to calculate the control amount of the intelligent vehicle at the current moment:

[0125]

[0126] The optimal steering angle δ of the intelligent vehicle at the current moment can be obtained opt , to control the operation of smart vehicles.

[0127] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. An intelligent vehicle path control method based on improved reinforcement learning adaptive fusion, characterized in that: The specific steps are as follows: Step 1: Establish the error model of intelligent vehicle: Where X e Describes the longitudinal error, Y e represents the lateral error, ψ e represents the vehicle heading angle error, L represents the vehicle wheelbase, v represents the vehicle speed, and δ represents the vehicle front wheel steering angle; Step 2: Design a fast natural logarithmic sliding mode controller for the intelligent vehicle, including the following steps: (2.1) Establish the fast natural logarithmic sliding surface of the vehicle: Where ζ0, ζ1 and ζ2 are all positive numbers, and ln(·) represents the natural logarithm function; (2.2) Establish the fast-converging sliding surface approach law: in are all positive numbers and satisfy (2.3) According to the sliding surface and reaching law, the steering angle of the vehicle sliding mode controller is calculated: Step 3: Design a nonlinear model predictive controller for an intelligent vehicle, including the following steps: (3.1) Establish a prediction model for nonlinear model prediction: Where Yte represents the lateral error of the fitting curve, b is the quadratic term coefficient of the polynomial curve, z(t) = k / (1+k 2 ), k is the slope of the reference point; (3.2) Establish the steering constraint predicted by the nonlinear model: d min ≤δ(k+n∣k)≤δ max n=1,2…N (7) Dd min ≤Δδ(k+n∣k)≤Δδ max (8) (3.3) Establishing the objective function of nonlinear model prediction (3.4) Establish the steering angle predicted by the nonlinear model: δ2(t)=δ2(t-1)+Δδ2(t) (10) Step 4: Establish the proximal policy optimization fusion controller of the intelligent vehicle: Where δ1 represents the output steering angle of fast sliding mode control, and δ2 represents the output steering angle of nonlinear model predictive control; Step 5: Use the proximal strategy optimization algorithm to quickly solve the problem in step 4 and The optimal solution is to predict the optimal state of the intelligent vehicle at the future moment and calculate the optimal parameters at the current moment. Step 6: Set the current time Substitute the optimal gain parameters into the proximal strategy optimization fusion control model of the intelligent vehicle to obtain the optimal expected steering angle δ of the intelligent vehicle opt , control the operation of the intelligent vehicle; determine whether to switch the reference point, jump to step 2 to continue the control at the next moment, until the intelligent vehicle reaches the end of the reference path.

2. According to claim 1, the intelligent vehicle path control method based on improved reinforcement learning adaptive fusion is characterized in that: The intelligent vehicle is an unmanned vehicle.

3. The intelligent vehicle path control method based on improved reinforcement learning adaptive fusion according to claim 1 is characterized in that: The step 5 comprises: (5.1) Establish the cumulative return factor: Where γ represents the discount factor, and 0≤γ≤1, R t represents the reward at time t; (5.2) The state value function and action value function are used to evaluate the expected returns of all possible decision sequences after the intelligent vehicle starts to take action: Where S t represents the state at time t, A t represents the action at time t; (5.3) Establish the advantage function under the selected action: A π (s t ,a t )=Q π (s t ,a t )-V π (s t ) (15) (5.4) Establish the estimation function of the advantage function: (5.5) Establish the policy gradient loss function: Where r represents the probability ratio of the new and old strategies, and ε represents the clipping factor; (5.5) Establish a state-decomposed attention mechanism to better focus on the impact of curvature and error on action selection: Where Q Ni , K Ni represents Q and K in the attention mechanism, and d represents the corresponding dimension; (5.6) Construct the loss function based on the physical information of the intelligent vehicle: in represents the error between the predicted model and the actual model, represents the data observation error, γ pinn and γ data represents the corresponding coefficient; Finally get and The optimal parameters 4. The intelligent vehicle path control method based on improved reinforcement learning adaptive fusion according to claim 1 is characterized in that: The step 6 includes: obtaining the optimal parameters from the near-end strategy optimization algorithm Bring in the fusion control algorithm to calculate the control amount of the intelligent vehicle at the current moment: Get the optimal steering angle δ of the intelligent vehicle at the current moment opt , to control the operation of smart vehicles.

Images