A reinforcement learning safety control method based on transfer learning

Abstract

A reinforcement learning safety control method based on transfer learning, which belongs to the field of intelligent control technology. The present invention solves the problem that the safety of controlling vehicle motion using existing deep reinforcement learning methods cannot be guaranteed. The method of the present invention is: step one, establishing a source system model for transfer learning; step two, establishing an affine dynamic system model for vehicle motion control, that is, a target system model; step three, using the source system model to learn the uncertainty in the affine dynamic system model for vehicle motion control; step four, migrating the control strategy of the source system model to the target system model, and controlling the vehicle motion in combination with the uncertainty obtained in step three. The method of the present invention can be applied to the field of intelligent control technology.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control technology, and specifically relates to a reinforcement learning safety control method based on transfer learning. Background Art

[0002] Reinforcement learning is a promising approach because it can effectively handle nonlinear, stochastic, unknown, or highly uncertain dynamics often encountered in modern control systems. Reinforcement learning methods provide powerful algorithms to obtain optimal or suboptimal controllers online through trial and error using only data without models. Recent studies have reported the advantages of reinforcement learning in solving complex control problems, including those of vehicle motion. However, a major drawback of reinforcement learning is the lack of stability and safety guarantees, which limits its application in real-world physical systems. The safety requirements of reinforcement learning control include two aspects: safety during training and safety after training. Safety after training focuses on the safety of the control strategy only after the training is completed, without considering the safety during training. In contrast, safety during training refers to the safety during the learning process, which is an important challenge due to the trial-and-error nature of reinforcement learning, which may lead to unsafe results. The problem of safety during training, also known as safe exploration, has received less attention from scholars, hindering the practical application of reinforcement learning.

[0003] In summary, the existing deep reinforcement learning methods do not fully consider the safety during training and the safety after training. Therefore, the safety of controlling vehicle motion using existing deep reinforcement learning methods is still not guaranteed. Therefore, research on how to ensure safety based on deep reinforcement learning control technology is of great scientific significance. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the safety of controlling vehicle motion using existing deep reinforcement learning methods cannot be guaranteed, and to propose a reinforcement learning safety control method based on transfer learning.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a reinforcement learning safety control method based on transfer learning, the method specifically comprises the following steps:

[0006] Step 1: Establish the source system model for transfer learning;

[0007] Step 2: Establish a vehicle motion control affine dynamic system model, that is, a target system model;

[0008] Step 3: Use the source system model to learn the uncertainty in the vehicle motion control affine dynamic system model;

[0009] Step 4: Migrate the control strategy of the source system model to the target system model, and control the vehicle motion in combination with the uncertainty obtained in step 3.

[0010] Furthermore, the specific process of step one is:

[0011]

[0012] Where ξ is the source system state, ξ∈Ξ, Ξ is the set of source system states, ν′ is the control input of the source system, ν′∈Ω, Ω is the set of source system control inputs, is the first-order derivative of ξ, and f(·) and g(·) are Lipschitz continuous functions.

[0013] Furthermore, the specific process of step 2 is as follows:

[0014] The affine dynamic system model of vehicle motion control is:

[0015]

[0016] Where x is the vehicle motion state, x = [βr] T , β is the vehicle sideslip angle, r is the vehicle yaw rate, and the superscript T represents the transposition. is the set of vehicle motion states, is the real number domain, m' is the dimension of the vehicle's motion state, u is the control input, u=[δ f δ r ] T , δ f is the steering angle of the vehicle’s front wheels, δ r is the steering angle of the vehicle's rear wheels, is the set of control inputs, n is the dimension of the control inputs, is the first derivative of x;

[0017]

[0018] Where v is the vehicle speed, m is the vehicle mass, μ represents the adhesion coefficient, K f Represents the front wheel curvature stiffness, L f Represents the distance from the vehicle's center of mass to the front axle, K r Represents the rear wheel curvature stiffness, L r Represents the distance from the vehicle's center of mass to the rear axle, I z Represents the vehicle's moment of inertia.

[0019] Furthermore, the source system model and the target system model are restored and transformed by differential coordinate transformation φ and input transformation ψ;

[0020] The target system model satisfies: For any The row rank of g(x) is full rank;

[0021] The differential coordinate transformation φ satisfies: for any ξ, is a non-singular matrix, and

[0022] The input transformation ψ satisfies:

[0023]

[0024] Among them, I n is the identity matrix, z is any compatible column vector, is the right inverse matrix of g(φ(ξ)).

[0025] Furthermore, the specific process of step three is:

[0026] Step 3.1 The nominal model of the vehicle motion control affine dynamic system model is:

[0027]

[0028] Comparing formula (2) with formula (5), formula (2) can be rewritten as:

[0029]

[0030] Among them, a(x)+b(x)u is the uncertainty of the system;

[0031] Then the source system model of formula (1) is reconstructed as:

[0032]

[0033] in, and is a Lipschitz continuous function in the reconstructed source system model;

[0034] Step 32: learning the uncertainty in the vehicle motion control affine dynamic system model based on the reconstructed source system model;

[0035] The specific process of step 32 is as follows:

[0036] Step 321: Initialize the state set ρ1 of the source system model, the maximum episode length N, the initial value of the uncertainty a(·) and the initial value of the uncertainty b(·), initialize the replay pool D1 to be empty, and initialize the replay pool D2 to be empty;

[0037] Step 322: Sample the initial state ξ0 according to ρ1 and initialize k=0;

[0038] Step 3: From the control strategy π(ν′ k |ξ k ) in the sampling action According to Solve the source system model controller

[0039]

[0040] sL f V(ξ)+L g V(ξ)ν′<δ

[0041] Among them, p is the state transition probability, L f V(ξ) and L g V(ξ) is the Lie derivative of V(ξ) along the f and g directions, respectively, and δ is the slack variable;

[0042] Step 324: Deploy the source system model controller According to v′ k Sampling state transition ξ k+1 ~p(ξ k+1 |ξ k ,v′ k ), where ξ k+1 is the state at time k+1;

[0043] The tuple (ξ k ,v′ k ,c k ,ξ k+1 ) is stored in the playback pool D1, c k is the cost at time k, and ξ k Substituting into formula (7) we get The tuple Stored in replay pool D2;

[0044] Step 325: Determine whether k is less than N;

[0045] If satisfied, set k=k+1 and return to step 323;

[0046] If not satisfied, execute step 326;

[0047] Step 326: Sample tuples from the playback pool D1 to train the control strategy, and sample tuples from the playback pool D2 to update the uncertainties a(·) and b(·);

[0048] Determine whether the iterative stopping condition is met. If so, obtain the final control strategy and uncertainties a(·) and b(·). If not, return to step 322 using the updated control strategy and updated uncertainties.

[0049] Furthermore, the sampling tuple from the playback pool D2 updates the uncertainties a(·) and b(·); the specific updating process is:

[0050]

[0051] in, and are the estimated values ​​of a(·) and b(·), respectively;

[0052]

[0053] in, and There are two function classes. is the loss function, N is the number of sampled tuples, Yes i The first derivative of .

[0054] Furthermore, the specific process of step 4 is as follows:

[0055] Step 41: Initialize the state set ρ2 of the target system model, the maximum episode length M, load the control strategy of the source system model and the uncertainty obtained in step 3;

[0056] Step 42: Sample the initial state x0 according to ρ2 and initialize k=0;

[0057] Step 43: Use differential coordinate transformation to transform x k Transform to ξ k ;

[0058] Step 4. Control strategy π(v′) from source system model k |ξ k ) in the sampling action According to Solving for the source system controller

[0059] Step 45: Execute strategy migration:

[0060]

[0061] Step 46: Deploy source and target system controller u k After that, update the vehicle motion state x k+1 ; Using x k+1 Return to step 43.

[0062] Furthermore, the equilibrium point of the target system model is the origin, that is, f(0)=0.

[0063] Furthermore, the control strategy is trained using SAC, PPO, DDPG and TRPO algorithms.

[0064] Furthermore, the training method of the control strategy is:

[0065] The source system model is discretized into the following form using the Euler discretization method:

[0066] ξ k+1 =ξ k +Tf(ξ k )+Tg(ξ k )ν′ k (13)

[0067] Among them, ξ k+1 is the state of the source system model at time k+1, ξ k is the state of the source system model at time k, ν′ k is the source system control input at time k, T is the discretization period;

[0068] The cost function c(ξ k ,v′ k ) is defined as follows:

[0069] c(ξ k ,v′ k )=ξ k T Rξ k +V T (k)QV(k) (14)

[0070] Among them, the superscript T represents transpose, and R and Q are both positive definite weight matrices;

[0071] According to c(ξ k ,ν′ k ) Establish control objectives π :

[0072]

[0073] where ρ π represents the trajectory distribution guided by the control strategy, β is an adjustable parameter, is the control strategy π(·|ξ k )’s entropy;

[0074] The optimal control strategy is obtained by minimizing the weighted sum of the cost function and entropy.

[0075] The beneficial effects of the present invention are:

[0076] The strategy training in the present invention is performed offline in the source system without any security threats. An auxiliary stability compensation method based on quadratic programming of the control Lyapunov function is introduced to ensure the stability of the system under reinforcement learning control. A supervised learning algorithm is introduced into the reinforcement learning framework to learn uncertainty. Uncertainty learning is implemented in the source system, which can effectively avoid unsafe behaviors during training. Migrating the strategy trained in the source system to the target system can ensure stability and safety when controlling vehicle motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of the control process of the method of the present invention;

[0078] Figure 2 It is a control framework diagram of the method of the present invention;

[0079] Figure 3 It is a schematic diagram of the evolution of data cascade and distribution;

[0080] Figure 4 is a vehicle model diagram;

[0081] Figure 5(a) shows the uncertainty learning result when the number of episodes is 80. Figure 1 ;

[0082] In the figure, the horizontal axis is the vehicle sideslip angle β, and the vertical axis is the vehicle yaw rate r;

[0083] Figure 5(b) shows the uncertainty learning result when the number of episodes is 80. Figure 2 ;

[0084] Figure 5(c) shows the uncertainty learning result when the number of episodes is 120. Figure 1 ;

[0085] Figure 5(d) shows the uncertainty learning result when the number of episodes is 120. Figure 2 ;

[0086] Figure 5(e) shows the uncertainty learning result when the number of episodes is 200. Figure 1 ;

[0087] Figure 5(f) shows the uncertainty learning result when the number of episodes is 200. Figure 2 ;

[0088] Figure 5(g) shows the uncertainty learning result when episode is 1160. Figure 1 ;

[0089] Figure 5(h) shows the uncertainty learning result when episode 1160 is reached. Figure 2;

[0090] Figure 6(a) is the source system control result;

[0091] In the figure, the horizontal axis is the vehicle sideslip angle β, and the vertical axis is the vehicle yaw rate r;

[0092] Figure 6(b) is the vehicle lateral control result of the target system. DETAILED DESCRIPTION

[0093] The present application is further described in detail below through specific implementations in combination with the accompanying drawings. Obviously, the described implementations are only part of the implementations of the present invention, not all of the implementations. Based on the implementations in the present invention, other implementations obtained by ordinary technicians in the field without making creative work are all within the scope of protection of the present invention.

[0094] Specific implementation method 1. Combination Figure 1 and Figure 2 The present embodiment describes a reinforcement learning safety control method based on transfer learning, and the method specifically includes the following steps:

[0095] Step 1: Establish the source system model for transfer learning;

[0096] The dynamics of the source system should be the same as the dynamics of the target system, which means that they use the same controlled plant. It means that the source system is not restricted by state, thus eliminating the security threat of the source system. In order to meet these two characteristics, the source system used for policy training must be a real system where security threats can be temporarily removed. This requirement can be achieved in various practical scenarios. Compared with virtual-to-real migration, the method proposed in this invention belongs to the category of real-to-real migration, and there is no need to build a virtual scene.

[0097] Step 2: Establish a vehicle motion control affine dynamic system model, that is, a target system model;

[0098] Step 3: Use the source system model to learn the uncertainty in the vehicle motion control affine dynamic system model;

[0099] Step 4: Migrate the control strategy of the source system model to the target system model, and control the vehicle motion in combination with the uncertainty obtained in step 3.

[0100] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the specific process of step 1 is as follows:

[0101]

[0102] Where ξ is the source system state, ξ∈Ξ, Ξ is the set of source system states, ν′ is the control input of the source system, ν′∈Ω, Ω is the set of source system control inputs, is the first-order derivative of ξ, and f(·) and g(·) are Lipschitz continuous functions.

[0103] The other steps and parameters are the same as those in the first embodiment.

[0104] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the specific process of step two is as follows:

[0105] The affine dynamic system model of vehicle motion control is:

[0106]

[0107] Where x is the vehicle motion state, x = [βr] T , β is the vehicle sideslip angle, r is the vehicle yaw rate, and the superscript T represents the transposition. is the set of vehicle motion states, is the real number domain, m' is the dimension of the vehicle's motion state, u is the control input, u=[δ f δ r ] T , δ f is the steering angle of the vehicle’s front wheels, δ r is the steering angle of the vehicle's rear wheels, is the set of control inputs, n is the dimension of the control inputs, is the first derivative of x;

[0108]

[0109] Where v is the vehicle speed, m is the vehicle mass, μ represents the adhesion coefficient, K f Represents the front wheel curvature stiffness, L f Represents the distance from the vehicle's center of mass to the front axle, K r Represents the rear wheel curvature stiffness, L r Represents the distance from the vehicle's center of mass to the rear axle, I z Represents the vehicle's moment of inertia.

[0110] The other steps and parameters are the same as those in the first or second embodiment.

[0111] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that the source system model and the target system model are restored and transformed by differentiable coordinate transformation φ and input transformation ψ;

[0112] The target system model satisfies: For any The row rank of g(x) is full rank;

[0113] The differentiable coordinate transformation φ satisfies: for any ξ, is a non-singular matrix, and

[0114] Lemma 1: The equation Ax=b is solvable if and only if rank(A,b)=rank(A) holds.

[0115] Lemma 2: Let A + is the generalized inverse matrix of matrix A. If the equation Ax=b has a solution, then its general solution can be expressed as x=A + b+(I n -A + A)z, where z and I n are any compatible column vector and identity matrix, respectively.

[0116] Lemma 3: is the right inverse of the matrix A, and if and only if A is of full row rank,

[0117] According to the above description, equation (4) is proposed to ensure the existence of the recovery transformation of the target system.

[0118] The input transformation ψ satisfies:

[0119]

[0120] Among them, I n is the identity matrix, z is any compatible column vector, is the right inverse matrix of g(φ(ξ)).

[0121] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0122] f(ξ) and g(ξ) in Eq. (4) are the exact dynamics models obtained based on the learned uncertainties.

[0123] Assuming φ(0) = 0, if the source system model can maintain asymptotic stability under the controller ν = k(ξ), then under the migration control strategy, the asymptotic stability and security of the target system model can be guaranteed:

[0124] u=ψ(φ -1 (x),k(φ -1 (x))).

[0125] Specific implementation mode 5: This implementation mode is described in conjunction with FIG5. This implementation mode is different from the specific implementation modes 1 to 4 in that the specific process of step 3 is:

[0126] Step 3.1 The nominal model of the vehicle motion control affine dynamic system model is:

[0127]

[0128] Comparing formula (2) with formula (5), formula (2) can be rewritten as:

[0129]

[0130] Among them, a(x)+b(x)u is the uncertainty of the system; and It is assumed to be time-invariant and depends only on the state of the system;

[0131] Then the source system model of formula (1) is reconstructed as:

[0132]

[0133] in, and is a Lipschitz continuous function in the reconstructed source system model;

[0134] Step 32: learning the uncertainty in the vehicle motion control affine dynamic system model based on the reconstructed source system model;

[0135] The specific process of step 32 is as follows:

[0136] Step 321: Initialize the state set ρ1 of the source system model, the maximum episode length N, the initial value of the uncertainty a(·) and the initial value of the uncertainty b(·), initialize the replay pool D1 to be empty, and initialize the replay pool D2 to be empty;

[0137] Step 322: Sample the initial state ξ0 according to ρ1 and initialize k=0;

[0138] Step 3: From the control strategy π(ν′ k |ξ k ) in the sampling action According to Solve the source system model controller

[0139] Define the function V(x):R n →R is a continuously differentiable function that satisfies:

[0140] And x≠0;

[0141] There exists u that satisfies: [L f V(x)+Lg V(x)u]<0,x∈χ;

[0142]

[0143] sL f V(ξ)+L g V(ξ)v′<δ

[0144] Among them, p is the state transition probability, L f V(ξ) and L g V(v) is the Lie derivative of V(v) along the f and g directions, respectively, and δ is the slack variable to ensure the feasibility of the QP problem;

[0145] That is, the quadratic programming method based on the control Lyapunov function is used to enhance stability. By using formula (7) to RL Make minimal modifications to stabilize the system. Minimal modifications ensure that the system stability is achieved while minimizing the impact on control performance;

[0146] Step 324: Deploy the source system model controller According to ν′ k Sampling state transition v k+1 ~p(ξ k+1 |ξ k ,v′ k ), where v k+1 is the state at time k+1;

[0147] The tuple (v k ,v′ k ,c k ,ξ k+1 ) is stored in the playback pool D1, c k is the cost at time k, and ξ k Substituting into formula (7) we get The tuple Stored in replay pool D2;

[0148] Step 325: Determine whether k is less than N;

[0149] If satisfied, set k=k+1 and return to step 323;

[0150] If not satisfied, execute step 326;

[0151] Step 326: Sample tuples from the playback pool D1 to train the control strategy, and sample tuples from the playback pool D2 to update the uncertainties a(·) and b(·);

[0152] Determine whether the iterative stopping condition is met. If so, obtain the final control strategy and uncertainties a(·) and b(·). If not, return to step 322 using the updated control strategy and updated uncertainties.

[0153] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0154] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the sampling tuple from the playback pool D2 updates the uncertainties a(·) and b(·); the specific updating process is:

[0155] Build two function classes using deep neural networks and To estimate the uncertainty a(·) and b(·) of the above labeled data. A neural network can be used to approximate the uncertainty, that is, and in and are the estimates of a(·) and b(·), respectively. Then, using the estimates of uncertainty, It can be expressed as:

[0156]

[0157] in, and are the estimated values ​​of a(·) and b(·), respectively;

[0158]

[0159] in, and There are two function classes. is the loss function (calculated as and The distance between them), N is the number of sampled tuples, Yes i The first derivative of .

[0160] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0161] The collected states and actions (ξ i ,v′ i ), i = 1, 2, ..., N can be numerically differentiated to approximate The labeled dataset D for uncertainty supervised regression is as follows:

[0162]

[0163] The dataset of formula (11) cannot be directly used for uncertainty learning because it violates the basic requirement that the data used for supervised learning should be independent and identically distributed, such as Figure 3 As shown in Figure 2. Reinforcement learning is a sequential decision problem. Each data in the trajectory obtained from a training cycle depends on the previous data, which leads to data cascading and violates the independent distribution property. Secondly, the system trajectory is related to the current policy. As the policy is improved, the data distribution under the improved policy in each cycle will change, violating the same distribution property.

[0164] In order to solve the problems of data concatenation and data distribution evolution at the same time, the present invention introduces the following techniques for uncertainty learning:

[0165] Data aggregation. After each reinforcement learning episode, the generated data trajectories are added to the existing dataset (11).

[0166] Data forgetting. When the amount of collected data exceeds the capacity of the playback buffer, part of the historical data set (Eq. (11)) is discarded according to the order of the time tags.

[0167] Random Sampling. In each training iteration, a mini-batch of data is randomly sampled from (11) for uncertainty supervised learning.

[0168] Obviously, data aggregation and random sampling can jointly solve the data cascade problem. For the problem of data distribution evolution, data forgetting is proposed to update the replay buffer. The buffer capacity can be empirically determined based on the policy convergence speed and the number of samples required for uncertainty learning. An appropriate buffer capacity can ensure that there is enough data for uncertainty learning, while making the data distribution in the buffer approximate the trajectory distribution caused by the current policy.

[0169] Specific implementation method 7: This implementation method is different from the specific implementation methods 1 to 6 in that the specific process of step 4 is as follows:

[0170] Step 41: Initialize the state set ρ2 of the target system model, the maximum episode length M, load the control strategy of the source system model and the uncertainty obtained in step 3;

[0171] Step 42: Sample the initial state x0 according to ρ2 and initialize k=0;

[0172] Step 43: Use differentiable coordinate transformation to transform x k Transform to ξ k ;

[0173] Step 4. Control strategy π(ν′) from source system model k |ξ k ) in the sampling action According to Solving for the source system controller

[0174] Step 4: Perform strategy migration according to formula (4):

[0175]

[0176] Step 46: Deploy source and target system controller u k After that, update the vehicle motion state x k+1 ; Using x k+1 Return to step 43.

[0177] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0178] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the equilibrium point of the target system model is the origin, that is, f(0)=0.

[0179] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0180] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the control strategy is trained using SAC, PPO, DDPG and TRPO algorithms.

[0181] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0182] Specific implementation method 10: This implementation method is different from the specific implementation methods 1 to 9 in that the training method of the control strategy is:

[0183] The source system model is discretized into the following form using the Euler discretization method:

[0184] ξ k+1 =ξ k +Tf(ξ k )+Tg(ξ k )ν′ k (13)

[0185] Among them, ξ k+1 is the state of the source system model at time k+1, ξ k is the state of the source system model at time k, ν′ k is the source system control input at time k, T is the discretization period;

[0186] The Markov decision process tuple of definition (13) is in, is the state space, It is the action space. p(ξ k+1 |ξ k ,ν′ k ) is the state transition probability determined by formula (13). is the cost function. γ∈[0,1] is the discount factor. ρ is the distribution of the initial state ξ0.

[0187] Considering the stability and energy minimization objectives, the cost function c(ξ k ,v′ k ) is defined as follows:

[0188] c(ξ k ,v′ k )=ξ k T Rξ k +V T (k)QV(k) (14)

[0189] Among them, the superscript T represents transpose, and R and Q are both positive definite weight matrices;

[0190] According to c(ξ k ,v′ k ) Establish control objectives other than stability and safety π :

[0191]

[0192] where ρ π represents the trajectory distribution guided by the control strategy, β is an adjustable parameter, is the control strategy π(·|ξ k )’s entropy;

[0193] The optimal control strategy is obtained by minimizing the weighted sum of the cost function and entropy.

[0194] Its steps and parameters are the same as those of the specific implementation modes 1 to 9.

[0195] Given an initial state With a control strategy u=π(x), the state trajectory of the closed-loop system can be determined as x(t)=xπ(x0,t). The present invention focuses on the safety-critical control problem under uncertainty. Given a safe state set and the initial state set The goal is to find a control strategy The system is guaranteed to be safe while asymptotically stabilizing to the equilibrium point x = 0. This problem can be expressed as:

[0196]

[0197]

[0198]

[0199]

[0200] The present invention provides a method for ensuring the safety of reinforcement learning control based on transfer learning, and its code is shown in Table 1:

[0201] Table 1

[0202]

[0203] Example

[0204] A four-wheel steering (4WS) vehicle is used to illustrate the effectiveness of the control algorithm proposed in this invention. The control objective is to maintain the lateral stability of the vehicle. Figure 4 As shown, the origin of the coordinate system [x, y] is fixed at the center of gravity (CG) of the vehicle. v is the driving speed, β is the sideslip angle, r is the yaw rate, δ f and δ r are the steering angles of the front and rear wheels respectively.

[0205] The state space model is described as follows:

[0206]

[0207] Where x = [βr] T is the state, u=[δ f δ r ] T is the control input, and

[0208]

[0209] The parameters of the vehicle are shown in Table 2.

[0210] Table 2 Vehicle parameters

[0211]

[0212] Since an excessively large sideslip angle may lead to an increase in uncertainty in the lateral stiffness and the appearance of a very significant nonlinear effect, a side slip angle constraint is imposed to ensure the safety of the vehicle's lateral control. In the simulation, the side slip angle constraint is -0.2<β<0.2.

[0213] To evaluate the performance of uncertainty learning, both matrix A and matrix B are perturbed by up to 20%. The design is as follows.

[0214]

[0215] Using the algorithm in Table 1, the reinforcement learning strategy training is carried out in the non-hazardous source system, and the uncertainty of the model is learned at the same time. The results of uncertainty learning are Figure 5(a) to Figure 5(h) As shown in . It can be observed from Figure 5(a), Figure 5(c), Figure 5(e), and Figure 5(g) that the training set (i.e., the sampled data represented by the points) and the test set (i.e., the system trajectories represented by the solid and dashed curves) remain consistent throughout the training process. In addition, the learned uncertainty has a small error near the system trajectory, as shown in Figure 5(b), Figure 5(d), Figure 5(f), and Figure 5(h). It can be seen that an accurate estimate of the model uncertainty is maintained throughout the training process, which is of great significance for data-driven reinforcement learning control.

[0216] Figure 6(a) and Figure 6(b) show the state trajectories of the source system under the trained reinforcement learning controller and the target system under the transferred controller. It is observed that the trained controller effectively stabilizes the source system to the origin. In addition, the controller trained on the source system can be transferred to the target system to drive the state to the origin and obey the safety constraints, thus ensuring safety after learning. In addition, the paradigm of training first and then transferring avoids direct exploration in potentially dangerous environments, thus ensuring safety during learning.

[0217] exist Figure 2 In the figure, the right side is the reinforcement learning control module. The policy training is performed offline in the source system without any security threats. An auxiliary stability compensation method based on quadratic programming of the control Lyapunov function is introduced to ensure the stability of the system under reinforcement learning control.

[0218] A supervised learning algorithm is introduced into the above reinforcement learning framework to learn uncertainty. Uncertainty learning is implemented in the source system, which can effectively avoid unsafe behaviors during training.

[0219] The left side is policy migration. The policy trained in the source system is migrated to the target system to ensure stability and security. The learned uncertainty is also transferred to the target system for policy migration.

[0220] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A reinforcement learning safety control method based on transfer learning, characterized in that: The method specifically comprises the following steps: Step 1: Establish the source system model for transfer learning; The specific process of step one is: Where ξ is the source system state, ξ∈Ξ, Ξ is the set of source system states, v' is the control input of the source system, v'∈Ω, Ω is the set of source system control inputs, is the first-order derivative of ξ, f(·) and g(·) are Lipschitz continuous functions; Step 2: Establish a vehicle motion control affine dynamic system model, that is, a target system model; The specific process of step 2 is as follows: The affine dynamic system model of vehicle motion control is: Where x is the vehicle motion state, x = [β r] T , β is the vehicle sideslip angle, r is the vehicle yaw rate, and the superscript T represents the transposition. is the set of vehicle motion states, is the real number domain, m' is the dimension of the vehicle's motion state, u is the control input, u=[δ f δ r ] T , δ f is the steering angle of the vehicle's front wheels, δ r is the steering angle of the vehicle's rear wheels, is the set of control inputs, n is the dimension of the control inputs, is the first derivative of x; Where v is the vehicle speed, m is the vehicle mass, μ represents the adhesion coefficient, K f Represents the front wheel curvature stiffness, L f Represents the distance from the vehicle's center of mass to the front axle, K r Represents the rear wheel curvature stiffness, L r Represents the distance from the vehicle's center of mass to the rear axle, I z Represents the vehicle's moment of inertia; The source system model and the target system model are restored and transformed by differential coordinate transformation φ and input transformation ψ; The target system model satisfies: For any The row rank of g(x) is full rank; The differential coordinate transformation φ satisfies: for any ξ, is a non-singular matrix, and The input transformation ψ satisfies: Among them, I n is the identity matrix, z is any compatible column vector, is the right inverse matrix of g(φ(ξ)); Step 3: Use the source system model to learn the uncertainty in the vehicle motion control affine dynamic system model; Step 4: Migrate the control strategy of the source system model to the target system model, and control the vehicle motion in combination with the uncertainty obtained in step 3.

2. According to the reinforcement learning safety control method based on transfer learning in claim 1, it is characterized in that: The specific process of step three is: Step 3.1 The nominal model of the vehicle motion control affine dynamic system model is: Comparing formula (2) with formula (5), formula (2) can be rewritten as: Among them, z(x)+b(x)u is the uncertainty of the system; Then the source system model of formula (1) is reconstructed as: in, and is a Lipschitz continuous function in the reconstructed source system model; Step 32: learning the uncertainty in the vehicle motion control affine dynamic system model based on the reconstructed source system model; The specific process of step 32 is as follows: Step 321: Initialize the state set ρ1 of the source system model, the maximum episode length N, the initial value of the uncertainty a(·) and the initial value of the uncertainty b(·), initialize the replay pool D1 to be empty, and initialize the replay pool D2 to be empty; Step 322: Sample the initial state ξ0 according to ρ1 and initialize k=0; Step 3: From the control strategy π(v' k |ξ k ) in the sampling action According to Solve the source system model controller Among them, p is the state transition probability, L f V(ξ) and L g V(ξ) is the Lie derivative of V(ξ) along the f and g directions, respectively, and δ is the slack variable; Step 324: Deploy the source system model controller According to v' k Sampling state transition ξ k+1 ~p(ξ k+1 |ξ k ,v' k ), where ξ k+1 is the state at time k+1; The tuple (ξ k ,v' k ,c k ,ξ k+1 ) is stored in the playback pool D1, c k is the cost at time k, and ξ k Substituting into formula (7) we get The tuple Stored in replay pool D2; Step 325: Determine whether k is less than N; If satisfied, set k=k+1 and return to step 323; If not satisfied, execute step 326; Step 326: Sample tuples from the playback pool D1 to train the control strategy, and sample tuples from the playback pool D2 to update the uncertainties a(·) and b(·); Determine whether the iterative stopping condition is met. If so, obtain the final control strategy and uncertainties a(·) and b(·). If not, return to step 322 using the updated control strategy and updated uncertainties.

3. A reinforcement learning safety control method based on transfer learning according to claim 2, characterized in that: The sampling tuple from the playback pool D2 updates the uncertainties a(·) and b(·); the specific updating process is: in, and are the estimated values ​​of a(·) and b(·), respectively; in, and There are two function classes. is the loss function, N is the number of sampled tuples, Yes i The first derivative of .

4. The reinforcement learning safety control method based on transfer learning according to claim 3 is characterized in that: The specific process of step 4 is as follows: Step 41: Initialize the state set ρ2 of the target system model, the maximum episode length M, load the control strategy of the source system model and the uncertainty obtained in step 3; Step 42: Sample the initial state x0 according to ρ2 and initialize k=0; Step 43: Use differential coordinate transformation to transform x k Transform to ξ k ; Step 4. Control strategy π(v' from source system model k |ξ k ) in the sampling action According to Solving for the source system controller Step 45: Execute strategy migration: Step 46: Deploy source and target system controller u k After that, update the vehicle motion state x k+1 ; Using x k+1 Return to step 43.

5. The reinforcement learning safety control method based on transfer learning according to claim 4 is characterized in that: The equilibrium point of the target system model is the origin, that is, f(0)=0.

6. A reinforcement learning safety control method based on transfer learning according to claim 5, characterized in that: The control strategy is trained using SAC, PPO, DDPG and TRPO algorithms.

7. The reinforcement learning safety control method based on transfer learning according to claim 6 is characterized in that: The training method of the control strategy is: The source system model is discretized into the following form using the Euler discretization method: x k+1 =ξ k +Tf(ξ k )+Tg(ξ k )n' k (13) Among them, ξ k+1 is the state of the source system model at time k+1, ξ k is the state of the source system model at time k, ν' k is the source system control input at time k, T is the discretization period; The cost function c(ξ k ,v' k ) is defined as follows: c(ξ k ,v' k )=ξ k T Rξ k +V T (k)QV(k) (14) Among them, the superscript T represents transpose, and R and Q are both positive definite weight matrices; According to c(ξ k ,v' k ) Establish control objectives π : where ρ π represents the trajectory distribution guided by the control strategy, β is an adjustable parameter, is the control strategy π(·|ξ k )’s entropy; The optimal control strategy is obtained by minimizing the weighted sum of the cost function and entropy.

Images