An adaptive optimal cooperative game control method for an intelligent agent

Abstract

The present application belongs to the field of multi-agent control, and relates to an adaptive optimal cooperative game control method for an agent. The method can keep the vehicle system stable during movement and make the performance index of cooperation optimal. First, noise signals are applied to two controllers in the cooperative game, and vehicle state data in a certain period of time is collected. Then, a pair of control strategies that can make the system stable is found as an initial control. Finally, strategy iteration is performed by using an adaptive dynamic programming method to obtain a cooperative game control strategy that can make the performance index of cooperation optimal. The present application can solve the cooperative game problem, has less calculation time, can guarantee system stability, and is globally optimal.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of multi-agent control, and relates to an adaptive optimal cooperative game control method for an agent. BACKGROUND

[0002] A multi-agent system is a collection of multiple agents, and its goal is to build a large and complex system into a small, mutually communicating and coordinating, and easy-to-manage system. At present, according to different optimization goals of the multi-agent, the learning tasks of the agent can be divided into three types: complete cooperation, complete competition and mixed environment. In the application, taking a vehicle stability control system as an example, a driver and a vehicle control system are regarded as two agents in complete cooperation, and the two agents jointly maintain the stability of the vehicle.

[0003] The vehicle stability control system can help the driver maintain the stability of the vehicle in various situations. When the vehicle speed is too fast, side slip, spin and rollover problems may occur when braking or steering. At this time, the vehicle stability control plays a role to assist the driver in controlling the vehicle balance and stability to avoid possible safety accidents. Therefore, the vehicle stability control research has very important significance.

[0004] At present, the research on the vehicle stability control algorithm can be classified according to the control target as follows:

[0005] 1) Taking yaw motion and center side slip angle as control target.

[0006] This method measures the target turning state of the driver according to the steering sensor, compares the actual running state of the vehicle with the expected value, and if there is a deviation, allocates the wheel driving force according to the pre-designed algorithm to obtain the expected yaw moment, so that the actual running state tracks the expected running state. The control algorithms used include sliding mode control, fuzzy control and quadratic linear optimal control.

[0007] 2) Taking slip ratio as control target.

[0008] The slip ratio range that ensures the safe driving of the vehicle when the vehicle is in driving or braking state is called normal region, and other cases are called emergency region. During the driving of the vehicle, the slip ratio change of the vehicle is monitored in real time, and when the slip ratio is in the emergency region, the controller controls according to the set control law. This way mainly prevents the side slip of the vehicle to ensure stable driving.

[0009] 3) Taking roll motion as control target.

[0010] This method uses the lateral load transfer ratio as an index to evaluate whether the vehicle is stable in the roll direction. When the lateral load transfer ratio is greater than 0.8, it is considered that the vehicle has the risk of rollover, and a predictive control algorithm is introduced to control the vehicle motion.

[0011] From the above method, it can be seen that the main purpose of the current vehicle control system is to ensure the stability of the normal driving of the vehicle and the tracking of the expected driving trajectory of the vehicle, but the optimization control problem of complete cooperation game between the driver and the vehicle system is not considered. SUMMARY

[0012] The present application aims at the stability problem in the vehicle system, and regards the driver and the vehicle controller as two intelligent agents of complete cooperation game, and uses the strategy of direct yaw control to ensure the stability of the system and make the performance index of cooperation optimal.

[0013] To achieve the above purpose, the technical scheme of the present application is as follows:

[0014] The present application proposes an adaptive optimal cooperative game control method, which can make the vehicle system keep stable during movement and make the performance index of cooperation optimal. First, noise signals are applied to the two controllers in the cooperative game, and the vehicle state data within a certain period of time is collected. Then, a pair of control strategies that can make the system stable is found as the initial control. Finally, the adaptive dynamic programming method is used for strategy iteration to obtain the cooperative game control strategy that can make the performance index of cooperation optimal.

[0015] The present application specifically includes the following steps:

[0016] An adaptive optimal cooperative game control method for intelligent agents, the steps are as follows:

[0017] Step 1: Construct a cooperative game problem of a vehicle system, as shown in the accompanying drawings. Figure 1

[0018] Consider a system equation

[0019]

[0020] Where u is the control input applied to the steering wheel by the driver, v is the control input of the vehicle yaw controller, f(x) is the known system matrix, g1(x) and g2(x) are the input matrices of the two controllers respectively. A detection input is applied to the two controllers respectively, u0 and v0, wherein u0 is the detection noise applied to the steering wheel controller, and v0 is the detection noise applied to the vehicle yaw controller. The state variables x of the system within a period of time are collected, Where y is the lateral position of the vehicle mass center, v y is the lateral velocity of the vehicle mass center, and ψ is the yaw angle, ​is the yaw rate. The vehicle dynamics model in Carsim7 is cited, and for the purpose of control, the model is simplified, assuming that the tire lateral force is a linear function of the tire side slip angle, f(x), g1(x), g2(x) satisfy the following formula respectively

[0021]

[0022]

[0023]

[0024] where m is the mass of the vehicle, v x is the forward speed of the vehicle, C αF and C αR are the front and rear tire cornering stiffness, r st is the steering ratio of the steering wheel angle to the front wheel angle, l F , l R are the wheel base of the vehicle, I z is the moment of inertia of the vehicle around the z axis.

[0025] Step 2: Use the policy iteration method in adaptive dynamic programming to obtain the optimal solution of the game controller. For the system equation (1), two controllers u, v are jointly optimized for a performance index J as follows:

[0026]

[0027] where r(x(t), u(t), v(t)) = q(x) + u T R1(x)u + u T R2(x)v + v T R3(x)v, R1(x), R2(x), R3(x) are weight matrices. According to the Bellman optimality principle, the generalized Hamilton equation can be obtained as follows:

[0028]

[0029] where the value function V is the optimal performance index under the optimal control law u * and v * control. The optimal control law of the two controllers is as follows:

[0030]

[0031]

[0032] Step 3: For the system equation with probe noise Define α i = u0-ui ,β i =v0-v i ,u i ,v i is the control law obtained after the ith iteration. Rewrite the system equation as:

[0033]

[0034] For any i ≥ 0, the derivative of the value function satisfies the following equation:

[0035]

[0036] Integrate equation (7) in the time interval [t, t+T] with the current time t and the integration time interval T to obtain the following equation:

[0037]

[0038] Use three basis functions to approximate the substitute value function V and the control strategies u, v, respectively.

[0039]

[0040]

[0041]

[0042] where φ j (x), ψ j (x) are different basis functions, N1, N2 are the number of basis functions, c i,j , ω i,j , σ i,j represent the weights of the three basis functions, respectively. Substitute equations (12), (13), and (14) into equation (11) to obtain the iterative equation as follows:

[0043]

[0044] where t k is a strictly increasing time sequence starting from 0 with a time interval of T, u i and v i are the control laws obtained in the last iteration.

[0045] Step 4: Firstly, the collected state variables are divided into N groups at time interval T, and the data is processed according to formula (15), then a polynomial function with an order of less than four is taken as a base function base, and the initial weight ω0, σ0 of the control strategy is determined. The values of ω, σ and c are iteratively updated through formula (15), and the iteration is stopped when the value function V converges, and the control strategy obtained after iteration is the optimal control method.

[0046] The beneficial effects of the present application are as follows:

[0047] The present application can solve the cooperative game problem, has less calculation time, can ensure system stability, and is globally optimal. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the automobile dynamics model used in the present application.

[0049] Figure 2 is the response curve of the optimal controller u.

[0050] Figure 3 is the response curve of the optimal controller v.

[0051] Figure 4 is the response curve of the lateral position y of the vehicle mass center under the input of the adaptive controller and the input without control.

[0052] Figure 5 is the lateral speed v y of the vehicle mass center under the input of the adaptive controller and the input without control.

[0053] Figure 6 is the response curve of the yaw angle ψ under the input of the adaptive controller and the input without control.

[0054] Figure 7 is the yaw rate under the input of the adaptive controller and the input without control.

[0055] Figure 8 is the surface graph of the value function V0 obtained by fixing the state variables x2 and x3 as 0 and drawing x1 and x4 in the interval [-0.5, 0.5], and the value function V 18 obtained after final iteration. DETAILED DESCRIPTION

[0056] Example 1:

[0057] Step 1: For a simplified automobile model Take m = 2257 kg, I z = 3254.9 kg.m 2r st = 17.8, l F = 1.330 m, l R = 1.616 m, C αF = C αR ≈ 84000 N / rad, available

[0058]

[0059]

[0060]

[0061] The following performance indices are chosen:

[0062]

[0063] Step 2: Choose the probing signals u0, v0 to apply to the steering controller and the vehicle yaw controller respectively.

[0064] u0 = sum(0.2*sin([1 3 7 11 13 15]*t))

[0065] v0 = sum(0.2*sin([1 3 7 9 11 13 15]*t))

[0066] Then collect the state variable data within 3.7s of the system

[0067] Step 3: Determine the initial weights of the three basis functions, the weights ω, σ are column vectors with 24 elements, and the weight c is a column vector with 20 elements, note that the selection of ω and σ should be able to make the system stable. Their initializations are as follows:

[0068] ω = [-1, 0, 0, -3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] T

[0069] σ = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] T

[0070] c = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] T

[0071] The collected state variables are divided into N = 370 data sets with a time interval of T = 0.01s. According to the requirements of formula (15), the integrals of each term within the 0.01s time interval are calculated using the collected data and the initialized basis function weights. Formula (15) can be regarded as AX = B, where X is a column vector with 68 elements composed of c, ω, and σ. The left side of formula (15) can form a matrix A with 370 rows and 68 columns, and the right side of formula (15) can form a matrix B with 370 rows and 1 column. Then, the weights of the three basis functions are obtained using the least squares method.

[0072] The first iteration is now complete. Then, using the updated control law basis function weights ω and σ, formula (15) is recalculated to obtain the control law basis function weights for the second iteration. This continues until the value function basis function weights c converge. It is considered that the difference in the Euclidean norm of the weight vector c between the two iterations is less than 0.00001, indicating that the weight vector c has converged. The control laws obtained after convergence for the two controllers are the optimal control laws.

[0073] Step 4: This method enables online learning. This paper first calculates the control law using data within 3.7 seconds, and then immediately applies the control law at 3.7 seconds, as shown in the attached diagram. Figure 2 Appendix Figure 3 As shown in the attached figure. Figure 4 Appendix Figure 5 Appendix Figure 6 Appendix Figure 7 As shown. The performance index values ​​under different initial conditions can be obtained based on the value function weight c. A plot is drawn using the value function obtained after the first iteration and the value function obtained after the last iteration. Figure 8 .

[0074] Implementation Results

[0075] 1) From the appendix Figure 2 Appendix Figure 3 Appendix Figure 4 Appendix Figure 5 As can be seen, the system moves under probe noise before 3.7s. After 3.7s, the dashed line represents the state variable change curve without control input, which is divergent. The solid line represents the state variable curve after control is applied, which converges to the origin. This proves that the obtained control law can stabilize the system.

[0076] 2) From the appendix Figure 6 As can be seen, after multiple iterations using this method and reaching a stable point, the system's value function is less than the initial value function in every initial state. This proves that this method can optimize the system's performance under two-player cooperative game conditions.

Claims

1. An adaptive optimal cooperative game control method for an agent, characterized in that, The steps are as follows: Step 1: Constructing the vehicle system cooperative game equation Wherein: u is the control input applied to the steering wheel by the driver, v is the control input of the vehicle yaw controller, f(x) is a known system matrix, g1(x) and g2(x) are the input matrices of the two controllers respectively; a probe input is applied to the two controllers respectively, u0 and v0, wherein u0 is the probe noise applied to the steering wheel controller, and v0 is the probe noise applied to the vehicle yaw controller; collecting state variables x of the system over a period of time, where: y is the lateral position of the vehicle's center of mass, v y is the lateral velocity of the vehicle's center of mass, ψ is the yaw angle, is the yaw rate; the vehicle dynamics model in Carsim7 is referenced, for control purposes, the model is simplified, assuming that the tire lateral force is a linear function of the tire side slip angle, f(x), g1(x), g2(x) satisfy the following formula respectively: where m is the vehicle mass, v x is the forward speed of the vehicle, C αF and C αR are the front and rear tire cornering stiffness, respectively, r st is the steering ratio of the steering wheel to the front wheel angle, l F and l R are the wheel base of the vehicle, respectively, I z is the moment of inertia of the vehicle about the z axis. Step 2: Using the policy iteration method in adaptive dynamic programming to obtain the optimal solution of the game controller; For the system equation (1), the two controllers u and v jointly optimize a performance index J as follows: where r(x(t),u(t),v(t)) = q(x) + u T R1(x)u + u T R2(x)v + v T R3(x)v, R1(x), R2(x), R3(x) are weight matrices; according to Bellman optimality principle, the generalized Hamilton equation is obtained as follows: where the value function V is the optimal performance index under the optimal control law u * and v * under the optimal control action The optimal control law of the two controllers is as follows: Step 3: System equation with probing noise applied Define a i = u0- u i , β i = v0- v i , u i , v i are the control laws obtained after the ith iteration; rewrite the system equation as: For any i > 0, the derivative of the value function satisfies the following equation: Integrate equation (7) in the time period [t, t+T] with the current time t and the integral time period T to obtain the following equation: Use three basis function approximations to replace the value function V and the control strategies u and v respectively; where φ j (x), ψ j (x) are different basis functions, N1, N2 are the number of basis functions, respectively, c i,j , ω i,j , σ i,j represent the weights of the three basis functions, respectively; substituting formula (12), formula (13), formula (14) into formula (11), the iterative equation is as follows: where t k is a strictly increasing time sequence starting from 0 with T as the time interval, u i is the control law obtained in the last iteration; and i is the control law obtained in the last iteration. Step 4: First, divide the collected state variables into N groups with a time interval T, process the data according to the needs of formula (15), then use a polynomial function with a degree of four or less as the basis function basis, and determine the initial weights ω0 and σ0 of the control strategy; Update the values of ω, σ and c through formula (15) until the value function V converges, then stop iteration, and the control strategy obtained after iteration is the optimal control method.

Images