A reinforcement learning-based method for shifting AMT in electric vehicles

Abstract

The present invention discloses a method for shifting up and down an automatic transmission (AMT) of an electric vehicle based on reinforcement learning, comprising the following steps: constructing a shifting network model, the shifting network model comprising a strategy network and an evaluation network; using vehicle speed, vehicle acceleration and accelerator pedal opening as input variables of the strategy network, and using a shifting instruction as an output variable of the strategy network; using vehicle speed, vehicle acceleration, accelerator pedal opening and the shifting instruction as input variables of an evaluation network, and the evaluation network outputting the value of the current shifting instruction; calculating a comprehensive evaluation index of the shifting strategy; optimizing the shifting network model with the minimum difference between the comprehensive evaluation index of the shifting strategy and the value of the current shifting instruction as an optimization goal, and obtaining an optimal shifting network model; and during vehicle driving, detecting vehicle speed, vehicle acceleration and accelerator pedal opening in real time as input variables of the optimal shifting network model, and obtaining a shifting instruction through the optimal shifting network model.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric vehicle shift control, and in particular relates to a reinforcement learning-based electric vehicle automatic transmission (AMT) shift control method. Background Art

[0002] In recent years, the severe global energy and environmental challenges have driven countries to accelerate the transformation of their transportation energy strategies. As one of these strategic emerging industries, the new energy vehicle industry has become a key breakthrough for my country in energy conservation and emission reduction, economic revitalization, and industrial restructuring.

[0003] The "Energy-Saving and New Energy Vehicle Industry Development Plan (2012-2020)" emphasizes my country's commitment to developing the new energy vehicle industry through a pure electric drive strategy. With its market share increasing year by year, the electric vehicle industry is rapidly developing and has become a key driver of the global automotive industry's transformation and development, as well as a key engine driving sustained global economic growth.

[0004] Automatic transmissions are one of the core assemblies for automated operation in electric vehicles. They significantly reduce driver fatigue and improve driving control and safety. Their control quality directly impacts the vehicle's handling, comfort, safety, power, and economy. They are a key indicator of a vehicle's overall technological level and a primary indicator of the international competitiveness of its core technologies, occupying a significant share of the automotive industry. With the continuous advancement of traditional automotive technology and the development of various new energy vehicles, the demand for automated control and matching technologies for automotive powertrains is increasing, and automatic transmission control technology is becoming increasingly complex. Gear decision-making is a core component of automatic transmission theory. Its primary function is to determine when to shift gears during driving, specifically selecting the optimal shift timing based on driver input, vehicle operating conditions, and driving environment. Its rationality directly impacts the vehicle's power, economy, and drivability. Improving the intelligence of automatic transmission gear decision-making systems is a key focus and research topic in the international automotive transmission field.

[0005] In this social context, traditional gear decision optimization technology has certain limitations: since the formulation process of traditional gear decision methods is mostly based on engine / motor characteristic bench tests, and the optimal shifting strategy is designed based on experimental data, its application requires a lot of manual calibration, and the development process has the disadvantages of long cycle, high cost and large workload; and because the actual driving conditions of the vehicle are quite different from the conditions set by the optimal rules, it is difficult for traditional gear decision methods to guarantee the optimal results. Summary of the Invention

[0006] This paper designs and develops a reinforcement learning-based method for shifting electric vehicle automatic transmission (AMT). The purpose of this invention is to apply reinforcement learning theory to electric vehicle gear decision-making and optimization, and comprehensively consider multiple vehicle performance factors such as power and economy in the gear shift decision. This can greatly shorten the development cycle of the gear shift strategy and quickly obtain the optimal gear decision strategy for the electric vehicle.

[0007] The technical solution provided by the present invention is:

[0008] A method for shifting up and down an AMT of an electric vehicle based on reinforcement learning comprises the following steps:

[0009] Constructing a shift network model, wherein the shift network model includes a strategy network and an evaluation network;

[0010] The vehicle speed, vehicle acceleration, and accelerator pedal opening are used as input variables of the strategy network, and the shift instruction is used as the output variable of the strategy network; the vehicle speed, vehicle acceleration, accelerator pedal opening, and the shift instruction are used as input variables of the evaluation network, and the evaluation network outputs the value of the current shift instruction;

[0011] Calculating a comprehensive evaluation index of the shift strategy; optimizing the shift network model with the difference between the comprehensive evaluation index of the shift strategy and the value of the current shift instruction minimized as an optimization goal to obtain an optimal shift network model;

[0012] The comprehensive evaluation index of the shift strategy is calculated by the following formula:

[0013] r=k1*r1+k2*r2+k3*r3+k4*r4;

[0014] Where r1 is the power evaluation score, r2 is the economy evaluation score, r3 is the shift principle evaluation score, and r4 is the shift interval evaluation score; k1, k2, k3, and k4 are the relative weights of each score.

[0015] During vehicle travel, the vehicle speed, vehicle acceleration and accelerator pedal opening are detected in real time as input variables of the optimal shift network model, and a shift instruction is obtained through the optimal shift network model.

[0016] Preferably, the calculation method of the dynamic evaluation score is:

[0017]

[0018] Among them, J is the vehicle impact degree, J_max is the maximum impact degree, and error_v is the velocity error.

[0019] Preferably, the economic evaluation score is calculated as follows:

[0020]

[0021] Among them, e0 is the current efficiency of the motor, and e is the motor efficiency after shifting.

[0022] Preferably, the shift principle evaluation score is calculated as follows:

[0023]

[0024] Preferably, the shift time interval evaluation score is calculated as follows:

[0025]

[0026] Among them, delta_t represents the time between the current gear shift and the previous gear shift.

[0027] Preferably, the shift network model is optimized using a PPO algorithm.

[0028] The beneficial effects of the present invention are:

[0029] (1) The electric vehicle AMT shift up / down method based on reinforcement learning provided by the present invention can improve the electric vehicle's cruising range while meeting the power requirements and driving comfort, reduce the energy consumption of the entire vehicle, and lower the production and use costs of the vehicle.

[0030] (2) The electric vehicle AMT shift up / down method based on reinforcement learning provided by the present invention uses reinforcement learning (RL) to optimize the decision-making process, which can achieve the purpose of freeing manpower and realizing automation in the process of seeking the optimal shift MAP.

[0031] (3) The electric vehicle AMT shifting method based on reinforcement learning provided by the present invention can be used for different vehicle models and has good versatility.

[0032] (4) The electric vehicle AMT shift up / down method based on reinforcement learning provided by the present invention uses Matlab and Simulink to jointly establish a gear decision control model and build a simulation platform, thereby realizing the forward shifting and modeling of the system software development process, improving development efficiency, shortening the development cycle, and reducing development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the electric vehicle AMT shift up and down method based on reinforcement learning described in the present invention.

[0034] Figure 2 This is a simplified structural diagram of a two-speed AMT pure electric vehicle according to the present invention.

[0035] Figure 3 This is a schematic diagram of the reinforcement learning algorithm described in the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0037] like Figure 1 As shown, the present invention provides an electric vehicle AMT shift up / down method based on reinforcement learning, and the specific implementation process is as follows.

[0038] 1. Determine the expected driving conditions, establish a vehicle longitudinal dynamics model in Simulink, and set various vehicle parameters and driving condition parameters.

[0039] The vehicle longitudinal dynamics model includes:

[0040] 1.1 Vehicle Model

[0041] This module uses the actual output torque of the gearbox, the braking torque of the vehicle and the slope as input to calculate the vehicle's real-time information parameters such as driving resistance, speed, mileage, wheel speed, acceleration, etc.

[0042] First, add the actual output torque of the gearbox and the braking torque of the vehicle, then divide it by the wheel radius to get the total traction of the vehicle. Then, according to the vehicle power balance formula, that is, total traction = rolling resistance + acceleration resistance + wind resistance + rolling resistance, get the acceleration resistance.

[0043] The calculation formulas for the four resistances are as follows:

[0044] Acceleration resistance:

[0045] F=δma

[0046] Where: m is the curb mass of the vehicle; δ is the vehicle rotational mass coefficient.

[0047] Wind resistance:

[0048]

[0049] Where: C D is the air resistance coefficient; A is the frontal area of ​​the vehicle;

[0050] Slope resistance:

[0051] F=mgsin[arctan(α)]

[0052] Rolling resistance:

[0053] F=mgcos[arctan(α)]*μ r

[0054] Where: μ r is the wheel rolling coefficient.

[0055] The above formula first determines the vehicle's acceleration, from which multiple wheel parameters can be derived. Differentiating the acceleration yields the vehicle's impact. Adding this to the velocity integral yields the vehicle's speed in m / s, which can then be converted to km / h. The velocity integral yields the vehicle's mileage. Dividing the speed by the wheel radius yields the wheel's angular velocity, which can be converted to the wheel's rotational speed.

[0056] 1.2 Driving and braking system model

[0057] The system uses the difference between the actual vehicle speed and the required speed to obtain a value between -1 and 1 using the PI function. Values ​​greater than 0 represent the accelerator pedal travel, which is then multiplied by the current maximum motor output torque to obtain the current required motor torque. Values ​​less than 0 represent the brake pedal travel, which is then multiplied by the manually set maximum vehicle braking force to obtain the current required vehicle braking torque.

[0058] The required vehicle speed here is imported from the simulation condition (NEDC condition).

[0059] 1.3 Motor and transmission system model

[0060] This module takes the motor's required torque transmitted by the TCU, the speed ratio transmitted by the gearbox, and the wheel speed as input, and calculates the actual output torque of the gearbox, the maximum output torque of the motor, the current motor speed, and the actual output torque of the motor.

[0061] The motor's actual output torque is obtained by delaying the motor's required torque from the TCU. The actual transmission output torque is obtained by multiplying the required torque by the transmission ratio and transmission efficiency. The motor's required speed is obtained by multiplying the wheel speed by the transmission ratio, which is also the actual motor's required speed.

[0062] In order to obtain the actual power consumption of the motor, it is necessary to obtain the motor efficiency at each moment, and obtain the motor efficiency graph by performing two-dimensional interpolation on the test data points.

[0063] 1.4 two-speed AMT model

[0064] 1.4.1 Shift actuator

[0065] The structure of the pure electric vehicle equipped with two-speed AMT studied in this invention is as follows: Figure 2 As shown in the figure, its transmission system consists of a permanent magnet synchronous motor, a two-speed AMT, a final reducer, a differential, and two output half shafts. Figure 1It can be seen that the AMT has two sets of meshing gears to form two gears of the automatic transmission, and is equipped with a shift actuator controlled by a small-power DC motor, which controls the rotation of the shift motor to move the synchronizer sleeve left and right, forcing the synchronizer to engage with the 1st or 2nd gear driven gear to achieve the purpose of engaging 1st or 2nd gear.

[0066] The shift command is sent to the shift actuator (executed by the shift motor), which then outputs two outputs: the shift time and whether the shift is complete. The speed, torque, and backshift status here refer to the state of the shift motor. When the shift is complete, the shift motor will backshift back to its initial state. If backshifting is not required, the shift has not yet been completed.

[0067] The synchronous resistance torque is the largest in the shift motor, so its resistance is considered alone, and the other resistances are ignored. The calculation formula is as follows:

[0068]

[0069] Where, x is the axial travel of the coupling sleeve; x0 is the starting position of the coupling sleeve; x1 is the position where the coupling sleeve is disengaged from the positioning pin; x2 is the shift completion position; F spr is the spring force; F f is the friction force in the initial stage; f is the friction coefficient; F f0 is other friction force; x spr is the spring compression stroke; k spr is the spring stiffness; R pin is the radius of the shallow pit of the positioning pin; r pin is the radius of the dowel pin head.

[0070] 1.4.2 Shift Motor Control

[0071] The PID controller uses the shift motor's feedback position (displacement in degrees, angular displacement) and the required displacement as inputs to determine the required speed. Note how the required displacement is calculated: when the shift motor has fully retracted, the shift is complete, and the required displacement is 0. If the shift motor has not fully retracted, the shift is still incomplete, and retraction is required. The retraction displacement is manually set.

[0072] Next, the shift motor's displacement is calculated. The required speed and the internally calculated current are used as inputs to calculate the shift motor's feedback position, or actual position. The current is multiplied by the torque coefficient to obtain the electromagnetic torque. The actual torque is obtained by subtracting the synchronizer sleeve's drag torque, and then divided by the moment of inertia to obtain the angular acceleration. Integrating the angular acceleration yields the actual angular velocity, which is then integrated again to obtain the actual angular displacement.

[0073] The most important calculation here is the current calculation for the shifting motor. This involves first using the desired and actual speeds to determine the desired current using the PID controller. The desired and actual currents are then combined to determine the control voltage (total voltage). The resistor loss voltage and the motor loss voltage are then calculated. The difference between these three is the actual voltage, which is then divided by the inductance to determine the actual current. This forms a closed loop.

[0074] In the vehicle longitudinal dynamics model, the vehicle parameters and driving condition parameters are set as follows:

[0075] BF=1000;% maximum braking force;

[0076] E=0.97;% transmission efficiency;

[0077] T_max=150% of the motor peak torque;

[0078] P_max=94.25% of the motor peak power;

[0079] n_0=6000;% The corresponding speed when the motor torque changes;

[0080] r=0.335;% wheel radius;

[0081] delta=1.05; % rotational mass conversion factor;

[0082] m=1500;% curb mass;

[0083] A=2.1;% frontal area;

[0084] C_d=0.28;%Drag coefficient;

[0085] rho=1.2258;%air density;

[0086] f=0.0085;% rolling friction resistance coefficient;

[0087] g = 0.981; % acceleration due to gravity;

[0088] FG=0;% slope;

[0089] i1=12;% first gear transmission ratio;

[0090] i2=8;% second gear transmission ratio;

[0091] r_0=0.003;% gearbox pin head radius;

[0092] Rm=0.005;%The radius of the gearbox's pinion pit;

[0093] Ff=4.2%;

[0094] Ks=0.2%;

[0095] s0=0.002% pedal idle travel;

[0096] E_2=0.94;%gearbox transmission efficiency;

[0097] i_2=3.6;%gearbox transmission ratio;

[0098] KT=0.215;% motor torque coefficient

[0099] Kw=0.02;% motor speed coefficient

[0100] JM=0.84;%motor moment of inertia

[0101] R=0.286;% motor armature resistance

[0102] L=0.0002;% motor inductance

[0103] E0=60*1000*3600;% battery power

[0104] e_battery = 0.8;

[0105] J_max=10; % maximum impact strength.

[0106] 2. Establish comprehensive evaluation indicators based on the gear shifting strategy objectives, construct a comprehensive evaluation function for the gear shifting strategy, and build a function model through Simulink.

[0107] 2.1 Dynamic evaluation indicators

[0108] 2.1.1 Speed ​​error error_v (required vehicle speed v 需 -Actual vehicle speed v 实 )

[0109] Speed ​​error error_v = required vehicle speed v 需 -Actual vehicle speed v 实 If the speed error is greater than 0, it means that the actual vehicle speed is low and does not meet the power requirement, so a negative score is given; if the speed error is equal to 0, it means that the power requirement is met, and a maximum score of 10 points is given; if the speed error is less than 0, it means that the actual vehicle speed is greater than the required speed. Although it does not fully meet the requirement, there is room for reduction if it exceeds the requirement. Compared with the power reserve, a slightly smaller positive score is given.

[0110] In summary, the evaluation score of speed error is:

[0111]

[0112] 2.1.2 Shock J (Derivative of acceleration with respect to time)

[0113] If J is greater than J_max, -10 points are given, otherwise 0 points are given, so the evaluation score of the impact degree is:

[0114]

[0115] In summary, the evaluation scores for dynamics are:

[0116]

[0117] 2.2 Economic evaluation indicators

[0118] The motor efficiency includes the current efficiency e0, the efficiency after downshifting e1, the efficiency after upshifting e2, and the efficiency after shifting e. The current gear position i determines whether the shift is upshifting or downshifting. If i is 1, it is 1st gear, then e = e2; if i = 2, then e = e1. Comparing e0 and e, if e0 is larger, the current gear efficiency is high and a positive score is given; if e0 is smaller, a negative score is given. Therefore, the evaluation score for economy is:

[0119]

[0120] 2.3 Evaluation indicators of shifting principles

[0121] To ensure compliance with actual shifting conditions, the agent must not downshift when in the lowest gear and not upshift when in the highest gear. Violations of this rule result in a score of -10, while compliance results in a score of 0. (However, the model already fundamentally ensures this rule through other methods, so r3 is always 0 during the simulation.) Therefore, the evaluation score for the shifting rule is:

[0122]

[0123] 2.4 Driving performance evaluation indicators

[0124] The shift interval delta_t represents the time between the current shift and the previous shift (the time of the first shift is not counted as a penalty). Frequent shifts while driving can cause excessive shock. Even if the shock is not severe, it can easily cause driver fatigue. Therefore, when the shift interval is less than 2s, a score of -10 is assigned, and otherwise a score of 0 is assigned. The drivability evaluation score is:

[0125]

[0126] In summary, the comprehensive score of all evaluation indicators is:

[0127] r=k1*r1+k2*r2+k3*r3+k4*r4;

[0128] Among them, k1, k2, k3, and k4 are the relative weights of each score, and the default value is 1.

[0129] 3. Determine the input and output variables of the gear shift decision process, introduce the agent module in Simulink and connect the input and output variables and the comprehensive evaluation function of the gear shift strategy.

[0130] Input variables: vehicle speed u, acceleration a, accelerator pedal opening t;

[0131] Output variable: shift instruction shift_temp,

[0132] 4. Create a reinforcement learning environment and build a neural network (shift network model) in the agent through Matlab instructions.

[0133] The constructed neural network includes actor (strategy network) and critic (value network), which are used to represent the strategy function and return prediction function respectively.

[0134] 5. Specify the agent training options, run the reinforcement learning training program, and complete the optimization of the shifting strategy.

[0135] The PPO algorithm in reinforcement learning adopts the PPO-truncation form and imposes restrictions in the objective function to ensure that the gap between the new parameters and the old parameters is not too large, that is,

[0136]

[0137] 6. Simulate the operating conditions in Simulink to view the agent's training results and working performance.

[0138] When simulating working conditions, relevant parameters such as current gear position, actual vehicle speed, required vehicle speed, reward function value (value of shift instruction), vehicle speed, acceleration, shift time, etc. are expressed in the form of curves in the simulation data checker for evaluation and acceptance.

[0139] like Figure 3 As shown, the training process of the shift network model is as follows: the vehicle speed, vehicle acceleration and accelerator pedal opening are used as input variables of the strategy network, and the shift instruction (action) is used as the output variable of the strategy network; the vehicle speed, vehicle acceleration and accelerator pedal opening and the shift instruction are used as input variables of the evaluation network, and the evaluation network outputs the value of the current shift instruction (action).

[0140] Calculate the comprehensive evaluation index (reward) of the shift strategy, that is, calculate the comprehensive score of all evaluation indicators through a formula; optimize the shift network model with the minimum difference between the comprehensive evaluation index (comprehensive score) of the shift strategy and the value of the current shift instruction output by the evaluation network as the optimization goal to obtain the optimal shift network model.

[0141] The present invention comprehensively considers multiple vehicle performance factors such as power and economy to establish a comprehensive evaluation index (reward function) for shifting strategies, applies reinforcement learning theory to electric vehicle gear decision-making and optimization, and relies on multi-software joint simulation platforms such as Matlab / Simulink to build an intelligent agent training and interactive environment, which can quickly obtain the optimal gear decision-making strategy for electric vehicles.

[0142] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for shifting up and down AMT of electric vehicles based on reinforcement learning, characterized in that: include: Constructing a shift network model, wherein the shift network model includes a strategy network and an evaluation network; The vehicle speed, vehicle acceleration, and accelerator pedal opening are used as input variables of the strategy network, and the shift instruction is used as the output variable of the strategy network; the vehicle speed, vehicle acceleration, accelerator pedal opening, and the shift instruction are used as input variables of the evaluation network, and the evaluation network outputs the value of the current shift instruction; Calculate the comprehensive evaluation index of the shift strategy; Taking the minimum difference between the comprehensive evaluation index of the shift strategy and the value of the current shift instruction as the optimization goal, the shift network model is optimized to obtain the optimal shift network model; The comprehensive evaluation index of the shift strategy is calculated by the following formula: r=k1*r1+k2*r2+k3*r3+k4*r4; Where r1 is the power evaluation score, r2 is the economy evaluation score, r3 is the shift principle evaluation score, and r4 is the shift time interval evaluation score; k1, k2, k3, and k4 are the relative weights of each score. During vehicle travel, the vehicle speed, vehicle acceleration, and accelerator pedal opening are detected in real time as input variables of the optimal shift network model, and a shift instruction is obtained through the optimal shift network model; The calculation method of the dynamic evaluation score is: Among them, J is the vehicle impact degree, J_max is the maximum impact degree, and error_v is the velocity error.

2. The electric vehicle AMT shift up / down method based on reinforcement learning according to claim 1 is characterized in that: The calculation method of the shift principle evaluation score is:

3. The electric vehicle AMT shift up / down method based on reinforcement learning according to any one of claims 1-2, characterized in that: The calculation method of the shift time interval evaluation score is: Among them, delta_t represents the time between the current gear shift and the previous gear shift.

4. The electric vehicle AMT shift up / down method based on reinforcement learning according to claim 3 is characterized in that: The shift network model is optimized using the PPO algorithm.

Images