Braking energy recovery control method for multi-axle electric heavy truck based on reinforcement learning
Through deep reinforcement learning technology, the problem of complexity of side slip and force distribution during braking of multi-axle electric heavy truck vehicles is solved, and more efficient and safe energy recovery control is achieved, improving the stability and safety of the vehicle.
Patent Information
- Application Number
- CN202411296257.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-18
AI Technical Summary
The existing technology cannot effectively solve the problems of side slip instability and complex braking force distribution in the braking process of multi-axis electric heavy truck vehicles, especially under load transfer and articulation constraints, and lack of targeted energy recovery control strategies. .
Using a multi-axis electric heavy truck braking energy recovery control method based on deep reinforcement learning, the deep reinforcement learning control framework is constructed, the agent's state, action and reward functions are defined, the multi-training environment is established, and empirical data is collected to train the deep reinforcement learning control strategy. , generate a braking energy recovery strategy to realize the output of dynamic control instructions.
The braking control efficiency and accuracy of multi-axis electric heavy truck vehicles are improved, the stability and safety of the vehicle are improved, and a more efficient and safe energy recovery process is achieved, reducing safety risks caused by accuracy deviations.
Smart Images

Figure CN118810453B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of vehicle electric braking technology. Specifically, the present invention is applied to the field of dynamic electric energy recovery braking of multi-axle electric heavy trucks, and in particular to a multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning. Background Art
[0002] Multi-axle heavy-duty trucks have excellent carrying capacity and play an important role in the transportation industry. Multi-axle heavy-duty trucks have a large mass, and their mass varies with the cargo capacity. Multi-axle heavy-duty trucks have many axles and a long wheelbase, so they have high requirements for vehicle energy recovery control and braking stability control.
[0003] Specifically, a multi-axle heavy truck consists of a tractor and a trailer.
[0004] Among them, the braking configuration of the tractor is as follows: the front axle uses a set of pneumatic brakes, the middle axle and the rear axle share a set of pneumatic brakes, and there is an electric motor on the left and right sides of the middle axle and the rear axle to provide electric braking force.
[0005] Among them, the braking configuration of the trailer is: three axles share a set of pneumatic brakes.
[0006] In addition, the braking components of the tractor and trailer are uniformly controlled by the vehicle control unit of the multi-axle heavy truck. Based on the braking configuration of the multi-axle heavy truck, the technical difficulties of its braking energy recovery control are mainly reflected in the following aspects:
[0007] First, electric heavy trucks have many brake control components and related components, including three motors, power batteries and three sets of pneumatic brakes, which are highly coupled with each other. It is necessary to dynamically coordinate the motor brakes and pneumatic brakes according to different expected braking intensities and vehicle conditions.
[0008] Secondly, due to the high center of mass, large inertia and large number of axles of multi-axle heavy trucks after loading, they are more likely to produce unstable conditions such as skidding when braking than passenger cars; the existing front and rear axle braking force distribution control strategy is more suitable for two-axle vehicles and cannot be directly applied to multi-axle heavy trucks.
[0009] Thirdly, to ensure driving safety, the articulation relationship between the tractor and the trailer must be fully considered, the corresponding constraints need to be met during the control process, and dangerous conditions such as folding need to be avoided.
[0010] It can be seen that the braking energy recovery control scheme of multi-axle electric heavy trucks cannot directly apply the existing energy recovery braking strategy. If it is directly applied to multi-axle electric heavy trucks without improvement, the following problems will occur:
[0011] 1. For example, Chinese patent CN117227491A discloses a method for coordinated control of regenerative braking of an electric tractor and an electric trailer, which divides the control during vehicle braking into modules to coordinate the regenerative braking process of the electric tractor and the electric trailer. However, this solution only evaluates the risk of folding of the tractor and trailer during braking, and does not provide specific control instructions or calculation methods for each control element of the vehicle under dangerous working conditions, and does not solve or avoid dangerous working conditions.
[0012] 2. For example, the hybrid vehicle control method based on multi-agent deep reinforcement learning disclosed in Chinese patent CN115793445B uses a deep reinforcement learning algorithm to establish a regenerative braking control strategy. Compared with the weighted sum optimization scheme with manual adjustment of weight coefficients, it has the advantages of strong flexibility, real-time performance and obvious optimization effect. However, this scheme focuses on the research of two-axle vehicles. For multi-axle vehicles with more complex configurations, the vehicle load is significantly transferred during braking, and the braking control is more difficult. It lacks a targeted braking energy recovery control strategy; in addition, this scheme only considers the longitudinal force of the vehicle during braking. If it is applied to a multi-axle semi-trailer, it is also necessary to consider the lateral braking stability, especially the folding instability condition, to ensure braking safety.
[0013] 3. For example, the Chinese patent CN116588055A discloses a composite braking distribution method and device for a multi-axle commercial vehicle, which uses a rule-based method to design an electro-hydraulic braking force distribution method, converting the braking force distribution problem under different braking intensities into a braking mode switching problem. This solution can take advantage of motor braking under some working conditions, but its simple mode division method does not cover all possible working conditions, the algorithm has poor robustness, and does not consider the force of the trailer under braking, so there are still risks in the braking process.
[0014] 4. In addition to the above solutions, there are other vehicle braking control methods that only consider the braking energy recovery rate or braking stability separately. This type of control strategy cannot be applied to the vehicle configuration of multi-axle electric heavy trucks.
[0015] 5. In addition, some studies have adopted the method of hierarchical control strategy, but this method increases the complexity and response time of the system, and may affect the control accuracy of the entire control system due to errors in a certain layer. The deviation in control accuracy has a significant impact on multi-axle electric heavy trucks, and may also cause multi-axle electric heavy trucks to fall into dangerous conditions.
[0016] In summary, there is no in-depth and targeted research on energy recovery control methods for multi-axle electric heavy trucks in the prior art. Summary of the invention
[0017] The object of the present invention is to provide a multi-axis electric heavy truck brake energy recovery control method based on reinforcement learning, thereby solving all or one of the above-mentioned problems existing in the prior art.
[0018] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:
[0019] The present invention provides a multi-axis electric heavy truck braking energy recovery control method based on reinforcement learning, comprising the following steps:
[0020] Constructing a braking energy recovery control framework based on deep reinforcement learning, wherein: configuring a deep reinforcement learning control strategy, defining the agent state, action and reward of the deep reinforcement learning control strategy, establishing a multi-dimensional training environment and collecting experience data, and training the deep reinforcement learning control strategy based on the agent state, action and reward, the multi-dimensional training environment and the experience data;
[0021] The trained deep reinforcement learning control strategy is deployed as a braking energy recovery strategy to a multi-axle electric heavy truck, and the braking energy recovery strategy is called to output control instructions for the multi-axle electric heavy truck according to the target input data of the multi-axle electric heavy truck.
[0022] In one embodiment, configuring a deep reinforcement learning control strategy includes:
[0023] Set up a deep neural network based on two value networks, one policy network and three target networks;
[0024] The three target networks correspond to the two value networks and the one strategy network respectively;
[0025] The policy network is used to take actions according to the state;
[0026] The value network is used to evaluate the quality of the corresponding action under the state;
[0027] The target network is used to cut off the bootstrapping phenomenon during network training.
[0028] In one embodiment, the agent state, action and reward of the deep reinforcement learning control strategy are defined, including:
[0029] The agent state is defined as:
[0030] ;
[0031] in, is the state of the agent; is the wheel speed, is the tractor speed; is the trailer speed; is the target braking deceleration; is the tractor acceleration; is the trailer acceleration; is the left wheel turning angle of the front axle of the tractor; is the heading angle of the tractor; is the heading angle of the trailer; is the yaw angular velocity of the tractor; is the trailer yaw angular velocity; The maximum chargeable power of the vehicle's current battery; is the speed of each motor, ; is the actual pressure of each air brake at present, ;
[0032] The actions of the agent are defined as: ;
[0033] in, is the action of the agent; is the target torque of each motor, is the target pressure of each air brake, ;
[0034] A braking effect reward function, a braking energy recovery rate reward function, a dangerous working condition reward function and a slip rate reward function are set, and the weighted sum of the braking effect reward function, the braking energy recovery rate reward function, the dangerous working condition reward function and the slip rate reward function is used as the total reward function.
[0035] In one embodiment, the braking effect reward function is:
[0036] ;
[0037] Among them, r1 is the braking effect reward; a1 is the weight coefficient.
[0038] In one embodiment, the braking energy recovery rate reward function is:
[0039] ;
[0040] in, Reward for braking energy recovery; is the motor braking power; is the air pressure brake power; is the weight coefficient.
[0041] In one embodiment, the dangerous working condition reward function is:
[0042] ;
[0043] in, Rewards for hazardous working conditions; is the heading angle difference between the tractor and trailer; is the yaw rate difference threshold; is the weight coefficient.
[0044] In one embodiment, the definition operation of the slip rate reward function includes:
[0045] Set up a preliminary fourth reward function:
[0046] ;
[0047] in, for a preliminary fourth award regarding slippage; is the Gaussian distribution function parameter; is the slip rate of each wheel of the vehicle, which is obtained from the vehicle speed and the wheel speed of each wheel; is the maximum road adhesion coefficient of the road surface; is the weight coefficient;
[0048] Set the locking penalty function: ;
[0049] in; Punishment for holding death; is the weight coefficient; A coefficient to control the growth rate of the locking penalty;
[0050] The slip rate reward function is obtained by integrating the fourth reward function and the locking penalty function of all wheels of the multi-axle electric heavy truck:
[0051] .
[0052] in, Bonus for slip rate.
[0053] In one embodiment, establishing a multivariate training environment and collecting empirical data includes:
[0054] Establish a high-precision multi-axle heavy truck simulation model;
[0055] Design the driving conditions of several multi-axle heavy trucks;
[0056] Perform simulation training of the high-precision multi-axle heavy truck vehicle simulation model under the several driving conditions, collect real-time data of the vehicle simulation model, and obtain vehicle status information;
[0057] Filter, reduce noise and calibrate the collected data;
[0058] Normalizing the data after filtering, noise reduction and calibration;
[0059] The normalized data are used as the empirical data.
[0060] In one embodiment, the training of the deep reinforcement learning control strategy based on the agent state, action and reward, the multi-training environment and experience data includes:
[0061] Initializing the neural network parameters of the deep reinforcement learning control strategy, and then looping through the training steps until the cumulative reward converges to a maximum;
[0062] The initialization includes:
[0063] Use two value networks and one policy network, namely: , ;in, and , is the policy network, s is the state, a is the action, and are the network parameters of the two value networks, are the network parameters of the policy network;
[0064] Use the three target networks corresponding to the three neural networks together: , , ;in, , , are two target value networks, is the target policy network, s is the state, a is the action, and are the network parameters of the two target value networks, are the network parameters of the target policy network;
[0065] Initialize the policy network and value network before training, and randomly select parameters: , , ;in, Representing the target value network Parameters Initialized as a value network Parameters ; Representing the target value network Parameters Initialized as a value network Parameters ; Represents the target policy network Network parameters Initialized as a policy network Parameters ;
[0066] When training the policy network and the value network, a quadruple is randomly drawn from the experience replay pool each time, denoted as ; Use subscript Indicates the current parameters of the neural network, using subscript Indicates the updated parameters.
[0067] In one embodiment, the training step includes:
[0068] Call the target network to make predictions;
[0069] Call two target value networks to make predictions;
[0070] Calculate TD target;
[0071] Call two value networks to make predictions;
[0072] Calculate TD error;
[0073] Update the value network;
[0074] The policy network and the three target networks are updated every k rounds.
[0075] The beneficial effects of the technical solution of the present invention are:
[0076] 1. The multi-axle electric heavy-duty truck braking energy recovery control method based on reinforcement learning described in the present invention can enable the multi-axle electric heavy-duty truck to continuously interact and explore with the road surface based on deep reinforcement learning, realize continuous learning of the optimal control strategy, and directly calculate the control instructions of each braking element according to the target braking deceleration input, and perform data-driven strategy optimization based on deep reinforcement learning to improve control efficiency and control accuracy.
[0077] 2. The reinforcement learning-based multi-axle electric heavy-duty truck braking energy recovery control method described in the present invention optimizes the braking control of the multi-axle heavy-duty truck based on the articulation constraints of the tractor and trailer, thereby directly improving the stability and safety of the vehicle, and achieving a more efficient and safe energy recovery process by uniformly controlling braking energy recovery and braking stability.
[0078] 3. The multi-axis electric heavy-duty truck brake energy recovery control method based on reinforcement learning described in the present invention does not require manual design of control strategies, and automatically performs strategy optimization based on data-driven and deep reinforcement learning. During the control process, it only focuses on the vehicle's motion state and omits the rule building of the internal complex coupling mechanism. Finally, the target braking deceleration is input to obtain end-to-end control actions. It has high computing efficiency and real-time performance, and responds quickly, further reducing the safety risks caused by accuracy during the control process. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0080] Figure 1 It is a schematic diagram of the vehicle dynamics model of a multi-axle electric heavy truck;
[0081] Figure 2 It is a schematic diagram of the steering process of a multi-axle electric heavy truck;
[0082] Figure 3 It is a schematic diagram of the stable steering of a multi-axle electric heavy truck;
[0083] Figure 4 It is a flow chart of a multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to Embodiment 1 of the present invention;
[0084] Figure 5 It is a logic flow diagram of the multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to Example 1 of the present invention;
[0085] Figure 6 It is a schematic diagram of the architecture of the braking energy recovery control framework in the braking energy recovery control method for a multi-axle electric heavy truck based on reinforcement learning according to Embodiment 1 of the present invention;
[0086] Figure 7 It is a schematic diagram of the internal neural network relationship of the deep reinforcement learning control strategy in the reinforcement learning-based multi-axle electric heavy truck braking energy recovery control method described in Example 1 of the present invention. DETAILED DESCRIPTION
[0087] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0088] In the description of the present invention, it should be noted that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.
[0089] The terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0090] In the description of the present invention, it should be noted that Figure 1~Figure 3 As shown in the figure, when designing this method, the feasibility of the braking energy recovery scheme for multi-axle heavy trucks based on the braking model was verified as follows:
[0091] Establish a braking model for multi-axle heavy trucks, such as Figure 1 As shown; among them, Respectively The longitudinal and lateral forces on each wheel are is the center of mass of the tractor, and The articulation point between the tractor and trailer ( and In the actual vehicle, they are overlapped. Figure 1 In order to clearly show the forces, the tractor and trailer are shown separately front and back). The hinge point is located between the tractor's center axle and the rear axle, and is on the tractor-trailer axis. is the articulation angle of the tractor and trailer;
[0092] Among them, the tractor body motion model is:
[0093] ;
[0094] ;
[0095] ;
[0096] in, is the total mass of the tractor, are the longitudinal velocity, lateral velocity and yaw rate of the tractor, respectively. is the front wheel turning angle, is the total moment of inertia of the tractor, They are the front axle track and the middle and rear axle track of the tractor respectively. are the distances from each axle of the tractor to the center of mass of the tractor, is the distance from the center of mass of the tractor to the articulation point, They are the longitudinal and transverse components of the articulation force of the trailer on the tractor respectively;
[0097] Among them, the trailer body motion model is:
[0098] ;
[0099] ;
[0100] ;
[0101] in, is the total mass of the tractor, are the longitudinal velocity, lateral velocity and yaw rate of the tractor respectively, is the total moment of inertia of the tractor, is the trailer axle track, are the distances from each axle of the trailer to the center of mass of the trailer, is the distance from the trailer's center of mass to the articulation point, They are the longitudinal and transverse components of the articulation force exerted by the tractor on the trailer respectively;
[0102] Among them, since the tractor and the trailer are connected by a hinge point through the saddle, when establishing the dynamic model of the multi-axle vehicle, the constraint characteristics of the articulation of the tractor and the trailer need to be considered. The longitudinal movement, lateral movement and yaw movement of the two have a constraint relationship at the hinge point. The constraint relationship is expressed as follows:
[0103] ;
[0104] ;
[0105] ;
[0106] In addition to the dynamic constraints, the multi-axle heavy truck braking model should also consider the characteristic constraints of the braking components, as follows:
[0107] ;
[0108] in, is the target torque of each motor, is the maximum torque of the motor, is the actual air pressure of each group of air brakes, is the maximum air pressure of each group of air brakes, The maximum chargeable power of the battery; in addition, during the braking energy recovery process, it should be ensured that the output torque of each motor cannot exceed the maximum torque of the motor, the air pressure of each group of pneumatic brake components cannot exceed the maximum air pressure, the output power of the motor cannot exceed the maximum charge power of the battery, and the maximum charge power of the battery is a dynamically changing value;
[0109] Among them, the kinematic constraints in the steering process should also be considered, such as the steering process diagram of a typical multi-axle heavy truck. Figure 2 The motion model of multi-axle heavy trucks is simplified in the paper. The middle axle and rear axle of the tractor are considered as one body, located between the two axles; the three axles of the trailer are considered as one body, located at the middle axle of the trailer. Of course, the simplification here will not affect the analysis of vehicle motion characteristics. Figure 2 When the tractor starts to turn and the front wheel turns, the instantaneous turning center of the tractor is formed. At this time, the trailer is still in a straight-line driving state. Figure 2 Part (a); As the turn continues, the trailer wheels slide sideways from the straight-ahead position to , the trailer turns instantaneously Approaching from infinity , this is the transition state Figure 2 Part (b); then, and Gradually approaching until the overlap, there is a stable circular motion without relative motion between the tractor and the semi-trailer, which is the stable state of steering for multi-axle heavy-duty tractors and trailers;
[0110] Furthermore, during the braking and steering process of a multi-axle heavy truck, the above-mentioned stable state should be maintained as much as possible to minimize the drag of the wheels, alleviate tire slip, and help suppress the risk of collapse; at this time, there is a one-to-one correspondence between the articulation angle of the tractor and the trailer and the steering angle of the front wheels of the tractor, and the following constraints should be met:
[0111] ;
[0112] in, For the left front wheel turning angle The target articulation angle for the tractor and trailer to maintain steady-state steering is set, and in the subsequent control process, the actual articulation angle is expected to be as close to the target value as possible.
[0113] In summary, by establishing a multi-axle heavy truck braking model, we can intuitively see the design difficulties of the control strategy of multi-axle heavy trucks; multi-axle heavy trucks control the vehicle braking control by controlling the motor and pneumatic components together, among which the pneumatic brake components need to control multiple wheel brakes at the same time, and because the wheels have nonlinear characteristics, the longitudinal and lateral coupling is complex, and it is difficult to decouple the longitudinal and lateral forces when designing control variables; at the same time, the braking control strategy for multi-axle heavy trucks has an under-driven characteristic, that is, the number of control variables is less than the number of degrees of freedom of the controlled object; in addition, since the braking process is a dynamically changing process, the control applied at a certain moment does not only affect the current state of the vehicle, but also affects the future state. Finally, there is currently a lack of absolutely accurate dynamic and kinematic models; various characteristics show that it is very difficult to explicitly build rules for dynamic control, at least it cannot be proven to be optimal, so the solution of this application came into being.
[0114] Embodiment 1: This embodiment provides a multi-axis electric heavy truck braking energy recovery control method based on reinforcement learning, such as Figure 4~Figure 7 As shown, the following steps are included:
[0115] The core function of this method is: for the complex multi-axis electric heavy-duty truck braking system configuration, advanced data-driven and deep reinforcement learning control algorithms are used to achieve braking energy recovery control, improve energy recovery efficiency and take into account braking stability. There is no need to manually design features. During the control process, only the vehicle motion state is concerned. End-to-end control actions can be obtained based on the target braking deceleration input. The specific steps are as follows:
[0116] S1. Establish a braking energy recovery control framework based on TD3:
[0117] In this step, a braking energy recovery control strategy based on TD3 is constructed. The entire strategy framework consists of a training environment, a multi-axle heavy-duty truck real vehicle, and a deep reinforcement learning control strategy. The training environment provides training support for the strategy, the multi-axle heavy-duty truck real vehicle is the execution unit for strategy deployment and application, and the deep reinforcement learning control strategy is the main control algorithm. The following steps are required in the framework construction process:
[0118] S10. Configure deep reinforcement learning control strategy, including:
[0119] In this step, six deep neural networks are set up, including two value networks (Critic) and one policy network (Actor), as well as the target networks corresponding to the above three networks, as follows:
[0120] Among them, the value network is used to evaluate the quality of a certain action a under a certain state s, which is an approximation of the value function of the current action;
[0121] Among them, the strategy network is used to make action a according to the state s, that is, output the corresponding action command to control the braking components of the vehicle;
[0122] Among them, the target network is used to cut off the "bootstrapping" phenomenon in the network training process, avoid bias propagation, and alleviate network overestimation.
[0123] S20. Define the state, action, and reward of the reinforcement learning agent corresponding to the strategy, including:
[0124] In this step, define the state of the agent , action a and 4 reward functions (r1~r4), and the final total reward function r is obtained according to the 4 reward functions, as follows:
[0125] S201. Define the state of the intelligent agent according to the dynamic model of multi-axle electric heavy truck braking. for:
[0126] ;
[0127] in:
[0128] is the wheel speed,
[0129] is the tractor speed; is the trailer speed;
[0130] is the target braking deceleration; is the tractor acceleration; is the trailer acceleration;
[0131] is the left wheel turning angle of the front axle of the tractor; is the heading angle of the tractor; is the heading angle of the trailer;
[0132] is the yaw angular velocity of the tractor; is the trailer yaw angular velocity;
[0133] The maximum chargeable power of the vehicle's current battery;
[0134] is the speed of each motor, ;
[0135] is the actual pressure of each air brake at present, .
[0136] S202: Define the actions of the agent based on the dynamics model of multi-axle electric heavy truck braking for:
[0137] ;
[0138] in:
[0139] is the target torque of each motor,
[0140] is the target pressure of each air brake, ;
[0141] It should be noted that, since the motor responds very quickly, its actual torque can usually stably follow the target torque, while the response speed of the air brake is relatively slow, and there is often a delay and steady-state difference between its actual pressure and the target pressure.
[0142] S203, setting four reward functions, respectively considering the follow-up realization of braking intention, braking energy recovery rate, folding danger and locking and skidding. Since the definition of the reward function directly affects the training effect of the control algorithm, the method combines the four reward functions and the penalty term to obtain the final reward function, which is as follows:
[0143] S2031. Set the first reward function (braking effect reward function) as follows:
[0144] ;
[0145] in, is the weight coefficient. The Gaussian error function is used in this function to measure the braking intensity error, and its value range is between 0 and 1;
[0146] It should be noted that in this function, when the braking intensity error is small, the amount subtracted from the total reward will actually be smaller, and the net reward obtained by the agent will be higher;
[0147] It should be noted that in order to ensure that the control strategy can effectively follow the driver's braking intention, the first reward function is set to the ability to follow the expected braking intensity / target deceleration. In this item, the braking decelerations of the tractor and trailer are considered separately, and the matching degree of the above two factors with the target deceleration is simultaneously evaluated in the reward function. Ultimately, it can ensure that both the tractor and the trailer can achieve the desired braking effect, while avoiding potential dangers caused by inconsistent braking.
[0148] S2032. Set the second reward function (brake energy recovery rate reward function) as follows:
[0149] ;
[0150] in:
[0151] is the motor braking power;
[0152] is the air pressure brake power;
[0153] is the weight coefficient;
[0154] It should be noted that the second reward function is the ratio of electric braking power to total braking power, which takes into account the braking energy recovery rate. The larger the proportion of electric braking power, the higher the braking energy recovery rate.
[0155] S2033. Set the third reward function (dangerous working condition reward function) as follows:
[0156] ;
[0157] in:
[0158] is the difference in heading angles between the tractor and trailer, i.e., the actual articulation angle;
[0159] is the yaw rate difference threshold, which is set according to actual applications and test data to ensure that the model has moderate sensitivity to risk;
[0160] is the weight coefficient;
[0161] It should be noted that this function comprehensively considers the constraints of the articulation angles of the tractor and trailer and the inconsistency of the yaw rate. The greater the difference, the higher the risk and the corresponding greater the reward and penalty.
[0162] It should be noted that the third reward function takes into account the risk of folding and can ensure that dangerous conditions such as folding and skidding do not occur during vehicle braking. Since the articulation angle between the tractor and the trailer cannot be accurately measured, the heading angle difference and yaw angular velocity difference between the tractor and the trailer are comprehensively considered in this function.
[0163] S2034. Set the preliminary fourth reward function as follows:
[0164] ;
[0165] in:
[0166] is the Gaussian distribution function parameter;
[0167] is the slip rate of each wheel of the vehicle, which is obtained from the vehicle speed and the wheel speed of each wheel;
[0168] is the maximum road adhesion coefficient of the road surface;
[0169] is the weight coefficient;
[0170] It should be noted that the fourth reward function is related to the slip rate. Ideally, the slip rate should be kept in an optimal range and locking should not occur as much as possible to ensure the stability and braking efficiency of the vehicle. In this function, a smooth function is used to provide rewards for the slip rate of each wheel.
[0171] It should be noted that the fourth reward function is constructed based on the Gaussian distribution function, which can ensure that the agent smoothly provides high rewards near the maximum adhesion rate (i.e., the road adhesion coefficient) and gradually reduces the reward as the slip rate deviates from the optimal value; in this function, a piecewise function is designed according to the functional relationship between the slip rate and the adhesion rate, and different parameters can be set on both sides of the maximum adhesion rate. In this way, the reward on the left side can decay faster, thereby motivating the agent to control the slip rate within a range closer to the road adhesion coefficient.
[0172] S2035. Considering the wheel locking situation, a penalty term, i.e., a penalty function, is set to introduce a penalty to the agent to reduce the reward when the wheel is close to locking (i.e., the slip rate is close to 1). The penalty function is as follows:
[0173] ;
[0174] in;
[0175] is the weight coefficient;
[0176] To control the locking penalty growth rate coefficient, when the slip rate As it approaches 1 (i.e. wheel locking), the penalty increases rapidly.
[0177] S2036. The fourth reward function and the locking penalty function of all wheels of the vehicle are combined to obtain the following final fourth reward function (slip rate reward function), as follows:
[0178] .
[0179] S2037. Finally, taking into account the various factors that affect the braking energy recovery control strategy, according to the different weights of the energy recovery target and the braking performance target, based on the weighted sum of the above four reward functions, the total reward function of the braking energy recovery control strategy is set as follows:
[0180] ;
[0181] in, are weight coefficients respectively, and their specific values are determined according to experimental parameters.
[0182] S30, establishing a training environment and collecting experience data, including:
[0183] In this step, a high-precision training environment is built according to the vehicle parameters of the multi-axle heavy truck, thereby reducing the actual vehicle test, improving the training efficiency and reducing the training cost; a variety of driving conditions are configured in the training environment, thereby fully covering a variety of real driving conditions of the multi-axle heavy truck; based on the different braking intensity requirements of the given training vehicle under different driving conditions, the experience data of the corresponding state and action are obtained, and the obtained data is pre-processed and stored in the experience pool; of course, the relevant actual vehicle control data can also be recorded and added to the experience pool after processing, as follows:
[0184] S301. Use the trucksim platform to build an intelligent training environment and establish a high-precision multi-axle heavy-duty truck simulation model to ensure that the parameters of the model are consistent with those of the real vehicle.
[0185] S302. Design 10 driving conditions for multi-axle heavy trucks. The training conditions fully cover the real driving conditions of multi-axle heavy trucks, including extreme and dangerous conditions.
[0186] The operating conditions include: low-adhesion straight-line driving, high- and low-adhesion interchange straight-line driving, straight-line open road, high-adhesion small-curvature curve driving, low-adhesion small-curvature curve driving, high- and low-adhesion interchange small-curvature curve driving, small-curvature curve open road, high-adhesion large-curvature curve driving, low-adhesion large-curvature curve driving and high- and low-adhesion interchange large-curvature curve driving;
[0187] Among them, the low adhesion coefficient in the working condition is set to 0.4, and the high adhesion road surface is set to 0.8; the curvature radius of the small curvature curve is set to 500m, and the curvature radius of the large curvature curve is set to 250m; each working condition is set according to the initial speed of 40km / h, 50km / h, 70km / h, and 80km / h, and the braking is performed at 25%, 50%, 75%, and 100% brake pedal opening.
[0188] S303, performing simulation training under the above-mentioned various working conditions, collecting real-time data of the vehicle through the data acquisition module, and obtaining detailed vehicle status information;
[0189] The collected information includes: the current state, action, reward and the state at the next moment, that is, ;
[0190] Among them, when collecting data, the information tuple obtained at each time step is Store them in the experience replay pool, and then randomly sample them to update the main network parameters;
[0191] Among them, the first data collection adopts a manually written braking control strategy to ensure that the initial braking energy recovery control strategy is theoretically correct; subsequently, the reinforcement learning control strategy is adopted to obtain training data according to the training process.
[0192] S304: After the data information is collected, the data is filtered, denoised and calibrated by the data filtering module in the data processing module to improve the authenticity and accuracy of the data;
[0193] Among them, the data processing module is used to normalize the data based on the Min-Max normalization method, thereby improving efficiency and ensuring the extraction of features with smaller values, and mapping the value ranges of different features to the same scale. The algorithm is as follows:
[0194] ;
[0195] in:
[0196] is the original data;
[0197] is the normalized data;
[0198] is the maximum value of the corresponding variable;
[0199] is the minimum value of the corresponding variable;
[0200] Among them, the original data is mapped to The interval makes the data more evenly distributed, avoids the increase of model training error caused by large differences in different feature value ranges, and improves accuracy and training efficiency.
[0201] S40, training of vehicle braking energy recovery strategy for multi-axle electric heavy trucks, including:
[0202] In this step, based on the above network, function, training scenario and data, the vehicle braking energy recovery strategy of the multi-axle electric heavy truck is trained, and the network parameters are updated and tuned, as follows:
[0203] Among them, TD3 uses two value networks and one strategy network, namely:
[0204] , ;in, and , is the policy network, s is the state, a is the action, and are the network parameters of the two value networks, are the network parameters of the policy network;
[0205] Among them, the three target networks corresponding to the three neural networks are used together as follows:
[0206] , , ;in, , , are two target value networks, is the target policy network, s is the state, a is the action, and are the network parameters of the two target value networks, are the network parameters of the target policy network;
[0207] Among them, the policy network and value network are initialized before training, and their parameters are randomly selected as follows:
[0208] , , ;in, Representing the target value network Parameters Initialized as a value network Parameters ; Representing the target value network Parameters Initialized as a value network Parameters ; Represents the target policy network Network parameters Initialized as a policy network Parameters ;
[0209] When training the policy network and the value network, a quadruple is randomly drawn from the experience replay pool each time, denoted as ; Use subscript Indicates the current parameters of the neural network, using subscript represents the updated parameters;
[0210] After completing the above initialization and preparation, the following training steps are performed. During the training process, the value network, policy network and target network are updated, and the network parameters are continuously updated iteratively to improve the level of each network and maximize the control reward;
[0211] The training steps include:
[0212] S401, call the target network to make predictions, as follows:
[0213] ;
[0214] Among them, the vector Each element of Independent extraction.
[0215] S402, call two target value networks to make predictions, as follows:
[0216] ;
[0217] .
[0218] S403, calculate the TD target as follows:
[0219] .
[0220] S404, call two value networks to make predictions, as follows:
[0221] ;
[0222] .
[0223] S405, calculate the TD error as follows:
[0224] ;
[0225] .
[0226] S406. Update the value network as follows:
[0227] ;
[0228] .
[0229] S407. Update the policy network and the three target networks every k rounds:
[0230] S4071, call the policy network to make predictions, as follows:
[0231] .
[0232] S4072, update the policy network as follows:
[0233] .
[0234] S4073. Update the parameters of the target network as follows:
[0235] ;
[0236] ;
[0237] .
[0238] S4074. Repeat the above steps using new strategy parameters until the cumulative reward converges to the maximum, and one training is completed.
[0239] S4075. Execute multiple training steps according to the results of the previous round of training to achieve multiple rounds of iterative training, obtain the optimal training results, and complete the training of the braking energy recovery control strategy.
[0240] It should be noted that, in practical applications, other reinforcement learning methods such as DDPG and PPO can be used to construct the braking energy recovery control strategy, but TD3 can improve learning stability and performance compared with the DDPG algorithm, and its key technology, the truncated dual Q learning algorithm, can better solve the overestimation problem; furthermore, compared with the PPO algorithm, TD3 is a different strategy with higher training efficiency; but in summary, the present invention does not make a unique limitation on the specific reinforcement learning method, and it can be selected according to specific needs.
[0241] S2. Deploy the control strategy to the heavy truck and optimize it:
[0242] In this step, the trained policy network is deployed to a multi-axle heavy truck to achieve optimal control of the braking energy recovery process;
[0243] The strategy network with the best result of reinforcement learning training is deployed to multi-axle electric heavy trucks, and the strategy network is called to output the air pressure of each air brake according to the expected braking intensity of the working condition. , and output each motor braking torque at the same time , and finally realize the braking energy recovery control;
[0244] When the strategy is applied, relevant status data is recorded to further optimize the control strategy.
[0245] In summary, this scheme is based on the multi-axle heavy-duty truck braking model, establishes constraints for the braking force distribution, defines the state quantity, control quantity and reward function of the intelligent agent, trains the deep reinforcement learning network based on the training environment and a large amount of data, and realizes efficient energy recovery through network deployment; in actual application, the control network deployed to the heavy-duty truck can reasonably distribute the vehicle's axles, as well as the electric braking torque and air brake pressure, and can also simultaneously adjust the braking torque difference between the left and right motors of the tractor's central axle, further optimizing the vehicle's braking stability.
[0246] It should also be noted that when a heavy-duty truck is traveling at a low speed (i.e., the driving speed is <3km / h), the electric braking power is too small, the braking time is short, the recovered energy is extremely small, the inherent working characteristics of the motor change greatly, and the motor cannot recover energy when the vehicle stops; therefore, the energy recovery control method of the present application is mainly used in the high-speed driving stage of multi-axle electric heavy-duty trucks (i.e., the driving speed ≥3km / h); when the multi-axle electric heavy-duty truck is in the low-speed driving stage (i.e., the driving speed <3km / h), the control method is not limited, for example, the air brake pressure can be controlled only by the brake pedal.
[0247] It should be noted that all the above examples are only for explaining the present invention and cannot limit the protection scope of the present invention.
[0248] Embodiment 2: This embodiment is based on the same inventive concept as the multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning described in Embodiment 1, and provides a multi-axle electric heavy truck braking energy recovery control system based on reinforcement learning, including:
[0249] A framework construction module is used to: construct a braking energy recovery control framework based on deep reinforcement learning, wherein: a deep reinforcement learning control strategy is configured, the agent state, action and reward of the deep reinforcement learning control strategy are defined, a multi-training environment is established and experience data is collected, and the deep reinforcement learning control strategy is trained based on the agent state, action and reward, the multi-training environment and experience data;
[0250] A strategy deployment module, used to: deploy the trained deep reinforcement learning control strategy as a braking energy recovery strategy to a multi-axle electric heavy truck;
[0251] The strategy calling module is used to: call the braking energy recovery strategy to output the control instructions of the multi-axle electric heavy truck according to the target input data of the multi-axle electric heavy truck.
[0252] Embodiment 3: This embodiment provides a computer-readable storage medium, including:
[0253] The storage medium is used to store computer software instructions used to implement the multi-axis electric heavy-duty truck braking energy recovery control method based on reinforcement learning described in the above-mentioned Example 1, which includes a program for executing the above-mentioned program set for the multi-axis electric heavy-duty truck braking energy recovery control method based on reinforcement learning; specifically, the executable program can be built into the multi-axis electric heavy-duty truck braking energy recovery control system based on reinforcement learning described in Example 2, so that the multi-axis electric heavy-duty truck braking energy recovery control system based on reinforcement learning can implement the multi-axis electric heavy-duty truck braking energy recovery control method based on reinforcement learning described in Example 1 by executing the built-in executable program.
[0254] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof.
[0255] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.
[0256] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0257] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this article.
[0258] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0259] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.
[0260] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.
[0261] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
[0262] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0263] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A multi-axis electric heavy truck braking energy recovery control method based on reinforcement learning, characterized in that: The following steps are involved: Constructing a braking energy recovery control framework based on deep reinforcement learning, wherein: configuring a deep reinforcement learning control strategy, defining the agent state, action and reward of the deep reinforcement learning control strategy, establishing a multi-dimensional training environment and collecting experience data, and training the deep reinforcement learning control strategy based on the agent state, action and reward, the multi-dimensional training environment and the experience data; Deploy the trained deep reinforcement learning control strategy as a braking energy recovery strategy to a multi-axle electric heavy truck, and call the braking energy recovery strategy to output a control instruction of the multi-axle electric heavy truck according to the target input data of the multi-axle electric heavy truck; The configuration of the deep reinforcement learning control strategy includes: setting a deep neural network based on two value networks, one policy network and three target networks; the three target networks correspond to the two value networks and one policy network respectively; the policy network is used to make actions according to the state; the value network is used to evaluate the quality of the corresponding action under the state; the target network is used to cut off the bootstrap phenomenon in the network training process; The defining of the agent state, action and reward of the deep reinforcement learning control strategy includes: defining the agent state as: ;in, is the state of the agent; is the wheel speed, is the tractor speed; is the trailer speed; is the target braking deceleration; is the tractor acceleration; is the trailer acceleration; is the left wheel turning angle of the front axle of the tractor; is the heading angle of the tractor; is the heading angle of the trailer; is the yaw angular velocity of the tractor; is the trailer yaw angular velocity; The maximum chargeable power of the vehicle's current battery; is the speed of each motor, ; is the actual pressure of each air brake at present, ; Define the actions of the agent as: ;in, is the action of the agent; is the target torque of each motor, is the target pressure of each air brake, ; Set a braking effect reward function, a braking energy recovery rate reward function, a dangerous working condition reward function and a slip rate reward function, and take the weighted sum of the braking effect reward function, the braking energy recovery rate reward function, the dangerous working condition reward function and the slip rate reward function as the total reward function.
2. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 1 is characterized in that: The braking effect reward function is: ; Among them, r1 is the braking effect reward; a1 is the weight coefficient.
3. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 2 is characterized in that: The braking energy recovery rate reward function is: ; in, Reward for braking energy recovery; is the motor braking power; is the air pressure brake power; is the weight coefficient.
4. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 3 is characterized in that: The dangerous working condition reward function is: ; in, Rewards for hazardous working conditions; is the heading angle difference between the tractor and trailer; is the yaw rate difference threshold; is the weight coefficient.
5. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 4 is characterized in that: The definition operation of the slip rate reward function includes: Set up a preliminary fourth reward function: ; in, for a preliminary fourth award regarding slippage; is the Gaussian distribution function parameter; is the slip rate of each wheel of the vehicle; is the maximum road adhesion coefficient of the road surface; is the weight coefficient; Set the locking penalty function: ; in; Punishment for holding death; is the weight coefficient; A coefficient to control the growth rate of the locking penalty; The slip rate reward function is obtained by integrating the fourth reward function and the locking penalty function of all wheels of the multi-axle electric heavy truck: ; in, Bonus for slip rate.
6. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 5 is characterized in that: The establishing of a multi-dimensional training environment and collecting empirical data includes: Establish a high-precision multi-axle heavy truck simulation model; Design the driving conditions of several multi-axle heavy trucks; Perform simulation training of the high-precision multi-axle heavy truck vehicle simulation model under the several driving conditions, collect real-time data of the vehicle simulation model, and obtain vehicle status information; Filter, reduce noise and calibrate the collected data; Normalizing the data after filtering, noise reduction and calibration; The normalized data are used as the empirical data.
7. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 1 is characterized in that: The step of training the deep reinforcement learning control strategy based on the agent state, action and reward, the multi-training environment and the experience data includes: Initializing the neural network parameters of the deep reinforcement learning control strategy, and then looping through the training steps until the cumulative reward converges to a maximum; The initialization includes: Use two value networks and one policy network, namely: , ;in, and , is the policy network, s is the state, a is the action, and are the network parameters of the two value networks, are the network parameters of the policy network; Use the three target networks corresponding to the three neural networks together: , , ;in, , , are two target value networks, is the target policy network, s is the state, a is the action, and are the network parameters of the two target value networks, are the network parameters of the target policy network; Initialize the policy network and value network before training, and randomly select parameters: , , ;in, Representing the target value network Parameters Initialized as a value network Parameters ; Representing the target value network Parameters Initialized as a value network Parameters ; Represents the target policy network Network parameters Initialized as a policy network Parameters ; When training the policy network and the value network, a quadruple is randomly drawn from the experience replay pool each time, denoted as ; Use subscript Indicates the current parameters of the neural network, using subscript Indicates the updated parameters.
8. The multi-axle electric heavy truck braking energy recovery control method based on reinforcement learning according to claim 7 is characterized in that: The training step comprises: Call the target network to make predictions; Call two target value networks to make predictions; Calculate TD target; Call two value networks to make predictions; Calculate TD error; Update the value network; The policy network and the three target networks are updated every k rounds.
