A method and system for energy management of a tractor-semitrailer based on uncertainty guidance

Abstract

The present application relates to a kind of based on uncertainty guidance's tractor semi-trailer energy management method and system, the method comprises: S1. Vehicle selection, the interactive information that intelligent agent is respectively and middle axle three gears, rear axle two gears TS is obtained, and experience sample is generated according to action, state and reward and next state;S2. The calculation of uncertainty, initialize K structure same single parameter independent value network Q1, Q2, …, Qk, wherein, k≥2;S3. The calculation of priority;S4. Energy management strategy based on uncertainty guidance priority experience replay.The present application not only sample utilization rate improves, high-value sample is preferentially learned, low-value sample is not wasted calculation, and also has higher training efficiency under long time multi-working condition training.

Description

Technical Field

[0001] This invention relates to the field of control technology for tractor-trailers, and in particular to an uncertainty-guided energy management method and system for tractor-trailers. Background Technology

[0002] Reinforcement learning algorithms have recently been a focus of research in energy management, as many complex energy management strategies are difficult to solve using traditional rule-based and optimization methods. The tractor-semi-trailer (TS), as one of the most promising low-carbon transportation solutions, has received widespread attention in recent years. The challenge facing BEV powertrain systems is to improve vehicle fuel economy while providing sufficient dynamic performance. Energy Management Strategy (EMS) is a mainstream approach widely used to address the aforementioned TS problems. EMS can more rationally select the motor speed and model, achieving effective energy coordination.

[0003] With the continuous development of EMS, there are more and more EMS methods applied to BEVs. The common technologies include the following: I. Rule-based EMS: Although rule-based EMS is simple, it has the following drawbacks: 1. Design rules require extensive expert experience as a reference; 2. The well-designed EMS has poor adaptability and cannot be widely applied to other different types of working conditions; 3. The design is difficult in complex working conditions; II. Optimized EMS systems are generally divided into local optima and global optima, and have the following drawbacks: 1. Requires global information to be known or unknown operating conditions to be obtained in advance; 2. It takes a long time to process, and most of them are not real-time. 3. High design difficulty under complex working conditions; III. While EMS based on deep reinforcement learning algorithms performs well in applications, it has the following drawbacks: 1. It lacks the ability to adaptively explore states based on model cognition, and cannot dynamically adjust the exploration intensity according to the uncertainty of the current state-action value, resulting in ineffective exploration in areas that have already been fully learned.

[0004] When rewards are sparse, a large number of TD errors are zero, and the agent cannot distinguish which samples are "more worth learning"; it is difficult to discover positive reward paths, resulting in poor sample utilization. Summary of the Invention

[0005] In view of the above problems, the present invention provides an uncertainty-guided energy management method and system for tractor semi-trailers, which not only improves sample utilization, prioritizes the learning of high-value samples, and avoids wasting computation on low-value samples, but also has higher training efficiency under long-term multi-condition training.

[0006] To achieve the above and other related objectives, the present invention provides the following technical solution: An uncertainty-guided energy management method for tractor-trailers, the method comprising: S1. Vehicle selection: Obtain interaction information between the intelligent agent and the middle axle in three gears and the rear axle in two gears TS respectively, and generate experience samples based on actions, states, rewards and the next state. S2. Uncertainty calculation: Initialize K independent value networks Q1, Q2, ..., Qk with identical structure and single parameters, where k≥2; S3. Priority calculation; S4. Energy management strategy based on uncertainty-guided priority experience replay.

[0007] Furthermore, in step S2, the calculation of the uncertainty includes: S21. Each Q i The network uses different initial weights and updates with different mini-batch sample subsets during training, and for each Q... i The target network Q corresponding to the network configuration i '; S22. For any sample (s, a, r, s´) in the experience pool, input the state s and action a into K Q-networks respectively, and output K Q-values. ; S23. Calculate the arithmetic mean of K Q values. ; S24. Calculate the standard deviation of the K Q values. , When each experience sample is stored in the priority experience pool, its calculated uncertainty U is also saved. The uncertainty value of the sample is not updated as training progresses.

[0008] Furthermore, the uncertainty U is treated as an inherent attribute of the sample, bound to the empirical sample data, and persistently stored in the priority empirical pool. During subsequent sampling and training processes, the value of the uncertainty U remains unchanged, without any gradient updates or recalculations.

[0009] Furthermore, during the interaction between the agent and the TS (Transmission Switch) at the three gears of the middle bridge and the two gears of the rear bridge, experience samples are collected. These experience samples include states, actions, rewards, next states, and termination flags.

[0010] Furthermore, in step S3, the priority calculation includes: S31. For any sample j in the experience pool, define its sampling priority P. j It is the product of the power of the absolute value of the time difference error and the power of the uncertainty. , Where, δ j Let U be the time difference error of sample j. j Let be the uncertainty of sample j, and α and β be constant parameters; S32. For sample j=(s,a,r,s´), calculate the target Q value. , Where r is the parameter in sample j, and γ is the weight coefficient. Output the Q value, where μ´ is the arithmetic mean; S33. Calculate the current Q estimate. ; S34. Calculate the time difference error. ; S35. Calculate the probability p of each sample being sampled according to priority. j , , Among them, P j K and P m K Sampling priority P j The components, where m is a positive integer.

[0011] Furthermore, in step S35, the probability p of each sample being sampled is calculated according to the priority. j To improve the selection of multiple samples using a prioritization method, an uncertainty in action value is introduced on top of the TD error to construct a comprehensive priority system, which is then assigned to p. j Training is performed by selecting empirical samples, which allows the agent to prioritize learning regions where predictions are inaccurate and cognition is uncertain.

[0012] Furthermore, in step S4, the energy management strategy based on uncertainty-guided priority experience replay includes: S41. The reinforcement learning agent interacts with the middle bridge at three gears and the rear bridge at two gears TS, storing the state, actions, and rewards during the interaction process into the experience pool; S42. During training, initialize K independent value networks with identical structures and single parameters, and calculate the uncertainty; S43. Calculate the sampling priority and sampling probability for each network, and select empirical samples from the empirical pool for training based on the sampling probability; S44. Verify the effectiveness of the trained EMS.

[0013] Furthermore, the states during the interaction process include vehicle speed, acceleration, and required torque.

[0014] Furthermore, the actions during the interaction process include the selection of twelve working modes, which can be divided into twelve different driving modes based on the different transmission ratios of the middle and rear axle motors.

[0015] To achieve the above and other related objectives, the present invention also provides an uncertainty-guided tractor-trailer energy management system, including a computer device programmed or configured to perform the steps of any of the uncertainty-guided tractor-trailer energy management methods described herein.

[0016] The present invention has the following positive effects: This invention maintains the uncertainty of state-action pairs calculated by multiple parallel value networks, and uses it in a weighted fusion with time difference error as the sampling priority of experience samples. This guides the agent to prioritize learning experience in knowledge-poor regions, significantly improving the sample efficiency, exploration ability and training stability of the deep deterministic policy gradient algorithm. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the vehicle configuration of the present invention; Figure 2 This is a schematic diagram illustrating the uncertainty calculation of the present invention; Figure 3 This is a schematic diagram illustrating the calculation of the priority of the present invention; Figure 4 This is a schematic diagram of the energy management strategy based on uncertainty-guided priority experience replay of the present invention.

[0018] The labels in the diagram are as follows: 1-Differential, 2-Middle and rear axle drive shafts, 3-Middle axle motor, 4-Rear axle motor. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] Example: An uncertainty-guided energy management method for tractor-trailers, the method comprising: S1. Vehicle selection: Obtain interaction information between the intelligent agent and the middle axle in three gears and the rear axle in two gears TS respectively, and generate experience samples based on actions, states, rewards and the next state. S2. Uncertainty calculation: Initialize K independent value networks Q1, Q2, ..., Qk with identical structure and single parameters, where k≥2; S3. Priority calculation; S4. Energy management strategy based on uncertainty-guided priority experience replay.

[0021] In this embodiment, as Figure 1 As shown, differential 1 is connected to rear axle motor 4 via middle and rear axle drive shaft 2.

[0022] In this embodiment, as Figure 2 As shown, in step S2, the calculation of the uncertainty includes: S21. Each Q i The network uses different initial weights and updates with different mini-batch sample subsets during training, and for each Q... i The target network Q corresponding to the network configuration i '; S22. For any sample (s, a, r, s´) in the experience pool, input the state s and action a into K Q-networks respectively, and output K Q-values. ; S23. Calculate the arithmetic mean of K Q values. ; S24. Calculate the standard deviation of the K Q values. , When each experience sample is stored in the priority experience pool, its calculated uncertainty U is also saved. The uncertainty value of the sample is not updated as training progresses.

[0023] In this embodiment, the uncertainty U is treated as an inherent property of the sample, bound to the empirical sample data, and persistently stored in the priority empirical pool. During subsequent sampling and training processes, the value of the uncertainty U remains unchanged, without any gradient updates or recalculations.

[0024] In this embodiment, during the interaction between the agent and the TS (Transmission Switch) at the middle bridge and the rear bridge, experience samples are collected. These experience samples include states, actions, rewards, next states, and termination flags.

[0025] In this embodiment, as Figure 3 As shown, in step S3, the priority calculation includes: S31. For any sample j in the experience pool, define its sampling priority P. j It is the product of the power of the absolute value of the time difference error and the power of the uncertainty. , Where, δ j Let U be the time difference error of sample j. j Let be the uncertainty of sample j, and α and β be constant parameters; S32. For sample j=(s,a,r,s´), calculate the target Q value. , Where r is the parameter in sample j, and γ is the weight coefficient. Output the Q value, where μ´ is the arithmetic mean; S33. Calculate the current Q estimate. ; S34. Calculate the time difference error. ; S35. Calculate the probability p of each sample being sampled according to priority. j , , Among them, P j K and P m K Sampling priority P j The components, where m is a positive integer.

[0026] In this embodiment, in step S35, the probability p of each sample being sampled is calculated according to the priority. j To improve the selection of multiple samples using a prioritization method, an uncertainty in action value is introduced on top of the TD error to construct a comprehensive priority system, which is then assigned to p. jTraining is performed by selecting empirical samples, which allows the agent to prioritize learning regions where predictions are inaccurate and cognition is uncertain.

[0027] In this embodiment, as Figure 4 As shown, in step S4, the energy management strategy based on uncertainty-guided priority experience replay includes: S41. The reinforcement learning agent interacts with the middle bridge at three gears and the rear bridge at two gears TS, storing the state, actions, and rewards during the interaction process into the experience pool; S42. During training, initialize K independent value networks with identical structures and single parameters, and calculate the uncertainty; S43. Calculate the sampling priority and sampling probability for each network, and select empirical samples from the empirical pool for training based on the sampling probability; S44. Verify the effectiveness of the trained EMS.

[0028] In this embodiment, the states during the interaction process include vehicle speed, acceleration, and required torque.

[0029] In this embodiment, the actions during the interaction process include the selection of twelve working modes. The selection of the twelve working modes is based on the different transmission ratios of the middle axle and rear axle motors, which can be divided into twelve different driving modes.

[0030] In this embodiment, the present invention provides an uncertainty-guided tractor-trailer energy management system, including a computer device programmed or configured to perform the steps of any of the uncertainty-guided tractor-trailer energy management methods described herein.

[0031] The present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform the aforementioned uncertainty-guided traction semi-trailer energy management method.

[0032] Any references to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0033] In summary, this invention not only improves sample utilization, prioritizes the learning of high-value samples, and avoids wasting computation on low-value samples, but also achieves higher training efficiency under long-term multi-condition training.

[0034] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An uncertainty-guided energy management method for tractor-trailers, characterized in that, The method includes: S1. Vehicle selection: Obtain interaction information between the intelligent agent and the middle axle in three gears and the rear axle in two gears TS respectively, and generate experience samples based on actions, states, rewards and the next state. S2. Uncertainty calculation: Initialize K independent value networks Q1, Q2, ..., Qk with identical structure and single parameters, where k≥2; S3. Priority calculation; S4. Energy management strategy based on uncertainty-guided priority experience replay.

2. The uncertainty-guided energy management method for tractor-trailers according to claim 1, characterized in that, In step S2, the calculation of the uncertainty includes: S21. Each Q i The network uses different initial weights and updates with different mini-batch sample subsets during training, and for each Q... i The target network Q corresponding to the network configuration i '; S22. For any sample (s, a, r, s´) in the experience pool, input the state s and action a into K Q-networks respectively, and output K Q-values. ； S23. Calculate the arithmetic mean of K Q values. ； S24. Calculate the standard deviation of the K Q values. ， When each experience sample is stored in the priority experience pool, its calculated uncertainty U is also saved. The uncertainty value of the sample is not updated as training progresses.

3. The uncertainty-guided energy management method for tractor-trailers according to claim 2, characterized in that: The uncertainty U is treated as an inherent property of the sample, bound to the empirical sample data, and persistently stored in the priority empirical pool. During subsequent sampling and training, the value of the uncertainty U remains unchanged, without any gradient updates or recalculations.

4. The uncertainty-guided energy management method for tractor-trailers according to claim 1, characterized in that: During the interaction between the agent and the TS (Transmission Switch) at the three gears of the middle bridge and the two gears of the rear bridge, experience samples are collected. These experience samples include states, actions, rewards, next states, and termination flags.

5. The uncertainty-guided energy management method for tractor-trailers according to claim 1, characterized in that, In step S3, the priority calculation includes: S31. For any sample j in the experience pool, define its sampling priority P. j It is the product of the power of the absolute value of the time difference error and the power of the uncertainty. ， Where, δ j Let U be the time difference error of sample j. j Let be the uncertainty of sample j, and α and β be constant parameters; S32. For sample j=(s,a,r,s´), calculate the target Q value. ， Where r is the parameter in sample j, and γ is the weight coefficient. Output the Q value, where μ´ is the arithmetic mean; S33. Calculate the current Q estimate. ； S34. Calculate the time difference error. ； S35. Calculate the probability p of each sample being sampled according to priority. j , ， Among them, P j K and P m K Sampling priority P j The components, where m is a positive integer.

6. The uncertainty-guided energy management method for tractor-trailers according to claim 5, characterized in that, In step S35, the probability p of each sample being sampled is calculated according to the priority. j To improve the selection of multiple samples using a prioritization method, an uncertainty in action value is introduced on top of the TD error to construct a comprehensive priority system, which is then divided into p-order priority systems. j Training is performed by selecting empirical samples, which allows the agent to prioritize learning regions where predictions are inaccurate and cognition is uncertain.

7. The uncertainty-guided energy management method for tractor-trailers according to claim 1, characterized in that, In step S4, the energy management strategy based on uncertainty-guided priority experience replay includes: S41. The reinforcement learning agent interacts with the middle bridge at three gears and the rear bridge at two gears TS, storing the state, actions, and rewards during the interaction process into the experience pool; S42. During training, initialize K independent value networks with identical structures and single parameters, and calculate the uncertainty; S43. Calculate the sampling priority and sampling probability for each network, and select empirical samples from the empirical pool for training based on the sampling probability; S44. Verify the effectiveness of the trained EMS.

8. The uncertainty-guided energy management method for tractor-trailers according to claim 7, characterized in that: The states during the interaction process include vehicle speed, acceleration, and required torque.

9. The uncertainty-guided energy management method for tractor-trailers according to claim 7, characterized in that: The actions during the interaction process include the selection of twelve working modes. These twelve working modes are selected based on the different transmission ratios of the middle and rear axle motors, which can be divided into twelve different driving modes.

10. An uncertainty-guided energy management system for a tractor-trailer, comprising computer equipment, characterized in that, The computer device is programmed or configured to perform the steps of the uncertainty-guided energy management method for tractor-trailers as described in any one of claims 1 to 9.

Images