Hybrid electric vehicle energy management method considering working condition recognition, computer readable storage medium and computer program product

Abstract

The invention discloses a hybrid electric vehicle energy management method considering working condition recognition, and the method comprises the steps: carrying out the real-time classification of vehicle driving working conditions based on a working condition recognition network of a long and short term memory attention mechanism, and designing a self-adaptive adversarial dual depth Q network AD-D3QN energy management algorithm based on the classification. According to the algorithm, dynamic planning expert demonstration experience and a reward weight self-adaptive adjustment mechanism are combined, and optimal distribution of engine and motor torque is achieved. The method has the advantages that working condition adaptability is improved, the training convergence speed is increased, and economical efficiency and stability are improved.

Description

Technical Field

[0001] This invention relates to the field of hybrid electric vehicle technology, and in particular to a hybrid electric vehicle energy management method that takes into account operating condition identification, a computer-readable storage medium, and a computer program product. Background Technology

[0002] The energy management methods used in hybrid electric vehicles (HEVs) largely determine the vehicle's fuel economy and power performance. Currently, common energy improvement methods can be mainly divided into three categories: rule-based methods, optimization-based methods, and learning-based methods.

[0003] Rule-based methods rely on manually set switching rules to allocate power output between the engine and motor. While simple to implement and computationally inexpensive, they lack adaptability and cannot dynamically adjust energy distribution strategies based on real-time driving conditions, resulting in limited fuel economy. Optimization-based methods, such as dynamic programming (DP), Pontryagin's optimal principle (PMP), and equivalent fuel minimization strategy (ECMS), can obtain relatively optimal or globally optimal solutions under known complete driving conditions, providing theoretical reference value. However, DP cannot run in real-time, PMP relies on an accurate model, and ECMS only considers instantaneous optima, depending on complete driving condition information or an accurate model, failing to meet the requirements of real-time operation and thus limiting their practical application.

[0004] In recent years, learning-based methods have gradually gained widespread attention in the field of hybrid vehicle energy management. Reinforcement learning (RL), with its characteristics of direct interaction with the environment and autonomous learning of optimal control strategies through trial and error, effectively eliminates the strong dependence on precise prior operating condition models and shows good theoretical potential. However, current mainstream deep reinforcement learning algorithms, such as DQN, DDPG, and their many variants, still face several significant bottlenecks in actual training: generally slow convergence speed, extreme sensitivity to hyperparameter combinations, drastic policy fluctuations in the early and middle stages of training, and low sample efficiency. More importantly, their generalization performance drops sharply under unseen or significantly offset driving conditions, making it difficult to meet the stringent requirements of control stability and reliability for long-term vehicle operation. At the same time, much existing research still focuses on improving overall economic indicators (equivalent fuel consumption, SOC maintenance capability, etc.), while paying insufficient attention to key engineering attributes such as dynamic perception and recognition of online operating conditions, efficiency improvement of the training process, and robust adaptability of the strategy under cross-scenario and cross-driving style conditions. This, to some extent, limits the progress of deep reinforcement learning methods in practical controller implementation.

[0005] Given the above limitations, it has gradually become clear that relying solely on blank-slate deep reinforcement learning is insufficient to simultaneously meet the multiple requirements of real-time performance, high cost-effectiveness, and strong generalization ability. A more engineering-feasible and widely explored approach is to organically integrate domain prior knowledge with data-driven reinforcement learning. Among them, the two most complementary sources of prior information are: (1) online identification and context awareness of driving conditions, which extract features such as speed-acceleration sequences and power demand patterns, and use classifiers or lightweight networks to provide condition category / feature representations in advance or synchronously, thereby giving the policy a certain situational adaptability; (2) high-quality expert demonstration data accumulated in engineering practice: rule controllers, DP global optimization results, ECMS real-time policies, excellent driver trajectories, etc., which are injected into the training process through various methods such as behavior cloning, imitation learning (GAIL), DQfD, reward shaping, and expert-guided exploration, so as to significantly accelerate convergence, smooth early policies, compress ineffective exploration space, and provide more physically meaningful initial behaviors. Therefore, how to systematically integrate the three modules of online driving condition recognition, expert demonstration experience guidance, and deep reinforcement learning for continuous optimization, and build a hybrid energy management strategy that performs well in terms of sample efficiency, training stability, real-time execution capability, cross-condition economy, and long-term reliability, remains an open core problem with significant theoretical value and outstanding engineering significance in the field of intelligent energy management of hybrid power. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a hybrid electric vehicle energy management method, a computer-readable storage medium, and a computer program product that take into account operating condition identification, thereby improving operating condition adaptability, accelerating training convergence speed, improving economy and stability, significantly reducing fuel consumption under different operating conditions and maintaining stable battery state of charge, and improving the economy, robustness and engineering application reliability of hybrid electric vehicles.

[0007] To address the aforementioned technical problems, this invention provides a hybrid electric vehicle energy management method considering operating condition identification. Based on a Long Short-Term Memory (LSTM) attention mechanism, an operating condition identification network performs real-time classification of vehicle driving conditions. Furthermore, an Adaptive Adversarial Dual Deep Q-Network (AD-D3QN) energy management algorithm is designed. This algorithm combines dynamic programming (DP) expert demonstration experience with an adaptive adjustment mechanism for reward weights to achieve optimal allocation of engine and motor torque, reducing energy consumption while meeting power requirements. The method includes the following steps:

[0008] Step S1: Construct a working condition recognition model based on the Long Short-Term Memory (LSTM) attention mechanism; S11, Working condition cleaning; The working condition data is collected and cleaned: first, the working condition is divided into multiple segments, then representative segments are selected for further segmentation, and finally, the segmented segments are labeled with the working condition type. S12. The speed segment of the driving condition is used as the feature input of the condition recognition model. The specific steps are as follows: 1) Discretization: The standard driving cycle is divided into discrete periods. Discretize in 5s; 2) Slice extraction: at a fixed length 80-second time window, time interval Extract segments from the discretized velocity sequence in 5 seconds. ,get , Multiple samples; 3) Sample balancing: Randomly select an equal number of samples from different working condition categories to avoid too many samples from a single working condition leading to training failure; Long Short-Term Memory (LSTM) uses forgetting gates, input gates, and output gates to enhance or remove information. Its gating mechanism is related to the input sequence. The relationship is represented as: (1); (2); (3); in: It is the sigmoid activation function; ,、 and These are the activation values ​​for the forget gate, input gate, and output gate, respectively, with a value range of [0,1]. Forgotten Gate Control how much of the cell state from the previous time step is retained ; Input gate Determine how many candidate states Add the current cell state; Output gate ; The update formulas for cell state and hidden state are as follows: (4); (5); (6); in: It is the hyperbolic tangent function; and , , , Here are the weight matrix and bias vector to be trained; S13, Training of the working condition recognition model; The working condition recognition models based on LSTM and attention mechanisms include: 1) Input layer: Receives the processed velocity sequence data, with a length of 80s; 2) The first LSTM layer and the Dropout(0.1) layer: return the complete temporal output to extract rich temporal features; during training, neurons are randomly masked with a 10% probability to prevent overfitting; 3) Self-attention layer: To make the structure lightweight and easy to deploy, the structure only uses one head and a Transformer layer with a 64-dimensional hidden size to process sequences. : ; ; ; in, These are learnable parameters; 4) The second LSTM layer and Dropout(0.1) layer: Based on the output of the first layer, further refinement is performed, returning only the hidden states at the end of the sequence; Dropout is also used to prevent overfitting. 5) Fully connected layer: Maps the features extracted by LSTM onto three neurons, which are then activated by Softmax to output the predicted probabilities for three operating conditions: urban, suburban, and highway. (10); The loss function is multivariate cross-entropy: (11); in: The total number of samples; For the first The sample at the th The true label of the similar working conditions, and For the model to the first The sample at the th The predicted probability of a class.

[0009] Step S2: Build an expert demonstration knowledge base based on DP; S21. Design of an expert experience base based on dynamic programming (DP); State vector There are three: vehicle speed State of charge and required torque ,Right now .

[0010] Action space It is constructed from a two-dimensional discrete combination of engine and motor torques, that is: Engine torque exist The values ​​between Nm are discrete into 22 values, representing the motor torque. exist The values ​​of Nm are discrete, with 37 possible values. For the energy management system of a hybrid electric vehicle, dynamic programming (DP) is used to generate the globally optimal trajectory under complex operating conditions. We construct an expert experience base for optimal action trajectories based on dynamic programming (DP) to provide high-quality initial data for deep Q-networks, accelerating their convergence to the optimal policy and improving training stability.

[0011] The optimal motion trajectory covers the entire driving cycle and is associated with a discretized grid of the state space. Correspondingly.

[0012] The reward function is set as follows: ; in: and These represent the factors influencing the reward function, namely, changes in fuel consumption rate and State of Charge (SOC). The large numerical value guides the DQN agent to prioritize reducing engine fuel consumption. Big data encourages intelligent agents to more closely align with the target SOC trajectory. The target SOC is represented by 0.6.

[0013] To simplify the analysis, the energy management process of the HEV is modeled as a discrete control problem, and its state transition equation is expressed as: ;in: At the current sampling time, Let be the state transition function of the hybrid system. and These represent the state variable and control variable at the current moment, respectively.

[0014] The control quantities at all times constitute a policy sequence, which includes the control quantities at each moment of the car's movement. Output control quantity : ; The control objective of EMS is to output a sequence with minimum equivalent fuel consumption while ensuring that the state vector satisfies the constraints, described as follows: ; ; in: The objective function for EMS optimization is... This represents the sum of the equivalent fuel consumption rates of N steps. This is a penalty function that describes the system constraints.

[0015] System constraints are described in this invention as follows: ; in, and These are the upper and lower bounds of the battery's State of Charge (SOC). It is the engine's maximum torque. and These are the minimum and maximum torques of the motor, respectively.

[0016] Integrate the reward function into the DP solution process and record the four data sets in the DP policy. , , To improve generalization ability, the data set is loaded into the DQN experience replay area. The specific steps are as follows: 1) Buffer initialization: Allocate a fixed-size buffer, using a priority queue or random access structure; 2) Data loading: Load all [ , , Quadruples are filled into the buffer; 3) Priority setting: Use priority experience replay to allocate initial priorities based on the reward magnitude; 4) Distribution balance: Adjust the sampling weights to ensure that the buffer covers all states of the driving cycle.

[0017] After integrating the expert buffer into the DQN training process, pre-training is performed first. In the initial few epochs, only small batches of data are sampled from the expert buffer to train the DQN. Subsequently, mixed sampling is performed, reducing the proportion of expert data and supplementing the remaining data with data collected online to further optimize the strategy. Simultaneously, the network is updated using the Q-learning loss function to update the Q-network.

[0018] Step S3: Operating condition identification and expert demonstration of the D3QN energy management algorithm; Step S31, D3QN algorithm; 1) Dual-depth Q-network; When updating a double deep Q network, the action corresponding to the maximum objective value is still selected from the evaluation network. The selected value is then used to calculate the target value through the target network. To obtain the corresponding Q value.

[0019] Due to the target network This method offers higher stability by transforming overestimation at each step into overestimation across multiple steps (every other step). This addresses the offset that occurs in the first step, thus mitigating the overestimation problem in the DQN algorithm. In the original DQN, the target value is calculated directly from the target network. Select the maximum value corresponding to the next state. Value. In dual-depth Q-networks, the first step is to evaluate the network... Find the action that maximizes the Q value: (13); Then, in the target network In this context, this action is used to calculate the target Q value: (14); 2) Adversarial deep Q-networks; Adversarial deep Q-networks divide the Q-network into two parts, allowing the agent to more effectively handle states that are weakly related to actions: the first part is only related to the state and is called the state-value function, denoted as: The second part relates to both state and action, and is called the dominance function, denoted as: .

[0020] The final action-value function is restated as follows: (15); in: For public network weights; Dedicated weights for value branches; Dedicated weights for the dominant branches.

[0021] To improve the identifiability of the output, the advantage function is centered, which eliminates the average deviation between actions. Therefore, the final form of the action-value function is: (16); State value function It will be updated in each update phase, thus affecting the Q value of all actions.

[0022] 3) Countering dual deep Q-networks; By combining the features of the aforementioned dueling double deep Q-network and adversarial deep Q-network, we obtain the basic framework of the adversarial double deep Q-network (D3QN). Based on the characteristics of the double deep Q-network, its objective value is defined as: (17); In the formula and The adversarial deep Q-network is used to calculate based on the state value function and the advantage function; The loss function for adversarial dual deep Q-networks is expressed as: (18); Next, the parameters are updated using gradient descent. To obtain the optimal action, use : (19); in, This represents the probability of random exploration.

[0023] Adversarial dual deep Q-networks use an adversarial architecture for both the evaluation and target networks, while using a dual deep Q-network architecture in the target value calculation. This eliminates overestimation and better distinguishes similar actions.

[0024] Step S32: Adjust the weights of the reward function; LSTM is used to identify different operating conditions. Since different operating conditions have different requirements for the reward function, the parameters of the reward function are selected accordingly. This is to find the appropriate parameters in the EMS reward function. (Fuel consumption weighting) and To find the optimal weight parameters for (SOC fluctuation weights), this invention proposes a two-stage mesh-based adaptive search method. For each different working condition, the evaluation function for each weight is as follows: (20); - (twenty one); in, Represents fuel consumption. Represents the SOC offset rate. Take 0.6, parameter range: .

[0025] Will The system divides the data into segments by a specific step size, trains and simulates all weight combinations under different operating conditions, and obtains the fuel consumption and SOC offset values ​​of specific weight combinations under specific operating conditions. This allows the system to determine which combination has the best energy-saving effect under that specific operating condition. The process includes the following steps: Step S321: Grid generation; direction: It is always 1; Direction: Divided into 36 segments, step size 28.30.

[0026] Step S322: Simulation evaluation; For each grid point ( , The Simulink model was trained and simulated in parallel under three typical loops: urban area, suburbs, and highway, to calculate the merging target. Recorded as a set of tuples ; Each working condition has 36 weight combinations, and there are a total of 108 weights across the three working conditions, requiring a total of 108 offline training sessions.

[0027] Step S323: Pareto frontier screening; For a specific operating condition, three points of interest are selected from the uniformity of the above tuples, representing an optimal fuel consumption, a stable state of charge (SOC), and a middle ground. These three points constitute candidate points. Therefore, there are nine sets of candidate points for the three working conditions.

[0028] For each group of candidate points In its neighborhood Perform a small-area grid scan inside: (twenty two); Pick 14. For each candidate point, three re-simulation points are generated, requiring 18 more simulation runs. Finally, all local simulation results are summarized, and Pareto front screening is performed again for each point to obtain a more refined solution set.

[0029] Based on the above assessment, among the points undergoing local fine-tuning, the light green dots represent... and The weighted combination is most suitable for this specific operating condition, resulting in the smallest SOC offset and fuel consumption.

[0030] In addition, the present invention also provides a computer system including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the energy management method as described above.

[0031] In addition, the present invention also provides a computer storage medium, which is a computer-readable storage medium, and stores a computer program thereon, which, when executed by a processor, implements the steps of the energy management method described above.

[0032] In addition, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the energy management method as described above.

[0033] The one or more technical solutions proposed in this invention have at least the following beneficial effects: 1) Improved adaptability to driving conditions: By using a driving condition recognition model based on LSTM and attention mechanism, typical driving conditions in urban areas, suburbs and highways can be distinguished in real time, and the corresponding energy management method can be dynamically selected to achieve adaptive optimization of energy distribution and enhance the adaptability to complex driving environments. 2) Accelerate training convergence speed: By introducing dynamic programming (DP) expert demonstration experience, reconstructing the experience replay pool and adopting a hybrid sampling method, the early exploration cost is effectively reduced, the convergence speed of the reinforcement learning algorithm is accelerated, and the convergence efficiency is improved by more than 15% compared with the traditional DQN. 3) Improved fuel economy and stability: By adaptively adjusting the weights of the reward function, the optimal balance between fuel economy and battery state of charge (SOC) is achieved while ensuring stability. Simulation results show that the proposed method reduces equivalent fuel consumption by approximately 16.6% and 11.2% compared to the DQN and DDPG methods, respectively, and differs from the global optimal result of dynamic programming by only 0.5%. 4) Verifying the feasibility of engineering applications: In the hardware-in-the-loop (HIL) platform experiment, the control signal output of the present invention is highly consistent with the simulation results, and the deviation of the main control quantities is less than 0.9%, proving that the method has good real-time performance and reliability, and can be directly applied to the energy management system of actual hybrid electric vehicles. Attached Figure Description

[0034] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a specific embodiment of the present invention; Figures 2(a), (b), and (c) are schematic diagrams of different training conditions after cleaning in specific embodiments of the present invention; Figure 3 This is a schematic diagram comparing a specific embodiment of the present invention with other identification methods; Figures 4(a), (b), and (c) are schematic diagrams of coarse grid scanning results in embodiments of the present invention; Figures 5(a), (b), and (c) are schematic diagrams of the coarse and fine Pareto front screening results in the embodiments of the present invention; Figure 6 This is a schematic diagram of the reward function of the energy management model under different operating conditions in an embodiment of the present invention; Figure 7 and Figure 8 This is a schematic diagram of the SOC and fuel consumption variation curves under WLTC conditions. Figure 9 This is a diagram illustrating the convergence speed of the reward curves. Figure 10-12 This is a schematic diagram showing the comparison results between HIL and simulation. Figure 13 This is a schematic diagram of the HIL platform according to a specific embodiment of the present invention; Figure 14 This is a schematic diagram illustrating an embodiment of the present invention that counters dual deep Q-networks. Detailed Implementation

[0035] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] This invention is based on a Hardware-in-the-Loop (HIL) platform, including: a host computer, a PXI chassis, processor boards, data acquisition boards, a CAN card, a low-voltage power supply, etc. HIL provides resources corresponding to the hardware I / O signals of the controller under test (DUT) and vehicle models and motor models corresponding to the DUT's control strategies. The test system mainly consists of three parts: the HIL hardware simulation platform, the host computer software platform, and the real-time simulation model. The overall system architecture of the HIL platform, including the controller, is as follows: Figure 13 As shown.

[0037] In the constructed HIL test platform, the various hardware devices and software platforms used together constitute a complete real-time simulation test system. The core computing and simulation unit in the system is the NI PXIe-63985 real-time processor, which integrates an Intel i5 processor, runs the Phar-Lap real-time operating system, supports the deployment of the VeriStand engine and simulation models, and is the central hub responsible for real-time model operation and data processing in the entire system. It is installed in a PXIe-2315G2 chassis, which provides multiple slots for installing various PXI standard cards, enabling high-speed, high-precision data bus connections and synchronous operation.

[0038] Within the real-time system, the PXI-8510 / 6 is an interface card for bus communication, supporting commonly used automotive communication protocols such as CAN, CANFD, and LIN to complete message exchange with the ECU under test. These interface cards are physically connected to the controller under test via transceiver cables such as TRC-8542 and TRC-8546 to achieve reliable electrical-level communication.

[0039] To meet the diverse signal input and output requirements of the system, the HIL platform is also equipped with the HRC20-1652 (S1 controller) analog and digital I / O control board. It supports multiple AI, AO, DI, DO signals and PWM channels, simulating the voltage, current, duty cycle, and other input signals required by the controller, and reading the controller's output status signals, thereby completing comprehensive testing and simulation of the device under test (DUT). Meanwhile, the HHT2005 resistive load simulation card is used to simulate load changes in various sensors or motors, such as temperature, voltage, and resistance variables. This type of load card automatically adjusts the resistance value according to requirements, thus providing a realistic external working environment.

[0040] To ensure signal quality and connection stability, the system is equipped with a dedicated signal conditioning box, which performs level conversion, filtering, and isolation on I / O signals to ensure the accuracy and anti-interference capability of analog signals. In addition, the system uses a 12V / 5V switching power supply as a low-voltage power supply module to provide stable power to various controllers, boards, and external devices, ensuring long-term reliable system operation.

[0041] At the software level, NI MAX, NI VeriStand, and NI TestStand are installed on the host computer. MAX is mainly responsible for hardware device identification and driver management, and is the most basic configuration tool. VeriStand is the core software platform of the real-time simulation system, used for deploying models, setting channel mappings, performing signal monitoring, fault injection, and real-time operation control. TestStand is mainly used for automating the test process, editing test sequences, setting conditional logic, automatically recording data, and generating standardized test reports, supporting Word and HTML formats to meet different output requirements.

[0042] The combination of these hardware and software components forms a complete HIL platform system with high reusability, strong scalability, and high testing accuracy, enabling the virtual environment operation and functional verification of complex controllers.

[0043] The hybrid control unit (HCU) used in the hardware-in-the-loop simulation platform serves as a signal generator, transmitting and receiving necessary CAN signals, switching signals, duty cycle signals, and running the energy management method designed in this invention.

[0044] like Figure 1As shown, this invention provides a hybrid electric vehicle energy management method considering operating condition identification. Based on a long short-term memory (LSTM) attention mechanism, an operating condition identification network performs real-time classification of vehicle driving conditions. Using this classification as a constraint, an adaptive adversarial dual deep Q-network (AD-D3QN) energy management algorithm is designed. The algorithm combines dynamic programming (DP) expert demonstration experience with an adaptive adjustment mechanism for reward weights to achieve optimal allocation of torque between the engine and motor. The method includes the following steps: Step S1: Construct a working condition recognition model based on the Long Short-Term Memory (LSTM) attention mechanism; S11, Cleaning under working conditions, as shown in Figures 2(a), (b), and (c); The working condition data is collected and cleaned: first, the working condition is divided into multiple segments, then representative segments are selected for further segmentation, and finally, the segmented segments are labeled with the working condition type to solve the problem of typical working condition sample data and obtain better training results. The speed segment of the driving condition is used as the feature input of the condition recognition model. The specific steps are as follows: 1) Discretization: The standard driving cycle is divided into discrete periods. Discretize in 5s; 2) Slice extraction: at a fixed length 80-second time window, time interval Extract segments from the discretized velocity sequence in 5 seconds. ,get , Multiple samples; 3) Sample balancing: Randomly select an equal number of samples from different working condition categories to avoid too many samples from a single working condition leading to training failure; Long Short-Term Memory (LSTM) uses forgetting gates, input gates, and output gates to enhance or remove information. Its gating mechanism is related to the input sequence. The relationship is represented as: (1); (2); (3); in: It is the sigmoid activation function; ,、 and These are the activation values ​​for the forget gate, input gate, and output gate, respectively, with a value range of [0,1]. Forgotten Gate Control how much of the cell state from the previous time step is retained ; Input gate Determine how many candidate states Add the current cell state; Output gate ; The update formulas for cell state and hidden state are as follows: (4); (5); (6); in: It is the hyperbolic tangent function; and , , , Here are the weight matrix and bias vector to be trained; S12, Model Training; The working condition recognition models based on LSTM and attention mechanisms include: 1) Input layer: Receives the processed velocity sequence data, with a length of 80s; 2) The first LSTM layer and the Dropout(0.1) layer: return the complete temporal output to extract rich temporal features; during training, neurons are randomly masked with a 10% probability to prevent overfitting; 3) Self-attention layer: To make the structure lightweight and easy to deploy, the structure only uses one head and a Transformer layer with a 64-dimensional hidden size to process sequences. : ; ; ; in, These are learnable parameters; 4) The second LSTM layer and Dropout(0.1) layer: Based on the output of the first layer, further refinement is performed, returning only the hidden states at the end of the sequence; Dropout is also used to prevent overfitting. 5) Fully connected layer: Maps the features extracted by LSTM onto three neurons, which are then activated by Softmax to output the predicted probabilities for three operating conditions: urban, suburban, and highway. (10); The loss function is multivariate cross-entropy: (11); in: The total number of samples; For the first The sample at the th Real labels for similar working conditions (urban / suburban / highway), and For the model to the first The sample at the th The predicted probability of each class. The total number of iterations was 300, with the training set accounting for 72%, the validation set for 8%, and the test set for 20%. The distribution of work condition classes was kept consistent during the partitioning process.

[0045] Meanwhile, this invention establishes working condition recognition models based on classic recognition methods such as K-means clustering and support vector machine supervised learning. The sample set used is consistent with that used in the LSTM model mentioned above. The comparison results are as follows: Figure 3 As shown in the figure, operating condition types 0, 1, and 2 represent urban, suburban, and highway operating conditions, respectively, and the operating condition marked by the red line is the baseline operating condition type.

[0046] In order to distinguish it from the working conditions used in the previous training, this test comparison selected a working condition dataset that is completely different from the training set and combined them. The composite working conditions are shown as working condition 1 and working condition 2.

[0047] The overall accuracy of working condition identification is equal to the percentage of correctly predicted samples out of the total number of samples. Table 1 lists the identification accuracy of each method, with the LSTM working condition identification model achieving the highest accuracy of 95.12%.

[0048] Table 1: .

[0049] Step S2: Build an expert demonstration knowledge base based on DP; S21. Design of an expert experience base based on dynamic programming (DP); State vector There are three: vehicle speed State of charge and required torque ,Right now .

[0050] Action space It is constructed from a two-dimensional discrete combination of engine and motor torques, that is: Engine torque exist The values ​​between Nm are discrete into 22 values, representing the motor torque. exist The values ​​of Nm are discrete, with 37 possible values. For the energy management system of a hybrid electric vehicle, dynamic programming (DP) is used to generate the globally optimal trajectory under complex operating conditions. We construct an expert experience base for optimal action trajectories based on dynamic programming (DP) to provide high-quality initial data for deep Q-networks, accelerating their convergence to the optimal policy and improving training stability.

[0051] The optimal motion trajectory covers the entire driving cycle and is associated with a discretized grid of the state space. Correspondingly.

[0052] The reward function is: ; in: and These represent the factors influencing the reward function, namely, changes in fuel consumption rate and State of Charge (SOC). The large numerical value guides the DQN agent to prioritize reducing engine fuel consumption. Big data encourages intelligent agents to more closely align with the target SOC trajectory. The target SOC is represented by 0.6.

[0053] To simplify the analysis, the energy management process of the HEV is modeled as a discrete control problem, and its state transition equation is expressed as: ; in: At the current sampling time, Let be the state transition function of the hybrid system. and These represent the state variable and control variable at the current moment, respectively.

[0054] The control quantities at all times constitute a policy sequence, which includes the control quantities at each moment of the car's movement. Output control quantity : ; The control objective of EMS is to output a sequence with minimum equivalent fuel consumption while ensuring that the state vector satisfies the constraints, described as follows: ; ; in: The objective function for EMS optimization is... This represents the sum of the equivalent fuel consumption rates of N steps. This is a penalty function that describes the system constraints.

[0055] System constraints are described in this invention as follows: ; in, and These are the upper and lower bounds of the battery's State of Charge (SOC). It is the engine's maximum torque. and These are the minimum and maximum torques of the motor, respectively.

[0056] Integrate the reward function into the DP solution process and record the four data sets in the DP policy. , , To improve generalization ability, the data set is loaded into the DQN experience replay area. The specific steps are as follows: 1) Buffer initialization: Allocate a fixed-size buffer, using a priority queue or random access structure; 2) Data loading: Load all [ , , Quadruples are filled into the buffer; 3) Priority setting: Use priority experience replay to allocate initial priorities based on the reward magnitude; 4) Distribution balance: Adjust the sampling weights to ensure that the buffer covers all states of the driving cycle.

[0057] After integrating the expert buffer into the DQN training pipeline, pre-training is performed first. For the initial 100 epochs, only mini-batch data is sampled from the expert buffer to train the DQN. Subsequently, mixed sampling is performed, reducing the proportion of expert data and supplementing the remaining portion with data collected online for further optimization. Simultaneously, the network is updated using the Q-learning loss function to update the Q-network.

[0058] Step S3: Operating condition identification and expert demonstration of the D3QN energy management algorithm; Step S31: D3QN algorithm for condition identification and expert demonstration; 1) Dual-depth Q-network; When updating a double deep Q network, the action corresponding to the maximum objective value is still selected from the evaluation network. The selected value is then used to calculate the target value through the target network. To obtain the corresponding Q value.

[0059] Due to the target network This method offers higher stability by transforming overestimation at each step into overestimation across multiple steps (every other step). This addresses the offset that occurs in the first step, thus mitigating the overestimation problem in the DQN algorithm. In the original DQN, the target value is calculated directly from the target network. Select the maximum value corresponding to the next state. Value. In dual-depth Q-networks, the first step is to evaluate the network... Find the action that maximizes the Q value: (13); Then, in the target network In this context, this action is used to calculate the target Q value: (14); 2) Adversarial deep Q-networks; Adversarial deep Q-networks divide the Q-network into two parts, allowing the agent to more effectively handle states that are weakly related to actions: the first part is only related to the state and is called the state-value function, denoted as: The second part relates to both state and action, and is called the dominance function, denoted as: .

[0060] The final action-value function is re-expressed as (15); in: For public network weights; Dedicated weights for value branches; Dedicated weights for the dominant branches.

[0061] To improve the identifiability of the output, the advantage function is centered, which eliminates the average deviation between actions. Therefore, the final form of the action-value function is: (16); State value function It will be updated in each update phase, thus affecting the Q value of all actions.

[0062] 2) Countering dual deep Q-networks; By combining the features of the aforementioned dueling double deep Q-network and adversarial deep Q-network, we obtain the basic framework of the adversarial double deep Q-network (D3QN). Based on the characteristics of the double deep Q-network, its objective value is defined as: (17); In the formula and The adversarial deep Q-network is used to calculate based on the state value function and the advantage function; The loss function for adversarial dual deep Q-networks is expressed as: (18); Next, the parameters are updated using gradient descent. To obtain the optimal action, use : (19); in, This represents the probability of random exploration.

[0063] Adversarial dual deep Q-networks use an adversarial architecture for both the evaluation and target networks, while using a dual deep Q-network architecture in the target value calculation. This eliminates overestimation and better distinguishes similar actions.

[0064] Regarding network parameters, the number of neurons in each hidden layer was set to 64, and the learning rate was 0.0003. During model training, the root mean square error loss function was used, and the Adam algorithm was used for iterative weight updates. The discount factor γ was set to 0.8, the initial exploration rate was set to 1, and the minimum exploration rate was set to 0.1.

[0065] like Figure 14 As shown, in general, Dueling DQN is responsible for splitting the Q-network into two branches: a state-value function and a dominance function. Double DQN is a dual-network strategy where the evaluation network selects actions and the target network evaluates the value of those actions. The idea behind D3QN is to use the Dueling architecture for both the evaluation and target networks, while using Double DQN for target value calculation. This eliminates overestimation and better distinguishes between similar actions.

[0066] Step S32: Adjust the weights of the reward function; LSTM is used to identify different operating conditions. Since different operating conditions have different requirements for the reward function, the parameters of the reward function are selected accordingly. This is to find the appropriate parameters in the EMS reward function. (Fuel consumption weighting) and To find the optimal weight parameters for (SOC fluctuation weights), this invention proposes a two-stage mesh-based adaptive search method. For each different working condition, the evaluation function for each weight is as follows: (20); - (twenty one); in, Represents fuel consumption. Represents the SOC offset rate. Take 0.6, parameter range: .

[0067] Will The system divides the data into segments by a specific step size, trains and simulates all weight combinations under different operating conditions, and obtains the fuel consumption and SOC offset values ​​of specific weight combinations under specific operating conditions. This allows the system to determine which combination has the best energy-saving effect under that specific operating condition. The process includes the following steps: Step S321: Grid generation; direction: It is always 1; Direction: Divided into 36 segments, step size 28.30.

[0068] Step S322: Simulation evaluation; For each grid point ( , The Simulink model was trained and simulated in parallel under three typical loops: urban area, suburbs, and highway, to calculate the merging target. Recorded as a set of tuples ; Each working condition has 36 weight combinations, and there are a total of 108 weights across the three working conditions, requiring a total of 108 offline training sessions.

[0069] Step S323: Pareto frontier screening; For a specific operating condition, three points of interest are selected from the rough set uniformity of the tuples mentioned above. These three points represent an optimal fuel consumption, a stable state of charge (SOC), and a middle compromise point (as shown in Figures 4(a), (b), and (c), where any point near the lower left corner of the figure is selected). These three points constitute candidate points. Therefore, there are nine sets of candidate points for the three working conditions.

[0070] For each group of candidate points In its neighborhood Perform a small-area grid scan inside: (twenty two); Pick 14. Each candidate point generates 3 re-simulation points, requiring 18 more simulations. Finally, all local simulation results are summarized, and Pareto front screening is performed again for each point to obtain a more refined solution set. The final results of each group of candidate points are shown in Figures 5(a), (b), and (c). Orange dots represent the 3 points selected from the coarse tuple set, and yellow dots are the re-simulation points.

[0071] As shown in Table 2, which compares the equivalent fuel consumption of different strategies, after the above evaluation, the light green dots represent the points where local fine-tuning was performed. and The weighted combination is most suitable for this specific operating condition, resulting in the smallest SOC offset and fuel consumption.

[0072] Table 2: ; Based on the simulation results above, the weights for each operating condition are as follows: Urban area: = 1, = 307; Suburbs: = 1, =250.7; High-speed: = 1, = 165.5. Based on the above three weight values, training was conducted under urban, suburban, and highway operating conditions to obtain the corresponding energy management models. The reward function during the training process is as follows: Figure 6 As shown, these three weights are well adapted to typical urban, suburban, and highway operating conditions.

[0073] Figure 7 and Figure 8 The SOC and fuel consumption curves under WLTC conditions are presented to show the comparison results of AD-D3QN with DQN, DDPG, and DP under the aforementioned unknown standard conditions.

[0074] As shown in Table 3, which compares the equivalent fuel consumption of different strategies, the AD-D3QN of this invention has an equivalent fuel consumption of only 4.23L per 100km, saving 11.07% and 16.1% of the equivalent fuel compared to DQN and DDPG, respectively, and differing from the optimal result of DP by only 0.5%. Because it combines LSTM condition recognition, DP expert demonstration, and an adaptive reward weight network, the AD-D3QN of this invention simultaneously achieves both the generalization ability of online real-time control and near-global optimal economy.

[0075] Table 3: .

[0076] Besides equivalent fuel consumption, the training time of the learning-based EMS is also an important indicator for evaluating the effectiveness of this energy management strategy. For example... Figure 9 This study compares the convergence speeds of DQN (without incorporating expert prior knowledge) and AD-D3QN under the same urban conditions. AD-D3QN converges in the 380th round, while DQN converges in the 450th round, representing a significant improvement in convergence time of 15.6%.

[0077] The comparison results between HIL and simulation are as follows: Figure 10-12 As shown, from Figure 10It was found that, except for a very few moments (during high-speed and acceleration phases) when the speed output in the HIL environment differed slightly from that in the simulation environment due to overall power boundary limitations, the two curves closely matched most of the time, indicating that the EMS strategy exhibited good speed following performance in both the HIL and simulation environments. Furthermore, from... Figures 11-12 It was found that the curves of battery state of charge and fuel consumption in the simulation and hardware-in-the-loop experiments were closely matched, with only a small deviation in the final results.

[0078] Table 4 shows the RMSE analysis of each signal. Under the real-time simulation system, there is a slight delay in communication between the controller and the real-time simulator, and some fluctuations in speed and output torque, but the overall error is within 0.9%. This HIL experiment shows that, with a simulation step size of 0.1s, the AD-D3QN algorithm of this invention has good real-time performance and effectiveness, providing reliable strategy support and deployment feasibility for energy management tasks in real vehicle scenarios.

[0079] Table 4 HIL and Simulation Error Analysis: .

[0080] This invention proposes a hybrid electric vehicle energy management method that considers driving condition identification. By using an LSTM-based driving condition identification model, it can distinguish different driving scenarios such as urban, suburban and highway driving in real time. By combining adversarial dual deep Q network (D3QN) and dynamic programming expert demonstration experience, the convergence speed and generalization performance of the energy management strategy are significantly improved.

[0081] In simulation verification, this energy management method achieved near-dynamic programming global optimal fuel economy under typical standard operating conditions, with equivalent fuel consumption differing from DP by only 0.5%, significantly outperforming existing DQN and DDPG methods. Simultaneously, the battery state of charge was stably maintained within a reasonable range, ensuring the vehicle's power performance and range.

[0082] In hardware-in-the-loop experiments, the energy management method demonstrated excellent real-time performance and reliability. The simulation results were highly consistent with the actual controller results, with errors controlled within 0.9%, verifying the feasibility and promotion value of the method in engineering applications.

[0083] In summary, this invention utilizes an LSTM attention mechanism-based driving condition recognition network for real-time identification of driving conditions, classifying them into three categories: urban, suburban, and highway. Based on the condition recognition results, different energy management models are dynamically selected, achieving adaptive switching to ensure optimal performance even in complex and changing driving scenarios. This invention combines adversarial structures with dual DQN to mitigate bias in action value estimation and improve the stability of reinforcement learning strategies. It introduces expert demonstration experience generated by dynamic programming, constructs a hybrid experience pool, and employs a phased hybrid sampling method to guide the agent in learning near-optimal control strategies. Addressing the varying demands for fuel economy and battery SOC stability under different driving conditions, this invention proposes an adaptive weight optimization method based on grid search and Pareto front screening. This dynamically adjusts the weight parameters of fuel consumption and SOC fluctuation in the reward function, ensuring a balance between high efficiency, energy saving, and battery life under different driving conditions.

[0084] The present invention also provides a computer system, including a memory, a processor, and a computer program stored in the memory, wherein the processor is used to execute the hybrid vehicle energy management method in the above embodiments.

[0085] The present invention also provides a computer-readable storage medium having computer-readable program instructions stored thereon, the computer-readable program instructions being used to execute the hybrid vehicle energy management method in the above embodiments.

[0086] The computer-readable storage medium provided by this invention is, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium is any tangible medium that contains or stores a program that is used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium is transmitted by any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0087] The present invention also provides a computer program product, including a computer program that is used by a processor to execute the hybrid vehicle energy management method in the above embodiments.

[0088] This invention not only improves the economy and stability of energy management in hybrid electric vehicles, but also takes into account generalizability and engineering feasibility, and has broad prospects for promotion and application.

[0089] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid electric vehicle energy management method considering operating condition identification, characterized in that: Based on the long short-term memory attention mechanism, a working condition recognition network is used to classify vehicle driving conditions in real time. Based on this, an adaptive adversarial dual deep Q network AD-D3QN energy management algorithm is designed. The algorithm combines dynamic programming expert demonstration experience and reward weight adaptive adjustment mechanism to achieve optimal allocation of engine and motor torque.

2. The hybrid electric vehicle energy management method considering operating condition identification according to claim 1, characterized in that: The operating condition recognition network based on the long short-term memory attention mechanism classifies vehicle driving conditions in real time, including the following steps: Step S1: Construct a work condition recognition model based on the long short-term memory attention mechanism; S11, Working condition cleaning; The working condition data is collected and cleaned: first, the working condition is divided into multiple segments, then representative segments are selected for further segmentation, and finally, the segmented segments are labeled with the working condition type. The speed segment of the driving condition is used as the feature input of the condition recognition model. The specific steps are as follows: 1) Discretization: The standard driving cycle is divided into discrete periods. Discretize in 5s; 2) Slice extraction: at a fixed length 80-second time window, time interval Extract segments from the discretized velocity sequence in 5 seconds. ,get , Multiple samples; 3) Sample balancing: Randomly select equal numbers of samples from different operating condition categories; Long short-term memory (LSTM) uses forgetting gates, input gates, and output gates to enhance or remove information; its gating mechanism is related to the input sequence. The relationship is represented as: (1)； (2)； (3)； in: It is the sigmoid activation function; ,、 and These are the activation values ​​for the forget gate, input gate, and output gate, respectively, with a value range of [0,1]. Forgotten Gate Control how much of the cell state from the previous time step is retained ; Input gate Determine how many candidate states Add the current cell state; Output gate ; The update formulas for cell state and hidden state are as follows: （4）； （5）； （6）； in: It is the hyperbolic tangent function; and , , , Here are the weight matrix and bias vector to be trained; S13, Model Training; Work condition recognition models based on long short-term memory attention mechanisms include: 1) Input layer: Receives the processed velocity sequence data, with a length of 80s; 2) The first LSTM layer and the Dropout(0.1) layer: return the complete time series output to extract rich temporal features; 3) Self-attention layer: Uses a Transformer layer with a 1-head and 64-dimensional hidden size to process sequences. : ； ； ； in, These are learnable parameters; 4) The second LSTM layer and Dropout(0.1) layer: Based on the output of the first layer, further refinement is performed, and only the hidden state at the end of the sequence is returned; 5) Fully connected layer: Maps the features extracted by LSTM onto three neurons, which are then activated by Softmax to output the predicted probabilities for three operating conditions: urban, suburban, and highway. (10)； The loss function is multivariate cross-entropy: (11)； in: The total number of samples; For the first The sample at the th The true label of the similar working conditions, and For the model to the first The sample at the th The predicted probability of a class.

3. The hybrid electric vehicle energy management method considering operating condition identification according to claim 1, characterized in that: The expert demonstration experience in dynamic programming includes the following specific steps: Step S2: Build an expert demonstration knowledge base based on dynamic programming; S21. Design of an expert experience base based on dynamic programming (DP); Dynamic programming generates globally optimal motion trajectories under complex working conditions. Construct an expert experience base for optimal action trajectories based on dynamic programming (DP); The optimal motion trajectory covers the entire driving cycle and is associated with a discretized grid of the state space. Correspondingly; The reward function is: ; in: and These represent the factors influencing the reward function, namely, changes in fuel consumption rate and State of Charge (SOC). Represents the target SOC; The energy management process of an HEV is modeled as a discrete control problem, and its state transition equation is expressed as: ； in: At the current sampling time, Let be the state transition function of the hybrid system. and These represent the state variable and control variable at the current moment, respectively. The control quantities at all times constitute a policy sequence, which includes the control quantities at each moment of the car's movement. Output control quantity : ； The control objective of EMS is to output a sequence with minimum equivalent fuel consumption while ensuring that the state vector satisfies the constraints, described as follows: ； ； in: The objective function for EMS optimization is... This represents the sum of the equivalent fuel consumption rates of N steps. A penalty function describing system constraints; The system constraints are described as follows: ； in: and These are the upper and lower bounds of the battery's State of Charge (SOC). It is the engine's maximum torque. and These are the minimum and maximum torques of the motor, respectively. The reward function is incorporated into the dynamic programming solution process, and the four data sets are recorded. , , To load the data set into the DQN experience replay area, the specific steps are as follows: 1) Buffer initialization: Allocate a fixed-size buffer, using a priority queue or random access structure; 2) Data loading: Load all [ , , Quadruples are filled into the buffer; 3) Priority setting: Use priority experience replay to allocate initial priorities based on the reward magnitude; 4) Distribution balance: Adjust the sampling weights to ensure that the buffer covers various states of the driving cycle; After integrating the expert buffer into the DQN training process, pre-training is performed first. In the initial few rounds, only small batches of data are sampled from the expert buffer to train the DQN. Subsequently, mixed sampling is performed, reducing the proportion of expert data and supplementing the remaining part with data collected online to further optimize the network. At the same time, the network is updated using the Q-learning loss function.

4. The hybrid electric vehicle energy management method considering operating condition identification according to claim 1, characterized in that: The adaptive adversarial dual deep Q network (AD-D3QN) energy management algorithm includes the following steps: Step S31, AD-D3QN algorithm; 1) Dual-depth Q-network; In evaluating the network Find the action that maximizes the Q value: (13)； In the target network In this context, this action is used to calculate the target Q value: (14)； 2) Adversarial deep Q-networks; The adversarial deep Q-network is divided into two parts. The first part is only related to the state and is called the state-value function, denoted as: The second part relates to both state and action, and is called the dominance function, denoted as: ; The action value function is expressed as: (15)； in: For public network weights; Dedicated weights for value branches; Dedicated weights for dominant branches; After centering the advantage function, the final action-value function is: (16)； State value function It is updated in each update phase, thereby affecting the Q value of all actions; 3) Countering dual deep Q-networks; By combining the features of the aforementioned dual deep Q-network and adversarial deep Q-network, we obtain the basic framework of the adversarial dual deep Q-network, whose objective value is defined as: (17)； In the formula and The adversarial deep Q-network is used to calculate based on the state value function and the advantage function; The loss function for adversarial dual deep Q-networks is expressed as: (18)； Update parameters using gradient descent. To obtain the optimal action, use : (19)； in, The probability of random exploration; Step S32: Adjust the weights of the reward function; A two-stage mesh-based adaptive search method is adopted: For each different working condition, the evaluation function for each weight is as follows: (20)； - (21)； in, Represents fuel consumption. Represents the SOC offset rate. Take 0.6, parameter range: ; Will The system divides the data into segments by a specific step size, trains and simulates all weight combinations under different operating conditions, and obtains the fuel consumption and SOC offset values ​​of specific weight combinations under specific operating conditions. This allows the system to determine which combination has the best energy-saving effect under that specific operating condition. The process includes the following steps: Step S321: Grid generation; direction: It is always 1; Direction: Divided into 36 segments, step size 28.30; Step S322: Simulation evaluation; For each grid point ( , Using Simulink models, parallel training and simulations were performed under different operating conditions to calculate the merging target. Recorded as a set of tuples ; Step S323: Pareto frontier screening; For a specific operating condition, three points of interest are selected from the uniformity of the above tuples, representing the optimal fuel consumption, the most stable SOC, and a middle compromise point, respectively. These three points constitute candidate points. gather; For each group of candidate points Set, in its neighborhood Perform a small-area grid scan inside: (22)； Pick 14. For each candidate point, generate 3 re-simulation points, and then train the simulation 18 times. Finally, summarize all local simulation results and re-select the Pareto front for each point to obtain a more refined solution set.

5. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the energy management method of claim 1.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the energy management method of claim 1.

7. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the energy management method of claim 1.

Images