Series hybrid vehicle multi-target energy optimization algorithm based on double-DQN algorithm

Abstract

The invention discloses a multi-target energy optimization algorithm for a series hybrid power vehicle based on a double depth Q network (DDQN), and belongs to the technical field of hybrid power vehicle energy management. According to the algorithm, the fuel economy is improved and the stability of the state of charge (SOC) of the power battery is kept while the dynamic property requirement of the whole vehicle is met. The method comprises the following steps: constructing a joint simulation model covering an engine, a generator, a power battery and a driving motor according to a coupling relationship between a whole vehicle power demand and multi-source power; a vehicle speed, a battery SOC and a system power demand are defined as a state space, engine generation power is used as an action variable, and an energy management controller based on a double-DQN architecture is designed. By separating action selection and value evaluation processes, the problem of Q value over-estimation in a traditional DQN algorithm is effectively suppressed. Meanwhile, a reward function fusing equivalent fuel consumption and SOC soft constraint is constructed, and tradeoff optimization between economy and a battery management target is achieved. A whole vehicle simulation platform is built in an MATLAB / Simulink environment, and training and verification are carried out in combination with typical cyclic working conditions. Results show that the algorithm can significantly improve fuel economy and SOC control performance under variable working conditions, and has good convergence characteristics and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of vehicle power control and energy management technology, specifically to a multi-objective energy optimization algorithm for series hybrid vehicles based on the dual DQN algorithm, used to achieve coordinated operation and optimized power allocation of the engine, motor and power battery. Background Technology

[0002] As the automotive industry continues to develop towards intelligence, electrification, and high efficiency, hybrid vehicles have demonstrated significant advantages in balancing power performance and fuel economy. With the widespread application of onboard electronic devices, driver assistance systems, and intelligent connectivity functions, higher demands are being placed on the stability of the power supply and the precision of energy management in vehicles. How to achieve comprehensive optimization of power performance, fuel economy, and battery health under varying traffic environments and complex operating conditions has become an important research direction in the field of hybrid power system control.

[0003] Currently, rule-based energy management strategies are widely used in mass-produced vehicles due to their simple structure, intuitive logic, and ease of implementation. However, these strategies rely on expert experience and extensive calibration tests, making them difficult to adapt to the diverse operating conditions in actual driving, thus limiting the vehicle's fuel economy and battery SOC maintenance capability. Optimal control methods (such as dynamic programming and model predictive control) can achieve global optimal power allocation under ideal conditions, but they are highly dependent on the accuracy of the system model, prediction of future operating conditions, and computational resources, making it difficult to meet the real-time computing requirements of the vehicle control unit. In recent years, reinforcement learning-based energy management methods have gradually become a research hotspot. These methods, through interactive learning with the environment and data-driven modeling, can achieve adaptive adjustment of control strategies, possessing good generalization ability and environmental adaptability. However, existing reinforcement learning algorithms still suffer from low training efficiency, unstable convergence processes, and a lack of physical constraints, requiring further optimization in algorithm structure and feature modeling.

[0004] To address the aforementioned problems, this invention proposes a multi-objective energy optimization algorithm for hybrid vehicles based on a dual-depth Q-network. This method combines vehicle energy flow modeling to construct a representation of the vehicle's operating state and action space. With fuel economy optimization and battery SOC range stability as dual objectives, it implements a reinforcement learning-driven adaptive energy management strategy. A vehicle co-simulation model is established based on the MATLAB / Simulink platform to verify the algorithm's control performance and generalization ability under typical urban and high-speed operating conditions. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN, which aims to solve the problems of engine-generator response lag, vehicle power performance requirements and multi-source power coupling, that is, to effectively improve fuel economy while ensuring power performance.

[0006] To achieve the above objectives, this invention provides a multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN, comprising the following steps:

[0007] Step 1: Construct a series hybrid vehicle powertrain model, including an engine-generator set, a power battery, an electric motor drive, and a module for calculating the vehicle's required power. Each module is interconnected through energy flow, and the vehicle's required power module calculates instantaneous power in real time based on vehicle speed, acceleration, road resistance, and special operational loads.

[0008] Step 2: A multi-dimensional feature fusion and adaptive learning mechanism is proposed to enhance the modeling accuracy and robustness of reinforcement learning under complex working conditions. By fusing multi-source features such as vehicle speed, battery SOC, demand power, engine speed gradient, road slope, and load intensity, a comprehensive state space that can fully represent the operating state is constructed. A hierarchical decision architecture is adopted, with the upper layer generating energy allocation coefficients based on task type and power trend, and the lower layer performing local power optimization by dual DQN controllers, realizing the synergy of macro and micro decision-making. At the same time, a dynamic confidence adjustment mechanism based on Q-value fluctuations and a constraint-guided action correction function are introduced to ensure the stability of the policy learning process and meet physical constraints such as power and SOC.

[0009] Step 3: Construct a state space using key variables such as vehicle speed, battery state of charge, and instantaneous power demand of the vehicle. Using engine power generation as the action set, design an energy management controller based on dual DQN. Utilize a separation mechanism between action selection and value evaluation to improve the stability of the learning process. Design a reward function with integrated fuel consumption and battery SOC maintenance as optimization objectives. Train the agent using a combination of experience replay and target network soft update methods to obtain a convergent and stable energy management strategy.

[0010] Step 4: Conduct multi-condition simulation verification in the MATLAB / Simulink environment, and evaluate the performance of the control strategy in terms of fuel economy and SOC stability through typical and compound operating condition tests.

[0011] Optionally, during the execution of step 1, without considering other unforeseen circumstances, the power balance equation for the driving force and resistance experienced by the series hybrid vehicle is as follows:

[0012]

[0013] in: —Engine power, -slope, —Vehicle weight, —Rolling resistance coefficient; —vehicle speed, —Air drag coefficient, —Windward area, — Rotational mass conversion factor —Acceleration.

[0014] To ensure the stability and reliability of the powertrain operation, a multi-dimensional constraint system needs to be constructed:

[0015]

[0016] In the formula, max and min correspond to the maximum and minimum power of the engine-generator unit and the maximum and minimum power of the motor, respectively.

[0017] Optionally, during the execution of step 2, to improve the modeling ability and learning efficiency of reinforcement learning in complex operating environments, this paper proposes a multi-dimensional feature fusion and adaptive learning mechanism to optimize the performance of the dual DQN strategy in energy management of series hybrid vehicles. This module achieves rapid adaptation and stable convergence of the strategy to complex operating conditions through state information enhancement, hierarchical decision-making, and adaptive learning rate adjustment.

[0018] In traditional state variables (vehicle speed) Battery SOC System power requirements Based on this, an engine speed gradient is introduced. Road slope estimate With work load intensity index

[0019] Construct the comprehensive state vector:

[0020]

[0021] Among them, the engine speed gradient is used to capture transient acceleration or deceleration characteristics, the gradient information is used to adjust the power distribution in advance, and the workload index reflects the changes in the intensity of vehicle tasks.

[0022] To improve the state representation capability, a feature-weighted normalization method is used to process the state in each dimension:

[0023]

[0024] in For feature weights, and These represent the state mean and standard deviation, respectively. After normalization, high-order correlation features between local states are extracted through convolutional encoding, thereby enhancing the policy's generalization ability to complex operating conditions.

[0025] To address the multi-scale decision-making problem in energy management, a framework of upper-level decision-making and lower-level control is constructed:

[0026] Upper-level decision-making: Dynamically generate energy allocation coefficients based on job type and power demand trends.

[0027]

[0028] This layer is responsible for global energy allocation planning, ensuring the rationality of energy use under different operating conditions.

[0029] Lower-level control: Locally optimal power control is performed by dual DQN controllers, outputting engine power and battery charging / discharging power, achieving synergy between task-level decision-making and energy-level optimization. This hierarchical structure effectively reduces the search space of direct control by a single-layer DQN, improving the convergence speed of the strategy in complex scenarios.

[0030] To address the fluctuation problem in Q-value estimation during reinforcement learning, a dynamic confidence coefficient is introduced. Based on the variance of the current Q value Adaptive learning rate adjustment Update frequency of the target network :

[0031]

[0032] in This represents the average Q-value of the current batch. This mechanism lowers the learning rate in the early stages of training to prevent unstable actions due to overestimation bias; and increases the learning rate in the later stages of training to promote rapid policy convergence.

[0033] To ensure the physical feasibility of the output action, an action constraint mapping function is introduced:

[0034]

[0035] in, This is the original output action of dual DQN. The feasible action range is calculated based on the current state, including engine power, battery charging and discharging power, and system power balance limits. This mechanism ensures that the strategy always outputs safe and achievable power commands, avoiding dangerous or invalid actions that could interfere with learning.

[0036] Step 2, the fusion and adaptive learning module, serves as a pre-optimization layer for the dual DQN algorithm. It enhances the policy's perception and generalization capabilities across multiple scenarios. Hierarchical decision-making and action constraints ensure the physical feasibility and task adaptability of the policy output. The dynamic confidence adaptive mechanism improves training stability and convergence speed, achieving end-to-end optimization from data features to action execution.

[0037] Optionally, during the execution of step 3, the key state variables of vehicle operation are constructed as multi-dimensional state space vectors, and the discretized engine power generation is used as the action set. The energy management control strategy is self-learned and optimized through a Double Deep Q-Network (DDQN).

[0038] Constructing state space vectors as follows:

[0039]

[0040] in, This represents the instantaneous traction power demand (kW) of the entire vehicle at the current moment. Indicates the battery's state of charge. This is the power output of the engine and drive motor at the previous moment, used to capture the dynamic inertia of the system; The current vehicle speed is (m / s); ΔSOC_(t-1) is the rate of change of SOC, used to determine the trend of battery energy change; This represents the current road slope angle.

[0041] Discretize the engine's power generation into M levels, and define the action set as follows:

[0042]

[0043] in For the first The target power generation capacity of the engine meets the following requirements:

[0044]

[0045] The granularity of the action space can be adjusted according to the system's dynamic characteristics and computational resource allocation. Finer granularity results in higher control precision, but also increases training complexity. To ensure training convergence and feasibility, 5–15 discrete power points are selected to cover the engine's available power range.

[0046] A dual-depth Q-network architecture is used to approximate the action value function. The control architecture consists of two parts: a behavior network and a target network. The parameters of the behavior network are... The target network parameters are The formula for calculating the target Q value is:

[0047]

[0048] in: The target Q value; For instant rewards; Discount factor; These are the Q-value estimation functions for the behavioral network and the target network, respectively.

[0049] The network is trained by minimizing the mean squared error loss function:

[0050]

[0051] in This is an experience replay pool with a capacity of 10. 5 ~10 6 Random small-batch sampling is used to improve data utilization and avoid training oscillations caused by temporal correlation between samples.

[0052] To ensure that the control output meets the conditions for safe operation, the action selection must undergo the following multi-dimensional validity checks:

[0053] Engine power constraints:

[0054]

[0055] Battery SOC constraint: Predict the next SOC; if it falls below the safe lower limit, disable pure electric mode; if it exceeds the upper limit, disable high charging power mode.

[0056]

[0057] Motor power and bus voltage constraints: Ensure that the actual power of the drive motor does not exceed the maximum allowable value, and that the bus voltage is not lower than the safety threshold.

[0058]

[0059] For actions that do not meet the constraints, their Q value is forced to a minimum value, thereby automatically avoiding the action during the decision-making process and ensuring the safety of the strategy throughout the entire operating range.

[0060] The agent selects actions according to a greedy policy, where ε gradually decays during training, transitioning from initial exploration-based behavior to optimal decision-making utilizing the learned policy. Once training is complete, the deployment phase sets ε to near zero, selecting only the action with the highest current Q-value to achieve stable control.

[0061] The design incorporates a composite reward function that integrates fuel consumption and battery SOC maintenance. Instantaneous fuel consumption rate, battery output power, and SOC deviation are used as components of the reward. The agent is trained through experience replay buffer and target network soft update mechanisms to achieve self-learning and optimization of energy management strategies. This enables the controller to maintain convergent and stable energy scheduling capabilities under multiple operating conditions, complex loads, and gradient changes.

[0062] A composite reward function is used to uniformly measure engine fuel consumption, battery output power, and SOC constraints.

[0063]

[0064] in: Engine instantaneous fuel consumption rate (g / s); This represents the battery output power (positive when discharging). The electrical energy equivalence factor converts electrical energy consumption into equivalent fuel quantity, taking into account charging and discharging efficiency and system losses. The lower heating value of fuel oil (kJ / g); These are soft constraint weights for SOC, used to guide SOC to remain at the reference value. Nearby. This reward function can simultaneously optimize fuel consumption and SOC steady-state deviation, and apply a larger negative reward when SOC exceeds a safe threshold to prevent overcharging or over-discharging.

[0065] During training, the reward signal is normalized and pruned to the upper and lower bounds, narrowed to the [−10,0] interval to avoid numerical explosion, and weights are applied accordingly. The relative importance of adjusting economic efficiency and SOC control objectives. Training was conducted using a capacity of 10... 5 ~10 6 The experience replay buffer is updated using random sampling with a batch size of 64 and a learning rate of 10. -3 Adam optimizer, discount factor, and target network soft update coefficient Update the target parameters as follows:

[0066]

[0067] in For online network parameters, These are the target network parameters. A greedy strategy is used during training to maintain a high exploration rate in the early stages. The learning rate is increased exponentially or linearly to 0.05 to balance the exploration and utilization relationship. For convergence monitoring and generalization improvement, periodic policy evaluation (e.g., every 1000 rounds) is performed using validation conditions not involved in training. The average reward, equivalent fuel consumption, and SOC deviation curves are calculated, and an early stop condition is triggered after several rounds without significant improvement to prevent overfitting. To improve policy robustness, various driving cycles, complex loads, and random slope disturbances are introduced for mixed training, and the optimal policy is obtained through multiple independent training sessions. If training is unstable or convergence is slow, the convergence speed is improved by reducing the learning rate, increasing the replay pool capacity, or using a prioritized replay mechanism. If the agent chooses extreme actions that lead to battery overcurrent or engine overload, a safety constraint filter is introduced during the action selection phase to ensure that the energy management strategy meets safety and durability requirements in engineering applications. Through the above design and training strategies, the resulting Dual Deep Q Network (DDQN) controller can achieve convergent, stable, robust, and efficient energy scheduling under various complex operating conditions, balancing fuel economy and battery life.

[0068] Optionally, during step 4, multi-condition simulation verification is conducted in the MATLAB / Simulink co-simulation environment to systematically evaluate the comprehensive performance of the control strategy in terms of fuel economy, SOC stability, and dynamic performance. First, a baseline comparison test is established based on standardized driving cycle cycles (such as NEDC and urban driving cycles) to ensure the rationality of energy distribution under normal driving conditions. Then, a special task cycle is introduced to reproduce extreme scenarios such as long-slope heavy loads, frequent acceleration and deceleration, and low-speed off-road driving (such as FTP75), thereby examining the robustness and adaptive capability of the controller under high loads and non-stationary conditions. To improve the statistical significance of the results, each driving condition is simulated multiple times under different initial SOC, load, and other conditions to form comparable statistical samples. In terms of performance evaluation, equivalent fuel consumption is used as the core economic indicator. This is achieved by converting electrical energy into fuel equivalent using an equivalence factor and combining it with the actual engine fuel consumption. Simultaneously, battery energy maintenance capability is measured by SOC final value error, average deviation, and fluctuation amplitude. Dynamics satisfaction rate is evaluated by vehicle speed tracking RMSE and acceleration response delay. The percentage of engine operation within the optimal BSFC range and the number of start-stop cycles are also statistically analyzed to reflect operating efficiency. Furthermore, SOC over-limit safety trigger events are monitored to ensure the strategy meets operational constraints. To verify the superiority of the proposed strategy, a horizontal comparison is conducted with the standard DQN algorithm. Under the same simulation platform and random seed, the relative improvement percentage of fuel consumption and SOC deviation is calculated, and confidence intervals are obtained through multiple repeated experiments. The final output includes time series curves of key quantities such as vehicle speed, SOC, and engine power, statistical tables of fuel consumption and SOC control effects, and box plots with confidence intervals, comprehensively demonstrating the performance and stability of the strategy under various operating conditions.

[0069] This invention provides a multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN. First, based on the vehicle's dynamic characteristics and multi-source power requirements, a system-level co-simulation platform including the engine, generator, power battery, and drive motor is established to characterize energy flow and power distribution relationships and achieve accurate modeling of key powertrain components. Then, a multi-dimensional feature fusion and adaptive learning mechanism is proposed to enhance the modeling accuracy and robustness of reinforcement learning under complex operating conditions. By fusing multi-source features such as vehicle speed, battery SOC, required power, engine speed gradient, road gradient, and load intensity, a comprehensive state space that fully characterizes the operating state is constructed. A hierarchical decision-making architecture is adopted, with the upper layer generating energy distribution coefficients based on task type and power trends, and the lower layer performing local power optimization by a dual DQN controller, achieving macro- and micro-level decision-making coordination. Simultaneously, a dynamic confidence adjustment mechanism based on Q-value fluctuations and a constraint-guided action correction function are introduced to ensure the stability of the strategy learning process and meet physical constraints such as power and SOC. Furthermore, key operating variables such as vehicle speed, battery state of charge, drive motor output power, and instantaneous power demand of the vehicle are jointly constructed into a multi-dimensional state space vector. Using the discretized result of engine power generation as the action set, an intelligent energy management controller based on dual DQN is designed. The convergence and stability of the strategy training are improved by introducing a separation structure between the action selection network and the target value network. Next, considering the multi-objective characteristics of energy management, a comprehensive reward function is constructed, with minimizing equivalent fuel consumption and maintaining SOC within the target range as optimization objectives. The agent is trained offline using an experience replay mechanism and a target network soft update algorithm, enabling it to obtain a stable and convergent control strategy under various operating conditions. Finally, verification tests are conducted in a MATLAB / Simulink co-simulation environment under multiple scenarios and random initial conditions, covering standard operating conditions, special task cycles, and random disturbance conditions. By analyzing performance indicators such as vehicle speed tracking accuracy, SOC fluctuation amplitude, and equivalent fuel consumption, the overall performance of the strategy in terms of fuel economy and SOC stability is systematically evaluated. Experimental results show that this method can achieve intelligent coordinated scheduling of multi-source power while meeting the overall vehicle power requirements, significantly improving fuel economy and energy utilization efficiency, and has good generalization and engineering application value. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1This is a flowchart illustrating a multi-objective energy optimization algorithm for series hybrid vehicles based on the dual DQN algorithm of the present invention.

[0072] Figure 2 This is the structure of the series hybrid power system of the present invention.

[0073] Figure 3 This invention relates to closed-loop control of engine speed and generator torque.

[0074] Figure 4 This is a network structure diagram of the energy management strategy of the dual DQN algorithm of the present invention.

[0075] Figure 5 This is a simulation result diagram comparing the average reward values ​​of the DQN and dual DQN control algorithms under the same working conditions, according to a specific embodiment of the present invention.

[0076] Figure 6 This is a simulation result diagram comparing the reward values ​​of the DQN and dual DQN control algorithms under the same working conditions, according to a specific embodiment of the present invention.

[0077] Figure 7 This is a simulation result of the speed following curve under NEDC conditions in a specific embodiment of the present invention.

[0078] Figure 8 This is a simulation result of the speed following curve under the FTP75 operating condition of a specific embodiment of the present invention.

[0079] Figure 9 The figure shows the simulation results of the SOC change trajectory under NEDC conditions, where the initial SOC value of the dual DQN control algorithm of this invention is 0.8 according to a specific embodiment.

[0080] Figure 10 The figure shows the power comparison simulation results of the dual DQN control algorithm of this invention with an initial SOC value of 0.8 under NEDC conditions.

[0081] Figure 11 The figure shows the simulation results of the SOC change trajectory under NEDC conditions, where the initial SOC value of the dual DQN control algorithm of this invention is 0.3 according to a specific embodiment.

[0082] Figure 12 The figure shows the power comparison simulation results of the dual DQN control algorithm of this invention with an initial SOC value of 0.3 under NEDC conditions.

[0083] Figure 13 The figure shows the simulation results of the SOC change trajectory of the trained agent under FTP75 conditions with an initial SOC value of 0.3, which is a specific embodiment of the present invention.

[0084] Figure 14 The figure shows the power comparison simulation results of the trained agent under FTP75 conditions with an initial SOC value of 0.3, which is a specific embodiment of the present invention. Detailed Implementation

[0085] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0086] The following explains the abbreviations of the English terms used in this invention:

[0087] DDQN (Double Deep Q-Network): A network with double deep Q-networks.

[0088] PID (Proportional Integral Derivative): Proportional-Integral-Derivative.

[0089] Please see Figure 1 This invention provides a multi-objective energy optimization algorithm for series hybrid vehicles based on Double Deep Q-Network (DDQN), comprising the following steps:

[0090] S1: Based on the vehicle's dynamic characteristics and multi-source power requirements, a system-level joint simulation platform covering the engine, generator, power battery, and drive motor is established to characterize the energy flow path and power distribution relationship, and to achieve accurate modeling of key powertrain components, providing a simulation environment for subsequent control strategy design and verification.

[0091] S2: A multi-dimensional feature fusion and adaptive learning mechanism is proposed to enhance the modeling accuracy and robustness of reinforcement learning under complex operating conditions. By fusing multi-source features such as vehicle speed, battery SOC, power demand, engine speed gradient, road slope, and load intensity, a comprehensive state space that can fully represent the operating state is constructed. A hierarchical decision architecture is adopted, with the upper layer generating energy allocation coefficients based on task type and power trend, and the lower layer performing local power optimization by dual DQN controllers, realizing the coordination of macro and micro decision-making. At the same time, a dynamic confidence adjustment mechanism based on Q-value fluctuations and a constraint-guided action correction function are introduced to ensure the stability of the policy learning process and meet physical constraints such as power and SOC.

[0092] S3: To address the multi-objective optimization characteristics of energy management, a comprehensive reward function is designed, taking the minimization of equivalent fuel consumption and maintaining the State of Charge (SOC) within the target range as optimization objectives, while also considering engine start-stop smoothness. Combining an experience replay mechanism and a target network soft update algorithm, the dual-DQN agent is trained offline to obtain a stable and convergent energy allocation strategy under various operating conditions, achieving intelligent coordinated scheduling of multi-source power.

[0093] S4: Based on the trained DDQN strategy, validation tests were conducted on the MATLAB / Simulink co-simulation platform under multiple scenarios and multiple initial SOC conditions, covering standard operating conditions such as NEDC and FTP-75, as well as special task cycles. By analyzing vehicle speed tracking accuracy, SOC changes and fluctuations, engine and motor power scheduling, engine start-stop frequency, and equivalent fuel consumption, the strategy's fuel economy, SOC stability, and power response performance were systematically evaluated. The results show that dual-DQN outperforms DQN in training convergence, strategy stability, and energy management effectiveness, validating its adaptability, generalization ability, and engineering application value.

[0094] The following provides further explanation in conjunction with the implementation steps:

[0095] S1. To achieve a systematic analysis of the power performance of series hybrid vehicles, this invention establishes a longitudinal dynamics model of the entire vehicle and prioritizes power performance optimization as a design objective. The main performance evaluation indicators include peak speed, maximum gradient capability, and acceleration response time, where acceleration response time typically refers to the time it takes for the vehicle to accelerate from a standstill to 32 km / h.

[0096] The longitudinal dynamics of the vehicle can be expressed as:

[0097]

[0098] in: —Engine power, -slope, —Vehicle weight, —Rolling resistance coefficient; —vehicle speed, —Air drag coefficient, —Windward area, — Rotational mass conversion factor —Acceleration. This model comprehensively considers rolling resistance, air resistance, gradient resistance, and the equivalent inertia of rotating mass, and can calculate the required driving power under different vehicle speeds, gradients, and acceleration conditions.

[0099] The peak speed of a vehicle is mainly determined by air resistance and rolling resistance. Assuming the vehicle travels in steady state on a straight road, and neglecting the effects of acceleration and gradient, the dynamic balance equation can be simplified to:

[0100]

[0101] Solving this equation will yield the vehicle's maximum steady-state speed. Peak speed is an important constraint in vehicle powertrain design, used to determine the upper limit of engine and motor power.

[0102] Maximum gradeability reflects the limit of a vehicle's ability to traverse inclines. It is typically tested under steady-state, uniform speed conditions on standard paved roads, while eliminating inertial forces caused by acceleration or deceleration. The longitudinal dynamic equilibrium equation is:

[0103]

[0104] Given the engine's maximum power and vehicle speed When the maximum gradient can be obtained, the solution can be found. :

[0105]

[0106] This indicator is used to evaluate a vehicle's ability to pass over ramps in special operating scenarios, providing a reliable basis for the design of the vehicle's power system.

[0107] Acceleration performance is typically quantified by the time taken to accelerate from a standstill to full speed or to accelerate again while in motion. When analyzing acceleration dynamics, gradient drag can be considered a secondary factor; the primary focus is on the vehicle's ability to overcome translational inertia. The longitudinal dynamic equations can be simplified to:

[0108]

[0109] Accelerate from a standstill to the target speed At that time, points can be used to obtain acceleration time. :

[0110]

[0111] This method can quantitatively evaluate the acceleration response of vehicles under different power configurations, providing a basis for powertrain matching and energy management strategies.

[0112] To ensure the effectiveness and generalization capability of energy management strategy training, a high-precision hybrid power system unit model is needed, including core components such as the engine, generator, drive motor, and power battery. This co-simulation model should not only accurately describe the efficiency characteristics of each subsystem under steady-state conditions, but also consider dynamic response, power boundaries, and constraints to realistically reproduce the operating behavior of actual vehicles.

[0113] The engine unit is the primary energy source in the hybrid power system. Its mathematical modeling is based on engine universal characteristic data, and a steady-state model is established through three-dimensional surface fitting of speed-torque-fuel consumption rate (BSFC). The instantaneous output power of the engine can be expressed as:

[0114]

[0115] in, and These are the engine output torque and speed, respectively. This represents the engine's efficiency under this operating condition. The fuel consumption rate is obtained through BSFC surface interpolation:

[0116]

[0117] To ensure that the simulation results conform to the inertial characteristics of a real power system, a first-order inertial element is introduced to describe the dynamic response of engine power, making engine power changes smoother and avoiding simulation instability caused by instantaneous jumps. At the same time, upper and lower limits are imposed on engine torque and speed to prevent simulation conditions from exceeding the limits, such as misfire caused by overload or excessively low speed.

[0118] The generator unit is connected to the engine and is used to convert mechanical energy into electrical energy. The torque and speed of the generator are constrained by the rated operating range. The rate of power change is limited by a first-order dynamic equation to reflect the inductive lag of the electromagnetic system and the response delay of the controller, so as to avoid generating unreasonable instantaneous power pulses.

[0119] The drive motor unit is responsible for the primary function of driving the vehicle. The model, based on the motor efficiency characteristic table, calculates the output power in real time at a given torque and speed.

[0120]

[0121] Maximum power and torque limits are imposed on the drive motor model to prevent overload operation or exceeding the controller's allowed operating range. This model allows for accurate assessment of motor energy loss and feedback efficiency, providing a reference for energy management strategies.

[0122] The state of charge (SOC) of the power battery cell is described by an electrochemical equivalent model over time, and the mathematical expression is as follows:

[0123]

[0124] in, This is the battery's rated voltage. The rated capacity of the battery is used. The model considers the nonlinear relationship between battery internal resistance, voltage, and SOC, as well as differences in charge and discharge efficiency. To avoid overcharging and over-discharging, the SOC range is limited to [specific range missing]. Between these parameters, maximum charge and discharge power constraints are applied to ensure battery lifespan and safety.

[0125] The goal of power flow allocation across the vehicle is to meet the power demands of each wheel. This is achieved through optimized scheduling to coordinate the output of power from multiple sources. The objective function for this optimization can be expressed as:

[0126]

[0127] In this process, the power constraints, speed and torque limits, and SOC range requirements of the engine, generator, motor, and battery must be met simultaneously to achieve efficient energy distribution and precise matching of instantaneous power for the entire vehicle.

[0128] The aforementioned co-simulation platform can simultaneously consider component efficiency curves, dynamic delays, and power boundary constraints, and can reproduce the power flow characteristics and energy distribution process of the entire vehicle under various driving conditions (such as standard cycles, special mission conditions, and random disturbance conditions). This platform not only provides multi-dimensional state variables and real-time power flow information, but also calculates equivalent fuel consumption and SOC deviation, thus providing high-precision environmental simulation and reward signals for training intelligent energy management strategies based on dual-DQN.

[0129] S2. To improve the modeling ability and learning efficiency of reinforcement learning in complex operating environments, this invention proposes a multi-dimensional feature fusion and adaptive learning mechanism to optimize the performance of the dual DQN strategy in energy management of series hybrid vehicles. This module not only enhances the ability to represent state information but also ensures the feasibility and robustness of the strategy output through hierarchical decision-making, adaptive learning rate adjustment, and action constraints. The detailed design of the module is as follows:

[0130] 1. Multidimensional feature fusion

[0131] Traditional energy management strategies only use vehicle speed Battery SOC System power requirements Using basic states as input is insufficient to fully reflect the dynamic characteristics of a vehicle under special operating conditions. To enhance state awareness, this module introduces the following extended features:

[0132] Engine speed gradient It reflects transient acceleration and deceleration characteristics and can predict engine load change trends in advance;

[0133] Road slope estimate Used for auxiliary power planning, adjusting the power output of the engine and battery in advance to ensure power performance;

[0134] Workload intensity index : Quantify the operational intensity of hybrid vehicles to reflect the impact of operational tasks on energy management.

[0135] Therefore, a comprehensive state vector is constructed:

[0136]

[0137] To improve the ability to represent states, this module first normalizes and weights the features of each state:

[0138]

[0139] in For feature weights, and These represent the state mean and standard deviation, respectively, determined through training or prior experience. After normalization and weighting, the state vector is input into the convolutional coding layer to extract local and cross-feature correlations, resulting in an enhanced high-dimensional state. This multi-dimensional feature fusion not only preserves the original state information but also incorporates vehicle transient dynamics, road and task environment information, enabling the strategy to possess sensitivity and generalization ability to complex operating conditions.

[0140] 2. Hierarchical decision-making framework

[0141] To address the multi-scale decision-making problem (i.e., the coupling of task-level demand and energy-level power allocation) in the energy management of series hybrid vehicles, this invention designs a two-layer control architecture that organically combines global energy allocation with local power optimization. The layered architecture structure and functions are as follows:

[0142] Upper-level task planning (global energy allocation): The upper-level controller performs global planning of the vehicle's energy usage based on the vehicle's operation type, power demand trends, and road gradient information. Its core objective is to determine the power allocation ratio between the engine and the battery.

[0143]

[0144] upper-level decision output As a reference constraint for the lower-level controller, the dual DQNs can consider both global planning and retain local optimization degrees of freedom when performing local power control.

[0145] In addition, the upper-level task planning also incorporates predictive models:

[0146]

[0147] Predicting the future The power requirements of each step are used to optimize energy distribution, making the strategy forward-looking and improving fuel economy and SOC regulation stability.

[0148] Lower-level local control (dual DQN power regulation): The lower-level controller is implemented by a dual DQN network, and its inputs are the enhanced state vector and energy distribution ratio after being constrained by the upper-level regulation. Output engine power and battery power:

[0149]

[0150] The lower-level control achieves a locally optimal strategy through dual DQN, with the main optimization objectives including:

[0151] Optimal fuel economy: Select the engine and battery power combination based on the principle of minimizing equivalent fuel consumption; Stable SOC: Ensure that the battery SOC fluctuates minimally within the target range; Fast transient response: Meet the transient power requirements such as acceleration, deceleration, and hill climbing.

[0152] The upper and lower layer collaboration mechanism is achieved through the following three aspects:

[0153] Information flow: Upper layer generates global energy allocation coefficient → Lower layer receives state and allocation reference → Dual DQN outputs local power command → Action constraint correction → Actual power execution.

[0154] Time scale: Upper-level decision-making can use a longer sampling period to plan global energy; lower-level decision-making uses a short sampling period to adjust power in real time to ensure transient response capability.

[0155] Feedback loop: The actual power and SOC information after the lower layer is executed are fed back to the upper layer to adjust the subsequent energy distribution, so that the hierarchical system forms a closed-loop optimization.

[0156] 3. Dynamic confidence adaptive mechanism

[0157] In reinforcement learning, Q-value estimation errors can lead to policy instability or overestimation bias. To address this, this invention proposes a dynamic confidence adaptive mechanism, which adjusts the confidence level based on the variance of the Q-value fluctuation. Adaptive learning rate adjustment Update frequency of the target network :

[0158]

[0159] in This represents the average Q-value of the current batch. and These represent the initial learning rate and the target network update interval, respectively. In the early stages of training, Larger A smaller Q-value leads to a lower learning rate and a lower target network update frequency, preventing the policy from making unstable actions due to Q-value noise; in the later stages of training, convergence, Increasing the learning rate accelerates the update of the target network, thus speeding up policy convergence. This mechanism involves lowering the learning rate in the early stages of training to prevent unstable actions due to overestimation bias, and increasing the learning rate in the later stages of training to promote rapid policy convergence.

[0160] 4. Constraint-guided action correction mechanism

[0161] To ensure the physical feasibility of the output action, an action constraint mapping function is introduced:

[0162]

[0163] in: This is the raw output of the dual DQN network; and The range of executable actions is calculated based on the current state, including the engine's maximum power, minimum power, battery charge and discharge limits, and system total power balance constraints. The function guarantees that the output action is within the physically permissible range. This mechanism prevents the policy from outputting invalid or dangerous actions, while also guiding the network to learn safer and more efficient energy management strategies through constraints.

[0164] S3. Construct key state variables of vehicle operation into a multi-dimensional state space vector to fully characterize the vehicle's power demand, energy storage state, and dynamic characteristics. The state space is defined as:

[0165]

[0166] in, The traction power requirement at the current moment can be calculated using the vehicle dynamics equations:

[0167]

[0168] This vector simultaneously contains information such as battery SOC, power output of the engine and motor at the previous moment, vehicle speed, rate of change of SOC, and slope angle, used to capture the dynamic inertia and energy change trends of the system. Based on this state description, the available power range of the engine is discretized into M power levels, constructing a set of actions.

[0169]

[0170] The discrete granularity is determined based on the power range and computational resources, typically selected as 5–15 levels to balance control accuracy and training convergence speed. To achieve multi-objective optimization, a comprehensive reward function is designed:

[0171]

[0172] The first term characterizes equivalent fuel consumption, the second term penalizes deviations from the target SOC range, and the third term suppresses the decrease in smoothness caused by frequent engine start-stop or power jumps. The controller employs a dual-depth Q-network (DDQN) architecture, comprising a behavioral network and a target network. The target Q-value is calculated using the following formula:

[0173]

[0174] The capacity of the experience replay pool is set to 10. 5 ~10 6 Random sampling is used to eliminate temporal correlation and improve training stability. During training, candidate actions undergo multi-dimensional validity checks to ensure the policy output meets constraints such as engine power, battery SOC, motor power, and bus voltage. Actions that do not meet the constraints are assigned a minimum Q-value, automatically avoiding them during decision-making. An ε-greedy strategy balances exploration and utilization, with ε gradually decaying during training, transitioning from random exploration to optimal decision-making based on the value function. The target network uses soft update rules. The behavior network is gradually approximated to ensure smooth training convergence. After training, ε is set to near zero, and only the action with the largest current Q value is selected to achieve stable control. In actual deployment, online constraint shielding and fault-tolerant mechanisms are combined to ensure the feasibility, robustness, and safety of the strategy under all operating conditions.

[0175] S4. The stability and learning effectiveness of the reinforcement learning strategy during training were evaluated using the cumulative reward fluctuation of a single training round as the convergence criterion. The New European Cyclic Circuit (NEDC) test was selected as the standard test condition for training and testing. The specific test curves are shown in Figure 4. Figure 7 As shown in the figure. Under this condition, policy training was performed based on DQN and dual DQN algorithms respectively, and the trend of the average reward in each round of training was recorded. The results are as follows. Figure 5As shown in the training curves, the DQN algorithm exhibits high reward values ​​in the early stages (approximately the first 50 rounds), but these values ​​fluctuate wildly with limited average reward growth, demonstrating convergence instability. This is primarily due to DQN's use of the maximum Q-value for target estimation, which easily leads to systematic overestimation bias, causing some actions to be misjudged as optimal before sufficient learning, resulting in artificially high immediate rewards. However, as training progresses, while the overall reward value of DQN increases, frequent fluctuations persist in the mid-to-late stages, indicating significant policy degradation and value function distortion issues in high-dimensional nonlinear environments. In contrast, the dual DQN algorithm has relatively low reward values ​​in the early training stages (the first 30 rounds), but the curve rises faster, demonstrating the policy's stable improvement capability under reasonable exploration and accurate evaluation mechanisms. In the mid-stage (rounds 30 to 100), the reward value continues to rise with significantly reduced fluctuations, and the average reward shows a steady growth trend, indicating steady policy convergence. In the later stage (after round 100), the reward value tends to stabilize and remains at a high level, indicating that the policy can achieve stable and efficient action selection in most situations. In summary, the trend of reward value changes reveals the algorithm's exploratory ability, estimation bias control, and convergence performance, verifying the applicability and advantages of dual DQN in hybrid power energy management tasks.

[0176] Based on the trained dual-DQN agent, vehicle speed tracking and energy management performance simulations were conducted under NEDC conditions. The results show that the strategy can effectively control the speed of the series hybrid vehicle to follow the target speed. During a continuous operation lasting 1220 seconds, except for a few rapid acceleration phases where the actual vehicle speed was slightly higher than the target speed, the overall speed change trend was basically consistent with the target speed, indicating that the dual-DQN has good tracking performance under steady-state operating conditions. In the hybrid system, the agent uses a deep reinforcement learning framework to dynamically adjust the engine output power to optimize the coordination between the battery state of charge (SOC) and the vehicle's power demand. When the initial battery SOC is low (e.g., 0.3), the agent senses insufficient battery energy and actively drives the engine to output maximum power to meet the driving power demand. Simultaneously, it rapidly charges the battery using redundant power, significantly improving the SOC. During this stage, the engine power far exceeds the vehicle's instantaneous load, and the battery power exhibits a significant positive charging effect, demonstrating the system's strong response capability to energy recovery. As the State of Charge (SOC) gradually recovers to near the target value of 0.6, the agent flexibly adjusts the engine power output according to environmental conditions, gradually reducing the charging power to prevent overcharging and ensuring a balance and economy in energy supply. When the SOC reaches or slightly exceeds the target value (e.g., 0.8), the agent prioritizes pure electric drive mode, reducing the engine output power to an extremely low level and providing only a small amount of electricity generation when necessary to maintain SOC stability, avoiding frequent engine start-stop and fuel waste. At this time, the battery mainly bears the driving load, and the battery power is negative, reflecting its state of discharge.

[0177] In the simulation platform, the FTP75 operating condition was input as an unknown operating condition into the offline-trained energy management strategy. The changes in vehicle speed following effect, system output power, and battery SOC are shown in the attached figure. Compared with other operating conditions, the FTP75 operating condition involves more frequent and drastic acceleration and deceleration, which can more realistically simulate the operating characteristics of hybrid vehicles in complex driving environments, placing higher demands on the response speed and collaborative control capabilities of the energy management strategy.

[0178] At high SOC (State of Charge), the total system output power is primarily contributed by battery discharge, significantly reducing the frequency of engine power intervention and power fluctuations, resulting in high overall stability of the powertrain. The energy management strategy prioritizes battery power to meet power demands during low load and moderate acceleration, avoiding frequent engine start-stop cycles and drastic power adjustments, thus effectively improving system energy efficiency. The SOC curve exhibits a slow and controlled downward trend, reflecting that the strategy, while ensuring power requirements, achieves reasonable regulation and constraint of battery state of charge, avoiding the problem of limited regenerative energy caused by excessively high SOC, and ensuring continuous and stable operation of the powertrain. Furthermore, the stable engine operation helps reduce peak fuel consumption, thereby improving overall vehicle fuel economy.

[0179] When the State of Charge (SOC) drops to the preset target range, the energy management strategy dynamically adjusts the power distribution ratio between the engine and battery, stabilizing the SOC and significantly reducing its fluctuations. During this phase, the output power curve exhibits a more balanced characteristic. By optimizing the engine operating point and rationally configuring battery discharge power, the system achieves a smooth transition in power response and further reduces fuel consumption. Overall, this strategy effectively improves the comprehensive energy efficiency and fuel economy of the powertrain system, while ensuring continuous vehicle power performance and extending battery life.

[0180] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN, characterized in that, Includes the following steps: Step 1: Construct a series hybrid vehicle powertrain model, including an engine-generator set, power battery, electric motor drive, and a module for calculating the vehicle's required power. Each module is interconnected through energy flow, and the vehicle's required power module calculates instantaneous power in real time based on vehicle speed, acceleration, road resistance, and special operating loads. Step 2: Propose a multi-dimensional feature fusion and adaptive learning mechanism, which constructs a comprehensive state space by fusing features such as vehicle speed, SOC, power demand, road slope, and load intensity; adopt a hierarchical decision architecture to achieve task-level and energy-level collaboration; and introduce a dynamic confidence adjustment and constraint-guided correction mechanism to improve the convergence stability and physical feasibility of reinforcement learning. Step 3: Construct a state space from key variables such as vehicle speed, battery state of charge, and instantaneous power demand of the vehicle. Use the engine power generation as the action set and design an energy management controller based on dual DQN. Improve the stability of the learning process by using the action selection and value evaluation separation mechanism. The reward function is designed with the combined fuel consumption and battery SOC maintenance as the optimization objectives. The agent is trained by combining experience replay and target network soft update to obtain a convergent and stable energy management strategy. Step 4: Conduct multi-condition simulation verification in the MATLAB / Simulink environment, and evaluate the performance of the control strategy in terms of fuel economy and SOC stability through typical and compound operating condition tests.

2. The multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN as described in claim 1, characterized in that, In step 1, the system-level simulation platform describes the energy transfer and conversion relationships between the engine, generator, battery and motor through modular modeling, ensuring that the vehicle power requirements are met under various operating conditions, while providing a realistic dynamic environment for subsequent energy management strategies.

3. The multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN as described in claim 1, characterized in that, In step 2, a multi-dimensional feature fusion and adaptive learning mechanism is proposed to enhance the modeling accuracy and robustness of reinforcement learning under complex working conditions. By fusing multi-source features such as vehicle speed, battery SOC, power demand, engine speed gradient, road slope, and load intensity, a comprehensive state space that can fully characterize the operating state is constructed. A hierarchical decision architecture is adopted, with the upper layer generating energy allocation coefficients based on task type and power trend, and the lower layer performing local power optimization by dual DQN controllers, achieving macro- and micro-decision coordination. At the same time, a dynamic confidence adjustment mechanism based on Q-value fluctuations and a constraint-guided action correction function are introduced to ensure the stability of the policy learning process and meet physical constraints such as power and SOC.

4. The multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN as described in claim 1, characterized in that, In step 3, the state space includes vehicle speed, SOC, battery power, and power allocation information from the previous moment; the action space consists of discrete values ​​of engine power generation; the reward function considers both fuel economy and SOC stability, setting corresponding penalty terms to avoid overcharging or over-discharging of SOC, ensuring battery health and energy utilization efficiency during long-term vehicle operation; the controller uses an experience replay mechanism to randomly sample state transition samples in batches to improve the diversity and utilization of training data; simultaneously, a target network soft update mechanism is adopted to smooth the parameter update process, reduce instability caused by gradient fluctuations during training, and thus obtain a stable and convergent energy management strategy.

5. The multi-objective energy optimization algorithm for series hybrid vehicles based on dual DQN as described in claim 1, characterized in that, In step 4, simulation verification covers training conditions, typical conditions, and extreme load conditions to verify the adaptability of the energy management strategy under different scenarios. The results show that the method can improve fuel economy and enhance SOC maintenance, and has strong generalization ability and robustness.

Images