A fuel cell vehicle energy management method based on constraint reinforcement learning

Abstract

The application provides a fuel cell vehicle energy management method based on constraint reinforcement learning. By introducing a constraint reinforcement learning framework, the application realizes the double decoupling of economic optimization and safety constraints in the action mechanism and time scale, thereby overcoming the problem of unclear target orientation and unstable learning caused by the coupling of multiple objectives through fixed weights in a single reward function in the traditional reward shaping method. The steps of the application are as follows: obtaining relevant state information of a fuel cell vehicle, establishing a fuel cell vehicle energy management system model, constructing an energy management strategy model based on constraint reinforcement learning, and finally training the energy management strategy based on constraint reinforcement learning in a simulation environment to obtain an optimal control strategy. The application significantly reduces the complexity and uncertainty of multi-objective collaborative learning through decoupling design, improves the convergence stability and convergence speed in the strategy training stage, and improves the running economy of the system while meeting the safety constraints.

Description

Technical Field

[0001] This invention belongs to the field of fuel cell vehicle energy management, and in particular relates to a fuel cell vehicle energy management method based on constraint reinforcement learning. Background Technology

[0002] With the advancement of low-carbonization and energy structure transformation in the transportation sector, fuel cell vehicles are gradually becoming an important development direction for new energy vehicles due to their advantages such as zero emissions, high energy conversion efficiency, and fast refueling. However, due to the inherent slow dynamic response characteristics of fuel cell systems, actual vehicles need to be equipped with secondary energy sources such as lithium batteries to meet transient power demands. For this hybrid energy storage system, in order to ensure safe system operation, improve vehicle economy, and extend the service life of key components, the key lies in designing an efficient energy management strategy to achieve coordinated control of multiple energy systems.

[0003] Currently, energy management strategies for fuel cell vehicles mainly include rule-based methods, optimization-based methods, and reinforcement learning-based methods. Rule-based strategies rely on human experience, resulting in limited adaptability and effectiveness. Optimization-based strategies can be divided into global optimization and instantaneous optimization. The former has high computational complexity, making it difficult to meet the needs of real-time applications. Although it possesses some real-time performance, it typically only obtains local optima and is highly sensitive to model accuracy and parameter settings. With the development of artificial intelligence technology, reinforcement learning-based energy management strategies are gradually gaining attention. This type of method learns the optimal control strategy through interaction between the agent and the environment, exhibiting strong adaptability.

[0004] However, existing reinforcement learning methods typically employ reward shaping, coupling economic objectives and security constraints into the same scalar reward function through pre-defined weight coefficients. Within this framework, the scalar action value function estimated by the Critic network simultaneously incorporates the combined impact of economic gains and security penalties on future discounted returns, obscuring the independent guiding role of these two types of objectives in policy optimization. This not only increases the difficulty and uncertainty of policy learning but also affects the policy's convergence speed and the effectiveness of multi-objective collaborative optimization. Summary of the Invention

[0005] To address the problems and needs in the background technology, the purpose of this invention is to provide an energy management method for fuel cell vehicles based on a constraint reinforcement learning framework. This invention improves the traditional soft actor-critic (SAC) algorithm by introducing a constraint reinforcement learning framework. Its core lies in decoupling the economic optimization objective from the safety constraints in terms of both the mechanism of action and the time scale. Specifically, the reward function is used only to describe economic objectives such as hydrogen consumption cost and energy source decay, allowing the value function to purely evaluate the long-term merits of actions from an economic perspective. Simultaneously, safety factors such as SOC management and power demand balance are incorporated into the construction of the policy optimization objective in the form of single-step penalties. An adaptive penalty intensity adjustment mechanism is designed to dynamically adjust the penalty intensity of SOC management to ensure that SOC remains within a safe range. Through this design, the policy update process is simultaneously guided by clear economic value from the Critic network and by immediate safety behavior correction from the penalty function. This dual decoupling design in terms of both the mechanism of action and the time scale significantly reduces the complexity of multi-objective learning, achieving faster convergence speed and better collaborative optimization results.

[0006] The technical solution adopted in this invention is as follows:

[0007] I. An Energy Management Method for Fuel Cell Vehicles Based on Constraint Reinforcement Learning

[0008] Step 1) Obtain the attribute parameters and operating status parameters of the fuel cell vehicle, and then construct a hybrid power system model of the fuel cell vehicle.

[0009] Step 2) Build a system simulation environment based on the hybrid power system model of fuel cell vehicle, construct an improved SAC algorithm based on the constraint reinforcement learning framework, train the improved SAC algorithm based on the constraint reinforcement learning framework in the system simulation environment, and obtain the optimal energy management strategy.

[0010] Step 3) Deploy the optimal energy management strategy in fuel cell vehicles to achieve energy management of fuel cell vehicles.

[0011] In step 1), the hybrid power system model includes a longitudinal vehicle dynamics model, a fuel cell hydrogen consumption model, a fuel cell life model, a second-order equivalent circuit model of the lithium battery, and an aging model.

[0012] In the improved SAC algorithm based on the constrained reinforcement learning framework, the reward function satisfies the following formula:

[0013]

[0014] in, Let be the reward function value at step t. The total equivalent hydrogen consumption at step t. The differential, These are the weighting coefficients for equivalent hydrogen consumption, fuel cell degradation, battery degradation, and efficiency compensation, respectively. This refers to the hydrogen consumption of the fuel cell; This is the equivalent hydrogen consumption of a lithium battery. To maximize the efficiency of the fuel cell system, For DC / AC converter efficiency; This refers to the battery's rated capacity. The change in the health status of a lithium battery is represented by LHV, which is the lower heating value of hydrogen. The charging efficiency of lithium batteries; It is the change in battery capacity.

[0015] In the improved SAC algorithm based on the constrained reinforcement learning framework, the loss function satisfies the following formula:

[0016]

[0017] in, The value of the loss function; Let be the expectation operator, representing the expectation of the state-action trajectory distribution; The action value function for choosing action a in state s; This is the entropy regularization coefficient; Let the probability distribution of the parameterized policy function choosing action a in state s be defined. For parameterized policy networks; The penalty intensity factor; The penalty function value is managed by the SOC. This represents the power supply and demand balance penalty function value.

[0018] SOC management penalty function value at time step t Satisfy the following formula:

[0019]

[0020]

[0021] in, Use the baseline penalty function; This is the penalty coefficient for SOC management; Basic penalty coefficient; This represents the maximum output power of the fuel cell system. Let be the output power of the fuel cell system at time step t; Let t be the state of charge at time step t; The target state of charge; Represents the state s at time step t; Let a represent the action at time step t.

[0022] The power supply and demand balance penalty function value at time step t Satisfy the following formula:

[0023]

[0024] in, This is the power supply and demand balance constraint factor; This refers to the maximum permissible discharge power of the lithium battery. Let t be the power required at step t; For DC / AC converter efficiency; Let be the output power of the fuel cell system at time step t.

[0025] Dynamically adjust the penalty intensity factor The value of .

[0026] The penalty intensity factor is dynamically adjusted using a PID controller. The value of is determined by the following formula:

[0027]

[0028]

[0029] in, These are the proportional feedback term, integral feedback term, and derivative feedback term of the PID controller at time step t during the k-th training round; and These are the smoothing factors for the proportional and differential terms, respectively; These are the corresponding proportional gain coefficient, integral gain coefficient, and differential gain coefficient, respectively. is the smoothed cost signal after low-pass filtering; d is the delay step used to calculate the differential term; A safe cost benchmark for training round k; The historical value of the integral feedback term at time step t-1 of the k-th training round; The smoothed cost signal history value at time step t-1 of the k-th training round; This represents the average SOC management cost. For the kd round of training Smoothing cost signal historical values ​​at time steps; Let be the penalty intensity factor for the k-th training round at time step t; This is the baseline value for the safe cost of the k-th training round; Let i be the total number of time steps in round i. Let i be the total set of time steps contained in round i; Let be the SOC management penalty value at time step t in round i.

[0030] The safety cost benchmark is modified using the following formula:

[0031]

[0032]

[0033]

[0034] in, This is a factor for determining violations; Threshold adjustment direction factor; Update the step size for the threshold; The safety cost baseline for training round k+1; This is the initial safety cost baseline value.

[0035] II. A Fuel Cell Vehicle Energy Management System Based on Constraint Reinforcement Learning

[0036] The data acquisition unit is used to collect and acquire the attribute parameters and operating status parameters of the fuel cell vehicle.

[0037] The simulation environment simulation unit is used to construct a hybrid power system model of the fuel cell vehicle based on the acquired attribute parameters and operating state parameters of the fuel cell vehicle, and then build a system simulation environment;

[0038] The policy generation unit is used to construct an improved SAC algorithm based on a constrained reinforcement learning framework and train it in a system simulation environment to obtain the optimal energy management policy.

[0039] The strategy deployment unit is used to deploy the optimal energy management strategy in fuel cell vehicles.

[0040] III. A computer device

[0041] The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the constrained reinforcement learning-based fuel cell vehicle energy management method.

[0042] The beneficial effects of this invention are:

[0043] In this invention, the reward function and action value function focus only on economic objectives, while safety objectives such as SOC management and power supply and demand balance participate in the construction of strategy optimization objectives in the form of single-step penalties. At the same time, an adaptive penalty intensity adjustment mechanism is designed to dynamically adjust the penalty intensity of SOC management to ensure that SOC is maintained within a safe range.

[0044] This invention improves upon traditional reinforcement learning energy management methods based on a constrained reinforcement learning framework. Specifically, it decouples economic objectives from security constraints in terms of their mechanism of action and time scale during policy optimization. This overcomes the problems of unclear objective orientation and unstable learning caused by coupling multiple objectives to a single reward function with fixed weights in existing methods.

[0045] Therefore, this invention effectively reduces the complexity and uncertainty of multi-objective collaborative learning, improves the convergence stability and convergence speed in the policy training phase, and enhances the system's operational economy while satisfying safety constraints. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the training process of a constraint reinforcement learning model.

[0047] Figure 2 This is a schematic diagram of the structure of a fuel cell vehicle system.

[0048] Figure 3 This is a comparison chart of the convergence characteristics of the method of this invention and the traditional SAC method during the training phase under a unified evaluation benchmark.

[0049] Figure 4 This is a comparison chart of the SOC of lithium batteries using the method of this invention and the traditional SAC method on the training set.

[0050] Figure 5 is a comparison of the output power of the fuel cell system using the method of this invention and the classic energy management algorithm on the validation set.

[0051] Figure 6 This is a flowchart of the method of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be further described below in conjunction with the accompanying drawings and actual experiments.

[0053] like Figure 6 As shown, the energy management method for fuel cell vehicles based on constraint reinforcement learning proposed in this invention includes the following steps:

[0054] Step 1) Fuel cell vehicles are equipped with auxiliary batteries. The system framework diagram of a fuel cell vehicle is as follows: Figure 2 As shown, the attribute parameters and operating state parameters of the fuel cell vehicle are obtained, and then a hybrid power system model of the fuel cell vehicle is constructed; among them, the attribute parameters include the vehicle mass. Windward area Wheel rolling resistance coefficient Operating parameters include air drag coefficient. air density Vehicle speed The hybrid power system model includes the vehicle's longitudinal dynamics model, the fuel cell hydrogen consumption model, the fuel cell lifespan model, the second-order equivalent circuit model of the lithium battery, and the aging model.

[0055] The hybrid power system model includes the vehicle's longitudinal dynamics model, the fuel cell hydrogen consumption model, the fuel cell life model, the second-order equivalent circuit model of the lithium battery, and the aging model.

[0056] The vehicle dynamics model satisfies the following formula:

[0057]

[0058] in, For the traction required by fuel cell vehicles, To accelerate the vehicle, For road slope, For drive power, For the required power, For the output power of the fuel cell system, For lithium battery output power, For motor efficiency, For DC / AC converter efficiency, For gear rotation efficiency; It is a symbolic function; m represents the total mass of the vehicle, and g represents the acceleration due to gravity. Indicates the air drag coefficient; Indicates the vehicle's frontal area; Indicates the vehicle's speed; These represent acceleration drag, gradient drag, rolling drag, and air drag, respectively.

[0059] The hydrogen consumption model for a fuel cell system satisfies the following formula:

[0060]

[0061] in, LHV is the hydrogen consumption rate, and LHV is the lower heating value of hydrogen. For the efficiency of the fuel cell system.

[0062] Fuel cell lifespan degradation is mainly caused by various harsh operating conditions, including start-stop cycles, low-power operation, drastic load fluctuations, and high-power, high-load operation. Therefore, the impact of these factors is quantified, and a discretization method is used to establish the following fuel cell aging model as the fuel cell lifespan model, satisfying the following formula:

[0063]

[0064]

[0065] in, The total performance degradation of the fuel cell is expressed as % (%). , , , Representing respectively in the Within each time step, the performance degradation caused by start-up and shutdown, low-power operation, load variation, and high-power conditions; T is the total number of time steps. For time step; Let be the operating status signal of the fuel cell system at time t. This is the operating status signal of the fuel cell system at time step t-1, where 0 represents shutdown and 1 represents normal operation. and These are the low power threshold and high power threshold of the fuel cell, respectively. This indicates the absolute value operation.

[0066] The second-order equivalent circuit model of a lithium battery satisfies the following formula:

[0067]

[0068]

[0069] Where SOC stands for State of Charge. The rated capacity of the battery; I and U are the load current and terminal voltage, respectively; For two polarization voltages The differential; There are two polarization internal resistances; Polarizing capacitor; The internal resistance is ohmic; This is the open-circuit voltage of the lithium battery.

[0070] The lithium battery aging model satisfies the following formula:

[0071]

[0072] Among them, SOH represents the health status of the lithium battery. The equivalent number of cycles to reach the end of life, where c is the multiplier. Here, z is the pre-exponential factor, z is the power-law factor, and R is the molar gas constant. This represents the internal average temperature. This represents the change in the health status of the lithium battery. The charging and discharging current of the lithium battery; The equivalent number of cycles at which a lithium battery reaches the end of its lifespan.

[0073] Step 2) Build a system simulation environment based on the hybrid power system model of fuel cell vehicle, construct an improved SAC algorithm based on the constraint reinforcement learning framework, and train the improved SAC algorithm based on the constraint reinforcement learning framework in the system simulation environment using multiple standard driving cycles to obtain the optimal energy management strategy.

[0074] In the improved SAC algorithm based on the constraint reinforcement learning framework, the state for ;in, Let be the state of charge at time t, the output power of the fuel cell system at time t-1, and the power demand at time t, respectively. This represents the change in power demand at time t. The output power of the fuel cell system is selected as the action variable. .

[0075] The reward function satisfies the following formula:

[0076]

[0077] in, Let be the reward function value at step t. The total equivalent hydrogen consumption at time step t. The differential, These are the weighting coefficients for equivalent hydrogen consumption, fuel cell degradation, battery degradation, and efficiency compensation, respectively. Hydrogen consumption for fuel cells; This is the equivalent hydrogen consumption of a lithium battery. The maximum efficiency of the fuel cell system is used here because the real-time calculation of the average efficiency violates the Markov property. Therefore, the maximum efficiency of the fuel cell system is used instead, and a fourth efficiency compensation term is introduced into the reward function. For DC / AC converter efficiency; This refers to the battery's rated capacity. The change in the health status of a lithium battery is represented by LHV, which is the lower heating value of hydrogen. The charging efficiency of lithium batteries; It is the change in battery capacity.

[0078] Existing reward functions for fuel cell vehicle energy management generally only optimize the actual hydrogen consumption of the fuel cell, ignoring the "equivalent hydrogen consumption" of lithium battery charging and discharging behavior. Discharging a lithium battery consumes energy stored in the battery, which comes from the hydrogen consumed during previous or subsequent charging, thus effectively increasing hydrogen consumption. Charging, on the other hand, stores the energy output from the fuel cell, which can be used to replace the fuel cell output during subsequent discharge, thus effectively reducing hydrogen consumption. However, existing methods do not convert this into hydrogen consumption units, leading to strategies that rely excessively on battery discharge for short-term energy savings, accelerating battery degradation and increasing long-term energy consumption. This invention designs an equivalent hydrogen consumption model for lithium batteries, uniformly quantifying the battery energy change ΔSOC into hydrogen consumption cost. This includes factoring in equivalent hydrogen consumption during discharge and converting hydrogen consumption savings during charging, integrating these into the reward function to achieve overall optimization of "fuel cell hydrogen consumption - battery equivalent hydrogen consumption," improving the overall energy economy throughout the entire lifecycle.

[0079] Establish an improved SAC algorithm based on a constraint reinforcement learning framework:

[0080]

[0081] in, The optimal energy management strategy to be solved is... The parameter is A policy network is used to generate control actions based on the system state. Represents the candidate parameter space of the policy network; Denotes the mathematical expectation, here representing the expectation of the experience replay pool. Calculate the expectation of the state distribution and the action distribution generated by the policy. For parameters An action value network is used to evaluate actions taken in state s. Future returns This is the entropy weighting coefficient, used to adjust the balance between the exploratory and deterministic aspects of the strategy. This represents the logarithmic probability of the policy's output action.

[0082] To solve the above-mentioned constrained optimization problem, this invention introduces a penalty intensity factor into the loss function. The loss function satisfies the following formula:

[0083]

[0084] in, The value of the loss function; Let be the expectation operator, representing the expectation of the state-action trajectory distribution; The action value function for selecting action a in state s; This is the entropy regularization coefficient; Let the probability distribution of the parameterized policy function choosing action a in state s be defined. For parameterized strategy functions; The penalty intensity factor; The penalty function value is managed by the SOC. This represents the power supply and demand balance penalty function value.

[0085] SOC management penalty function value at time step t Satisfy the following formula:

[0086]

[0087]

[0088] in, The baseline penalty function (i.e., the traditional penalty function); This is the penalty coefficient for SOC management; Basic penalty coefficient; This represents the maximum output power of the fuel cell system. Let t be the output power of the fuel cell system at step t; Let t be the state of charge at step t; The target state of charge; Represents the state s at time step t; Let a represent the action at time step t.

[0089] The power supply and demand balance penalty function value at time step t Satisfy the following formula:

[0090]

[0091] in, This is the power supply and demand balance constraint factor, and its value is set to 60 here to enforce the power balance constraint. This refers to the maximum permissible discharge power of the lithium battery. Let t be the power required at step t; For DC / AC converter efficiency; Let be the output power of the fuel cell system at step t.

[0092] This invention utilizes a traditional penalty function Introducing dynamic motion coefficients and This addresses the problem that the original function lacked sufficient differentiation for actions with different power levels, making it difficult to guide the strategy to generate actions with higher safety awareness during training. Furthermore, given the physical limitation that the maximum output power of a fuel cell is much lower than the peak power of a lithium battery, when the system faces high power demands and is in a low SOC state, boundary-triggered remedial strategies alone are insufficient to prevent the SOC from continuously dropping. The improved SOC management penalty function, when the SOC is low (e.g., SOC < 0.5), encourages the fuel cell to output higher power earlier to prevent over-discharge of the lithium battery by reducing the penalty value for high-power actions. Conversely, when the SOC is high, it suppresses fuel cell output by increasing the penalty value for high-power actions to avoid overcharging of the lithium battery.

[0093] In one feasible implementation, the penalty intensity factor can be dynamically adjusted using the gradient descent method. The value of .

[0094] In one feasible implementation, a PID controller can be used to dynamically adjust the penalty intensity factor. The value of is determined by the following formula:

[0095]

[0096]

[0097] in, These are the proportional feedback term, integral feedback term, and derivative feedback term of the PID controller at time step t during the k-th training round; and These are the smoothing factors for the proportional and differential terms, respectively; These are the corresponding proportional gain coefficient, integral gain coefficient, and differential gain coefficient, respectively. is the smoothed cost signal after low-pass filtering; d is the delay step used to calculate the differential term; The safety cost baseline for training round k, i.e., the control error, is specifically the average SOC management cost of the first four rounds. Compared with current security cost benchmarks difference; The historical value of the integral feedback term at time step t-1 of the k-th training round; The smoothed cost signal history value at time step t-1 of the k-th training round; This represents the average SOC management cost. For the kd round of training Smoothing cost signal historical values ​​at time steps; Let be the penalty intensity factor for the k-th training round at time step t; This is the baseline value for the safe cost of the k-th training round; Let i be the total number of time steps in round i. Let i be the total set of time steps contained in round i; Let be the SOC management penalty value at time step t in round i.

[0098] In one feasible implementation, the safety cost benchmark is a preset fixed value.

[0099] In one feasible implementation, the safety cost benchmark can be modified using the following formula:

[0100]

[0101]

[0102]

[0103] in, This is a violation determination factor used to determine whether the battery SOC has exceeded the preset boundary in the last four rounds; The threshold adjustment direction factor is determined based on the relative relationship between the number of violations and the average SOC management cost; The threshold update step size is a preset value; The safety cost baseline for training round k+1; This is the initial safety cost baseline value.

[0104] when Below If a State of Charge (SOC) violation occurs, the safety cost baseline is reduced; Higher than If there is no SOC violation, the security cost benchmark is relaxed; otherwise, the security cost benchmark remains unchanged.

[0105] This invention constructs a method for dynamically adjusting the penalty intensity factor using a PID controller. With safety cost benchmark The self-tuning-based collaborative adjustment mechanism dynamically scales the penalty intensity factor to keep the battery operating within the safe and feasible SOC range.

[0106] During the training of the SAC model, policy optimization is achieved by minimizing this loss function, where the power supply and demand balance penalty term is included. and SOC management penalty items By incorporating a single-step penalty term into the construction of the policy optimization objective, the policy can obtain an instantaneous safety correction gradient. Forced constraints are implemented through preset fixed high-weight coefficients, and the penalty intensity factor is used. The global scaling factor, serving as the SOC management penalty term, is dynamically adjusted by an adaptive penalty intensity adjustment mechanism to ensure that the SOC remains within the constraint range. This invention, through the construction of this loss function, achieves a dual decoupling of the economic optimization objective and the safety constraint in terms of both the mechanism of action and the time scale, thereby significantly reducing the complexity of multi-objective learning.

[0107] One feasible implementation method is, for example Figure 1 As shown, the improved SAC algorithm based on a constraint reinforcement learning framework is trained using multiple standard driving cycles in a system simulation environment to obtain the optimal energy management strategy, including:

[0108] (1) Initialization phase:

[0109] Set the algorithm hyperparameters, including the total number of training iterations E and the sampling batch size B; initialize the policy network parameters. Critic network parameters and its target network parameters Set the initial values ​​of the Lagrange multipliers. and initial safety cost benchmark Set the PID controller parameters, including the proportional gain. Integral gain Differential gain And the delay step d.

[0110] (2) Experience collection and storage:

[0111] In each training round k, the simulation environment state is reset. In each time step t, the policy network adjusts the current state accordingly. Output Action After the environment performs an action, it returns to the next state. Economic rewards ; this quadruple empirical data Store in the experience replay pool middle.

[0112] (3) Iterative update of network parameters:

[0113] When the sample size in the experience replay pool is greater than the sampling batch B, periodically from... A batch of samples is randomly drawn from the dataset. The Critic network is updated by minimizing the time-series error. :

[0114]

[0115] in, For future value expectations, This is the action the current policy network outputs for the next state. Then, the loss function value is minimized. Update the Actor network and the target network Perform a soft update:

[0116]

[0117] in, This is a soft update coefficient; This is an assignment operator, indicating a parameter update operation. Finally, the temperature factor is automatically adjusted based on entropy constraints. .

[0118] (4) Penalty intensity factor Real-time adjustment:

[0119] When the number of training epochs k > 8, adaptive penalty factor adjustment is enabled. The PID controller adjusts the penalty factor based on the deviation between the current average SOC management cost and the safety cost benchmark. Calculate proportional, integral, and differential feedback terms, and update the penalty intensity factor in real time. .

[0120] (5) Safety cost benchmark Self-tuning:

[0121] When the number of training rounds k>8, at the end of each round, the security cost baseline self-tuning logic is triggered (i.e., the security cost baseline is adjusted) based on the SOC violations and average SOC management cost performance of the most recent four rounds.

[0122] Step 3) Deploy the optimal energy management strategy in fuel cell vehicles to achieve energy management of fuel cell vehicles.

[0123] Finally, to illustrate the superiority of the proposed method, this embodiment compares and analyzes it with existing technologies. First, a longitudinal comparison is made between the proposed method (Constrained SAC) and two reward-based SAC benchmark strategies: SAC PN using the improved SOC management penalty function and SAC_PO using the traditional SOC management penalty function. Then, a horizontal comparison is made between the proposed method and classic energy management algorithms. The performance statistics of each method are shown in Tables 1 and 2, respectively.

[0124] Table 1. Performance comparison of the method of this invention with the traditional SAC-based method on the training set.

[0125]

[0126] Table 2 Performance comparison of the method of this invention and classical energy management algorithms on the validation set.

[0127]

[0128] The longitudinal comparison results show that: Figure 3 This indicates that the algorithm has the fastest convergence speed and the best performance; Figure 4 The improved SOC penalty function shows that both Constrained SAC and SAC_PN can effectively maintain SOC within the safe range. Table 1's quantitative results indicate that Constrained SAC, with an equivalent hydrogen consumption of 4382 g / 100 km, outperforms SAC-PN, although slightly higher than SAC-PO, but the latter sacrifices SOC safety. More importantly, Constrained SAC achieves low energy consumption while maintaining SOC stability, and exhibits excellent overall driving costs (including hydrogen consumption, fuel cell and lithium battery degradation costs), achieving the best balance between energy consumption, lifespan, and safety. SAC-PN is slightly less economical due to its reward-shaping mechanism during multi-objective optimization; while SAC-PO has significant shortcomings in constraint compliance and fuel cell protection.

[0129] The results of the horizontal comparison show that: Figure 5 The results show that the DP algorithm produces the most stable fuel cell output power, followed by Constrained SAC, while the power fluctuations of TD3 and AECM accelerate stack aging. Table 2 shows that the equivalent hydrogen consumption of Constrained SAC is only 3846 g / 100 km, 0.41% lower than DP, achieving near-globally optimal fuel economy. The higher average fuel cell efficiency indicates that it operates more within its high-efficiency range, while TD3 and AECM are inferior to DP and Constrained SAC in both efficiency and hydrogen consumption. Considering both hydrogen consumption and degradation costs, the driving cost index shows that Constrained SAC significantly outperforms TD3 and AECM, approaching the optimal level of DP. These results fully validate the performance advantages of the Constrained SAC strategy and its strong generalization ability to unknown operating conditions.

[0130] The embodiments of the present invention have been described and illustrated above in conjunction with the accompanying drawings, but are not limited to the above-described embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes and modifications can be made based on the concept of the present invention.

Claims

1. A method for energy management of fuel cell vehicles based on constrained reinforcement learning, characterized in that, Includes the following steps: Step 1) Obtain the attribute parameters and operating status parameters of the fuel cell vehicle, and then construct a hybrid power system model of the fuel cell vehicle. Step 2) Build a system simulation environment based on the hybrid power system model of fuel cell vehicle, construct an improved SAC algorithm based on the constraint reinforcement learning framework, train the improved SAC algorithm based on the constraint reinforcement learning framework in the system simulation environment, and obtain the optimal energy management strategy. Step 3) Deploy the optimal energy management strategy in fuel cell vehicles to achieve energy management of fuel cell vehicles.

2. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 1, characterized in that, In step 1), the hybrid power system model includes a longitudinal vehicle dynamics model, a fuel cell hydrogen consumption model, a fuel cell life model, a second-order equivalent circuit model of the lithium battery, and an aging model.

3. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 1, characterized in that, In the improved SAC algorithm based on the constrained reinforcement learning framework, the reward function satisfies the following formula: in, Let be the reward function value at step t. The total equivalent hydrogen consumption at step t. The differential, These are the weighting coefficients for equivalent hydrogen consumption, fuel cell degradation, battery degradation, and efficiency compensation, respectively. This refers to the hydrogen consumption of the fuel cell; This is the equivalent hydrogen consumption of a lithium battery. To maximize the efficiency of the fuel cell system, For DC / AC converter efficiency; This refers to the battery's rated capacity. The change in the health status of a lithium battery is represented by LHV, which is the lower heating value of hydrogen. The charging efficiency of lithium batteries; It is the change in battery capacity.

4. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 1, characterized in that, In the improved SAC algorithm based on the constrained reinforcement learning framework, the loss function satisfies the following formula: in, The value of the loss function; Let be the expectation operator, representing the expectation of the state-action trajectory distribution; The action value function for choosing action a in state s; This is the entropy regularization coefficient; Let the probability distribution of the parameterized policy function choosing action a in state s be defined. For parameterized policy networks; The penalty intensity factor; For SOC management penalty function value, This represents the power supply and demand balance penalty function value.

5. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 4, characterized in that, SOC management penalty function value at time step t Satisfy the following formula: in, Use the baseline penalty function; This is the penalty coefficient for SOC management; Basic penalty coefficient; This represents the maximum output power of the fuel cell system. Let be the output power of the fuel cell system at time step t; Let t be the state of charge at time step t; The target state of charge; Represents the state s at time step t; Let a represent the action at time step t.

6. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 4, characterized in that, The power supply and demand balance penalty function value at time step t Satisfy the following formula: in, This is the power supply and demand balance constraint factor; This refers to the maximum allowable discharge power of the lithium battery. Let t be the power required at time step t; For DC / AC converter efficiency; Let be the output power of the fuel cell system at time step t.

7. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 4, characterized in that, Dynamically adjust the penalty intensity factor The value of .

8. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 4, characterized in that, The penalty intensity factor is dynamically adjusted using a PID controller. The value of is determined by the following formula: in, These are the proportional feedback term, integral feedback term, and derivative feedback term of the PID controller at time step t during the k-th training round; and These are the smoothing factors for the proportional and differential terms, respectively; These are the corresponding proportional gain coefficient, integral gain coefficient, and differential gain coefficient, respectively. is the smoothed cost signal after low-pass filtering; d is the delay step used to calculate the differential term; A safe cost benchmark for training round k; The historical value of the integral feedback term at time step t-1 of the k-th training round; The smoothed cost signal history value at time step t-1 of the k-th training round; This represents the average SOC management cost. For the kd round of training Smoothing cost signal historical values ​​at time steps; Let be the penalty intensity factor for the k-th training round at time step t; This is the baseline value for the safe cost of the k-th training round; Let i be the total number of time steps in round i. Let i be the total set of time steps contained in round i; Let be the SOC management penalty value at time step t in round i.

9. The energy management method for fuel cell vehicles based on constraint reinforcement learning according to claim 8, characterized in that, The safety cost benchmark is modified using the following formula: in, This is a factor for determining violations; Threshold adjustment direction factor; Update the step size for the threshold; The safety cost baseline for training round k+1; This is the initial safety cost baseline value.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the energy management method for fuel cell vehicles based on constraint reinforcement learning as described in any one of claims 1 to 9.

Images