A reinforcement learning energy management method for fuel cell vehicles considering operating condition prediction

Abstract

The present invention provides a fuel cell vehicle reinforcement learning energy management method considering working condition prediction, the method comprising: using four different types of classic driving conditions to construct comprehensive driving condition data, segmenting the comprehensive driving condition data, extracting characteristic parameters of the conditions to perform principal component analysis, and clustering the driving cycle segments by a K-means clustering method; after obtaining three different types of driving cycle segment sets, offline training a Bi-LSTM-based working condition recognition module and a Markov-based vehicle speed prediction module, wherein data sets representing three different road characteristics are selected in the vehicle speed prediction module to respectively train three different Markov matrices; extracting working condition characteristic parameters of the vehicle speed prediction sequence, converting them into equivalent factor adjustment coefficients #imgabs0#, and inputting #imgabs1# as a state into a DDPG energy management algorithm, and the fuel cell power P fc As an action, set a suitable reward function to achieve comprehensive consideration and multi-objective optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of fuel cells, and in particular to a fuel cell vehicle reinforcement learning energy management method considering operating condition prediction. Background Art

[0002] At present, the world's energy depletion and environmental crisis are becoming increasingly serious, and cars are one of the important carriers of energy consumption and air pollution, so the optimization of automobile energy efficiency, energy conservation and emission reduction are the current research hotspots. Among them, hydrogen fuel cells, as an emerging energy generation technology, are applied to automobiles, with many advantages such as zero emissions, no pollution, low noise, high energy efficiency, renewable energy utilization and fast fuel filling. Therefore, this technical route has broad prospects in the field of new energy vehicles, and it is also one of the feasible solutions to alleviate the problems of non-renewable energy consumption and pollutant gas emissions.

[0003] At present, the corresponding technology of hydrogen fuel cell vehicles in China still needs to be improved, and there are still some technical gaps in energy management solutions. Energy management (EMS) is one of the core technologies of hydrogen fuel cell vehicles, which has a great impact on the fuel economy, fuel cell life, power, and comfort of the whole vehicle. It is also a research difficulty and hot spot in the field of fuel cell vehicles. Fuel cell vehicles generally use all-electric power systems, but if only fuel cells provide power for the whole vehicle, there will be some problems, such as time-delay response, inability to recover braking energy, and thus inability to meet power requirements in time and energy cannot be recovered. In the current fuel cell vehicle power system architecture, batteries or supercapacitors are added as auxiliary power sources to cope with sudden power demands and perform braking energy recovery at the same time. The power system also changes from a single power source to multiple power sources, so a more efficient energy management strategy is needed to allocate the required power to each energy source to ensure the economy and power of the whole vehicle and improve the performance of the whole vehicle.

[0004] If the three factors of equivalent hydrogen consumption, SOC offset, and fuel cell life need to be considered, the rule-based strategy needs to rely on expert experience for calibration and cannot achieve global optimization. The energy management strategy based on optimization has poor real-time performance and has limitations for multi-objective optimization problems. A new strategy is needed to meet the multi-objective, high-real-time, and optimal energy management needs.

[0005] Because fuel cells are used in trucks and need to face complex working conditions such as frequent switching between urban roads, rural roads, and highways, a single energy management algorithm cannot meet the complex working conditions well.

[0006] There are many existing energy management strategies, but they all focus on optimizing the algorithm itself, and fail to give full play to the advantages of combining vehicle speed condition prediction and energy management. If the future vehicle speed is taken into account in the energy management algorithm, and the bidirectional long short-term memory network Bi-LSTM is used to simultaneously consider the driving state information before and after, the vehicle will be forward-looking during driving, and the accuracy of condition identification can be effectively improved for the situation where there is a drastic change in the vehicle speed condition before and after, thereby improving energy efficiency. Therefore, the present invention provides a fuel cell vehicle reinforcement learning energy management method considering condition prediction. Summary of the invention

[0007] The present invention provides a fuel cell vehicle reinforcement learning energy management method taking into account operating condition prediction, which solves the technical problems mentioned in the above background technology.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A fuel cell vehicle reinforcement learning energy management method considering operating condition prediction, the method comprising the following steps;

[0010] S01. Four different types of classic driving conditions are used to construct comprehensive driving condition data that can reflect the characteristics of various road conditions. The comprehensive driving condition data is divided into segments of 100 seconds each. The principal component analysis (PCA) is performed by extracting the condition characteristic parameters of the conditions. The driving cycle segments are clustered by the K-means clustering method and clustered into three types: low-speed, medium-speed, and high-speed conditions.

[0011] S02, after obtaining three different types of driving cycle segments, offline training is performed on the Bi-LSTM-based working condition recognition module and the Markov-based vehicle speed prediction module;

[0012] S03. Extract the working condition characteristic parameters of the vehicle speed prediction sequence, such as the average vehicle speed and the vehicle speed standard deviation, and convert them into equivalent factor adjustment coefficients , and As a state input to the DDPG energy management algorithm, the fuel cell power P fc As actions, a suitable reward function is set, including hydrogen consumption, SOC fluctuation, and fuel cell degradation, to achieve a comprehensive consideration of the optimization problem of achieving multiple objectives.

[0013] As a further technical solution of the present invention, the step of using four different types of classic driving conditions to construct comprehensive driving condition data that can reflect the characteristics of various road conditions includes:

[0014] Condition data preprocessing: Four typical road conditions, NEDC, WLTC, CLTC-P, and FTP75, are combined as a comprehensive driving condition data that can reflect the characteristics of various road conditions, which is used for subsequent training of condition recognition and prediction algorithms;

[0015] NEDC (New European Driving Cycle) is the European endurance standard test condition, which includes 4 urban cycles and 1 suburban cycle; the urban cycle lasts 780 seconds with a maximum speed of 50 km / h; the suburban cycle lasts 400 seconds with a maximum speed of 120 km / h;

[0016] The WLTC standard working condition is closer to the actual road driving conditions. Its complete test cycle consists of four stages: low speed, medium speed, high speed and ultra-high speed, which lasts a total of 1800s, including 235s of idling time, 23266m of travel, an average speed of 46.5km / h and a maximum speed of 131.3km / h.

[0017] CLTC-P (China light-duty vehicle test cycle-passenger car) passenger car test conditions, including three speed ranges: low speed, medium speed and high speed, with a total duration of 1800s, a total mileage of 14480m, a maximum speed of 114km / h and an average speed of 28.96km / h;

[0018] FTP75 (Federal Test Procedure) is a standard issued by the U.S. Energy Administration for testing the economy and emissions of passenger cars in urban conditions. It is used to evaluate the emissions and fuel economy of light cars and light trucks. The complete FTP75 operating cycle takes 1874 seconds, with a theoretical driving distance of 17.77 km, an average speed of 34.12 km / h, and a maximum speed of 91.25 km / h. It includes three parts: cold start transient stage, steady state stage, and hot start transient stage.

[0019] As a further technical solution of the present invention, the step of dividing the comprehensive driving condition into segments of 100 seconds each and performing principal component analysis (PCA) by extracting characteristic parameters of the condition includes:

[0020] The comprehensive driving condition data is set to 100s per segment; and the condition characteristic parameters can reflect the characteristic information of each driving cycle segment; we use 14 parameters such as average speed, maximum speed, average acceleration, maximum acceleration, average deceleration, maximum deceleration, idle time ratio, acceleration time ratio, deceleration time ratio, uniform speed time ratio, speed standard deviation, acceleration standard deviation, mileage, and running time to describe the characteristics of each driving cycle segment;

[0021] However, there are too many characteristic parameters of the 14 working conditions, and there are information overlaps and correlations between these characteristic parameters, which makes the large amount of calculation unfavorable for analysis. In order to reduce the complexity of the analysis, principal component analysis (PCA) is used for dimensionality reduction;

[0022] For 100s of comprehensive driving condition data with 14 condition characteristic parameters, the dimension is reduced to 3 dimensions. The steps of principal component analysis for dimension reduction processing include:

[0023] 1) Organize the original data into a 100-row 14-column matrix X;

[0024] 2) Zero-mean each row of X, that is, subtract the mean of this row;

[0025] 3) Find the covariance matrix ;

[0026] 4) Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0027] 5) Arrange the eigenvectors into a matrix by row from top to bottom according to the corresponding eigenvalues, and take the first 3 rows to form the matrix P;

[0028] 6) That is, the data after dimension reduction to 3 dimensions;

[0029] In this process, in order to ensure the integrity of the information after dimensionality reduction, the selected principal components must satisfy the cumulative contribution rate of more than 80%. We finally selected the first three principal components with the largest contribution rate, and the cumulative contribution rate reached more than 80%, which is enough to cover most of the working condition characteristic information.

[0030] As a further technical solution of the present invention, the step of clustering the driving cycle segments by using the K-means clustering method comprises:

[0031] The driving cycle segments were classified using the K-means clustering algorithm. In this process, we set the number of categories K=3 and assigned an initial cluster center point to each category by calculating the Euclidean distance from each sample point to the cluster center. ;

[0032] Each point is divided into each category according to the minimum distance principle. After the first clustering is completed, for each cluster, its cluster center is recalculated. The new cluster center is the mean of all data points in the cluster. The calculation formula is as follows:

[0033] ;

[0034] in, It is A set of data points that are clustered, is the number of data points in the set. The above formula updates and redistributes each cluster point until the cluster center tends to be stable.

[0035] Finally, after PCA was used to reduce the dimension of the operating characteristic parameters of the operating condition data to obtain the operating condition data containing three principal component information, the 100 road driving cycle segments were divided into three categories through the K-means clustering algorithm, which were used to train the Markov transfer matrix for vehicle speed condition prediction in the following text.

[0036] As a further technical solution of the present invention, after obtaining the three different types of driving cycle segment sets, the step of offline training the Bi-LSTM-based working condition recognition module and the Markov-based vehicle speed prediction module includes:

[0037] Bi-LSTM consists of forward LSTM and backward LSTM. Forward LSTM processes the input sequence in chronological order to capture information from the past to the future, while backward LSTM processes the input sequence from the latest time point to the earliest time point to capture information from the future to the past. Each LSTM in this bidirectional structure contains an input gate, a forget gate, an output gate, and a unit state. The input of the model is 14 operating condition feature parameters, and the output is the operating mode, which is divided into three types: low speed, medium speed, and high speed. The architecture of LSTM is:

[0038] Forget gate: The forget gate determines which information is forgotten. Its expression is:

[0039] ;

[0040] in, is the sigmoid function, is the weight coefficient of the forget gate, Represents the hidden state at the previous moment and the current input state splicing, where There are 14 working condition characteristic parameters, including the real-time speed and acceleration of the vehicle, is the bias term;

[0041] Input gate: The input gate determines what new information needs to be added to the cell state;

[0042] ;

[0043] ;

[0044] in, and is the weight matrix of the input gate, and the dimension is adjusted according to the number of input parameters. and is the bias term, is the candidate cell state;

[0045] Unit state update: Update the cell state by combining the results of the forget gate and the input gate;

[0046] ;

[0047] Output gate: determines what value to output based on the updated cell state;

[0048] ;

[0049] ;

[0050] in, is the weight parameter of the output gate, is the bias term, is the output state at the current moment;

[0051] Bi-LSTM calculates bidirectional hidden states by running the LSTM layer forward and backward along the time axis. The forward LSTM calculates sequentially from the first element to the last element of the sequence, while the backward LSTM does the opposite. The two hidden states are concatenated together to form the final bidirectional hidden state. The bidirectional LSTM captures the contextual information before and after each time step in the sequence, thereby providing a more comprehensive feature representation.

[0052] In the vehicle speed condition prediction module, the Markov vehicle speed prediction method is used to identify three working conditions: low speed, medium speed, and high speed. Each mode corresponds to a Markov transfer matrix. Then, the three driving cycle fragments clustered in step S01 are used to train three Markov matrices (low speed Markov transfer matrix, medium speed Markov transfer matrix, and high speed Markov transfer matrix) offline in a targeted manner. The prediction step size is set to n, and Markov outputs the predicted vehicle speed information of n steps.

[0053] After completing the vehicle speed prediction considering the working condition identification, a p-step predicted vehicle speed sequence will be output.

[0054] As a further technical solution of the present invention, in step S03, the DDPG algorithm is based on the Actor-Critic framework, so the algorithm contains Actor and Critic networks, and each network has its corresponding target network, so the DDPG algorithm includes four networks, namely Actor network , Critic Network , Target Actor Network and Target Critic Network , the update process of the DDPG algorithm, the update method of the target network and the purpose of introducing the target network;

[0055] The DDPG training process includes:

[0056] Initialize the critic network and actor networks Parameters and ;

[0057] Initialize the corresponding target network parameters, , ;

[0058] Initialize the experience playback D, set the capacity to K, and set the number of loop rounds to M;

[0059] According to the current status , input to the current network, according to the current network strategy and noise Select the action, , generating action Afterwards, get rewarded , and enter the next state , and store state-action pairs Go to Experience Replay D;

[0060] Take a random batch from D , target value ;

[0061] in is the value function at the next moment, is the next moment strategy function, , is the network weight parameter at the next moment;

[0062] Update the critic parameters, and its loss is: ;

[0063] Update the actor network: ;

[0064] Soft update target network: .

[0065] As a further technical solution of the present invention, in step S03, a DDPG deep reinforcement learning energy management strategy integrating working condition identification and vehicle speed condition prediction is used;

[0066] Set the state collection:

[0067] ;

[0068] The power required for the vehicle, is the power battery status, FOH is the fuel cell life;

[0069] Set action set:

[0070] ;

[0071] is the fuel cell power;

[0072] Set the reward function:

[0073] ;

[0074] R is the reward function, is the fuel cell hydrogen consumption rate; λ is the equivalent factor adjustment coefficient calculated by the p-step predicted speed sequence output by the speed prediction part, , in the training process, the optimal energy management strategy is achieved by selecting the appropriate k value; after the training is completed, in the actual operation process, λ is adjusted in real time according to the specific predicted vehicle speed sequence it receives; To predict the standard deviation of the speed series, To predict the average speed of the speed sequence; is the output power of the power battery; LHV is the lower heating value of hydrogen; is the fuel cell life degradation value, is the weight factor to consider the fuel cell lifetime degradation;

[0075] In the process of implementing DDPG-based energy management, the environment first inputs the state into the agent's Actor. The agent selects the corresponding action, obtains real-time rewards, and then transitions to the state S' at the next moment. The experience samples are stored in the sample pool. When the number of samples in the sample pool reaches the preset number, the earliest experience is removed. The DDPG algorithm continuously allows the agent to perform actions, collect data on interactions with the environment, including states and rewards, and update network parameters, so that the agent gradually learns the optimal action strategy and achieves optimal energy efficiency optimization.

[0076] The beneficial effects of the present invention are as follows: compared with the energy management algorithm based on rules and optimization, the present invention adopts an energy management strategy based on the deep deterministic policy gradient DDPG algorithm. Rule-based algorithms usually rely on pre-set fixed rules and the experience of engineers, while the optimization objectives and constraints of the optimization algorithm are relatively fixed, and the real-time performance is relatively poor. The DDPG algorithm can adaptively adjust the strategy through continuous interaction and learning with the environment. In addition, the energy management process involves many interrelated and coupled factors and uncertainties, while the DDPG algorithm can handle high-dimensional state space and continuous action space problems, and continuously control the output power of fuel cells, and can effectively adapt to the energy management scenario of fuel cell vehicles. In addition, the energy management strategy based on optimization may be limited by complexity when solving the optimal solution, and sometimes only a local optimal solution can be obtained. In addition, the optimization-based algorithm requires an accurate mathematical model, while the DDPG-based algorithm can learn and optimize without understanding the system model.

[0077] At the same time, compared with the traditional DDPG-based algorithm, the present invention adds a fusion of bi-LSTM-based working condition identification and Markov vehicle speed prediction modules, so that it can adapt to complex road conditions. And the bi-LSTM algorithm used is composed of two LSTMs compared to the traditional LSTM classification algorithm, which can capture bidirectional information at the same time, so that the prediction process can consider past and future vehicle speed information at the same time. For example, when identifying urban working conditions with frequent starts and stops, bi-LSTM can take into account the previous start-stop conditions and subsequent possible change trends at the same time, so as to more accurately identify this type of working condition, which can effectively improve the recognition accuracy and reduce information omissions. After the working condition identification model based on bi-LSTM identifies the working condition type, the three Markov state transfer matrices trained with low-speed, medium-speed and high-speed data can better predict the vehicle speed according to different road conditions, improve the speed prediction accuracy, and thus improve the energy efficiency of the energy management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0079] Figure 1 A schematic diagram of a fuel cell vehicle power system provided by an embodiment of the present invention is shown.

[0080] Figure 2 A diagram of a reinforcement learning energy management strategy for integrating operating condition identification and prediction provided by an implementation of the present invention is shown.

[0081] Figure 3 The Bi-LSTM architecture diagram provided by an embodiment of the present invention is shown.

[0082] Figure 4 The energy management architecture diagram of a DDPG fuel cell vehicle provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] 1.1 Fuel Cell Vehicle Power System Architecture:

[0085] The research object of the present invention is a fuel cell vehicle, which is powered by a power battery and a fuel cell and driven by a drive motor. The fuel cell system is connected to the main circuit after being stabilized by a DC / DC converter, and the power battery is directly connected to the main circuit. After the fuel cell power and the power battery power are output, the drive motor is controlled by the motor controller to drive the vehicle. Figure 1 As shown:

[0086] 1.2 Longitudinal dynamics model:

[0087] The present invention is about the energy management of fuel cell vehicles. The lateral stability is not considered yet. The longitudinal dynamics of the vehicle need to be analyzed and modeled. The vehicle will be subject to four kinds of resistance during driving: air resistance, slope resistance, rolling resistance and acceleration resistance.

[0088] Vehicle driving force It can be expressed as:

[0089] ;

[0090] in, is the air resistance, is the slope resistance, is the rolling resistance, is the acceleration resistance; its calculation equations are:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] in, For the quality of fuel cell vehicles; is the acceleration due to gravity; Road slope; is the air density; is the frontal area of ​​the vehicle; is the air resistance coefficient; is the speed of the vehicle; is the rolling resistance coefficient; is the rotational mass conversion factor.

[0096] 1.3 Lithium battery model:

[0097] The lithium battery model adopts the first-order Rint model, which treats the lithium battery as an ideal voltage source and a structure connected in series with an internal resistor, while ignoring the polarization phenomenon of the battery. The structure is simple. In this model, the output power of the power battery for:

[0098] ;

[0099] Battery Current for:

[0100] ;

[0101] The battery state of charge SOC is updated using the ampere-hour integration method:

[0102] ;

[0103] in, is the open circuit voltage; is the current; is the internal resistance of the battery.

[0104] 1.4 Fuel cell model:

[0105] In the process of constructing the fuel cell hydrogen consumption model, the fuel cell hydrogen consumption model is expressed as:

[0106] ;

[0107] In the above formula, is the fuel cell system power, is the fuel cell system efficiency, The lower heating value of hydrogen .

[0108] 1.5 Fuel cell degradation model:

[0109] Considering fuel cell degradation, four main degradation conditions are considered to evaluate the degradation degree of the fuel cell stack. Degradation mechanism under dynamic cycle, degradation mechanism under start / stop cycle, degradation mechanism under idling condition, and degradation mechanism under high power load. It is expressed by the formula:

[0110] ;

[0111] Degradation mechanism under start / stop cycle:

[0112] ;

[0113] Degradation mechanism under dynamic cycle:

[0114] ;

[0115] Degradation mechanisms under idling conditions:

[0116] ;

[0117] Degradation mechanisms under high power loads:

[0118] ;

[0119] Among them, a, b, c, and d are constants, which are empirical parameters obtained through calibration in the experiment. After normalization, the remaining life of the fuel cell is expressed as:

[0120] ;

[0121] See also Figure 2 , as an embodiment of the present invention, a fuel cell vehicle reinforcement learning energy management method considering operating condition prediction is provided, and the method comprises the following steps;

[0122] S01. Four different types of classic driving conditions are used to construct comprehensive driving condition data that can reflect the characteristics of various road conditions. The comprehensive driving condition data is divided into segments of 100 seconds each. The principal component analysis (PCA) is performed by extracting the condition characteristic parameters of the conditions. The driving cycle segments are clustered by the K-means clustering method and clustered into three types: low-speed, medium-speed, and high-speed conditions.

[0123] S02, after obtaining three different types of driving cycle segments, offline training is performed on the Bi-LSTM-based working condition recognition module and the Markov-based vehicle speed prediction module;

[0124] S03. Extract the working condition characteristic parameters of the vehicle speed prediction sequence, such as the average vehicle speed and the vehicle speed standard deviation, and convert them into equivalent factor adjustment coefficients , and As a state input to the DDPG energy management algorithm, the fuel cell power P fc As actions, a suitable reward function is set, including hydrogen consumption, SOC fluctuation, and fuel cell degradation, to achieve a comprehensive consideration of the optimization problem of achieving multiple objectives.

[0125] In this embodiment, the step of using four different types of classic driving conditions to construct comprehensive driving condition data that can reflect various road condition characteristics includes:

[0126] Preprocessing of working condition data: The driving condition of the car is obtained by post-processing the actual driving data, which reflects the trend of time changing with speed. First, in order to fully reflect the road driving conditions and improve the accuracy of identifying all road conditions, we need to construct the driving cycle of the sample, combining the four typical road conditions of NEDC, WLTC, CLTC-P, and FTP75 as a comprehensive driving condition three-dimensional data that can reflect the characteristics of various road conditions, which is used for subsequent training of the algorithm for condition identification and prediction;

[0127] NEDC (New European Driving Cycle) is the European endurance standard test condition, which includes 4 urban cycles and 1 suburban cycle; the urban cycle lasts 780 seconds with a maximum speed of 50 km / h; the suburban cycle lasts 400 seconds with a maximum speed of 120 km / h;

[0128] The WLTC standard working condition is closer to the actual road driving conditions. Its complete test cycle consists of four stages: low speed, medium speed, high speed and ultra-high speed, which lasts a total of 1800s, including 235s of idling time, 23266m of travel, an average speed of 46.5km / h and a maximum speed of 131.3km / h.

[0129] CLTC-P (China light-duty vehicle test cycle-passenger car) passenger car test conditions, including three speed ranges: low speed, medium speed and high speed, with a total duration of 1800s, a total mileage of 14480m, a maximum speed of 114km / h and an average speed of 28.96km / h;

[0130] FTP75 (Federal Test Procedure) is a standard issued by the U.S. Energy Administration for testing the economy and emissions of passenger cars in urban conditions. It is used to evaluate the emissions and fuel economy of light cars and light trucks. The complete FTP75 operating cycle takes 1874 seconds, with a theoretical driving distance of 17.77 km, an average speed of 34.12 km / h, and a maximum speed of 91.25 km / h. It includes three parts: cold start transient stage, steady state stage, and hot start transient stage.

[0131] In this embodiment, the step of dividing the comprehensive driving condition into segments of 100 seconds each and performing principal component analysis (PCA) by extracting the characteristic parameters of the driving condition includes:

[0132] The comprehensive driving condition data constructed by four typical road conditions cannot be directly applied to the training of the condition recognition model and the Markov transfer matrix for vehicle speed prediction. It needs to be processed in segments, and the setting can be assumed to be 100s per segment; and the condition characteristic parameters can reflect the characteristic information of each driving cycle segment; we use 14 parameters such as average speed, maximum speed, average acceleration, maximum acceleration, average deceleration, maximum deceleration, idle time ratio, acceleration time ratio, deceleration time ratio, uniform speed time ratio, speed standard deviation, acceleration standard deviation, mileage, and running time to describe the characteristics of each driving cycle segment;

[0133] However, there are too many characteristic parameters of the 14 working conditions, and there are information overlaps and correlations between these characteristic parameters, which makes the large amount of calculation unfavorable for analysis. In order to reduce the complexity of the analysis, principal component analysis (PCA) is used for dimensionality reduction;

[0134] For 100s of comprehensive driving condition data with 14 condition characteristic parameters, the dimension is reduced to 3 dimensions. The steps of principal component analysis for dimension reduction processing include:

[0135] 1) Organize the original data into a 100-row 14-column matrix X;

[0136] 2) Zero-mean each row of X, that is, subtract the mean of this row;

[0137] 3) Find the covariance matrix ;

[0138] 4) Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0139] 5) Arrange the eigenvectors into a matrix by row from top to bottom according to the corresponding eigenvalues, and take the first 3 rows to form the matrix P;

[0140] 6) That is, the data after dimension reduction to 3 dimensions;

[0141] In this process, in order to ensure the integrity of the information after dimensionality reduction, the selected principal components must satisfy the cumulative contribution rate of more than 80%. We finally selected the first three principal components with the largest contribution rate, and the cumulative contribution rate reached more than 80%, which is enough to cover most of the working condition characteristic information.

[0142] In this embodiment, the step of clustering the driving cycle segments by using the K-means clustering method includes:

[0143] Assuming that the comprehensive working condition is 10,000 seconds, it can be divided into 100 driving cycle segments. The 100 driving cycle segments after PCA dimensionality reduction need to be divided into three categories: low-speed, medium-speed, and high-speed working conditions, so as to train the three types of Markov transfer matrices in the vehicle speed prediction module. Use the K-means clustering algorithm to classify the driving cycle segments. In this process, we determine the number of classifications K=3, and specify an initial cluster center point for each category, by calculating the Euclidean distance from each sample point to the cluster center;

[0144] ;

[0145] Each point is divided into each category according to the minimum distance principle. After the first clustering is completed, for each cluster, its cluster center is recalculated. The new cluster center is the mean of all data points in the cluster. The calculation formula is as follows:

[0146] ;

[0147] in, It is A set of data points that are clustered, is the number of data points in the set. The above formula updates and redistributes each cluster point until the cluster center tends to be stable.

[0148] Finally, after PCA was used to reduce the dimension of the operating characteristic parameters of the operating condition data to obtain the operating condition data containing three principal component information, the 100 road driving cycle segments were divided into three categories through the K-means clustering algorithm, which were used to train the Markov transfer matrix for vehicle speed condition prediction in the following text.

[0149] In this embodiment, after obtaining the three different types of driving cycle segment sets, the step of offline training the Bi-LSTM-based working condition recognition module and the Markov-based vehicle speed prediction module includes:

[0150] Considering the characteristics of complex road conditions, adding a condition recognition module before the speed prediction module can effectively improve the speed prediction accuracy. The LSTM network itself is designed to solve the gradient vanishing or gradient exploding problems that are prone to occur in traditional RNN (recurrent neural network) when processing long sequences. The road condition recognition model of the present invention adopts a method based on a bidirectional long short-term memory network Bi-LSTM. The Bi-LSTM architecture is as follows Figure 3 As shown in the figure, Bi-LSTM consists of forward LSTM and backward LSTM. The forward LSTM processes the input sequence in chronological order and captures information from the past to the future, while the backward LSTM processes the input sequence from the latest time point to the earliest time point and captures information from the future to the past. This means that the model can not only predict the current vehicle speed condition based on the past driving status, but also refer to the future driving status information to more comprehensively capture the changing trends of parameters such as vehicle speed and acceleration, and adapt to some sudden changes in road conditions, so as to make more accurate judgments. Each LSTM of this bidirectional structure contains an input gate, a forget gate, an output gate, and a unit state. The input of the model is 14 working condition feature parameters, and the output is a working condition mode, which is divided into three types: low speed, medium speed, and high speed. Compared with the traditional LSTM, the prediction effect is more accurate and more adaptable to the recognition of complex road conditions. The architecture of LSTM is:

[0151] Forget gate: The forget gate determines which information is forgotten. Its expression is:

[0152] ;

[0153] in, is the sigmoid function, is the weight coefficient of the forget gate, Represents the hidden state at the previous moment and the current input state splicing, where There are 14 working condition characteristic parameters, including the real-time speed and acceleration of the vehicle, is the bias term;

[0154] Input gate: The input gate determines what new information needs to be added to the cell state;

[0155] ;

[0156] ;

[0157] in, and is the weight matrix of the input gate, and the dimension is adjusted according to the number of input parameters. and is the bias term, is the candidate cell state;

[0158] Unit state update: Update the cell state by combining the results of the forget gate and the input gate;

[0159] ;

[0160] Output gate: determines what value to output based on the updated cell state;

[0161] ;

[0162] ;

[0163] in, is the weight parameter of the output gate, is the bias term, is the output state at the current moment;

[0164] Bi-LSTM calculates bidirectional hidden states by running the LSTM layer forward and backward along the time axis. The forward LSTM calculates sequentially from the first element to the last element of the sequence, while the backward LSTM does the opposite. The two hidden states are concatenated together to form the final bidirectional hidden state. The bidirectional LSTM captures the contextual information before and after each time step in the sequence, thereby providing a more comprehensive feature representation.

[0165] In the vehicle speed condition prediction module, the Markov vehicle speed prediction method is used to identify three working conditions: low speed, medium speed, and high speed. Each mode corresponds to a Markov transfer matrix. Then, the three driving cycle fragments clustered in step S01 are used to train three Markov matrices (low speed Markov transfer matrix, medium speed Markov transfer matrix, and high speed Markov transfer matrix) offline in a targeted manner. The prediction step size is set to n, and Markov outputs the predicted vehicle speed information of n steps.

[0166] After completing the vehicle speed prediction considering the working condition identification, a p-step predicted vehicle speed sequence will be output.

[0167] See also Figure 4 In this embodiment, in step S03, the DDPG algorithm is based on the Actor-Critic framework, so the algorithm contains Actor and Critic networks, and each network has its corresponding target network, so the DDPG algorithm includes four networks, namely Actor network , Critic Network , Target Actor Network and Target Critic Network , the update process of the DDPG algorithm, the update method of the target network and the purpose of introducing the target network;

[0168] The DDPG training process includes:

[0169] Initialize the critic network and actor networks Parameters and ;

[0170] Initialize the corresponding target network parameters, , ;

[0171] Initialize the experience playback D, set the capacity to K, and set the number of loop rounds to M;

[0172] According to the current status , input to the current network, according to the current network strategy and noise Select the action, , generating action Afterwards, get rewarded , and enter the next state , and store state-action pairs Go to Experience Replay D;

[0173] Take a random batch from D , target value ;

[0174] in is the value function at the next moment, is the next moment strategy function, , is the network weight parameter at the next moment;

[0175] Update the critic parameters, and its loss is: ;

[0176] Update the actor network: ;

[0177] Soft update target network: .

[0178] In this embodiment, in step S03, a DDPG deep reinforcement learning energy management strategy is integrated with operating condition identification and vehicle speed condition prediction;

[0179] Set the state collection:

[0180] ;

[0181] The power required for the vehicle, is the power battery status, FOH is the fuel cell life;

[0182] Set action set:

[0183] ;

[0184] is the fuel cell power; compared with the energy management algorithm based on DQN and Q-learning to optimize its Q table, the action space of the energy management algorithm based on DDPG is continuous, which can better meet the actual fuel cell power output requirements;

[0185] Set the reward function:

[0186] ;

[0187] R is the reward function, is the fuel cell hydrogen consumption rate; λ is the equivalent factor adjustment coefficient calculated by the p-step predicted speed sequence output by the speed prediction part, , in the training process, the optimal energy management strategy is achieved by selecting the appropriate k value; after the training is completed, in the actual operation process, λ is adjusted in real time according to the specific predicted vehicle speed sequence it receives; To predict the standard deviation of the speed series, To predict the average speed of the speed sequence; is the output power of the power battery; LHV is the lower heating value of hydrogen; is the fuel cell life degradation value, is the weight factor to consider the fuel cell lifetime degradation;

[0188] In the process of implementing DDPG-based energy management, the environment first inputs the state into the agent's Actor. The agent selects the corresponding action, obtains real-time rewards, and then transitions to the state S' at the next moment. The experience samples are stored in the sample pool. When the number of samples in the sample pool reaches the preset number, the earliest experience is removed. The DDPG algorithm continuously allows the agent to perform actions, collect data on interactions with the environment, including states and rewards, and update network parameters, so that the agent gradually learns the optimal action strategy and achieves optimal energy efficiency optimization.

[0189] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0190] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A fuel cell vehicle reinforcement learning energy management method considering operating condition prediction, characterized in that: The method The steps include: S01. Four different types of classic driving conditions are used to construct comprehensive driving condition data that can reflect the characteristics of various road conditions. The comprehensive driving condition data is divided into segments of 100 seconds each. The principal component analysis is performed by extracting the characteristic parameters of the driving conditions. The driving cycle segments are clustered by the K-means clustering method and clustered into three types: low-speed, medium-speed, and high-speed conditions. S02, after obtaining three different types of driving cycle segments, offline training is performed on the Bi-LSTM-based working condition recognition module and the Markov-based vehicle speed prediction module; S03, extract the working condition characteristic parameters of the vehicle speed prediction sequence, convert them into the equivalent factor adjustment coefficient λ, and input λ as a state into the energy management algorithm of DDPG, the fuel cell power P fc As actions, and setting appropriate reward functions, including hydrogen consumption, SOC fluctuation and fuel cell degradation, to achieve a comprehensive consideration of the optimization problem of achieving multiple objectives; The step of clustering the driving cycle segments by using the K-means clustering method comprises: Use K-means clustering algorithm to classify driving cycle segments, determine the number of categories K = 3, and assign an initial cluster center point to each category, by calculating the Euclidean distance from each sample point to the cluster center; Each point is divided into each category according to the minimum distance principle. After the first clustering is completed, for each cluster, its cluster center is recalculated. The new cluster center is the mean of all data points in the cluster. The calculation formula is as follows: Among them, S i is the set of data points of the ith cluster, |S i | is the number of data points in the set. The above formula updates and redistributes each cluster point until the cluster center tends to be stable; After obtaining the three different types of driving cycle segment sets, the steps of offline training the Bi-LSTM-based working condition recognition module and the Markov-based vehicle speed prediction module include: Bi-LSTM consists of forward LSTM and backward LSTM. Forward LSTM processes the input sequence in chronological order and captures information from the past to the future, while backward LSTM processes the input sequence from the latest time point to the earliest time point and captures information from the future to the past. Each LSTM in this bidirectional structure contains an input gate, a forget gate, an output gate, and a unit state. The input of the model is 14 working condition feature parameters, and the output is the working condition mode, which is divided into three types: low speed, medium speed, and high speed. The architecture of LSTM is: Forget gate: The forget gate determines which information is forgotten. Its expression is: f t =σ(W f ·[h t-1 ,x t ]+b f ); Where σ is the sigmoid function, W f is the weight coefficient of the forget gate, [h t-1 ,x t ] represents the hidden state h at the previous moment t-1 and the current input state x t splicing, where x t are 14 working condition characteristic parameters, b f is the bias term; Input gate: The input gate determines what new information needs to be added to the cell state; i t =σ(W i ·[h t-1 ,x t ]+b i ); Among them, W i and W c is the weight matrix of the input gate, and the dimension is adjusted according to the number of input parameters. i and b c is the bias term, is the candidate cell state; Unit state update: Update the cell state by combining the results of the forget gate and the input gate; Output gate: determines what value to output based on the updated cell state; the t =σ(W o ·[h t-1 ,x t ]+b o ); h t =o t *tanh(C t ); Among them, W o is the weight parameter of the output gate, b o is the bias term, h t is the output state at the current moment; Bi-LSTM calculates bidirectional hidden states by running the LSTM layer forward and backward along the time axis. The forward LSTM calculates sequentially from the first element to the last element of the sequence, while the backward LSTM does the opposite. The two hidden states are concatenated together to form the final bidirectional hidden state. The bidirectional LSTM captures the contextual information before and after each time step in the sequence, thereby providing a more comprehensive feature representation. In the vehicle speed condition prediction module, the Markov vehicle speed prediction method is used to identify the three working conditions of low speed, medium speed and high speed. Each mode corresponds to a Markov transfer matrix. Then, the three driving cycle segments clustered in step S01 are used to train three Markov matrices offline in a targeted manner. The prediction step length is set to n, and Markov outputs the predicted vehicle speed information of n steps. After completing the vehicle speed prediction considering the working condition identification, a p-step predicted vehicle speed sequence will be output.

2. A fuel cell vehicle reinforcement learning energy management method considering operating condition prediction according to claim 1, characterized in that: The steps of using four different types of classic driving conditions to construct comprehensive driving condition data that can reflect the characteristics of various road conditions include: The four typical road conditions of NEDC, WLTC, CLTC-P and FTP75 are combined as a comprehensive driving condition data that can reflect the characteristics of various road conditions, which is used for subsequent training of condition recognition and prediction algorithms; The NEDC cycle is the European endurance standard test cycle, which includes 4 urban cycles and 1 suburban cycle. The urban cycle lasts for 780 seconds with a maximum speed of 50 km / h; the suburban cycle lasts for 400 seconds with a maximum speed of 120 km / h. The WLTC standard working condition is closer to the actual road driving conditions. Its complete test cycle consists of four stages: low speed, medium speed, high speed and ultra-high speed, which lasts a total of 1800s, including 235s of idling time, 23266m of travel, an average speed of 46.5km / h and a maximum speed of 131.3km / h. CLTC-P passenger car test conditions include three speed ranges: low speed, medium speed and high speed, with a total duration of 1800s, a total mileage of 14480m, a maximum speed of 114km / h and an average speed of 28.96km / h; FTP75 is a standard issued by the U.S. Energy Administration for testing the economy and emissions of passenger cars in urban conditions. It is used to evaluate the emissions and fuel economy of light cars and light trucks. The complete FTP75 operating cycle takes 1874 seconds, with a theoretical driving distance of 17.77 km, an average speed of 34.12 km / h, and a maximum speed of 91.25 km / h. It includes three parts: cold start transient stage, steady state stage, and hot start transient stage.

3. A fuel cell vehicle reinforcement learning energy management method considering operating condition prediction according to claim 1, characterized in that: The step of dividing the comprehensive driving condition data into segments of 100 seconds each and performing principal component analysis by extracting characteristic parameters of the driving condition comprises: The comprehensive driving condition data is set to 100s per segment; the condition characteristic parameters reflect the characteristic information of each driving cycle segment; the characteristics of each driving cycle segment are described by 14 parameters including average speed, maximum speed, average acceleration, maximum acceleration, average deceleration, maximum deceleration, idle time ratio, acceleration time ratio, deceleration time ratio, uniform speed time ratio, speed standard deviation, acceleration standard deviation, mileage, and running time; For 100s of comprehensive driving condition data with 14 condition characteristic parameters, the dimension is reduced to 3 dimensions. The steps of principal component analysis for dimension reduction processing include: 1) Organize the original data into a 100-row 14-column matrix X; 2) Zero-mean each row of X, that is, subtract the mean of this row; 3) Find the covariance matrix 4) Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix; 5) Arrange the eigenvectors into a matrix by row from top to bottom according to the corresponding eigenvalues, and take the first 3 rows to form the matrix P; 6) Y = PX is the data after dimension reduction to 3 dimensions; In this process, in order to ensure the integrity of the information after dimensionality reduction, the selected principal components must satisfy the cumulative contribution rate of more than 80%. Finally, the top three principal components with the largest contribution rates are selected, and the cumulative contribution rate reaches more than 80%, which is sufficient to cover most of the operating condition characteristic information.

4. A fuel cell vehicle reinforcement learning energy management method considering operating condition prediction according to claim 1, characterized in that: In step S03, a DDPG deep reinforcement learning energy management strategy is integrated with operating condition recognition and vehicle speed condition prediction; Set the state collection: S={p m ,SOC,FOH}; P m is the vehicle's required power, SOC is the power battery state, and FOH is the fuel cell life; Set action set: A={P fc }; P fc is the fuel cell power; Set the reward function: R is the reward function, is the fuel cell hydrogen consumption rate; λ is the equivalent factor adjustment coefficient calculated by the p-step predicted speed sequence output by the speed prediction part, During the training process, the optimal energy management strategy is achieved by selecting a suitable k value. After the training is completed, during the actual operation, λ is adjusted in real time according to the specific predicted vehicle speed sequence it receives; where p std is the standard deviation of the predicted speed sequence, p ave is the average speed of the predicted speed sequence; p bat is the output power of the power battery; LHV is the lower heating value of hydrogen; D fc is the fuel cell life degradation value, α is the weight factor considering the fuel cell life degradation; In the process of implementing DDPG-based energy management, the environment first inputs the state into the agent's Actor. The agent selects the corresponding action, obtains real-time rewards, and then transitions to the state S' at the next moment. The experience samples are stored in the sample pool. When the number of samples in the sample pool reaches the preset number, the earliest experience is removed. The DDPG algorithm continuously allows the agent to perform actions, collect data on interactions with the environment, including states and rewards, and update network parameters, so that the agent gradually learns the optimal action strategy and achieves optimal energy efficiency optimization.

Images