Energy management method combining meta-learning and near-end strategy optimization algorithm

Abstract

The invention discloses an energy management method combining meta-learning and a near-end strategy optimization algorithm, and belongs to the technical field of automation. According to the method, gradient updating is carried out on a series of different lithium battery / super-capacitor hybrid vehicle tasks through internal circulation, and a set of parameters with high adaptability and generalization ability are obtained; and after the internal circulation is completed, the global parameters of the basic EMS controller are updated again by integrating the feedback of a plurality of tasks in the external circulation, so that the basic EMS controller can adapt to the characteristics of the new lithium battery / super-capacitor hybrid vehicle. Parameter fine tuning is carried out in the meta-test stage, and a new energy management strategy which aims at a new HEV control object and is excellent in control performance is rapidly generated. When facing a new hybrid energy system, the method provided by the invention can realize control performance similar to that of an energy management strategy based on traditional reinforcement learning only by carrying out a small number of gradient descent calculation steps, and provides a solution for developing a hybrid power system energy management strategy with high adaptability and high industrial efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of automation technology, and in particular relates to an energy management method combining meta-learning and proximal strategy optimization algorithms. Background Art

[0002] For new energy hybrid vehicles, a reasonable energy management strategy (EMS) helps save energy, reduce consumption and improve battery life. Common energy management strategy design algorithms, including fuzzy logic, dynamic programming (DP), model predictive control (MPC), reinforcement learning (RL), etc., can only face a single task, that is, after the algorithm training is completed, it can only be used to control a specific vehicle. If the energy management strategy is applied to other vehicles, even if there are only slight changes in the energy parameters, such as changes in lithium battery capacity and changes in the maximum output power constraint of the lithium battery, the entire control strategy needs to be completely retrained. The multiple training processes of these algorithms are independent of each other, and the experience gained from the previous training process cannot be accumulated. The repetitive and lengthy training process leads to inefficiency. In order to solve this problem, it is urgent to quickly adapt the meta-energy management strategy based on the meta-reinforcement learning algorithm to the new application scenario, so as to quickly generate a new hybrid vehicle energy management strategy. Summary of the invention

[0003] The purpose of the present invention is to solve the limitation that traditional energy management strategies cannot adapt to new application scenarios, the processes are independent of each other, the experience gained from previous training processes cannot be accumulated, and the repetitive and lengthy training process leads to low efficiency. The present invention provides an energy management method that combines meta-learning and proximal policy optimization algorithms. The method of the present invention uses a model-agnostic meta-learning algorithm (MAML) and a proximal policy optimization (PPO) reinforcement learning algorithm to design a meta-energy management strategy that can quickly adapt to new control objects. This strategy can achieve control performance with only a small number of gradient descent steps when facing new hybrid energy systems, thereby significantly improving the adaptability of the control system and the efficiency of industrial applications.

[0004] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0005] An energy management method combining meta-learning and proximal policy optimization algorithm, comprising the following steps:

[0006] S1. Build a basic EMS controller as a basic controller that can adapt to a variety of lithium battery and supercapacitor hybrid vehicles;

[0007] S2. In the meta-training task set preparation stage, a set of lithium battery and supercapacitor hybrid vehicle tasks of different batteries and supercapacitors are sampled according to the distribution to form a task set, and the rated capacity variation range of the actual hybrid vehicle lithium battery, the rated capacity variation range of the supercapacitor, the peak power variation range of the lithium battery and the peak power variation range of the supercapacitor are set according to the key variables of the lithium battery and supercapacitor hybrid vehicle. Within the set four variation ranges, several different parameter combinations are randomly selected, and each parameter combination is used to describe a lithium battery and supercapacitor hybrid vehicle;

[0008] S3. In the meta-training phase, a plurality of different lithium battery and supercapacitor hybrid vehicle tasks are randomly sampled from the task set to form a batch, and the basic EMS controller parameters are updated by the inner loop operation on each batch;

[0009] S4. After the inner loop operation is completed, the basic EMS controller parameters updated by the inner loop operation are used to re-update the global parameters of the basic EMS controller in the outer loop operation by integrating the feedback of multiple tasks, and the global parameters of the basic EMS controller updated by the outer loop operation are used as the final basic EMS controller parameters to obtain the trained basic EMS controller;

[0010] S5. In the meta-testing phase, a group of lithium battery and supercapacitor hybrid vehicle tasks with different batteries and supercapacitors are sampled according to the distribution to form a new task set, and test tasks are randomly selected from the new task set. A new parameter combination different from that in the meta-training phase is generated for each test task, and the trained basic EMS controller is fine-tuned on each new parameter combination for energy management. After fine-tuning, a new energy management strategy corresponding to each test task is formed.

[0011] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0012] Preferably, in step S1, a convolutional neural network reinforcement learning controller is constructed as the main controller of the lithium battery and supercapacitor hybrid vehicle, which includes an input layer, a first convolutional layer, a second convolutional layer, a fully connected layer and an output layer;

[0013] In the convolutional neural network reinforcement learning controller, the input layer is used to obtain input features, the first convolution layer performs shallow feature extraction on the input features to obtain first output features, the second convolution layer performs deep feature extraction on the first output features to obtain second output features, the second output features are flattened and input into the fully connected layer to obtain third output features, and the output layer outputs the third output features;

[0014] The input features include vehicle speed, acceleration, battery charge state, supercapacitor charge state, current required power, vehicle position, vehicle acceleration direction, vehicle steering angle, vehicle load state, and external temperature.

[0015] Preferably, in step S2, the key variables include the rated capacity of the lithium battery, the rated capacity of the supercapacitor, the peak value of the lithium battery and the peak power of the supercapacitor, and the equivalent resistance of the lithium battery is taken as a related dependent variable.

[0016] Preferably, the same reward function is set for the basic EMS controllers of different types of lithium battery and supercapacitor hybrid vehicles:

[0017] R t =ω1r1+ω2r2+ω3r3+ω4r4+ω5r5

[0018]

[0019] r2=BSOC t -BSOC0

[0020]

[0021] Among them, R t represents the reward function at the tth time node in the whole process; ω1, ω2, ω3, ω4, ω5 represent the weight parameters of each part of the reward function; r1, r2, r3, r4, r5 represent the feedback of different performance indicators in lithium battery and supercapacitor hybrid vehicles respectively; represents the total power demand of the vehicle at the tth time node; p t Indicates the actual output power provided by the lithium battery and supercapacitor at the tth time node; BSOC t Indicates the state of charge of the lithium battery at the tth time node; BSOC0 indicates the initial charge of the battery; Represents the output current of the lithium battery at the tth time node; Represents the output current of the lithium battery at the t-1th time node; Indicates the change of the state of charge of the supercapacitor at the tth time node; USOC t Indicates the state of charge of the supercapacitor at the tth time node; USOC min and USOC max They represent the upper and lower limits of the capacitor charge respectively.

[0022] Preferably, in step S3, the specific process of updating the basic EMS controller parameters by the inner loop operation is as follows:

[0023] S31. Randomly initialize the basic EMS controller parameters, including weights and biases of all neural network layers;

[0024] S32. Randomly select a batch from a preset task set, where each task in a batch represents a specific vehicle configuration and driving environment;

[0025] S33: For the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j , based on the current parameters θ, run the basic EMS controller T time steps to obtain the feedback data D of the environment ij t ,a t ,r t >, where s t ,a t ,r t Represent the state, action and reward of the t-th time step respectively;

[0026] S34: Calculating an advantage function at each time step based on the feedback data, where the advantage function is used to represent the relative performance of the current strategy at each time step;

[0027] Advantage function at the tth time step The calculation formula is:

[0028]

[0029] Among them, γ i-t represents the discount weight of the reward at the i-th time step relative to the t-th time step; r i represents the reward obtained at the i-th time step; V φ (s t ) is the state s at the t-th time step t The value function under ;

[0030] S35: Based on the advantage function and the feedback data, the maximization formula is used to perform gradient update, and the current parameter θ is updated to θ′ ij :

[0031]

[0032] Among them, |D ij | indicates feedback data D ij The size of f θ (a i |s i ) represents a given state s i When we execute action a based on strategy θ i probability; ​Represents a given state s i Based on the strategy θ ij Execute action a i probability; Indicates that in state s i Next, perform action a i Based on the current strategy The advantage function of ij Represents the i-th batch B i The parameters after gradient update for the jth lithium battery and supercapacitor hybrid vehicle task in; α represents the inner loop learning rate;

[0033] S36: Continuously iterate the gradient update to make the gradient updated parameter θ′ ij Able to gradually adapt to the needs of the current task.

[0034] Preferably, in step S4, the specific process of updating the basic EMS controller parameters by the outer loop operation is as follows:

[0035] S41: The input variables are the updated parameters of the gradients of each lithium battery and supercapacitor hybrid vehicle task in a batch, aggregating each lithium battery and supercapacitor hybrid vehicle task in a pre-built validation set The performance on the validation set is calculated by calculating the loss function L for each lithium battery and supercapacitor hybrid vehicle task on the validation set. ij (θ′ ij ):

[0036]

[0037] in, is the predicted value of the basic EMS controller, y i is the expected output; Represents the validation set size;

[0038] S42: Compute meta-gradients based on the loss function for each lithium battery and supercapacitor hybrid vehicle task Used to update global parameters

[0039]

[0040] in, Represents the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j The loss function of the global parameter The gradient of

[0041] S43: Use the meta-gradient descent method to update the global parameters as follows:

[0042]

[0043] Among them, β represents the outer loop learning rate;

[0044] S44: Use updated global parameters Evaluate the performance of the base EMS controller on the validation set of each task to ensure that the global update helps generalization rather than just adapting to a single task;

[0045] S45: Iteratively update global parameters to optimize the performance of the base EMS controller on all Li-ion and supercapacitor hybrid vehicle tasks in the entire batch.

[0046] Preferably, in step S5, the specific process of fine-tuning in the meta-testing phase is as follows:

[0047] S51: Global parameters obtained from meta-training As initial parameters, initialize the weights and biases of the basic EMS controller;

[0048] S52: sampling test data from the test data set, using the vehicle state parameter as an input feature of the basic EMS controller, and using the expected control output power distribution as a target output of the basic EMS controller;

[0049] S53: Fast adaptive fine-tuning of the basic EMS controller on the test task, using the initial parameters from S51 Perform gradient updates on the basic EMS controller to generate feedback data for the new environment Calculate the loss function using feedback data from the new environment The specific form is as follows:

[0050]

[0051] in, Represent the new state, new action and new reward of the t-th time step respectively; |D i |Represents feedback data of the new environment D i size; Indicates that in the new parameter Under the basic EMS controller The predicted value at time ;

[0052] S54: After the gradient has descended N times, the gradient is calculated based on the loss and the parameters of the basic EMS controller are updated as follows: So that it can select appropriate actions according to the current state and strategy and achieve efficient energy management:

[0053]

[0054] in, Represents the loss function Basic EMS controller parameters The gradient of

[0055] S55: With current parameters As the basic EMS controller parameters, run the fine-tuned basic EMS controller, execute T time steps, obtain the data of the running process, and calculate the performance indicators of the basic EMS controller on the test task.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The method of the present invention provides an efficient, fast and effective method for designing an energy management system for a lithium battery / supercapacitor hybrid electric vehicle. The method of the present invention can quickly adapt to new tasks and generate an energy management strategy (EMS) with good control performance through a combination of meta-learning and reinforcement learning. By recycling energy from supercapacitors and reducing fluctuations in lithium battery output current, the method of the present invention achieves reduced energy consumption and extended battery life. The successful application of the method of the present invention will bring important progress and breakthroughs to the field of hybrid electric vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the steps of the method of the present invention;

[0059] Figure 2 It is a schematic diagram of the overall framework of the method of the present invention;

[0060] Figure 3 Schematic diagram of inner loop and outer loop in the meta-training stage of the method of the present invention;

[0061] Figure 4 A comparison chart of the reward values ​​of the method of the present invention and the benchmark algorithm in the gradient descent stage;

[0062] Figure 5 This is a comparison diagram of lithium battery power consumption when the method of the present invention is iterated 5 times and other methods;

[0063] Figure 6 This is a comparison diagram of lithium battery power consumption when the method of the present invention is iterated 10 times and other methods;

[0064] Figure 7 This is a comparison chart of lithium battery power consumption when the method of the present invention is iterated 15 times and other methods;

[0065] Figure 8 This is a comparison diagram of the supercapacitor power recovery of the method of the present invention and other methods when the number of iterations is 5;

[0066] Fig. 9 This is a comparison diagram of the supercapacitor power recovery of the method of the present invention and other methods when the number of iterations is 10;

[0067] Fig.10 This is a comparison diagram of the supercapacitor power recovery of the method of the present invention and other methods when the number of iterations is 15;

[0068] Fig.11 This is a comparison diagram of the average instantaneous fluctuation of lithium battery output current when the method of the present invention is iterated 5 times and other methods;

[0069] Fig.12 This is a comparison diagram of the average instantaneous fluctuation of lithium battery output current when the method of the present invention is iterated 10 times and other methods;

[0070] Fig.13 This is a comparison diagram of the average instantaneous fluctuation of lithium battery output current of the method of the present invention and other methods when the number of iterations is 15. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0072] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0073] like Figure 1 As shown, in a preferred implementation of the present invention, the energy management method combining meta-learning and proximal strategy optimization algorithm includes the following steps S1 to S5. The specific implementation process is described in detail below.

[0074] S1. Build a basic EMS controller as a basic controller that can adapt to a variety of lithium battery and supercapacitor hybrid electric vehicles (HEV).

[0075] It should be noted that in step S1 of the present invention, a convolutional neural network reinforcement learning controller is constructed and used as the main controller of a lithium battery and supercapacitor hybrid vehicle (HEV) to process input data from various sensors of the vehicle and output corresponding control signals. Its main structure includes an input layer, a first convolutional layer, a second convolutional layer, a fully connected layer, and an output layer. Among them, the input layer is used to receive vehicle state parameters, the charge state (SoC) of the battery and supercapacitor; the first convolutional layer and the second convolutional layer use nonlinear activation functions (such as ReLU or Tanh) to capture the complex relationship between input data and output control commands; the output layer is designed as a linear or activation function layer to generate actual control commands and power allocation strategies for lithium batteries and supercapacitors.

[0076] In the convolutional neural network reinforcement learning controller, the input layer is used to obtain the input feature X, the first convolution layer performs shallow feature extraction on the input feature to obtain the first output feature C1, the second convolution layer performs deep feature extraction on the first output feature to obtain the second output feature C2, the second output feature is flattened and input into the fully connected layer to obtain the third output feature h1, and the output layer outputs the third output feature.

[0077] In this embodiment, the input feature dimension of the input layer is 10, and the input feature X is specifically as follows:

[0078] X=[v,a,SOC battery ,SOC capacitor ,P demand ,x,θ,φ,L,T ext ]

[0079] Where, v represents the vehicle speed; a represents the acceleration; SOC battery Indicates the battery charge state; SOC capacitor Indicates the charge status of the supercapacitor; P demand represents the current required power; x represents the vehicle position; θ represents the vehicle acceleration direction; φ represents the vehicle steering angle; L represents the vehicle load state; T ext Indicates the outside temperature.

[0080] The first convolutional layer uses 32 3×3 convolution kernels to extract features, and the output dimension is (32, H, W), where H and W are the height and width of the input feature respectively. Each input feature is convolved with 32 different convolution kernels to generate 32 output features. The convolution result of each convolution kernel is added with the corresponding bias term and the ReLU activation function is applied to the output, setting negative values ​​to zero and keeping positive values ​​unchanged:

[0081]

[0082] Among them, W1,k represents the kernel-th convolution kernel of the first convolution layer; b 1,k Represents the kernel bias term of the first convolutional layer.

[0083] The second convolutional layer uses 64 3×3 convolution kernels to extract features. The output dimension is (64, H, W). The 32 input features are convolved with 64 different convolution kernels to generate 64 output features. The convolution result of each convolution kernel is added with the corresponding bias term and the ReLU activation function is applied to the output:

[0084]

[0085] Among them, W 2,k represents the kernel-th convolution kernel of the second convolutional layer; b 2,k Represents the kernel bias term of the second convolutional layer.

[0086] The 64 output features are flattened and input into the fully connected layer, first multiplied by the weight matrix W3, then added to the bias term b3, and finally the Relu activation function generates 128 output features:

[0087] h1=ReLU(W3·flatten(C2)+b3)

[0088] Among them, flatten represents the flattening operation.

[0089] Multiply the 128 output features of the fully connected layer by the weight matrix W4 and add them to the bias term b4 to obtain the output features:

[0090] h2=W4·h1+b4

[0091] The two control signals in the output feature h2 represent the power distribution of the lithium battery and the power distribution of the supercapacitor respectively.

[0092] Finally, the output feature h2 is output by the output layer.

[0093] It should be noted that the energy management method (meta-EMS) combining meta-learning and proximal policy optimization algorithm of the present invention adopts the MAML algorithm in its core structure. By performing inner and outer loop training on a series of different lithium battery / supercapacitor hybrid vehicle tasks, the gradient is updated, and the parameters of the basic EMS controller are optimized so that this set of parameters has the best generalization ability. At the same time, the basic EMS controller is capable of adapting to the energy management of new lithium battery / supercapacitor hybrid vehicles, and finally a trained basic EMS controller is obtained. The method meta-EMS of the present invention adopts the proximal policy optimization (PPO) algorithm as the framework. The specific algorithm can be divided into three stages: meta-training task set preparation, meta-training and meta-testing, such as Figure 2 The specific details will be described later.

[0094] S2. In the meta-training task set preparation stage, a set of lithium battery and supercapacitor hybrid electric vehicle (HEV) tasks of different batteries and supercapacitors are sampled according to the distribution ε to form a task set H S ={H1,H2,…,H N}, and according to the key variables of the lithium battery and supercapacitor hybrid vehicle, the rated capacity variation range of the actual hybrid vehicle lithium battery, the rated capacity variation range of the supercapacitor, the peak power variation range of the lithium battery, and the peak power variation range of the supercapacitor are set. Within the set four variation ranges, several different parameter combinations are randomly selected, and each parameter combination is used to describe a lithium battery and supercapacitor hybrid vehicle. Among them, H1, H2, …, H N They represent the 1st, 2nd, …, Nth lithium battery and supercapacitor hybrid vehicle tasks in the task set respectively.

[0095] It should be noted that in step S2 of the present invention, the key variables include the rated capacity Q of the lithium battery. b 、Rated capacity of supercapacitor Q c , peak value of lithium battery and the peak power of the supercapacitor The equivalent resistance of the lithium battery is taken as a relevant dependent variable.

[0096] It should be noted that each lithium battery and supercapacitor hybrid electric vehicle (HEV) task in the task set in step S2 can be regarded as a task with a length of l. i The Markov decision process (MDP) is expressed as i ,A i ,R i ,γ i ,P i >, where is a set of states; Indicates the power demand of lithium battery and supercapacitor hybrid vehicles; Indicates the charge state of the lithium battery; Indicates the charge state of the supercapacitor; represents the finite set of actions for the i-th batch, which is determined by the output power of the i-th batch of lithium batteries and the output power of the supercapacitors in the i-th batch Composition; R i is the reward function for all tasks in the i-th batch; γ i is the discount factor for the i-th batch; P i is the state transition rate of the i-th batch.​

[0097] In this embodiment, the same reward function is set for the basic EMS controllers of different types of lithium battery and supercapacitor hybrid vehicles:

[0098] R t =ω1r1+ω2r2+ω3r3+ω4r4+ω5r5

[0099]

[0100] r2=BSOC t -BSOC0

[0101]

[0102] Among them, R t represents the reward function at the tth time node in the whole process; ω1, ω2, ω3, ω4, ω5 represent the weight parameters of each part of the reward function, which are 1, 10, 1 / 20, 1 and 1 respectively. The weight parameter setting is used to balance the difference in dimensions between each item and control the reward function value (R t ) is in a reasonable range to improve the performance of the algorithm. The weight parameter can also be further adjusted according to the importance of different goals. r1, r2, r3, r4, and r5 represent the feedback of different performance indicators in lithium battery and supercapacitor hybrid vehicles respectively; represents the total power demand of the vehicle at the tth time node; p t Indicates the actual output power provided by the lithium battery and supercapacitor at the tth time node; BSOC t Indicates the state of charge of the lithium battery at the tth time node; BSOC0 indicates the initial charge of the battery; Represents the output current of the lithium battery at the tth time node; Represents the output current of the lithium battery at the t-1th time node; Indicates the change of the state of charge of the supercapacitor at the tth time node; USOC t Indicates the state of charge of the supercapacitor at the tth time node; USOC min and USOC max They represent the upper and lower limits of the capacitor charge respectively.

[0103] It should be noted that each lithium battery and supercapacitor hybrid vehicle in step S2 has a unique parameter configuration to ensure the diversity and coverage of the task set to improve the generalization ability of the basic EMS controller. In this embodiment, 100 different parameter combinations are randomly selected. In addition to randomly selecting parameter combinations, various parameters used in the training process need to be set. For example, the learning rate α = 0.001, the batch size B = 64 for each iteration during the training process, and the number of training rounds N = 100.

[0104] S3. In the meta-training phase, multiple different lithium battery and supercapacitor hybrid vehicle tasks are randomly sampled from the task set to form a batch, and the basic EMS controller parameters are updated by the inner loop operation on each batch.

[0105] It should be noted that in step S3, in order to ensure the diversity and coverage of the task set and improve the generalization ability of the basic EMS controller, the embodiment of the present invention randomly selects 4 different lithium battery and supercapacitor hybrid vehicle tasks from the task set to form a batch. In the training phase, the internal and external loops are combined, and the final training goal is to improve the top-level EMS model M meta The performance in different tasks and situations makes the top-level EMS model have the best generalization ability when facing different HEV tasks. The top-level EMS model refers to a global, universal control strategy in the energy management system, which can quickly adapt and optimize control under different tasks or scenarios. This model is usually trained by meta-learning, can quickly adjust parameters according to the data of various tasks, and adapt to different lithium battery and supercapacitor hybrid vehicle configurations. For the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j , j = 1:4, perform inner loop operation to update the basic EMS controller parameters.

[0106] like Figure 3 As shown in Figure 1, the inner loop is the core part of Model-Agnostic Meta-Learning (MAML), and its main purpose is to quickly adapt to the specific needs of each task. In the inner loop, the parameters of the base EMS controller are updated with a small number of gradient steps for each sampled task. In each inner loop, for the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j , using the current parameter θ i The strategy is used as the basic EMS controller for the jth lithium battery and supercapacitor hybrid vehicle task H j Then, observe the state s at the tth time step t And the feedback from the environment (specifically the reward r at the tth time step tand the state s at the t+1th time step t+1 ), calculate the advantage function of the t-th time step The parameters θ of the basic EMS controller are updated by gradient descent.

[0107] It should be noted that the specific process of the inner loop operation in step S3 of the present invention is as follows:

[0108] S31. Randomly initialize the basic EMS controller parameters θ, including the weights and biases of all neural network layers;

[0109] S32. Randomly select a batch from a preset task set, where each task in a batch represents a specific vehicle configuration and driving environment;

[0110] S33: For the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j , based on the current parameters θ, run the basic EMS controller T time steps to obtain the feedback data D of the environment ij t ,a t ,r t >, where s t ,a t ,r t Represent the state, action and reward of the t-th time step respectively;

[0111] S34: Calculating an advantage function at each time step based on the feedback data, where the advantage function is used to represent the relative performance of the current strategy at each time step;

[0112] Advantage function at the tth time step The calculation formula is:

[0113]

[0114] Among them, γ i-t represents the discount weight of the reward at the i-th time step relative to the t-th time step; r i represents the reward obtained at the i-th time step; V φ (s t ) is the state s at the t-th time step t The value function under ;

[0115] S35: Based on the advantage function and the feedback data, the maximization formula is used to perform gradient update, and the current parameter θ is updated to θ′ ij :

[0116] ​

[0117] Among them, |D ij | indicates feedback data D ij The size of f θ (a i |s i ) represents a given state s i When we execute action a based on strategy θ i probability; Represents a given state s i Based on the strategy θ ij Execute action a i probability; Indicates that in state s i Next, perform action a i Based on the current strategy The advantage function of ij Represents the i-th batch B i The parameters after gradient update of the jth lithium battery and supercapacitor hybrid vehicle task in; α represents the inner loop learning rate;

[0118] S36: Continuously iterate the gradient update to make the gradient updated parameter θ′ ij Able to gradually adapt to the needs of the current task.

[0119] It should be noted that after each gradient update, the strategy of the basic EMS controller will be adjusted to gradually improve the performance in a specific task.

[0120] S4. When the inner loop operation is completed, the basic EMS controller parameters updated by the inner loop operation are used to re-update the global parameters of the basic EMS controller in the outer loop operation by integrating the feedback of multiple tasks, and the global parameters of the basic EMS controller updated by the outer loop operation are used as the final basic EMS controller parameters to obtain the trained basic EMS controller.

[0121] It should be noted that if Figure 3 As shown, the outer loop operation performed in step S4 must be performed after all inner loops in S3 are completed. The key to the outer loop is to use the θ′ obtained in the inner loop ij To update the basic EMS controller global parameters This enables the basic EMS controller to have stronger generalization capabilities when facing different hybrid vehicle tasks.

[0122] In step S4, the specific process of updating the basic EMS controller parameters by the outer loop operation is as follows:

[0123] S41: The input variables are the updated parameters of the gradients of each lithium battery and supercapacitor hybrid vehicle task in a batch, aggregating each lithium battery and supercapacitor hybrid vehicle task in a pre-built validation set The performance on the validation set is calculated for each lithium battery and supercapacitor hybrid vehicle task:

[0124]

[0125] in, is the predicted value of the basic EMS controller, y i is the expected output; Represents the validation set size.

[0126] It should be noted that the validation set From Task H i The data in is divided independently, and each task can be divided into a validation set to evaluate the updated θ′ ij generalization ability on this task.

[0127] S42: Compute meta-gradients based on the loss function for each lithium battery and supercapacitor hybrid vehicle task Used to update global parameters

[0128]

[0129] in, Represents the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task Η j The loss function of the global parameter gradient.

[0130] It should be noted that the above loss function measures the updated basic EMS controller parameter θ′ ij The error between the predicted value and the true value on the validation set.

[0131] S43: Use the meta-gradient descent method to update the global parameters as follows:

[0132]

[0133] Among them, β represents the outer loop learning rate, which determines the step size of global parameter update.

[0134] It should be noted that the goal of the update is to minimize the loss function of all tasks, thereby optimizing the performance of the basic EMS controller.

[0135] S44: Use updated global parameters Evaluate the performance of the base EMS controller on the validation set of each task to ensure that the global update helps generalization rather than just adapting to a single task;

[0136] S45: Iteratively update global parameters to optimize the performance of the base EMS controller on all Li-ion and supercapacitor hybrid vehicle tasks in the entire batch.

[0137] S5. In the meta-test phase, a set of lithium battery and supercapacitor hybrid vehicle tasks with different batteries and supercapacitors are sampled according to the distribution ε to form a new task set Randomly select test tasks from the new task set, generate a new parameter combination different from the meta-training stage for each test task, fine-tune the trained basic EMS controller on each new parameter combination for energy management, and form a new energy management strategy corresponding to each test task after fine-tuning.

[0138] It should be noted that in the meta-test phase of the present invention, the performance of the trained basic EMS controller needs to be evaluated and verified to adjust the parameters of the basic EMS controller. In the meta-test phase of this embodiment, first, a set of lithium battery and supercapacitor hybrid electric vehicle (HEV) tasks of different batteries and supercapacitors are sampled according to the distribution ε to form a new task set Then, test tasks are randomly selected from the new task set, and new HEV parameters different from those in the meta-training stage are configured for each test task, including the rated capacity of the lithium battery, the rated capacity of the supercapacitor, the peak power of the lithium battery, and the peak power of the supercapacitor, etc. For each new set of lithium battery and supercapacitor HEV tasks, the corresponding basic EMS controller (i.e., M1 to M5) is trained for energy management.

[0139] It should be noted that, in the meta-testing stage of the present invention, the specific process of fine-tuning is as follows:

[0140] S51: Global parameters obtained from meta-training As initial parameters, initialize the weights and biases of the basic EMS controller;

[0141] S52: sampling test data from the test data set, using the vehicle state parameter as an input feature of the basic EMS controller, and using the expected control output power distribution as a target output of the basic EMS controller;

[0142] S53: Fast adaptive fine-tuning of the basic EMS controller on the test task, using the initial parameters from S51 Perform gradient updates on the basic EMS controller to generate feedback data for the new environment Calculate the loss using feedback data from the new environment The specific form is as follows:

[0143]

[0144] in, Represent the new state, new action and new reward of the t-th time step respectively; |D i |Represents feedback data of the new environment D i size; Indicates that in the new parameter Under the basic EMS controller The predicted value at time ;

[0145] S54: After the gradient has descended N times, the gradient is calculated based on the loss and the parameters of the basic EMS controller are updated as follows: So that it can select appropriate actions according to the current state and strategy and achieve efficient energy management:

[0146]

[0147] in, Represents the loss function Basic EMS controller parameters The gradient of

[0148] S55: With current parameters As the basic EMS controller parameters, run the fine-tuned basic EMS controller, execute T time steps, obtain the data of the running process, and calculate the performance indicators of the basic EMS controller on the test task.

[0149] In order to better demonstrate the specific implementation and technical effects of the present invention, the energy management method combining meta-learning and proximal strategy optimization algorithm shown in steps S1 to S5 in the above preferred implementation is applied to a specific example.

[0150] Example

[0151] The specific implementation process of the energy management method combining meta-learning and proximal strategy optimization algorithm adopted in the embodiment of the present invention is as described above and will not be repeated here.

[0152] In order to verify the performance of the energy management method combining meta-learning and proximal policy optimization algorithm proposed in the present invention, two groups of experiments were conducted in the embodiments of the present invention. In the first group of experiments, the adaptability of the method meta-EMS of the present invention was compared with the two baseline models from random parameter training and transfer learning to prove the advantage of the method of the present invention in terms of adaptability. In the second group of experiments, the control performance of the method meta-EMS of the present invention was compared with the traditional reinforcement learning algorithm to evaluate its performance in terms of control performance. Specifically, in the first group of experiments, the migration efficiency of the method of the present invention was tested, and the adaptability of the basic EMS controller obtained by meta-training to new tasks was compared with the random parameter model and the pre-trained model obtained by transfer learning training in the meta-test phase. In the second group of experiments, the control strategy finally obtained in the meta-test phase was compared with the control strategy based on the traditional reinforcement learning algorithm, including Q learning (Q-learning) and deep Q network (DQN) algorithm, to evaluate its performance in terms of control performance. Among them, the pre-trained model specifically refers to selecting an HEV as a test object, whose parameters are shown in Table 1, and using the parameters obtained by training the HEV through the PPO algorithm to initialize the model; the random parameter model specifically refers to the model obtained by randomly initializing the parameters in the PPO strategy model.

[0153] Table 1. Parameters of specific lithium battery / supercapacitor hybrid vehicles selected for the experiment

[0154]

[0155] In this embodiment, four driving conditions are selected, namely, crowded urban roads (Manhattan bus driving cycle, MBDC), general urban roads (Urban Dynamics Driving Program, UDDS), suburban roads (West Virginia Suburban Driving Program, WVUSUB) and highways (Highway Fuel Economy Certification Test, HWFET). Figure 4 As shown, the control performance of the method of the present invention is compared with two adaptability baselines, the random parameter model and the pre-trained model. The experiment was conducted using the same HEV as in the first set of experiments. Figures 5 to 13 The performance of meta-EMS after 5, 10, and 15 gradient updates is compared with that of Q-learning, deep Q-learning (DQN), and DDPG. The parameter ratios between lithium batteries and supercapacitors are similar (Q b / Q c The values ​​are similar, and at the same time, the two vehicles The values ​​are similar), but there are significant differences in the values.

[0156] The method of the present invention can adapt to new tasks more quickly and provide better control performance. The method of the present invention can also design EMS through fewer gradient descent steps to improve the efficiency and practicality of the method of the present invention.

[0157] Table 2. Energy management strategy performance comparison

[0158]

[0159] The results show that the method of the present invention has similar performance to the DQN algorithm when only 5 gradient steps are passed, which is much better than the Q-Learning algorithm. As the number of gradient descent steps increases, the control performance of the method of the present invention gradually improves and exceeds the DQN algorithm. Finally, after 15 gradient descent steps, although the method of the present invention does not exceed the DDPG algorithm, the results are very close. In addition, the main advantage of the method of the present invention is that the EMS design can be achieved by a small amount of gradient descent in the meta-training stage, without the need for 100 iterations of learning each time to obtain the EMS.

[0160] The effectiveness and superiority of the method of the present invention are demonstrated by comparing the performance with the traditional reinforcement learning baseline algorithm. The method of the present invention can achieve comparable control performance with a small number of gradient steps, and after gradually increasing the gradient descent steps, its performance gradually surpasses the baseline algorithm. This shows that the method of the present invention can adapt to new tasks more quickly and provide better control performance. In addition, the method of the present invention can also design EMS with fewer gradient descent steps, improving the efficiency and practicality of the algorithm.

[0161] In summary, in the present invention, the meta-training stage uses the MAML algorithm for meta-training, the purpose of which is to train a basic EMS controller that can quickly adapt to new scenarios. It is trained through multiple different tasks (each task corresponds to a specific driving situation), and each task will perform several gradient updates to find a good strategy; in the meta-testing stage, after completing the meta-training, the basic EMS controller is applied to new tasks (i.e., new driving environments). At this stage, the basic EMS controller needs to demonstrate its ability to quickly adapt to new tasks. The specific approach is to perform only a small amount of gradient updates on the new tasks, and then evaluate the performance of the basic EMS controller; in the performance evaluation, after a small amount of gradient updates, the difference in control performance between the meta-energy management strategy and the traditional strategy is compared, including comparing indicators such as the power consumption of lithium batteries and the power recovery of supercapacitors. In the result analysis, the performance of the meta-energy management strategy after different gradient update steps is specifically analyzed, and compared with the benchmark strategy to determine the adaptability and efficiency of the meta-learning strategy.

[0162] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An energy management method combining meta-learning and proximal strategy optimization algorithm, characterized in that: The following steps are involved: S1. Build a basic EMS controller as a basic controller that can adapt to a variety of lithium battery and supercapacitor hybrid vehicles; S2. In the meta-training task set preparation stage, a set of lithium battery and supercapacitor hybrid vehicle tasks of different batteries and supercapacitors are sampled according to the distribution to form a task set, and the rated capacity variation range of the actual hybrid vehicle lithium battery, the rated capacity variation range of the supercapacitor, the peak power variation range of the lithium battery and the peak power variation range of the supercapacitor are set according to the key variables of the lithium battery and supercapacitor hybrid vehicle. Within the set four variation ranges, several different parameter combinations are randomly selected, and each parameter combination is used to describe a lithium battery and supercapacitor hybrid vehicle; S3. In the meta-training phase, a plurality of different lithium battery and supercapacitor hybrid vehicle tasks are randomly sampled from the task set to form a batch, and the basic EMS controller parameters are updated by the inner loop operation on each batch; S4. After the inner loop operation is completed, the basic EMS controller parameters updated by the inner loop operation are used to re-update the global parameters of the basic EMS controller in the outer loop operation by integrating the feedback of multiple tasks, and the global parameters of the basic EMS controller updated by the outer loop operation are used as the final basic EMS controller parameters to obtain the trained basic EMS controller; S5. In the meta-testing phase, a group of lithium battery and supercapacitor hybrid vehicle tasks with different batteries and supercapacitors are sampled according to the distribution to form a new task set, and test tasks are randomly selected from the new task set. A new parameter combination different from that in the meta-training phase is generated for each test task, and the trained basic EMS controller is fine-tuned on each new parameter combination for energy management. After fine-tuning, a new energy management strategy corresponding to each test task is formed.

2. The energy management method combining meta-learning and proximal strategy optimization algorithm as claimed in claim 1, characterized in that: In step S1, a convolutional neural network reinforcement learning controller is constructed as a main controller of a lithium battery and supercapacitor hybrid vehicle, which includes an input layer, a first convolutional layer, a second convolutional layer, a fully connected layer, and an output layer; In the convolutional neural network reinforcement learning controller, the input layer is used to obtain input features, the first convolution layer performs shallow feature extraction on the input features to obtain first output features, the second convolution layer performs deep feature extraction on the first output features to obtain second output features, the second output features are flattened and input into the fully connected layer to obtain third output features, and the output layer outputs the third output features; The input features include vehicle speed, acceleration, battery charge state, supercapacitor charge state, current required power, vehicle position, vehicle acceleration direction, vehicle steering angle, vehicle load state, and external temperature.

3. The energy management method combining meta-learning and proximal strategy optimization algorithm as claimed in claim 1, characterized in that: In step S2, the key variables include the rated capacity of the lithium battery, the rated capacity of the supercapacitor, the peak power of the lithium battery and the peak power of the supercapacitor, and the equivalent resistance of the lithium battery is taken as a related dependent variable.

4. The energy management method combining meta-learning and proximal strategy optimization algorithm as claimed in claim 1, characterized in that: The same reward function is set for the basic EMS controllers of different types of lithium battery and supercapacitor hybrid vehicles: R t =ω1r1+ω2r2+ω3r3+ω4r4+ω5r5 r2=BSOC t -BSOC0 Among them, R t represents the reward function at the tth time node in the whole process; ω1, ω2, ω3, ω4, ω5 represent the weight parameters of each part of the reward function; r1, r2, r3, r4, r5 represent the feedback of different performance indicators in lithium battery and supercapacitor hybrid vehicles respectively; represents the total power demand of the vehicle at the tth time node; p t Indicates the actual output power provided by the lithium battery and supercapacitor at the tth time node; BSOC t Indicates the state of charge of the lithium battery at the tth time node; BSOC0 indicates the initial charge of the battery; Represents the output current of the lithium battery at the tth time node; Represents the output current of the lithium battery at the t-1th time node; Indicates the change of the state of charge of the supercapacitor at the tth time node; USOC t Indicates the state of charge of the supercapacitor at the tth time node; USOC min and USOC max They represent the upper and lower limits of the capacitor charge respectively.

5. In the energy management method combining meta-learning and proximal strategy optimization algorithm as claimed in claim 1, in step S3, the specific process of updating the basic EMS controller parameters by the inner loop operation is as follows: S31. Randomly initialize the basic EMS controller parameters, including weights and biases of all neural network layers; S32. Randomly select a batch from a preset task set, where each task in a batch represents a specific vehicle configuration and driving environment; S33: For the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j , based on the current parameters θ, run the basic EMS controller T time steps to obtain the feedback data D of the environment ij t ,a t ,r t >, where s t ,a t ,r t Represent the state, action and reward of the t-th time step respectively; ​ S34: Calculating an advantage function at each time step based on the feedback data, where the advantage function is used to represent the relative performance of the current strategy at each time step; Advantage function at the tth time step The calculation formula is: Among them, γ i-t represents the discount weight of the reward at the i-th time step relative to the t-th time step; r i represents the reward obtained at the i-th time step; V φ (s t ) is the state s at the t-th time step t The value function under ; S35: Based on the advantage function and the feedback data, the maximization formula is used to perform gradient update, and the current parameter θ is updated to θ i ′ j : Among them, |D ij | indicates feedback data D ij The size of f θ (a i |s i ) represents a given state s i When we execute action a based on strategy θ i probability; Represents a given state s i Based on the strategy θ ij Execute action a i probability; Indicates that in state s i Next, perform action a i Based on the current strategy The advantage function of i ′ j Represents the i-th batch B i The parameters after gradient update for the jth lithium battery and supercapacitor hybrid vehicle task in; α represents the inner loop learning rate; S36: Continuously iterate the gradient update to make the gradient updated parameter θ i ′ j Able to gradually adapt to the needs of the current task.

6. In the energy management method combining meta-learning and proximal strategy optimization algorithm as claimed in claim 5, in step S4, the specific process of updating the basic EMS controller parameters by the outer loop operation is as follows: S41: The input variables are the updated parameters of the gradients of each lithium battery and supercapacitor hybrid vehicle task in a batch, aggregating each lithium battery and supercapacitor hybrid vehicle task in a pre-built validation set The performance on the validation set is calculated by calculating the loss function L for each lithium battery and supercapacitor hybrid vehicle task on the validation set. ij (θ i ′ j ): in, is the predicted value of the basic EMS controller, y i is the expected output; Represents the validation set size; S42: Compute meta-gradients based on the loss function for each lithium battery and supercapacitor hybrid vehicle task Used to update global parameters in, Represents the i-th batch B i The jth lithium battery and supercapacitor hybrid vehicle task H j The loss function of the global parameter The gradient of S43: Use the meta-gradient descent method to update the global parameters as follows: Among them, β represents the outer loop learning rate; S44: Use updated global parameters Evaluate the performance of the base EMS controller on the validation set of each task to ensure that the global update helps generalization rather than just adapting to a single task; S45: Iteratively update global parameters to optimize the performance of the base EMS controller on all Li-ion and supercapacitor hybrid vehicle tasks in the entire batch.

7. The energy management method combining meta-learning and proximal strategy optimization algorithm as claimed in claim 6, characterized in that: In step S5, the specific process of fine-tuning in the meta-test phase is as follows: S51: Global parameters obtained from meta-training As initial parameters, initialize the weights and biases of the basic EMS controller; S52: sampling test data from the test data set, using the vehicle state parameter as an input feature of the basic EMS controller, and using the expected control output power distribution as a target output of the basic EMS controller; S53: Fast adaptive fine-tuning of the basic EMS controller on the test task, using the initial parameters from S51 Perform gradient updates on the basic EMS controller to generate feedback data for the new environment Calculate the loss function using feedback data from the new environment The specific form is as follows: in, Represent the new state, new action and new reward of the t-th time step respectively; |D i |Represents feedback data of the new environment D i size; Indicates that in the new parameter Under the basic EMS controller The predicted value at time ; S54: After the gradient has descended N times, the gradient is calculated based on the loss and the parameters of the basic EMS controller are updated as follows: So that it can select appropriate actions according to the current state and strategy and achieve efficient energy management: in, Represents the loss function Basic EMS controller parameters The gradient of S55: With current parameters As the basic EMS controller parameters, run the fine-tuned basic EMS controller, execute T time steps, obtain the data of the running process, and calculate the performance indicators of the basic EMS controller on the test task.

Images