Pulse power supply system control method and system based on safe deep reinforcement learning

Abstract

The application discloses a kind of based on safe depth reinforcement learning's pulse power supply system control method and system, control method includes: based on depth reinforcement learning model generates the real-time control action of pulse power supply system;Based on nonlinear model predictive controller generates the real-time guide action of pulse power supply system;Determine whether real-time control action satisfies safety constraint: if yes, based on real-time control action, real-time guide action and first risk action shield function calculate safety control action, and based on first reward function calculate the reward value of depth reinforcement learning model;Otherwise, based on real-time control action, real-time guide action and second risk action shield function calculate safety control action, and based on second reward function calculate the reward value of depth reinforcement learning model.The application is applied to pulse power supply system control field, and voltage sag under the effective realization of mixed energy storage system under pulse power load can be effectively inhibited and the effective management of system temperature rise.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pulse power supply system control, and particularly relates to a pulse power supply system control method and system based on safe deep reinforcement learning. BACKGROUND

[0002] Pulse power load is a kind of special load with high peak-to-average ratio, and its application fields include industrial precision manufacturing, frontier scientific research, intelligent power grid facilities, emerging technologies and many other important fields. Representative applications include electric spark machining and superconducting fault current limiter. With the rapid development of modern industrial technology, vehicle-mounted pulse loads on mobile platforms such as vehicles and aircraft have become a research and application hotspot due to their outstanding mobility, strong environmental adaptability and compact structure design characteristics.

[0003] Vehicle-mounted pulse power supply system is the main energy source of pulse load, and has the following three typical characteristics: first, some vehicle-mounted pulse power supply systems such as vehicle power supply systems are independent power systems, which are usually powered by battery energy storage systems. These systems may face power density or energy density problems in isolated operation mode due to the lack of power support from the public power grid. Second, due to the space constraints of mobile platforms, various power supply devices need to be densely deployed in limited volume, which not only greatly limits the capacity of the battery energy storage system but also causes additional thermal management problems. Third, when facing pulse load impact, vehicle-mounted pulse power supply systems are more likely to experience power quality deterioration, including voltage sag.

[0004] The hybrid energy storage system integrated with supercapacitors and batteries provides an effective solution to the above problems. In vehicle-mounted pulse power supply systems, the dynamic energy management of the hybrid energy storage system mainly faces two challenges: first, the pulse load usually requires the voltage sag of the hybrid energy storage system to be as small as possible, because it will seriously affect the working characteristics of the vehicle-mounted pulse load, such as radar detection accuracy and distance, and in severe cases, the device may stop working. Another potential challenge is the abnormal temperature rise of the hybrid energy storage system, because it not only increases additional heat dissipation and reduces energy efficiency, but also easily causes thermal runaway problems in severe cases; at the same time, abnormal temperature rise can also accelerate the aging of the battery and supercapacitor, reducing the service life of the hybrid energy storage system. Therefore, it is urgent to study the dynamic energy management strategy of the hybrid energy storage system to improve the voltage sag and abnormal temperature rise problems in the vehicle-mounted pulse power supply system.

[0005] Currently, in the field of dynamic energy management of battery-supercapacitor hybrid energy storage system (BESS), the related work can be summarized as model-based and model-free methods. Model-based methods mainly achieve the optimal energy management of BESS by solving optimization problems. Optimization-based methods are typical achievements of model-based methods, including dynamic programming, Pontryagin's minimum principle, convex optimization, heuristic algorithms, and model predictive control. However, the problems of model-based methods, such as long computation time and dependence on accurate models, seriously hinder their practical application, and they are more used as offline or slow-time scale simulation tools. Another approach expects to achieve dynamic energy management of BESS by directly observing system control variables (such as energy storage output current) and state variables (such as voltage), thereby avoiding or reducing dependence on system models, which has given rise to model-free methods. Rule-based methods and frequency decomposition-based methods are two typical model-free methods. The former establishes the relationship between system control variables and state variables through certain rules. These rules can be deterministic, such as logic and threshold-based methods, proportional-integral-derivative control-based methods, and droop control. However, some control parameters, such as threshold values and droop coefficients, are highly dependent on human experience. The latter uses frequency domain analysis methods such as filtering and wavelet transform to decompose the total load current or power into high-frequency and low-frequency components and allocate power according to the characteristics of batteries and supercapacitors. The selection of cutoff frequency has a significant impact on the effectiveness of energy management strategies, and some scholars have pointed out that inappropriate cutoff frequency can accelerate the aging rate of batteries. In addition, the energy management strategies developed by the above two model-free methods are static and not the best energy management method. Another type of model-free method is learning-based method, such as deep learning, reinforcement learning, and deep reinforcement learning (DRL), which learns the control rules of BESS from historical data or the environment. Since it does not depend on the accurate mathematical model of the system and does not require prior knowledge, it is becoming increasingly popular. In particular, deep reinforcement learning achieves autonomous strategy learning in complex environments by deeply integrating the representation ability of deep learning and the decision optimization mechanism of reinforcement learning.Deep reinforcement learning (DRL) has shown great performance in the problems of battery and supercapacitor current sharing control under pulsed power load, optimal power distribution of hybrid energy storage to ensure voltage stability, and power imbalance solution using hybrid energy storage system, thanks to the powerful exploration ability of the agent and various algorithms such as Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), etc. The method of deep reinforcement learning combines the advantages of optimization-based methods, which can seek the best energy management strategy through reasonable reward design, and fully plays its advantages of no model in actual deployment, and quickly generates actions. Therefore, deep reinforcement learning is a very potential method and needs further research.

[0006] The existing technologies and methods mainly face three deficiencies in the control problem of hybrid energy storage under pulsed power load. First, action safety: unsafe actions generated by DRL will cause overcharge or overdischarge of hybrid energy storage, which is unacceptable for energy management of hybrid energy storage system; some documents try to propose safe reinforcement learning methods, such as adding a safety layer, adding a CBF function, etc., although this can ensure safety to some extent, but it will also filter out the best action. Second, slow training speed: the inefficient and blind exploration of the agent may make the training process slow; in addition, insufficient generalization: the trained agent may not be able to adapt to changes in the peak of pulsed power load, and needs to be retrained. SUMMARY

[0007] In view of the deficiencies in the prior art described above, the present application provides a pulsed power supply system control method and system based on safe deep reinforcement learning, which can effectively realize the rapid suppression of voltage sag and effective management of system temperature rise of the hybrid energy storage system under pulsed power load.

[0008] To achieve the above purpose, the present application provides a pulsed power supply system control method based on safe deep reinforcement learning, comprising the following steps:

[0009] An equivalent analysis model of the pulsed power supply system is established, and system data is collected;

[0010] A deep reinforcement learning model is constructed, and real-time control actions of the pulsed power supply system are generated based on the system data and the deep reinforcement learning model;

[0011] construct a nonlinear model predictive controller, and generate real-time guiding actions of the pulse power supply system based on the system data and the nonlinear model predictive controller;

[0012] determine whether the real-time control actions meet safety constraints:

[0013] If yes, calculate safety control actions based on the real-time control actions, the real-time guiding actions and a first risk action shielding function, and calculate a reward value of the deep reinforcement learning model based on a first reward function;

[0014] Otherwise, calculate safety control actions based on the real-time control actions, the real-time guiding actions and a second risk action shielding function, and calculate a reward value of the deep reinforcement learning model based on a second reward function;

[0015] control the pulse power supply system based on the safety control actions.

[0016] To achieve the above-mentioned purpose, the application further provides a pulse power supply system control system based on safety deep reinforcement learning, the control system comprising:

[0017] an equivalent model construction unit for establishing an equivalent analysis model of the pulse power supply system;

[0018] a data acquisition unit for acquiring system data of the pulse power supply system;

[0019] a deep reinforcement learning model for generating real-time control actions of the pulse power supply system according to the system data;

[0020] a nonlinear model predictive controller for generating real-time guiding actions of the pulse power supply system according to the system data;

[0021] a risk action shielding unit for determining whether the real-time control actions meet safety constraints: if yes, calculating safety control actions based on the real-time control actions, the real-time guiding actions and a first risk action shielding function, and calculating a reward value of the deep reinforcement learning model based on a first reward function; otherwise, calculating safety control actions based on the real-time control actions, the real-time guiding actions and a second risk action shielding function, and calculating a reward value of the deep reinforcement learning model based on a second reward function;

[0022] a system control unit for controlling the pulse power supply system based on the safety control actions.

[0023] Compared with the prior art, the application has the following beneficial technical effects:

[0024] 1.The application is aimed at the problem that single deep reinforcement learning may output unsafe actions, and proposes a risk action shielding method based on nonlinear model predictive control, which intervenes in the actions output by the deep reinforcement learning model in real time, and gradually makes the actions within the safe action domain;

[0025] 2.In the preferred scheme, the application proposes a training process soft guiding mechanism to solve the problem of slow training speed caused by inefficient exploration of single deep reinforcement learning, uses a nonlinear model predictive controller to guide the actions of deep reinforcement learning, and additionally adds a penalty for risky actions, so that deep reinforcement learning gradually learns the optimal strategy;

[0026] 3.The application is aimed at the problem that single deep reinforcement learning does not perform well when the peak value of the pulse power load changes, and by combining the rolling optimization of nonlinear model predictive control and the exploration ability of deep reinforcement learning, the proposed control method has a certain generalization when the peak value of the pulse load changes, so that the voltage sag of the hybrid energy storage system under the pulse power load can be effectively suppressed and the temperature rise of the system can be effectively managed. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor based on the structures shown in these drawings.

[0028] Figure 1 The flow chart of the pulse power supply system control method based on safe deep reinforcement learning in embodiment 1 of the application;

[0029] Figure 2 The composition and structure diagram of the vehicle-mounted pulse load in embodiment 1 of the application;

[0030] Figure 3 The equivalent circuit diagram of the hybrid energy storage system based on a first-order RC circuit in embodiment 1 of the application;

[0031] Figure 4 The control principle diagram of the vehicle-mounted pulse load based on safe deep reinforcement learning in embodiment 1 of the application;

[0032] Figure 5 The bus voltage waveform diagram in the simulation and experiment process in embodiment 1 of the application;

[0033] Figure 6 The battery monomer current waveform diagram in the simulation and experiment process in embodiment 1 of the application;

[0034] Figure 7 Figure 1 is a super capacitor cell current waveform chart in the simulation and experiment process of embodiment 1 of the present application;

[0035] Figure 8 Figure 2 is a battery cell temperature waveform chart in the simulation and experiment process of embodiment 1 of the present application;

[0036] Figure 9 Figure 3 is a super capacitor cell temperature waveform chart in the simulation and experiment process of embodiment 1 of the present application;

[0037] Figure 10 Figure 4 is a structure block diagram of the pulse power supply system control system based on safety deep reinforcement learning in embodiment 2 of the present application.

[0038] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0040] In addition, the technical solutions of various embodiments of the present application can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0041] Embodiment 1

[0042] As Figure 1 shown is a pulse power supply system control method based on safety deep reinforcement learning disclosed in the present embodiment, which mainly includes the following steps:

[0043] An equivalent analysis model of the pulse power supply system is established, and system data is collected;

[0044] A deep reinforcement learning model is constructed, and real-time control actions of the pulse power supply system are generated based on the system data and the deep reinforcement learning model;

[0045] A nonlinear model predictive controller is constructed, and real-time guide actions of the pulse power supply system are generated based on the system data and the nonlinear model predictive controller;

[0046] It is judged whether the real-time control actions meet the safety constraints:

[0047] If yes, a safe control action is calculated based on the real-time control action, the real-time guidance action and a first risk action masking function, and a reward value of the deep reinforcement learning model is calculated based on a first reward function;

[0048] Otherwise, a safe control action is calculated based on the real-time control action, the real-time guidance action and a second risk action masking function, and a reward value of the deep reinforcement learning model is calculated based on a second reward function;

[0049] The pulse power supply system is controlled based on the safe control action.

[0050] Reference Figure 2 The pulse power supply system in the embodiment is a battery semi-active hybrid energy storage system, which includes a battery module, a DC-DC converter, a super capacitor module and a pulse load. The battery module is connected to a DC bus through the DC-DC converter to supply power to the pulse load, and the super capacitor module is directly connected in parallel to the DC bus. The battery semi-active hybrid energy storage system topology has significant advantages. The converter capacity design can be based on the average power of the load, thereby greatly reducing the system construction cost. At the same time, the battery is isolated from the load through the DC-DC converter, which can effectively prolong the service life of the battery.

[0051] In the specific implementation process, the equivalent analysis model of the pulse power supply system mainly includes load modeling, equivalent electric model construction and equivalent thermal model construction.

[0052] The battery semi-active hybrid energy storage system of the embodiment supplies power to the load through the DC bus. Therefore, according to the characteristics of the load, the load is modeled as a pulse current source, which is:

[0053]

[0054] wherein, is the load current, , is the maximum and minimum values of the load current, is the pulse period, is the number of pulses, is the time, is the duty cycle of the pulse, .

[0055] In order to reflect the dynamic characteristics of the battery and the super capacitor, the battery and the super capacitor are modeled based on the first-order RC model in the pulse working condition. The equivalent circuit is as shown in Figure 3 . Figure 3 In the formula, the subscripts respectively represent the battery monomer LIB or the super capacitor monomer SC, is the temperature of the battery monomer LIB or the super capacitor monomer SC, ​This refers to the state of charge of a single battery cell (LIB) or a single supercapacitor cell (SC). This refers to the open-circuit voltage of a single battery cell (LIB) or a single supercapacitor cell (SC). This refers to the terminal voltage of the battery cell LIB or the supercapacitor cell SC. This refers to the current of a single battery cell (LIB) or a single supercapacitor cell (SC). This refers to the equivalent series resistance of a single battery cell (LIB) or a single supercapacitor cell (SC). This refers to the equivalent parallel resistance of a single battery cell (LIB) or a single supercapacitor cell (SC). This refers to the parallel capacitance of a single battery cell (LIB) or a single supercapacitor cell (SC). This refers to the terminal voltage of a single battery cell (LIB) or a single supercapacitor cell (SC) in the RC circuit.

[0056] According to the appendix Figure 3 The reference direction is shown, and the relationship between the above parameters can be established according to Kirchhoff's voltage law as follows:

[0057] ;

[0058] Further transformation of the above equation yields:

[0059] ;

[0060] in, It is the time constant of the RC element in a first-order RC circuit.

[0061] To reflect the thermal effects of hybrid energy storage during charging and discharging, this embodiment establishes thermal models for individual battery cells and supercapacitor cells. Considering simulation accuracy, parameter identification, and computational complexity, the model uses the lumped mass parameter of the battery and supercapacitor cells, while neglecting internal heat transfer within the cells and only considering heat exchange with the external environment, primarily including heat generation during charging and discharging and heat dissipation in the surrounding medium. The heat dissipation power of the battery and supercapacitor cells is mainly related to the temperature difference, i.e.:

[0062] ;

[0063] in, This refers to the power dissipation of a single battery cell (LIB) or a single supercapacitor cell (SC). This is the heat dissipation constant of a single battery cell (LIB) or a single supercapacitor cell (SC). The temperature of the surrounding medium can be considered a constant.

[0064] For the battery monomer, its heat production generally includes two parts, one part is the irreversible heat generated by the battery internal resistance, and the other part is the reversible heat generated by the entropy change reaction of the battery, so the heat production model of the battery monomer is:

[0065]

[0066] wherein, is the total heat production of the battery monomer LIB in the charging and discharging process, is the entropy heat coefficient of the battery monomer LIB;

[0067] For the super capacitor monomer, the total heat production is mainly irreversible heat, so the heat production model of the super capacitor monomer is:

[0068]

[0069] wherein, is the total heat production of the super capacitor monomer SC in the charging and discharging process;

[0070] In summary, according to the heat production and heat dissipation of the battery monomer and the super capacitor monomer, the temperature rise power can be further calculated as:

[0071]

[0072] wherein, is the heat capacity parameter of the battery monomer LIB or the super capacitor monomer SC, with the unit of W / K.

[0073] The system data collected in this embodiment includes the battery monomer state of charge, the battery monomer temperature, the battery monomer output current, the super capacitor monomer state of charge, the super capacitor monomer temperature, the super capacitor monomer output current, the DC bus voltage, and the DC-DC converter inductance current.

[0074] In this embodiment, the process of generating real-time control actions by the deep reinforcement learning model (hereinafter referred to as "DRL agent") is specifically as follows:

[0075] Firstly, the currently collected battery monomer state of charge, battery monomer temperature, super capacitor monomer state of charge, super capacitor monomer temperature and pulse load output current are input into the deep reinforcement learning model as the current state to obtain the initial control action wherein, the initial control action includes the current values of the battery monomer and the super capacitor monomer;

[0076] Then, Gaussian noise is added on the basis of the initial control action to balance exploration and utilization, and the real-time control action is obtained as:

[0077] ​​​​

[0078] wherein, is a real-time control action, is a standard deviation of Gaussian noise.

[0079] In the specific implementation process, the deep reinforcement learning framework adopted by the DRL agent can be DDPG algorithm, or TD3 algorithm, SAC algorithm, etc. DDPG algorithm, TD3 algorithm, and SAC algorithm are all Actor-Critic frameworks. In specific application, the exploration intensity design for DRL agents using different algorithms includes low exploration capability (LEC), medium exploration capability (MEC), and high exploration capability (HEC). For TD3 algorithm and DDPG algorithm, the exploration capability is controlled by adjusting the standard deviation, and for SAC algorithm, the exploration capability of the algorithm is affected by designing the entropy weight. In addition, in order to fully consider the influence of different initialization conditions of Actor network and Critic network in the algorithm on the training process, 10 repeated experiments are conducted for each algorithm, and finally the average index can be used to show the performance of the algorithm, and the optimal algorithm is used as the deep reinforcement learning framework of the DRL agent. Preferably, in order to speed up the training, the action and state value in the DRL agent training process are normalized data, and a scaling layer is additionally added after the Actor network, which is used to map the normalized action value output by the Actor to the actual action interval.

[0080] Taking the DRL agent using DDPG algorithm as the deep reinforcement learning framework as an example, the core idea is to optimize the cooperative learning of deterministic policy gradient and Q value approximation, combined with target network and experience replay mechanism, to realize stable heterogeneous policy learning in continuous action space. In the state , the normalized value of the deterministic action and its real value , i.e. the initial control action, are obtained through the Actor network; in order to make the DDPG algorithm better balance exploration and utilization, Gaussian noise is added on the basis of , i.e. the processed action ; the action acts on the environment and observes the new state , the immediate reward and the termination flag Done, which can be the battery state of charge out of bounds, super capacitor temperature out of limits, DC bus voltage collapse, pulse load period end, etc.; repeat the above process until the termination condition is met. And after each iteration step, the 5-tuple is transferred to the buffer , the maximum size of which is , and a sample of size N is randomly sampled from it.

[0081] During training, the Actor network is used to generate actions, while the Critic network is evaluated using the Bellman equation. To ensure a stable training process, a stable Q-value is typically obtained using the target network, i.e.:

[0082] ;

[0083] in, For strategy of value, As a strategy, The discount factor is used to weigh immediate rewards against future rewards; correspondingly, the Q-value of the Critic network can be expressed as:

[0084] ;

[0085] in, The Q value is calculated by the network. As a reward for the current action, For the state at the next moment, For strategy;

[0086] The loss function can then be expressed as:

[0087] ;

[0088] Finally, gradient descent is used to update the parameters of the Critic network. ,Right now:

[0089] ;

[0090] in, This is the learning rate.

[0091] The updates to the Actor network depend on the expected return of the policy, which is:

[0092] ;

[0093] in, For the environment, Indicates action By strategy Generation; by applying the chain rule to the expected reporting function. J To update the policy function , right The gradient can be approximated as:

[0094] ;

[0095] in, The expectation is that the policy The down-sampled action The upper computation, is the sampled probability distribution;

[0096] In the specific application process, in order to facilitate the calculation, The gradient of is updated by batch samples, that is:

[0097] ;

[0098] Finally, the gradient ascent method is used to update the parameters of the Actor network , that is:

[0099] ;

[0100] Wherein, is the learning rate.

[0101] As a preferred embodiment, in order to ensure the stability in the training process, the parameters of the target network are updated in a soft manner, that is, the parameters of the target network And Are not directly assigned by the Actor and Critic network parameters, but are updated gradually by the following formula:

[0102] ;

[0103] ;

[0104] Wherein , indicates the update coefficient of the parameters.

[0105] In the specific application process, the state space and the action space are two core concepts for defining the agent and the environment (i.e. the hybrid energy storage system in this embodiment), and their structure and properties directly affect the design of the algorithm and the training effect, so they need to be carefully defined. The method of this embodiment is mainly used for the control of hybrid energy storage, so the action of the DRL agent can be regarded as a control instruction. Before defining the action space, the energy distribution relationship of the system needs to be analyzed. In Figure 2 , a plurality of battery monomers and super capacitor monomers are connected in series and parallel to form a battery pack and a super capacitor module, and according to Kirchhoff's current law, the current equation at the bus node is:

[0106] ;

[0107] Wherein, is the output current of the converter; is the number of parallel super capacitor monomers, so the bus voltage can be represented as:

[0108] ;

[0109] wherein, is the number of series connection of supercapacitor cells, is the converter output voltage;

[0110] According to the power conservation and the battery charge and discharge state, the relationship between the converter input and output side power can be further expressed as:

[0111] ;

[0112] wherein, and are the number of series and parallel connection of supercapacitor cells, respectively, is the energy conversion efficiency of the converter.

[0113] From the hybrid energy storage electric heating coupling model, the output current of the battery and the supercapacitor cell not only affects its own terminal voltage, but also has an important influence on the energy balance of the system. Therefore, the current of the battery and the supercapacitor cell is selected as the action of the agent, i.e. On this basis, the limit value of the action has a clear physical meaning, i.e. the lower limit of the action corresponds to the discharge current of the battery and the supercapacitor cell, respectively , , and the upper limit of the action corresponds to the charging current , , i.e.

[0114] ;

[0115] The state space is a set containing all possible states of the hybrid energy storage and each state is a unique environment description. In the control problem, the state is generally selected as the parameter that can be directly measured and contains rich information. In this embodiment, the state of charge, temperature of the battery cell, state of charge, temperature of the supercapacitor cell and load are selected as the state, i.e. the state space is:

[0116]

[0117] In deep reinforcement learning, clearly defining the upper and lower bounds of the state space can help constrain the behavior of the agent and ensure that it operates within a reasonable range, thereby improving the stability and efficiency of learning. That is, the upper and lower bounds of the state space are:

[0118] ;

[0119] ;

[0120] wherein, , are lower and upper bounds of the state space, , are lower bounds of the state of charge of the battery cells LIB or supercapacitor cells SC, , are lower bounds of the temperature of the battery cells LIB or supercapacitor cells SC, , are lower and upper bounds of the load current, , are upper bounds of the state of charge of the battery cells LIB or supercapacitor cells SC, , are upper bounds of the temperature of the battery cells LIB or supercapacitor cells SC.

[0121] Reasonable reward shaping is essential for fast training of DRL and stable policy learning. In the process of reward shaping, the present research mainly focuses on the satisfaction of the quality of pulse load power supply, the temperature rise of battery and supercapacitor cells, and safety constraints. Some types of pulse loads (such as microwave emission) often require that the bus voltage fluctuation be limited within a certain range, that is, the change of bus voltage relative to the bus reference voltage within a certain range, as shown in the following formula:

[0122] ;

[0123] wherein, is the bus voltage fluctuation, is the bus reference voltage;

[0124] Therefore, the corresponding bus voltage fluctuation reward can be designed according to the bus voltage fluctuation , which is:

[0125] ;

[0126] wherein, , are upper and lower limits of the bus voltage fluctuation, , is a reward constant.

[0127] According to the electro-thermal coupling model, the temperature of the battery cells and supercapacitor cells will change during charging or discharging. The high-frequency fluctuation of the pulse load will significantly increase the temperature rise of the hybrid energy storage system, which not only increases the additional heat dissipation demand, thereby increasing unnecessary energy loss, but also exacerbates the aging of the battery and supercapacitor. Therefore, the temperature rise suppression reward is shaped by the present embodiment to reduce heat generation on the one hand and to reduce temperature changes as much as possible on the other hand. The temperature rise suppression reward is specifically:

[0128] ;

[0129] wherein, 、 is a weight coefficient.

[0130] In addition, the DRL agent may violate the constraints when generating actions, and the embodiment makes the DRL agent learn stably in training and tend to generate safe actions by a safety constraint penalty term, and sets a penalty based on whether the constraints are met, that is:

[0131] ;

[0132] ;

[0133] ;

[0134] wherein, is a penalty of the constraint , is a penalty coefficient of the constraint , ; is a system safety constraint matrix, is an upper limit of the terminal voltage of the super capacitor unit SC, respectively represent (battery SOC upper limit constraint), (battery SOC lower limit constraint), (super capacitor SOC upper limit constraint), (super capacitor SOC lower limit constraint), (super capacitor voltage upper limit constraint).

[0135] Therefore, the total reward of the DRL agent in the process of interacting with the environment can be expressed as .

[0136] In the embodiment, the nonlinear model predictive controller (hereinafter referred to as “NMPC”) is mainly used as part of the risk action shielding model. Model predictive control combines optimization and feedback control ideas and has the characteristics of naturally suppressing disturbances. In the hybrid energy storage system considering electro-thermal coupling, dynamic energy management and thermal-electric safety collaborative control are realized through a rolling optimization mechanism. Specifically, at a certain time , the NMPC and the DRL obtain the same initial state , the DRL generates a real-time control action, and the NMPC generates a real-time guide action. Specifically, the NMPC uses an electro-thermal coupling model to predict a future period of time (a prediction time domain, denoted as system behavior, the optimal control sequence within the prediction horizon is obtained by solving a pre-defined optimization problem The current time to the prediction horizon. Subsequently, only the first few steps of the optimal sequence (control horizon, denoted as ) are actually applied to the system. With and the first steps of the control sequence, the entire prediction window is rolled forward and the optimization and procedure are repeated at the next time instant, realizing the rolling optimization of the hybrid energy storage system.

[0137] The nonlinear model predictive controller in this embodiment is specifically:

[0138] ;

[0139] wherein, represents the cost function value of the system at time , and represents the expression of the cost function; the first term of the above formula is mainly the dynamic constraint of the system; in addition, this embodiment also considers the dynamic constraint of the state of charge of the battery cell and the super capacitor cell, that is:

[0140] ;

[0141] wherein represents the duration of the current, represents the rated capacity of the battery or super capacitor cell. In the second term of the above formula constraint, represents the entire state space, represents the entire action space; in the third term of the constraint, represents the terminal state constraint set, which mainly constrains the final state of charge of the super capacitor cell, has the following form:

[0142] ;

[0143] wherein, represents the minimum value of the expected state of charge of the super capacitor at the terminal state, represents the maximum value of the expected state of charge of the super capacitor at the terminal state. The fourth term of the constraint represents the system state at time m ; the fifth term of the constraint is the constraint on the state and action.

[0144] The main idea of the risk action shielding model in this embodiment is to determine whether the real-time control action generated by the DRL satisfies the safety constraint, and combine the real-time control action generated by the DRL with the real-time guide action and a preset risk action shielding function, thereby generating a safe action . Wherein, the process of judging whether the real-time control action meets the safety constraints is: substituting the current values of the battery monomer and the super capacitor monomer corresponding to the real-time control action into the simulation calculation of the pulse power supply system, and judging whether the simulation result meets the battery SOC upper limit constraint, the battery SOC lower limit constraint, the super capacitor SOC upper limit constraint, the super capacitor SOC lower limit constraint and the super capacitor voltage upper limit constraint at the same time, if yes, it means that the real-time control action meets the safety constraints, otherwise it does not meet.

[0145] When the real-time control action meets the safety constraints, the process of calculating the safe control action based on the first risk action shielding function is:

[0146] First, iteratively search for the critical safe action in the search space with the real-time control action as the upper and lower limits In the specific application process, the bisection method can be used to search for the critical safe action in the search space. After each search, it is judged whether the critical safe action is in the safe domain by using the safety constraints. If yes, update it as the new upper limit or lower limit of the search space and search again until the critical safe action searched results in any constraint being violated, then output the critical safe action searched last time as ;

[0147] Secondly, calculate the safe control action based on the critical safe action and the real-time guide action , which is:

[0148] ;

[0149] Wherein, represents the scaling factor, and satisfies ;

[0150] When the real-time control action does not meet the safety constraints, the process of calculating the safe control action based on the second risk action shielding function is:

[0151] ;

[0152] Wherein, represents the scaling factor, satisfies .

[0153] In addition, according to the distance between the real-time control action and the safe control action , the real-time action penalty term is constructed , as a soft guiding mechanism of DRL training. And the real-time action penalty term is different according to whether the real-time control action satisfies the safety constraint, specifically:

[0154] When the real-time control action satisfies the safety constraint, that is, the real-time action penalty term in the first reward function is specifically:

[0155] ;

[0156] Wherein, is the first proportional coefficient;

[0157] In the second reward function, the real-time action penalty term is specifically:

[0158] ;

[0159] Wherein, is the second proportional coefficient, and .

[0160] Therefore, after considering the risk action shielding model, the reward of the DRL agent (i.e. the first reward function and the second reward function in the embodiment) is Although the action generated by the DRL agent is manually shielded when the action is applied for safety considerations, it does not directly affect the parameters of the network. The action shielding operation in the embodiment is fed back to the training process of the network in the form of reward-state pairs. Obviously, such updates are also relatively mild.

[0161] Referring to Figure 4 , the embodiment generates a risk action using a deep reinforcement learning method , generates a safety action using a risk shielding function, and uses a current-voltage-current three-level control on the control strategy. The specific process is as follows:

[0162] First, compare the safety control action as a reference value with the actually measured current values of the battery monomer and the super capacitor monomer, and generate an outer loop voltage reference signal after passing through a PID controller.

[0163] Second, use the outer loop voltage reference signal and the actually measured DC bus voltage to generate an inner loop current reference signal after passing through a PID controller.

[0164] Finally, use the inner loop current reference signal and the actually measured DC-DC converter inductor current The duty cycle is generated by a PID controller to obtain the switching control signals S1 and S2 of the DC-DC converter, and the DC-DC converter is controlled based on the control signals S1 and S2.

[0165] Reference Figure 5 , Figure 6 , Figure 7 , Figure 8 and Figure 9 are bus voltage waveform diagrams, battery cell current waveform diagrams, supercapacitor cell current waveform diagrams, battery cell temperature waveform diagrams, and supercapacitor cell temperature waveform diagrams in the simulation and experimental processes of the method of the embodiment, respectively. Figure 5 It can be seen that the control method of the embodiment can quickly realize the rapid suppression of bus voltage sag under pulse load conditions and maintain the bus voltage near 200V. The simulation and implementation processes verify the effectiveness of the control method of the embodiment. According to Figure 6 and Figure 7 It can be seen that the battery cell action current and the supercapacitor action current generated by the method of the embodiment can realize the safe control of the hybrid energy storage system under pulse load, avoiding the overcharging or discharging of the battery or supercapacitor cell caused by the risk action of the conventional deep reinforcement learning algorithm. According to Figure 8 and Figure 9 It can be seen that the control algorithm of the embodiment can realize the active thermal management of the battery and supercapacitor cells. Under the pulse load conditions shown in the embodiment, the temperature rise of the battery cell and the supercapacitor cell is only 0.8℃ and 0.4℃.

[0166] It is worth noting that although each step in the embodiment Figure 1 is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated herein, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0167] Embodiment 2

[0168] Based on the pulse power supply system control method based on safe deep reinforcement learning in embodiment 1, the embodiment discloses a pulse power supply system control system based on safe deep reinforcement learning, referring to Figure 10The pulse power supply system control system comprises an equivalent model construction unit, a data acquisition unit, a deep reinforcement learning model, a nonlinear model predictive controller, a risk action shielding unit and a system control unit, and specifically:

[0169] The equivalent model construction unit is configured to establish an equivalent analysis model of the pulse power supply system.

[0170] The data acquisition unit is configured to acquire system data of the pulse power supply system.

[0171] The deep reinforcement learning model is configured to generate a real-time control action of the pulse power supply system according to the system data.

[0172] The nonlinear model predictive controller is configured to generate a real-time guide action of the pulse power supply system according to the system data.

[0173] The risk action shielding unit is configured to determine whether the real-time control action satisfies a safety constraint: if yes, a safety control action is calculated based on the real-time control action, the real-time guide action and a first risk action shielding function, and a reward value of the deep reinforcement learning model is calculated based on a first reward function; otherwise, a safety control action is calculated based on the real-time control action, the real-time guide action and a second risk action shielding function, and a reward value of the deep reinforcement learning model is calculated based on a second reward function.

[0174] The system control unit is configured to control the pulse power supply system according to the safety control action.

[0175] In this embodiment, the specific working processes and working principles of the equivalent model construction unit, the data acquisition unit, the deep reinforcement learning model, the nonlinear model predictive controller, the risk action shielding unit and the system control unit are the same as those in the method of embodiment 1, and thus the above will not be described herein. Each unit module can be realized by software, hardware or a combination thereof, and can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each unit module.

[0176] The above description is only preferred embodiments of the present application, and does not limit the protection scope of the present application, and any equivalent structural transformation made according to the contents of the present application and the drawings, or direct / indirect application in other related technical fields is included in the protection scope of the present application.

Claims

1. A method for controlling a pulse power supply system based on safe deep reinforcement learning, characterized in that, The method comprises the following steps: An equivalent analysis model of the pulse power supply system is established, and system data is collected, wherein: The pulse power supply system comprises a battery module, a DC-DC converter, a super capacitor module and a pulse load, the battery module is connected to a DC bus through the DC-DC converter to supply power to the pulse load, and the super capacitor module is connected in parallel to the DC bus; The system data comprises battery cell state of charge, battery cell temperature, battery cell output current, super capacitor cell state of charge, super capacitor cell temperature, super capacitor cell output current, DC bus voltage and DC-DC converter inductance current; A deep reinforcement learning model is constructed, and real-time control actions of the pulse power supply system are generated based on the system data and the deep reinforcement learning model; A nonlinear model predictive controller is constructed, and real-time guide actions of the pulse power supply system are generated based on the system data and the nonlinear model predictive controller; It is judged whether the real-time control actions meet safety constraints: If yes, a safety control action is calculated based on the real-time control actions, the real-time guide actions and a first risk action shielding function, and a reward value of the deep reinforcement learning model is calculated based on a first reward function; Otherwise, a safety control action is calculated based on the real-time control actions, the real-time guide actions and a second risk action shielding function, and a reward value of the deep reinforcement learning model is calculated based on a second reward function; The pulse power supply system is controlled based on the safety control action.

2. The method of claim 1, wherein, The process of obtaining the real-time control action is: inputting the current collected battery monomer state of charge, the battery monomer temperature, the super capacitor monomer state of charge, the super capacitor monomer temperature and the pulse load output current as current state into the deep reinforcement learning model to obtain an initial control action wherein the initial control action comprises current values of the battery monomer and the super capacitor monomer; Gaussian noise is added to the initial control action to obtain the real-time control action, which is: wherein for real-time control of actions, is the standard deviation of the Gaussian noise.

3. The method of claim 2, wherein, The process of judging whether the real-time control action meets safety constraints is: The current values of the battery cells and the super capacitor cells corresponding to the real-time control action are substituted into the pulse power supply system for simulation calculation, and it is judged whether the simulation results meet the battery SOC upper limit constraint, the battery SOC lower limit constraint, the super capacitor SOC upper limit constraint, the super capacitor SOC lower limit constraint and the super capacitor voltage upper limit constraint at the same time, if yes, it means that the real-time control action meets the safety constraints, otherwise, it does not meet the safety constraints.

4. The control method of the pulse-powered system based on security-depth reinforcement learning according to claim 1 or 2 or 3, characterized in that, When the real-time control action meets the safety constraints, the process of calculating the safety control action based on the first risk action shielding function is: First, in real-time control action , real-time guidance action Iterative search for critical safety action within the search space of the upper and lower limits ; Second, based on the critical safety action with real-time guidance action computing safety control action for: wherein denotes the coefficient of expansion, and satisfies ; When the real-time control action does not meet the safety constraints, the process of calculating the safety control action based on the second risk action shielding function is: wherein denotes the coefficient of expansion, satisfies .

5. The control method of the pulse-powered system based on security-depth reinforcement learning according to claim 1 or 2 or 3, characterized in that, The first reward function and the second reward function are: wherein, is a bus voltage fluctuation reward, is a temperature rise suppression reward, is a safety constraint penalty term, is a real-time action penalty term; The bus voltage fluctuation reward is: wherein, is a bus voltage fluctuation, is a bus voltage, is a bus reference voltage, , is an upper limit, a lower limit of the bus voltage fluctuation, , is a reward constant; The temperature rise suppression reward is: wherein, respectively represent a battery cell LIB or a supercapacitor cell SC, is the temperature of the battery cell LIB or the supercapacitor cell SC, t is the time, is the total heat generation of the battery cell LIB or the supercapacitor cell SC during the charging and discharging process, , is a weight coefficient; The safety constraint penalty term is: wherein is a penalty for constraining , is a penalty for constraining , ; is a system safety constraint matrix, , is a state of charge of the battery cell LIB, of the super capacitor cell SC, , is an upper limit, a lower limit of the state of charge of the battery cell LIB, , is an upper limit, a lower limit of the state of charge of the super capacitor cell SC, is an end voltage of the super capacitor cell SC, is an upper limit of the end voltage of the super capacitor cell SC; In the first reward function, the real-time action penalty term is: wherein, is a real-time control action, is a safety control action, is a first proportional coefficient; In the second reward function, the real-time action penalty term is: wherein is a second proportionality coefficient, and .

6. The control method of the pulse-powered system based on security-depth reinforcement learning according to claim 1 or 2 or 3, characterized in that, The process of controlling the pulse power supply system based on the safety control action is: The safety control action is first compared with the actually measured current values of the battery monomer and the super capacitor monomer as a reference value, and a voltage reference signal of an outer ring is generated after a PID controller ; Secondly, the outer loop voltage reference signal and the actual measured DC bus voltage passes through a PID controller to generate an inner loop current reference signal ; Finally, the inner loop current reference signal and the measured DC-DC converter inductor current The duty cycle is generated through the PID controller to obtain the switching control signals S1 and S2 of the DC-DC converter, and the DC-DC converter is controlled based on the control signals S1 and S2.

7. The control method of the pulse-powered system based on security-depth reinforcement learning according to claim 1 or 2 or 3, characterized in that, The process of establishing the equivalent analysis model comprises: The load is modeled as a pulse current source, which is: wherein, is the load current, , is the maximum and minimum of the load current, is the pulse period, is the number of pulses, is the duty cycle of the pulses, t is time; An equivalent electrical model is constructed, which is: wherein, respectively represent a battery cell LIB or a supercapacitor cell SC, is the end voltage of the battery cell LIB or the supercapacitor cell SC, is the open circuit voltage of the battery cell LIB or the supercapacitor cell SC, is the current of the battery cell LIB or the supercapacitor cell SC, is the equivalent series resistance of the battery cell LIB or the supercapacitor cell SC, is the equivalent parallel resistance of the battery cell LIB or the supercapacitor cell SC, is the time constant of the RC element in the first order RC circuit, is the parallel capacitance of the battery cell LIB or the supercapacitor cell SC; An equivalent thermal model is constructed, which comprises: wherein, Pdiss is the dissipated power of the battery cell LIB or supercapacitor cell SC; Pdiss is the dissipated power of the battery cell LIB or supercapacitor cell SC; Tamb is the temperature of the surrounding medium; wherein, is the total heat generation of the battery cell LIB during the charging and discharging process, is the entropy heat coefficient of the battery cell LIB; wherein, Qtotal is the total heat generated by the supercapacitor cell SC during the charging and discharging process; The temperature rise power is: wherein, Cth is the heat capacity parameter of the battery cell LIB or supercapacitor cell SC.

8. A safe deep reinforcement learning based control system for an impulse power supply system, characterized in that, The control system comprises: an equivalent model construction unit configured to establish an equivalent analysis model of the pulse power supply system; a data acquisition unit configured to acquire system data of the pulse power supply system; a deep reinforcement learning model configured to generate a real-time control action of the pulse power supply system according to the system data; a nonlinear model predictive controller configured to generate a real-time guide action of the pulse power supply system according to the system data; a risk action shielding unit configured to determine whether the real-time control action satisfies a safety constraint; if yes, a safety control action is calculated based on the real-time control action, the real-time guide action and a first risk action shielding function, and a reward value of the deep reinforcement learning model is calculated based on a first reward function; otherwise, a safety control action is calculated based on the real-time control action, the real-time guide action and a second risk action shielding function, and a reward value of the deep reinforcement learning model is calculated based on a second reward function; a system control unit configured to control the pulse power supply system according to the safety control action.

Images