Intelligent flow battery control model construction method, control method and device

Abstract

The invention belongs to the technical field of flow batteries, and provides an intelligent flow battery control model construction method, an intelligent flow battery control method and an intelligent flow battery control device. Constructing a state space and a behavior space based on time steps based on the pre-processed parameters and preset given positive and negative electrode pump frequencies; calculating an instant reward based on a predefined reward function; constructing a state transition matrix according to the state space, the behavior space, the instant reward and the next state space; wherein the next state space is obtained through the state space and the behavior space; training a control model based on the state transition matrix and a predefined optimization objective function to obtain a constructed control model; an instant reward is calculated through a predefined reward function, so that the control model can accurately evaluate the effect of the current action at each time step, and instant feedback is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of liquid flow batteries, and in particular relates to an intelligent liquid flow battery control model construction method, a control method and a device. Background Art

[0002] Liquid flow battery is a new type of electrochemical energy storage device, whose main features are high energy conversion efficiency, environmental friendliness, good safety performance, deep discharge, etc. However, the performance of liquid flow battery is affected by many factors, such as electrolyte temperature, tank and pipeline pressure, stack flow rate, tank liquid level, stack voltage, stack current, pump frequency feedback, cumulative charge capacity, cumulative discharge capacity, charge and discharge efficiency, etc. Therefore, how to effectively control these parameters to improve the performance of liquid flow battery is an important issue facing current liquid flow battery technology.

[0003] The existing control methods of liquid flow batteries mainly adjust the operating state of liquid flow batteries by manually setting parameters or using simple control algorithms, such as PID control. However, these methods often cannot achieve precise control of liquid flow batteries, nor can they achieve coordinated optimization of multiple parameters, such as electrolyte temperature, tank and pipeline pressure, stack flow rate, tank liquid level, stack voltage, stack current, pump frequency feedback, cumulative charge capacity, cumulative discharge capacity, charge and discharge efficiency, etc., which limits the performance of the battery. Summary of the invention

[0004] In view of the problems in the background technology, the present invention proposes a smart liquid flow battery control model construction method, a control method and a device.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present disclosure proposes a method for constructing a smart flow battery control model based on reinforcement learning, comprising:

[0007] Obtain various parameters of the liquid flow battery operation state, and construct a time-step-based state space and behavior space based on the preprocessed parameters and the preset positive and negative electrode pump frequencies;

[0008] Calculating an immediate reward based on a predefined reward function; wherein the reward function includes a reward function within each time step; and the immediate reward is calculated based on the reward function within each time step;

[0009] Constructing a state transfer matrix according to the state space, the behavior space, the immediate reward and the next state space; wherein the next state space is obtained through the state space and the behavior space;

[0010] The control model is trained based on the state transfer matrix and a predefined optimization objective function to obtain a constructed control model.

[0011] Preferably, the method for preprocessing the parameters is normalization.

[0012] Preferably, the method further comprises: taking restrictive measures on the behavior space.

[0013] Preferably, the reward function also includes a reward function for each charge and discharge cycle;

[0014] The method further comprises:

[0015] Calculate the total reward through the reward function of each charge and discharge cycle;

[0016] The model is evaluated and adjusted based on the total reward.

[0017] Preferably, the method further comprises: folding the state space on a timeline.

[0018] Preferably, the method further comprises: training the control model through a deep learning network.

[0019] Preferably, the training of the control model based on the state transfer matrix and a predefined optimization objective function includes:

[0020] Cut the state transfer matrix into pieces to obtain a plurality of sub-matrices;

[0021] The sub-matrix is ​​randomly selected, and the control model is trained using a backtracking learning method and a predefined optimization objective function.

[0022] In a second aspect, the present disclosure proposes a smart flow battery construction device based on reinforcement learning, comprising:

[0023] A state space and behavior space construction module is used to obtain various parameters of the flow battery operation state, and to construct a state space and behavior space based on a time step based on the preprocessed parameters and the preset positive and negative electrode pump frequencies;

[0024] An immediate reward building block for calculating immediate rewards based on a predefined reward function;

[0025] A state transfer matrix construction module, used to construct a state transfer matrix according to the state space, the behavior space, the immediate reward and the next state space; wherein the next state space is obtained through the state space and the behavior space;

[0026] The control model construction module is used to train the control model based on the state transfer matrix and a predefined optimization objective function to obtain a constructed control model.

[0027] In a third aspect, the present disclosure proposes a smart flow battery control method based on reinforcement learning, comprising:

[0028] Obtain various parameters of the operating status of the flow battery; wherein the parameters include: stack electrolyte temperature, stack pressure, stack flow rate, tank liquid level, pump frequency feedback, stack voltage and stack current;

[0029] Inputting the parameters into the control model constructed in the first aspect to obtain the positive and negative electrode pump frequency settings;

[0030] According to the given positive and negative electrode pump frequencies, the pump frequency of the flow battery is adjusted.

[0031] In a fourth aspect, the present disclosure proposes an intelligent liquid flow battery control device based on reinforcement learning, comprising:

[0032] A parameter acquisition module is used to acquire various parameters of the operating state of the flow battery; wherein the parameters include: stack inlet electrolyte temperature, stack inlet pressure, stack inlet flow, tank liquid level, pump frequency feedback, stack voltage and stack current;

[0033] A parameter analysis module, used for inputting the parameters into the control model constructed in the first aspect to obtain the positive and negative electrode pump frequency settings;

[0034] The execution module is used to adjust the pump frequency of the flow battery according to the positive and negative electrode pump frequencies.

[0035] Beneficial effects of the present invention:

[0036] 1. The method of the present invention calculates the instant reward through a predefined reward function, so that the control model can accurately evaluate the effect of the current action at each time step and provide instant feedback. It can not only help the control model to quickly adjust the strategy, but also comprehensively consider multiple performance indicators (such as charging amount, discharging amount, number of faults, number of warnings, etc.), avoiding the limitations brought by the optimization of a single indicator, thereby improving learning efficiency and decision-making accuracy.

[0037] 2. The method of the present invention constructs a state transfer matrix, which records the state, action, reward and state of the next time step at each time step, forming a complete state transfer history. It not only simplifies complex problems, making it easier for the control model to understand and process system behavior, but also enhances its dynamic adaptability and generalization ability, so that the control model can dynamically adapt to changes in the system and adjust the strategy in time to ensure efficient operation under different working conditions. Through the synergy of immediate rewards and state transfer matrices, it is ensured that the control model not only focuses on short-term effects, but also takes into account long-term effects, achieving a global optimal solution, thereby significantly improving the performance and reliability of the flow battery energy storage system.

[0038] 3. The method of the present invention monitors the key operating parameters of the flow battery in real time (such as electrolyte temperature, tank and pipeline pressure, stack inlet flow, tank liquid level, stack voltage, stack current, pump frequency feedback, cumulative charge capacity, cumulative discharge capacity and charge and discharge efficiency), and analyzes these operating parameters in combination with a trained control model. It can realize real-time monitoring of the operating status of the flow battery and dynamically adjust the pump frequency, and can achieve maximum system-level charge and discharge efficiency, thereby improving the energy utilization efficiency of the flow battery, reducing energy waste, and improving the energy storage capacity of the flow battery. It can achieve maximum balance between charge and discharge efficiency and discharge capacity, thereby increasing the service life of the flow battery and reducing maintenance costs.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become obvious from the description, or be understood by implementing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flow chart showing a method for constructing a smart flow battery control model based on reinforcement learning according to the present invention is shown;

[0042] Figure 2 A framework diagram of a smart liquid flow battery construction device based on reinforcement learning of the present invention is shown;

[0043] Figure 3 A flow chart of a smart liquid flow battery control method based on reinforcement learning of the present invention is shown;

[0044] Figure 4 A framework diagram of an intelligent liquid flow battery control device based on reinforcement learning of the present invention is shown;

[0045] Figure 5 A schematic diagram of the device structure of the present invention is shown. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Reference Figure 1 As shown, a method for constructing an intelligent liquid flow battery control model based on reinforcement learning specifically includes the following steps:

[0048] S11, obtaining various parameters of the liquid flow battery operation state, and constructing a state space and a behavior space based on a time step based on the preprocessed parameters and the preset positive and negative electrode pumping frequencies;

[0049] Specifically, the parameters include: stack inlet electrolyte temperature (T), stack inlet pressure (P), stack inlet flow (F), tank liquid level (L), pump frequency feedback (QB), stack voltage (V) and stack current (C). It is understandable that these parameters can be either historical data extracted from past operation records or current data collected instantly through a real-time monitoring system.

[0050] Let po represent the positive pole of the system, ne represent the negative pole of the system, and t represent the time step (t∈N+, the time granularity of the time step depends on the shortest period of data acquisition), then:

[0051] The electrolyte inlet temperature (T) uses the data of two temperature measurement points of the electrolyte positive and negative electrode main pipes in a process system as the state evaluation point, namely:

[0052] The stack inlet pressure (P) uses the data of two pressure measurement points in the positive and negative main pipes of the electrolyte in a process system and their difference as the state evaluation point, namely: in, Indicates pressure difference;

[0053] The stack inflow flow (F) uses the data of two flow measurement points in the positive and negative main pipes of the electrolyte in a process system as the state evaluation points, namely:

[0054] The stack tank liquid level (L) uses the liquid level measurement point data of the positive and negative electrolyte tanks in a set of process systems as the status evaluation point, namely:

[0055] Pump frequency feedback (QB) uses the liquid level measurement point data of the positive and negative electrolyte storage tanks in a set of process systems as the status evaluation point, namely:

[0056] The stack voltage (V) is the real-time voltage of each stack. The total number of stacks in a process system is k, and the voltage status evaluation point of a process system is:

[0057] The stack current (C) is the real-time current of each stack. The total number of stacks in a process system is k, and the voltage state evaluation point of a process system is:

[0058] The method for preprocessing the parameters is data normalization. Specifically, in order to eliminate the influence of the original data magnitude on the algorithm, a homogenization algorithm is used to unify the dimensions, that is, data normalization, and its expression is: In the formula, x represents the real-time value of any state evaluation point, L scal Indicates the lower limit of the evaluation point range (reference collection point instrument range), H scal Indicates the upper limit of the evaluation point range (reference collection point instrument range), X scal Represents the result after normalization of x data.

[0059] After preprocessing, the state space based on time steps is:

[0060]

[0061] In the formula, scal represents data normalization processing, and t represents the time step.

[0062] The behavior space (A) of a flow battery energy storage process system includes: positive and negative electrode pump frequency settings (QS). Let po represent the positive electrode of the system, ne represent the negative electrode of the system, and t represent the time step (t∈N+, consistent with the time step of the state space), then the behavior space based on the time step is:

[0063]

[0064] In the formula, Indicates the lower limit of the positive and negative pump frequency (reference pump frequency control output range), Indicates the upper limit value of the positive and negative pump frequency setting (reference pump frequency control quantity output range), and the positive and negative pump frequency control quantity output ranges remain consistent.

[0065] As a preferred embodiment of the present disclosure, in order to prevent the difference between the output time steps of the pump frequency control from being too large, or the pump frequency being set too small to cause the inability to charge and discharge, or the pump frequency being set too large to cause the pump to be damaged, it is necessary to take restrictive measures on the behavior space, that is, to take restrictive measures on the control of the pump frequency at each time step, and the expression is:

[0066]

[0067] In the formula, a is the given maximum allowable pump frequency value; d is the given minimum allowable pump frequency value; b is the given maximum value of the pump frequency difference between steps; c is the given minimum value of the pump frequency difference between steps. t The algorithm calculates the pump frequency value for the current time step t, is the actual pump frequency value at the previous time step t-1, Calculate the pump frequency value for the constraint at the current time step t, is the final execution pump frequency value of the current time step t;

[0068] Specifically, the meaning of the expression of restriction measures is:

[0069] If the calculated pump frequency value A at the current time step t t Less than the minimum allowable pump frequency value d, or the current If it is less than the minimum allowable pump frequency value d, Set to d. This ensures that the pump frequency is not too low, preventing the situation where charging and discharging cannot occur.

[0070] If the calculated pump frequency value A at the current time step t t Greater than the maximum allowable pump frequency value a, or the current If it is greater than the maximum allowable pump frequency value a, Set to a. This ensures that the pump frequency is not too high, thus damaging the system.

[0071] If the calculated pump frequency value A at the current time step t t The pump frequency execution value at the previous time step t-1 If the difference is less than -, Reduce b. It can prevent the pump frequency from suddenly dropping too much, which will damage the stability and life of the pump itself, pipelines and stack.

[0072] If the calculated pump frequency value A at the current time step t t The pump frequency execution value at the previous time step t-1 If the difference is greater than b, Increase b. It can prevent the pump frequency from rising too much suddenly, which will damage the stability and life of the pump itself, pipelines and stack.

[0073] If the calculated pump frequency value A at the current time step t tThe pump frequency given value at the previous time step t-1 If the difference is between - and c, then keep The difference in pump frequency between steps is too small, and the impact on the system can be ignored, and the optimization degree of the algorithm can also be ignored. This constraint can ensure the stability of the algorithm and reduce unnecessary action overhead.

[0074] If the calculated pump frequency value A at the current time step t t The pump frequency execution value at the previous time step t-1 If the difference is between [-,-c] or [c,b], execute A t The pump frequency value within this range does not affect the safety and stability of the system, and can be directly executed without any constraint changes. It is a direct and effective action execution step of the system and algorithm.

[0075] S12, calculating an immediate reward based on a predefined reward function;

[0076] Specifically, the reward function (R) of a flow battery energy storage process system is defined as: the system maximum capacity (MaxC) AC side cumulative charge amount (CG), the AC side cumulative discharge amount (DCG), the AC side charge and discharge efficiency (EF), the AC side system efficiency (EP) including the pump energy consumption, the system alarm (ALM), and the related functions of the fault (ER). Among them, the AC side charge amount (CG) and discharge amount (DCG) of the energy storage converter are obtained by the statistics of the energy storage converter or the electricity meter, and the pump energy consumption is measured by the total power supply meter of the pump system; the charge amount and discharge amount in a single cycle are normalized:

[0077]

[0078] Where t represents the time step, CG t -CG t-1 、DCG t -DCG t-1 is the actual charge and discharge amount of the system between time steps, normalized to the (0,1) interval according to the ratio of the system maximum capacity MaxC.

[0079] The calculation method of the AC side charging and discharging efficiency (EF) of the energy storage converter is:

[0080]

[0081] Among them, τ represents the charge and discharge cycle step (τ∈N+, each τ contains multiple time steps t), and the charge and discharge amount is calculated based on the cumulative charge / discharge difference at the end of each charge and discharge cycle and the end of the next cycle, and the charge and discharge efficiency EF under the cycle is calculated τ .

[0082] The calculation method of AC side system efficiency (EP) taking into account pump energy consumption is:

[0083]

[0084] in, Indicates the pump energy consumption at the end of the current charge and discharge cycle, Indicates the pump energy consumption at the end of the last charge and discharge cycle. The ratio of the charge amount to the remaining discharge amount plus the pump energy consumption is used as the AC side system efficiency EP τ .

[0085] System alarms and faults are set according to the number. Assume that there are at most M system alarms and K system faults, normalized to the interval (0,1), and the calculation method is:

[0086]

[0087] In the formula, m t , k t Indicates the number of alarms and faults actually generated by the system at time step t

[0088] Since efficiency calculation can only be performed in a complete charge-discharge cycle, one charge-discharge cycle contains several time steps, and the system selects an action from the action space for execution at each time step. A relatively immediate reward function is required to evaluate the pros and cons of the selection, thereby guiding the selection and execution of the next action. Therefore, the reward function calculation method is as follows:

[0089]

[0090] In the formula, R t is the reward function for each time step, R τ is the reward function for each charge-discharge cycle, is the weight coefficient. By setting different weight coefficient values, the focus indicators of the reward function and the balance relationship between various indicator quantities can be adjusted. T is the total number of time steps contained in each charge and discharge cycle.

[0091] Specifically, the meaning of the expression of the reward function is: through R t Calculate the immediate reward and use R at each step of execution t As an optimization function, R is accumulated in a complete charge and discharge cycle t , get the total reward, and assign the total reward to R after the cycle ends t , and is carried into the next charge-discharge cycle for iterative optimization calculation. The total reward is used to evaluate the performance of the entire charge-discharge cycle and is used as a reference to adjust and evaluate model parameters.

[0092] S13, constructing a state transfer matrix according to the state space, the behavior space, the immediate reward and the next state space; wherein the next state space is obtained through the state space and the behavior space;

[0093] Specifically, the state space (S) is used as input, the behavior space (A) is used as the state transfer fulcrum, the next state space (S_), the reward function is calculated immediately (R), and S_ is used as the next input S, and the cycle is repeated to form a state transfer matrix:

[0094] ((S0,A0,R0,S1),(S1,A1,R1,S2),(S2,A2,R2,S3),…)

[0095] Among them, the next state space is based on the current state S t and the action taken A t Sure.

[0096] As a preferred embodiment of the present disclosure, the state space can be folded on the timeline to optimize the problem of insufficient state change caused by slowly varying physical quantities such as pressure and temperature, specifically:

[0097]

[0098] Where y represents the length of the time step fold.

[0099] By merging the states of multiple consecutive time steps into a collapsed state It can capture the changing trends of slowly varying physical quantities (such as pressure and temperature) more comprehensively.

[0100] S14: training a control model based on the state transfer matrix and a predefined optimization objective function to obtain a constructed control model.

[0101] Specifically, the control model is trained by a deep reinforcement learning algorithm, such as: Q-learning, Deep QNetwork, Double DeepQNetwork, Dueling Deep QNetwork, Deep DeterministicPolicy Gradient, Advantage Actor Critic, Proximal Policy Optimization, Distributed Proximal Policy Optimization, etc.

[0102] This disclosure takes minimizing the reward function as the optimization goal of the algorithm, and aims to maximize the charging amount, discharging amount, AC side efficiency, and pump consumption AC side efficiency through optimizing the control strategy, and minimize faults and warnings, so as to achieve the purpose of optimizing control, and take into account the impact of any behavior on the future. The expression is:

[0103]

[0104] Where η represents the discount factor.

[0105] The method disclosed in the present invention calculates the instant reward through a predefined reward function, so that the control model can accurately evaluate the effect of the current action at each time step and provide instant feedback, which can not only help the control model to quickly adjust the strategy, but also comprehensively consider multiple performance indicators (such as charging amount, discharging amount, number of faults, number of warnings, etc.), avoiding the limitations brought by the optimization of a single indicator, thereby improving learning efficiency and decision accuracy. In addition, a state transfer matrix is ​​constructed according to the state space, behavior space, instant reward and next state space. The state transfer matrix records the state, action, reward and state of the next time step at each time step, forming a complete state transfer history, which not only simplifies complex problems and makes it easier for the control model to understand and process system behavior, but also enhances its dynamic adaptability and generalization ability, so that the control model can dynamically adapt to changes in the system, adjust the strategy in time, and ensure that it can maintain efficient operation under different working conditions. Through the synergistic effect of instant rewards and state transfer matrices, it is ensured that the control model not only pays attention to short-term effects, but also takes into account long-term effects, and achieves the global optimal solution, thereby significantly improving the performance and reliability of the liquid flow battery energy storage system.

[0106] The disclosure supports normalizing other related indicators not listed in this article to the (0,1) interval and incorporating them into the reward function through a weight system to optimize these indicators. For example, parameters such as the pressure balance at the positive and negative electrode inlets and the pressure balance in the positive and negative electrode storage tanks can be optimized. Specifically, these additional indicators are normalized using the average variance of the pressure and used as part of the reward function to help the model more comprehensively evaluate and optimize the overall performance of the system.

[0107] In the present disclosure, all time steps t involved refer to the same thing, and all charge-discharge cycles τ involved refer to the same thing. However, it is not specified that the actual time intervals are exactly the same, and the present disclosure is applicable to time steps and charge-discharge cycles with any different actual time intervals.

[0108] Reference Figure 2 As shown, based on the same inventive concept, the present disclosure also proposes a smart flow battery construction device based on reinforcement learning, comprising:

[0109] The state space and behavior space construction module 110 is used to obtain various parameters of the flow battery operation state, and to construct a state space and behavior space based on a time step based on the preprocessed parameters and the preset positive and negative electrode pumping frequencies;

[0110] An instant reward building module 120, for calculating an instant reward based on a predefined reward function;

[0111] A state transfer matrix construction module 130, configured to construct a state transfer matrix according to the state space, the behavior space, the immediate reward, and the next state space; wherein the next state space is obtained through the state space and the behavior space;

[0112] The control model construction module 140 is used to train the control model based on the state transfer matrix and a predefined optimization objective function to obtain a constructed control model.

[0113] Reference Figure 3 As shown, the present disclosure also proposes a smart flow battery control method based on reinforcement learning, including:

[0114] S21, obtaining various parameters of the operating state of the flow battery; wherein the parameters include: battery stack inlet electrolyte temperature, battery stack inlet pressure, battery stack inlet flow rate, tank liquid level, pump frequency feedback, battery stack voltage and battery stack current;

[0115] S22, inputting the parameters into the control model constructed above to obtain the positive and negative electrode pump frequency settings;

[0116] S23. According to the given positive and negative electrode pump frequencies, the pump frequency of the flow battery is adjusted to achieve the maximum system-level charge and discharge efficiency, the maximum discharge capacity, and the maximum balanced charge and discharge efficiency and discharge capacity.

[0117] Reference Figure 4 As shown, based on the same inventive concept, the present disclosure also proposes an intelligent liquid flow battery control device based on reinforcement learning, comprising:

[0118] The parameter acquisition module 210 is used to acquire various parameters of the operating state of the flow battery; wherein the parameters include: stack inlet electrolyte temperature, stack inlet pressure, stack inlet flow rate, tank liquid level, pump frequency feedback, stack voltage and stack current;

[0119] The parameter analysis module 220 is used to input the parameters into the control model constructed above to obtain the positive and negative electrode pump frequency settings;

[0120] The execution module 230 is used to adjust the pump frequency of the flow battery according to the positive and negative electrode pump frequencies.

[0121] By real-time monitoring of the key operating parameters of the flow battery (such as electrolyte temperature, tank and pipeline pressure, stack inlet flow, tank liquid level, stack voltage, stack current, pump frequency feedback, cumulative charge capacity, cumulative discharge capacity and charge and discharge efficiency), and analyzing these operating parameters in combination with the trained control model, it is possible to achieve real-time monitoring of the operating status of the flow battery and dynamically adjust the pump frequency, thereby achieving maximum system-level charge and discharge efficiency, thereby improving the energy utilization efficiency of the flow battery, reducing energy waste, and improving the energy storage capacity of the flow battery, and achieving maximum balance between charge and discharge efficiency and discharge capacity, thereby increasing the service life of the flow battery and reducing maintenance costs.

[0122] Reference Figure 5 As shown, based on the same inventive concept, the present disclosure also proposes a device, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and when the processor runs the computer instructions, it executes the above-mentioned reinforcement learning-based intelligent liquid flow battery control model construction method and / or control method.

[0123] Based on the same inventive concept, the present disclosure also proposes a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the above-mentioned intelligent liquid flow battery control model construction method and / or control method based on reinforcement learning can be implemented.

[0124] Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0125] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0126] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an intelligent liquid flow battery control model based on reinforcement learning, characterized in that: include: Obtain various parameters of the liquid flow battery operation state, and construct a time-step-based state space and behavior space based on the preprocessed parameters and the preset positive and negative electrode pump frequencies; Calculating an immediate reward based on a predefined reward function; wherein the reward function includes a reward function within each time step; and the immediate reward is calculated based on the reward function within each time step; Constructing a state transfer matrix according to the state space, the behavior space, the immediate reward and the next state space; wherein the next state space is obtained through the state space and the behavior space; The control model is trained based on the state transfer matrix and a predefined optimization objective function to obtain a constructed control model.

2. According to claim 1, a method for constructing a smart flow battery control model based on reinforcement learning is characterized in that: The method of preprocessing the parameters is normalization.

3. The method for constructing a smart flow battery control model based on reinforcement learning according to claim 1, characterized in that: The method further includes: taking restrictive measures on the behavior space.

4. The method for constructing a smart flow battery control model based on reinforcement learning according to claim 1, characterized in that: The reward function also includes a reward function for each charge and discharge cycle; The method further comprises: Calculate the total reward through the reward function of each charge and discharge cycle; The model is evaluated and adjusted based on the total reward.

5. The method for constructing a smart flow battery control model based on reinforcement learning according to claim 1, characterized in that: The method further includes folding the state space on a timeline.

6. The method for constructing a smart flow battery control model based on reinforcement learning according to claim 1, characterized in that: The method also includes: training the control model through a deep learning network.

7. The method for constructing a smart flow battery control model based on reinforcement learning according to claim 1, characterized in that: The training of the control model based on the state transfer matrix and a predefined optimization objective function includes: Cut the state transfer matrix into pieces to obtain a plurality of sub-matrices; The sub-matrix is ​​randomly selected, and the control model is trained using a backtracking learning method and a predefined optimization objective function.

8. A smart flow battery construction device based on reinforcement learning, characterized in that: include: A state space and behavior space construction module is used to obtain various parameters of the flow battery operation state, and to construct a state space and behavior space based on a time step based on the preprocessed parameters and the preset positive and negative electrode pump frequencies; An immediate reward building block for calculating immediate rewards based on a predefined reward function; A state transfer matrix construction module, used to construct a state transfer matrix according to the state space, the behavior space, the immediate reward and the next state space; wherein the next state space is obtained through the state space and the behavior space; The control model construction module is used to train the control model based on the state transfer matrix and a predefined optimization objective function to obtain a constructed control model.

9. A smart flow battery control method based on reinforcement learning, characterized in that: include: Obtain various parameters of the operating status of the flow battery; wherein the parameters include: stack electrolyte temperature, stack pressure, stack flow rate, tank liquid level, pump frequency feedback, stack voltage and stack current; Inputting the parameters into the control model constructed according to any one of claims 1 to 7 to obtain the positive and negative electrode pump frequency settings; According to the given positive and negative electrode pump frequencies, the pump frequency of the flow battery is adjusted.

10. An intelligent liquid flow battery control device based on reinforcement learning, characterized in that: include: A parameter acquisition module is used to acquire various parameters of the operating state of the flow battery; wherein the parameters include: stack inlet electrolyte temperature, stack inlet pressure, stack inlet flow, tank liquid level, pump frequency feedback, stack voltage and stack current; A parameter analysis module, used for inputting the parameters into the control model constructed according to any one of claims 1 to 7 to obtain the positive and negative electrode pump frequency settings; The execution module is used to adjust the pump frequency of the flow battery according to the positive and negative electrode pump frequencies.

Images