Method and device for controlling energy storage converter based on artificial intelligence

Abstract

The invention relates to the technical field of data acquisition and control, in particular to an energy storage converter control method and device based on artificial intelligence, and the method comprises the steps: inputting a real-time state at a current moment into a reinforcement learning model, obtaining a target action of an energy storage converter, the load demand and the power grid frequency of the power grid side are determined; and executing the target action at the current moment to realize the control of the energy storage converter. According to the technical scheme, the control precision of the energy storage converter in various operation states can be improved.

Description

Technical Field

[0001] This application relates to the field of data acquisition and control technology, and in particular to an artificial intelligence-based energy storage converter control method and device. Background Technology

[0002] Energy storage converters are an indispensable and important component of energy storage systems. They connect the energy storage power source and the power grid, converting direct current (DC) to alternating current (AC) or vice versa, to meet the grid's charging and discharging requirements for the energy storage system and ensure the efficient and stable operation of the energy storage power source.

[0003] Currently, patent application CN108134409A discloses a control method, device, storage medium, and processor for an energy storage converter. The method includes: acquiring the active droop characteristics and reactive droop characteristics of the energy storage converter; detecting the real-time output active power and reactive power of the energy storage converter; and adjusting the frequency and voltage of the microgrid system where the energy storage converter is located based on the active droop characteristics, reactive droop characteristics, active power, and reactive power.

[0004] The above method regulates the frequency and voltage of the energy storage converter by monitoring its operating status. The operating status of the energy storage converter includes active power droop characteristics, reactive power droop characteristics, and real-time output active and reactive power. However, the above method ignores the impact of different operating states of the energy storage converter on the control accuracy, and cannot guarantee the control accuracy of the energy storage converter under each operating state, resulting in poor control accuracy of the energy storage converter. Summary of the Invention

[0005] To address the technical problem of poor control accuracy in energy storage converters, this application provides an artificial intelligence-based control method and device for energy storage converters, which can improve the control accuracy of energy storage converters under various operating conditions.

[0006] In a first aspect, this application provides an artificial intelligence-based control method for an energy storage converter. The control method includes: inputting the real-time state of the current moment into a reinforcement learning model to obtain a target action of the energy storage converter, the target action including charging power and discharging power; the real-time state including the remaining power and health of the energy storage power source, as well as the load demand and grid frequency on the grid side; executing the target action of the current moment to achieve control of the energy storage converter; wherein the reward value of the reinforcement learning model is a weighted sum of multiple sub-reward values, and the update method of the reinforcement learning model includes: collecting reward vectors of each historical state in an experience pool, the reward vectors including each sub-reward value; clustering the historical states to obtain multiple state clusters; obtaining the predicted state for the next moment; determining a target cluster based on the similarity between the predicted state and each state cluster; and selecting training samples from the target cluster based on the reward vectors of the historical states to update the reinforcement learning model.

[0007] Preferably, the input of the reinforcement learning model is the real-time state at any time, and the output is a set of reward values ​​at that time. The set of reward values ​​includes the reward values ​​of each preset action, and the preset action corresponding to the maximum reward value corresponds to the target action.

[0008] Preferably, the sub-reward values ​​include power conversion reward, frequency consistency reward, and load supply reward; the power conversion reward is the ratio of the power connected to the grid to the power consumed by the energy storage power source; the frequency consistency reward is negatively correlated with the difference between the grid frequency and the preset frequency after the target action is performed; the load supply reward is negatively correlated with the difference between the power supply and load demand on the grid side after the target action is performed.

[0009] Preferably, selecting training samples from the target cluster based on the reward vector of historical states includes: calculating the attention level of each sub-reward value at the current time, and the sub-reward value... attention for:

[0010] , The sub-reward value after performing the target action at the current moment. The average sub-reward value over a preset time period prior to the current moment. The Sigmoid activation function is used; the sub-reward values ​​in the reward vector are weighted and summed based on the attention level to obtain the selection score of each training sample in the target cluster; the first M training samples are selected in descending order of selection score to update the reinforcement learning model.

[0011] Preferably, historical moments The selection score of the training samples for:

[0012] , For any sub-reward value, The set of all sub-reward values. For child reward value attention, The sum of attention received for all sub-reward values. For a historic moment Sub-reward values ​​in training samples The value of .

[0013] Preferably, the step of clustering historical states to obtain multiple state clusters includes: using the Euclidean distance between historical states as the clustering distance, and using the K-means algorithm to cluster the historical states to obtain multiple clusters, wherein the multiple clusters correspond to multiple state clusters; wherein the number of clusters is determined using the elbow method.

[0014] Preferably, obtaining the predicted state at the next moment includes: obtaining the time series of each dimension of the real-time state within a historical time period; performing polynomial fitting on the time series to obtain the fitting curve of the dimension; inputting the next moment into the fitting curve to obtain the predicted value of the dimension at the next moment; and concatenating the predicted values ​​of all dimensions at the next moment to obtain the predicted state at the next moment.

[0015] Preferably, determining the target cluster based on the similarity between the predicted state and each state cluster includes: calculating the mean of all historical states in the state cluster to obtain the cluster center of the state cluster; calculating the similarity between the predicted state and the state cluster, and taking the state cluster corresponding to the maximum similarity as the target cluster, wherein the similarity is negatively correlated with the Euclidean distance between the predicted state and the cluster center.

[0016] Preferably, a training sample includes historical states. Adjacent historical states and historical status target action And multiple sub-reward values, updating the reinforcement learning model includes: historical states Input the reinforcement learning model to obtain the target action. Reward value ; to connect adjacent historical states Input the reinforcement learning model, output the reward value for each preset action, and record the maximum reward value as . The reward value for the training sample is obtained by weighted summation of multiple sub-reward values. Constructing the loss function for: ; , and For each sub-reward value, , and The preset weights for each sub-reward value, The adjustment coefficients are used to update the reinforcement learning model using gradient descent.

[0017] In a second aspect, this application also provides an artificial intelligence-based energy storage converter control device, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the artificial intelligence-based energy storage converter control method according to the first aspect of this application is implemented.

[0018] The technical solution of this application has the following beneficial technical effects:

[0019] By inputting the real-time status of the energy storage converter into the reinforcement learning model, the target action of the energy storage converter is directly output. Executing the target action can complete the control of the energy storage converter at the current moment. The reward value of the reinforcement learning model is a weighted sum of power conversion reward, frequency consistency reward and load supply reward, which constrains the control effect of the target action from different evaluation criteria.

[0020] Furthermore, since the remaining power, health status, grid load demand, and grid frequency are constantly changing in real-time, the reinforcement learning model needs to be updated after executing the target action to improve the control accuracy of the energy storage converter under various operating conditions. When updating the reinforcement learning model, it is first necessary to obtain the predicted state for the next time step. The state category for the next time step is determined by comparing the predicted state with the state clusters in the experience pool. Training samples are selected from the state clusters (i.e., target clusters) corresponding to the state category. During the selection of training samples, the values ​​of each sub-reward value in the training samples are taken into account to ensure the effectiveness of the training samples in updating the reinforcement learning model. This ensures that the updated reinforcement learning model can output accurate target actions in the next time step and achieve better control performance. Attached Figure Description

[0021] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0022] Figure 1 This is a flowchart of an artificial intelligence-based energy storage converter control method according to an embodiment of this application;

[0023] Figure 2 This is a flowchart of a reinforcement learning model update method according to an embodiment of this application;

[0024] Figure 3This is a structural block diagram of an artificial intelligence-based energy storage converter control device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that when the terms "first," "second," etc., are used in the claims, description, and drawings of this application, they are only used to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" used in the description and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0027] According to a first aspect of this application, this application provides an artificial intelligence-based energy storage converter control method. Figure 1 This is a flowchart of an artificial intelligence-based energy storage converter control method according to an embodiment of this application. Figure 1 As shown, the AI-based energy storage converter control method includes steps S101 to S103, which are described in detail below.

[0028] S101, collect the real-time status at the current moment. The real-time status includes the remaining power and health of the energy storage power source, as well as the load demand and grid frequency on the grid side.

[0029] In one embodiment, the real-time status reflects the operating status of the energy storage converter at the current moment; the real-time status includes the remaining power and health of the energy storage power source, as well as the load demand and grid frequency on the grid side.

[0030] The remaining power capacity reflects the percentage of currently available power in the energy storage power source, and can be used to determine charging and discharging needs. Its value range is [value missing]. arrive The health score is used to reflect the health level of the energy storage power source, and its value range is: arrive The load demand is used to reflect the current external load demand for electricity on the grid side; the grid frequency is used to reflect the voltage frequency and current frequency on the grid side. When the difference between the grid frequency and the preset frequency is large, the energy storage power supply needs to maintain the stability of the grid side voltage frequency and current frequency through charging and discharging.

[0031] The methods for obtaining the remaining power and health status of the energy storage power source are well-known to those skilled in the art and will not be elaborated here.

[0032] In this way, the remaining power and health status of the energy storage power source at the current moment, as well as the load demand and grid frequency on the grid side, are collected, accurately reflecting the real-time status of the energy storage converter.

[0033] S102, input the real-time state of the current moment into the reinforcement learning model to obtain the target action of the energy storage converter, the target action including charging power and discharging power.

[0034] In one embodiment, the input of the reinforcement learning model is the real-time state at any time, and the output is a set of reward values ​​at that time. The set of reward values ​​includes the reward values ​​of each preset action, and the preset action corresponding to the maximum reward value corresponds to the target action.

[0035] One preset action includes a charging power and a discharging power. The number of preset actions is related to the number of types of charging power and discharging power. For example, if there are 5 types of charging power and 6 types of discharging power, then by arranging and combining the charging power and discharging power, a total of 30 preset actions can be obtained.

[0036] Specifically, the reinforcement learning model is a DQN network, which includes an input layer, multiple hidden layers, and an output layer. The number of neurons in the input layer is equal to the dimension of the real-time state, and it is used to receive the real-time state. The multiple hidden layers are used to perform dimension transformation on the real-time state. The number of neurons in the output layer is equal to the number of preset actions, and it is used to map the dimension transformation result to the reward value of each preset action.

[0037] Thus, by inputting the real-time state obtained in step S101 into the reinforcement learning model, the target action at the current moment can be obtained, and the target action includes charging power and discharging power.

[0038] S103 executes the target action at the current moment to realize the control of the energy storage converter.

[0039] In one embodiment, the charging and discharging power of the energy storage converter are adjusted according to the target action to complete the control of the energy storage converter at the current moment.

[0040] Thus, for any given moment, the real-time state is input into the reinforcement learning model to obtain the target action, and the control of the energy storage converter at that moment is completed based on the target action. Therefore, the control accuracy of the energy storage converter is closely related to the reinforcement learning model; in order to ensure the control accuracy of the energy storage converter, the reinforcement learning model needs to be updated after the control at any given moment ends and before the control at the next adjacent moment begins, that is to say, the reinforcement learning model needs to be updated at every moment.

[0041] Please see Figure 2 This is a flowchart of a reinforcement learning model update method according to an embodiment of this application. Figure 2 As shown, the method for updating the reinforcement learning model includes steps S201 to S204, which are described in detail below.

[0042] S201, Collect the reward vectors of each historical state in the experience pool, wherein the reward vectors include each sub-reward value.

[0043] It should be noted that the reward value of the reinforcement learning model includes multiple sub-reward values, namely, power conversion reward, frequency consistency reward, and load supply reward. The power conversion reward is the ratio of the power supplied to the grid to the power consumed by the energy storage power source. The larger the power conversion reward, the higher the utilization rate of the energy storage power source and the better the control effect of the target action. The frequency consistency reward is negatively correlated with the difference between the grid frequency and the preset frequency after the target action is executed. That is, the larger the frequency consistency reward, the closer the grid frequency is to the preset frequency, and the better the control effect of the target action. The load supply reward is negatively correlated with the difference between the power supply and load demand after the target action is executed. That is, the larger the load supply reward, the closer the power supply is to the load demand, and the better the control effect of the target action.

[0044] Specifically, the reward value is a weighted sum of multiple sub-reward values. Satisfying the relation:

[0045] , and These are the preset weights for the energy conversion reward and the energy conversion bonus, respectively. and These are the preset weights for frequency consistency rewards and frequency consistency rewards, respectively. and These are the preset weights for load supply rewards and load supply rewards, respectively.

[0046] Understandably, a sub-reward value corresponds to an evaluation criterion for control effectiveness; reward value The overall control effect and reward value are obtained by combining multiple evaluation criteria. The larger the value, the better the overall control effect of the target action; , and The values ​​are all .

[0047] It should also be noted that experience replay is a common technique in the updating process of reinforcement learning models. Experience replay stores information such as historical states and target actions from multiple historical moments as training samples in an experience pool. Each time the target action is executed, a new training sample is added to the experience pool. For example, historical moments... The training samples are ,in, For a historic moment The historical state, For historical state The target action, To perform the target action Posthistoric Moment The historical state, , and For historical state Execute the target action Multiple sub-reward values ​​were subsequently collected. Among them, This is the reward value for energy conversion. The frequency consistency reward value. Provide reward values ​​to the load.

[0048] In one embodiment, the reward vector for each historical state in the experience pool is obtained, the reward vector including each sub-reward value; for each historical moment... Regarding the training samples, historical states The reward vector is .

[0049] In this way, we obtain the sub-reward value of the reinforcement learning model in each historical state; if a sub-reward value is larger, it means that the reinforcement learning model can achieve better results in the aspect corresponding to that sub-reward value in that historical state.

[0050] S202, clustering historical states to obtain multiple state clusters.

[0051] In one embodiment, clustering historical states to obtain multiple state clusters includes: using the Euclidean distance between historical states as the clustering distance, and using the K-means algorithm to cluster the historical states to obtain multiple clusters, wherein the multiple clusters correspond to multiple state clusters; wherein the number of clusters is determined using the elbow method.

[0052] Among them, the K-means algorithm is a distance-based clustering algorithm that can group historical states with small cluster distances into a single state cluster. It is a well-known technique in the art and will not be elaborated here.

[0053] In this way, by classifying all historical states, we can obtain multiple state clusters, and a state cluster can be regarded as a state category.

[0054] S203, obtain the predicted state for the next time step, and determine the target cluster based on the predicted state and the similarity between each state cluster.

[0055] In one embodiment, the predicted state can reflect the operating state of the energy storage converter at the next adjacent time. Specifically, obtaining the predicted state at the next time includes: acquiring the time series of each dimension of the real-time state within a historical time period; performing polynomial fitting on the time series to obtain the fitting curve of the dimension; inputting the next time into the fitting curve to obtain the predicted value of the dimension at the next time; and concatenating the predicted values ​​of all dimensions at the next time to obtain the predicted state at the next time.

[0056] The real-time status includes four dimensions: the remaining power and health of the energy storage power source, as well as the load demand and grid frequency on the grid side. Taking the remaining power as an example, the input of the fitting curve is any time, and the output is the predicted value of the remaining power at that time.

[0057] The fitted curve is a fourth-order polynomial, and the implementer can adjust the order of the fitted curve according to the specific scenario.

[0058] In one embodiment, after obtaining the predicted state for the next moment, the state category of the energy storage converter at the next moment can be determined by comparing the predicted state with each state cluster. Specifically, determining the target cluster based on the similarity between the predicted state and each state cluster includes: calculating the mean of all historical states in the state cluster to obtain the cluster center of the state cluster; calculating the similarity between the predicted state and the state cluster, and taking the state cluster corresponding to the maximum similarity as the target cluster, wherein the similarity is negatively correlated with the Euclidean distance between the predicted state and the cluster center.

[0059] Among them, the predicted state and the state cluster similarity for: , For predicting states and state clusters The Euclidean distance between the cluster centers.

[0060] Thus, the state category to which the predicted state belongs in the next moment is determined.

[0061] S204, Select training samples from the target cluster based on the reward vector to update the reinforcement learning model.

[0062] In one embodiment, in order to ensure that the reinforcement learning model can output accurate target actions in the predicted state at the next time step and obtain better control performance, it is necessary to select training samples from the target cluster (the target cluster and the predicted state have the same state category) to train the reinforcement learning model.

[0063] For a training sample in the target cluster In other words, each sub-reward value corresponds to an evaluation criterion for control effect. If the sub-reward value The smaller the value, the more likely it is to represent the target action in the training sample. In sub-reward value The corresponding evaluation criteria cannot achieve good evaluation results; therefore, if training samples are used... The updated reinforcement learning model is still unable to improve the sub-reward values. To achieve good evaluation results on the corresponding evaluation criteria; that is, to ensure that the updated reinforcement learning model can achieve good results in sub-reward values. If a good evaluation result is achieved on the corresponding evaluation criteria, a sub-reward value should be selected from the target cluster. A large number of training samples are required; therefore, in order to ensure that the updated reinforcement learning model achieves good evaluation results on all evaluation criteria, it is necessary to select training samples based on the sub-reward values ​​in the training samples.

[0064] Specifically, selecting training samples from the target cluster based on the reward vector of historical states includes: calculating the attention level of each sub-reward value at the current moment, and the sub-reward value... attention for:

[0065] , The sub-reward value after performing the target action at the current moment. The average sub-reward value over a preset time period prior to the current moment. The Sigmoid activation function is used; the sub-reward values ​​in the reward vector are weighted and summed based on the attention level to obtain the selection score of each training sample in the target cluster; the first M training samples are selected in descending order of selection score to update the reinforcement learning model.

[0066] The preset time period includes the 30 moments preceding the current moment; M equals the number of updates the strong chemical model can complete at adjacent moments; historical moments The selection score of the training samples for: , For any sub-reward value, The set of all sub-reward values. For child reward value attention, The sum of attention received for all sub-reward values. For a historic moment Sub-reward values ​​in training samples The value of .

[0067] Understandably, in the formula for calculating attention, Used to reflect the sub-reward value and average sub-reward value after performing the target action at the current moment. The differences between them; if This indicates that after performing the target action at the current moment, the sub-reward value... If good control has been achieved on the corresponding evaluation criteria, then the reward value for the sub-item should be reduced. attention, For a smaller value; conversely, if This indicates that after performing the target action at the current moment, the sub-reward value... If the corresponding evaluation criteria have not achieved good control results, then the reward value for the sub-item should be increased. attention, It is a relatively large value.

[0068] Thus, based on the value of each sub-reward after executing the target action at the current moment, different attention is assigned to different evaluation criteria, thereby calculating the selection score of each training sample in the target cluster. The larger the selection score, the better the control effect of the training sample on each evaluation criterion, and the more effective the update of the reinforcement learning model.

[0069] In one embodiment, after selecting training samples, the reinforcement learning model can be updated using the training samples. Specifically, the training samples include historical states. Adjacent historical states and historical status target action And multiple sub-reward values, the updating of the reinforcement learning model includes: historical states Input the reinforcement learning model to obtain the target action. Reward value ; to connect adjacent historical states Input the reinforcement learning model, output the reward value for each preset action, and record the maximum reward value as . The reward value for the training sample is obtained by weighted summation of multiple sub-reward values. Constructing the loss function The following relation is satisfied:

[0070] ; , and For each sub-reward value, , and The preset weights for each sub-reward value, The adjustment coefficient is used to update the reinforcement learning model using gradient descent. The value is 0.2.

[0071] In this way, the reinforcement learning model is updated at the current moment, and the updated reinforcement learning model is used to output the target action at the next moment, ensuring the control accuracy of the energy storage converter at the next moment.

[0072] The above specific embodiments illustrate the technical principles and implementation details of an artificial intelligence-based energy storage converter control method of this application. By inputting the real-time state of the energy storage converter into a reinforcement learning model, the target action of the energy storage converter is directly output. Executing the target action completes the control of the energy storage converter at the current moment. The reward value of the reinforcement learning model is a weighted sum of power conversion reward, frequency consistency reward, and load supply reward, constraining the control effect of the target action from different evaluation criteria. Furthermore, since the remaining power, health status, grid load demand, and grid frequency are constantly changing in real-time, the reinforcement learning model needs to be updated after executing the target action to improve the control accuracy of the energy storage converter under various operating conditions. When updating the reinforcement learning model, it is first necessary to obtain the predicted state for the next time step. The state category for the next time step is determined by comparing the predicted state with the state clusters in the experience pool. Training samples are selected from the state clusters (i.e., target clusters) corresponding to the state category. During the selection of training samples, the values ​​of each sub-reward value in the training samples are taken into account to ensure the effectiveness of the training samples in updating the reinforcement learning model. This ensures that the updated reinforcement learning model can output accurate target actions in the next time step and achieve better control performance.

[0073] According to a second aspect of this application, this application also provides an artificial intelligence-based energy storage converter control device. Figure 3 This is a structural block diagram of an artificial intelligence-based energy storage converter control device according to an embodiment of this application. Figure 3 As shown, the device 50 includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the artificial intelligence-based energy storage converter control method according to the first aspect of this application. The device also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface. Their configuration and functions are known in the art and will not be described further here.

[0074] In this application, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, system, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store required information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this application can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0077] In this application, "multiple" refers to two or more.

[0078] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0079] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0080] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0081] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An artificial intelligence-based control method for energy storage converters, characterized in that, The control method includes: The current real-time state is input into the reinforcement learning model to obtain the target action of the energy storage converter. The target action includes charging power and discharging power. The real-time state includes the remaining power and health of the energy storage power supply, as well as the load demand and grid frequency on the grid side. To execute the target action at the current moment and achieve control of the energy storage converter; The reward value of the reinforcement learning model is a weighted sum of multiple sub-reward values. The update method of the reinforcement learning model includes: collecting the reward vector of each historical state in the experience pool, wherein the reward vector includes each sub-reward value; clustering the historical states to obtain multiple state clusters; obtaining the predicted state at the next time step, and determining the target cluster based on the similarity between the predicted state and each state cluster; and selecting training samples from the target cluster based on the reward vector of the historical state to update the reinforcement learning model.

2. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, The input to the reinforcement learning model is the real-time state at any time, and the output is the set of reward values ​​at that time. The set of reward values ​​includes the reward value of each preset action, and the preset action corresponding to the maximum reward value corresponds to the target action.

3. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, The sub-reward values ​​include power conversion rewards, frequency consistency rewards, and load supply rewards; The energy conversion reward is the ratio of the energy fed into the grid to the energy consumed by the energy storage power source. The frequency consistency reward is negatively correlated with the difference between the grid frequency on the grid side and the preset frequency after the target action is performed; The load supply reward is negatively correlated with the difference between the power supply and load demand on the grid side after the target action is performed.

4. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, Selecting training samples from the target cluster based on the reward vector of historical states includes: Calculate the attention level of each sub-reward value at the current moment, sub-reward value attention for: , The sub-reward value after performing the target action at the current moment. The average sub-reward value over a preset time period prior to the current moment. Use the Sigmoid activation function; The selection score of each training sample in the target cluster is obtained by weighted summation of the sub-reward values ​​in the reward vector based on the attention level. The top M training samples are selected in descending order of their scores to update the reinforcement learning model.

5. The energy storage converter control method based on artificial intelligence according to claim 4, characterized in that, Historical moment The selection score of the training samples for: , For any sub-reward value, The set of all sub-reward values. For child reward value attention, The sum of attention received for all sub-reward values. For a historic moment Sub-reward values ​​in training samples The value of .

6. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, The clustering of historical states yields multiple state clusters, including: The Euclidean distance between historical states is used as the clustering distance, and the K-means algorithm is used to cluster the historical states to obtain multiple clusters, which correspond to multiple state clusters. The number of clusters was determined using the elbow method.

7. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, The process of obtaining the predicted state for the next moment includes: The time series of each dimension of the real-time status is obtained within a historical time period, and the time series is fitted with a polynomial to obtain the fitting curve of each dimension. Input the next time step into the fitted curve to obtain the predicted value of the dimension at the next time step; By concatenating the predicted values ​​of all dimensions for the next time step, we obtain the predicted state for the next time step.

8. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, The target cluster is determined based on the similarity between the predicted state and each state cluster, including: Calculate the mean of all historical states in the state cluster to obtain the cluster center of the state cluster; The similarity between the predicted state and the state cluster is calculated, and the state cluster corresponding to the maximum similarity is taken as the target cluster. The similarity is negatively correlated with the Euclidean distance between the predicted state and the cluster center.

9. The energy storage converter control method based on artificial intelligence according to claim 1, characterized in that, A training sample includes historical states. Adjacent historical states and historical status target action And multiple sub-reward values, updating the reinforcement learning model includes: historical status Input the reinforcement learning model to obtain the target action. Reward value ; Adjacent historical states Input the reinforcement learning model, output the reward value for each preset action, and record the maximum reward value as . ; The reward value for the training sample is obtained by weighted summation of multiple sub-reward values. Constructing the loss function for: ; , and For each sub-reward value, , and The preset weights for each sub-reward value, The adjustment coefficients are used to update the reinforcement learning model using gradient descent.

10. An artificial intelligence-based energy storage converter control device, characterized in that, It includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the artificial intelligence-based energy storage converter control method according to any one of claims 1 to 9.

Images