A hierarchical collaborative control method for multiple energy storage systems based on deep reinforcement learning

Abstract

The present invention discloses a multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning, comprising the following steps: S1, analyzing the energy relationship between observables of the traction power supply system; S2, deep reinforcement learning algorithm based on fuzzy guidance; S3, multi-energy storage system hierarchical collaborative control method; S4, intelligent agent framework based on FLG-DDPG algorithm. The present invention adopts the above-mentioned multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning, proposes a central-site-device hierarchical control architecture for the application scenario of multiple energy storage systems, configures different control strategies at different levels, and realizes collaborative optimization control between different levels and between energy storage systems, so as to achieve the best overall energy saving effect.

Description

Technical Field

[0001] The present invention relates to the field of energy storage control technology, and in particular to a hierarchical collaborative control method for multiple energy storage systems based on deep reinforcement learning. Background Art

[0002] Urban rail trains widely use regenerative braking technology, that is, during the braking process, the traction motor of the train will work in the generator state. In addition to being used for the auxiliary power supply of the train itself, the remaining electricity generated will be fed back to the traction power supply network to interact with nearby traction trains. Due to the unidirectional conductivity of the substation traction rectifier unit, when the regenerative braking energy generated by the braking train cannot be fully absorbed, the traction network voltage will increase. When the starting voltage of the train brake resistor is reached, the brake resistor starts to consume energy. If the network voltage is still not effectively controlled, regeneration failure may occur as the network voltage continues to rise, that is, air braking is put into use. If the above situation occurs, the regenerative braking energy of the train cannot be fully utilized.

[0003] The ground-based energy storage system of urban rail transit connects the energy storage device to the traction network through a bidirectional converter. Through the effective control of the energy management strategy (EMS), the remaining regenerative braking energy of the braking train is absorbed. In the past, when designing energy management strategies, whether it is based on the indirect reflection of the train status based on the changes in the traction network or directly obtaining the train status through other means, when it is judged that there is residual regenerative braking energy, it is hoped that the energy storage system can be quickly charged and absorbed, otherwise the remaining energy will be consumed in the braking resistor. However, for multi-substation and multi-energy storage systems, the circulating current characteristics caused by the difference in no-load voltage between substations are ignored, resulting in the energy storage charging amount not being entirely regenerative braking energy, but also including the energy output of other substations caused by the circulating current characteristics. And because the energy storage capacity is always limited, if the circulating current energy occupies too much capacity, it may also cause the remaining regenerative braking energy to be unable to be absorbed and finally consumed in the braking resistor. Therefore, for the application scenario of multiple substations equipped with multiple energy storage systems, the above two angles will jointly affect the final energy-saving performance.

[0004] The energy management strategies based on deep reinforcement learning proposed in the past all have corresponding problems. The most common problem is the design of agent rewards, and the reward design problem itself is one of the most important problems in deep reinforcement learning. The energy management strategies based on deep reinforcement learning proposed in the past mainly use energy saving rate, energy storage system charge, train brake resistor loss, etc. as the basis for reward function design. On the one hand, in actual engineering problems, energy saving rate, train brake resistor loss and other states cannot be observed online. Due to factors such as communication performance and train information security, it is difficult for the ground energy storage system to achieve real-time communication with the train, so the above information is difficult to obtain or evaluate in real time. On the other hand, since the energy management strategies currently proposed almost do not consider the problem of no-load voltage differences between multiple substations, the charge of the energy storage system is not entirely the residual regenerative braking energy, but also includes the energy output of the substation. Therefore, the energy-saving performance cannot be well reflected by the charge and discharge of the energy storage system. Based on the above, due to the difficulties in the reward function design process in the actual engineering application of the energy management strategy based on deep reinforcement learning, it is difficult for the agent to learn a better strategy online to control the energy storage system to achieve better energy-saving effects. Summary of the invention

[0005] The purpose of the present invention is to provide a hierarchical collaborative control method for multiple energy storage systems based on deep reinforcement learning, propose a central-site-device hierarchical control architecture for multiple energy storage system application scenarios, configure different control strategies at different levels, and realize collaborative optimization control between different levels and between energy storage systems to achieve the best overall energy saving effect.

[0006] To achieve the above object, the present invention provides a multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning, comprising the following steps:

[0007] S1. Analyze the energy relationship between observable quantities in the traction power supply system;

[0008] S2, deep reinforcement learning algorithm based on fuzzy guidance;

[0009] S3. Hierarchical collaborative control method for multiple energy storage systems;

[0010] S4. Agent framework based on FLG-DDPG algorithm.

[0011] Preferably, in step S1, under actual engineering conditions, the real-time observables generally only include two types of ground information, namely, the substation and the energy storage system. Ignoring line loss and converter loss, the energy destination of the substation includes the traction energy provided to the train and the charging amount of the energy storage system, as shown in the following formula:

[0012] E sub =Esub-tra +E sub-ess (1)

[0013] The energy changes of the energy storage system are divided into charging and discharging. The charging amount includes the energy provided by the substation, the remaining regenerative braking energy from the train, and the energy provided by other energy storage systems, as shown in the following formula:

[0014] E char =E sub-ess +E tra-ess +E ess-ess (2)

[0015] The discharge of the energy storage system includes the traction energy provided to the train and the energy provided for charging other energy storage systems, as shown in the following formula:

[0016] E dis =E ess-tra +E ess-ess (3)

[0017] Define the total system energy consumption as E sys The total train traction energy consumption is E tra Based on the above energy relationship, the following formula can be obtained:

[0018]

[0019] Assuming that the total train traction demand energy consumption is a constant, the system energy consumption E sys The smaller it is, the more the energy storage system absorbs the regenerative braking energy, that is, E tra-ess The larger the value, and based on the energy conservation relationship, if the system energy consumption is negative, then E tra-ess It must not be zero, which means that the energy storage system has absorbed the regenerative braking energy, and the smaller the system energy consumption at this time, it also reflects that the energy storage system absorbs more regenerative braking energy.

[0020] Preferably, in step S2, a deep deterministic policy gradient algorithm (DDPG) is used. The DDPG algorithm adds a deep neural network as a policy network on the basis of the original value function network, that is, the Q network, to determine the unique mapping relationship from the state space to the action space. In any state, only a unique action strategy corresponds to the policy network, thereby realizing the continuous control process of the energy storage system. The intelligent agent updates two deep neural networks simultaneously during the learning process, as shown in the following formula, namely, the value function network, i.e., the evaluator Critic, and the policy network, i.e., the executor Actor,

[0021]

[0022] Based on the above fuzzy-guided reward rules combined with the DDPG algorithm, a deep reinforcement learning algorithm based on fuzzy guidance is proposed. For the control of the energy storage system, the input state state of the reinforcement learning agent is the observable quantity in the urban rail traction power supply system; the execution action action of the agent is the continuous control of various quantities including thresholds and power of the energy storage system; the instant reward signal reward obtained by the agent is given based on the above fuzzy guidance rules. At the same time, in order to further improve the generalization ability of the initial value network and the stability of training, the exploration noise and experience replay mechanism are added.

[0023] Preferably, in step S3, based on the proposed deep reinforcement algorithm based on fuzzy guidance, a hierarchical system control strategy for multiple energy storage systems is proposed. For the application scenario of multiple energy storage systems, in order to achieve the best global energy-saving effect and reduce the dimension of centralized control parameters as much as possible, while ensuring the reliability of system control performance, a central-site-device hierarchical control architecture is proposed, and different control strategies are configured at different levels to achieve collaborative optimization control between different levels and between energy storage systems, so as to achieve the best overall energy-saving effect;

[0024] The fuzzy-guided deep reinforcement learning algorithm proposed in this patent is executed at the central level; the site level can adopt any single energy storage system energy management strategy. Considering the controller performance and actual engineering conditions, the energy management strategy based on no-load voltage identification, which is currently more commonly used, is adopted here; the device-level control adds a power compensation loop on the basis of the original voltage and current dual closed-loop structure, and the compensation power is determined by the central-level intelligent agent.

[0025] Preferably, in step S4, in the intelligent agent framework definition, the action is the compensation power of the energy storage system, i.e., Pcom in the step S3 framework;

[0026] The states include substation output energy, power, energy storage SOC, power, charge and discharge, and energy within the time step;

[0027] s=[E sub ,P sub ,SOC,P ess ,E char ,E dis ,...] (6)

[0028] The reward includes the basic reward item based on the fuzzy rule guidance in Section 2.2 above, and the penalty item.

[0029] r=r FLG +k1·r VOL (7).

[0030] Therefore, the present invention adopts the above-mentioned multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning, and its technical effects are as follows:

[0031] (1) Based on the central-site-device hierarchical control architecture, a fuzzy-guided deep reinforcement learning algorithm is proposed at the central level to coordinate the control of the compensation power of each device-level power compensation loop and effectively cooperate with the original site-level energy management strategy to improve the regenerative braking energy absorption effect.

[0032] (2) Compared with the traditional single-level energy management strategy, this method can effectively absorb the remaining regenerative braking energy of the train and make full use of the absorbed regenerative braking energy during traction conditions, thereby reducing the system output energy consumption and achieving the goal of energy saving and consumption reduction.

[0033] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The charging guidance part fuzzy reward and punishment surface;

[0035] Figure 2 The discharge guidance part fuzzy reward and punishment surface;

[0036] Figure 3 Deep reinforcement learning algorithm based on fuzzy guidance;

[0037] Figure 4 Hierarchical collaborative control architecture for multiple energy storage systems;

[0038] Figure 5 Intelligent agent Actor-Cirtic network construction;

[0039] Figure 6 Reinforcement learning agent training process (number of training segments: 200; training time: 10h);

[0040] Figure 7 Reinforcement learning agent controls behavior changes; (a) charging and discharging process (training segment: 1); (b) charging and discharging process (training segment: 172). DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0042] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0043] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the main idea or essential features of the present invention. Therefore, from all points of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0044] In addition, it should be understood that although this specification is described according to the implementation modes, not every implementation mode includes only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation modes that can be understood by those skilled in the art. These other implementation modes are also covered within the protection scope of the present invention.

[0045] It should also be understood that the specific embodiments described above are only used to explain the present invention, and the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, which should be covered by the protection scope of the present invention / invention.

[0046] Technologies, methods, and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0047] The disclosed contents of the prior art documents cited in the present specification are incorporated into the present invention by reference in their entirety and are therefore part of the disclosure of the present invention.

[0048] Embodiment 1

[0049] As shown in the figure, the present invention provides a hierarchical collaborative control method for multiple energy storage systems based on deep reinforcement learning. Based on actual engineering conditions, the absorption of regenerative braking energy is roughly inferred according to the energy relationship between actual observables, thereby guiding the reward design of the intelligent agent, and proposing a deep reinforcement learning algorithm based on fuzzy guidance. Based on the proposed algorithm, a hierarchical collaborative control method for multiple energy storage systems based on deep reinforcement learning is proposed for multiple energy storage systems.

[0050] Step 1: Energy relationship between observable quantities in the traction power supply system

[0051] Under actual engineering conditions, real-time observables generally only include two types of ground information: substation and energy storage system. Ignoring line losses and converter losses, the energy destination of the substation includes the traction energy provided to the train and the charging amount of the energy storage system. As shown in the following formula:

[0052] E sub =E sub-tra +E sub-ess (1)

[0053] The energy changes of the energy storage system are divided into charging and discharging. The charging amount includes the energy provided by the substation, the remaining regenerative braking energy from the train, and the energy provided by other energy storage systems. As shown in the following formula:

[0054] E char =E sub-ess +E tra-ess +E ess-ess (2)

[0055] The discharge of the energy storage system includes the traction energy provided to the train and the energy provided for charging other energy storage systems, as shown in the following formula:

[0056] E dis =E ess-tra +E ess-ess (3)

[0057] Define the total system energy consumption as E sys The total train traction energy consumption is E tra Based on the above energy relationship, the following formula can be obtained:

[0058]

[0059] Assuming that the total train traction demand energy consumption is a constant, the system energy consumption E sys The smaller it is, the more the energy storage system absorbs the regenerative braking energy, that is, E tra-ess The larger the value, the larger the energy consumption. Based on the energy conservation relationship, if the system energy consumption is negative, then E tra-ess It must not be zero, which means that the energy storage system has absorbed the regenerative braking energy, and the smaller the system energy consumption at this time, it also reflects that the energy storage system has absorbed more regenerative braking energy.

[0060] Step 2: Deep reinforcement learning algorithm based on fuzzy guidance

[0061] Based on the above analysis of energy relationships, a fuzzy-guided agent learning method is proposed. Instead of designing an accurate reward function in the traditional method, the corresponding fuzzy rules are designed based on "expert experience" according to the energy relationship between observable quantities to guide the reinforcement learning agent to learn.

[0062] Taking the charging process guidance as an example, during the net charging process of the energy storage system (i.e., the total charge of all energy storage systems within the time step ΔE char Greater than the discharge capacity ΔE dis ), the absorption of regenerative braking energy is determined based on the system energy consumption within the time step. sys When it is negative, the regenerative braking energy must be absorbed. At this time, the intelligent agent is rewarded based on the charging amount and the capacity of the energy storage system; when the system energy consumption is non-negative, it indicates that there is a traction energy demand or a strong circulation characteristic in the system. At this time, the penalty is based on the charging amount and the ESS capacity state. Combining the fuzzy control principle, the fuzzy rules for charging guidance are shown in Table 1 below.

[0063] Table 1 Fuzzy guidance reward rules based on considerable quantity - charging guidance

[0064]

[0065] VS, M, VB in the above table are very small, medium and very large in the classic fuzzy control rules; while NB, NS, O, PS, PB correspond to negative large, negative small, zero, positive small and positive large respectively. According to the above description, it can be seen that based on the different variables in the table, different degrees of rewards or penalties will be given to the intelligent agent, thereby guiding the intelligent agent to control the energy storage system to continue to increase charging or weaken charging until it exits the charging mode.

[0066] Based on the same idea, the discharge guidance rules are designed, and the fuzzy rules are shown in Table 2. During the net discharge of the energy storage system (that is, the total charge of all energy storage systems within the time step ΔEchar is less than the discharge ΔEdis), the traction energy demand of the train can be roughly judged based on the substation energy consumption within the time step. During the net discharge of the energy storage system, when the substation energy consumption ΔEsub is still large (that is, corresponding to M or VB), it means that there is a traction energy demand at this time, and the intelligent agent will be rewarded based on the discharge amount and the capacity of the energy storage system, thereby enhancing the discharge guidance; when the substation energy consumption is very small (VS), it indicates that the traction energy demand is small at this time or there is regenerative braking energy in the system that needs to be absorbed. At this time, the discharge amount and the capacity state of the energy storage system are punished, thereby weakening the discharge guidance.

[0067] Table 2 Fuzzy-guided reward rules based on considerable quantity - discharge guidance

[0068]

[0069] Since the controlled object is an energy storage system, in addition to the common methods of controlling the charge and discharge voltage thresholds in the past, in actual applications, it is often hoped that the current and input power of the energy storage system can be controlled, which belongs to a continuous control problem. At present, in the field of energy storage system control for urban rail regenerative braking energy absorption, most of the methods based on deep reinforcement learning algorithms use DQN (deep Q learning) algorithms, which cannot be used for continuous control problems. Therefore, this method will use the deep deterministic policy gradient algorithm (DDPG). The DDPG algorithm adds a deep neural network as a policy network on the basis of the original value function network, that is, the Q network, to determine the unique mapping relationship from the state space to the action space. Under any state, the policy network only corresponds to a unique action strategy, thereby realizing the continuous control process of the energy storage system. In the process of learning, the agent updates two deep neural networks at the same time, as shown in the following formula, which are the value function network, i.e., the evaluator (Critic) and the policy network, i.e., the executor (Actor).

[0070]

[0071] The DDPG algorithm itself and its update process will not be elaborated here. This patent proposes a deep reinforcement learning algorithm based on fuzzy guidance based on the above-mentioned fuzzy-guided reward rules combined with the DDPG algorithm. For the control of the energy storage system, the input state state of the reinforcement learning agent is the observable quantity in the urban rail traction power supply system; the execution action action of the agent is the control of the energy storage system, which may include continuous control of various quantities including thresholds and power; the instant reward signal reward obtained by the agent is given based on the above-mentioned fuzzy guidance rules. At the same time, in order to further improve the generalization ability of the initial value network and the stability of training, an exploration noise and experience playback mechanism is added. The principle flow chart of the deep reinforcement learning algorithm based on fuzzy guidance proposed in this patent is as follows Figure 3 shown.

[0072] The following table shows the training process of the deep neural network, where x is the input feature (original feature + one feature extraction result) and y is the label (the sum of the train power in the power supply section at the same time).

[0073] Table 3 Reinforcement learning algorithm training based on fuzzy guidance

[0074]

[0075]

[0076] Step 3: Hierarchical collaborative control method for multiple energy storage systems

[0077] Based on the proposed deep reinforcement algorithm based on fuzzy guidance, a hierarchical system control strategy for multiple energy storage systems is proposed. For the application scenarios of multiple energy storage systems, in order to achieve the best global energy saving effect and reduce the dimension of centralized control parameters as much as possible, while ensuring the reliability of system control performance, a central-site-device hierarchical control architecture is proposed. Different control strategies are configured at different levels to achieve collaborative optimization control between different levels and between energy storage systems to achieve the best overall energy saving effect. Multi-energy storage system hierarchical collaborative control architecture Figure 4 shown.

[0078] The fuzzy-guided deep reinforcement learning algorithm proposed in this patent is executed at the central level; the site level can adopt any single energy storage system energy management strategy. Considering the controller performance and actual engineering conditions, the energy management strategy based on no-load voltage identification, which is currently more commonly used, is adopted here; the device-level control adds a power compensation loop on the basis of the original voltage and current dual closed-loop structure, and the compensation power is determined by the central-level intelligent agent.

[0079] Step 4: Agent framework based on FLG-DDPG algorithm

[0080] In the definition of the intelligent agent framework, the action is the compensation power of the energy storage system, which is Pcom in the above architecture;

[0081] The status includes substation output energy, power, energy storage SOC, power, charge and discharge, and energy within the time step;

[0082] s=[E sub ,P sub ,SOC,P ess ,E char ,E dis ,...] (6)

[0083] The reward includes a basic reward item guided by the fuzzy rules in Section 2.2 above, and a penalty item.

[0084] r=r FLG +k1·r VOL (7)

[0085] The completed intelligent agent framework is as follows Figure 5 Shown

[0086] Case analysis:

[0087] Take a subway line in Beijing as an example, where ground energy storage systems are installed in substations A and B. Therefore, the action of the reinforcement learning agent is defined as the compensation power of the energy storage system installed in stations A and B, as shown in formula (8).

[0088] a=[P essA_com ,PessB_com ]=μ(s t ) (8)

[0089] The state includes observable substation output energy, power, energy storage SOC, power, charge and discharge capacity, and energy within the time step, and is preprocessed into a standardized vector, as shown in Equation (9).

[0090] s=[E sub ,P sub ,SOC,P ess ,E char ,E dis ,...] (9)

[0091] The agent's reward is obtained based on the proposed fuzzy rules, which can be referred to in the previous section.

[0092] Based on the MATLAB Simulink simulation environment, the hierarchical cooperative control of multiple energy storage systems was simulated and verified, and the intelligent agent was trained on a small scale. Figure 6 It can be seen that the learning process of the intelligent agent shows good convergence characteristics.

[0093] Depend on Figure 7 It can be seen that the intelligent agent has obviously undergone positive behavioral changes after a period of learning, and has achieved better control effects after a shorter learning process compared to previous methods.

[0094] Table 4 shows the simulation verification results of the system energy consumption, train braking resistor loss energy, and substation energy saving rate in the above example analysis when different currently common energy management strategies are adopted.

[0095] Table 4. Simulation verification results of different energy management strategies (departure interval: 420s)

[0096]

[0097] It can be seen that the multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning proposed in this patent has effectively improved the energy saving rate compared with traditional energy storage system control, such as energy management strategies based on no-load voltage identification and offline train power control. At the same time, the train braking resistance loss is significantly reduced, thereby achieving better energy-saving effects for the ground-based energy storage system.

[0098] Therefore, the present invention adopts the above-mentioned multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning, proposes a central-site-device level hierarchical control architecture for the application scenarios of multiple energy storage systems, configures different control strategies at different levels, and realizes collaborative optimization control between different levels and between energy storage systems, so as to achieve the best overall energy-saving effect.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning, the method adopts a central-site-device hierarchical control architecture, configures different control strategies at different levels, and is characterized in that: The following steps are involved: S1. Acquire two types of ground information of the substation and the energy storage system, analyze the energy relationship between observable quantities of the traction power supply system according to the two types of ground information of the substation and the energy storage system, and determine the fuzzy guidance rule based on the energy relationship between the observable quantities; S2. Execute the deep reinforcement learning algorithm based on fuzzy guidance at the central level. For the control of the energy storage system, the input state state of the reinforcement learning agent is the observable quantity in the traction power supply system; the execution action action of the agent is to continuously control various quantities of the energy storage system including threshold value and power; the instant reward signal reward obtained by the agent is given based on the above fuzzy guidance rules; S3, the compensation power of the energy storage system installed in the substation outputted at the central level is used to control the energy storage device at the device level; S4, site level execution of any energy storage system energy management strategy; S5. Device-level control adds a power compensation loop based on the original voltage-current dual closed-loop structure, and the compensation power is determined by the central-level intelligent agent.

2. According to claim 1, a multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning is characterized in that: Analyzing the energy relationship between observable quantities in the traction power supply system includes Ignoring line losses and converter losses, the energy destination of the substation includes the traction energy provided to the train and the charging amount of the energy storage system, as shown in the following formula: AND sub =And sub-tra +E sub-ess (1); The energy changes of the energy storage system are divided into charging and discharging. The charging amount includes the energy provided by the substation, the remaining regenerative braking energy from the train, and the energy provided by other energy storage systems, as shown in the following formula: AND char =And sub-ess +E tra-ess +E ess-ess (2); The discharge of the energy storage system includes the traction energy provided to the train and the energy provided for charging other energy storage systems, as shown in the following formula: AND dis =And ess-tra +E ess-ess (3); The total system energy consumption is E sys The total train traction energy consumption is E tra Based on formulas (1), (2) and (3), the energy relationship between observable quantities in the traction power supply system is obtained as follows: The total train traction demand energy consumption is a constant, then the system energy consumption E sys The smaller it is, the more the energy storage system absorbs the regenerative braking energy, that is, E tra-ess The larger the value, and based on the energy conservation relationship, if the system energy consumption is negative, then E tra-ess It must not be zero, which means that the energy storage system has absorbed the regenerative braking energy, and the smaller the system energy consumption at this time, it also reflects that the energy storage system has absorbed more regenerative braking energy.

3. According to claim 2, a multi-energy storage system hierarchical collaborative control method based on deep reinforcement learning is characterized in that: The deep reinforcement learning algorithm is specifically: The deep deterministic policy gradient algorithm DDPG, as shown in formula (5), is composed of the value function network, i.e., the evaluator Critic, and the policy network, i.e., the executor Actor. Based on "expert experience", corresponding fuzzy rules are designed according to the energy relationship between observable quantities. The deep deterministic policy gradient algorithm DDPG is guided by fuzzy rules to obtain a deep reinforcement learning algorithm based on fuzzy guidance.

Images