Rail transit energy storage system control method, model, device and storage medium

Abstract

The application discloses a kind of rail transit energy storage systems control method, model, equipment and storage medium, the method includes determining offline charging and discharging action according to the state of energy storage system based on offline algorithm, determining online charging and discharging action according to the state of energy storage system based on deep reinforcement learning algorithm, the fusion proportion of the offline charging and discharging action and the online charging and discharging action is obtained according to the number of communication delay and delay degree, the offline charging and discharging action and online charging and discharging action are fused according to the fusion proportion and output to energy storage system, the fusion proportion is obtained according to the number of communication delay and delay degree in the embodiment of the application, the offline charging and discharging action and online charging and discharging action are fused according to the fusion proportion and output to energy storage system, can also be normally operated under different communication environments, improve the robustness of system.

Description

Technical Field

[0001] This invention relates to the field of energy storage system control technology, and in particular to control methods, models, equipment and storage media for rail transit energy storage systems. Background Technology

[0002] Rail transit is a crucial component of transportation systems, and urban rail transit is a type of rail transit. With the rapid development of urban rail transit, its power consumption has increased significantly. Therefore, reducing the traction energy consumption of urban rail transit is of great importance to energy conservation and emission reduction across society. Improving the ability of urban rail traction power supply systems to receive regenerative energy and fully utilizing the regenerative braking energy of trains is key to reducing the energy consumption of urban rail transit systems. However, currently, the regenerative energy absorption capacity of urban rail power supply systems is very limited: most traction substations use diode unidirectional rectification, and regenerative braking energy cannot be fed back to the AC power grid; if there is no traction train near the braking train to absorb regenerative energy, the braking energy is wasted on the braking resistor. Utilizing the regenerative energy of trains through energy storage systems is of great significance to the sustainable development of the urban rail industry.

[0003] Considering the frequent braking and high braking power of urban rail trains, supercapacitor energy storage devices, with their high power density, have been widely researched and applied in the rail transit field. However, on the one hand, the power and position of urban rail trains change in real time, and the parameters and topology of the traction power supply system have nonlinear and time-varying characteristics, making the entire optimization model extremely complex. On the other hand, the voltage level of urban rail power supply systems is relatively low, and changes in various system operating parameters can have a significant impact on energy transmission, thus affecting the energy efficiency of the energy storage system. If the characteristics of the train, line, and substation are not comprehensively considered, and the charging and discharging actions of the energy storage system are not adjusted in real time, the energy efficiency of the energy storage system will fluctuate significantly with external conditions. This can even exacerbate energy waste when train intervals are long, which is a bottleneck limiting the large-scale application of energy storage systems in urban rail transit. Therefore, it is crucial to fully consider the characteristics of trains, energy storage devices, lines, and substations, optimize the energy flow of the urban rail power supply system, and improve the energy efficiency of the energy storage system.

[0004] Existing energy storage device energy management strategies are mostly fixed threshold strategies, such as Figure 1 As shown, a fixed charging threshold Uchar and a fixed discharging threshold Udis are set using an offline optimization algorithm. When the traction grid voltage is greater than the charging threshold, the energy storage device charges; when the traction grid voltage is lower than the discharging threshold, the energy storage device discharges. This method fails to fully consider the characteristics of the train, energy storage device, line, and substation, resulting in low charging and discharging efficiency and high regeneration failure rate. To improve the charging efficiency of the energy storage system, some scholars have proposed a dynamic voltage-following charging threshold dynamic adjustment strategy, such as... Figure 2As shown, it dynamically maintains the train's terminal voltage at the critical value of the braking resistor's starting voltage based on the train's position and power, thereby maximizing energy interaction between trains and improving the energy efficiency of the energy storage system.

[0005] None of the above algorithms can achieve globally optimal control. Some scholars have considered that solving the optimal control strategy for energy storage devices is a sequential decision optimization problem, such as... Figure 3 As shown, a reinforcement learning algorithm is introduced to adjust the control parameters of the energy storage device online to adapt to changes in the power supply system operating conditions, enabling the energy storage system to play a good role in energy saving and voltage stabilization. However, the robustness of the algorithm is poor. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a control method, model, device and storage medium for a rail transit energy storage system to solve the technical problem of poor robustness of existing energy storage control methods.

[0007] The technical solution proposed in this invention is as follows:

[0008] The first aspect of this invention provides a control method for a rail transit energy storage system, comprising: determining offline charging and discharging actions based on the state of the energy storage system using an offline algorithm; determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm; obtaining a fusion ratio between the offline charging and discharging actions and the online charging and discharging actions based on the number and degree of communication delays; and fusing the offline charging and discharging actions and the online charging and discharging actions according to the fusion ratio and outputting the fusion to the energy storage system.

[0009] Optionally, the method of determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm includes: receiving the state of the energy storage system and the offline charging and discharging actions; using the offline charging and discharging actions as the initial values ​​of the neural network and training the neural network with training data, wherein the neural network outputs an action-value function based on the state of the energy storage system; and obtaining online charging and discharging actions based on the action-value function and a greedy strategy.

[0010] Optionally, the method of determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm further includes: storing used training data and randomly selecting training data from the used training data to retrain the neural network.

[0011] Optionally, before determining the offline charging and discharging action based on the state of the energy storage system using an offline algorithm, the method further includes: obtaining the operating range of the energy storage system, wherein the state of the energy storage system includes the substation state, train state, and energy storage device state within the operating range.

[0012] Optionally, the effective range of the energy storage system includes: selecting a central substation; determining whether the impact of the train at different locations on the terminal voltage of the central substation is greater than a threshold voltage; if it is greater than the threshold voltage, then determining that the effective range includes the central substation and the substation where the train is located.

[0013] Optionally, obtaining the fusion ratio of the offline charging / discharging action and the online charging / discharging action based on the number and degree of communication delay includes: obtaining the correspondence between any number and degree of communication delay and the fusion ratio through pre-training; and obtaining the fusion ratio of the offline charging / discharging action and the online charging / discharging action based on the correspondence and the number and degree of communication delay.

[0014] Optionally, the step of obtaining the correspondence between the number and degree of arbitrary communication delays and the fusion ratio through pre-training includes: initializing the fusion ratio; obtaining the online charging and discharging actions based on the state of the energy storage system under arbitrary communication delays and degrees; obtaining the offline charging and discharging actions based on the state of the energy storage system; calculating the fused charging and discharging actions based on the online charging and discharging actions, the offline charging and discharging actions, and the fusion ratio; executing the offline charging and discharging actions and the fused charging and discharging actions respectively to obtain a first reward signal based on the fused charging and discharging actions and a second reward signal based on the offline charging and discharging actions; updating the fusion ratio based on the first reward signal and the second reward signal, wherein if the first reward signal is greater than the second reward signal, the fusion ratio is increased, and if the first reward signal is less than the second reward signal, the fusion ratio is decreased; repeating the step of updating the fusion ratio until the rate of change of the fusion ratio reaches a termination value.

[0015] A second aspect of this invention proposes a control model for a rail transit energy storage system, comprising: an offline generalization module for determining offline charging and discharging actions based on the state of the energy storage system using an offline algorithm; a deep reinforcement learning module for determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm; and a robustness enhancement module for obtaining the fusion ratio of the offline charging and discharging actions and the online charging and discharging actions based on the number and degree of communication delays, fusing the offline charging and discharging actions and the online charging and discharging actions according to the fusion ratio, and outputting the fusion to the energy storage system.

[0016] A third aspect of the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the control method of the rail transit energy storage system as described in the first aspect and any one of the first aspects of the present invention.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the control method for a rail transit energy storage system as described in the first aspect and any one of the first aspects of the present invention.

[0018] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages:

[0019] This invention provides a control method, model, device, and storage medium for a rail transit energy storage system. It determines offline charging and discharging actions based on the energy storage system's state using an offline algorithm, and determines online charging and discharging actions based on the energy storage system's state using a deep reinforcement learning algorithm. The fusion ratio of the offline and online charging and discharging actions is obtained based on the number and degree of communication delays. The offline and online charging and discharging actions are then fused according to this fusion ratio and output to the energy storage system. This invention enables normal operation under different communication environments, improving the system's robustness. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the fixed threshold strategy in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the framework of the dynamic voltage following charging threshold dynamic adjustment strategy in an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the framework of the globally optimal control strategy in an embodiment of the present invention;

[0024] Figure 4 This is a topology diagram of the rail transit energy storage system in an embodiment of the present invention;

[0025] Figure 5 This is a flowchart of the control method for the rail transit energy storage system in an embodiment of the present invention;

[0026] Figure 6 This is a flowchart illustrating the training process of the offline generalization module in an embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram of the framework of the offline simulation model in an embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram of the framework for offline optimization of charge and discharge threshold curves in an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of the offline pattern table in an embodiment of the present invention;

[0030] Figure 10 This is a schematic diagram of the framework for pattern mining and strategy formulation in an embodiment of the present invention;

[0031] Figure 11 This is a network training framework diagram of the deep reinforcement learning algorithm in an embodiment of the present invention;

[0032] Figure 12 This is a flowchart illustrating the process of obtaining the effective range in an embodiment of the present invention;

[0033] Figure 13 This is a flowchart illustrating the training process of the robustness enhancement model in this embodiment of the invention.

[0034] Figure 14 This is a block diagram of the control model of the rail transit energy storage system in an embodiment of the present invention;

[0035] Figure 15 This is a block diagram of the control model of another rail transit energy storage system in an embodiment of the present invention;

[0036] Figure 16 This is a flowchart illustrating the control model of the rail transit energy storage system in an embodiment of the present invention.

[0037] Figure 17 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention;

[0038] Figure 18 This is a schematic diagram of the structure of the storage medium in an embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The application scenario of the control method for a rail transit energy storage system according to an embodiment of the present invention is a ground-based energy storage device based on information interaction. The system topology diagram of the train power supply system including the ground-based energy storage device is as follows. Figure 4 As shown, the energy storage system includes a management system and ground-based energy storage. The energy storage system is installed in the substation and connected in parallel to the DC bus via a bidirectional buck / boost topology. Train status, substation status, and SC status are transmitted to the energy storage system via communication.

[0041] This invention provides a control method for a rail transit energy storage system, such as... Figure 5 As shown, it includes:

[0042] Step S100: Determine offline charging and discharging actions based on the state of the energy storage system using an offline algorithm. Specifically, an offline generalization module is constructed based on the offline algorithm. The offline generalization module is an analytical approach that takes the state as input and the decision as output. For example, an initial offline generalization module is obtained based on offline training and pattern mining. The training process of the offline generalization module is as follows... Figures 6 to 10 As shown, the system comprises four parts: an offline simulation model, offline optimization of charge / discharge threshold curves, an expert system and analysis of optimization results, and pattern mining and strategy formulation. Based on typical operating conditions of the offline simulation model of the train power supply system, the initial offline generalization module is trained to obtain optimized offline charge / discharge threshold curves. Subsequently, an offline pattern table is obtained through the expert system. Finally, patterns are mined and extracted, and then a strategy is formulated to obtain the final offline generalization model. The input of the offline generalization model is the power and location of adjacent trains, as well as the SOC of the energy storage system. The output is the offline charge / discharge action of the current energy storage device, i.e., the charge / discharge threshold. Specifically, based on the offline simulation model and offline optimization algorithms, such as genetic algorithms and dynamic programming, the optimal charge / discharge threshold curves are optimized under various operating conditions, resulting in a large amount of data with the power and location of adjacent trains and the SOC of the energy storage system as inputs, and the optimal charge / discharge threshold as the output. The expert system is a computer-based judgment system that uses existing experience and knowledge as rules to replace human decision-making. The expert system automatically extracts data segments with patterns, automatically describes the relationship between the pattern and the input, and integrates the patterns. Then, the patterns are classified according to whether they are linear or not, and the nonlinear patterns are further explored. That is, by solving the analytical problem, the analytical solution form between the input and output is established. The process of pattern integration has decomposed the global optimization problem into local optimization problems, and analytical solutions can be obtained.

[0043] Step S200: Determine online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm. Specifically, the deep reinforcement learning algorithm employs the DQN (Deep Q-Learning) algorithm, using a neural network to approximate the action-value function. The action selection strategy of the deep reinforcement learning algorithm is an ε-greedy strategy, that is, selecting the action with the largest action-value function with a certain probability ε, and randomly selecting other strategies with a probability of 1-ε. The parameters in the network are updated using gradient descent. Through continuous iteration, the action corresponding to the final largest action-value function can be made the optimal action. The online charging and discharging action, as the output value calculated by the deep reinforcement learning algorithm, is obtained after acquiring the state of the energy storage system, based on the analysis of the deep reinforcement learning algorithm, i.e., the charging and discharging threshold based on online learning.

[0044] Step S300: Obtain the fusion ratio of offline charging / discharging actions and online charging / discharging actions based on the number and severity of communication delays. In urban rail transit applications, delay is generally the sum of the sending program processing delay + transmission delay + receiving program processing delay. The program processing delay is relatively fixed, but the transmission delay is not. Packet loss mainly occurs during electromagnetic wave transmission and may be caused by interference from other strong electromagnetic fields. Through training, the higher the number and severity of communication delays, the smaller the fusion ratio becomes.

[0045] Step S400: Merge offline charging / discharging actions and online charging / discharging actions according to the fusion ratio and output them to the energy storage system. Specifically, the output ratio of offline charging / discharging actions and online charging / discharging actions is determined according to the fusion ratio. The smaller the fusion ratio, the larger the proportion of offline charging / discharging actions, and the smaller the proportion of online charging / discharging actions. Conversely, the larger the fusion ratio, the smaller the proportion of offline charging / discharging actions, and the larger the proportion of online charging / discharging actions. If there is significant communication loss, i.e., the reinforcement learning state is incomplete, the online charging / discharging action results output by the deep reinforcement learning algorithm will be poor. Therefore, when the communication state is poor, the proportion of offline charging / discharging actions can be increased to maintain stable output and enhance the robustness of the system.

[0046] This invention provides a control method for a rail transit energy storage system. It determines offline charging and discharging actions based on the system's state using an offline algorithm, and determines online charging and discharging actions based on the system's state using a deep reinforcement learning algorithm. The method obtains a fusion ratio between the offline and online charging and discharging actions based on the number and degree of communication delays. This fusion ratio allows the system to operate normally under different communication environments, improving its robustness.

[0047] In one embodiment, a deep reinforcement learning algorithm is used to determine online charging and discharging actions based on the state of the energy storage system. This includes: receiving the state of the energy storage system and offline charging and discharging actions; using the offline charging and discharging actions as initial values ​​for a neural network and training the neural network with training data; the neural network outputting an action-value function based on the state of the energy storage system; and obtaining online charging and discharging actions based on the action-value function and a greedy strategy. Specifically, the neural network is a Q-network, trained using a gradient descent algorithm. The algorithm updates the network parameters θ by minimizing the root mean square error between the target network and the Q-network. The action-value function represents the relationship between the action taken in the current state and the resulting reward. In this embodiment, a neural network is used to simulate the action-value function; that is, inputting the current state s, the output can obtain the current Q(s, a), where a represents the corresponding action, i.e., obtaining the action-value function for any action in the current state. Specifically, the action-value function represents the charging / discharging threshold. After receiving offline charging / discharging actions, the action-value function corresponding to each action is assigned the maximum value. This means the offline charging / discharging action is used as the initial value for the neural network. In this case, the deep reinforcement learning algorithm is highly likely to select this action, reducing the trial-and-error process and introducing the idea of ​​behavior cloning to improve the algorithm's generalization ability. After receiving the state of the energy storage system, the Q-network outputs the action-value function based on the system's state. Then, a greedy strategy is used to obtain the online charging / discharging action. The greedy strategy means selecting the action with the maximum action-value function in the current state with a certain probability; otherwise, a random action is selected. The initial values ​​for the neural network can also be obtained from offline training, but such initial values ​​are model-dependent, requiring retraining for different models. However, the offline algorithm directly obtains the analytical relationship between the input and output, independent of the model. Its output is used as the initial value, eliminating the need for offline training each time.

[0048] In one embodiment, determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm further includes: storing used training data and randomly selecting training data from the used training data to retrain the neural network. Specifically, the neural network is a Q-network, which is trained using a gradient descent algorithm. The gradient descent algorithm updates the network parameters θ by minimizing the root mean square error between the target network and the Q-network. The target network and the Q-network are identical, obtained by copying the Q-network. The gradient descent algorithm is calculated as follows:

[0049]

[0050] N is the mini-batch size used to perform the gradient descent algorithm, θ- is the weight of the target network, θ is the weight of the Q network, and s k a k For the current state and the current action, s k ′、a k ′ represents the state and action at the next moment, r k γ represents the current reward signal, and γ is the algorithm parameter. To break the correlation between training data and improve algorithm stability, used training data, i.e., experience data tuples, are stored in the experience replay pool, and the data in the experience replay pool is randomly sampled during training.

[0051] See Figure 11 In this embodiment of the invention, used training data is stored in an experience replay pool. The experience replay module is a database that stores multiple experience data tuples, and one experience data tuple is a complete training data set (s). k ,a k ,r k ,s k+1 These represent the current state, the optimal action in the current state, the reward in the current state, and the next state, respectively. The algorithm for training with an experience replay pool is as follows:

[0052] Initialize the experience replay pool and initialize the Q-network based on random weights θ;

[0053] The target network Q′ is initialized based on zero-weight θ.

[0054] repeat:

[0055] The initial running state s based on the offline simulation model;

[0056] repeat:

[0057] In state s, action a is selected according to the ε-greedy strategy;

[0058] Perform action a in the offline simulation model;

[0059] The system state s′ and reward signal r at the next moment are obtained by solving the circuit equations of the offline simulation model.

[0060] Store the state transition tuple <,a,r,s′> in the experience replay pool;

[0061] Sample mini-batch state transition arrays from the experience replay pool;

[0062] The parameters θ of the Q network are updated by performing gradient descent on equation (1);

[0063] Execute θ-←θ every n steps;

[0064] The process continues until s reaches a termination state, such as when the gradient of gradient descent approaches 0 or the iteration reaches the upper limit.

[0065] The algorithm continues until the termination condition is met, meaning that the termination state is met at each step.

[0066] This invention stores used training data and randomly selects training data from the used training data to retrain the neural network. That is, it stores the experience data tuples in the experience replay pool and randomly samples the data during training. Compared with discarding the incoming data immediately after an update, this method wastes training data and increases the correlation between the two training sessions, which is not conducive to model training. This invention breaks the correlation between training data and improves the stability of the algorithm.

[0067] In one embodiment, before determining the offline charging and discharging actions based on the state of the energy storage system using an offline algorithm, the method further includes: obtaining the effective range of the energy storage system, where the state of the energy storage system includes the substation state, train state, and energy storage device state within the effective range. Specifically, based on circuit theory, upstream and downstream tracking is performed on the current distribution of the urban rail traction power supply system to quantitatively characterize the proportional relationship between the substation and braking train current and the traction train current; based on the results of current tracking, upstream and downstream tracking is performed on the power distribution of the urban rail traction power supply system to obtain the specific allocation coefficients between the substation output power, braking train power, traction train power, and line losses. Through energy flow analysis, the power flow path of the system can be displayed intuitively and quantitatively, thereby enabling real-time division of the effective range of the energy storage system and real-time calculation of the transmission ratio within different energy ranges. Based on the power of adjacent trains, the largest energy control area, i.e., the effective range, is calculated and output. The effective range is used to determine the scale that the deep reinforcement learning algorithm needs to learn. The substation state, train state, and energy storage device state within this effective range are learned as a whole. Selecting an appropriate effective range is beneficial for the rapid convergence of the deep reinforcement learning algorithm.

[0068] In one embodiment, obtaining the effective range of the energy storage system includes: selecting a central substation; determining whether the impact of the train at different locations on the terminal voltage of the central substation is greater than a threshold voltage; if it is greater than the threshold voltage, then determining that the effective range includes both the central substation and the substation where the train is located. Specifically, as... Figure 12 As shown, an arbitrary central substation is selected as the controlled object. The electrical coupling strength is defined based on the magnitude of the voltage fluctuation in the central substation's grid caused by the train running in the vicinity of the substation. Subsequently, with the train power fixed at the maximum operating power, strong coupling intervals are searched to the left and right respectively; the affected interval is the strong coupling interval. Uoc is defined as the output voltage of the central substation when there is no train running on the line, Umid is the output voltage of the central substation when the train is outputting its maximum operating power at different positions, and Ulim is the threshold voltage for determining strong and weak coupling intervals, which can be selected arbitrarily, but is generally chosen as 5V. When the train's influence on the central substation's terminal voltage is determined to be greater than the threshold voltage, it is determined to be a strong coupling interval; otherwise, it is determined to be a weak coupling interval.

[0069] This invention determines the effective range of the energy storage system based on the impact of the train on the terminal voltage of the central substation at different locations, thereby achieving zoned control, avoiding the problem of overly complex information processing in the algorithm, and improving the convergence ability and computation speed of the algorithm.

[0070] In one embodiment, obtaining the fusion ratio of offline charging / discharging actions and online charging / discharging actions based on the number and degree of communication delays includes:

[0071] Step S310: Obtain the correspondence between the number and degree of arbitrary communication delays and the fusion ratio through pre-training. Specifically, run the fused offline charging and discharging actions and online charging and discharging actions with different fusion ratios in a simulation environment, and find the optimal fusion ratio corresponding to the current number and degree of communication delays based on the running results. Map this fusion ratio to the current number and degree of communication delays to form a correspondence. This correspondence is implemented through a neural network. The input of the neural network is the number and degree of communication delays, and the output is the fusion ratio.

[0072] Step S320: Based on the correspondence, obtain the fusion ratio of offline charging / discharging actions and online charging / discharging actions according to the number and degree of communication delays. After obtaining the correspondence between the number and degree of any communication delay and the fusion ratio through pre-training, in actual operation, the fusion ratio of offline charging / discharging actions and online charging / discharging actions is obtained based on the current actual number and degree of communication delays. This embodiment of the invention considers the problem of communication delay, thereby improving the robustness of the deep reinforcement learning algorithm.

[0073] In one embodiment, the corresponding relationship between any communication delay quantity, delay degree, and fusion ratio is obtained through pre-training, including:

[0074] Step S311: Initialize the fusion ratio. Initialize the fusion ratio k = 1

[0075] Step S312: Under any communication delay quantity and delay degree, obtain the online charge-discharge action according to the state of the energy storage system. The state of the energy storage system is obtained through an offline simulation model. After obtaining the state of the energy storage system, use the deep reinforcement learning algorithm to obtain the online charge-discharge action.

[0076] Step S313: Obtain the offline charge-discharge action according to the state of the energy storage system. After obtaining the state of the energy storage system, use the offline algorithm to obtain the offline charge-discharge action.

[0077] Step S314: Calculate the fusion charge-discharge action based on the online charge-discharge action, offline charge-discharge action, and fusion ratio. Specifically, the fusion charge-discharge action is a2, and the calculation formula is: a2 = a * k + a1 * (1 - k), where a is the online charge-discharge action and a1 is the offline charge-discharge action.

[0078] Step S315: Execute the offline charge-discharge action and the fusion charge-discharge action respectively to obtain the first reward signal based on the fusion charge-discharge action and the second reward signal based on the offline charge-discharge action. The execution process is carried out in the offline simulation model. After executing the corresponding action, the corresponding reward signal can be obtained. The reward signal is the feedback of the environment to the agent's action. In the embodiment of the present invention, the energy saving rate of the energy storage device is mainly concerned. Therefore, the reward signal is the energy saving rate within the time step T (the time step T is the time interval for executing the algorithm once, and the energy saving rate is: the energy output by the energy storage device / the energy output by the substation).

[0079] Step S316: Update the fusion ratio based on the first reward signal and the second reward signal. Among them, if the first reward signal is greater than the second reward signal, increase the fusion ratio; if the first reward signal is less than the second reward signal, decrease the fusion ratio. Specifically, the update formula is: k = k - c1 * (r2 - r1), (r2 > r1) and k = k + c2 * d(r1), (r2 < r1), r1 is the first reward signal, r2 is the second reward signal, c1 and c2 are the update steps, which can be adjusted according to the situation. When r2 is greater than r1, when the value of the fusion ratio k is updated to k - c1 * (r2 - r1), when r2 is less than r1, when the value of the fusion ratio k is updated to k + c2 * d(r1).

[0080] Step S317: Repeat the step of updating the fusion ratio until the change rate of the fusion ratio reaches the termination value. For example, when the change rate of the fusion ratio k is less than a number, such as 0.001, end the training process.

[0081] Specifically, the above pre-training steps are implemented through a neural network. By training the neural network, a robust enhancement model is obtained, with the input being the number and degree of delay, and the output being the optimal fusion ratio k. The training process of the robust enhancement model is as follows: Figure 13 As shown. This embodiment of the invention obtains the optimal fusion ratio through pre-training, enabling it to achieve the best output even under poor communication conditions, thus enhancing the system's robustness.

[0082] This invention also proposes a control model for a rail transit energy storage system, such as... Figure 14 As shown, the device includes:

[0083] The offline generalization module is used to determine offline charging and discharging actions based on the state of the energy storage system using an offline algorithm. For details, please refer to the corresponding sections of the above method embodiments; they will not be repeated here.

[0084] The deep reinforcement learning module is used to determine online charging and discharging actions based on the state of the energy storage system using deep reinforcement learning algorithms. For details, please refer to the corresponding sections of the above method embodiments, which will not be repeated here.

[0085] The robustness enhancement module is used to obtain the fusion ratio of offline charging / discharging actions and online charging / discharging actions based on the number and degree of communication delays. It then fuses the offline and online charging / discharging actions according to this ratio and outputs them to the energy storage system. For details, please refer to the corresponding sections of the above method embodiments; they will not be repeated here.

[0086] This invention provides a control model for a rail transit energy storage system. It determines offline charging and discharging actions based on the system's state using an offline algorithm, and determines online charging and discharging actions based on the system's state using a deep reinforcement learning algorithm. The model obtains a fusion ratio between the offline and online charging and discharging actions based on the number and degree of communication delays. This fusion ratio allows the system to operate normally under different communication environments, improving its robustness.

[0087] In one embodiment, the deep reinforcement learning module includes:

[0088] The receiving module is used to receive the status of the energy storage system and its offline charging and discharging actions.

[0089] The network module is used to use offline charging and discharging actions as initial values ​​for the neural network and train the neural network using training data. The neural network outputs an action-value function based on the state of the energy storage system.

[0090] The strategy module is used to obtain online charging and discharging actions based on action-value functions and greedy strategies.

[0091] In one embodiment, such as Figure 15 As shown, the control model of the rail transit energy storage system also includes an experience playback module, which is used to store the training data that has been used and to randomly extract training data from the training data that has been used to retrain the neural network.

[0092] In one embodiment, such as Figure 15 As shown, the control model of the rail transit energy storage system also includes a real-time partitioning module, which is used to obtain the operating range of the energy storage system. The status of the energy storage system includes the substation status, train status and energy storage device status within the operating range.

[0093] In one embodiment, the real-time partitioning module includes:

[0094] The selection module is used to select the central substation.

[0095] The judgment module is used to determine whether the impact of the train on the terminal voltage of the central substation at different locations is greater than the threshold voltage. If it is greater than the threshold voltage, the effective range is determined to include both the central substation and the substation where the train is located.

[0096] In one embodiment, the robustness enhancement module includes:

[0097] The pre-training module is used to obtain the correspondence between the number of communication delays, the degree of delay, and the fusion ratio through pre-training.

[0098] The proportional output module is used to obtain the fusion ratio of offline charging and discharging actions and online charging and discharging actions based on the correspondence and according to the number and degree of communication delay.

[0099] In one embodiment, the pre-training module includes:

[0100] The initialization module is used to initialize the fusion ratio.

[0101] The first action acquisition module is used to acquire online charging and discharging actions based on the state of the energy storage system under any communication delay quantity and delay level.

[0102] The second action acquisition module is used to acquire offline charging and discharging actions based on the state of the energy storage system; and to calculate fused charging and discharging actions based on online charging and discharging actions, offline charging and discharging actions, and fusion ratio.

[0103] The execution module is used to execute offline charging and discharging actions and fused charging and discharging actions respectively, and obtain a first reward signal based on the fused charging and discharging actions and a second reward signal based on the offline charging and discharging actions.

[0104] The update module is used to update the fusion ratio based on the first reward signal and the second reward signal. If the first reward signal is greater than the second reward signal, the fusion ratio is increased; if the first reward signal is less than the second reward signal, the fusion ratio is decreased.

[0105] The repeat module is used to repeatedly update the fusion ratio until the rate of change of the fusion ratio reaches the termination value.

[0106] In one embodiment, the workflow of the control model of the rail transit energy storage system of this invention is as follows: Figure 16 As shown, it includes:

[0107] Step 1: Call the real-time partitioning model and determine the scale that needs to be trained.

[0108] Step 2: Call the offline generalization model and use its output as the initial grid for the deep reinforcement learning model.

[0109] Step 3: Take the state s (within the effective range determined by the real-time partitioning model: substation no-load voltage and output current, train position and power, and remaining power of the energy storage device) as input, use a greedy algorithm as the action selection strategy, use the reward generated by the action as feedback, and use gradient descent as the parameter update algorithm to iteratively update the parameters of the neural network model and train the deep neural network model.

[0110] Step 4: At the same time, at regular intervals, the complete training data and network parameters are stored in the experience replay module. During training, samples are randomly taken from the experience replay module to break the correlation between the training data before and after.

[0111] Step 5: The offline generalization module determines the offline charging and discharging actions (Action 1) based on the state of the energy storage system, and the deep reinforcement learning module determines the online charging and discharging actions (Action 2) based on the state of the energy storage system.

[0112] Step 6: Call the robustness enhancement module to obtain a suitable fusion ratio k based on the current data transmission delay. The action 1 output by the deep reinforcement learning module and the action 2 output by the offline generalization model are fused with the k value to output the final charging and discharging threshold action, and the k value is continuously updated in real time with small increments.

[0113] Step 7: The actual physical system operates according to the final charge and discharge threshold output and calculates the reward information, and feeds the reward information back to the deep reinforcement learning module for learning.

[0114] This invention's control model for a rail transit energy storage system, based on the DQN reinforcement learning algorithm and incorporating an offline generalization model, a real-time partitioning module, an experience replay module, and a robustness enhancement module, achieves online real-time global optimal control of the energy storage system for the first time. Specifically, addressing the issue that existing global optimization algorithms cannot operate online in real time, a behavior cloning approach is introduced, using the offline generalization model as the initial input for reinforcement learning, thus improving the algorithm's generalization ability. To avoid the problem of the algorithm processing overly complex information, a real-time partitioning control concept is introduced, proposing a real-time partitioning module to improve the algorithm's convergence ability and computation speed. A well-designed neural network is used to fit the action-value function, and techniques such as "experience replay" and "independent target networks" are incorporated into the algorithm to improve convergence speed. Considering issues such as communication latency, a robustness enhancement module is proposed for the first time, enhancing the robustness of the reinforcement learning algorithm.

[0115] This invention also provides an electronic device, such as... Figure 17As shown, the system includes a memory 12 and a processor 11, which are interconnected. The memory 12 stores computer instructions, and the processor 11 executes these computer instructions to perform the control method for the rail transit energy storage system as described in the above-described method embodiment of the present invention. The processor 11 and the memory 12 can be connected via a bus or other means. The processor 11 can be a central processing unit (CPU). The processor 11 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The memory 12, as a non-transitory computer storage medium, can be used to store non-transitory software programs, non-transitory executable programs, and modules, such as the corresponding program instructions / modules in the embodiments of the present invention. The processor 11 executes various functional applications and data processing by running non-transitory software programs, instructions, and modules stored in the memory 12, thereby implementing the control method of the rail transit energy storage system in the above method embodiments. The memory 12 may include a program storage area and a data storage area. The program storage area may store the application program required for operating the device and at least one function; the data storage area may store data created by the processor 11, etc. Furthermore, the memory 12 may include a high-speed random access memory 12, and may also include non-transitory memory 12, such as at least one disk storage device 12, a flash memory device, or other non-transitory solid-state memory 12. In some embodiments, the memory 12 may optionally include memory 12 remotely located relative to the processor 11, and these remote memories 12 can be connected to the processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. One or more modules are stored in the memory 12 and, when executed by the processor 11, perform the control method of the rail transit energy storage system as described in the above method embodiments. Specific details of the above electronic equipment can be understood by referring to the corresponding descriptions and effects in the above method embodiments, and will not be repeated here.

[0116] This invention also provides a computer-readable storage medium, such as... Figure 18As shown, a computer program 13 is stored on the storage medium. When executed by a processor, this program implements the steps of the control method for the rail transit energy storage system in the above embodiments. The storage medium also stores audio and video stream data, feature frame data, interactive request signaling, encrypted data, and preset data sizes. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program 13 can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0117] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method for a rail transit energy storage system, characterized in that, include: Based on the offline algorithm, offline charging and discharging actions are determined according to the state of the energy storage system; Based on the state of the energy storage system, online charging and discharging actions are determined using a deep reinforcement learning algorithm. The fusion ratio of the offline charging / discharging action and the online charging / discharging action is obtained based on the number and degree of communication delay. The offline charging and discharging actions and the online charging and discharging actions are fused together according to the fusion ratio and output to the energy storage system; The step of obtaining the fusion ratio of the offline charging / discharging action and the online charging / discharging action based on the number and degree of communication delay includes: The correspondence between the number and degree of arbitrary communication delays and the fusion ratio can be obtained through pre-training; Based on the aforementioned correspondence, the fusion ratio of the offline charging / discharging action and the online charging / discharging action is obtained according to the number and degree of communication delay. The step of obtaining the correspondence between the number of communication delays, the degree of delay, and the fusion ratio through pre-training includes: Initialize the fusion ratio; Under any amount and degree of communication delay, the online charging and discharging actions are obtained based on the state of the energy storage system; The offline charging and discharging actions are obtained based on the status of the energy storage system; The fused charge / discharge action is calculated based on the online charge / discharge action, the offline charge / discharge action, and the fusion ratio. The offline charging and discharging action and the fused charging and discharging action are executed respectively to obtain a first reward signal based on the fused charging and discharging action and a second reward signal based on the offline charging and discharging action; The fusion ratio is updated based on the first reward signal and the second reward signal, wherein if the first reward signal is greater than the second reward signal, the fusion ratio is increased, and if the first reward signal is less than the second reward signal, the fusion ratio is decreased. Repeat the step of updating the fusion ratio until the rate of change of the fusion ratio reaches the termination value.

2. The control method for the rail transit energy storage system according to claim 1, characterized in that, The method for determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm includes: Receive the status of the energy storage system and the offline charging and discharging actions; The offline charging and discharging actions are used as the initial values ​​for the neural network, and the neural network is trained using training data. The neural network outputs an action-value function based on the state of the energy storage system. Online charging and discharging actions are obtained based on action-value functions and greedy strategies.

3. The control method for the rail transit energy storage system according to claim 2, characterized in that, The method for determining online charging and discharging actions based on the state of the energy storage system using a deep reinforcement learning algorithm also includes: The used training data is stored, and training data is randomly drawn from the used training data to retrain the neural network.

4. The control method for the rail transit energy storage system according to claim 1, characterized in that, Before determining offline charging and discharging actions based on the state of the energy storage system using an offline algorithm, the following steps are also included: The effective range of the energy storage system is obtained, and the state of the energy storage system includes the state of the substation, the train, and the energy storage device within the effective range.

5. The control method for the rail transit energy storage system according to claim 4, characterized in that, The effective range of the energy storage system includes: Select a central substation; Determine whether the impact of the train at different locations on the terminal voltage of the central substation is greater than the threshold voltage; If the voltage exceeds the threshold voltage, the effective range is determined to include both the central substation and the substation where the train is located.

6. A control model for a rail transit energy storage system, characterized in that, include: The offline generalization module is used to determine offline charging and discharging actions based on the state of the energy storage system using an offline algorithm; The deep reinforcement learning module is used to determine online charging and discharging actions based on the state of the energy storage system using deep reinforcement learning algorithms. The robustness enhancement module is used to obtain the fusion ratio of the offline charging and discharging action and the online charging and discharging action based on the number and degree of communication delay, and to fuse the offline charging and discharging action and the online charging and discharging action according to the fusion ratio and output them to the energy storage system. The robustness enhancement module includes: The pre-training module is used to obtain the correspondence between the number and degree of arbitrary communication delays and the fusion ratio through pre-training; The proportional output module is used to obtain the fusion ratio of offline charging and discharging actions and online charging and discharging actions based on the correspondence and according to the number and degree of communication delay; The pre-training module includes: The initialization module is used to initialize the fusion ratio; The first action acquisition module is used to acquire online charging and discharging actions based on the state of the energy storage system under any communication delay quantity and delay level. The second action acquisition module is used to acquire offline charging and discharging actions based on the state of the energy storage system; and to calculate fused charging and discharging actions based on online charging and discharging actions, offline charging and discharging actions, and fusion ratio. The execution module is used to execute offline charging and discharging actions and fused charging and discharging actions respectively, and to obtain a first reward signal based on the fused charging and discharging actions and a second reward signal based on the offline charging and discharging actions. The update module is used to update the fusion ratio based on the first reward signal and the second reward signal, wherein if the first reward signal is greater than the second reward signal, the fusion ratio is increased, and if the first reward signal is less than the second reward signal, the fusion ratio is decreased. The repeat module is used to repeatedly update the fusion ratio until the rate of change of the fusion ratio reaches the termination value.

7. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the control method for the rail transit energy storage system as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the control method for the rail transit energy storage system as described in any one of claims 1 to 5.

Images