A method for charge and discharge control of energy storage system based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's energy storage system charge-discharge control method, comprising the following steps: obtaining energy storage control state, forms continuous charge-discharge action flow;Continuous charge-discharge action flow is divided, forms cross-border weaving track;Get original sub-track, adjust charge-discharge power distribution, form boundary conservation shadow sub-track;Using Diaster algorithm decomposes original sub-track and boundary conservation shadow sub-track, determines closable responsibility and unclosed responsibility;Obtain modified action feasible region;When another boundary responsibility is closed, connect overlapping responsibility unit to form closed sub-track, reusing Diaster algorithm, update modified action feasible region, obtain final agent reward;Based on final agent reward, updated modified action feasible region generates target charge-discharge action of next control time.This application can quasi-dynamic revision charge-discharge action range, improve energy storage system charge-discharge control rationality.

Description

Technical Field

[0001] This invention relates to the field of energy storage system control technology, and in particular to a charging and discharging control method for energy storage systems based on reinforcement learning. Background Technology

[0002] With the increasing proportion of renewable energy connected to the grid and the continued application of peak-valley electricity pricing mechanisms, energy storage systems are gradually taking on tasks such as peak shaving and valley filling, demand control, power regulation, and operating cost optimization. Existing energy storage system charging and discharging control methods typically construct the energy storage operating state based on state of charge, battery temperature, load power, time-of-use pricing, and public grid switching power, and use rule-based control, model predictive control, or reinforcement learning models to generate charging, discharging, and standby actions. Reinforcement learning methods can evaluate charging and discharging actions through operational rewards and improve energy storage control performance using long-term rewards, making them suitable for energy storage operation scenarios with complex state changes and coupled control objectives.

[0003] Existing reinforcement learning-based energy storage control methods typically allocate rewards according to fixed time steps or calculate trajectory rewards uniformly after the completion of a full scheduling cycle. However, the demand statistics window and the battery cycle have different start and end boundaries, and a single charge / discharge action may simultaneously affect the current demand statistics result and the battery cycle that has not yet ended. When the demand statistics window ends first while the battery cycle is still continuing, or when the battery cycle closes first while the demand statistics window has not yet ended, existing methods struggle to distinguish between established reward responsibilities and unclosed reward responsibilities, easily distributing delayed rewards evenly across all control positions, leading to distorted attribution of action contributions.

[0004] Therefore, how to provide a reinforcement learning-based method for controlling the charging and discharging of energy storage systems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a charging and discharging control method for energy storage systems based on reinforcement learning. This invention constructs cross-boundary woven trajectories and boundary-conserving shadow sub-trajectories, uses the Diaster algorithm to separate closable and unclosable responsibility quantities, and dynamically corrects the action feasible region accordingly. This can improve the accuracy of delayed reward attribution, reduce the impact of boundary mismatch on charging and discharging decisions, and enhance the rationality and stability of charging and discharging control of energy storage systems.

[0006] A method for controlling the charging and discharging of an energy storage system based on reinforcement learning according to an embodiment of the present invention includes the following steps: The energy storage control state is obtained, the basic action feasible domain is constructed, the target charging and discharging action is obtained and executed through a reinforcement learning model, forming a continuous charging and discharging action flow; The continuous charging and discharging action flow is divided according to the demand statistics window boundary and the battery cycle boundary, forming overlapping responsibility units. When one boundary responsibility is closed first, the other boundary responsibility is continued across the boundary, forming a cross-boundary weaving trajectory. The original sub-trajectory is obtained by dividing the cross-boundary weaving trajectory with overlapping responsibility units. Under the condition that the energy storage boundary state and energy balance remain unchanged, the charging and discharging power distribution is adjusted to form the boundary-conserving shadow sub-trajectory. The Diaster algorithm is used to decompose the original sub-trajectory and the boundary-conserving shadow sub-trajectory. The initial agent reward is obtained based on the difference in expected reward between adjacent sub-trajectories. The boundary-conserving liability is obtained based on the difference in implicit reward between the two types of sub-trajectories. The closable liability and the unclosable liability are then determined. The feasible region of the basic action is modified based on the unclosed liability quantity to obtain the modified action feasible region; When another boundary responsibility is closed, the overlapping responsibility units are connected to form a closed sub-trajectory. The Diaster algorithm is re-adopted, and the action feasible region is updated and corrected in combination with the closable responsibility quantity to obtain the final agent reward. Based on the final agent reward and the updated feasible domain of the corrected action, the target charging and discharging action for the next control moment is generated.

[0007] Optionally, obtaining the basic action feasible region and the continuous charge / discharge action flow specifically includes: Collect the operating data and external dispatch data of the energy storage system at the current control time, and correlate them according to the current control time to form the energy storage control status; The basic action feasible domain is constructed based on the energy storage control state. The energy storage control state is input into the reinforcement learning model to obtain candidate charging and discharging actions. The candidate charging and discharging actions are then subjected to action constraint processing based on the basic action feasible domain to obtain the target charging and discharging action. The target charging and discharging action is converted into the corresponding charging and discharging power control command of the energy storage converter. The energy storage system is controlled to execute the charging and discharging power control command, and the corresponding action execution result is recorded. The energy storage control state, target charging and discharging action and action execution result formed at the continuous control time are arranged according to the control time to form a continuous charging and discharging action stream.

[0008] Optionally, obtaining the cross-boundary weaving trajectory and overlapping responsibility units specifically includes: Read the control time and action execution result corresponding to each target charging and discharging action in the continuous charging and discharging action stream, determine the demand statistics window boundary according to the demand statistics cycle, determine the battery cycle boundary according to the direction of state of charge change and the range of state of charge change, and arrange the demand statistics window boundary and the battery cycle boundary according to the control time to form a responsibility boundary sequence; Based on the responsibility boundary sequence, the continuous charging and discharging action flow is processed to construct responsibility units, forming overlapping responsibility units. Double boundary responsibility mapping is performed on the overlapping responsibility units to obtain demand boundary responsibility and cyclic boundary responsibility, and the responsibility status of demand boundary responsibility and cyclic boundary responsibility is determined. The responsibility status includes open state and closed state. Determine the closing order of demand boundary responsibility and cyclic boundary responsibility. When one boundary responsibility is closed first, the other boundary responsibility that is in an open state continues across the boundary and continues to associate with the next overlapping responsibility unit. Select overlapping responsibility units that bear the same open boundary responsibility. Determine the connection relationship between adjacent overlapping responsibility units according to the continuity of charge state. Connect overlapping responsibility units in sequence according to the execution order of target charging and discharging actions. Terminate the connection when the other boundary responsibility is closed to form a cross-boundary weaving trajectory.

[0009] Optionally, obtaining the original sub-trajectory and the boundary-conserving shadow sub-trajectory specifically includes: In the cross-boundary weaving trajectory, an overlapping responsibility unit is identified where one boundary responsibility closes first, and the next overlapping responsibility unit connected to the overlapping responsibility unit is identified along the cross-boundary continuation direction of another boundary responsibility. The two overlapping responsibility units are combined according to the connection order in the cross-boundary weaving trajectory to form the original sub-trajectory. The starting energy storage control state and the ending energy storage control state of the original sub-trajectory are extracted to form the energy storage boundary state. Within an overlapping responsibility unit where a boundary responsibility is closed first, the energy transfer-out control position is determined, and within the next overlapping responsibility unit, the energy transfer-in control position is determined. Part of the charging and discharging energy carried by the energy transfer-out control position is transferred to the energy transfer-in control position, and the charging and discharging power distribution of the original sub-trajectory is reconstructed to form the shadow charging and discharging power distribution. The shadow trajectory is generated based on the shadow charging and discharging power distribution. The shadow trajectory and the original sub-trajectory are checked for boundary conservation and energy balance. If either check result does not meet the corresponding conservation condition, the charging and discharging energy transfer amount is corrected, and the power distribution is reconstructed and the shadow trajectory is generated again. If both check results meet the corresponding conservation conditions, the shadow trajectory is determined as the boundary-conserved shadow sub-trajectory.

[0010] Optionally, obtaining the closed and unclosed liability quantities specifically includes: The original sub-trajectory and the corresponding boundary-conserving shadow sub-trajectory are combined to form a coupled trajectory pair. The overlapping responsibility unit and the next overlapping responsibility unit that closes the boundary responsibility first are determined. The responsibility splitting position is set between the two overlapping responsibility units. The responsibility migration interval is limited according to the energy transfer-out control position and the energy transfer-in control position. Establish a synchronous position mapping according to the control position order in the coupled trajectory pair, mark the control position outside the responsibility migration interval as non-migration control position, set implicit reward increment equal value constraint for the corresponding non-migration control positions in the two types of sub-trajectories, set implicit reward difference allocation mark for the control position within the responsibility migration interval, and form a set of coupled reward constraint. The original trajectory reward is determined based on the action execution result of the original sub-trajectory, and the shadow trajectory reward is determined based on the shadow action execution result of the boundary-conserving shadow sub-trajectory. The coupled trajectory pairs, synchronization position mapping, coupled reward constraint set, original trajectory reward, and shadow trajectory reward are jointly input into the Diaster algorithm. The original prefix sub-trajectory and shadow prefix sub-trajectory are constructed step by step along the synchronization position. The expected implicit reward corresponding to each original prefix sub-trajectory and each shadow prefix sub-trajectory is jointly estimated. During the implicit reward allocation process, the implicit reward increment corresponding to the non-migration control position is kept the same. Only the implicit reward increment within the responsibility migration interval is adjusted so that the cumulative result of the implicit reward increment of each control position in the original sub-trajectory is consistent with the original trajectory reward, and the cumulative result of the implicit reward increment of each control position in the boundary-conserving shadow sub-trajectory is consistent with the shadow trajectory reward. According to the control position order, the expected implicit reward of the current original prefix sub-trajectory is subtracted from the expected implicit reward of the previous original prefix sub-trajectory to obtain the initial agent reward corresponding to the current control position. The expected implicit reward increments of the original prefix sub-trajectory and the shadow prefix sub-trajectory within the responsibility migration interval are compared according to the synchronization position, and the difference between the two expected implicit reward increments is determined as the boundary conservation responsibility quantity corresponding to the current control position. Based on the responsibility status corresponding to the overlapping responsibility unit where a boundary responsibility is closed first, the initial agency reward is decomposed into responsibility, the boundary conservation responsibility quantity is determined as the closable responsibility quantity, and the remaining quantity after deducting the closable responsibility quantity from the initial agency reward is determined as the unclosable responsibility quantity.

[0011] Optionally, obtaining the feasible domain of the correction action specifically includes: For unclosed liability quantities, perform liability action mapping processing and directional liability determination processing to obtain the liability maintenance direction, liability inhibition direction, and liability correction intensity; Based on the direction of responsibility maintenance, the direction of responsibility inhibition, and the intensity of responsibility correction, the feasible domain of basic actions is subjected to responsibility boundary correction processing to form a responsibility correction boundary; The boundary of responsibility correction is merged with the feasible domain of basic actions to form the feasible domain of corrective actions.

[0012] Optionally, obtaining the final agent reward specifically includes: When another boundary responsibility reaches a closed state, the overlapping responsibility units that have closed one boundary responsibility first, the overlapping responsibility units formed during the cross-boundary continuation period, and the overlapping responsibility units that have reached a closed state of another boundary responsibility are extracted along the cross-boundary weaving trajectory. They are spliced ​​together according to the connection order of the overlapping responsibility units to form a closed sub-trajectory. Using the complete trajectory reward corresponding to the closed sub-trajectory as the decomposition target and the closable responsibility quantity as the closed responsibility locking item, the Diaster reward decomposition is re-executed. The splitting positions are set sequentially along the control positions in the closed sub-trajectory, dividing the closed sub-trajectory into prefix sub-trajectories and remaining sub-trajectories. The implicit rewards of the prefix sub-trajectories and remaining sub-trajectories are allocated respectively. The remaining complete trajectory reward is allocated to the control position corresponding to the cross-boundary continuation period of the unclosed responsibility quantity. Based on the difference in expected implicit rewards between adjacent prefix sub-trajectories, the closed proxy responsibility quantity is obtained. The closed agent responsibility quantity is compared with the open responsibility quantity at the corresponding control position. The responsibility correction result is determined based on the numerical difference and positive / negative direction of the two. The feasible domain of the corrected action is then updated based on the responsibility correction result to form the updated feasible domain of the corrected action. According to the same control position, the closable responsibility quantity and the closing agent responsibility quantity are merged to obtain the final agent reward, and the final agent reward is then associated with the updated feasible domain of the corrective action.

[0013] Optionally, the generation of the target charging / discharging action at the next control moment specifically includes: The final agent reward is summarized according to the control position. Based on the positive or negative direction of the final agent reward and the charging or discharging direction of the corresponding target charging or discharging action, the action selection tendency at the next control moment is determined. The energy storage control state at the next control moment is obtained and input into the reinforcement learning model to generate the next candidate charging and discharging action. The target action direction is selected from the updated corrected action feasible domain according to the action selection tendency. When the charging and discharging direction of the next candidate charging and discharging action is consistent with the target action direction, it is retained. When they are inconsistent, it is adjusted to the target action direction to form a direction-corrected candidate charging and discharging action. The candidate charge / discharge actions for direction correction are matched with the updated feasible domain of the corrected actions to form the target charge / discharge action for the next control moment.

[0014] The beneficial effects of this invention are: This invention addresses the problem that the demand statistics window and battery cycle have different boundaries, and the operational impact of charging and discharging actions cannot be evaluated at the same control moment. It introduces the Diaster algorithm to decompose trajectory rewards. The Diaster algorithm can allocate delayed rewards generated after the demand statistics window ends or the battery cycle closes to the corresponding control positions, and generate initial surrogate rewards based on the expected implicit rewards differences between adjacent sub-trajectories. This avoids the confusion caused by the average allocation method, which leads to differences in the contributions of different charging and discharging actions, and improves the accuracy of the correspondence between long-term operational rewards and specific charging and discharging actions.

[0015] This invention does not directly employ the conventional Diaster algorithm. Instead, it constructs a cross-boundary weaving trajectory based on the overlap between demand boundary responsibility and cyclic boundary responsibility. Under the condition that the energy storage boundary state and energy balance remain unchanged, it constructs a boundary-conserving shadow sub-trajectory. By forming a coupled trajectory pair with the original sub-trajectory and the boundary-conserving shadow sub-trajectory, it locks the implicit reward increment of non-migrating control positions, allocating reward differences only within the responsibility migration interval. Through these improvements, it can eliminate the interference of energy storage boundary state, net energy change, and unadjusted control position on the reward comparison results. It extracts the reward change caused by the redistribution of charging and discharging energy across overlapping responsibility units as the boundary-conserving responsibility quantity, and further distinguishes between closable and unclosable responsibility quantities.

[0016] This invention utilizes unclosed responsibility quantities to dynamically correct the feasible region of basic actions, allowing unclosed demand boundary responsibilities or cyclic boundary responsibilities to continue constraining the charging and discharging direction and power range. After another boundary responsibility is closed, the invention locks the closable responsibility quantity and re-applies the Diaster algorithm to correct the unclosed responsibility quantity, obtaining the final surrogate reward and the updated corrected feasible region of actions. This reduces invalid charging and discharging direction switching and power allocation deviations caused by boundary responsibility mismatches, improves the coordination between the target charging and discharging action and the demand control target, battery cycle state, and energy storage safety boundary at the next control moment, and thus enhances the accuracy, stability, and continuity of energy storage system charging and discharging control. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a reinforcement learning-based energy storage system charging and discharging control method proposed in this invention. Figure 2 This is a schematic diagram of the boundary conservation shadow sub-trajectory coupled Diaster reward decomposition of the energy storage system charging and discharging control method based on reinforcement learning proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-2 A method for charging and discharging control of an energy storage system based on reinforcement learning includes the following steps: The energy storage control state is obtained, the basic action feasible domain is constructed, the target charging and discharging action is obtained and executed through a reinforcement learning model, forming a continuous charging and discharging action flow; The continuous charging and discharging action flow is divided according to the demand statistics window boundary and the battery cycle boundary, forming overlapping responsibility units. When one boundary responsibility is closed first, the other boundary responsibility is continued across the boundary, forming a cross-boundary weaving trajectory. The original sub-trajectory is obtained by dividing the cross-boundary weaving trajectory with overlapping responsibility units. Under the condition that the energy storage boundary state and energy balance remain unchanged, the charging and discharging power distribution is adjusted to form the boundary-conserving shadow sub-trajectory. The Diaster algorithm is used to decompose the original sub-trajectory and the boundary-conserving shadow sub-trajectory. The initial agent reward is obtained based on the difference in expected reward between adjacent sub-trajectories. The boundary-conserving liability is obtained based on the difference in implicit reward between the two types of sub-trajectories. The closable liability and the unclosable liability are then determined. The feasible region of the basic action is modified based on the unclosed liability quantity to obtain the modified action feasible region; When another boundary responsibility is closed, the overlapping responsibility units are connected to form a closed sub-trajectory. The Diaster algorithm is re-adopted, and the action feasible region is updated and corrected in combination with the closable responsibility quantity to obtain the final agent reward. Based on the final agent reward and the updated feasible domain of the corrected action, the target charging and discharging action for the next control moment is generated.

[0020] In this embodiment, obtaining the basic feasible domain of actions and the continuous charge-discharge action flow specifically includes: The system collects operational data and external dispatch data of the energy storage system at the current control time, and correlates them according to the current control time to form an energy storage control state. The energy storage system is an energy storage and regulation system composed of energy storage batteries, energy storage converters, and energy storage control units, used to store electrical energy and perform charging, discharging, and standby operations according to charging and discharging control commands. The operational data includes state of charge, battery health status, battery temperature, current charging and discharging power, and current charging and discharging direction. The external dispatch data includes power load power, public grid switching power, time-of-use electricity price, current peak demand, and remaining duration of the demand statistics window. The energy storage control state is a set of state data reflecting the current operating conditions and current dispatch conditions of the energy storage system. The rated capacity of the energy storage battery is set to 1000kWh, the rated charging power and rated discharging power of the energy storage converter are both set to 500kW, the allowable state of charge range is set to 20% to 90%, the upper limit of the allowable battery temperature is set to 45℃, the charging efficiency is set to 95%, the discharging efficiency is set to 94%, and the interval between adjacent control moments is set to 5min. A basic action feasible domain is constructed based on the energy storage control state. The energy storage control state is input into the reinforcement learning model to obtain candidate charging and discharging actions. The candidate charging and discharging actions are then subjected to action constraint processing based on the basic action feasible domain to obtain the target charging and discharging action. The basic action feasible domain is the range of charging and discharging actions that the energy storage system is allowed to perform at the current control moment, including the allowable charging power range, the allowable discharging power range, standby actions, and charging and discharging direction switching conditions. The candidate charging and discharging actions are the action direction and action power output by the reinforcement learning model based on the energy storage control state. The action constraint processing is to retain the candidate charging and discharging actions that are within the basic action feasible domain and adjust the candidate charging and discharging actions that are outside the basic action feasible domain to the corresponding action boundary. The reinforcement learning model includes a state input layer, a state feature layer, and an action output layer. The state input layer receives a fixed-length numerical vector corresponding to the energy storage control state. The state feature layer performs weighted mapping and nonlinear feature extraction on the fixed-length numerical vector. The action output layer outputs candidate charging and discharging actions based on the extraction results. The candidate charging and discharging actions include charging and discharging direction and charging and discharging power. The action output layer uses a hyperbolic tangent function to limit the output value to between -1 and 1. Negative values ​​correspond to charging actions, zero values ​​correspond to standby actions, and positive values ​​correspond to discharging actions. The absolute value of the output value is used to determine the candidate charging and discharging power. The training data comes from the historical operation records and online operation records of the energy storage system. The training samples include the energy storage control state and candidate charging and discharging actions. The trainable parameters of the state feature layer and the action output layer are updated according to the training loss. The training control parameters include the learning rate, batch sample size, and maximum number of training rounds. Model training stops when the change in training loss corresponding to consecutive training rounds is not greater than the training convergence threshold. The reinforcement learning model has 10 input nodes in its state input layer, two fully connected feature extraction layers in its state feature layer (64 and 32 feature nodes respectively), and one action output node in its action output layer. The model is trained using 18,000 historical training records and 2,400 online training records. The learning rate is set to 0.001, the batch size to 128, and the maximum number of training epochs to 300. Training is stopped when the loss change is no greater than 0.0001 over 20 consecutive training epochs. The target charging and discharging action is converted into a corresponding charging and discharging power control command for the energy storage converter. The energy storage system is controlled to execute the charging and discharging power control command, and the corresponding action execution results are recorded. The energy storage control state, target charging and discharging action, and action execution results formed during continuous control are arranged according to the control time to form a continuous charging and discharging action stream. The energy storage converter is an energy conversion device that realizes bidirectional conversion between DC power from the energy storage battery and AC power from the grid according to the charging and discharging power control command, and adjusts the charging and discharging power of the energy storage system. The action execution results are the energy storage operation change results and grid exchange change results formed after the energy storage system executes the target charging and discharging action. The energy storage operation change results include changes in state of charge, charging and discharging power, charging and discharging direction, and battery temperature. The grid exchange change results include changes in public grid exchange power and current peak demand. The continuous charging and discharging action stream is a data sequence arranged according to the control time and maintaining the correspondence between the energy storage control state, target charging and discharging action, and action execution results.

[0021] In this embodiment, obtaining the cross-boundary weaving trajectory and overlapping responsibility units specifically includes: The system reads the control time and execution result of each target charging and discharging action in the continuous charging and discharging action stream. It determines the demand statistics window boundary based on the demand statistics cycle and the battery cycle boundary based on the direction of state of charge change and the state of charge change interval. The demand statistics window boundary and the battery cycle boundary are arranged according to the control time to form a responsibility boundary sequence. The demand statistics window boundary includes the start boundary and the end boundary of the demand statistics cycle. The battery cycle boundary includes the cycle stage boundary formed when the direction of state of charge change changes and the cycle closure boundary formed when the state of charge change interval is closed. The direction of state of charge change is determined according to the increase or decrease relationship of the state of charge corresponding to adjacent control times. The state of charge change interval is the interval formed by the change of the state of charge from the state of charge value corresponding to the cycle stage boundary to the state of charge value corresponding to another cycle stage boundary. Based on the responsibility boundary sequence, the continuous charging and discharging action flow is processed to construct responsibility units, forming overlapping responsibility units. Double boundary responsibility mapping is performed on the overlapping responsibility units to obtain demand boundary responsibility and cyclic boundary responsibility, and the responsibility status of demand boundary responsibility and cyclic boundary responsibility is determined. The responsibility status includes open state and closed state. The responsibility unit construction process is as follows: A continuous charge / discharge action flow is divided according to the time interval defined by adjacent responsibility boundaries in the responsibility boundary sequence, and the energy storage control state, target charge / discharge action, and action execution result within the same time interval are associated. The overlapping responsibility unit is a continuous charge / discharge action segment that simultaneously corresponds to the demand statistics window and the battery cycle. The dual-boundary responsibility mapping is as follows: Demand boundary responsibility is generated based on the time attribution relationship between the overlapping responsibility unit and the demand statistics window boundary, and cycle boundary responsibility is generated based on the state of charge change relationship between the overlapping responsibility unit and the battery cycle boundary. The demand boundary responsibility is the delayed reporting responsibility undertaken by the overlapping responsibility unit for its respective demand statistics window, and the cycle boundary responsibility is the delayed reporting responsibility undertaken by the overlapping responsibility unit for its respective battery cycle. The open state indicates that the corresponding boundary responsibility has not yet been closed, and the closed state indicates that the corresponding boundary responsibility has been closed. The closure order of demand boundary responsibility and cyclic boundary responsibility is determined. When one boundary responsibility is closed first, the other boundary responsibility that is in an open state continues across the boundary and continues to be associated with the next overlapping responsibility unit. Overlapping responsibility units that bear the same open state boundary responsibility are selected. The connection relationship between adjacent overlapping responsibility units is determined according to the continuity of charge state. Overlapping responsibility units are connected sequentially according to the execution order of the target charging and discharging action. The connection is terminated when the other boundary responsibility is closed, forming a cross-boundary weaving trajectory. The cross-boundary continuation means that after one boundary responsibility is closed, the other boundary responsibility that is in an open state continues to be associated with the next overlapping responsibility unit. The cross-boundary weaving trajectory is the responsibility trajectory formed by the overlapping responsibility units that bear the same open state boundary responsibility being connected sequentially according to the execution order of the target charging and discharging action.

[0022] In a series of continuous charge and discharge operations, a total of 12 demand statistics windows are identified, each containing 3 control positions. At the same time, 3 cycle stage boundaries and 2 cycle closure boundaries are identified based on the direction of charge state change. The demand statistics window boundaries and battery cycle boundaries are merged and arranged to form 15 responsibility boundaries. The time intervals defined by adjacent responsibility boundaries form 14 overlapping responsibility units, each containing 1 to 3 control positions.

[0023] In this embodiment, obtaining the original sub-trajectory and the boundary-conserving shadow sub-trajectory specifically includes: In the cross-boundary weaving trajectory, an overlapping responsibility unit is identified where a boundary responsibility closes first, and the next overlapping responsibility unit connected to the overlapping responsibility unit is identified along the cross-boundary continuation direction of another boundary responsibility. The two overlapping responsibility units are combined according to the connection order in the cross-boundary weaving trajectory to form an original sub-trajectory. The starting energy storage control state and the ending energy storage control state of the original sub-trajectory are extracted to form the energy storage boundary state. The original sub-trajectory crosses the closing position of a boundary responsibility. The two overlapping responsibility units in the original sub-trajectory jointly undertake the same open state boundary responsibility. When the demand boundary responsibility closes first, the open state boundary responsibility is a cyclic boundary responsibility. When the cyclic boundary responsibility closes first, the open state boundary responsibility is a demand boundary responsibility. The starting energy storage control state is taken from the overlapping responsibility unit where a boundary responsibility closes first, and the ending energy storage control state is taken from the next overlapping responsibility unit. Within an overlapping responsibility unit where a boundary responsibility is closed first, an energy transfer-out control position is determined. Within the next overlapping responsibility unit, an energy transfer-in control position is determined. A portion of the charging and discharging energy carried by the energy transfer-out control position is transferred to the energy transfer-in control position. The charging and discharging power distribution of the original sub-trajectory is reconstructed, forming a shadow charging and discharging power distribution. The target charging and discharging actions corresponding to the energy transfer-out control position and the energy transfer-in control position have the same charging and discharging direction and jointly bear the same open state boundary responsibility. The amount of charging and discharging energy transferred is determined based on the charging and discharging energy that can be reduced at the energy transfer-out control position and the charging and discharging energy that can be increased at the energy transfer-in control position. Based on the duration of the control interval and the charging efficiency or discharging efficiency, the amount of charging and discharging energy transferred is converted into the charging and discharging power adjustment amount of the two control positions respectively. The absolute value of the charging and discharging power at the energy transfer-out control position is reduced, and the absolute value of the charging and discharging power at the energy transfer-in control position is increased. The charging and discharging power of the remaining control positions in the original sub-trajectory remains unchanged, and the adjusted charging and discharging power is within the feasible domain of the basic action. A shadow trajectory is generated based on the shadow charging and discharging power distribution. Boundary conservation and energy balance checks are performed between the shadow trajectory and the original sub-trajectory. If either check result fails to meet the corresponding conservation condition, the charging and discharging energy transfer amount is corrected, and the power distribution is reconstructed and the shadow trajectory is generated again. If both checks meet the corresponding conservation conditions, the shadow trajectory is determined as a boundary-conserved shadow sub-trajectory. The shadow trajectory uses the initial energy storage control state of the original sub-trajectory as its starting state, maintaining the overlapping responsibility unit composition, control position sequence, control interval duration, external scheduling data, demand boundary responsibility attribution, cycle boundary responsibility attribution, and boundary responsibility closure sequence unchanged from the original sub-trajectory. The shadow charging and discharging... The power distribution is written to the corresponding control position, and candidate trajectories formed by shadow charging and discharging actions, shadow energy storage control states, and shadow action execution results are generated in sequence. The boundary conservation verification compares the termination energy storage control state of the shadow trajectory with the termination energy storage control state of the original sub-trajectory. The energy balance verification compares the net energy change of the shadow trajectory with the net energy change of the original sub-trajectory. The boundary conservation shadow sub-trajectory and the original sub-trajectory have the same initial energy storage control state, termination energy storage control state, and net energy change. Between the boundary conservation shadow sub-trajectory and the original sub-trajectory, only the distribution difference of charging and discharging power between the overlapping responsibility unit where a boundary responsibility is closed first and the next overlapping responsibility unit is retained.

[0024] This invention constructs a boundary-conserving shadow sub-trajectory by migrating a portion of the charge and discharge energy between the boundary responsibility-first closing position and the cross-boundary continuation position, while maintaining the energy storage boundary state, net energy change, and boundary responsibility attribution unchanged. This ensures that the original sub-trajectory and the shadow sub-trajectory retain only the difference in charge and discharge power distribution, providing a stable reference basis for identifying reward changes caused by cross-boundary power adjustments, and improving the accuracy of responsibility quantity extraction and the reliability of trajectory reward decomposition.

[0025] In this embodiment, obtaining the closed and unclosed liability quantities specifically includes: The original sub-trajectory and the corresponding boundary-conserving shadow sub-trajectory are combined to form a coupled trajectory pair. The overlapping responsibility unit and the next overlapping responsibility unit are determined when a boundary responsibility is closed first. A responsibility splitting position is set between the two overlapping responsibility units. The responsibility migration interval is defined according to the energy transfer-out control position and the energy transfer-in control position. The two types of sub-trajectories in the coupled trajectory pair have the same initial energy storage control state, final energy storage control state, net energy change, overlapping responsibility unit connection sequence, external scheduling data, demand boundary responsibility attribution, and cyclic boundary responsibility attribution. Only the distribution difference of charging and discharging power between the overlapping responsibility unit and the next overlapping responsibility unit when a boundary responsibility is closed first is retained between the two types of sub-trajectories. The responsibility splitting position is used to separate the closed boundary responsibility and the open state boundary responsibility that continues across the boundary. The responsibility migration interval is the continuous trajectory interval between the energy transfer-out control position and the energy transfer-in control position. A synchronous position mapping is established according to the control position order in the coupled trajectory pair. Control positions outside the responsibility migration interval are marked as non-migration control positions. Implicit reward increment equal value constraints are set for corresponding non-migration control positions in the two types of sub-trajectories. Implicit reward difference allocation marks are set for control positions within the responsibility migration interval, forming a set of coupled reward constraints. The synchronous position mapping is used to maintain the correspondence between control positions with the same control order in the original sub-trajectory and the boundary-conserving shadow sub-trajectory. The implicit reward increment equal value constraints are used to restrict the Diaster algorithm from allocating the same implicit reward increment to the corresponding control positions where no change in charging or discharging power has occurred. The implicit reward difference allocation marks are used to allow the Diaster algorithm to allocate different implicit reward increments only in the energy transfer-out control position, the energy transfer-in control position, and the responsibility migration interval. The original trajectory reward is determined based on the action execution result of the original sub-trajectory, and the shadow trajectory reward is determined based on the shadow action execution result of the boundary-conserving shadow sub-trajectory. The coupled trajectory pairs, synchronization position mapping, coupled reward constraint set, original trajectory reward, and shadow trajectory reward are jointly input into the Diaster algorithm. The original prefix sub-trajectory and shadow prefix sub-trajectory are constructed step by step along the synchronization position. The expected implicit reward corresponding to each original prefix sub-trajectory and each shadow prefix sub-trajectory is jointly estimated. During the implicit reward allocation process, the implicit reward increment corresponding to the non-migration control position is kept the same. Only the implicit reward increment within the responsibility migration interval is adjusted so that the cumulative result of the implicit reward increment of each control position in the original sub-trajectory is consistent with the original trajectory reward, and the cumulative result of the implicit reward increment of each control position in the boundary-conserving shadow sub-trajectory is consistent with the shadow trajectory reward. The joint estimation is to process the original sub-trajectory and the boundary-conserving shadow sub-trajectory simultaneously in the same Diaster reward decomposition process, and use the coupled reward constraint set to limit the implicit reward allocation range of the two types of sub-trajectories. According to the control position order, the expected implicit reward of the current original prefix sub-trajectory is subtracted from the expected implicit reward of the previous original prefix sub-trajectory to obtain the initial proxy reward corresponding to the current control position. The expected implicit reward increments of the original prefix sub-trajectory and the shadow prefix sub-trajectory within the responsibility migration interval are compared according to the synchronization position. The difference between the two expected implicit reward increments is determined as the boundary conservation responsibility quantity corresponding to the current control position. The initial proxy reward is the expected reward increment formed after the original sub-trajectory adds the current control position. The boundary conservation responsibility quantity is the implicit reward change caused by the migration of charging and discharging energy from the overlapping responsibility unit where one boundary responsibility has been closed to the next overlapping responsibility unit when the energy storage boundary state, net energy change, external scheduling data, and boundary responsibility attribution remain unchanged. Since non-migration control positions are constrained by the same value of implicit reward increment, the boundary conservation responsibility quantity does not include the reward difference generated by control positions where no charging and discharging power change has occurred. Based on the responsibility status corresponding to the overlapping responsibility unit where a boundary responsibility is closed first, the initial agent reward is decomposed into responsibilities. The boundary conservation responsibility quantity is determined as the closable responsibility quantity, and the remaining quantity after deducting the closable responsibility quantity from the initial agent reward is determined as the unclosable responsibility quantity. The closable responsibility quantity is the reward increment that can complete the responsibility attribution based on the already formed boundary reward. The unclosable responsibility quantity is the reward increment that still corresponds to another open boundary responsibility and needs to continue across the boundary. The closable responsibility quantity and the unclosable responsibility quantity maintain the control position and positive and negative direction corresponding to the initial agent reward. The Diaster algorithm is a trajectory reward decomposition algorithm used to divide a complete trajectory into prefix sub-trajectories and residual sub-trajectories at the trajectory segmentation position, assign implicit rewards to the prefix sub-trajectories and residual sub-trajectories respectively, so that the combination result of the two implicit rewards matches the reward of the complete trajectory, and determine the agent reward corresponding to each control position based on the difference in expected implicit rewards corresponding to adjacent prefix sub-trajectories.

[0026] This invention constructs a coupled comparison relationship between the original sub-trajectory and the boundary-conserving shadow sub-trajectory, constraining the Diaster algorithm to allocate reward differences only within the responsibility migration interval. This eliminates the interference of non-migration control positions and changes in energy storage boundary states on reward judgment, enabling accurate extraction of boundary-conserving responsibility quantities generated by the cross-boundary migration of charging and discharging energy, and distinguishing between closable and unclosable responsibility quantities. This improves the accuracy of delayed reward attribution and the reliability of charging and discharging decisions.

[0027] In this embodiment, obtaining the feasible domain of the correction action specifically includes: The unclosed responsibility quantities are processed by responsibility action mapping and directional responsibility determination to obtain the responsibility maintenance direction, responsibility suppression direction, and responsibility correction intensity. The responsibility action mapping process establishes an association between the unclosed responsibility quantity and the corresponding target charging / discharging action in the original sub-trajectory according to the control position, and collects the unclosed responsibility quantities according to the charging direction, discharging direction, and standby state corresponding to the target charging / discharging action. The directional responsibility determination process accumulates the unclosed responsibility quantities corresponding to each action direction. When the accumulation result is positive, the corresponding action direction is determined as the responsibility maintenance direction. When the accumulation result is negative, the corresponding action direction is determined as the responsibility suppression direction. The responsibility correction intensity is determined according to the ratio of the accumulated result of the absolute value of the unclosed responsibility quantity in the corresponding action direction to the sum of the accumulated results of the absolute values ​​of the unclosed responsibility quantities in all action directions. Based on the responsibility retention direction, responsibility suppression direction, and responsibility correction intensity, responsibility boundary correction processing is performed on the feasible domain of basic actions to form a responsibility correction boundary. The responsibility boundary correction processing involves retaining the allowable charging and discharging power range corresponding to the responsibility retention direction, compressing the allowable charging and discharging power range corresponding to the responsibility suppression direction according to the responsibility correction intensity, and the compression process proceeds from the maximum allowable power boundary corresponding to the responsibility suppression direction toward the standby power boundary. During the open state boundary responsibility retention open state, the target charging and discharging action is restricted from switching from the responsibility retention direction to the responsibility suppression direction. The responsibility correction boundary and the basic action feasible domain are merged within the domain to form the corrected action feasible domain. The domain fusion means retaining charging actions, discharging actions and standby actions that simultaneously satisfy the basic action feasible domain and the responsibility correction boundary, deleting actions that exceed the basic action feasible domain, and keeping the state of charge limit, battery temperature limit, rated charging and discharging power limit and public grid switching power limit corresponding to the basic action feasible domain unchanged.

[0028] In this embodiment, obtaining the final agent reward specifically includes: When another boundary responsibility reaches a closed state, the overlapping responsibility units that have closed first, the overlapping responsibility units formed during the cross-boundary continuation, and the overlapping responsibility units that have reached a closed state of another boundary responsibility are extracted along the cross-boundary weaving trajectory. They are spliced ​​together according to the connection order of the overlapping responsibility units to form a closed sub-trajectory. The other boundary responsibility is the open state boundary responsibility belonging to the unclosed responsibility quantity. The closed sub-trajectory starts from the overlapping responsibility unit that has closed first and ends at the overlapping responsibility unit that has reached a closed state of another boundary responsibility. The closed sub-trajectory includes all overlapping responsibility units associated with the cross-boundary continuation of the unclosed responsibility quantity, and maintains the control position order and boundary responsibility association relationship corresponding to each overlapping responsibility unit. Using the complete trajectory reward corresponding to the closed sub-trajectory as the decomposition target and the closable responsibility quantity as the closed responsibility locking item, the Diaster reward decomposition is re-executed. The splitting position is set sequentially along the control position in the closed sub-trajectory, dividing the closed sub-trajectory into a prefix sub-trajectory and a remaining sub-trajectory. The implicit rewards of the prefix sub-trajectory and the remaining sub-trajectory are allocated respectively, so that the combination result of the two implicit rewards is consistent with the complete trajectory reward. The implicit reward increment of the control position corresponding to the closable responsibility quantity remains unchanged. The remaining complete trajectory reward is allocated to the control position corresponding to the cross-boundary continuation period of the unclosed responsibility quantity. The closed proxy responsibility quantity is obtained according to the expected implicit reward difference of adjacent prefix sub-trajectories. The closed responsibility locking item is used to prevent the already completed closable responsibility quantity from being changed when the Diaster reward decomposition is re-executed. The closed proxy responsibility quantity is the reward increment re-determined based on the complete trajectory reward of the closed sub-trajectory after another boundary responsibility reaches the closed state. The closed proxy responsibility quantity is compared with the unclosed responsibility quantity at the corresponding control position. The responsibility correction result is determined based on the difference in their values ​​and their positive and negative directions. The feasible domain of the corrected action is then updated based on the responsibility correction result. When the positive and negative directions of the closed proxy responsibility quantity and the unclosed responsibility quantity are the same, the responsibility holding direction and the responsibility suppression direction are retained, and the compression degree of the corresponding charge and discharge power boundary is adjusted according to the difference in their absolute values. When the positive and negative directions of the closed proxy responsibility quantity and the unclosed responsibility quantity are different, the restriction formed by the unclosed responsibility quantity on the corresponding charge and discharge direction is removed, the corresponding charge and discharge power boundary is restored to the range defined by the basic feasible domain of the action, and the responsibility holding direction and the responsibility suppression direction are re-determined according to the positive and negative directions of the closed proxy responsibility quantity to form the updated feasible domain of the corrected action. According to the same control position, the closable responsibility quantity and the closing proxy responsibility quantity are merged to obtain the final proxy reward. The final proxy reward is then associated with the updated corrected action feasible domain. The final proxy reward includes the closable responsibility quantity that has been assigned when one boundary responsibility is closed first, and the closing proxy responsibility quantity determined by Diaster reward decomposition after another boundary responsibility reaches the closed state. The final proxy reward maintains the corresponding control position and positive / negative direction unchanged. The updated corrected action feasible domain is used to limit the candidate charging and discharging actions allowed to be generated at the next control moment.

[0029] This invention locks the already assigned closable responsibility quantity after another boundary responsibility is closed, and performs Diaster reward decomposition using the closed sub-trajectory to correct the unclosed responsibility quantity across the boundary, avoiding the repeated allocation of closed rewards and improving the accuracy of delayed responsibility assignment. At the same time, it adjusts the charging and discharging power boundary according to the responsibility correction result, so that the range of the next control action conforms to the demand control and battery cycle state, thereby improving the stability and rationality of the charging and discharging control of the energy storage system.

[0030] In this embodiment, the generation of the target charging / discharging action at the next control moment specifically includes: The final agent reward is summarized according to the control position. Based on the positive or negative direction of the final agent reward and the charging or discharging direction of the corresponding target charging or discharging action, the action selection tendency at the next control moment is determined. The action selection tendency is used to characterize the selection priority relationship between the charging direction, the discharging direction and the standby state at the next control moment. A positive final agent reward increases the selection priority of the corresponding charging or discharging direction, and a negative final agent reward decreases the selection priority of the corresponding charging or discharging direction. The energy storage control state at the next control moment is obtained and input into the reinforcement learning model to generate the next candidate charging and discharging action. The target action direction is selected from the updated corrected action feasible domain according to the action selection tendency. When the charging and discharging direction of the next candidate charging and discharging action is consistent with the target action direction, it is retained. When they are inconsistent, it is adjusted to the target action direction to form a direction-corrected candidate charging and discharging action. The direction correction candidate charge / discharge action is matched with the updated corrected action feasible domain. The action domain matching includes: when the direction correction candidate charge / discharge action is within the updated corrected action feasible domain, the corresponding charge / discharge power is retained; when the direction correction candidate charge / discharge action exceeds the updated corrected action feasible domain, the corresponding charge / discharge power is adjusted to the power boundary defined by the updated corrected action feasible domain; when the updated corrected action feasible domain does not retain the corresponding charge / discharge direction, the direction correction candidate charge / discharge action is adjusted to a standby action to form the target charge / discharge action at the next control moment.

[0031] Example 1: To verify the feasibility of the present invention in practice, the present invention was applied to a commercial and industrial energy storage system with a rated capacity of 1000kWh and a rated charge / discharge power of 500kW. The energy storage system includes a lithium iron phosphate energy storage battery, an energy storage converter, and an energy storage control unit. The allowable state of charge range is set to 20% to 90%, the charging efficiency is set to 95%, the discharging efficiency is set to 94%, the control interval is set to 5 minutes, and the duration of the demand statistics window is set to 15 minutes. The power of the electrical load connected to the energy storage system fluctuates between 280kW and 780kW, and it simultaneously undertakes the tasks of demand peak control, time-of-use pricing adjustment, and battery cycle protection.

[0032] During the test, some batteries were still in the continuous discharge phase when the demand statistics window ended, and some batteries' demand statistics windows were not yet closed when the cycle closed. Conventional reinforcement learning control methods distribute the complete trajectory reward evenly to each control position, making it difficult to distinguish between the demand boundary responsibility that has been closed and the cycle boundary responsibility that still needs to be continued across the boundary. This can easily lead to premature switching of charging and discharging directions and power allocation deviations.

[0033] The energy storage control unit collects data on state of charge, battery temperature, current charging / discharging power, load power, power exchanged with the public grid, time-of-use pricing, current peak demand, and remaining duration of the demand statistics window to form the energy storage control state. Candidate charging / discharging actions are then obtained through a reinforcement learning model. These candidate actions are constrained by the basic action feasible region to form the target charging / discharging action. The energy storage control state, target charging / discharging action, and action execution results corresponding to each control moment are arranged in the control sequence to form a continuous charging / discharging action flow.

[0034] The energy storage control unit divides the continuous charging and discharging action flow according to the demand statistics window boundary and the battery cycle boundary, forming overlapping responsibility units. When the demand boundary responsibility is closed first while the cycle boundary responsibility remains open, the cycle boundary responsibility is extended to the next overlapping responsibility unit, and the overlapping responsibility units that bear the same cycle boundary responsibility are connected to form a cross-boundary weaving trajectory.

[0035] An initial sub-trajectory is formed by selecting an overlapping responsibility unit with a pre-closed demand boundary responsibility and its connected next overlapping responsibility unit. The discharge powers of the two overlapping responsibility units are 280kW and 120kW, respectively. A portion of the discharge energy from the previous overlapping responsibility unit is transferred to the next overlapping responsibility unit, adjusting the discharge power to 226kW and 174kW, respectively, while keeping the charging and discharging powers at other control positions unchanged. The difference between the adjusted trajectory's final state of charge and the original sub-trajectory's final state of charge is 0.08 percentage points, with a net energy change difference of 0.12kWh, satisfying the boundary conservation and energy balance conditions. Therefore, a boundary-conserved shadow sub-trajectory is formed.

[0036] The original sub-trajectory and the boundary-conserving shadow sub-trajectory are paired to form a coupled trajectory pair. The Diaster algorithm is restricted to assigning the same implicit reward increment to non-migrating control positions, allowing reward differences only within the responsibility migration interval. The expected trajectory reward of the original sub-trajectory is 0.731, and the expected trajectory reward of the boundary-conserving shadow sub-trajectory is 0.684, resulting in a boundary-conserving responsibility amount of 0.047. The initial agent reward corresponding to the original sub-trajectory is 0.103, and after deducting the boundary-conserving responsibility amount, the unclosed responsibility amount is 0.056.

[0037] Based on the unclosed responsibility quantity, the discharge direction is determined as the responsibility maintenance direction, and the charging direction is determined as the responsibility suppression direction. The upper limit of the allowable charging power is compressed from 220kW to 82kW. After the cyclic boundary responsibility is closed, the corresponding overlapping responsibility units are connected to form a closed sub-trajectory. The Diaster reward decomposition is re-executed with the closable responsibility quantity as the locking term, resulting in a closed surrogate responsibility quantity of 0.062 and a final surrogate reward of 0.109. At the next control moment, the reinforcement learning model generates candidate charging and discharging actions of 136kW. After processing by action selection bias and the updated corrected action feasible region, it is adjusted to discharging 168kW, reducing the power exchanged by the public grid from 716kW to 548kW.

[0038] Using the same load data, initial state of charge (SBC), and energy storage device parameters, a comparison was made between the fixed-rule control method, the single-trajectory Diaster reinforcement learning method, and the present invention. The fixed-rule control method directly determines the charging, discharging, and standby actions of the energy storage system based on preset SBC thresholds, load power thresholds, and time-of-use pricing rules. The single-trajectory Diaster reinforcement learning method uses the Diaster algorithm to decompose the delayed reward of a single continuous charge / discharge trajectory and uses the resulting agent reward for reinforcement learning to make charge / discharge decisions. The test data includes 960 demand statistics windows and 114 battery cycles; the results are shown in Table 1. Table 1. Comparison of the overall operational performance of different energy storage system charge and discharge control methods

[0039] As shown in the table, under the same load data, initial state of charge, and energy storage device parameters, the number of windows exceeding the target demand value in this invention is 19, which is a reduction of 73.97% and 60.42% respectively compared to 73 in the fixed rule control method and 48 in the single-trajectory Diaster reinforcement learning method; the mean absolute error of delayed reward assignment is reduced from 0.134 and 0.081 to 0.036, a reduction of 73.13% and 55.56% respectively; the number of cross-boundary responsibility error assignments is reduced from 126 and 74 to 21; the average number of charge / discharge direction switching per battery cycle is reduced from 1.93 and 1.44 to 0.82; the number of candidate actions exceeding the allowable range is reduced from 186 and 137 to 52; and the relative value of the overall operating cost is reduced to 91.9. This indicates that this invention can improve the accuracy of cross-boundary delayed reward assignment, reduce invalid charge / discharge direction switching and action overruns, and improve the demand control effect and overall operating economy of the energy storage system.

[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A charging and discharging control method for an energy storage system based on reinforcement learning, characterized in that, Includes the following steps: The energy storage control state is obtained, the basic action feasible domain is constructed, the target charging and discharging action is obtained and executed through a reinforcement learning model, forming a continuous charging and discharging action flow; The continuous charging and discharging action flow is divided according to the demand statistics window boundary and the battery cycle boundary, forming overlapping responsibility units. When one boundary responsibility is closed first, the other boundary responsibility is continued across the boundary, forming a cross-boundary weaving trajectory. The original sub-trajectory is obtained by dividing the cross-boundary weaving trajectory with overlapping responsibility units. Under the condition that the energy storage boundary state and energy balance remain unchanged, the charging and discharging power distribution is adjusted to form the boundary-conserving shadow sub-trajectory. The Diaster algorithm is used to decompose the original sub-trajectory and the boundary-conserving shadow sub-trajectory. The initial agent reward is obtained based on the difference in expected reward between adjacent sub-trajectories. The boundary-conserving liability is obtained based on the difference in implicit reward between the two types of sub-trajectories. The closable liability and the unclosable liability are then determined. The feasible region of the basic action is modified based on the unclosed liability quantity to obtain the modified action feasible region; When another boundary responsibility is closed, the overlapping responsibility units are connected to form a closed sub-trajectory. The Diaster algorithm is re-adopted, and the action feasible region is updated and corrected in combination with the closable responsibility quantity to obtain the final agent reward. Based on the final agent reward and the updated feasible domain of the corrected action, the target charging and discharging action for the next control moment is generated.

2. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The acquisition of the basic feasible region of action and the continuous charge / discharge action flow specifically includes: Collect the operating data and external dispatch data of the energy storage system at the current control time, and correlate them according to the current control time to form the energy storage control status; The basic action feasible domain is constructed based on the energy storage control state. The energy storage control state is input into the reinforcement learning model to obtain candidate charging and discharging actions. The candidate charging and discharging actions are then subjected to action constraint processing based on the basic action feasible domain to obtain the target charging and discharging action. The target charging and discharging action is converted into the corresponding charging and discharging power control command of the energy storage converter. The energy storage system is controlled to execute the charging and discharging power control command, and the corresponding action execution result is recorded. The energy storage control state, target charging and discharging action and action execution result formed at the continuous control time are arranged according to the control time to form a continuous charging and discharging action stream.

3. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The specific methods for obtaining the cross-boundary weaving trajectory and overlapping responsibility units include: Read the control time and action execution result corresponding to each target charging and discharging action in the continuous charging and discharging action stream, determine the demand statistics window boundary according to the demand statistics cycle, determine the battery cycle boundary according to the direction of state of charge change and the range of state of charge change, and arrange the demand statistics window boundary and the battery cycle boundary according to the control time to form a responsibility boundary sequence; Based on the responsibility boundary sequence, the continuous charging and discharging action flow is processed to construct responsibility units, forming overlapping responsibility units. Double boundary responsibility mapping is performed on the overlapping responsibility units to obtain demand boundary responsibility and cyclic boundary responsibility, and the responsibility status of demand boundary responsibility and cyclic boundary responsibility is determined. The responsibility status includes open state and closed state. Determine the closing order of demand boundary responsibility and cyclic boundary responsibility. When one boundary responsibility is closed first, the other boundary responsibility that is in an open state continues across the boundary and continues to associate with the next overlapping responsibility unit. Select overlapping responsibility units that bear the same open boundary responsibility. Determine the connection relationship between adjacent overlapping responsibility units according to the continuity of charge state. Connect overlapping responsibility units in sequence according to the execution order of target charging and discharging actions. Terminate the connection when the other boundary responsibility is closed to form a cross-boundary weaving trajectory.

4. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The acquisition of the original sub-trajectory and the boundary-conserving shadow sub-trajectory specifically includes: In the cross-boundary weaving trajectory, an overlapping responsibility unit is identified where one boundary responsibility closes first, and the next overlapping responsibility unit connected to the overlapping responsibility unit is identified along the cross-boundary continuation direction of another boundary responsibility. The two overlapping responsibility units are combined according to the connection order in the cross-boundary weaving trajectory to form the original sub-trajectory. The starting energy storage control state and the ending energy storage control state of the original sub-trajectory are extracted to form the energy storage boundary state. Within an overlapping responsibility unit where a boundary responsibility is closed first, the energy transfer-out control position is determined, and within the next overlapping responsibility unit, the energy transfer-in control position is determined. Part of the charging and discharging energy carried by the energy transfer-out control position is transferred to the energy transfer-in control position, and the charging and discharging power distribution of the original sub-trajectory is reconstructed to form the shadow charging and discharging power distribution. The shadow trajectory is generated based on the shadow charging and discharging power distribution. The shadow trajectory and the original sub-trajectory are checked for boundary conservation and energy balance. If either check result does not meet the corresponding conservation condition, the charging and discharging energy transfer amount is corrected, and the power distribution is reconstructed and the shadow trajectory is generated again. If both check results meet the corresponding conservation conditions, the shadow trajectory is determined as the boundary-conserved shadow sub-trajectory.

5. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The specific methods for obtaining the closed and unclosed liability quantities include: The original sub-trajectory and the corresponding boundary-conserving shadow sub-trajectory are combined to form a coupled trajectory pair. The overlapping responsibility unit and the next overlapping responsibility unit that closes the boundary responsibility first are determined. The responsibility splitting position is set between the two overlapping responsibility units. The responsibility migration interval is limited according to the energy transfer-out control position and the energy transfer-in control position. Establish a synchronous position mapping according to the control position order in the coupled trajectory pair, mark the control position outside the responsibility migration interval as non-migration control position, set implicit reward increment equal value constraint for the corresponding non-migration control positions in the two types of sub-trajectories, set implicit reward difference allocation mark for the control position within the responsibility migration interval, and form a set of coupled reward constraint. The original trajectory reward is determined based on the action execution result of the original sub-trajectory, and the shadow trajectory reward is determined based on the shadow action execution result of the boundary-conserving shadow sub-trajectory. The coupled trajectory pairs, synchronization position mapping, coupled reward constraint set, original trajectory reward, and shadow trajectory reward are jointly input into the Diaster algorithm. The original prefix sub-trajectory and shadow prefix sub-trajectory are constructed step by step along the synchronization position. The expected implicit reward corresponding to each original prefix sub-trajectory and each shadow prefix sub-trajectory is jointly estimated. During the implicit reward allocation process, the implicit reward increment corresponding to the non-migration control position is kept the same. Only the implicit reward increment within the responsibility migration interval is adjusted so that the cumulative result of the implicit reward increment of each control position in the original sub-trajectory is consistent with the original trajectory reward, and the cumulative result of the implicit reward increment of each control position in the boundary-conserving shadow sub-trajectory is consistent with the shadow trajectory reward. According to the control position order, the expected implicit reward of the current original prefix sub-trajectory is subtracted from the expected implicit reward of the previous original prefix sub-trajectory to obtain the initial agent reward corresponding to the current control position. The expected implicit reward increments of the original prefix sub-trajectory and the shadow prefix sub-trajectory within the responsibility migration interval are compared according to the synchronization position, and the difference between the two expected implicit reward increments is determined as the boundary conservation responsibility quantity corresponding to the current control position. Based on the responsibility status corresponding to the overlapping responsibility unit where a boundary responsibility is closed first, the initial agency reward is decomposed into responsibility, the boundary conservation responsibility quantity is determined as the closable responsibility quantity, and the remaining quantity after deducting the closable responsibility quantity from the initial agency reward is determined as the unclosable responsibility quantity.

6. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The specific steps to obtain the feasible region of the correction action include: For unclosed liability quantities, perform liability action mapping processing and directional liability determination processing to obtain the liability maintenance direction, liability inhibition direction, and liability correction intensity; Based on the direction of responsibility maintenance, the direction of responsibility inhibition, and the intensity of responsibility correction, the feasible domain of basic actions is subjected to responsibility boundary correction processing to form a responsibility correction boundary; The boundary of responsibility correction is merged with the feasible domain of basic actions to form the feasible domain of corrective actions.

7. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The specific details of obtaining the final agent reward include: When another boundary responsibility reaches a closed state, the overlapping responsibility units that have closed one boundary responsibility first, the overlapping responsibility units formed during the cross-boundary continuation period, and the overlapping responsibility units that have reached a closed state of another boundary responsibility are extracted along the cross-boundary weaving trajectory. They are spliced ​​together according to the connection order of the overlapping responsibility units to form a closed sub-trajectory. Using the complete trajectory reward corresponding to the closed sub-trajectory as the decomposition target and the closable responsibility quantity as the closed responsibility locking item, the Diaster reward decomposition is re-executed. The splitting positions are set sequentially along the control positions in the closed sub-trajectory, dividing the closed sub-trajectory into prefix sub-trajectories and remaining sub-trajectories. The implicit rewards of the prefix sub-trajectories and remaining sub-trajectories are allocated respectively. The remaining complete trajectory reward is allocated to the control position corresponding to the cross-boundary continuation period of the unclosed responsibility quantity. Based on the difference in expected implicit rewards between adjacent prefix sub-trajectories, the closed proxy responsibility quantity is obtained. The closed agent responsibility quantity is compared with the open responsibility quantity at the corresponding control position. The responsibility correction result is determined based on the numerical difference and positive / negative direction of the two. The feasible domain of the corrected action is then updated based on the responsibility correction result to form the updated feasible domain of the corrected action. According to the same control position, the closable responsibility quantity and the closing agent responsibility quantity are merged to obtain the final agent reward, and the final agent reward is then associated with the updated feasible domain of the corrective action.

8. The method for charging and discharging control of an energy storage system based on reinforcement learning according to claim 1, characterized in that, The generation of the target charging / discharging action at the next control moment specifically includes: The final agent reward is summarized according to the control position. Based on the positive or negative direction of the final agent reward and the charging or discharging direction of the corresponding target charging or discharging action, the action selection tendency at the next control moment is determined. The energy storage control state at the next control moment is obtained and input into the reinforcement learning model to generate the next candidate charging and discharging action. The target action direction is selected from the updated corrected action feasible domain according to the action selection tendency. When the charging and discharging direction of the next candidate charging and discharging action is consistent with the target action direction, it is retained. When they are inconsistent, it is adjusted to the target action direction to form a direction-corrected candidate charging and discharging action. The candidate charge / discharge actions for direction correction are matched with the updated feasible domain of the corrected actions to form the target charge / discharge action for the next control moment.

Images