A Multi-Charging Station Regulation Method Based on Master-Slave Game Theory and Multi-Agent Reinforcement Learning

By employing master-slave game theory and multi-agent reinforcement learning, the problems of strategy competition and high computational load in the regulation of multiple charging stations are solved, achieving effective execution of station power and preservation of individual differences, thus adapting to the needs of power grid regulation.

CN123121858APending Publication Date: 2026-11-17HEFEI UNIV OF TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611622672.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing multi-charging station control methods are unable to reflect the differences between stations, leading to policy competition. Traditional methods have high computational requirements, reinforcement learning actions exceed the equipment's scope, and policy networks have difficulty reusing commonalities and preserving individual differences.

Method used

Employing a master-slave game theory and multi-agent reinforcement learning approach, this method generates planned power and response quantities for charging stations by combining centralized training with decentralized execution through a non-cooperative game between the load aggregator leader and the charging station followers. Multi-agent reinforcement learning is then used for online decision-making and equipment control.

Benefits of technology

It improves the feasibility of target power at power stations, reduces online computation latency, realizes physical constraints and individual differences in policy actions, and ensures the effective execution of charging and discharging control commands for electric vehicles and energy storage devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123121858A_ABST
    Figure CN123121858A_ABST
Patent Text Reader

Abstract

This invention discloses a multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning, comprising: collecting power grid and station data; calculating the power-energy boundaries of equipment, stations, and aggregation layers; generating planned power and flexible capacity for station adjustments; constructing a master-slave Markov game with load aggregators as leaders and charging stations as non-cooperative followers; configuring agent observations, actions, and rewards; the leader outputs the target adjustment amount and screening threshold; stations generate response amounts and feature values; and the adjustment amounts are allocated after screening and sorting; the target power of the stations is sent to the equipment for execution, the state is updated, and state transition samples are generated; network parameters are updated offline, and parameters are loaded online to achieve rolling aggregation control. This invention can take into account both the common state representation of multiple stations and individual difference decision-making, can utilize multi-agent reinforcement learning to complete online decision-making, and can convert policy action constraints into actual charging and discharging control commands.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of vehicle charging and discharging control and scheduling technology, specifically to a multi-charging station control method, device, and storage medium based on master-slave game theory and multi-agent reinforcement learning. Background Technology

[0002] With the large-scale access of electric vehicles, multiple geographically dispersed charging stations with different operating statuses are gradually equipped to provide adjustable loads and bidirectional charging and discharging capabilities to the power grid. Load aggregators can aggregate the adjustable capabilities of multiple charging stations, determine the aggregation layer power adjustment target based on the power grid aggregation control instructions, aggregation reference power, and the real-time feasible power range of each charging station, and then distribute the power adjustment target to multiple charging stations for execution.

[0003] Existing multi-site aggregation and control methods typically treat each charging station as a homogeneous resource uniformly controlled by the aggregator. This makes it difficult to reflect the differences between different stations in terms of equipment access scale, remaining energy, available flexible capacity, and the impact of execution regulation on subsequent operation. When multiple stations determine their response strategies based on their local operation status, the strategies of each station will affect each other through a common target regulation allocation process, resulting in an objective non-cooperative competitive relationship between stations.

[0004] When using traditional two-level optimization or iterative game theory methods, it is necessary to repeatedly solve the leader problem and multiple follower problems in each control period. When faced with scenarios such as random access of electric vehicles, changes in expected departure time, changes in power grid control commands, and rolling updates of the adjustable capacity of power stations, nested iterative solutions are difficult to simultaneously meet the requirements of online computation timeliness and multi-agent strategy adaptation.

[0005] Existing reinforcement learning control methods still suffer from the problem of disconnect between actions and physical constraints of equipment. If the actions of the agent are restricted only according to the rated power or static aggregate capacity, the adjustable capacity of the site may be overestimated because the current energy, remaining access time, and off-site target energy are not taken into account. As a result, the generated site power command cannot be executed by the electric vehicles or stationary energy storage devices that are actually connected.

[0006] Furthermore, if each charging station is configured with a completely independent strategy feature extraction network, each strategy network needs to repeatedly learn local state mappings with the same physical control semantics, such as time periods, flexible capacity, energy margin, and aggregated regulation signals. If each charging station shares a complete strategy network, it is difficult to preserve the decision differences between different stations in terms of resource capacity, energy constraints, response capabilities, and regulation impact. Therefore, it is also necessary to maintain the differences in strategy outputs of each charging station while reusing the common state feature extraction parameters of multiple stations.

[0007] To address these issues, this application proposes a multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning. This method can characterize the master-slave decision-making relationship between aggregators and competing multi-charging stations, take into account both the common state representation of multiple stations and individual difference decision-making, utilize multi-agent reinforcement learning to complete online decision-making, and convert policy action constraints into actual charging and discharging control commands, thereby solving the aforementioned technical problems. Summary of the Invention

[0008] The main objective of this invention is to provide a multi-charging station regulation method based on master-slave game theory and multi-agent reinforcement learning, in order to solve the technical problems mentioned in the background art, such as strategy competition among multiple independently operated charging stations, large computational load of traditional nested game theory online solution, reinforcement learning actions that may exceed the real-time executable range of the equipment, and difficulty in reusing common state mapping and preserving individual decision differences in multi-station policy networks.

[0009] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: A method for regulating multi-charging stations based on master-slave game theory and multi-agent reinforcement learning includes: Step S1: Collect grid aggregation control commands and equipment operation data, calculate the feasible charging and discharging power boundary of the equipment, aggregate to obtain the power and energy boundaries of the power station and aggregation layer, and generate the planned power and flexible capacity for upward and downward adjustment of the power station; Step S2: Construct a master-slave non-cooperative Markov game with the load aggregator as the leader and each charging station as a non-cooperative follower, and set the observations, actions and technical performance rewards for each agent. Step S3: The leader generates the target adjustment amount and the response screening threshold for each direction, and each follower generates the response amount and the comprehensive adjustment characteristic value based on the local observations of their own station; Step S4: Filter the stations according to the response screening threshold of the current regulation direction, sort them according to the comprehensive regulation characteristic value and allocate the target regulation amount to obtain the station allocation regulation amount and unmet regulation amount; Step S5: Generate the target power of the station from the planned power and the allocation adjustment amount, allocate it to the equipment to perform charge and discharge control, read back the power and energy status, calculate the reward and form a state transition sample; Step S6: The policy network and value sub-network are updated offline using a multi-agent reinforcement learning algorithm with centralized training and decentralized execution. Each follower concatenates the features of the original local observations and the output of the shared state encoder, and generates actions through individual decoders and independent Beta action output layers. After training, the policy network parameters and normalized scale parameters are loaded and fixed, and online rolling aggregation control is implemented.

[0010] Preferably, the specific operation process of step S1 includes: Step S11: Collect grid aggregation control commands, aggregation reference power, expected departure time of electric vehicles at each charging station, end time of the current scheduling cycle of fixed energy storage equipment at each charging station and terminal target energy, connection indication of equipment in each charging station, current energy, upper and lower limits of energy, target energy, rated charging and discharging power and charging and discharging efficiency; Step S12: With charging power as positive and discharging power as negative, establish the energy state transition relationship of the equipment based on the equipment's charging power, discharging power, charging efficiency, discharging efficiency, and the length of the control period. The specific formula is as follows: In the formula, This indicates the charging station index. Indicates the first Equipment index within each charging station. Indicates the current control period number; Indicates the first The first charging station Each device during the time period The charging and discharging power is determined according to the convention that charging power is positive and discharging power is negative; and These represent the energy of the device at the start of the current control period and at the start of the next control period, respectively. and These represent the charging efficiency and discharging efficiency of the device, respectively. Indicates the length of the control period; Represents the variable Perform the positive part operation, that is, take the larger value between the variable and zero; Step S13: Calculate the lower limit of dynamic energy, the specific formula is as follows: In the formula, Indicates the number of times the current action is performed. The first charging station The dynamic energy lower limit that each device needs to meet at the start of the next control period; This indicates the lower limit of the static energy allowed by the device; This indicates the target energy that the device needs to achieve at a target time, or the terminal target energy of a fixed energy storage device; This indicates the rated charging power of the device; Indicates charging efficiency; This indicates the expected departure time slot number for electric vehicles, or the end time slot number for the current scheduling cycle of fixed energy storage devices; This indicates the remaining charging time between the execution of the current control period action and the expected departure time; the expected departure time or cycle end time is calculated from the start of the corresponding period, and the remaining time is only calculated for devices that are still connected and have not yet reached that time. Step S14: Calculate the upper limit of the charging power and the lower limit of the signed charging and discharging power of the device during the current control period, respectively, according to the following formulas: In the formula, and They represent the first The first charging station Each device during the time period The upper bound of charging power and the lower bound of signed charging and discharging power; and These represent the rated charging power and rated discharging power of the device, respectively, both of which are non-negative values. This indicates the maximum allowable static energy of the device; This indicates a device connection indicator; the value is 1 when the device is connected and 0 when it is not connected. It represents the difference between the dynamic energy lower limit and the current energy during the next control period; when it is positive, the minimum charging demand is determined according to the charging efficiency, and when it is not greater than zero, the dischargeable boundary is determined according to the discharging efficiency. Step S15: Based on the sum of the lower and upper bounds of the charging and discharging power of the currently connected devices in the same charging station, determine the feasible power range of the same charging station. Then, aggregate the lower limit of the dynamic energy and the upper limit of the static energy of the devices to obtain the lower limit of the station energy and the upper limit of the station energy, and further obtain the energy boundary of the aggregation layer. Step S16. Generate the planned power of the power station within the feasible power range of the power station according to the preset planned power generation rules; Step S17: Use the difference between the upper end of the feasible power range of the power station and the planned power of the power station as the upward adjustment of flexible capacity, and use the difference between the planned power of the power station and the lower end of the range as the downward adjustment of flexible capacity.

[0011] Preferably, step S2 specifically includes: Step S21: Set the load aggregator as the leader agent and each charging station as a follower agent. Execute the leader decision, follower parallel decision, response screening and adjustment allocation, and physical execution in sequence. After being given a leader action, each follower makes a non-cooperative decision with the goal of increasing the expected cumulative reward of its station. Step S22: The leader observes the grid aggregation control command, aggregation reference power, power and energy boundary of the power station, planned power, actual power, flexible capacity, unmet adjustment amount and time information of the previous period, and outputs the target adjustment amount and the screening threshold for upward and downward adjustment responses; each follower observes the power, energy and flexible capacity information of its own power station, the characteristic value of the basic regulation impact, time information and the leader's target adjustment amount, and outputs the ratio of upward and downward adjustment responses and the corresponding strategy adjustment coefficient; Step S23: The leader reward consists of a weighted negative value of aggregated power tracking deviation, unmet regulation amount, and aggregated regulation impact index; the follower reward consists of a weighted positive value of the station response contribution index and a weighted negative value of the station power tracking deviation, equipment target energy deviation, and station regulation impact index; the basic regulation impact characteristic value is determined based on at least one normalized physical quantity among station energy margin, target energy urgency, available flexible capacity, and executable power boundary; the aggregated power tracking deviation is compared with the grid reference power corresponding to the external grid command, and when the command is an increment relative to the aggregated reference power, the two are added together to obtain the grid reference power.

[0012] Preferably, step S3 specifically includes: Step S31: The leader agent generates target adjustment normalization action, response screening threshold normalization action and response screening threshold normalization action based on its observations; determines the mapping boundary of the target adjustment amount based on the sum of the planned power of each station and the aggregated feasible power range, and maps the two threshold normalization actions to the response screening threshold and response screening threshold respectively. Step S32: Each follower agent generates, in parallel, an upward adjustment ratio, an upward adjustment policy adjustment coefficient, a downward adjustment ratio, and a downward adjustment policy adjustment coefficient based on its local observations; the current target adjustment direction is determined based on the sign of the target adjustment amount, and the response amount ratio and policy adjustment coefficient corresponding to the current target adjustment direction are selected; Step S33: Multiply the selected response ratio by the flexible capacity of the corresponding charging station in the current target adjustment direction to obtain the response amount; calculate the comprehensive adjustment characteristic value based on the basic adjustment influence characteristic value of the corresponding station, the selected strategy adjustment coefficient, and the preset characteristic adjustment amplitude upper limit.

[0013] Preferably, step S4 specifically includes: Step S41: Determine the current target adjustment direction based on the sign of the target adjustment amount, and select the corresponding response screening threshold; remove stations whose comprehensive adjustment feature value is greater than the response screening threshold, form a candidate response set for the remaining stations, and arrange them in ascending order according to the comprehensive adjustment feature value. If the feature values ​​are the same, they are sorted according to the preset station identifier. Step S42: Set the absolute value of the target adjustment amount as the initial remaining unallocated adjustment amount, determine the station allocation adjustment amount of each charging station in turn according to the sorting result of the candidate response set, and deduct the remaining unallocated adjustment amount after each allocation; when the sum of all response amounts in the candidate response set is less than the absolute value of the target adjustment amount, record the difference between the two as the unmet adjustment amount. Based on the ranking of the candidate response set, the station allocation adjustment amount for each charging station is determined according to the following formula: In the formula, This represents the aggregation layer target adjustment amount generated by the leader agent relative to the aggregation plan power; The sign representing the target adjustment amount of the aggregation layer; when When >0, the direction d takes the "+" sign. When <0, direction d takes the "-" sign; n represents the current station's sorting position in the candidate response set. Indicates the sorting position preceding the current station; This represents the response quantity of the station at the sorting position n in the selected adjustment direction d; This represents the station allocation adjustment amount corresponding to the station at the sorting position n; This indicates positive part operation; when n=1, the summation of empty spaces in the formula is treated as zero; Step S43: Add the station allocation adjustment amount of each charging station to the corresponding station planned power to obtain the station target power; when the target adjustment amount is zero, do not perform response screening and adjustment amount allocation, set the station allocation adjustment amount of each charging station to zero, and make the station target power equal to the corresponding station planned power.

[0014] Preferably, step S5 specifically includes: Step S51: Determine the power allocation coefficient based on the relative position of the target power of the power station within the feasible power range of the power station. Interpolate the corresponding aggregated target power within the aggregated feasible power range of electric vehicles and stationary energy storage respectively. Then allocate the power of each device within the feasible charge and discharge power boundary of each device. When the total executable power of the devices is sufficient to reach the target power of the power station, make the sum of the device power consistent with the target power. When the width of the feasible power range of the power station is zero, execute directly according to the unique feasible power of each device. When the width of the aggregated feasible power range of electric vehicles or stationary energy storage is zero, allocate power according to the unique endpoint of the corresponding range. Step S52: Send the charging and discharging power command of the corresponding equipment to the charging pile controller and the energy storage converter configured in the station, read the actual power and energy status of the equipment after execution, summarize the actual power at the grid connection point of the station, and use the updated operating data for the calculation of the feasible power boundary in the next control period. Step S53: Calculate the technical performance bonus based on the deviation between the aggregated actual power and the grid reference power, the unmet regulation amount, the deviation between the actual power of the power station and the target power, the equipment energy deviation, and the power station regulation impact amount; Step S54: Summarize the observations of each agent, the corresponding decision-making stage state, action, log probability of the old policy action, reward, next observation, next state, and termination flag to form a state transition sample.

[0015] Preferably, step S6 specifically includes: Step S61: After completing action generation, response screening and adjustment allocation and physical execution at each joint decision-making step, record the state transition samples of each agent as the corresponding training trajectory. Step S62: Set up value sub-networks corresponding to the leader and each follower respectively. The leader value sub-network reads the centralized state of the leader's decision-making stage, and the follower value sub-network reads the centralized state of the follower's decision-making stage that includes the leader's actions. Each policy network reads the corresponding local observations, outputs the Beta distribution shape parameters of each normalized action dimension and samples the actions. Calculate the policy probability ratio based on the probability of the updated policy and the old policy for the same action. Step S63: Calculate the temporal difference residual and generalized advantage estimate based on the reward sequence, state value and termination flag of each agent, and update the policy network using the truncated policy probability ratio objective function; use the sum of the generalized advantage estimate and the state value saved during trajectory acquisition as the reward target, and update the value sub-network based on the mean square error between the predicted state value and the reward target. Step S64: Using the same round of sampling trajectory, update the policy network and value sub-network of each agent under the condition that the old policy parameters are fixed. Repeat offline trajectory collection and network update until the preset training termination condition is reached. Save the trained policy network parameters and normalized scale parameters as online execution parameters. The strategy is updated sequentially according to the trajectory cache of each follower agent; the update of the first... When a follower agent is encountered, the shared state encoder parameters are updated based on the policy loss of that follower agent, as well as the parameters related to the first follower agent. The individual decoder parameters and independent Beta action output layer parameters corresponding to each charging station are set up, while keeping the individual decoder parameters and independent Beta action output layer parameters of other charging stations unchanged. After the preset training termination condition is reached, the shared state encoder parameters, the individual decoder parameters corresponding to each charging station, the independent Beta action output layer parameters, and the normalized scale parameters are saved as the online execution parameters of each follower agent. During the online phase, the same set of shared state encoder parameters and individual parameters corresponding to each charging station are loaded, and the loaded parameters are kept fixed.

[0016] Preferably, the specific operation steps of step S62 include: Step S621: Each follower strategy network shares a state encoder and is configured with an individual decoder and an independent Beta action output layer corresponding to each charging station. The shared state encoder receives local observations of each station and outputs local encoded features. All follower agents use the same set of encoder parameters, but the shared state encoder only receives local observations of the corresponding charging station itself each time, and does not transmit local observations or local encoded features between different follower agents. Step S622: Concatenate the original local observations of each follower with the corresponding local encoded features into vectors, and input the vectors into the individual decoder of the station to obtain the individual policy features; Step S623: The independent Beta action output layer of the station generates two shape parameters for each action dimension based on individual policy characteristics through positive value mapping. Normalized actions located in the [0,1] interval are sampled according to the corresponding Beta distribution and used to determine the ratio of up and down adjustment of response and the corresponding policy adjustment coefficient.

[0017] Preferably, step S6 further includes: Step S65: Load and fix the trained strategy network parameters and normalized scale parameters on the load aggregator control platform and the corresponding charging station controller. Do not call the centralized value network module during the online phase. When each online control cycle begins or the operating status update conditions are met, re-execute steps S11 to S17 before the next action is generated. Then, generate actions, filter responses, allocate adjustment amounts, and issue equipment control commands in sequence. Use the readback operating data for the next control cycle.

[0018] Preferably, the operating status update condition in step S65 is that at least one of the following changes: device connection status, energy status, expected departure time, target energy, current scheduling cycle end time of fixed energy storage device, and terminal target energy; the load aggregator control platform or the corresponding charging station controller detects the change, and triggers the re-execution of steps S11 to S17 after the change is detected and before the next strategy action is generated.

[0019] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0020] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0021] As can be seen from the above technical solution, this invention provides a multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning. Compared with the prior art, this invention has the following advantages: 1. This invention calculates feasible power boundaries hourly based on device access status, current energy, and electric vehicle departure targets or fixed energy storage terminal targets, and generates planned power and response quantities for the site within these boundaries. This constrains strategy actions to the current physical boundaries, improving the executability of the site target power. It also avoids generating invalid control commands that exceed the hardware and energy constraints of electric vehicles and fixed energy storage devices, reducing the probability of control command execution failures and power tracking mismatches. Under the conditions that the device target energy is achievable and the connection status and operating data are valid, it prioritizes meeting the vehicle charging needs and the energy requirements of the energy storage terminal.

[0022] 2. By setting an interactive sequence in which the load aggregator makes the initial decision and multiple charging stations respond in parallel, this invention unifies the non-cooperative competition between the leader and followers based on local physical operating states into a continuous state transition process. This reduces the computational burden of repeatedly solving nested optimization models in the online phase, ultimately reducing the real-time computational latency of online rolling regulation. It adapts to the response requirements of high-frequency aggregation regulation commands on the grid side and achieves coordinated regulation under non-cooperative game among multiple charging stations without the need for large-scale online iterative optimization.

[0023] 3. This invention forms a closed loop by screening thresholds for directional responses, sorting comprehensive adjustment features, allocating adjustment amounts to the charging station, generating target power for the charging station, allocating equipment power, and updating energy state. This enables game-theoretic actions and reinforcement learning outputs to be converted into control commands that can be sent to the charging pile controller and optional energy storage converters. This establishes a complete link from the multi-agent reinforcement learning decision output to the execution of the underlying equipment at the charging station, effectively implementing the upper-layer aggregated control intentions to the lower-layer charging and discharging equipment control quantities, and completing a complete time-series closed loop for power regulation and iterative energy state updates.

[0024] 4. This invention employs a centralized training and decentralized execution approach. During the offline training phase, it utilizes the centralized states of different decision-making stages for value evaluation and saves the policy network parameters and normalized scaling parameters of each agent after reaching the preset training termination condition. In the online phase, the online execution parameters are loaded, and the leader agent and each follower agent generate actions decentralizedly based on corresponding local observations. This enables rolling aggregation and control, transferring the complex game optimization solution process to the offline training stage. Online, only forward inference is completed, balancing multi-agent game optimization capabilities with online engineering deployment capabilities. It can continuously follow power grid control commands and changes in the status of station equipment to complete time-period rolling optimization and adjustment.

[0025] 5. This invention enables each follower policy network to share state encoding parameters while retaining an individual decoder and independent Beta action output layer for each charging station, thus reusing common state mappings and preserving individual decision differences. During the online phase, the original local observations, local encoding features, and actions of each station are not transmitted between followers; each follower still generates differentiated responses based on its own station's resource capacity, energy constraints, and local operating state. This data isolation is limited to the followers and does not imply that centralized states are unnecessary during centralized training, nor does it constitute a guarantee of data privacy for all participants.

[0026] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0027] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall process of the multi-charging station aggregation and control method of the present invention. Figure 2 This is a schematic diagram of the process architecture of the master-slave non-cooperative decision-making and multi-agent reinforcement learning of the present invention. Figure 3 This is a schematic diagram of the planned power, flexible capacity and online control closed-loop process of the power station according to the present invention. Figure 4 This is a schematic diagram of the state transition samples and multi-agent reinforcement learning training process of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] For details in the embodiments, please refer to Figures 1 to 4 .

[0030] like Figure 1 As shown in the figure, the multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning proposed in this embodiment of the invention includes the following steps: Step S1: Calculate the feasible power boundary for equipment and site Collect grid aggregation and control commands and electric vehicle operation data connected to each charging station; for charging stations equipped with fixed energy storage devices, also collect the operation data of the fixed energy storage devices; calculate the feasible charging and discharging power boundaries of each device based on the operation data and aggregate them to obtain the feasible power range of each charging station; generate the station's planned power within the station's feasible power range according to the preset planned power generation rules; calculate the upward and downward flexible capacity of each charging station based on the station's planned power and the upper and lower endpoints of the station's feasible power range, and sum them to obtain the aggregated upward and downward flexible capacity.

[0031] Step S11: The control targets are a load aggregator and multiple charging stations. The i-th charging station includes a collection of electric vehicles connected to charging piles and a station controller, and may further include fixed energy storage devices. At the start of each control period, the grid aggregation control command, aggregation reference power, and actual power of each station are collected; each station controller reads the device connection indication, current energy, upper and lower energy limits, rated charging and discharging power, charging and discharging efficiency, and target energy. For electric vehicles, the expected departure time is also read; for fixed energy storage devices, the end time of the current scheduling cycle and the terminal target energy are read. Device power is positive for charging and negative for discharging, and the length of the control period is denoted as . .

[0032] In this embodiment, the grid reference power represents the expected actual power of the aggregation layer given by the external grid aggregation control command: when the command is an absolute power, it is used directly; when the command is a signed increment relative to the aggregation reference power, the increment is added to the aggregation reference power to obtain the grid reference power. A positive increment indicates an increase in power consumption, and a negative increment indicates a decrease in power consumption or an increase in power discharged to the grid. The grid reference power is different from the target adjustment amount selected by the leader according to the strategy; the aggregation tracking deviation is always based on the external grid reference power and is not replaced by the leader's self-selected target.

[0033] Step S12: Establish the energy state transition relationship based on the device's charging power, discharging power, and corresponding efficiency. The first station The energy of each device is updated according to equation (1) between adjacent control periods.

[0034] (1) in, This indicates the charging station index. Indicates the first Equipment index within each charging station. Indicates the current control period number; Indicates the first The first charging station Each device during the time period The charging and discharging power is determined according to the convention that charging power is positive and discharging power is negative; and These represent the energy of the equipment at the start of the current control period and at the start of the next control period, respectively; and These represent the charging efficiency and discharging efficiency of the device, respectively, both of which are preset parameters greater than 0 and not greater than 1; Indicates the length of the control period; Represents the variable Perform positive part operations.

[0035] Step S13: To ensure that the equipment reaches the target energy at the specified time, for electric vehicles, the dynamic energy lower limit shown in Equation (2) is constructed based on the expected departure time, target energy and remaining charging time. For fixed energy storage equipment, the expected departure time in Equation (2) is replaced with the end time of the current scheduling cycle, and the target energy is replaced with the terminal target energy.

[0036] (2) in, This indicates that after executing the charging and discharging action during the current control period, the first... The first charging station The dynamic energy lower limit that each device needs to meet at the start of the next control period; Indicates the lower limit of the static energy allowed by the device; This indicates the target energy that the equipment needs to achieve at the target time; Indicates the device's rated charging power; Indicates charging efficiency; This indicates the estimated departure time slot number for the electric vehicle; This indicates the remaining rechargeable time between the execution of the current control action and the expected departure time. For stationary energy storage devices, this will... Replace it with the segment number at the end of the current scheduling period, and... Replace with the terminal target energy of the stationary energy storage device. The expected departure or end time of the cycle is calculated from the start of the corresponding time period, and the remaining time is only calculated for devices that are still connected and have not yet reached that time; devices that have reached the target time will no longer use this remaining time formula.

[0037] To maintain consistency with the current energy boundary of the site, the current dynamic energy lower limit is denoted as: Equation (1) represents the minimum energy requirement at the start of the current time period; Equation (2) represents the minimum energy requirement at the start of the next time period after the current action is performed.

[0038] Step S14: Adjust the rated charging power according to the margin between the current energy and the upper limit of energy and the device connection status to obtain the upper limit of device charging power shown in Equation (3).

[0039] (3) in, Indicates the first The first charging station The maximum allowable energy for each device; This indicates that the device is in the control period. The current energy at the beginning. Combining the current energy with the insufficient amount or discharge margin of the dynamic energy lower limit of the next period, the power lower limit is calculated by back-calculating according to the charging efficiency or discharging efficiency, and the device signed charging and discharging power lower limit shown in equation (4) is obtained; when the device is not connected, the charging power upper limit and the signed charging and discharging power lower limit of the corresponding device are both set to zero.

[0040] (4) in, and They represent the first The first charging station Each device during the time period The upper and lower bounds of the power; and These represent the device’s rated charging power and rated discharging power, respectively, both of which are non-negative values. This indicates the device connection status. Among them, When the difference is positive, the device must be charged to meet the dynamic energy lower limit for the next time period, with the power lower limit calculated based on charging efficiency; when the difference is not greater than zero, the dischargeable power is calculated based on discharge efficiency. Devices that do not support reverse discharge have their rated discharge power set to zero.

[0041] Before calculating the power boundary, confirm that both the current energy and the target energy are within the equipment's static energy range, the equipment is connected, and the target time has not yet been reached. If the calculated lower power bound is greater than the upper bound, the target energy constraint is no longer achievable under the current access conditions: do not aggregate this empty interval as a feasible interval; fix the equipment's current time-period execution power to the maximum charging power not exceeding the static energy upper limit and rated power, and set both the upper and lower bounds of this equipment used for station power aggregation to this fixed execution value, so that it no longer provides adjustable flexible capacity, and record the predicted target energy deficit. This emergency execution value is only used to ensure physical safety and does not indicate that the original target energy can be met.

[0042] Step S15: Place the first The set of currently connected and controllable devices at a charging station is denoted as . Summing the lower and upper limits of the equipment power respectively, we obtain the feasible power range of the station shown in equations (5) and (6).

[0043] (5) (6) The dynamic energy lower limit and static energy upper limit of currently connected and controllable devices within the same charging station are aggregated to obtain the first... Each charging station during the period The lower bound and upper bound of the aggregated energy of the power station: (7) (8) In the formula, and They represent the first Each charging station during the period The lower bound and upper bound of the aggregated energy of the station.

[0044] Furthermore, by summing the lower and upper bounds of the aggregated energy at each charging station, we obtain the lower and upper bounds of the aggregated energy layer: (9) (10) in, and These represent the load aggregators during the time period. The corresponding lower and upper bounds of the aggregation energy.

[0045] Step S16: Generate the planned power of the power station within the feasible power range. In one offline training implementation, taking the midpoint of the range as the center, multiply one-tenth of the range width by a random number uniformly sampled within [-1,1] and add it to the midpoint. The result is then truncated to the range and used as the planned power for the current time period; this power remains unchanged after generation within the same time period. During online operation, a reference power is provided by the power station's predetermined charging plan and truncated to the current feasible power range; this power remains unchanged when the reference power is already within the range. When the range width is zero, the planned power is taken from its unique endpoint.

[0046] Step S17: with The power station's planned power is represented by the charging power. An increase in the charging power relative to the planned power is considered an upward adjustment, and a decrease is considered a downward adjustment. When the total power of the station equals the planned power, the aggregator does not adjust the station power, and the station adjustment impact is 0. The upward and downward flexible capacity adjustments of the station are calculated according to equations (11) and (12).

[0047] (11) (12) in, Indicates the first Each charging station can increase its flexible capacity for adjusting charging power relative to its planned power. Indicates the first Each charging station has flexible capacity that allows for reducing charging power or increasing discharging power relative to the station's planned power. and All are non-negative values.

[0048] The upward and downward flexible capacity adjustments for each charging station are summed to obtain the aggregated upward and downward flexible capacity. When the aggregator's target adjustment amount is zero, response screening and adjustment allocation are not performed; the station allocation adjustment amount for each station is set to zero, but the calculated physical flexible capacity is still used as an observation for the next leader decision. In one implementation, steps S11 to S17 are executed according to a preset control cycle; in another implementation, execution is triggered when the equipment access status, energy status, or target constraints change, to update the station's feasible power range and flexible capacity for the next control period.

[0049] After completing the boundary and flexible capacity calculations in step S1, the operational constraints and adjustability potential of each charging station have been determined. Using this physical data as input, the next step is to build a master-slave game-theoretic intelligent agent system to achieve aggregated regulation. To realize the leader-follower game interaction between the load aggregator and multiple charging stations, and the multi-agent reinforcement learning modeling, the corresponding system framework for intelligent agent observation, action, and reward settings is as follows: Figure 2 As shown; after the game generates the adjustment decision, the planned power and flexible capacity of the power plants output in step S1 are used to complete the power plant screening and sorting, adjustment allocation, and further parsed down into equipment charging and discharging commands, completing the entire online time-series closed loop of energy state iterative update, as follows. Figure 3 As shown; each round of regulation and interaction generates state transition samples. Based on these samples, iterative optimization of the policy network and value sub-network is completed, achieving a complete learning deployment process of offline training and online parameter loading and inference, as follows. Figure 4 As shown, the following steps are used to explain the above process in detail.

[0050] Step S2: Construct a master-slave non-cooperative Markov game Build by load aggregator and A master-slave non-cooperative Markov game consisting of several charging stations, in which... Using load aggregators and charging stations as different decision-making entities, the observations, actions, and rewards of each agent are defined. The multi-station aggregation and control process is described through a leader-first, follower-response sequential decision-making approach, referencing... Figure 2 .

[0051] Step S21: Set the load aggregator as the leader agent and the multiple charging stations as follower agents. A complete joint decision-making process includes, in sequence, a leader decision-making sub-stage, a parallel decision-making sub-stage for multiple followers, a response selection and regulation allocation sub-stage, and a physical execution sub-stage. The leader agent first generates a target regulation amount and a response selection threshold for each direction based on the aggregation state; then, given the leader's target regulation amount, each follower agent independently generates a response strategy based on the local physical state of its station, and forms a non-cooperative competition by comprehensively ranking the regulation feature values ​​and allocating the regulation amount.

[0052] Step S22: The leader agent's observations include grid aggregation control commands, aggregation upward and downward flexible capacity adjustments, aggregation planned power and actual power, aggregation energy boundary, upward and downward flexible capacity adjustments for each substation, energy boundary for each substation, planned power for each substation, aggregation reference power, unmet regulation amounts from the previous control period, and time information. In one implementation, the observations also include a predicted control demand sequence for several future control periods. The leader's actions include target regulation amount normalization actions, upward response screening threshold normalization actions, and downward response screening threshold normalization actions. The i-th follower agent's local observations include the current substation's upward and downward flexible capacity adjustments, energy boundary, substation planned power and actual power, substation reference power window, current energy margin relative to the dynamic energy lower limit and target energy, current substation's upward and downward basic regulation impact characteristic values, leader target regulation amount, and time information. The follower's actions include the upward response amount ratio, upward strategy adjustment coefficient, downward response amount ratio, and downward strategy adjustment coefficient.

[0053] To reduce the impact of different dimensions on policy training, power, energy, comprehensive regulation feature values, and response screening thresholds are normalized using corresponding physical boundaries or preset scales; intraday times are encoded using sine, cosine, and intra-time position. The basic regulation impact feature values ​​of each site are only written into the local observations of the corresponding follower agent, and not into the policy observations of the leader agent.

[0054] Step S23: The leader agent's reward consists of the aggregated power tracking deviation index, the unmet regulation amount index, and the aggregated regulation impact index; the follower agent's reward consists of the station response contribution index, the station power tracking deviation index, the equipment target energy deviation index, and the station regulation impact amount. Each agent's training objective is to maximize its own discounted expected cumulative reward. All indicators are calculated based on grid control data and equipment operation data, and normalized using a preset scale.

[0055] The leader agent's reward is determined according to the following formula: (13) In the formula, This represents the reward of the leader agent during time period t; This represents the non-negative tracking error obtained by normalizing the absolute difference between the aggregated actual power and the grid reference power using a preset positive power scale. This indicates that the normalization index of the absolute value of the adjustment amount relative to the target adjustment amount is not met; it is zero when the target adjustment amount is zero. Indicates the impact of aggregation adjustment indicators; , and These are non-negative weights. Weights used for aggregate power tracking error are positive.

[0056] The reward for the i-th follower agent is determined according to the following formula: (14) In the formula, This represents the reward of the i-th follower agent during time period t; Indicates the station's response contribution indicators; This represents the normalized tracking deviation of the actual power of the power station relative to the target power of the power station; This indicates the non-negative normalized insufficiency of the energy after the device performs the current action relative to the dynamic energy lower limit of the next time period. When the target time is reached, the assessment is based on the target energy. Indicates the amount of influence of station regulation; , , and The weights are non-negative.

[0057] The impact of station regulation and the aggregated regulation impact index are determined according to the following formula: (15) In the formula, This represents the characteristic value of the basic adjustment influence of the i-th charging station in time period t and direction d; Indicates the amount of adjustment allocated to the station; Indicates the length of the control period; This indicates the preset energy normalization scale; This represents a preset positive number to prevent the denominator from being zero. The characteristic value of the basic regulation effect is determined by at least one normalized physical quantity among the following: station energy margin, target energy urgency, available flexible capacity, and executable power boundary.

[0058] Master-follower non-cooperative game theory represents a sequential game structure where the leader agent acts first, and each follower agent chooses its own strategy after being given the leader's action, without sharing local observations. Multi-agent reinforcement learning is used to update the strategies of each agent on repeated interaction samples, without setting additional policy equilibrium tests or equilibrium passing conditions. For a specific master-follower non-cooperative game architecture, please refer to [reference needed]. Figure 2 .

[0059] The non-cooperative relationship adopted in this method is reflected in the fact that, given the actions of the leader agent, each follower agent aims to maximize its own expected cumulative reward, and incorporates the strategies of other follower agents as part of the environmental state transition. The corresponding non-cooperative response strategy combination is approximated through offline multi-agent reinforcement learning, and no explicit equilibrium iteration solution or equilibrium test is performed in the online stage.

[0060] Step S3: Generate and constrain the actions of each agent. The actions of each agent are generated in the order of the leader agent making decisions first and the follower agents making decisions in parallel. The normalized actions of the leader agent are converted into target adjustment quantities, upward adjustment response screening thresholds, and downward adjustment response screening thresholds. The normalized actions of each follower agent are converted into the ratio of upward and downward adjustment of response quantities and corresponding policy adjustment coefficients. The corresponding action dimension is selected according to the current target adjustment direction to obtain the response quantity and comprehensive adjustment characteristic value that do not exceed the flexible capacity of the corresponding site.

[0061] Step S31: The leader strategy network reads the leader's observations and outputs the target adjustment normalization action and the normalization actions for the upward and downward adjustment response screening thresholds. The planned power of each site is summed to obtain the aggregated planned power, and the aggregated feasible power range is obtained by summing the feasible power ranges of each site. The target adjustment normalization action is mapped to the target adjustment amount with the aggregated planned power as the zero point and the aggregated upward and downward adjustments of flexible capacity as positive and negative boundaries. The two threshold normalization actions are mapped to zero and the upper limit of the preset feature scale, respectively, to obtain the upward adjustment response screening threshold and the downward adjustment response screening threshold.

[0062] Step S32: After the same leader action is generated, each follower strategy network outputs in parallel the proportion of increasing response amount, the adjustment coefficient of increasing strategy, the proportion of decreasing response amount, and the adjustment coefficient of decreasing strategy based on their respective local observations. Each normalized action is located in [0,1]. When the target adjustment amount is positive, the upward adjustment dimension is selected; when the target adjustment amount is negative, the downward adjustment dimension is selected; when the target adjustment amount is zero, response screening and adjustment amount allocation are not performed.

[0063] Step S33: Convert the response ratio and strategy adjustment coefficient of the i-th station into the physical response and comprehensive adjustment characteristic value in the current target adjustment direction d according to equations (16) and (17), where d∈{+, }

[0064] (16) (17) Where d represents the adjustment direction determined by the sign of the target adjustment amount; This represents the response of the i-th charging station in time period t and direction d; This indicates the flexible capacity of the charging station in direction d; Indicates the proportion of the response volume. represents the policy adjustment coefficient, both of which are located in [0,1]. Indicates the influence of baseline regulation on eigenvalues; This indicates the upper limit of the preset feature adjustment range; This represents the comprehensive regulation characteristic value. , and The same dimensionless numerical scale is used. When the direction is "+", the upward flexible capacity is taken; when the direction is "-", the downward flexible capacity is taken, consistent with the aforementioned directional capacity symbols.

[0065] Step S4: Perform response filtering, sorting, and adjustment allocation. Select the corresponding response screening threshold according to the direction of the target adjustment amount, and screen out the stations whose comprehensive adjustment characteristic value exceeds the response screening threshold; form a candidate response set for the remaining stations and sort them from low to high according to the comprehensive adjustment characteristic value, and determine the station allocation adjustment amount for each charging station in turn with the target adjustment amount as the amount to be allocated.

[0066] Step S41: Select the corresponding response screening threshold based on the sign of the target regulation. The load aggregator removes stations whose comprehensive regulation characteristic value is greater than the response screening threshold, and forms a candidate response set for the remaining stations, which are then arranged in ascending order according to their comprehensive regulation characteristic value; if the characteristic values ​​are the same, a fixed sorting order is determined according to the preset station identifier.

[0067] Step S42: Set the absolute value of the target regulation amount as the initial remaining unallocated regulation amount. Accept the station response amounts sequentially according to the sorting result of the candidate response set. The station allocation regulation amount corresponding to the sorting position n is determined according to formula (18). After each allocation, deduct the remaining unallocated regulation amount. If the sum of all response amounts in the candidate response set is less than the absolute value of the target regulation amount, accept all candidate response amounts and record the difference as the unmet regulation amount.

[0068] (18) in, This represents the target adjustment amount generated by the leader agent during the control period t, relative to the sum of the planned power of each charging station; The sign representing the target adjustment amount; when When the value is greater than 0, the target adjustment direction d is "+", indicating an increase in charging power or a decrease in discharging power; when... When <0, the target adjustment direction d is "-", indicating a decrease in charging power or an increase in discharging power; when When =0, no response filtering or adjustment allocation is performed, and the adjustment allocation for each charging station is zero. n represents the current station's ranking position in the candidate response set. Indicates the sorting position preceding the current station; This represents the response quantity of the station at the sorting position n in the selected adjustment direction d; This represents the station allocation adjustment amount corresponding to the station at the sorting position n.

[0069] Step S43: Add the station allocation adjustment amount of the i-th station to the corresponding station planned power to obtain the station target power shown in equation (19); when the target adjustment amount is zero, the station target power is equal to the station planned power.

[0070] (19) in, , and Let represent the target power, planned power, and allocated regulation amount of the i-th power station, respectively. For power stations whose comprehensive regulation characteristic value exceeds the response screening threshold, do not enter the candidate response set, do not receive regulation allocation, or whose target regulation amount is zero, their allocated regulation amount is set to zero, and the target power of the power station is equal to the planned power of the power station.

[0071] Step S5: Execute device control and generate state transition samples The station allocation adjustment amount of each charging station is added to the corresponding station planned power to obtain the station target power. The station target power is allocated to the electric vehicles connected in the station, and if the station is equipped with fixed energy storage equipment, it is also allocated to the fixed energy storage equipment. Charge and discharge control is executed, the equipment energy state and the actual power of the station are updated, the leader agent reward and the reward of each follower agent are calculated, and state transition samples are formed.

[0072] Step S51: For the first For each charging station, a station power allocation coefficient is calculated based on the relative position of the station's target power within its feasible power range. Using this coefficient, the aggregated target power for electric vehicles is interpolated within the aggregated feasible power range, and when configuring fixed energy storage equipment, the aggregated target power for fixed energy storage is interpolated within the aggregated feasible power range. Then, based on the feasible charge / discharge power boundaries of each device, the corresponding aggregated target power is allocated to each device, and the charge / discharge power of each device is limited to the real-time feasible charge / discharge power boundaries calculated in step S14. When the total executable power of the devices within the station is sufficient to reach the station's target power, the sum of the charge / discharge power of each device corresponds to the station's target power. When the width of the station's feasible power range is zero, the relative position ratio is not calculated, and the power is directly allocated according to the unique feasible power of each device; when the width of the aggregated power range for a certain type of device is zero, power is allocated to that type of device according to its unique endpoint.

[0073] Step S52: Send the electric vehicle equipment-level power command to the charging pile controller; if the station is equipped with fixed energy storage equipment, send the fixed energy storage equipment-level power command to the energy storage converter. Read the actual power of the equipment and the updated energy status, and update the equipment energy according to formula (1); summarize the actual power of the equipment at the grid connection point of the station into the actual power of the station, and use it as the input for recalculating the feasible power boundary in the next control period.

[0074] Step S53: Calculate the leader agent reward according to Equation (20) based on the normalized tracking deviation of the aggregated actual power relative to the grid reference power, the unmet regulation index, and the aggregated regulation impact index.

[0075] (20) in, This represents the reward of the leader agent during time period t; This represents the non-negative tracking error obtained by normalizing the absolute difference between the aggregated actual power and the grid reference power using a preset positive power scale. This indicates a normalized index representing the absolute value of the unmet adjustment amount relative to the target adjustment amount. Indicates the impact of aggregation adjustment indicators; , and The weights are non-negative. When the target adjustment is zero, the unmet adjustment index is set to zero; otherwise, it is calculated as the ratio of the absolute value of the unmet adjustment to the target adjustment. The weight used for aggregate power tracking deviation is positive, and the other weights are non-negative.

[0076] The reward for the i-th follower agent is calculated according to equation (21).

[0077] (twenty one) in, This represents the reward of the i-th follower agent during time period t; Indicates the station's response contribution indicators; Indicates the power tracking deviation index of the power station; This indicates the deviation index of the equipment's target energy. Indicates the amount of influence of station regulation; , , and The weights are non-negative. The station response contribution index is the ratio of the absolute value of the station's allocated adjustment amount to the corresponding directional flexible capacity; when the corresponding directional flexible capacity is zero, this index is zero. The station power tracking deviation index is determined by normalizing the absolute difference between the station's actual power and the station's target power. The equipment target energy deviation index is determined by normalizing the non-negative insufficiency of the equipment's actual energy after performing the current action relative to the dynamic energy lower limit of the next time period: for each controlled equipment in this time period, the dynamic energy lower limit is subtracted from the positive part of the energy after execution, then summed and divided by the station's preset positive energy scale; when there is no controlled equipment, it is zero. In the time period when the equipment's target time is reached, the target energy is used as the assessment lower limit, and the insufficiency of the same target is no longer repeatedly added.

[0078] (twenty two) In equation (22), This represents the characteristic value of the basic adjustment influence of the i-th charging station in time period t and direction d; Indicates the amount of adjustment allocated to the station; Indicates the length of the control period; This indicates the preset energy normalization scale; This represents a preset positive number to prevent the denominator from being zero. When the station's allocated adjustment amount is zero, the corresponding station adjustment impact amount is zero.

[0079] Step S54: After completing leader decision-making, follower decision-making, response screening and adjustment allocation, and physical execution, summarize the current local observations, corresponding decision-making stage centralized state, actions, log probability of actions under the corresponding old strategy, rewards, local observations for the next time period, corresponding centralized state for the next time period, and termination flag for the leader agent and multiple follower agents respectively; at the same time, summarize the station's feasible power range, station planned power, station response quantity, comprehensive adjustment characteristic value, station allocated adjustment quantity, station target power, station actual power, power tracking deviation, and equipment energy update quantity to form a state transition sample for the same control period.

[0080] Each joint decision step generates actions and forms corresponding state transition records by all follower agents. For stations whose comprehensive adjustment feature values ​​exceed the response screening threshold, do not enter the candidate response set, or do not receive adjustment allocation, rewards are still calculated based on their actual execution results and written to the corresponding trajectory cache; when the target adjustment amount is zero, the station allocation adjustment amount for each station is zero.

[0081] Step S6: Centralized training and distributed execution of the strategy Employing a multi-agent reinforcement learning approach with centralized training and distributed execution, referencing Figure 4 State transition samples formed during the same control period are written into the corresponding trajectory cache according to the agent identifier; a multi-agent proximal policy optimization algorithm is used to calculate the temporal difference residuals and generalized advantage estimates of the leader agent and each follower agent respectively, update the corresponding policy network using the truncated policy objective, and update the corresponding value sub-network using the reward objective; state transition sample collection and network parameter update are repeatedly performed in the offline training environment until the preset training termination condition is reached; the policy network parameters and normalized scale parameters of each agent when the preset training termination condition is reached are saved, and the policy network parameters and normalized scale parameters are loaded in the online rolling aggregation control stage, so that the leader agent and each follower agent generate actions in a dispersed manner according to the corresponding local observations.

[0082] Step S61: During the training phase, fix the old policy parameters used for the current round of trajectory acquisition, and interact with the training environment according to steps S1 to S5. At each complete joint decision-making step, write the local observations, actions, log probabilities of old policy actions, rewards, corresponding decision-making phase central states, next local observations, next central states, and termination flags of the leader agent and each follower agent into a leader trajectory cache and a... Each follower trajectory is cached by station identifier and the time correspondence between trajectories is maintained by using a common control period identifier.

[0083] Step S62: Employ a multi-agent proximal policy optimization algorithm with centralized training and distributed execution. The centralized value network module includes one leader value sub-network and N follower value sub-networks. The leader value sub-network reads the centralized state before the leader's action is generated and outputs the state value corresponding to the leader's reward. Each follower value sub-network reads the centralized state of the follower's decision-making stage, which includes the leader's action, and outputs the state value corresponding to the corresponding follower's reward. The leader policy network reads the leader's local observations and outputs two Beta distribution shape parameters for each normalized action dimension. Each follower policy network shares a common state encoder and sets up an individual decoder and an independent Beta action output layer corresponding to each charging station. Each follower policy network takes local observations of the site as input and generates normalized actions through a shared state encoder, a corresponding individual decoder, and an independent Beta action output layer.

[0084] In one implementation, the leader policy network and each value sub-network include an input normalization layer, two fully connected hidden layers, and an output layer; the shared state encoder and each individual decoder each include an input normalization layer and two fully connected hidden layers. Each fully connected hidden layer includes 64 hidden units and employs linear transformation, modified linear unit activation, and layer normalization. The leader policy output layer corresponds to the target adjustment amount and the up and down adjustment response screening thresholds; each independent Beta action output layer includes parameter output branches that generate a first shape parameter and a second shape parameter, respectively, corresponding to the output up and down adjustment response ratios and corresponding policy adjustment coefficients; each value sub-network outputs a scalar state value corresponding to the expected cumulative reward of the corresponding agent. The number of layers and hidden units mentioned above are adjustable implementation parameters.

[0085] (twenty three) in, This indicates that the agent index needs to be updated; Represents intelligent agents During the period The probability ratio of the strategies; This indicates the strategy after parameter updates. This indicates the old strategy used in the current trajectory acquisition and kept frozen within the preset update rounds; and Representing intelligent agents respectively During the period Actions and local observations.

[0086] Step S63: Perform policy updates for the leader agent and each follower agent separately: First, calculate the temporal difference residual according to Equation (24) based on the agent's reward, the corresponding value subnetwork output, and the termination flag. Then, recursively calculate the generalized advantage estimate using Equation (25). Finally, use the truncated policy objective shown in Equation (26) to suppress the magnitude of a single policy update by limiting the range of change in the probability ratio between the old and new policies. When the termination flag is 1, stop bootstrapping for the next state value and the advantage at the next moment.

[0087] (twenty four) (25) (26) in, Indicates the use of updating the intelligent agent Policy network parameters The truncation strategy proxies the objective function; Indicates an agent that needs updating. The strategy network parameters; This indicates that the empirical mean is calculated for each time series sample within the current training batch. Indicates the variable Cut off to interval During training, maximize The target for policy updates is its inverse, which is used as the policy loss function in the specific implementation. This indicates the agent index to be updated, where Represents the leader intelligent agent. Indicates the first A follower intelligent agent; Represents intelligent agents During the period The reward received, when hour, ,when hour, ; and Representing intelligent agents respectively The centralized state corresponding to the current decision-making stage and the next decision-making stage; Indicates the corresponding value subnetwork based on the centralized state Output state value; Represents intelligent agents During the period Temporal difference residuals; Represents intelligent agents During the period Generalized advantage estimation; This indicates a termination flag, which is used when the current state transitions to a termination transition. ,otherwise ; Indicates the discount factor; Represents the parameters for generalized dominance estimation; Indicates the truncation parameter; The policy probability ratio is defined by expression (23); Indicates the variable Cut off to interval ; This indicates the agents within the current training batch. Calculate the mean of all time series samples.

[0088] Equations (24) to (26) all take any agent to be updated as the computational object. Equation (26) is the truncated policy objective to be maximized, and its inverse is used as the policy loss when it is achieved; the mean value is calculated for all time-series samples of the agent in the current training batch. The sum of the generalized advantage estimation and the state value output and stored by the corresponding value sub-network during trajectory acquisition is used as the reward objective, and the mean square error between the predicted state value and the reward objective is used as the value loss.

[0089] Furthermore, in step S62, the first The shared state encoding relationship of a follower policy network is determined according to equation (27): (27) In the formula, From 1 to Any integer in the array represents the index of the follower agent and its corresponding charging station. This represents the control period number; the function on the right side of the equation represents the parameters. A shared state encoder whose input is the first... A follower agent during the time period The local observations of the charging station are given; the variables on the left side of the equation represent the local coded features output by the shared state encoder. All follower agents use the same set of parameters φ, but the shared state encoder only receives the local observations of the corresponding charging station itself each time, and does not transmit local observations or local coded features between different follower agents.

[0090] The original local observations and local coded features are concatenated and then input into the first... The individual decoder corresponding to each charging station obtains the individual strategy features according to equation (28): (28) In the formula, the comma in square brackets indicates vector concatenation; the function on the right side of the equation represents the parameters. The i-th charging station individual decoder; the variable on the left side of the equation represents the individual policy feature output by the decoder. The trajectory sample is updated only by the i-th follower agent to preserve the resource capacity, energy constraints, basic regulation impact characteristics and local physical state differences of the corresponding station.

[0091] By the The independent Beta action output layer of each charging station generates Beta distribution parameters for each normalized action dimension based on individual strategy characteristics and samples actions, which are determined according to equation (29): (29) In the formula, m represents the normalized action dimension index; the function on the right side of the equation has the following parameters. The parameter output branch corresponding to the m-th action dimension in the independent Beta action output layer of the i-th charging station; and These represent the first and second shape parameters obtained through positive value mapping, respectively; This represents the normalized action obtained by sampling according to the corresponding Beta distribution, with values ​​in the interval [0,1]. The action dimension of the i-th follower is 4, corresponding to the proportion of increasing the response amount, the proportion of increasing the policy adjustment coefficient, the proportion of decreasing the response amount, and the proportion of decreasing the policy adjustment coefficient, respectively.

[0092] Step S64: After each trajectory cache reaches its preset length or after the training round ends, extract the trajectories sampled in the same round from the leader's trajectory cache and each follower's trajectory cache, respectively, and update each agent sequentially while keeping the old policy parameters frozen. Update the... When there is a follower agent, the policy loss of that follower agent is backpropagated through the corresponding individual decoder, the independent Beta action output layer, and the shared state encoder to update the parameters of the shared state encoder and the first follower agent. The individual decoder parameters and independent Beta action output layer parameters for each charging station remain unchanged in this update. Each follower agent is updated sequentially, ensuring that the trajectory samples of all follower agents act on the same set of shared state encoder parameters. After completing the preset number of rounds of updates, the corresponding policy network and value sub-network parameters are replaced, and trajectory collection and network parameter updates continue in the offline training environment until the preset training termination condition is met. During training, when the number of training steps reaches the preset save interval, the leader policy network parameters, shared state encoder parameters, individual decoder parameters and independent Beta action output layer parameters for each charging station, value sub-network parameters, and normalized scale parameters are saved as training checkpoints. After the preset training termination condition is met, the leader policy network parameters, shared state encoder parameters, individual decoder parameters and independent Beta action output layer parameters for each charging station, and normalized scale parameters are saved separately as online execution parameters.

[0093] In one training parameter configuration, both the policy network and the value sub-network use the Adam optimizer, and both the policy learning rate and the value learning rate are set to 5×10. -4 The discount factor is set to 0.99, the generalized advantage estimation parameter is set to 0.95, the truncation parameter is set to 0.2, and 15 rounds of proximal policy optimization iterations are performed for each batch of trajectories. The upper limit of the gradient norm is set to 10. Eight parallel training environments are used, and each training round includes 288 continuous control periods. The above values ​​are feasible and preferred configurations, which can be adjusted according to the control cycle and the scale of the training site. The preset training termination condition can be that the number of training steps or training rounds reaches a preset upper limit, or it can further include at least one of the following: the average change in reward within a preset sliding window is not greater than a preset reward change threshold, and the update magnitude of the policy network parameters is not greater than a preset parameter change threshold.

[0094] Step S65: After reaching the preset training termination condition and saving the online execution parameters, load the trained leader policy network parameters and corresponding normalized scale parameters into the load aggregator control platform; load the same set of trained shared state encoder parameters, as well as the individual decoder parameters, independent Beta action output layer parameters, and normalized scale parameters corresponding to the current charging station into each charging station controller; or, create follower policy instances for each charging station in the load aggregator control platform, so that each follower policy instance references the same set of shared state encoder parameters and loads the corresponding individual parameters respectively. During the online phase, the centralized value network module is not invoked, and the policy network parameters and normalized scale parameters of each agent remain fixed; each charging station only inputs its local observations into the corresponding follower policy instance, without transmitting local observations, local encoded features, or normalized actions to other charging stations. At the beginning of each online control cycle, or when changes occur in equipment connection status, energy status, expected departure time, target energy, the end time of the current scheduling cycle of fixed energy storage equipment, or the terminal target energy, the load aggregator control platform or the corresponding charging station controller detects the changes and re-executes steps S11 to S17 before the next strategy action is generated. This re-collects grid aggregation control instructions and equipment operation data, updates the dynamic energy lower limit, equipment feasible charging and discharging power boundary, station feasible power range, station energy boundary, aggregation layer energy boundary, station planned power, and upward and downward flexible capacity. Then, the strategy action generation, response screening and adjustment allocation, equipment-level control instruction issuance, and operation data readback are executed sequentially. The readback data is used as the input for the next online control cycle to achieve online rolling aggregation control.

[0095] The above method is applicable to charging stations that include both electric vehicles and stationary energy storage devices, as well as charging stations that only include electric vehicles. The number of network layers, the number of hidden units, the length of the regulation period, and the length of the training batch can be set according to the control cycle and the scale of the equipment, but the physical boundaries of the equipment, the decision-making order of leader-first and follower-parallel response, the screening threshold for responses in different directions, the sorting of comprehensive regulation feature values, the closed loop of the station's allocation of regulation to the equipment execution, and the time-period correspondence of the trajectories of each agent remain unchanged.

[0096] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0097] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0098] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the multi-charging station control methods based on master-slave game theory and multi-agent reinforcement learning in the above embodiments.

[0099] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0100] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements the above-mentioned multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning.

[0101] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0102] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0103] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0104] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0107] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0108] The number of training rounds, learning rate, hidden layer width, and numerical normalization scale mentioned above can be selected based on the control period, device rated parameters, and training configuration. During the online phase, the loaded network parameters and normalization scale parameters are fixed. The directional response screening threshold is generated by the normalization action of the leader policy network according to the mapping rules described in step S31, and is used for response screening in step S41; the response screening threshold is dynamically determined based on leader observations and policy outputs, rather than a fixed threshold that remains unchanged throughout each control period.

[0109] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning, characterized in that, include: Step S1: Collect grid aggregation control commands and equipment operation data, calculate the feasible charging and discharging power boundary of the equipment, aggregate to obtain the power and energy boundaries of the power station and aggregation layer, and generate the planned power and flexible capacity for upward and downward adjustment of the power station; Step S2: Construct a master-slave non-cooperative Markov game with the load aggregator as the leader and each charging station as a non-cooperative follower, and set the observations, actions and technical performance rewards for each agent. Step S3: The leader generates the target adjustment amount and the response screening threshold for each direction, and each follower generates the response amount and the comprehensive adjustment characteristic value based on the local observations of their own station; Step S4: Filter the stations according to the response screening threshold of the current regulation direction, sort them according to the comprehensive regulation characteristic value and allocate the target regulation amount to obtain the station allocation regulation amount and unmet regulation amount; Step S5: Generate the target power of the station from the planned power and the allocation adjustment amount, allocate it to the equipment to perform charge and discharge control, read back the power and energy status, calculate the reward and form a state transition sample; Step S6: The policy network and value sub-network are updated offline using a multi-agent reinforcement learning algorithm with centralized training and decentralized execution. Each follower concatenates the original local observations with the features output by the shared state encoder, and generates actions through individual decoders and independent Beta action output layers. After training, the policy network parameters and normalized scaling parameters are loaded and fixed, and online rolling aggregation control is implemented.

2. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 1, characterized in that, The specific operation process of step S1 includes: Step S11: Collect grid aggregation control commands, aggregation reference power, expected departure time of electric vehicles at each charging station, end time of the current scheduling cycle of fixed energy storage equipment at each charging station and terminal target energy, connection indication of equipment in each charging station, current energy, upper and lower limits of energy, target energy, rated charging and discharging power and charging and discharging efficiency; Step S12: With charging power as positive and discharging power as negative, establish the energy state transition relationship of the equipment based on the equipment's charging power, discharging power, charging efficiency, discharging efficiency, and the length of the control period. The specific formula is as follows: In the formula, This indicates the charging station index. Indicates the first Equipment index within each charging station. Indicates the current control period number; Indicates the first The first charging station Each device during the time period The charging and discharging power is determined according to the convention that charging power is positive and discharging power is negative; and These represent the energy of the device at the start of the current control period and at the start of the next control period, respectively. and These represent the charging efficiency and discharging efficiency of the device, respectively. Indicates the length of the control period; Represents the variable Perform the positive part operation, that is, take the larger value between the variable and zero; Step S13: Calculate the lower limit of dynamic energy, the specific formula is as follows: In the formula, Indicates the number of times the current action is performed. The first charging station The dynamic energy lower limit that each device needs to meet at the start of the next control period; This indicates the lower limit of the static energy allowed by the device; This indicates the target energy that the device needs to achieve at a target time, or the terminal target energy of a fixed energy storage device; This indicates the rated charging power of the device; Indicates charging efficiency; This indicates the expected departure time slot number for electric vehicles, or the end time slot number for the current scheduling cycle of fixed energy storage devices; This indicates the remaining charging time between the execution of the current control period action and the expected departure time; the expected departure time or cycle end time is calculated from the start of the corresponding period, and the remaining time is only calculated for devices that are still connected and have not yet reached that time. Step S14: Calculate the upper limit of the charging power and the lower limit of the signed charging and discharging power of the device during the current control period, respectively, according to the following formulas: In the formula, and They represent the first The first charging station Each device during the time period The upper bound of charging power and the lower bound of signed charging and discharging power; and These represent the rated charging power and rated discharging power of the device, respectively, both of which are non-negative values. This indicates the maximum allowable static energy of the device; This indicates a device connection indicator; the value is 1 when the device is connected and 0 when it is not connected. It represents the difference between the dynamic energy lower limit and the current energy during the next control period; when it is positive, the minimum charging demand is determined according to the charging efficiency, and when it is not greater than zero, the dischargeable boundary is determined according to the discharging efficiency. Step S15: Based on the sum of the lower and upper bounds of the charging and discharging power of the currently connected devices in the same charging station, determine the feasible power range of the same charging station. Then, aggregate the lower limit of the dynamic energy and the upper limit of the static energy of the devices to obtain the lower limit of the station energy and the upper limit of the station energy, and further obtain the energy boundary of the aggregation layer. Step S16. Generate the planned power of the power station within the feasible power range of the power station according to the preset planned power generation rules; Step S17: Use the difference between the upper end of the feasible power range of the power station and the planned power of the power station as the upward adjustment of flexible capacity, and use the difference between the planned power of the power station and the lower end of the range as the downward adjustment of flexible capacity.

3. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 2, characterized in that, Step S2 specifically includes: Step S21: Set the load aggregator as the leader agent and each charging station as a follower agent. Execute the leader decision, follower parallel decision, response screening and adjustment allocation, and physical execution in sequence. After being given a leader action, each follower makes a non-cooperative decision with the goal of increasing the expected cumulative reward of its station. Step S22: The leader observes the grid aggregation control command, aggregation reference power, power and energy boundary of the power station, planned power, actual power, flexible capacity, unmet adjustment amount and time information of the previous period, and outputs the target adjustment amount and the screening threshold for upward and downward adjustment responses; each follower observes the power, energy and flexible capacity information of its own power station, the characteristic value of the basic regulation impact, time information and the leader's target adjustment amount, and outputs the ratio of upward and downward adjustment responses and the corresponding strategy adjustment coefficient; Step S23: The leader reward consists of a weighted negative value of aggregated power tracking deviation, unmet regulation amount, and aggregated regulation impact index; the follower reward consists of a weighted positive value of the station response contribution index and a weighted negative value of the station power tracking deviation, equipment target energy deviation, and station regulation impact index; the basic regulation impact characteristic value is determined based on at least one normalized physical quantity among station energy margin, target energy urgency, available flexible capacity, and executable power boundary; the aggregated power tracking deviation is compared with the grid reference power corresponding to the external grid command, and when the command is an increment relative to the aggregated reference power, the two are added together to obtain the grid reference power.

4. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 3, characterized in that, Step S3 specifically includes: Step S31: The leader agent generates target adjustment normalization action, response screening threshold normalization action and response screening threshold normalization action based on its observations; determines the mapping boundary of the target adjustment amount based on the sum of the planned power of each station and the aggregated feasible power range, and maps the two threshold normalization actions to the response screening threshold and response screening threshold respectively. Step S32: Each follower agent generates, in parallel, an upward adjustment ratio, an upward adjustment policy adjustment coefficient, a downward adjustment ratio, and a downward adjustment policy adjustment coefficient based on its local observations; the current target adjustment direction is determined based on the sign of the target adjustment amount, and the response amount ratio and policy adjustment coefficient corresponding to the current target adjustment direction are selected; Step S33: Multiply the selected response ratio by the flexible capacity of the corresponding charging station in the current target adjustment direction to obtain the response amount; calculate the comprehensive adjustment characteristic value based on the basic adjustment influence characteristic value of the corresponding station, the selected strategy adjustment coefficient, and the preset characteristic adjustment amplitude upper limit.

5. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 4, characterized in that, Step S4 specifically includes: Step S41: Determine the current target adjustment direction based on the sign of the target adjustment amount, and select the corresponding response screening threshold; remove stations whose comprehensive adjustment feature value is greater than the response screening threshold, form a candidate response set for the remaining stations, and arrange them in ascending order according to the comprehensive adjustment feature value. If the feature values ​​are the same, they are sorted according to the preset station identifier. Step S42: Set the absolute value of the target adjustment amount as the initial remaining unallocated adjustment amount, determine the station allocation adjustment amount of each charging station in turn according to the sorting result of the candidate response set, and deduct the remaining unallocated adjustment amount after each allocation; when the sum of all response amounts in the candidate response set is less than the absolute value of the target adjustment amount, record the difference between the two as the unmet adjustment amount. Based on the ranking of the candidate response set, the station allocation adjustment amount for each charging station is determined according to the following formula: In the formula, This represents the aggregation layer target adjustment amount generated by the leader agent relative to the aggregation plan power; The sign representing the target adjustment amount of the aggregation layer; when When >0, the direction d takes "+", when When d < 0, the direction d takes the "-" value; n represents the current station's sorting position in the candidate response set. Indicates the sorting position preceding the current station; This represents the response quantity of the station at the sorting position n in the selected adjustment direction d; This represents the station allocation adjustment amount corresponding to the station at the sorting position n; This indicates positive part operation; when n=1, the summation of empty spaces in the formula is treated as zero; Step S43: Add the station allocation adjustment amount of each charging station to the corresponding station planned power to obtain the station target power; when the target adjustment amount is zero, do not perform response screening and adjustment amount allocation, set the station allocation adjustment amount of each charging station to zero, and make the station target power equal to the corresponding station planned power.

6. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 5, characterized in that, Step S5 specifically includes: Step S51: Determine the power allocation coefficient based on the relative position of the target power of the power station within the feasible power range of the power station. Interpolate the corresponding aggregated target power within the aggregated feasible power range of electric vehicles and stationary energy storage respectively. Then allocate the power of each device within the feasible charge and discharge power boundary of each device. When the total executable power of the devices is sufficient to reach the target power of the power station, make the sum of the device power consistent with the target power. When the width of the feasible power range of the power station is zero, execute directly according to the unique feasible power of each device. When the width of the aggregated feasible power range of electric vehicles or stationary energy storage is zero, allocate power according to the unique endpoint of the corresponding range. Step S52: Send the charging and discharging power command of the corresponding equipment to the charging pile controller and the energy storage converter configured in the station, read the actual power and energy status of the equipment after execution, summarize the actual power at the grid connection point of the station, and use the updated operating data for the calculation of the feasible power boundary in the next control period. Step S53: Calculate the technical performance bonus based on the deviation between the aggregated actual power and the grid reference power, the unmet regulation amount, the deviation between the actual power of the power station and the target power, the equipment energy deviation, and the power station regulation impact amount; Step S54: Summarize the observations of each agent, the corresponding decision-making stage state, action, log probability of the old policy action, reward, next observation, next state, and termination flag to form a state transition sample.

7. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 6, characterized in that, Step S6 specifically includes: Step S61: After completing action generation, response screening and adjustment allocation and physical execution at each joint decision-making step, record the state transition samples of each agent as the corresponding training trajectory. Step S62: Set up value sub-networks corresponding to the leader and each follower respectively. The leader value sub-network reads the centralized state of the leader's decision-making stage, and the follower value sub-network reads the centralized state of the follower's decision-making stage that includes the leader's actions. Each policy network reads the corresponding local observations, outputs the Beta distribution shape parameters of each normalized action dimension and samples the actions. Calculate the policy probability ratio based on the probability of the updated policy and the old policy for the same action. Step S63: Calculate the temporal difference residual and generalized advantage estimate based on the reward sequence, state value and termination flag of each agent, and update the policy network using the truncated policy probability ratio objective function; use the sum of the generalized advantage estimate and the state value saved during trajectory acquisition as the reward target, and update the value sub-network based on the mean square error between the predicted state value and the reward target. Step S64: Using the same round of sampling trajectory, update the policy network and value sub-network of each agent under the condition that the old policy parameters are fixed. Repeat offline trajectory collection and network update until the preset training termination condition is reached. Save the trained policy network parameters and normalized scale parameters as online execution parameters.

8. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 7, characterized in that, The specific steps of step S62 include: Step S621: Each follower strategy network shares a state encoder and is configured with an individual decoder and an independent Beta action output layer corresponding to each charging station. The shared state encoder receives local observations of each station and outputs local encoded features. Step S622: Concatenate the original local observations of each follower with the corresponding local encoded features into vectors, and input the vectors into the individual decoder of the station to obtain the individual policy features; Step S623: The independent Beta action output layer of the station generates two shape parameters for each action dimension based on individual policy characteristics through positive value mapping. Normalized actions located in the [0,1] interval are sampled according to the corresponding Beta distribution and used to determine the ratio of up and down adjustment of response and the corresponding policy adjustment coefficient.

9. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 7, characterized in that, Step S6 further includes: Step S65: Load and fix the trained strategy network parameters and normalized scale parameters on the load aggregator control platform and the corresponding charging station controller. Do not call the centralized value network module during the online phase. When each online control cycle begins or the operating status update conditions are met, re-execute steps S11 to S17 before the next action is generated. Then, generate actions, filter responses, allocate adjustment amounts, and issue equipment control commands in sequence. Use the readback operating data for the next control cycle.

10. The multi-charging station control method based on master-slave game theory and multi-agent reinforcement learning as described in claim 9, characterized in that, The operating status update condition in step S65 is that at least one of the following changes: device connection status, energy status, expected departure time, target energy, current scheduling cycle end time of fixed energy storage equipment, and terminal target energy. The load aggregator control platform or the corresponding charging station controller detects the change and triggers the re-execution of steps S11 to S17 after the change is detected and before the next strategy action is generated.