Vehicle-network interactive cooperative scheduling method and system based on multiple agents

Abstract

The invention relates to the technical field of electric power marketization scheduling, in particular to a multi-agent-based vehicle-network interactive cooperative scheduling method and system. The method comprises the following steps: firstly, constructing a layered federated architecture of a power grid layer, an aggregator layer and a user layer, wherein the power grid layer generates a region-level charging and discharging price interval and a frequency regulation instruction by adopting a depth deterministic strategy gradient algorithm; the aggregator layer uses a multi-agent depth deterministic strategy gradient model to realize cooperative scheduling and power market bidding decision between charging stations; and the user layer optimizes charging and discharging behaviors of the electric vehicle based on a deep Q network algorithm under a federated learning framework. According to the system, cross-level collaborative optimization and closed-loop control are realized through a power grid dispatching center module, an aggregator decision module and a user terminal optimization module. According to the method, the problem of low cooperation efficiency caused by multi-subject benefit games in the power environment is effectively solved, and the system economy, the operation stability and the renewable energy consumption capability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of power market dispatching technology, and more specifically, to a vehicle-grid interactive collaborative dispatching method and system based on multi-agent systems. Background Technology

[0002] With the deepening of energy structure transformation, the large-scale integration of electric vehicles into the power grid has become an important trend. The massive number of electric vehicles not only puts pressure on the power grid's power balance but also creates considerable distributed energy storage resources due to their clustering effect. Against this backdrop, promoting the two-way interaction between electric vehicles and the power grid will help enhance the grid's regulation capabilities and facilitate the consumption of new energy sources.

[0003] To effectively schedule these distributed resources, existing technologies typically employ a centralized control architecture. A central control system generates and issues charging and discharging commands based on preset rules and optimization algorithms, requiring the terminals to passively execute them. While this approach can achieve basic load regulation in simple scenarios, its core flaw lies in treating the power grid, aggregators, and users, which have autonomous decision-making characteristics, as undifferentiated execution units. This rigid control mode is difficult to adapt to the real-time and dynamic requirements of multi-subject distributed decision-making, resulting in significant command response deviations, low coordination efficiency, and difficulty in achieving optimal allocation of system resources in actual operation. Therefore, there is a need to develop an intelligent scheduling method that can achieve collaborative optimization among multiple stakeholders and adapt to dynamic market changes. Summary of the Invention

[0004] One objective of this invention is to provide a vehicle-grid interactive cooperative scheduling method and system based on multi-agent technology. This method constructs a hierarchical federated architecture comprising a grid layer, an aggregator layer, and a user layer. It employs a deep deterministic policy gradient algorithm to generate grid-side price and frequency adjustment commands, utilizes a multi-agent deep deterministic policy gradient algorithm to achieve coordinated bidding and power allocation among charging stations, and optimizes user-side charging and discharging behavior based on a deep Q-network and federated learning framework. This addresses the problems mentioned in the background art, namely: In a market-oriented electricity environment, the centralized dispatching and coordination system suffers from low efficiency and difficulty in achieving optimal resource allocation due to the interplay of interests and strategic conflicts among multiple stakeholders.

[0005] To achieve the above objective, the method includes the following steps: S1. Divide the vehicle-grid interactive collaborative scheduling system into a grid layer, an aggregator layer, and a user layer, and perform state modeling for each layer based on a Markov decision process. S2: The power grid layer collects real-time electricity market information and power grid operation status data, constructs a decision model through a deep deterministic strategy gradient algorithm, generates regional charging and discharging price ranges and frequency adjustment instructions, and sends them to the aggregator layer. S3. The aggregation layer constructs a multi-agent deep deterministic policy gradient model, treating each charging station as an independent agent. Each intelligent agent collects local electric vehicle queue status and electricity price information and receives instructions from the power grid layer; Each intelligent agent generates the charging and discharging power allocation and bid price through a distributed Actor network based on the received instructions and collected information. The aggregation layer evaluates the joint actions of all agents through a centralized Critic network and coordinates policy conflicts between different agents based on the evaluation results. S4. Each terminal in the user layer receives the strategy from the aggregation layer and collects local vehicle status and user travel needs. It generates charging and discharging behavior decisions that meet the battery charging and discharging rate constraints and travel time constraints through the deep Q network algorithm, and uploads them to the aggregation layer. S5. The aggregation layer calculates the deviation between the actual load and the planned load of each charging station based on and updates its own model, and feeds back the deviation information to the power grid layer. S6. The power grid layer adjusts the parameters of the decision model based on the deviation information and updates the global scheduling strategy.

[0006] The design concept of the above technical solution stems from a deep reconstruction of the shortcomings of the traditional centralized scheduling architecture: relying solely on a single control center to directly manage massive numbers of terminals will lead to the "curse of dimensionality" and "policy conflict" due to the inability to handle high-dimensional state spaces and multi-agent game dynamics, resulting in scheduling instructions deviating from reality and response delays. To address this, this solution pioneers a layered federated architecture, using the grid-layer DDPG algorithm to achieve global guidance of market and security constraints, replacing the rigid unified instruction issuance; without the multi-agent collaborative mechanism of the aggregation layer MADDPG, it is impossible to resolve the resource competition conflict between charging stations, nor can the benefits of joint actions be evaluated from a centralized Critic's global perspective; and the combination of lightweight DQN and federated learning at the user layer breaks through the bottleneck of the difficulty in simultaneously achieving individual privacy and collaborative training. Without this design, distributed policy optimization cannot be achieved while protecting user data.

[0007] Based on this, the agent will form a sample tuple with the state, action, reward and next state generated by each interaction and store it in a fixed-size experience replay pool. During model training, the agent will randomly sample a small batch of samples from the pool for learning.

[0008] In another technical solution, the aggregated user policy gradient adopts a federated averaging algorithm, and the weight allocation is positively correlated with the historical policy execution accuracy of each terminal. The accuracy calculation formula is the ratio of actual response power to planned response power.

[0009] This technical solution effectively addresses the training instability and convergence difficulties caused by the strong correlation between sequence samples in traditional reinforcement learning through the experience replay mechanism. By breaking the temporal correlation between samples and improving data utilization, it solves the problems of training instability and convergence difficulties. Without this design, the agent struggles to efficiently extract stable policies from historical experience, easily getting trapped in local optima or policy oscillations. The introduction of the federated averaging algorithm specifically addresses the heterogeneity challenge in distributed terminal collaborative training. By implementing a precision-weighted gradient aggregation mechanism, it ensures the dominant role of high-reliability terminals in global model optimization while suppressing the interference of low-precision terminal noise on the training process. This avoids the model bias problem caused by traditional averaging aggregation strategies ignoring terminal differences, thus significantly improving the robustness and practicality of the global policy while ensuring convergence efficiency.

[0010] The second objective of this invention is to provide a vehicle-grid interactive collaborative scheduling method and system based on multi-agent systems, including a power grid dispatch center module, an aggregator decision-making module, and a user terminal optimization module; The power grid dispatch center module is used to execute steps S2 and S6; The aggregator decision module is used to execute steps S3 and S5; The user terminal optimization module is used to execute step S4; Each module uses a federated learning framework for encrypted parameter sharing and dynamically optimizes the scheduling strategy through a closed-loop feedback mechanism.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By employing a hierarchical federated multi-agent reinforcement learning architecture, this invention achieves a unified approach to "global optimization, distributed decision-making, and privacy protection" at the technical level. Traditional methods either rely on centralized optimization at the expense of response speed and privacy, or employ fully distributed decision-making, making it difficult to guarantee overall system optimization. This invention, through global state perception and instruction generation at the power grid layer, multi-agent game coordination at the aggregator layer, and local optimization and federated learning at the user layer, maintains the real-time performance and flexibility of distributed decision-making while ensuring global optimization of system economy and security through cross-layer collaboration. Furthermore, by utilizing the federated learning framework for collaboration at the parameter level rather than the data level, it fundamentally eliminates the risk of user privacy leakage.

[0012] 2. This invention innovatively couples multi-agent collaboration with the electricity market mechanism, solving the dynamic adaptability problem of resource allocation in a market-based environment. Most existing methods treat market signals as fixed boundary conditions, making them difficult to adapt to real-time fluctuating market environments. By using market information such as nodal prices and supply-demand gaps as input states for deep reinforcement learning, and integrating economic benefits, instruction tracking, and collaborative benefits through reward function design, the scheduling strategy can respond to market changes in real time. This maximizes the comprehensive benefits of all market participants while meeting grid security constraints, achieving seamless integration between technological implementation and market mechanisms. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall process structure of the vehicle-network interactive collaborative scheduling method and system based on multi-agent communication of the present invention; Figure 2 This is a schematic diagram of the specific process of step S3 of the present invention; Figure 3 This is a schematic diagram of the specific process of step S4 of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Here are some explanations of technical terms: Markov decision processes (MDPs) are a mathematical framework for describing sequential decision problems. Their core feature is the Markov property, which states that the state of the system at the next time step depends only on the current state and the action taken, and is independent of the historical states. This property provides a computable theoretical basis for reinforcement learning algorithms. Deep Deterministic Policy Gradient (DDPG) is a deep reinforcement learning algorithm for continuous action spaces. It adopts an Actor-Critic architecture and is trained using deterministic policy gradients. It is suitable for scenarios where a single agent interacts with the environment. Multi-agent Deep Deterministic Policy Gradient (MADDPG) is an extension of DDPG in a multi-agent environment. It adopts a centralized training and distributed execution framework and coordinates the policy learning of multiple agents through a global Critic network to solve the problems of environmental non-stationarity and inter-agent cooperation. Deep reinforcement learning networks (DQN) are reinforcement learning algorithms that combine deep neural networks with Q-learning. Through experience replay and a stable training process of the target network, they can handle high-dimensional state spaces and learn optimal action policies.

[0016] Currently, in the context of a market-based electricity market, centralized dispatching suffers from low efficiency and difficulty in achieving optimal resource allocation due to multi-agent interest games and strategy conflicts. One objective of this invention is to provide a vehicle-grid interactive collaborative dispatching method and system based on multi-agent interaction. See [link to relevant documentation]. Figure 1 As shown, it includes the following steps: S1. Divide the vehicle-grid interactive collaborative scheduling system into a grid layer, an aggregator layer, and a user layer, and perform state modeling for each layer based on a Markov decision process. S2: The power grid layer collects real-time electricity market information and power grid operation status data, constructs a decision model through a deep deterministic strategy gradient algorithm, generates regional charging and discharging price ranges and frequency adjustment instructions, and sends them to the aggregator layer. S3. The aggregation layer constructs a multi-agent deep deterministic policy gradient model, treating each charging station as an independent agent. Each intelligent agent collects local electric vehicle queue status and electricity price information and receives instructions from the power grid layer; Each intelligent agent generates the charging and discharging power allocation and bid price through a distributed Actor network based on the received instructions and collected information. The aggregation layer evaluates the joint actions of all agents through a centralized Critic network and coordinates policy conflicts between different agents based on the evaluation results. S4. Each terminal in the user layer receives the strategy from the aggregation layer and collects local vehicle status and user travel needs. It generates charging and discharging behavior decisions that meet the battery charging and discharging rate constraints and travel time constraints through the deep Q network algorithm, and uploads them to the aggregation layer. S5. The aggregation layer calculates the deviation between the actual load and the planned load of each charging station based on and updates its own model, and feeds back the deviation information to the power grid layer. S6. The power grid layer adjusts the parameters of the decision model based on the deviation information and updates the global scheduling strategy.

[0017] By combining multi-agent deep reinforcement learning with federated learning, the ultimate goal of improving the overall economy and scheduling efficiency of the system is achieved while ensuring the interests and privacy of all parties involved.

[0018] Step S1 of this invention involves constructing a hierarchical federated multi-agent reinforcement learning architecture and modeling the system state. Specifically, the primary task of this step is to lay the foundation for the entire collaborative scheduling method in terms of system architecture and theoretical model, and to clarify the functional division and information interaction relationships of each layer.

[0019] First, the entire vehicle-grid interactive and coordinated dispatch system is divided into three layers: the grid layer, the aggregator layer, and the user layer. This division is based on the roles, decision-making granularity, and information scope of each entity in the electricity market. The grid layer, as the global coordinator, is responsible for perceiving the system status from a macro perspective and generating guiding instructions; the aggregator layer, as the intermediate layer bridging the upper and lower layers, is responsible for coordinating the dispatching behavior of multiple charging stations within its management scope; and the user layer, composed of a massive number of electric vehicle terminals, is the final execution unit.

[0020] In terms of state modeling, a probabilistic model of states, actions, rewards, and state transitions for the entire system is constructed based on Markov decision processes. Specifically, the grid layer is responsible for modeling and perceiving the global state, and its state space includes high-dimensional operational data such as nodal electricity prices, regional supply and demand gaps, renewable energy output, current system frequency deviation, and total load curves. The aggregator layer and the user layer handle the local states they observe. For example, the state of the aggregator layer includes instructions issued by the grid layer, real-time load of its charging stations, electric vehicle queuing status within the stations, and local electricity price information; the state of the user layer includes the current battery health status, battery state of charge, remaining driving range, user's planned departure time, and estimated driving range.

[0021] Since the aggregator layer and user layer cannot obtain complete global information, their decision-making environment suffers from information incompleteness. Therefore, this step introduces a partially observable Markov decision process to address this problem. Within this framework, each agent estimates the true state of the environment based on its observed local state history, thereby making a decision. Through the above architecture and modeling, a clear theoretical framework and system boundaries are provided for subsequent steps applying deep reinforcement learning algorithms to achieve hierarchical, distributed collaborative optimization.

[0022] Step S2 of this invention is to generate dispatch instructions at the power grid layer. Specifically, the primary task of this step is to use a deep deterministic strategy gradient algorithm to generate regional-level charging and discharging price guidance ranges and frequency adjustment instructions that meet security constraints based on global electricity market information and power grid operating status, and then send them down to the aggregator layer to guide the charging and discharging behavior of the lower-level entities.

[0023] The power grid layer first collects and processes key information in real time through its global status acquisition unit. The collected data includes electricity market information, such as nodal marginal electricity prices, regional supply-demand gap power values, and real-time output of renewable energy sources such as wind farms and photovoltaic power plants. At the same time, it collects power grid operation status data, the core of which is the current frequency deviation value of the monitoring system. This value directly reflects the real-time balance of active power in the power grid and is a key indicator to ensure the safe and stable operation of the power grid.

[0024] Subsequently, a deep deterministic policy gradient model was constructed as the core decision engine. This model comprises two core neural networks: an Actor network and a Critic network. The Actor network, acting as the policy function, takes a preprocessed global high-dimensional state vector as input and outputs continuous adjustment commands. These commands specifically include the upper and lower limits of the regional electric vehicle charging and discharging price range, measured in yuan per kilowatt-hour, and the frequency adjustment command power value, measured in megawatts. It is particularly important to emphasize that all output commands must meet strict grid security constraints, with the frequency adjustment command ensuring that the system frequency deviation is maintained within a safe operating range of ±0.2 Hz.

[0025] The Critic network, acting as the value function, takes the current state vector and the action instructions output by the Actor network as joint inputs and outputs a scalar Q-value. This Q-value is used to evaluate the expected long-term cumulative reward obtained after executing the corresponding adjustment instruction under the current grid state, thereby judging the merits of the action from the perspective of global returns.

[0026] The reward function design comprehensively considers multiple objectives, including safety, economy, and command smoothness. The safety constraint reward provides a positive reward when the system frequency deviation is less than or equal to ±0.2 Hz; a severe negative reward is given if this safety threshold is exceeded. The economic reward is positively correlated with the reduction in the regional supply-demand gap and the successful absorption of renewable energy, aiming to incentivize dispatching behavior to prioritize the absorption of new energy and balance supply and demand. The smoothness reward is negatively correlated with the fluctuation amplitude of dispatching commands between continuous time steps, aiming to avoid drastic changes in power commands and ensure the safe operation of power equipment and system stability.

[0027] Model training employs an experience replay mechanism and a target network update strategy. The agent explores and interacts in the environment, storing the resulting state, action, reward, and next-state sample tuple in a fixed-size experience replay pool. During training, mini-batch samples are randomly sampled from this pool. The Critic network is updated by minimizing the mean squared error loss between its predicted Q-value and the target Q-value, which is calculated by the target Critic network. The Actor network is updated following the deterministic policy gradient theorem, aiming to maximize the Q-value evaluated by the Critic network. Parameters are updated by calculating the gradient of the Q-value with respect to the action and combining it with the policy gradient. The target network replicates parameters from the main network using a soft update method, periodically updating the main network's parameters to the target network at a very small rate, thus stabilizing the entire training process.

[0028] After training, the model enters the online application phase. The Actor network in the DDPG model training and decision-making unit at the power grid layer generates regional-level charging / discharging price ranges and frequency regulation commands that satisfy all safety and economic constraints, based on the latest global state information collected in real time. These commands are then encrypted and distributed to all aggregator decision modules at the lower layer through the command issuance and feedback receiving unit, thus initiating a new round of hierarchical collaborative scheduling.

[0029] like Figure 2 As shown, step S3 of the present invention is to coordinate the scheduling behavior of charging stations at the aggregation layer. Specifically, the primary task of this step is to construct a multi-agent deep deterministic policy gradient model, so that each charging station acts as an independent agent. Based on receiving instructions from the power grid layer and combining local state information, it makes distributed decisions on its own charging and discharging power allocation scheme and electricity market bidding price. It also coordinates group policy conflicts through a centralized evaluation mechanism to achieve collaborative optimization among charging stations.

[0030] To achieve this goal, a multi-agent deep deterministic policy gradient model needs to be constructed first. This model employs a centralized training and distributed execution framework, comprising a distributed Actor network corresponding one-to-one with each charging station agent and a shared centralized Critic network. Each charging station agent's Actor network takes the charging station's local state information and instructions from the power grid as input. The local state includes constraints such as the real-time charging demand of the electric vehicle queue, waiting status, the current total load of the charging station, and the station's capacity limit. The output of the Actor network is the charging station's continuous decision-making actions, specifically including the total charging and discharging power to be allocated and the planned electricity price to bid in the electricity market. It is particularly important to emphasize that the allocated charging and discharging power must meet the charging demand constraints of the electric vehicle queue while not exceeding the charging station's own capacity limit; this is a critical operational constraint to ensure both user demand and local safety.

[0031] The centralized Critic network is responsible for evaluating the overall effectiveness of the joint actions of multiple agents. This network takes as joint input the local state information of all charging station agents, grid-level instructions, and the decision actions output by all Actor networks, and outputs a global Q-value. This Q-value is used to comprehensively evaluate the expected long-term cumulative reward that all charging stations can obtain after taking joint actions under the current global state, providing a global perspective and collaborative guidance for the policy optimization of each Actor network.

[0032] The reward function is designed to guide each charging station agent to pursue its own interests while also considering the overall system's collaborative goals. The reward function comprises three core components: a command tracking reward, which incentivizes the matching degree between each charging station's charging and discharging power and the commands issued by the grid layer; the higher the matching degree, the greater the reward value. An economic benefit reward, calculated based on the difference between the bid price and the real-time electricity price, and the final transaction volume, aims to maximize the economic benefits for aggregators participating in the electricity market. A collaborative reward, on the other hand, encourages balanced power distribution among charging stations, preventing uneven resource allocation where some stations are overloaded while others are idle.

[0033] Through this MADDPG model, each charging station agent can make independent decisions based on its own locally observed information, while the centralized Critic network evaluates the overall effect of these decisions using global information and guides each agent to adjust its strategy through gradient updates. This design maintains the flexibility and privacy of distributed decision-making while effectively coordinating policy conflicts between different charging stations through a centralized evaluation mechanism, ultimately achieving collaborative optimization within the aggregator layer and proactive response to commands from the power grid layer.

[0034] like Figure 3 As shown, step S4 of the present invention optimizes the participation behavior of electric vehicles at the user layer. Specifically, the primary task of this step is to enable each electric vehicle terminal to autonomously optimize its charging and discharging behavior based on the strategy suggestions provided by the aggregator layer, combined with the user's own travel needs and the real-time status of the vehicle, using a lightweight deep reinforcement learning network algorithm. The learning results are shared with the upper layer through a federated learning framework instead of the original data, thereby achieving collaboration with the system strategy while ensuring user privacy.

[0035] The electric vehicle terminal first receives strategy parameters from the aggregator layer, including recommended charging and discharging periods and price coefficients. Simultaneously, the terminal acquires two types of key data through its local status acquisition unit: first, vehicle status data, including battery health, maximum battery capacity, current battery state of charge, and remaining driving range; second, user travel demand data, including planned departure time and estimated driving distance. This local information collectively constitutes the state input for decision-making.

[0036] Based on this information, each electric vehicle terminal builds and runs its local Deep Q-Network (DQN) model for behavior optimization. This DQN model comprises two identical neural networks: a Q-Network for real-time decision-making and a target Q-Network for stable training. The Q-Network employs a lightweight, fully connected network structure, encoding the aforementioned state information into feature vectors and outputting the Q-value estimate for each candidate action. The candidate action space consists of discrete charging / discharging periods and discrete power levels (e.g., 0.5kW, 1kW, 2kW, etc.). All actions must meet the hard constraint that the battery charge / discharge rate does not exceed 1C and ensure that the required charging amount is completed before the user's planned departure time. The target Q-Network periodically replicates parameters from the Q-Network to calculate a stable target Q-value.

[0037] The reward function of the DQN model is designed to balance user interests and system needs, and includes three core parts: economic rewards encourage users to charge during low electricity price periods and discharge during high electricity price periods to gain benefits; demand fulfillment rewards ensure that the vehicle has enough charge to reach the expected driving range before departure; and strategy synergy rewards provide positive incentives for behaviors that follow the aggregator's guidance strategy.

[0038] The model is trained using an experience replay mechanism and temporal difference learning. The terminal stores the experience tuples (state, action, reward, next state) generated from each decision in a local experience pool. During training, batch samples are randomly drawn, and the network parameters are updated by minimizing the error between the current Q-network's prediction and the target Q-value. The formula for calculating the target Q-value is: ; In the formula, The target Q value represents the agent's current state. Next action The target estimate of the total cumulative discounted return that can be obtained afterward; This represents an immediate reward, which is given by the agent while it is in a certain state. Next action Then, the environment immediately provides a reward signal; The discount factor is a constant between 0 and 1 (0 ≤ 1 ... <1), which determines how much importance the agent attaches to future rewards. The closer it is to 0, the more "short-sighted" the agent is, and the more it values ​​immediate rewards. The closer it is to 1, the more "visionary" the agent is, and the more it will consider long-term future returns. This represents the maximum expected future return for the next state. Operation means that the agent is in a state Next, select and evaluate the action that yields the maximum Q value, which represents the current value of the best future reward the agent can obtain starting from the next state; This represents the maximum Q-value output of the target Q-network for the next state; This indicates the "next state," the state entered after the current action is executed, the local state updated after the electric vehicle executes the current charging and discharging action, the updated user travel demand information, and the strategy parameters that the aggregator layer may update. This represents "all possible actions in the next state", that is, all candidate charging and discharging behaviors that an electric vehicle can choose after entering the next state s.

[0039] Most importantly, the terminals do not upload raw data containing sensitive information. Instead, they encrypt the policy gradients generated during training and upload them to the aggregator layer using a federated learning framework. The aggregator layer performs federated averaging of the gradients uploaded by all terminals to generate a global gradient before distributing it to each terminal. Each terminal then fine-tunes its local DQN model parameters based on this global gradient, thereby achieving collaborative optimization between user behavior and the aggregator's strategy, while strictly protecting the privacy and security of user data.

[0040] Step S5 of this invention is strategy aggregation and deviation feedback. Specifically, the primary task of this step is to process the learning information uploaded by the user layer, optimize the scheduling strategy of the aggregation layer, calculate the deviation between actual operation and plan, and feed back the corrected load deviation information to the power grid layer, thereby forming a bottom-up closed-loop feedback channel to provide data support for the dynamic adjustment of the global strategy.

[0041] The aggregator layer first receives policy information uploaded from each electric vehicle terminal in the user layer. This information includes the charging and discharging policy gradients generated by each terminal through the DQN algorithm and their corresponding execution status data, rather than the raw sensitive data. Policy aggregation and model updates employ a federated averaging algorithm to weighted aggregate the user layer policy gradients, generating a global policy gradient. The weight allocation is positively correlated with the historical policy execution accuracy of each terminal; specifically, the accuracy value is equal to the ratio of actual response power to planned response power, with terminals having higher accuracy receiving a larger weight proportion during aggregation. Subsequently, the generated global policy gradient is injected into the aggregator layer's local MADDPG model to update the parameters of its Actor and Critic networks, achieving federated collaborative optimization of model parameters. This allows the aggregator layer's decisions to better reflect the actual response characteristics of the user layer.

[0042] To optimize local scheduling strategies, the aggregator layer simultaneously constructs a local experience pool, storing sample tuples consisting of user-level strategies, joint actions of charging stations, global rewards, and the next state. The global reward includes economic benefits and collaborative benefits. Economic benefits are calculated as the product of the bid price difference and the transaction volume, while collaborative benefits assess the degree of cooperation among multiple charging stations. Through an experience replay mechanism, the aggregator layer can continuously optimize its local scheduling strategy using these samples, improving the accuracy and adaptability of its decisions.

[0043] Load deviation calculation is another core task in this step. The aggregated layer statistically analyzes the original deviation between the actual load and planned load of each charging station within its jurisdiction and performs multi-dimensional correction analysis. First, capacity constraint correction is performed, weighting the deviation based on the real-time capacity margin of each charging station (calculated as the capacity limit minus the current actual load). Second, response delay correction is performed, compensating for deviations caused by communication or execution delays based on the historical response time distribution characteristics of the electric vehicle cluster. The final deviation report includes the corrected total load deviation ΔP, the time-period deviation percentages for peak, off-peak, and low-peak periods, an analysis of the main causes of the deviation (such as capacity constraints or response delays), and real-time capacity margin information for each charging station.

[0044] The deviation report is ultimately fed back to the command issuing and feedback receiving unit at the power grid level via a secure communication link. This deeply analyzed and corrected deviation data provides the power grid level with accurate and reliable input for dynamically adjusting the global scheduling strategy in the next step, thus completing a closed-loop feedback cycle from local execution to global optimization.

[0045] Step S6 of this invention is a global strategy iterative update. Specifically, the primary task of this step is to enable the power grid layer to dynamically adjust its global scheduling strategy based on the load deviation information fed back by the aggregation layer. Through a closed-loop feedback mechanism, the multi-level vehicle-grid interactive scheduling effect is continuously optimized, ultimately achieving the goal of hierarchical, linked, and efficient collaborative control in the power market environment.

[0046] The power grid layer receives load deviation reports uploaded by the aggregator decision module through its command issuance and feedback receiving unit. This report includes key information such as the total load deviation ΔP after capacity constraints and response delay corrections, the time-period deviation percentages for peak, off-peak, and low-peak periods, a detailed analysis of the main causes of deviations (e.g., capacity-constrained percentage, delayed response percentage), and the real-time capacity margin of each charging station. This data reflects the differences between the actual execution at the lower level and the original planned commands within the previous scheduling cycle, and the reasons for these differences.

[0047] Based on this deviation information, the DDPG model training and decision-making unit at the power grid level initiates a dynamic adjustment process for the global strategy. First, in-depth analysis of the deviation data is performed to identify systematic deviation patterns. For example, if the proportion of negative deviations during peak hours is high and the main reason is capacity constraints, it indicates that the aggregator layer's regulation capacity during peak electricity consumption has reached its limit; if the deviation analysis shows that delayed response is dominant, it indicates that the actual response speed at the user layer is slow. These analytical results provide a clear direction for strategy adjustment.

[0048] Subsequently, the grid layer incorporates deviation information as a crucial environmental feedback signal into the training process of its DDPG algorithm. Specifically, when evaluating the Q-value of state-action pairs, the Critic network considers the impact of instruction execution deviations, giving lower evaluations to actions with large negative deviations. The Actor network, through policy gradient updates, learns to generate scheduling instructions that better adapt to the actual response characteristics of lower layers. For example, when it is identified that the capacity of charging stations in a certain area is generally limited, the subsequently generated charging and discharging price range will be appropriately widened to give the aggregator layer more operational space; when a severe response delay is detected, instructions will be issued earlier or the requirements for timing accuracy will be reduced.

[0049] Through continuous learning and optimization based on actual deviations, the grid-level dispatching strategy constantly adapts to the actual response characteristics of the underlying entities, gradually reducing the deviation between commands and execution. The entire system achieves comprehensive, multi-level collaborative optimization from the grid level to the aggregator level and then to the user level through this closed-loop iterative mechanism of "command issuance - execution feedback - strategy update - re-command issuance." Ultimately, in the context of the electricity market, it achieves the system operation goals of hierarchical control, coordinated response, and efficient collaboration, significantly improving the accuracy, economy, and reliability of vehicle-grid interactive dispatching.

[0050] The second objective of this invention is to provide a vehicle-grid interactive collaborative scheduling method and system based on multi-agent systems. The system comprises a power grid dispatch center module, an aggregator decision-making module, and a user terminal optimization module. Each module achieves encrypted parameter sharing through a federated learning framework. Through a closed-loop feedback mechanism of "power grid layer - aggregator layer - user layer," the system dynamically optimizes the vehicle-grid interactive scheduling strategy, reducing peak-valley differences in the power grid and improving system economy.

[0051] The system also includes a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program is used to implement the steps of the vehicle-network interactive cooperative scheduling method and system based on multi-agent interaction.

[0052] The system also includes a computer device, which includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs to implement the steps of the vehicle-network interactive cooperative scheduling method and system based on multi-agent intelligence.

[0053] The power grid dispatch center module is used for global dispatch command generation and dynamic optimization. It includes: a global state acquisition unit, which collects real-time electricity market information such as node electricity prices, supply and demand gaps, and renewable energy output, as well as operating status data such as grid frequency deviation and total load curve, forming a high-dimensional state vector; a DDPG model training and decision-making unit, which is used to build a deep deterministic policy gradient algorithm model, including an Actor network and a Critic network. The Actor network outputs regional charging and discharging price ranges and frequency adjustment commands based on the global state vector, and the Critic network evaluates the global Q value of the commands; and a command issuance and feedback receiving unit, which encrypts and issues the generated dispatch commands to the aggregator decision module and receives the load deviation reports uploaded by it as input for global policy iterative updates.

[0054] The aggregator decision-making module is used for collaborative scheduling and strategy aggregation of charging stations. It includes: an information interaction unit, which receives instructions from the power grid dispatch center module and collects policy gradients and execution data uploaded by the user terminal optimization module; a MADDPG model processing unit, which constructs a multi-agent deep deterministic policy gradient model with each charging station as an independent agent. The distributed Actor sub-unit outputs charging and discharging power allocation and bidding prices based on power grid instructions and local states, while the centralized Critic unit inputs the states and joint actions of all charging stations to calculate the global Q value; a federated aggregation and model update unit, which uses a federated averaging algorithm to perform weighted aggregation of user terminal policy gradients, generates a global gradient and injects it into the MADDPG model, and updates the Actor and Critic network parameters; and a load deviation calculation and feedback unit, which calculates the original deviation between the actual load and the planned load of each charging station, and after capacity constraint correction and response delay correction, generates a deviation report and feeds it back to the power grid dispatch center module.

[0055] The user terminal optimization module is used for local charging and discharging behavior optimization, including: a local state acquisition unit, which collects vehicle status such as current battery health status and remaining driving range, as well as user travel demand data such as planned departure time and estimated driving range; a DQN model decision unit, which outputs Q values ​​for discrete charging and discharging periods and power levels based on the strategy parameters provided by the aggregator and the local state vector; an experience pool and gradient generation unit, which stores state-action-reward-next state samples, updates Q network parameters through a time-series difference algorithm, generates policy gradients, and uploads them to the aggregator decision module; and a federated parameter receiving unit, which receives the global gradients issued by the aggregator, fine-tunes the local DQN model parameters, and achieves collaborative optimization with the aggregator's strategy.

[0056] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle-network interactive cooperative scheduling method and system based on multi-agent systems, characterized in that, Includes the following steps: S1. Divide the vehicle-grid interactive collaborative scheduling system into a grid layer, an aggregator layer, and a user layer, and perform state modeling for each layer based on a Markov decision process. S2: The power grid layer collects real-time electricity market information and power grid operation status data, constructs a decision model through a deep deterministic strategy gradient algorithm, generates regional charging and discharging price ranges and frequency adjustment instructions, and sends them to the aggregator layer. S3. The aggregation layer constructs a multi-agent deep deterministic policy gradient model, treating each charging station as an independent agent. Each intelligent agent collects local electric vehicle queue status and electricity price information and receives instructions from the power grid layer; Each intelligent agent generates the charging and discharging power allocation and bid price through a distributed Actor network based on the received instructions and collected information. The aggregation layer evaluates the joint actions of all agents through a centralized Critic network and coordinates policy conflicts between different agents based on the evaluation results. S4. Each terminal in the user layer receives the strategy from the aggregation layer and collects local vehicle status and user travel needs. It generates charging and discharging behavior decisions that meet the battery charging and discharging rate constraints and travel time constraints through the deep Q network algorithm, and uploads them to the aggregation layer. S5. The aggregation layer calculates the deviation between the actual load and the planned load of each charging station based on and updates its own model, and feeds back the deviation information to the power grid layer. S6. The power grid layer adjusts the parameters of the decision model based on the deviation information and updates the global scheduling strategy.

2. The vehicle-to-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The deep deterministic policy gradient algorithm includes an Actor network and a Critic network; The Actor network takes electricity market information and power grid operation status data as input and outputs the regional charging and discharging price range and frequency adjustment command. The Critic network takes electricity market information, power grid operation status data, and action commands output by the Actor network as joint inputs, and outputs a Q value to evaluate the quality of the commands.

3. The vehicle-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The multi-agent deep deterministic policy gradient model includes a distributed Actor network and a centralized Critic network. The distributed Actor network corresponds to each charging station's intelligent agent, which is used to generate charging and discharging power allocation and bidding price based on grid commands and local conditions; The centralized Critic network is used to calculate the global Q-value based on the states and actions of all agents, and to evaluate the overall benefit of joint actions.

4. The vehicle-to-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The multi-agent deep deterministic policy gradient model evaluates agent behavior through a reward function, which includes instruction tracking reward, economic benefit reward, and collaborative reward. The instruction tracking reward is positively correlated with the matching degree between the charging and discharging power of each charging station and the grid instruction; The economic reward is positively correlated with the difference between the bid price and the real-time electricity price, as well as the transaction volume. The collaborative reward is positively correlated with the balance of power distribution among charging stations.

5. The vehicle-to-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The agent combines the state, action, reward and next state generated by each interaction into a sample tuple and stores it in a fixed-size experience replay pool. During model training, the agent randomly samples a small batch of samples from this pool for learning.

6. The vehicle-to-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The vehicle status includes battery health status, battery capacity limit, and remaining driving range; the user's travel needs include planned departure time and estimated driving distance.

7. The vehicle-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The deep Q-network algorithm includes a Q-network and a target Q-network; The Q-network adopts a convolutional neural network or a fully connected network structure, and outputs the Q-value of each candidate action after inputting the state vector; The target Q-network has the same structure as the Q-network, and its parameters are updated by periodically copying the Q-network parameters.

8. The vehicle-to-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 7, characterized in that: The candidate actions include discrete charging and discharging periods and discrete power levels, and satisfy battery charging and discharging rate constraints and travel time constraints.

9. The vehicle-network interactive cooperative scheduling method and system based on multi-agent communication as described in claim 1, characterized in that: The aggregated user policy gradient adopts a federated averaging algorithm. The weight allocation is positively correlated with the historical policy execution accuracy of each terminal. The accuracy calculation formula is the ratio of actual response power to planned response power.

10. A vehicle-network interactive cooperative scheduling method and system based on multi-agent communication as described in claims 1-9, characterized in that, It includes a power grid dispatch center module, an aggregator decision-making module, and a user terminal optimization module; The power grid dispatch center module is used to execute steps S2 and S6; The aggregator decision module is used to execute steps S3 and S5; The user terminal optimization module is used to execute step S4; Each module uses a federated learning framework for encrypted parameter sharing and dynamically optimizes the scheduling strategy through a closed-loop feedback mechanism.

Images