Privacy protection type multi-agent electric vehicle charging scheduling method and system

Abstract

The invention relates to the technical field of charging scheduling, and provides a privacy protection type multi-agent electric vehicle charging scheduling method and system, and the method comprises the steps: enabling each charging pile agent to determine an adjustment ratio through a deep reinforcement learning method of flexible motion-evaluation, performing privacy protection confusion processing on the arrival time and the departure time through a discrete Gaussian noise mechanism; based on the confused arrival time and departure time, a centralized training-decentralized execution multi-agent deep reinforcement learning method is adopted, and a plurality of charging pile agents cooperatively determine the charging energy of the charging piles in each time period; the method comprises the following steps: acquiring photovoltaic generating capacity and fixed energy consumption of a light storage charging station, combining charging energy of a charging pile, and determining charging and discharging capacity of an energy storage system in each time period through a deep reinforcement learning algorithm of an energy storage system agent; and the multi-target requirements of user privacy protection, station profit optimization, load smoothing and the like are met.

Description

Technical Field

[0001] This invention belongs to the field of charging scheduling technology, and in particular relates to a privacy-preserving multi-agent electric vehicle charging scheduling method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the increasing prevalence of photovoltaic (PV) charging stations, efficient electric vehicle (EV) charging reservation and energy dispatch algorithms are crucial for improving EV charging satisfaction while increasing operational revenue. EV users send reservation data (such as arrival / departure times and demand) to PV charging stations via their terminal devices to reserve charging times. Based on this data, PV charging station operators can schedule optimal EV charging times using the station's charging piles and transmit the calculated EV reservation data and charging plans to a PV charging station aggregator. However, this private data transmitted to the aggregator could be exploited by potential attackers, leading to the leakage of EV users' sensitive personal information (i.e., private location information based on EV arrival and departure times) and the energy consumption patterns of the PV charging stations, thus raising serious privacy and security issues.

[0004] Most existing traditional EV charging scheduling algorithms are based on physical models or static rules, which cannot accurately model intermittent photovoltaic (PV) power generation and random EV charging behavior, making it difficult to maintain flexibility and real-time performance in complex dynamic environments. Deep reinforcement learning (DRL), as a method without a clear mathematical model of the system, is more suitable for dealing with changing system states. However, existing DRL and privacy protection research mainly focuses on traditional electric vehicle charging stations, and lacks systematic solutions for the special scenarios of photovoltaic charging stations.

[0005] Currently, no research has developed an EV scheduling framework based on DRL that can simultaneously address the following two tasks: privacy protection of EV charging data and photovoltaic charging station operation data; and maximizing the profits of photovoltaic charging stations. Summary of the Invention

[0006] To address the technical problems mentioned above, this invention provides a privacy-preserving multi-agent electric vehicle charging scheduling method and system. It constructs a three-level deep reinforcement learning scheduling framework, including differential privacy noise injection, multi-agent charging scheduling based on centralized training and distributed execution, and ESS charging and discharging scheduling. This enables refined scheduling of EV charging and ESS charging and discharging processes within photovoltaic charging stations, meeting multiple objectives such as user privacy protection, site profit optimization, and load smoothing.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a privacy-preserving multi-agent electric vehicle charging scheduling method, comprising: The system obtains the arrival and departure times of EV users' reservations. Through a flexible action-evaluation deep reinforcement learning method, each charging pile agent determines the adjustment ratio. The arrival and departure times are obfuscated for privacy protection using a discrete Gaussian noise mechanism to obtain the obfuscated arrival and departure times. Based on the confused arrival and departure times, a multi-agent deep reinforcement learning method with centralized training and distributed execution is adopted, in which multiple charging pile agents work together to determine the charging energy of the charging pile in each time period. By acquiring the photovoltaic power generation and the fixed energy consumption of the photovoltaic-storage charging station, and combining the charging energy of the charging pile, the charging and discharging amount of the energy storage system in each time period is determined through a deep reinforcement learning algorithm of the energy storage system agent.

[0008] Furthermore, the reward function of the flexible action-evaluation deep reinforcement learning method is the difference between the degree of confusion and the adverse cost caused by changes in the range of confusion.

[0009] Furthermore, the privacy-preserving obfuscation processing steps include: Calculate the updated arrival time confusion range based on the adjustment ratio. Confusing the scope with departure time : ; ; ; Where n represents the nth EV connected to the charging pile intelligent agent; This represents the upper limit of the confusion range of the departure time after the update of the (n-1)th EV. and These represent the arrival and departure times reserved by EV users, respectively. Represent a constant. This indicates the total duration of the charging scheduling cycle. This represents the total number of electric vehicles participating in the charging schedule at a certain charging station; Represents the largest integer value greater than x. Represents the smallest integer value less than x; The adjustment ratio representing the range of confusion associated with arrival time; The action of adjusting the ratio of the confusion range of departure time. To reach the upper limit of the time confusion range, The upper limit of the range for obfuscating departure time; Based on the updated arrival time confusion range and departure time confusion range, the confused arrival and departure times are generated using a confusion function based on discrete Gaussians.

[0010] Furthermore, the reward function of the centralized training-decentralized execution multi-agent deep reinforcement learning method is the difference between the charging revenue of the photovoltaic-storage charging station and the unpleasant cost of EV users not meeting their charging needs; The state of the centralized training-distributed execution multi-agent deep reinforcement learning method includes the electricity price during the current scheduling period, the remaining charging capacity of the currently connected EV, the remaining available charging time of the currently connected EV, and the charging speed control parameters.

[0011] Furthermore, the reward function of the deep reinforcement learning algorithm of the energy storage system agent includes the negative operating cost of the photovoltaic charging station and the negative penalty cost of the deviation between the actual net energy consumption of the photovoltaic charging station and the target value. The state of the deep reinforcement learning algorithm of the energy storage system agent includes the current electricity purchase price, the battery energy state of the energy storage system, the net energy consumption of the photovoltaic-storage charging station, and the photovoltaic power generation.

[0012] A second aspect of the present invention provides a privacy-preserving multi-agent electric vehicle charging scheduling system, comprising: The differential privacy module is configured to: obtain the arrival and departure times of EV users' reservations; determine the adjustment ratio for each charging pile agent through a flexible action-evaluation deep reinforcement learning method; and perform privacy-preserving obfuscation on the arrival and departure times through a discrete Gaussian noise mechanism to obtain the obfuscated arrival and departure times. The multi-agent collaborative scheduling module is configured to: based on the confused arrival and departure times, adopt a multi-agent deep reinforcement learning method of centralized training and decentralized execution, and multiple charging pile agents collaboratively determine the charging energy of the charging pile in each time period. The energy storage system scheduling module is configured to: acquire the photovoltaic power generation and the fixed energy consumption of the photovoltaic-energy storage charging station, combine the charging energy of the charging pile, and determine the charging and discharging amount of the energy storage system in each time period through a deep reinforcement learning algorithm of the energy storage system agent.

[0013] Furthermore, the reward function of the flexible action-evaluation deep reinforcement learning method is the difference between the degree of confusion and the adverse cost caused by changes in the range of confusion.

[0014] Furthermore, the privacy-preserving obfuscation processing steps include: Calculate the updated arrival time confusion range based on the adjustment ratio. Confusing the scope with departure time : ; ; ; Where n represents the nth EV connected to the charging pile intelligent agent; This represents the upper limit of the confusion range of the departure time after the update of the (n-1)th EV. and These represent the arrival and departure times reserved by EV users, respectively. Represent a constant. This indicates the total duration of the charging scheduling cycle. This represents the total number of electric vehicles participating in the charging schedule at a certain charging station; Represents the largest integer value greater than x. Represents the smallest integer value less than x; The adjustment ratio representing the range of confusion associated with arrival time; The action of adjusting the ratio of the confusion range of departure time. To reach the upper limit of the time confusion range, The upper limit of the range for obfuscating departure time; Based on the updated arrival time confusion range and departure time confusion range, the confused arrival and departure times are generated using a confusion function based on discrete Gaussians.

[0015] Furthermore, the reward function of the centralized training-decentralized execution multi-agent deep reinforcement learning method is the difference between the charging revenue of the photovoltaic-storage charging station and the unpleasant cost of EV users not meeting their charging needs; The state of the centralized training-distributed execution multi-agent deep reinforcement learning method includes the electricity price during the current scheduling period, the remaining charging capacity of the currently connected EV, the remaining available charging time of the currently connected EV, and the charging speed control parameters.

[0016] Furthermore, the reward function of the deep reinforcement learning algorithm of the energy storage system agent includes the negative operating cost of the photovoltaic charging station and the negative penalty cost of the deviation between the actual net energy consumption of the photovoltaic charging station and the target value. The state of the deep reinforcement learning algorithm of the energy storage system agent includes the current electricity purchase price, the battery energy state of the energy storage system, the net energy consumption of the photovoltaic-storage charging station, and the photovoltaic power generation.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a three-level deep reinforcement learning scheduling framework, including differential privacy noise injection, multi-agent charging scheduling based on centralized training and decentralized execution, and ESS charging and discharging scheduling, to achieve fine-grained scheduling of EV charging and ESS charging and discharging processes within photovoltaic charging stations, meeting multiple objectives such as user privacy protection, site profit optimization, and load smoothing.

[0018] This invention employs differential privacy technology to obfuscate the arrival and departure appointment times of EV users, effectively protecting user privacy while preventing the risk of user privacy data leakage.

[0019] This invention utilizes a multi-agent deep reinforcement learning algorithm based on centralized training-distributed execution (CTDE) to achieve real-time and refined decision-making on the charging power of EVs at each charging pile, thereby maximizing the profits of charging stations while ensuring that users' charging needs are met.

[0020] This invention uses a dedicated ESS scheduling deep reinforcement learning agent to smoothly control the net load of the site, further reducing operating costs and peak shaving and valley filling, and effectively avoiding the risks caused by drastic fluctuations in grid load. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of a privacy-preserving multi-agent electric vehicle charging scheduling method according to Embodiment 1 of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] Example 1 This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method.

[0026] The leakage of privacy data such as EV arrival / departure times and solar charging station energy consumption data will harm the interests of EV users and solar charging station operators.

[0027] Improving the profitability of photovoltaic charging stations can help accelerate their deployment and meet the growing demand for EV charging.

[0028] This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method, which adopts a privacy-preserving multi-agent deep reinforcement learning (DRL) framework to achieve collaborative scheduling of electric vehicle (EV) charging and energy storage system (ESS) within photovoltaic (PV) charging stations.

[0029] This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method that protects the privacy of EV arrival / departure scheduled time data and photovoltaic charging station energy consumption patterns, while simultaneously enabling profitable operation of photovoltaic charging stations by scheduling EV charging.

[0030] This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method. The overall idea is to construct a three-level deep reinforcement learning scheduling framework, including differential privacy noise injection, multi-agent charging scheduling based on centralized training-distributed execution (CTDE), and ESS charging and discharging scheduling, so as to realize fine-grained scheduling of EV charging and ESS charging and discharging processes in photovoltaic charging stations, and meet multiple objectives such as user privacy protection, site profit optimization, and load smoothing.

[0031] This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method, such as... Figure 1 As shown, it includes the following steps: Step 1: Differential privacy noise injection.

[0032] Specifically, the system receives raw charging reservation data submitted by EV users through their terminal devices, including the arrival and departure times of each EV. Then, based on a set privacy budget parameter ε, a discrete Gaussian difference privacy mechanism is used to inject random noise into the arrival and departure times of each EV. First, the standard deviation σ of the discrete Gaussian noise is calculated based on the preset privacy budget parameter ε. This standard deviation σ is then used to generate discrete Gaussian noise, which is added to the original arrival and departure times to obtain a privacy-protected time window. A deep reinforcement learning agent is established. The agent's state space includes the confusion time window range of the current EV and its relative positional relationship with the reservation times of adjacent EVs. The action space is defined as an adjustment factor for the standard deviation σ of the discrete Gaussian noise. A reward function for the reinforcement learning agent is established, including a positive privacy confusion reward (i.e., the time offset after noise injection) and a negative charging window overlap penalty (a larger penalty is given if the injected noise causes the charging windows to overlap). Finally, the Soft Actor-Critic (SAC) algorithm is used to train the agent, continuously iterating and updating the parameters to ultimately output a privacy-protected and non-conflicting reservation time window for each EV.

[0033] In this embodiment, each charging pile agent in the photovoltaic charging station executes the first-level DRL algorithm, using the original arrival time of the nth EV reserved by a certain charging pile agent. With departure time For input; obfuscation range by adjusting arrival time. Confusing the scope with departure time To maximize the arrival and departure times of the nth EV reservation connected to the charging pile intelligent agent ( , The obfuscation is handled by ensuring non-overlapping obfuscation ranges with the (n-1)th and (n+1)th EVs; using discrete Gaussian... Using discrete Gaussian noise The data is obfuscated as discrete values.

[0034] Step 101: For each charging station reserved ,in This represents the total number of electric vehicles participating in the charging schedule at a given charging station. Let n be the number of electric vehicles in the set. The state space of the corresponding charging pile intelligent agent is as follows: ; in, and They represent and Confusing the scope, , ; The degree of confusion range disruption indicates the confusion range caused by the previous (n-1) electric vehicle. Changes leading to The degree of change in the range of confusion is calculated using the following formula: ,in, for The rate of change is used to obfuscate the departure time of the (n-1)th EV. Action value; EV charging reservation time window is [ ].

[0035] Step 102: Define the action space of the charging pile intelligent agent as the action space of the nth charging pile intelligent agent. In the state space, the agent determines the nth vehicle according to the following formula. New scope of confusion: First, define the action of adjusting the ratio: ;in, Indicates the minimum ratio Limits and maximum ratios Limitations on the scope of confusion The action of adjusting the ratio; Indicates the minimum ratio Limits and maximum ratios Limitations on the scope of confusion The adjustment ratio action; an action value greater than (or less than) 1 will lead to an increase (or decrease) in the range of confusion; Then, using and Calculate the new confusion range and The minimum and maximum constraint expressions are as follows: ; ; ; ; in, It is a constant, serving as a safety margin for the non-overlapping confusion range. This indicates the total duration of the charging scheduling cycle. This represents the total number of electric vehicles participating in the charging schedule at a certain charging station; Represents the largest integer value greater than x. Represents the smallest integer value less than x, used to convert continuous time into the corresponding discrete time; Then, the scope of confusion regarding action changes. and It will be used as input, fed into a discrete Gaussian-based system. In the obfuscation function, the obfuscated arrival times are then generated respectively. and departure time .

[0036] Discrete Gaussian Given Assuming the query function , For the input data space, query function For any input data A query can find a set of integers. The definite value here represents the arrival or departure time of the electric vehicle (EV). In differential privacy analysis, the input data space is taken as... Any two input data and As a comparison, among them To and Adjacent input data refers to two sets of data that differ only in the arrival or departure time of a single electric vehicle. Query function. sensitivity Defined as when the input data is... Change to At that time, the maximum change in the query results, i.e., satisfying Based on this, a random algorithm is defined. for: Discrete Gaussian noise ; in, Indicates the sensitivity of the query function. The parameter representing the standard deviation of discrete Gaussian noise. This refers to a privacy budget. Indicates input data Execute query function The deterministic query result obtained, in this invention, is the arrival time or departure time of the electric vehicle (EV) (represented in integer form of discrete time slots). Corresponding to Discrete Gaussian By sending EV time data Add from Gaussian distribution Sampled discrete Gaussian noise To protect this data.

[0037] Under the above mechanism, by reasonably setting noise parameters Randomization algorithm Able to meet -Differential privacy requirements, among which The parameter representing the probability of privacy failure is used to characterize the allowable upper bound of the relaxation of differential privacy constraints under extremely low probability conditions. Its value ranges from... The following upper and lower bound relationships are satisfied: ; in, It's a probability symbol, representing the probability of something happening. Here... Represents the discrete Gaussian distribution Medium sampling After that, satisfied The probability value, This indicates sampling. After that, satisfied The probability value.

[0038] In the first level, a discrete Gaussian is used. Methods to protect EV arrival time and departure time Privacy, specifically, using discrete Gaussian-based... obfuscation function ,Will and In the scope of confusion and The internal parts are confused as and As shown below: ; ; The confusion function is defined as follows: .

[0039] Step 103: Define the agent's reward function as the difference between the degree of confusion (DOE) and the adverse cost caused by changes in the range of confusion. The degree of confusion (DOE) is calculated as the absolute deviation between the confused and unconfused appointment times. ; The sum of the unpleasant costs reflecting the disruption of the confusion range of adjacent EVs is expressed as: ;in, Reflects due to Arrival time caused Changes leading to departure time The extent of disruption to the scope of confusion, ; This reflects the fact that Caused departure time Changes in arrival time The extent of disruption to the scope of confusion, .

[0040] Step 104: The charging pile intelligent agent is trained using a Soft Actor-Critic (SAC) deep reinforcement learning algorithm. Through continuous iterative updates, it ultimately obtains EV reservation obfuscated time window data that meets differential privacy requirements and is mutually non-conflicting. When deployed in production, the privacy-protected arrival and departure reservation data will be used. and It is provided to the second-level module for EV charging scheduling to maximize the revenue of photovoltaic charging stations.

[0041] Step 2: Multi-agent DRL collaborative EV charging scheduling based on CTDE.

[0042] After obtaining the EV arrival and departure time windows after privacy protection in step 1, multiple EV charging station agents are established, each independently responsible for scheduling the charging power of several EVs. The agent's state space includes: the EV arrival and departure time windows after privacy protection by the first module; the electricity price for the current scheduling period; the remaining charging capacity of the currently connected EVs in this period; the arrival time of the EV after privacy protection provided by the first module; the remaining available charging time of the EV from the current moment to its planned departure time; and the control parameters currently used to adjust the charging power. The agent's action space is defined as the EV charging rate ratio for each scheduling period (e.g., every 5 minutes) (a continuous variable with values ​​between 0 and 1), multiplied by the maximum power of the EV charging station to determine the actual charging power of the EV in that period. The agent's reward function includes a charging station profit term, i.e., the difference between electricity sales revenue and electricity purchase cost in each period, and an EV user satisfaction penalty term, i.e., the penalty incurred when the actual charging amount of the EV is lower than the reserved demand. A Centralized Training-Decentralized Execution (CTDE) approach is used to train the agents. The training process is as follows: During the centralized training phase, the state-action-reward interaction data of all agents are stored in a central experience replay pool; each agent independently updates its Actor / Critic network parameters using the SAC algorithm based on the data samples shared in the central experience replay pool; during the actual execution phase, each agent executes its own independent decision-making strategy. After training, the optimal EV charging rate allocation strategy is output to guide the actual charging process.

[0043] Step 201: Construct multiple charging pile intelligent agents within the photovoltaic charging station. All charging pile intelligent agents collaborate to complete EV charging scheduling through the multi-agent deep reinforcement learning (MADRL) method. The goal is to maximize the revenue of the photovoltaic charging station with a scheduling interval of five minutes.

[0044] Step 202: Define the state space of the agent: for the scheduling period Each scheduling period lasts 5 minutes, and a day can be divided into 288 scheduling periods. (Charging piles) p During the period t The state space representation is as follows: ; in, for t Electricity prices corresponding to the time period; It is the remaining charging capacity of the currently connected EV, i.e. t Time-limited charging stations p The remaining charging energy of all EVs at that location is represented as ,in, yes t Time-limited charging stations p The demand for all EV requests at the location, express t Charging stations during the time period p The charging capacity, Indicates charging station p Confusion regarding EV arrival time; Indicates the remaining available charging time for the currently connected EV, i.e. t Charging stations during the time period p Remaining time before all EVs are fully charged; These are charging speed control parameters used to adjust... t Charging stations during the time period p All EV charging speeds.

[0045] Step 203: All charging pile smart agents cooperate to calculate the charging amount for each time period.

[0046] t Time-limited charging stations p The action space is defined as follows: Among them, actions For time period t The EV charging energy distribution coefficient can be determined. t Time-limited charging stations p All EV (EVs) charging energy at the location In this embodiment, each scheduling interval Δt is used as the unit, and the action is the charging power ratio. It can also be equivalently written as the charging energy allocation coefficient within that interval (both are converted through Δt) to calculate the requested EVs energy that satisfies the following formula: ; in, Indicates charging station p Considering maximum charging capacity Minimum charging energy, Charging piles p exist t Available charging energy during the period ; For Heaviside step function, exist When positive (or zero), it equals 1 (or 0); and No additional penalty term is needed in the agent's reward function.

[0047] Step 204, the second-level charging pile intelligent agent in t Reward function for a given time period Revenue from photovoltaic and energy storage charging stations Negative costs associated with completing the required charging energy for EVs The sum is: ; ; ; in, Indicates in t Charging stations during the time period p By selling electricity to EV users The revenue obtained from the photovoltaic-storage charging station; This represents the inconvenience cost incurred by EV users for not meeting their charging needs, i.e. t The requested charging demand of EVs during the time period will be used for the remaining time. When the internal needs are not fully met, at the charging station p The inconvenience and cost of charging EVs; parameters The charging speed of EVs can be controlled by using penalties to regulate action generation.

[0048] In summary, each charging station's intelligent entity p Observe its condition Determine the action set Receive a reward Then transition to the new state. In the CTDE-based MADRL framework, each charging station's intelligent agent... p Data tuples Stored in the central experience replay pool This centralized data collection approach enables each charging station agent to update its neural network using data shared from other agents, thereby fostering a collaborative learning environment and maximizing the profitability of the photovoltaic-storage charging station.

[0049] Step 3: ESS charge and discharge scheduling.

[0050] Based on the EV charging load data output in step 2, combined with the charging station's base load and photovoltaic power generation forecast, the net load of the charging station in each scheduling period is calculated. A dedicated agent for ESS (Energy Storage System) charging and discharging decisions is established. The agent's state space includes: the current net load of the charging station; the current State of Energy (SoE) of the ESS; and the current grid purchase price information. The agent's action space is defined as the continuous power decision of the ESS in the current period: charging (positive power), discharging (negative power), or holding (zero power). The agent's reward function is designed as a combination of the following two: energy cost reward, encouraging charging during off-peak hours and discharging during peak hours to maximize the reduction of electricity purchase costs; and load smoothing reward, encouraging the agent to control net load fluctuations within a small range through ESS charging and discharging actions, thereby further protecting privacy and reducing load impact on the grid side. The agent is trained using the SAC algorithm, and the network parameters are updated iteratively until convergence. The optimal charging and discharging decision strategy for the ESS in each period is output. This ESS decision strategy is then applied to a real-world environment to achieve reduced operating costs and smoothed load patterns.

[0051] In this embodiment, an ESS intelligent agent is constructed in the photovoltaic charging station. The charging power, fixed energy consumption of the photovoltaic-storage charging station and PV power generation of the second level for each time period are scheduled at a five-minute interval. By optimizing the charging and discharging decisions, the operating energy consumption cost of the photovoltaic charging station is minimized, and the privacy of the station's energy consumption mode is further protected.

[0052] Step 301: Define the time period of the ESS charging and discharging decision-making agent. t The state space is: ;in, express t Electricity purchase price for a given period of time; express t The State of Energy (SOE) of the ESS during the -1 time period; express t PV power generation forecast for a given time period; for charging stations Where p is the number of the charging station. It is a set consisting of the numbers of all charging stations, totaling [number missing]. One charging station, t Net energy consumption of photovoltaic-storage charging stations during a given period Represented as: ,in, express t Time-limited charging stations p The amount of electricity used to charge EVs express t Predicted stationary energy consumption of photovoltaic-storage charging stations during specific time periods.

[0053] Step 302: Based on state information, the ESS agent calculates the charge and discharge amounts within its allowed SOE range, thereby reducing the operating energy consumption cost of the photovoltaic-storage charging station. The action space of the ESS agent during time period t is defined as follows: ; Among them, actions The charging coefficient controls the charging amount during time period t. and discharge quantity ; Make the photovoltaic and energy storage charging station during the time period t Actual energy consumption and expected energy consumption Deviation between (for privacy protection) ,in, The target value is cost-effective, and it covers the entire scheduling range. The expected net energy consumption of the photovoltaic-storage charging station is shown below: ; in, It is a time period t The electricity purchase price at that time It is a time period t The estimated net energy consumption of the photovoltaic-storage charging station; in addition... The input data is embedded into the reward function of the third-level DRL agent to minimize the net energy consumption cost of the photovoltaic-storage-charging station, while hiding the energy consumption mode of the photovoltaic-storage-charging station through the charging and discharging of the ESS.

[0054] The specific charge and discharge amounts of the ESS are calculated to ensure they fall within the ESS's SOE range. Energy consumption mode of the built-in photovoltaic charging station: , ; in, , Indicates time period The maximum energy amplitude that the internal ESS allows to participate in charge and discharge regulation is used to limit the adjustable charge and discharge range of the ESS within the current time period; Indicates the maximum capacity of the ESS; Indicates the lower limit of ESS capacity; when When it is the correct time, It equals 1 if it is equal to 1, otherwise it is 0; in the above formula, express ; Set t The allowed charging and discharging range of the ESS during the specified time period, while also taking into account SOE limitations. ESS capacity limit And SOE of integrated PV power generation ( = + ); similar to and The function, and The reward function of the ESS agent does not require any penalty for SOE violation.

[0055] Step 303 t Reward function of ESS agent for time period This is represented as the negative operating cost of photovoltaic charging stations. Actual net energy consumption of photovoltaic charging stations versus target value Negative penalty cost of deviation ,Right now: ; ; ;in, express t Operating costs of net energy consumption of photovoltaic charging stations during specific time periods. Due to ESS charging and discharging, the net energy consumption is reduced from the original value. The revised actual net energy consumption of the photovoltaic charging station is expressed as follows: ; Used for evaluation Deviation The degree, parameters Control the degree of deviation.

[0056] Step 304, the ESS agent at any time t By smoothing the net energy consumption of photovoltaic charging stations to their target value, the operating energy cost of photovoltaic charging stations can be minimized. At the same time, smoothing the net energy consumption of each time period to the target value can hide the real load changes and prevent external entities from making "unauthorized learning" or commercial inferences through energy consumption data, thus protecting the privacy of the energy consumption of photovoltaic charging stations.

[0057] Finally, the proposed three-level DRL model must be implemented in sequence to achieve the following objectives at each level: (1) maximize the confusion of EV reservation data (Level 1); (2) maximize the revenue of the photovoltaic charging station using the EV data confused at Level 1 (Level 2); (3) minimize the operating energy cost of the photovoltaic charging station based on the charging power of each time period summarized at Level 2 (Level 3).

[0058] This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method. By constructing a three-level deep reinforcement learning scheduling framework, it can perform more granular collaborative optimization of the charging process of electric vehicles (EVs) and the charging and discharging of energy storage systems (ESS) in photovoltaic charging stations, thereby achieving efficient scheduling under privacy protection.

[0059] This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling method. It employs differential privacy technology to obfuscate the arrival and departure appointment times of EV users, effectively protecting user privacy while preventing the risk of user data leakage. Secondly, it utilizes a multi-agent deep reinforcement learning algorithm based on centralized training-distributed execution (CTDE) to achieve real-time, refined decision-making on the charging power of each charging pile, thereby maximizing charging station profits while ensuring that user charging needs are met. Furthermore, a dedicated ESS scheduling deep reinforcement learning agent is used to smoothly control the net load of the station, further reducing operating costs and peak shaving, effectively avoiding the risks caused by drastic fluctuations in grid load.

[0060] Example 2 This embodiment provides a privacy-preserving multi-agent electric vehicle charging scheduling system, including a server and a client connected to it.

[0061] The client is used to display the charging power decisions for each EV and the charging and discharging scheduling results of the ESS in real time, for charging station operators to monitor and manage.

[0062] A database is established on the server to store EV reservation information, charging load prediction data, ESS status data, and training and decision-making data for each module.

[0063] The servers include: The differential privacy module is configured to: obtain the arrival and departure times of EV users' reservations; determine the adjustment ratio for each charging pile agent through a flexible action-evaluation deep reinforcement learning method; and perform privacy-preserving obfuscation on the arrival and departure times through a discrete Gaussian noise mechanism to obtain the obfuscated arrival and departure times. The multi-agent collaborative scheduling module is configured to: based on the confused arrival and departure times, adopt a multi-agent deep reinforcement learning method of centralized training and decentralized execution, and multiple charging pile agents collaboratively determine the charging energy of the charging pile in each time period. The energy storage system scheduling module is configured to: acquire the photovoltaic power generation and the fixed energy consumption of the photovoltaic-energy storage charging station, combine the charging energy of the charging pile, and determine the charging and discharging amount of the energy storage system in each time period through a deep reinforcement learning algorithm of the energy storage system agent.

[0064] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A privacy-preserving multi-agent electric vehicle charging scheduling method, characterized in that, include: The system obtains the arrival and departure times of EV users' reservations. Through a flexible action-evaluation deep reinforcement learning method, each charging pile agent determines the adjustment ratio. The arrival and departure times are obfuscated for privacy protection using a discrete Gaussian noise mechanism to obtain the obfuscated arrival and departure times. Based on the confused arrival and departure times, a multi-agent deep reinforcement learning method with centralized training and distributed execution is adopted, in which multiple charging pile agents work together to determine the charging energy of the charging pile in each time period. By acquiring the photovoltaic power generation and the fixed energy consumption of the photovoltaic-storage charging station, and combining the charging energy of the charging pile, the charging and discharging amount of the energy storage system in each time period is determined through a deep reinforcement learning algorithm of the energy storage system agent.

2. The privacy-preserving multi-agent electric vehicle charging scheduling method as described in claim 1, characterized in that, The reward function of the flexible action-evaluation deep reinforcement learning method is the difference between the degree of confusion and the adverse cost caused by changes in the range of confusion.

3. The privacy-preserving multi-agent electric vehicle charging scheduling method as described in claim 1, characterized in that, The privacy protection obfuscation process includes the following steps: Calculate the updated arrival time confusion range based on the adjustment ratio. Confusing the scope with departure time : ; ; ; Where n represents the nth EV connected to the charging pile intelligent agent; This represents the upper limit of the confusion range of the departure time after the update of the (n-1)th EV. and These represent the arrival and departure times reserved by EV users, respectively. Represent a constant. This indicates the total duration of the charging scheduling cycle. This represents the total number of electric vehicles participating in the charging schedule at a certain charging station; This represents the largest integer value greater than x. Represents the smallest integer value less than x; The adjustment ratio representing the range of confusion associated with arrival time; The action of adjusting the ratio of the confusion range of departure time. To reach the upper limit of the time confusion range, The upper limit of the range for obfuscating departure time; Based on the updated arrival time confusion range and departure time confusion range, the confused arrival and departure times are generated using a confusion function based on discrete Gaussians.

4. The privacy-preserving multi-agent electric vehicle charging scheduling method as described in claim 1, characterized in that, The reward function of the centralized training-decentralized execution multi-agent deep reinforcement learning method is the difference between the charging revenue of the photovoltaic-storage charging station and the discomfort cost of EV users not meeting their charging needs. The state of the centralized training-distributed execution multi-agent deep reinforcement learning method includes the electricity price during the current scheduling period, the remaining charging capacity of the currently connected EV, the remaining available charging time of the currently connected EV, and the charging speed control parameters.

5. The privacy-preserving multi-agent electric vehicle charging scheduling method as described in claim 1, characterized in that, The reward function of the deep reinforcement learning algorithm of the energy storage system agent includes the negative operating cost of the photovoltaic charging station and the negative penalty cost of the deviation between the actual net energy consumption of the photovoltaic charging station and the target value. The state of the deep reinforcement learning algorithm of the energy storage system agent includes the current electricity purchase price, the battery energy state of the energy storage system, the net energy consumption of the photovoltaic-storage charging station, and the photovoltaic power generation.

6. A privacy-preserving multi-agent electric vehicle charging scheduling system, characterized in that, include: The differential privacy module is configured to: obtain the arrival and departure times of EV users' reservations; determine the adjustment ratio for each charging pile agent through a flexible action-evaluation deep reinforcement learning method; and perform privacy-preserving obfuscation on the arrival and departure times through a discrete Gaussian noise mechanism to obtain the obfuscated arrival and departure times. The multi-agent collaborative scheduling module is configured to: based on the confused arrival and departure times, adopt a multi-agent deep reinforcement learning method of centralized training and decentralized execution, and multiple charging pile agents collaboratively determine the charging energy of the charging pile in each time period. The energy storage system scheduling module is configured to: acquire the photovoltaic power generation and the fixed energy consumption of the photovoltaic-energy storage charging station, combine the charging energy of the charging pile, and determine the charging and discharging amount of the energy storage system in each time period through a deep reinforcement learning algorithm of the energy storage system agent.

7. A privacy-preserving multi-agent electric vehicle charging scheduling system as described in claim 6, characterized in that, The reward function of the flexible action-evaluation deep reinforcement learning method is the difference between the degree of confusion and the adverse cost caused by changes in the range of confusion.

8. A privacy-preserving multi-agent electric vehicle charging scheduling system as described in claim 6, characterized in that, The privacy protection obfuscation process includes the following steps: Calculate the updated arrival time confusion range based on the adjustment ratio. Confusing the scope with departure time : ; ; ; Where n represents the nth EV connected to the charging pile intelligent agent; This represents the upper limit of the confusion range of the departure time after the update of the (n-1)th EV. and These represent the arrival and departure times reserved by EV users, respectively. Represent a constant. This indicates the total duration of the charging scheduling cycle. This represents the total number of electric vehicles participating in the charging schedule at a certain charging station; This represents the largest integer value greater than x. Represents the smallest integer value less than x; The adjustment ratio representing the range of confusion associated with arrival time; The action of adjusting the ratio of the confusion range of departure time. To reach the upper limit of the time confusion range, The upper limit of the range for obfuscating departure time; Based on the updated arrival time confusion range and departure time confusion range, the confused arrival and departure times are generated using a confusion function based on discrete Gaussians.

9. A privacy-preserving multi-agent electric vehicle charging scheduling system as described in claim 6, characterized in that, The reward function of the centralized training-decentralized execution multi-agent deep reinforcement learning method is the difference between the charging revenue of the photovoltaic-storage charging station and the discomfort cost of EV users not meeting their charging needs. The state of the centralized training-distributed execution multi-agent deep reinforcement learning method includes the electricity price during the current scheduling period, the remaining charging capacity of the currently connected EV, the remaining available charging time of the currently connected EV, and the charging speed control parameters.

10. A privacy-preserving multi-agent electric vehicle charging scheduling system as described in claim 6, characterized in that, The reward function of the deep reinforcement learning algorithm of the energy storage system agent includes the negative operating cost of the photovoltaic charging station and the negative penalty cost of the deviation between the actual net energy consumption of the photovoltaic charging station and the target value. The state of the deep reinforcement learning algorithm of the energy storage system agent includes the current electricity purchase price, the battery energy state of the energy storage system, the net energy consumption of the photovoltaic-storage charging station, and the photovoltaic power generation.

Images