Region V2G charging pile charging and discharging regulation and control method and device based on reinforcement learning

Abstract

The invention relates to the technical field of power system load management and control, in particular to a transformer area V2G charging pile charging and discharging regulation and control method and device based on reinforcement learning. Specifically, the method comprises the following steps: grouping load data to obtain a plurality of groups of to-be-analyzed load data; carrying out feature analysis on the plurality of groups of to-be-analyzed load data through peak-valley period division to obtain peak power consumption periods and valley power consumption periods of each group; then, an optimal scheduling model of the V2G charging piles is constructed; and obtaining an optimal charging and discharging strategy of each group of V2G charging piles through a reinforcement learning algorithm and an optimal scheduling model. Through the method provided by the invention, strategy support can be provided for charging and discharging scheduling of the V2G charging pile in the transformer area, the peak-valley difference of the transformer in the transformer area is reduced, the electricity charging cost of the electric vehicle is reduced, and the method has relatively high practicability and operability.

Description

Technical Field

[0001] The present invention relates to the technical field of power system load control, and in particular to a method and device for controlling charging and discharging of a V2G charging pile in an area based on reinforcement learning. Background Art

[0002] With the in-depth implementation of the energy revolution strategy and the explosive growth of distributed power generation such as photovoltaic and wind power, the balance of power supply and demand in the power grid faces new challenges: first, the contradiction of seasonal peak load is prominent; second, the peak load regulation pressure of the power grid increases year by year. Since the peak of winter in 2019, my country's power supply has once again become relatively tight, and some areas even need to use power outages to alleviate the tight supply situation. Under the new situation, the implementation of traditional administrative measures for orderly power consumption is becoming increasingly difficult. Therefore, it is necessary to innovate systems, mechanisms and technologies, tap into the adjustable load resources on the demand side, and manage and utilize these adjustable load resources.

[0003] Electric vehicles are a type of transportation vehicle powered by power batteries. According to survey data, the total battery capacity of 5,000 electric vehicles is equivalent to a large energy storage power station. Therefore, the rational use of the energy storage characteristics of electric vehicles can realize charging compensation for the power grid during peak hours, reducing the power supply pressure of the power grid; increasing the power load during the off-peak hours of the power grid, improving the line load rate, and achieving the effect of peak shaving and valley filling, energy saving and emission reduction. However, to achieve energy interaction between electric vehicles and the power grid, the key lies in how to achieve charging and discharging regulation through the V2G charging pile. Therefore, how to formulate a scientific and reasonable V2G charging pile charging and discharging scheduling strategy to achieve energy interaction with the power grid is a current hot issue.

[0004] From the current research progress, the existing V2G charging pile control technology mainly targets the interests of the grid side, with the ultimate goal of smoothing the grid load fluctuations, without considering the actual adjustable capacity of users and their willingness to participate. For example, Liu Boyan proposed a research on the economic dispatch strategy of microgrids with battery swap stations based on demand response, which only established a multi-objective double-layer charging and discharging real-time dispatch model for coordination between battery swap stations and the grid, and reduced the peak-to-valley difference of the grid through coordinated dispatching, without considering the actual situation of electric vehicle users.

[0005] Therefore, a method and equipment for controlling the charging and discharging of V2G charging piles in a substation is needed, which comprehensively considers both electric vehicle users and the power grid. Summary of the invention

[0006] The purpose of the present invention is to overcome the problem that the existing V2G charging pile control technology in the prior art mainly focuses on the interests of the power grid side and lacks consideration of electric vehicle users, and to provide a V2G charging pile charging and discharging control method based on reinforcement learning.

[0007] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0008] A method for controlling charging and discharging of V2G charging piles in a substation based on reinforcement learning comprises the following steps:

[0009] S1: Load data grouping: input the load data to be analyzed, and divide it into several groups according to the preset grouping rules, and then normalize each group of load data to be analyzed to obtain several groups of normalized load data;

[0010] The load data to be analyzed includes the daily load data of the area and the V2G charging pile of the preset time length;

[0011] S2: Peak-valley time division: Perform feature analysis on several groups of normalized load data to obtain the peak power consumption period and valley power consumption period of each group of normalized load data;

[0012] S3: Construction of control model: Taking reducing the load fluctuation of the power grid in the substation area and reducing the charging cost of electric vehicle users as the optimization objective function; taking the charging and discharging power constraints of V2G charging piles, the battery capacity constraints of electric vehicles, and the charging period as constraints, construct the optimization scheduling model of V2G charging piles;

[0013] S4: Reinforcement learning optimization solution: Through the reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, the optimal charging and discharging strategy of each group of V2G charging piles is solved and output.

[0014] As a preferred solution of the present invention, the preset grouping rule in S1 is:

[0015] The substation load data is divided into 8 groups according to spring working days, spring non-working days, summer working days, summer non-working days, autumn working days, autumn non-working days, winter working days, and winter non-working days.

[0016] As a preferred embodiment of the present invention, S1 comprises the following steps:

[0017] S11: Inputting load data to be analyzed for a preset time length;

[0018] S12: grouping the load data to be analyzed according to season and whether it is a working day, and obtaining eight groups of data: spring working day, spring non-working day, summer working day, summer non-working day, autumn working day, autumn non-working day, winter working day, and winter non-working day;

[0019] S13: Calculate the baseline load data of the area after removing the V2G charging pile load. The calculation formula is:

[0020] P L0i (t) = P L1i (t)-Pvi (t)(t=1,2,…,8*M),

[0021] Where P L0i (t) represents the baseline load of the area at time t on the i-th day, P L1i (t) represents the total load of the area at time t on the i-th day, P vi (t) represents the V2G charging pile load data at time t on the i-th day, and M is the average daily sampling number;

[0022] S14: normalize each group of area baseline load data to obtain several groups of normalized load data, and the calculation formula is:

[0023] P' L0i (t) = [P L0i (t)-min(P Lj )] / [max(P Lj )-min(P Lj )](t=1,2,…,8*M),

[0024] Where P' L0i (t) represents the normalized baseline load of the area at time t on the i-th day; max(P Lj ) represents the maximum load value of the area in the jth group of data; min(P Lj ) represents the minimum load value of the substation in the jth group of data.

[0025] As a preferred embodiment of the present invention, S2 comprises the following steps:

[0026] S21: averaging the eight groups of load data to be analyzed to obtain the daily average load at 8*M moments; the specific calculation formula is as follows:

[0027]

[0028] Where P LAj (t) represents the average load of the jth group of data at the tth moment in the substation; N represents the number of days included in the jth group of data;

[0029] S22: Divide the peak power consumption period of the substation according to the following rules:

[0030] Rule 1: If the average load of the load data to be analyzed is greater than 0.8 at time t, then time t is the peak time;

[0031] Rule 2: If the average load of a non-peak hour adjacent to a peak hour is greater than 0.7, then the hour is a sub-peak hour;

[0032] Rule 3: Divide the connected peak and sub-peak hours into peak electricity consumption periods;

[0033] S23: Divide the off-peak power consumption period of the substation area according to the following rules:

[0034] Rule 1: If the average load of the load data to be analyzed is less than 0.3 at time t, then time t is the valley moment;

[0035] Rule 2: If the average load of a non-valley moment adjacent to a valley moment is less than 0.4, then this moment is the second valley moment;

[0036] Rule 3: Divide the connected off-peak and second-off-peak times into off-peak electricity consumption periods.

[0037] As a preferred embodiment of the present invention, S3 comprises the following steps:

[0038] S31: Optimization objective function construction:

[0039] The expression of the optimization objective function is:

[0040] obj0=min{obj1+obj2},

[0041] Optimization target 1-obj1:

[0042] obj1=min{P max -P min},

[0043] Where P max Indicates the maximum value of the daily load data of the substation, P min Indicates the minimum value of the daily load data of the substation;

[0044] Optimization target 2-obj2:

[0045] obj2=min{Q I C I -Q O C O},

[0046] In the formula, Q I Indicates the charging power of the electric vehicle, Q O Indicates the discharge capacity of the electric vehicle, C I Indicates the charging electricity cost per unit of electricity, C O Indicates the discharge capacity per unit of electricity;

[0047] S32: Constraint Construction:

[0048] Constraint 1: Charge and discharge power constraints of V2G charging piles:

[0049] P omax ≤Pc(t)≤P Imax ,

[0050] Where Pc(t) represents the charge and discharge power at time t, Pc(t)>0 represents charging, and Pc(t)<0 represents discharging; P omax Indicates the maximum discharge power; P Imax Indicates the maximum charging power;

[0051] Constraint 2: Electric vehicle charging capacity constraint:

[0052] Q I -Q o ≥Q min ,

[0053] In the formula, Q min Refers to the user's minimum charging power requirement;

[0054] Constraint 3: Charge and discharge period constraints:

[0055] V2G charging piles only discharge during peak hours of the power grid and give priority to charging during off-peak hours of the power grid.

[0056] As a preferred embodiment of the present invention, S4 comprises the following steps:

[0057] S41: Initialization:

[0058] Construct a decision table T with M rows and 9 columns, take the average load of each group of V2G charging piles as the initial value, construct the initialization charge and discharge curve of the V2G charging piles; and initialize the charging electricity price and cycle parameters at each time;

[0059] The data in the i-th row of the decision table T represents the decision value at the i-th moment of the day, M is the average number of daily sampling times, and the 9 columns correspond to the output actions of 100% power charging, 75% power charging, 50% power charging, 25% power charging, no charging and discharging, 25% power discharging, 50% power discharging, 75% power discharging, and 100% power discharging, respectively; the initial values ​​of the data in the decision table T are all 0; initializing the loop parameters includes initializing the count variable i=1, the accumulated reward value Ra=0, the number of loop iterations Loop=1, and setting the maximum number of loops;

[0060] S42: Action selection:

[0061] Select actions according to preset probabilities:

[0062] Set probability: take the maximum value of the data in the i-th row of the decision table T as the output action of the V2G charging pile at the i-th moment; when the data in the i-th row of the decision table T are all 0, randomly select an output action;

[0063] 1-Set probability: randomly select an output action;

[0064] S43: Reward value calculation:

[0065] According to the optimization objective function, the target value of the V2G charging pile after regulation is calculated, and then the target value before regulation is subtracted from the target value after regulation to obtain the reward value R i ;

[0066] If the constraint condition is not satisfied after taking the action, the reward value R i = -10;

[0067] S44: Iterative Update:

[0068] Updating the decision table T by using a Q-learning algorithm;

[0069] S45: Loop iteration judgment:

[0070] Number of loop iterations Loop = Loop + 1;

[0071] If the Loop reaches the set maximum number of cycles, it goes to S46;

[0072] If Loop is less than the maximum number of cycles, the charging and discharging curve of the V2G charging pile is initialized, and i=1, the cumulative reward value R a =0, the number of loop iterations Loop=1, and then enter S42;

[0073] S46: Control result output:

[0074] According to the decision table T, the optimal control strategy for each moment under different groups is output.

[0075] As a preferred embodiment of the present invention, the S44 comprises the following steps:

[0076] S441: Update the data in row i of decision table T according to the following formula:

[0077] T(i,a)=R i +0.9×max(T(i+1)),

[0078] In the formula, T(i,a) represents the data in the i-th row and a-th column of the decision table, and max(T(i+1)) represents the maximum value in the i+1-th row of the decision table T;

[0079] S442: i=i+1,R a =R a +R i ; R a is the accumulated reward value;

[0080] If i<96, then go to S42; if i=96, then go to S443;

[0081] S443: Update the decision table according to the following formula:

[0082] T(k,a k )=T(k,a k )+R a / (8*M)(k=1,2,...,8*M-1),

[0083] In the formula, a k Represents the action taken at the kth moment in the iteration process.

[0084] As a preferred solution of the present invention, the optimal control strategy at the i-th moment is the output action corresponding to the column number where the maximum value of the i-th row data in the decision table T is located.

[0085] A V2G charging pile charging and discharging control device based on reinforcement learning, comprising at least one processor and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] The present invention obtains several groups of load data to be analyzed by grouping load data; then, by dividing the peak and valley time periods, the characteristics of several groups of load data to be analyzed are analyzed respectively to obtain the peak power consumption period and the valley power consumption period of each group; then, the optimization scheduling model of V2G charging piles is constructed; and the optimal charging and discharging strategy of V2G charging piles of each group is obtained by reinforcement learning algorithm and optimization scheduling model. The method provided by the present invention can provide strategic support for the charging and discharging scheduling of V2G charging piles in the substation area, reduce the peak-valley difference of transformers in the substation area, and reduce the charging cost of electric vehicles, which has strong practicality and operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A schematic diagram of a flow chart of a method for controlling charging and discharging of a V2G charging pile in a substation based on reinforcement learning according to Embodiment 1 of the present invention;

[0089] Figure 2 This is a schematic diagram of the peak-valley time period division result in a method for controlling charging and discharging of a V2G charging pile in a substation based on reinforcement learning according to Embodiment 2 of the present invention;

[0090] Figure 3 A schematic diagram of V2G optimization scheduling in a method for controlling charging and discharging of V2G charging piles in a substation based on reinforcement learning according to Embodiment 2 of the present invention;

[0091] Figure 4 This is a structural schematic diagram of a reinforcement learning-based charging and discharging control device for a V2G charging pile in a substation described in Example 3 of the present invention, which utilizes a reinforcement learning-based charging and discharging control method for a V2G charging pile in a substation described in Example 1. DETAILED DESCRIPTION

[0092] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0093] Example 1

[0094] like Figure 1 As shown, a method for controlling charging and discharging of a V2G charging pile in a station area based on reinforcement learning includes the following steps:

[0095] S1: Load data grouping: input the load data to be analyzed, and divide it into several groups according to the preset grouping rules, and then normalize each group of load data to be analyzed to obtain several groups of normalized load data.

[0096] The load data to be analyzed includes daily load data of the area and the V2G charging pile for a preset time length.

[0097] S2: Peak and valley time division: Perform feature analysis on several groups of normalized load data respectively to obtain the peak power consumption period and valley power consumption period of each group of normalized load data.

[0098] S3: Construction of control model: Taking reducing the load fluctuation of the power grid in the substation area and reducing the charging cost of electric vehicle users as the optimization objective function; taking the charging and discharging power constraints of V2G charging piles, the battery capacity constraints of electric vehicles, and the charging period as constraints, construct an optimal scheduling model for V2G charging piles.

[0099] S4: Reinforcement learning optimization solution: Through the reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, the optimal charging and discharging strategy of each group of V2G charging piles is solved and output.

[0100] Example 2

[0101] This embodiment is a specific implementation of the method for controlling charging and discharging of a V2G charging pile in a substation based on reinforcement learning described in Example 1, and includes the following steps:

[0102] S1: Load data grouping: input the load data to be analyzed, and divide it into several groups according to the preset grouping rules, and then normalize each group of load data to be analyzed to obtain several groups of normalized load data.

[0103] S11: Inputting load data to be analyzed for a preset time length; the load data to be analyzed includes daily load data of the area and the V2G charging pile for a preset time length.

[0104] S12: The load data to be analyzed are grouped according to season and whether it is a working day, and eight groups of data are obtained: spring working days, spring non-working days, summer working days, summer non-working days, autumn working days, autumn non-working days, winter working days, and winter non-working days.

[0105] S13: Calculate the baseline load data of the area after removing the V2G charging pile load. The calculation formula is:

[0106] P L0i (t) = P L1i (t)-P vi (t)(t=1,2,…,8*M),

[0107] Where P L0i (t) represents the baseline load of the area at time t on the i-th day, P L1i (t) represents the total load of the area at time t on the i-th day, P vi (t) represents the V2G charging pile load data at time t on the i-th day, 8 represents 8 groups of data, and M is the average daily sampling times. For example, if sampling is performed once per hour, then M=24, and if sampling is performed 4 times per hour, then M=96.

[0108] S14: normalize each group of area baseline load data to obtain several groups of normalized load data, and the calculation formula is:

[0109] P' L0i (t) = [P L0i (t)-min(P Lj )] / [max(P Lj )-min(P Lj )](t=1,2,…,8*M),

[0110] Where P' L0i (t) represents the normalized baseline load of the area at time t on the i-th day; max(P Lj ) represents the maximum load value of the area in the jth group of data; min(P Lj ) represents the minimum load value of the substation in the jth group of data.

[0111] S2: Peak and valley time division: Perform feature analysis on several groups of normalized load data respectively to obtain the peak power consumption period and valley power consumption period of each group of normalized load data.

[0112] S21: Calculate the average of the 8 groups of load data to be analyzed to obtain the average load at 8*M moments. The specific calculation formula is as follows:

[0113]

[0114] Where P LAj (t) represents the average load of the j-th group of data at the t-th time; N represents the number of days included in the j-th group of data.

[0115] S22: Divide the peak power consumption period of the substation according to the following rules:

[0116] Rule 1: If the average load of the load data to be analyzed is greater than 0.8 at time t, then time t is the peak time;

[0117] Rule 2: If the average load of a non-peak hour adjacent to a peak hour is greater than 0.7, then the hour is a sub-peak hour;

[0118] Rule 3: Classify the consecutive peak and sub-peak hours as peak electricity consumption periods.

[0119] S23: Divide the off-peak power consumption period of the substation area according to the following rules:

[0120] Rule 1: If the average load of the load data to be analyzed is less than 0.3 at time t, then time t is the valley moment;

[0121] Rule 2: If the average load of a non-valley moment adjacent to a valley moment is less than 0.4, then this moment is the second valley moment;

[0122] Rule 3: Divide the connected off-peak and second-off-peak times into off-peak electricity consumption periods.

[0123] For example, Figure 2 The load curve shown is divided into peak and valley periods, and the results are peak periods of 0:15-6:00, 16:30-23:00 ( Figure 2 The black line area), the low period is 9:30-15:30 ( Figure 2 medium grey area).

[0124] S3: Construction of control model: Taking reducing the load fluctuation of the power grid in the substation area and reducing the charging cost of electric vehicle users as the optimization objective function; taking the charging and discharging power constraints of V2G charging piles, the battery capacity constraints of electric vehicles, and the charging period as constraints, construct an optimal scheduling model for V2G charging piles.

[0125] S31: Optimization objective function construction:

[0126] Optimization goal 1:

[0127] For the substation, its control goal is to reduce load fluctuations, that is, to reduce the daily peak-to-valley difference. Therefore, the optimization goal 1 is to reduce the peak-to-valley difference of the substation load obj1:

[0128] obj1=min{P max -P min},

[0129] Where P max Indicates the maximum value of the daily load data of the substation, P min Indicates the minimum value of the daily load data of the substation.

[0130] Optimization goal 2:

[0131] For electric vehicle users, one of the goals of participating in vehicle-grid interaction is to minimize the charging cost, so optimization objective 2 is to reduce the charging cost obj2:

[0132] obj2=min{Q I C I -Q O C O},

[0133] In the formula, Q I Indicates the charging power of the electric vehicle, Q O Indicates the discharge capacity of the electric vehicle, C I Indicates the charging electricity cost per unit of electricity, C O Indicates the discharge capacity per unit of electricity.

[0134] In summary, the charging and discharging optimization control objective function obj0 of the V2G charging pile is as follows:

[0135] obj0=min{obj1+obj2},

[0136] S32: Constraint Construction:

[0137] Constraint 1: Charge and discharge power constraints of V2G charging piles:

[0138] P omax ≤Pc(t)≤P Imax ,

[0139] Where Pc(t) represents the charge and discharge power at time t, Pc(t)>0 represents charging, and Pc(t)<0 represents discharging; P omax Indicates the maximum discharge power; P Imax Indicates the maximum charging power.

[0140] Constraint 2: In order to ensure the normal use of electric vehicle users, the net power charged into the V2G should be guaranteed to reach the target value after charging and discharging. Furthermore, the electric vehicle charging capacity constraint is:

[0141] Q I -Q o ≥Qmin ,

[0142] In the formula, Q min Refers to the user's minimum charging power requirement.

[0143] Constraint 3: Charge and discharge period constraints:

[0144] V2G charging piles only discharge during peak hours of the power grid and give priority to charging during off-peak hours of the power grid.

[0145] S4: Reinforcement learning optimization solution: Through the reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, the optimal charging and discharging strategy of each group of V2G charging piles is solved and output.

[0146] S41: Initialization:

[0147] Construct a decision table T with M rows and 9 columns, take the average load of each group of V2G charging piles as the initial value, construct the initialization charge and discharge curve of the V2G charging pile; and initialize the charging electricity price and cycle parameters at each time.

[0148] Among them, the data in the i-th row in the decision table T represents the decision value at the i-th moment of the day, and the 9 columns correspond to the output actions of 100% power charging, 75% power charging, 50% power charging, 25% power charging, no charging and discharging, 25% power discharging, 50% power discharging, 75% power discharging, and 100% power discharging respectively; the initial values ​​of the data in the decision table T are all 0; initializing the loop parameters includes initializing the counting variable i=1, the accumulated reward value Ra=0, the number of loop iterations Loop=1, and setting the maximum number of loops.

[0149] S42: Action selection:

[0150] Select actions according to preset probabilities:

[0151] Set the probability (set to 80% in this embodiment): take the maximum value of the i-th row of data in the decision table T as the output action of the V2G charging pile at the i-th moment; when the i-th row of data in the decision table T are all 0, randomly select an output action.

[0152] 1-Set probability (20% in this embodiment): randomly select an output action.

[0153] S43: Reward value calculation:

[0154] According to the optimization objective function, the target value of the V2G charging pile after regulation is calculated, and then the target value before regulation is subtracted from the target value after regulation to obtain the reward value R i .

[0155] If the constraint condition is not satisfied after taking the action, the reward value R i =-10.

[0156] S44: Iterative Update:

[0157] The decision table T is updated by using the Q-learning algorithm.

[0158] S441: Update the data in row i of decision table T according to the following formula:

[0159] T(i,a)=R i +0.9×max(T(i+1)),

[0160] In the formula, T(i,a) represents the data in the i-th row and a-th column of the decision table, and max(T(i+1)) represents the maximum value in the i+1-th row of the decision table T;

[0161] S442: i=i+1,R a =R a +R i ; R a is the accumulated reward value;

[0162] If i<96, then go to S42; if i=96, then go to S443;

[0163] S443: Update the decision table according to the following formula:

[0164] T(k,a k )=T(k,a k )+R a / (8*M)(k=1,2,...,8*M-1),

[0165] Where a k Represents the action taken at the kth moment in the iteration process.

[0166] S45: Loop iteration judgment:

[0167] The number of loop iterations Loop=Loop+1.

[0168] If the Loop reaches the set maximum number of cycles, it goes to S46;

[0169] If Loop is less than the maximum number of cycles, the charging and discharging curve of the V2G charging pile is initialized, and i=1, the cumulative reward value R a =0, the number of loop iterations Loop=1, and then enter S42.

[0170] S46: Control result output:

[0171] According to the decision table T, the optimal control strategy for each moment under different groups is output.

[0172] Furthermore, the optimal control strategy at the i-th moment is the output action corresponding to the column number where the maximum value of the i-th row data in the decision table T is located.

[0173] For example, after adjusting the charging load of V2G in the substation, the results are as follows: Figure 3 In the figure, positive charging and discharging power indicates charging, and negative power indicates discharging.

[0174] Example 3

[0175] like Figure 4 As shown, a V2G charging pile charging and discharging control device based on reinforcement learning in a substation includes at least one processor, a memory connected to the at least one processor in communication, and at least one input and output interface connected to the at least one processor in communication; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the V2G charging pile charging and discharging control method based on reinforcement learning in a substation described in the above embodiment. The input and output interface may include a display, a keyboard, a mouse, and a USB interface for inputting and outputting data.

[0176] Those skilled in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc. Various media that can store program codes.

[0177] When the above-mentioned integrated unit of the present invention is implemented in the form of a software functional unit and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for controlling charging and discharging of V2G charging piles in a station area based on reinforcement learning, characterized in that: The steps include: S1: Load data grouping: input the load data to be analyzed, and divide it into several groups according to the preset grouping rules, and then normalize each group of load data to be analyzed to obtain several groups of normalized load data; The load data to be analyzed includes the daily load data of the area and the V2G charging pile of the preset time length; S2: Peak-valley time division: Perform feature analysis on several groups of normalized load data to obtain the peak power consumption period and valley power consumption period of each group of normalized load data; S3: Construction of control model: Taking reducing the load fluctuation of the power grid in the substation area and reducing the charging cost of electric vehicle users as the optimization objective function; taking the charging and discharging power constraints of V2G charging piles, the battery capacity constraints of electric vehicles, and the charging period as constraints, construct the optimization scheduling model of V2G charging piles; S4: Reinforcement learning optimization solution: Through the reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, the optimal charging and discharging strategy of each group of V2G charging piles is solved and output.

2. According to a reinforcement learning-based V2G charging pile charging and discharging control method in a substation area, the method is characterized in that: The preset grouping rule in S1 is: The substation load data is divided into 8 groups according to spring working days, spring non-working days, summer working days, summer non-working days, autumn working days, autumn non-working days, winter working days, and winter non-working days.

3. According to a reinforcement learning-based V2G charging pile charging and discharging control method in a substation area according to claim 2, it is characterized in that: The S1 comprises the following steps: S11: Inputting load data to be analyzed for a preset time length; S12: grouping the load data to be analyzed according to season and whether it is a working day, and obtaining eight groups of data: spring working day, spring non-working day, summer working day, summer non-working day, autumn working day, autumn non-working day, winter working day, and winter non-working day; S13: Calculate the baseline load data of the area after removing the V2G charging pile load. The calculation formula is: P L0i (t)=P L1i (t)-P vi (t)(t=1,2,…,8*M), Where P L0i (t) represents the baseline load of the area at time t on the i-th day, P L1i (t) represents the total load of the area at time t on the i-th day, P vi (t) represents the V2G charging pile load data at time t on the i-th day, and M is the average daily sampling number; S14: normalize each group of area baseline load data to obtain several groups of normalized load data, and the calculation formula is: P' L0i (t)=[P L0i (t)-min(P Lj )] / [max(P Lj )-min(P Lj )](t=1,2,…,8*M), Where P' L0i (t) represents the normalized baseline load of the area at time t on the i-th day; max(P Lj ) represents the maximum load value of the area in the jth group of data; min(P Lj ) represents the minimum load value of the substation in the jth group of data.

4. According to a reinforcement learning-based V2G charging pile charging and discharging control method in a substation area according to claim 3, it is characterized in that: The S2 comprises the following steps: S21: averaging the eight groups of load data to be analyzed to obtain the daily average load at 8*M moments; the specific calculation formula is as follows: Where P LAj (t) represents the average load of the jth group of data at the tth moment in the substation; N represents the number of days included in the jth group of data; S22: Divide the peak power consumption period of the substation according to the following rules: Rule 1: If the average load of the load data to be analyzed is greater than 0.8 at time t, then time t is the peak time; Rule 2: If the average load of a non-peak hour adjacent to a peak hour is greater than 0.7, then the hour is a sub-peak hour; Rule 3: Divide the connected peak and sub-peak hours into peak electricity consumption periods; S23: Divide the off-peak power consumption period of the substation area according to the following rules: Rule 1: If the average load of the load data to be analyzed is less than 0.3 at time t, then time t is the valley moment; Rule 2: If the average load of a non-valley moment adjacent to a valley moment is less than 0.4, then this moment is the second valley moment; Rule 3: Divide the connected off-peak and second-off-peak times into off-peak electricity consumption periods.

5. According to a reinforcement learning-based V2G charging pile charging and discharging control method in a substation area, the method is characterized in that: The S3 comprises the following steps: S31: Optimization objective function construction: The expression of the optimization objective function is: obj0=min{obj1+obj2}, Optimization target 1-obj1: obj1=min{P max -P min }, Where P max Indicates the maximum value of the daily load data of the substation, P min Indicates the minimum value of the daily load data of the substation; Optimization target 2-obj2: obj2=min{Q I C I -Q O C O }, In the formula, Q I Indicates the charging power of the electric vehicle, Q O Indicates the discharge capacity of the electric vehicle, C I Indicates the charging electricity cost per unit of electricity, C O Indicates the discharge capacity per unit of electricity; S32: Constraint Construction: Constraint 1: Charge and discharge power constraints of V2G charging piles: P omax ≤Pc(t)≤P Imax , Where Pc(t) represents the charge and discharge power at time t, Pc(t)>0 represents charging, and Pc(t)<0 represents discharging; P omax Indicates the maximum discharge power; P Imax Indicates the maximum charging power; Constraint 2: Electric vehicle charging capacity constraint: Q I -Q o ≥Q min , In the formula, Q min Refers to the user's minimum charging power requirement; Constraint 3: Charge and discharge period constraints: V2G charging piles only discharge during peak hours of the power grid and give priority to charging during off-peak hours of the power grid.

6. The method for controlling charging and discharging of a V2G charging pile in a substation based on reinforcement learning according to claim 1, characterized in that: The S4 comprises the following steps: S41: Initialization: Construct a decision table T with M rows and 9 columns, take the average load of each group of V2G charging piles as the initial value, construct the initialization charge and discharge curve of the V2G charging piles; and initialize the charging electricity price and cycle parameters at each time; The data in the i-th row of the decision table T represents the decision value at the i-th moment of the day, M is the average number of daily sampling times, and the 9 columns correspond to the output actions of 100% power charging, 75% power charging, 50% power charging, 25% power charging, no charging and discharging, 25% power discharging, 50% power discharging, 75% power discharging, and 100% power discharging, respectively; the initial values ​​of the data in the decision table T are all 0; initializing the loop parameters includes initializing the count variable i=1, the accumulated reward value Ra=0, the number of loop iterations Loop=1, and setting the maximum number of loops; S42: Action selection: Select actions according to preset probabilities: Set probability: take the maximum value of the data in the i-th row of the decision table T as the output action of the V2G charging pile at the i-th moment; when the data in the i-th row of the decision table T are all 0, randomly select an output action; 1-Set probability: randomly select an output action; S43: Reward value calculation: According to the optimization objective function, the target value of the V2G charging pile after regulation is calculated, and then the target value before regulation is subtracted from the target value after regulation to obtain the reward value R i ; If the constraint condition is not satisfied after taking the action, the reward value R i = -10; S44: Iterative Update: Updating the decision table T by using a Q-learning algorithm; S45: Loop iteration judgment: Number of loop iterations Loop = Loop + 1; If the Loop reaches the set maximum number of cycles, it goes to S46; If Loop is less than the maximum number of cycles, the charging and discharging curve of the V2G charging pile is initialized, and i=1, the cumulative reward value R a =0, the number of loop iterations Loop=1, and then enter S42; S46: Control result output: According to the decision table T, the optimal control strategy for each moment under different groups is output.

7. A method for controlling charging and discharging of V2G charging piles in a substation based on reinforcement learning according to claim 6, characterized in that: The S44 comprises the following steps: S441: Update the data in row i of decision table T according to the following formula: T(i,a)=R i +0.9×max(T(i+1)), In the formula, T(i,a) represents the data in the i-th row and a-th column of the decision table, and max(T(i+1)) represents the maximum value in the i+1-th row of the decision table T; S442: i=i+1,R a =R a +R i ; R a is the accumulated reward value; If i<96, then go to S42; if i=96, then go to S443; S443: Update the decision table according to the following formula: T(k,a k )=T(k,a k )+R a / (8*M)(k=1,2,...,8*M-1), In the formula, a k Represents the action taken at the kth moment in the iteration process.

8. A method for controlling charging and discharging of V2G charging piles in a substation based on reinforcement learning according to claim 7, characterized in that: The optimal control strategy at the i-th moment is the output action corresponding to the column number where the maximum value of the i-th row data in the decision table T is located.

9. A V2G charging pile charging and discharging control device based on reinforcement learning, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of claims 1 to 8.

Images