A method and device for regulating the charging and discharging of V2G charging piles in a transformer substation based on reinforcement learning

Abstract

This invention relates to the field of power system load management technology, specifically a method and device for regulating the charging and discharging of V2G charging piles in distribution transformer areas based on reinforcement learning. Specifically, this invention groups load data to obtain several sets of load data to be analyzed; then, by dividing peak and valley periods, it performs feature analysis on each set of load data to obtain the peak and valley electricity consumption periods for each group; next, it constructs an optimized scheduling model for V2G charging piles; and through reinforcement learning algorithms and the optimized scheduling model, it obtains the optimal charging and discharging strategy for each group of V2G charging piles. The method provided by this invention can provide strategy support for the charging and discharging scheduling of V2G charging piles in distribution transformer areas, reducing the peak-valley difference of transformers and reducing the charging cost of electric vehicles, demonstrating strong practicality and operability.

Description

Technical Field

[0001] This invention relates to the field of power system load management technology, and in particular to a method and equipment for regulating the charging and discharging of V2G charging piles in distribution areas based on reinforcement learning. Background Technology

[0002] With the deepening implementation of the energy revolution strategy and the explosive growth of distributed generation such as photovoltaic and wind power, the power grid's supply and demand balance faces new challenges: firstly, seasonal peak load contradictions are prominent; secondly, the pressure on the power grid to regulate peak loads is increasing year by year. Starting from the winter peak of 2019, my country's power supply has once again experienced a relatively tight situation, with some areas even needing to resort to power rationing to alleviate the supply shortage. Under the new circumstances, the traditional administrative measures for orderly electricity use are becoming increasingly difficult to implement. Therefore, it is necessary to innovate systems, mechanisms, and technologies to explore adjustable load resources on the demand side and manage and utilize these adjustable load resources.

[0003] Electric vehicles (EVs) are vehicles powered by batteries. According to research data, the total battery capacity of 5,000 EVs is equivalent to a large energy storage power station. Therefore, rationally utilizing the energy storage characteristics of EVs can compensate for grid loads during peak hours, reducing grid pressure; and increase load during off-peak hours, improving line load rates, thus achieving peak shaving, valley filling, energy conservation, and emission reduction. However, the key to achieving energy interaction between EVs and the grid lies in how to regulate charging and discharging through V2G charging stations. Therefore, developing a scientific and reasonable V2G charging station scheduling strategy to achieve energy interaction with the grid is a current hot topic.

[0004] Current research indicates that existing V2G charging pile control technologies primarily target the interests of the power grid, aiming to mitigate grid load fluctuations without considering users' actual controllability and willingness to participate. For example, Liu Baiyan's research on demand-response-based economic dispatch strategies for microgrids with battery swapping stations only establishes a multi-objective, two-layer real-time charging and discharging dispatch model that coordinates battery swapping stations with the power grid. While aiming to reduce peak-valley differences in the power grid through coordinated dispatch, it does not consider the actual situation of electric vehicle users.

[0005] Therefore, there is a need for a comprehensive method and equipment for regulating the charging and discharging of V2G charging piles in the distribution area, taking into account both electric vehicle users and the power grid. Summary of the Invention

[0006] The purpose of this invention is to overcome the limitations of existing V2G charging pile control technologies, which mainly focus on the interests of the power grid side and lack consideration for electric vehicle users. This invention provides a reinforcement learning-based method for controlling the charging and discharging of V2G charging piles in distribution areas.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] A method for regulating the charging and discharging of V2G charging piles in a transformer substation based on reinforcement learning includes the following steps:

[0009] S1: Load data grouping: Input the load data to be analyzed, divide it into several groups according to the preset grouping rules, and then normalize each group of load data to be analyzed to obtain several groups of normalized load data.

[0010] The load data to be analyzed includes the daily load data of the transformer substations and V2G charging piles for a preset time period.

[0011] S2: Peak and off-peak time period division: Perform feature analysis on several groups of normalized load data to obtain the peak and off-peak electricity consumption periods of each group of normalized load data;

[0012] S3: Regulation Model Construction: The optimization objective function is to reduce the load fluctuation of the power grid in the distribution area and reduce the charging cost of electric vehicle users; the optimization scheduling model of V2G charging piles is constructed with the constraints of charging and discharging power of V2G charging piles, charging amount of electric vehicles, and charging and discharging time period as constraints.

[0013] S4: Reinforcement Learning Optimization Solution: Using reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, solve for the optimal charging and discharging strategy of each group of V2G charging piles and output it.

[0014] As a preferred embodiment of the present invention, the preset grouping rule in S1 is as follows:

[0015] The load data of the transformer area is divided into 8 groups based on spring working days, spring non-working days, summer working days, summer non-working days, autumn working days, autumn non-working days, winter working days, and winter non-working days.

[0016] As a preferred embodiment of the present invention, step S1 includes the following steps:

[0017] S11: Input the load data to be analyzed for a preset time period;

[0018] S12: Group the load data to be analyzed according to season and whether it is a working day to obtain eight groups of data: Spring working days, Spring non-working days, Summer working days, Summer non-working days, Autumn working days, Autumn non-working days, Winter working days, and Winter non-working days.

[0019] S13: Calculate the baseline load data of the transformer area after removing the load of V2G charging piles. The calculation formula is:

[0020] P L0i (t)=P L1i (t)-Pvi (t) (t=1,2,…,8*M),

[0021] In the formula, P L0i (t) represents the baseline load of the transformer area at time t on day i, P L1i (t) represents the total load of the transformer area at time t on day i, P vi (t) represents the V2G charging pile load data at time t on day i, and M is the average number of daily samplings;

[0022] S14: Normalize the baseline load data for each transformer area to obtain several sets of normalized load data. The calculation formula is as follows:

[0023] P' L0i (t)=[P L0i (t)-min(P Lj )] / [max(P Lj )-min(P Lj (t=1,2,…,8*M),

[0024] In the formula, P' L0i (t) represents the normalized base load of the transformer area at time t on day i; max(P Lj ) represents the maximum load value of the transformer area in the j-th data group; min(P Lj ) represents the minimum load value of the transformer area in the j-th data group.

[0025] As a preferred embodiment of the present invention, step S2 includes the following steps:

[0026] S21: Average the load data from the 8 sets to be analyzed to obtain the daily average load at 8*M time points; the specific calculation formula is as follows:

[0027]

[0028] In the formula, P LAj (t) represents the average load of the j-th data set at time t; N represents the number of days included in the j-th data set;

[0029] S22: Peak electricity consumption periods for distribution transformer areas shall be defined according to the following rules:

[0030] Rule 1: If the average load of the load data to be analyzed is greater than 0.8 at time t, then time t is the peak time;

[0031] Rule 2: If the average load is greater than 0.7 during off-peak hours adjacent to peak hours, then that time is considered the next peak hour.

[0032] Rule 3: Divide consecutive peak hours and sub-peak hours into peak electricity consumption periods;

[0033] S23: The off-peak electricity consumption periods for the transformer substations shall be divided according to the following rules:

[0034] Rule 1: If the average load of the load data to be analyzed is less than 0.3 at time t, then time t is the trough time;

[0035] Rule 2: If the average load is less than 0.4 during a non-low-peak time adjacent to a low-peak time, then that time is the next low-peak time.

[0036] Rule 3: Divide consecutive off-peak hours and the next lowest off-peak hours into off-peak electricity usage periods.

[0037] As a preferred embodiment of the present invention, step S3 includes the following steps:

[0038] S31: Optimize the objective function construction:

[0039] The expression for the objective function to be optimized is:

[0040] obj0 = min{obj1 + obj2},

[0041] Optimization objective 1-obj1:

[0042] obj1=min{P max -P min},

[0043] In the formula, P max P represents the maximum daily load data for the transformer area. min This represents the minimum value of the daily load data for the transformer substation.

[0044] Optimization objective 2-obj2:

[0045] obj2=min{Q I C I -Q O C O},

[0046] In the formula, Q I Q represents the amount of electricity a car can be charged. O C represents the discharge capacity of an electric vehicle. I C represents the charging cost per unit of electricity. O The discharge fee per unit of electricity;

[0047] S32: Constraint Construction:

[0048] Constraint 1: Charging and discharging power constraints of V2G charging piles:

[0049] P omax ≤Pc(t)≤P Imax ,

[0050] In the formula, Pc(t) represents the charging and discharging power at time t, Pc(t)>0 indicates charging, and Pc(t)<0 indicates discharging; P omax P represents the maximum discharge power. Imax Indicates the maximum charging power;

[0051] Constraint 2: Electric vehicle charging capacity constraint:

[0052] Q I -Q o ≥Q min ,

[0053] In the formula, Q min This refers to the user's minimum charging capacity requirement;

[0054] Constraint 3: Charge / discharge period constraint:

[0055] V2G charging piles only discharge during peak grid hours and prioritize charging during off-peak grid hours.

[0056] As a preferred embodiment of the present invention, step S4 includes the following steps:

[0057] S41: Initialization:

[0058] Construct an M-row, 9-column decision table T, using the average load of each group of V2G charging piles as the initial value, to construct the initial charging and discharging curves of the V2G charging piles; and initialize the charging price and cycle parameters at each time point.

[0059] In the decision table T, the i-th row represents the decision value at the i-th moment of the day, M is the daily average sampling number, and the 9 columns correspond to the output actions of 100% power charging, 75% power charging, 50% power charging, 25% power charging, no charging / discharging, 25% power discharging, 50% power discharging, 75% power discharging, and 100% power discharging, respectively. The initial values ​​of the data in the decision table T are all 0. The initialization of the loop parameters includes initializing the counter variable i = 1, the cumulative reward value Ra = 0, the loop iteration count Loop = 1, and setting the maximum number of loops.

[0060] S42: Action Selection:

[0061] Action selection is based on preset probabilities:

[0062] The probability is set as follows: the maximum value of the data in the i-th row of the decision table T is taken as the output action of the V2G charging pile at the i-th time; when all the data in the i-th row of the decision table T are 0, an output action is randomly selected.

[0063] 1-Set probability: Randomly select an output action;

[0064] S43: Reward Value Calculation:

[0065] Based on the optimization objective function, calculate the target value of the V2G charging pile after adjustment, and then subtract the target value after adjustment from the target value before adjustment to obtain the reward value R. i ;

[0066] If the constraint is not satisfied after taking the action, then the reward value R is set to... i =-10;

[0067] S44: Iterative Update:

[0068] The decision table T is updated using the Q-learning algorithm;

[0069] S45: Loop Iteration Judgment:

[0070] Loop iteration count = Loop + 1;

[0071] If the loop reaches the set maximum number of iterations, proceed to S46;

[0072] If the loop count is less than the maximum number of loops, initialize the charging and discharging curve of the V2G charging station, let i = 1, and accumulate the reward value R. a =0, Loop iteration count = 1, then enter S42;

[0073] S46: Output of regulation results:

[0074] Output the optimal control strategy for each time step under different groupings based on decision table T.

[0075] As a preferred embodiment of the present invention, S44 includes the following steps:

[0076] S441: Update the i-th row of decision table T according to the following formula:

[0077] T(i,a)=R i +0.9×max(T(i+1)),

[0078] In the formula, T(i,a) represents the data in the i-th row and a-th column of the decision table, and max(T(i+1)) represents the maximum value in the (i+1)-th row of the decision table T;

[0079] S442: i = i + 1, R a =R a +R i ;R a This refers to the cumulative reward value.

[0080] If i < 96, proceed to S42; if i = 96, proceed to S443.

[0081] S443: Update the decision table according to the following formula:

[0082] T(k,a k )=T(k,a k )+R a / (8*M)(k=1,2,...,8*M-1),

[0083] In the formula, a k This represents the action taken at the k-th moment during the iteration process.

[0084] As a preferred embodiment of the present invention, the optimal control strategy at time i is the output action corresponding to the column number of the maximum value of the data in the i-th row of decision table T.

[0085] A V2G charging pile charging and discharging control device based on reinforcement learning includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the methods described above.

[0086] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0087] This invention obtains several groups of load data to be analyzed by grouping load data; then, by dividing the load data into peak and valley periods, it performs feature analysis on each group to obtain the peak and valley electricity consumption periods for each group; next, it constructs an optimized scheduling model for V2G charging piles; and through reinforcement learning algorithms and the optimized scheduling model, it obtains the optimal charging and discharging strategy for each group of V2G charging piles. The method provided by this invention can provide strategy support for the charging and discharging scheduling of V2G charging piles in a transformer substation, reducing the peak-valley difference of transformers and reducing the charging cost of electric vehicles, demonstrating strong practicality and operability. Attached Figure Description

[0088] Figure 1 This is a flowchart illustrating a reinforcement learning-based method for regulating the charging and discharging of V2G charging piles in a distribution area, as described in Embodiment 1 of the present invention.

[0089] Figure 2 This is a schematic diagram of the peak-valley time period division results in the reinforcement learning-based V2G charging pile charging and discharging control method described in Embodiment 2 of the present invention.

[0090] Figure 3 This is a schematic diagram of V2G optimization scheduling in a V2G charging pile charging and discharging control method based on reinforcement learning, as described in Embodiment 2 of the present invention.

[0091] Figure 4 This is a schematic diagram of the structure of a V2G charging pile charging and discharging control device based on reinforcement learning, which utilizes the V2G charging and discharging control method based on reinforcement learning described in Embodiment 1, as described in Embodiment 3 of the present invention. Detailed Implementation

[0092] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0093] Example 1

[0094] like Figure 1 As shown, a method for regulating the charging and discharging of V2G charging piles in a transformer substation based on reinforcement learning includes the following steps:

[0095] S1: Load data grouping: Input the load data to be analyzed and divide it into several groups according to the preset grouping rules. Then, normalize each group of load data to be analyzed to obtain several groups of normalized load data.

[0096] The load data to be analyzed includes the daily load data of the transformer substations and V2G charging piles for a preset time period.

[0097] S2: Peak and off-peak time periods: Perform feature analysis on several groups of normalized load data to obtain the peak and off-peak electricity consumption periods for each group of normalized load data.

[0098] S3: Regulation Model Construction: The optimization objective function is to reduce the load fluctuation of the power grid in the distribution area and reduce the charging cost of electric vehicle users. The optimization scheduling model of V2G charging piles is constructed with the constraints of V2G charging pile charging and discharging power, electric vehicle charging amount, and charging and discharging time period.

[0099] S4: Reinforcement Learning Optimization Solution: Using reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, solve for the optimal charging and discharging strategy of each group of V2G charging piles and output it.

[0100] Example 2

[0101] This embodiment is a specific implementation of the reinforcement learning-based V2G charging pile charging and discharging control method described in Embodiment 1, including the following steps:

[0102] S1: Load data grouping: Input the load data to be analyzed and divide it into several groups according to the preset grouping rules. Then, normalize each group of load data to be analyzed to obtain several groups of normalized load data.

[0103] S11: Input the load data to be analyzed for a preset time period; the load data to be analyzed includes the daily load data of the transformer area and V2G charging piles for a preset time period.

[0104] S12: Group the load data to be analyzed according to season and whether it is a working day to obtain eight groups of data: Spring working days, Spring non-working days, Summer working days, Summer non-working days, Autumn working days, Autumn non-working days, Winter working days, and Winter non-working days.

[0105] S13: Calculate the baseline load data of the transformer area after removing the load of V2G charging piles. The calculation formula is:

[0106] P L0i (t)=P L1i (t)-P vi (t) (t=1,2,…,8*M),

[0107] In the formula, P L0i (t) represents the baseline load of the transformer area at time t on day i, P L1i (t) represents the total load of the transformer area at time t on day i, P vi (t) represents the V2G charging pile load data at time t on day i, 8 represents 8 sets of data, and M is the average number of daily samplings. For example, if sampling is done once per hour, then M = 24; if sampling is done 4 times per hour, then M = 96.

[0108] S14: Normalize the baseline load data for each transformer area to obtain several sets of normalized load data. The calculation formula is as follows:

[0109] P' L0i (t)=[P L0i (t)-min(P Lj )] / [max(P Lj )-min(P Lj (t=1,2,…,8*M),

[0110] In the formula, P' L0i (t) represents the normalized base load of the transformer area at time t on day i; max(P Lj ) represents the maximum load value of the transformer area in the j-th data group; min(P Lj ) represents the minimum load value of the transformer area in the j-th data group.

[0111] S2: Peak and off-peak time periods: Perform feature analysis on several groups of normalized load data to obtain the peak and off-peak electricity consumption periods for each group of normalized load data.

[0112] S21: Average the load data from the eight sets of data to be analyzed to obtain the average load over 8*M time periods. The specific calculation formula is as follows:

[0113]

[0114] In the formula, P LAj (t) represents the average load of the j-th data set at time t; N represents the number of days included in the j-th data set.

[0115] S22: Peak electricity consumption periods for distribution transformer areas shall be defined according to the following rules:

[0116] Rule 1: If the average load of the load data to be analyzed is greater than 0.8 at time t, then time t is the peak time;

[0117] Rule 2: If the average load is greater than 0.7 during off-peak hours adjacent to peak hours, then that time is considered the next peak hour.

[0118] Rule 3: Divide consecutive peak hours and sub-peak hours into peak electricity consumption periods.

[0119] S23: The off-peak electricity consumption periods for the transformer substations shall be divided according to the following rules:

[0120] Rule 1: If the average load of the load data to be analyzed is less than 0.3 at time t, then time t is the trough time;

[0121] Rule 2: If the average load is less than 0.4 during a non-low-peak time adjacent to a low-peak time, then that time is the next low-peak time.

[0122] Rule 3: Divide consecutive off-peak hours and the next lowest off-peak hours into off-peak electricity usage periods.

[0123] For example, for Figure 2 The load curve shown is divided into peak and valley periods, with peak periods being 0:15-6:00 and 16:30-23:00. Figure 2 The area marked by the black line in the middle of the map is at its lowest point, from 9:30 to 15:30. Figure 2 (Middle gray line area).

[0124] S3: Regulation Model Construction: The optimization objective function is to reduce the load fluctuation of the power grid in the distribution area and reduce the charging cost of electric vehicle users. The optimization scheduling model of V2G charging piles is constructed with the constraints of V2G charging pile charging and discharging power, electric vehicle charging amount, and charging and discharging time period.

[0125] S31: Optimize the objective function construction:

[0126] Optimization Goal 1:

[0127] For the transformer substation, the control objective is to reduce load fluctuations, i.e., to reduce the daily peak-to-valley difference. Therefore, the optimization objective 1 is to reduce the peak-to-valley difference of the substation load, obj1.

[0128] obj1=min{P max -P min},

[0129] In the formula, P max P represents the maximum daily load data for the transformer area. min This represents the minimum daily load data for the transformer area.

[0130] Optimization Goal 2:

[0131] For electric vehicle users, one of the goals of participating in vehicle-to-grid interaction is to minimize charging costs. Therefore, optimization objective 2 is to reduce charging costs (obj2).

[0132] obj2=min{Q I C I -Q O C O},

[0133] In the formula, Q I Q represents the amount of electricity a car can be charged. O C represents the discharge capacity of an electric vehicle. I C represents the charging cost per unit of electricity. O This represents the discharge fee per unit of electricity.

[0134] In summary, the objective function obj0 for optimizing the charging and discharging control of V2G charging piles is as follows:

[0135] obj0 = min{obj1 + obj2},

[0136] S32: Constraint Construction:

[0137] Constraint 1: Charging and discharging power constraints of V2G charging piles:

[0138] P omax ≤Pc(t)≤P Imax ,

[0139] In the formula, Pc(t) represents the charging and discharging power at time t, Pc(t)>0 indicates charging, and Pc(t)<0 indicates discharging; P omax P represents the maximum discharge power. Imax This indicates the maximum charging power.

[0140] Constraint 2: To ensure the normal usage needs of electric vehicle users, the net charge input should reach the target value after V2G charging and discharging regulation. Further, there are constraints on electric vehicle charging volume:

[0141] Q I -Q o ≥Qmin ,

[0142] In the formula, Q min This refers to the user's minimum charging power requirement.

[0143] Constraint 3: Charge / discharge period constraint:

[0144] V2G charging piles only discharge during peak grid hours and prioritize charging during off-peak grid hours.

[0145] S4: Reinforcement Learning Optimization Solution: Using reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, solve for the optimal charging and discharging strategy of each group of V2G charging piles and output it.

[0146] S41: Initialization:

[0147] Construct an M-row, 9-column decision table T, using the average load of each group of V2G charging piles as the initial value, to construct the initial charging and discharging curves of the V2G charging piles; and initialize the charging price and cycle parameters at each time point.

[0148] In the decision table T, the i-th row represents the decision value at the i-th moment of the day, and the nine columns correspond to the output actions of 100% power charging, 75% power charging, 50% power charging, 25% power charging, no charging / discharging, 25% power discharging, 50% power discharging, 75% power discharging, and 100% power discharging, respectively. The initial values ​​of the data in the decision table T are all 0. The initialization of the loop parameters includes initializing the counter variable i = 1, the cumulative reward value Ra = 0, the loop iteration count Loop = 1, and setting the maximum number of loops.

[0149] S42: Action Selection:

[0150] Action selection is based on preset probabilities:

[0151] Set the probability (in this embodiment, it is set to 80%): take the maximum value of the data in the i-th row of the decision table T as the output action of the V2G charging pile at the i-th moment; wherein, when all the data in the i-th row of the decision table T are 0, an output action is randomly selected.

[0152] 1- Set the probability (20% in this example): Randomly select an output action.

[0153] S43: Reward Value Calculation:

[0154] Based on the optimization objective function, calculate the target value of the V2G charging pile after adjustment, and then subtract the target value after adjustment from the target value before adjustment to obtain the reward value R. i .

[0155] If the constraint is not satisfied after taking the action, then the reward value R is set to... i =-10.

[0156] S44: Iterative Update:

[0157] The decision table T is updated using the Q-learning algorithm.

[0158] S441: Update the i-th row of decision table T according to the following formula:

[0159] T(i,a)=R i +0.9×max(T(i+1)),

[0160] In the formula, T(i,a) represents the data in the i-th row and a-th column of the decision table, and max(T(i+1)) represents the maximum value in the (i+1)-th row of the decision table T;

[0161] S442: i = i + 1, R a =R a +R i ;R a This refers to the cumulative reward value.

[0162] If i < 96, proceed to S42; if i = 96, proceed to S443.

[0163] S443: Update the decision table according to the following formula:

[0164] T(k,a k )=T(k,a k )+R a / (8*M)(k=1,2,...,8*M-1),

[0165] In the formula a k This represents the action taken at the k-th moment during the iteration process.

[0166] S45: Loop Iteration Judgment:

[0167] The number of loop iterations is Loop = Loop + 1.

[0168] If the loop reaches the set maximum number of iterations, proceed to S46;

[0169] If the loop count is less than the maximum number of loops, initialize the charging and discharging curve of the V2G charging station, let i = 1, and accumulate the reward value R. a =0, Loop iteration count = 1, then enter S42.

[0170] S46: Output of regulation results:

[0171] Output the optimal control strategy for each time step under different groupings based on decision table T.

[0172] Furthermore, the optimal control strategy at time i is the output action corresponding to the column number of the maximum value of the data in the i-th row of decision table T.

[0173] For example, after adjusting the charging load of V2G in the distribution area, the result is as follows: Figure 3 As shown in the diagram, a positive charge / discharge power indicates charging, while a negative power indicates discharging.

[0174] Example 3

[0175] like Figure 4 As shown, a reinforcement learning-based V2G charging pile charging and discharging control device includes at least one processor, a memory communicatively connected to the at least one processor, and at least one input / output interface communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the reinforcement learning-based V2G charging pile charging and discharging control method described in the foregoing embodiment. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data.

[0176] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0177] When the integrated units of this invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for regulating the charging and discharging of V2G charging piles in a transformer substation based on reinforcement learning, characterized in that, Includes the following steps: S1: Load data grouping: Input the load data to be analyzed, divide it into several groups according to the preset grouping rules, and then normalize each group of load data to be analyzed to obtain several groups of normalized load data. The load data to be analyzed includes the daily load data of the transformer substations and V2G charging piles for a preset time period. S2: Peak and off-peak time period division: Perform feature analysis on several groups of normalized load data to obtain the peak and off-peak electricity consumption periods of each group of normalized load data; S3: Regulation Model Construction: The optimization objective function is to reduce the load fluctuation of the power grid in the distribution area and reduce the charging cost of electric vehicle users; the optimization scheduling model of V2G charging piles is constructed with the constraints of charging and discharging power of V2G charging piles, charging amount of electric vehicles, and charging and discharging time period as constraints. The expression for the optimization objective function is: obj0 = min{obj1 + obj2}, Optimization target obj1: obj1=min{P max -P min }, In the formula, P max P represents the maximum daily load data for the transformer area. min This represents the minimum value of the daily load data for the transformer substation. Optimization target obj2: obj2=min{Q I C I -Q O C O }, In the formula, Q I Q represents the amount of electricity a car can be charged. O C represents the discharge capacity of an electric vehicle. I C represents the charging cost per unit of electricity. O The discharge fee per unit of electricity; S4: Reinforcement Learning Optimization Solution: Using reinforcement learning algorithm and the optimization scheduling model of the V2G charging piles, solve for the optimal charging and discharging strategy of each group of V2G charging piles and output it; S4 includes the following steps: S41: Initialization: Construct an M-row, 9-column decision table T, using the average load of each group of V2G charging piles as the initial value, to construct the initial charging and discharging curves of the V2G charging piles; and initialize the charging price and cycle parameters at each time point. In the decision table T, the i-th row represents the decision value at the i-th moment of the day, M is the daily average sampling number, and the 9 columns correspond to the output actions of 100% power charging, 75% power charging, 50% power charging, 25% power charging, no charging / discharging, 25% power discharging, 50% power discharging, 75% power discharging, and 100% power discharging, respectively. The initial values ​​of the data in the decision table T are all 0. The initialization of the loop parameters includes initializing the counter variable i = 1, the cumulative reward value Ra = 0, the loop iteration count Loop = 1, and setting the maximum number of loops. S42: Action Selection: Action selection is based on preset probabilities: The probability is set as follows: the maximum value of the data in the i-th row of the decision table T is taken as the output action of the V2G charging pile at the i-th time; when all the data in the i-th row of the decision table T are 0, an output action is randomly selected. (1-Set probability): Randomly select an output action; S43: Reward Value Calculation: Based on the optimization objective function, calculate the target value of the V2G charging pile after adjustment, and then subtract the target value after adjustment from the target value before adjustment to obtain the reward value R. i ; If the constraint is not satisfied after taking the action, then the reward value R is set to... i =-10; S44: Iterative Update: The decision table T is updated using the Q-learning algorithm; S45: Loop Iteration Judgment: Loop iteration count = Loop + 1; If the loop reaches the set maximum number of iterations, proceed to S46; If the loop count is less than the maximum number of loops, initialize the charging and discharging curve of the V2G charging station, let i = 1, and accumulate the reward value R. a =0, Loop iteration count = 1, then enter S42; S46: Output of regulation results: Output the optimal control strategy for each time step under different groupings based on decision table T.

2. The method for regulating the charging and discharging of V2G charging piles in a distribution area based on reinforcement learning according to claim 1, characterized in that, The preset grouping rule in S1 is: The load data of the transformer area is divided into 8 groups based on spring working days, spring non-working days, summer working days, summer non-working days, autumn working days, autumn non-working days, winter working days, and winter non-working days.

3. The method for regulating the charging and discharging of V2G charging piles in a distribution area based on reinforcement learning according to claim 2, characterized in that, S1 includes the following steps: S11: Input the load data to be analyzed for a preset time period; S12: Group the load data to be analyzed according to season and whether it is a working day to obtain eight groups of data: Spring working days, Spring non-working days, Summer working days, Summer non-working days, Autumn working days, Autumn non-working days, Winter working days, and Winter non-working days. S13: Calculate the baseline load data of the transformer area after removing the load of V2G charging piles. The calculation formula is: P L0i (t)=P L1i (t)-P vi (t),t=1,2,…,8×M, In the formula, P L0i (t) represents the baseline load of the transformer area at time t on day i, P L1i (t) represents the total load of the transformer area at time t on day i, P vi (t) represents the V2G charging pile load data at time t on day i, and M is the average number of daily samplings; S14: Normalize the baseline load data for each transformer area to obtain several sets of normalized load data. The calculation formula is as follows: P' L0i (t)=[P L0i (t)-min(P Lj )] / [max(P Lj )-min(P Lj )],t=1,2,…,8×M, In the formula, P' L0i (t) represents the normalized base load of the transformer area at time t on day i; max(P Lj ) represents the maximum load value of the transformer area in the j-th data group; min(P Lj ) represents the minimum load value of the transformer area in the j-th data group.

4. The method for regulating the charging and discharging of V2G charging piles in a distribution area based on reinforcement learning according to claim 3, characterized in that, S2 includes the following steps: S21: Average the 8 sets of load data to be analyzed to obtain the daily average load for 8×M time periods; the specific calculation formula is as follows: In the formula, P LAj (t) represents the average load of the j-th data set at time t; N represents the number of days included in the j-th data set; S22: Peak electricity consumption periods for distribution transformer areas shall be defined according to the following rules: Rule 1: If the average load of the load data to be analyzed is greater than 0.8 at time t, then time t is the peak time; Rule 2: If the average load is greater than 0.7 during off-peak hours adjacent to peak hours, then that time is considered the next peak hour. Rule 3: Divide consecutive peak hours and sub-peak hours into peak electricity consumption periods; S23: The off-peak electricity consumption periods for the transformer substations shall be divided according to the following rules: Rule 1: If the average load of the load data to be analyzed is less than 0.3 at time t, then time t is the trough time; Rule 2: If the average load is less than 0.4 during a non-low-peak time adjacent to a low-peak time, then that time is the next low-peak time. Rule 3: Divide consecutive off-peak hours and the next lowest off-peak hours into off-peak electricity usage periods.

5. The method for regulating the charging and discharging of V2G charging piles in a distribution area based on reinforcement learning according to claim 1, characterized in that, S3 includes the following steps: S31: Optimize the construction of the objective function; S32: Constraint Construction: Constraint 1: Charging and discharging power constraints of V2G charging piles: P omax ≤Pc(t)≤P Imax , In the formula, Pc(t) represents the charging and discharging power at time t, Pc(t)>0 indicates charging, and Pc(t)<0 indicates discharging; P omax P represents the maximum discharge power. Imax Indicates the maximum charging power; Constraint 2: Electric vehicle charging capacity constraint: Q I -Q o ≥Q min , In the formula, Q min This refers to the user's minimum charging capacity requirement; Constraint 3: Charge / discharge period constraint: V2G charging piles only discharge during peak grid hours and prioritize charging during off-peak grid hours.

6. The method for regulating the charging and discharging of V2G charging piles in a distribution area based on reinforcement learning according to claim 1, characterized in that, S44 includes the following steps: S441: Update the i-th row of decision table T according to the following formula: T(i,a)=R i +0.9×max(T(i+1)), In the formula, T(i,a) represents the data in the i-th row and a-th column of the decision table, and max(T(i+1)) represents the maximum value in the (i+1)-th row of the decision table T; S442: i = i + 1, R a =R a +R i ;R a This refers to the cumulative reward value. If i < 96, proceed to S42; if i = 96, proceed to S443. S443: Update the decision table according to the following formula: T(k,a k )=T(k,a k )+R a / (8×M),k=1,2,...,8×M-1, In the formula, a k This represents the action taken at the k-th moment during the iteration process.

7. The method for regulating the charging and discharging of V2G charging piles in a distribution area based on reinforcement learning according to claim 6, characterized in that, The optimal control strategy at time i is the output action corresponding to the column number of the maximum value in the i-th row of decision table T.

8. A V2G charging pile charging and discharging control device based on reinforcement learning, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

Images