A q-learning electric vehicle group charging scheduling method based on clustering state space dimension reduction

Abstract

The application discloses a kind of Q-learning electric vehicle group charging scheduling methods based on clustering state space dimension reduction, it is related to electric vehicle cluster scheduling technical field, first based on user charging willingness and electric vehicle cluster charging power determines scheduling range, then constructs Markov decision model, then embedding K-means clustering state space dimension reduction module in reinforcement learning algorithm obtains clustering reinforcement learning algorithm, finally solve model output optimal scheduling strategy;The application can compress high-dimensional state space to constant complexity by clustering, solve the "dimension disaster" problem of traditional reinforcement learning in large-scale electric vehicle group collaborative scheduling, and use non-neural network inference architecture, with significant advantages in computational efficiency and data utilization, both reduce hardware resource consumption and deployment cost, but also can quickly converge under small sample condition, reduce dependence on training data size.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric vehicle cluster scheduling, and more particularly to a Q-learning electric vehicle group charging scheduling method based on clustering state space dimension reduction. BACKGROUND

[0002] At present, in actual scenarios, the electric vehicle cluster scheduling problem is difficult to model, and the reinforcement learning method which does not need to rely on traditional model construction and only learns the optimal scheduling strategy through repeated interaction with the environment has gradually shown its advantages in solving this problem. However, the traditional reinforcement learning method faces the "dimension disaster" problem of high-dimensional state space in the large-scale electric vehicle group collaborative scheduling scenario.

[0003] Therefore, how to design a state space dimension reduction mechanism with spatiotemporal coupling characteristics to reduce the dimension of the state space and thus reduce the difficulty of modeling the electric vehicle cluster scheduling problem is a problem that needs to be solved by those skilled in the art. SUMMARY

[0004] Therefore, the present application provides a Q-learning electric vehicle group charging scheduling method based on clustering state space dimension reduction to solve the problems in the background art.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A Q-learning electric vehicle group charging scheduling method based on clustering state space dimension reduction comprises:

[0007] S1, determining the electric vehicle cluster scheduling range based on the user's charging intention and the charging power of the electric vehicle cluster;

[0008] S2, constructing the electric vehicle charging scheduling decision as a Markov decision model based on the electric vehicle cluster scheduling range;

[0009] S3, obtaining a reinforcement learning algorithm, embedding a state space dimension reduction module of K-means clustering in the framework of the reinforcement learning algorithm to obtain a clustering reinforcement learning algorithm;

[0010] S4, solving the constructed Markov decision model by using the clustering reinforcement learning algorithm, and outputting an optimal scheduling strategy so that the electric vehicle charging scheduling decision is optimally scheduled under the optimal scheduling strategy.

[0011] Optionally, based on the user's charging intention and the charging power of the electric vehicle cluster, the schedulable range of the electric vehicle group participating in demand response scheduling is obtained, which is represented as:

[0012] ;

[0013] ;

[0014] ;

[0015] ;

[0016] in, It is the minimum battery state that meets the minimum commuting needs of electric vehicle owners; and These represent the minimum and maximum charging rates, respectively. and These represent the minimum and maximum discharge rates, respectively. and They represent t The upper and lower limits of the dispatchable range of the electric vehicle fleet at any given time. For the first i The battery status of an electric vehicle The charging rate during the charging and discharging process of an electric vehicle. The discharge rate during the charging and discharging process of an electric vehicle. Let t be the schedulable range of the electric vehicle group.

[0017] Optionally, the electric vehicle charging scheduling decision is constructed as a Markov decision model, including:

[0018] The formula for defining the state of an agent is:

[0019] ;

[0020] in Indicates the state of the agent. t Indicates the current time period. Indicates the first i The average battery state of each electric vehicle cluster Indicates the electricity price for charging;

[0021] The formula for the agent's actions is:

[0022] ;

[0023] in, Indicates the action of the intelligent agent. This indicates the low-power charging capacity of the charging station. This indicates the high-power charging capacity of the charging station;

[0024] The formula for rewarding an agent is:

[0025] ;

[0026] wherein, represents the maximum electricity price, represents the total charging power executed by the current cluster, represents the maximum power, represents the ideal target electricity level set, represents the lower limit of the battery state threshold, represents a heavy parameter for balancing the relative importance of cost, electricity proximity, and low electricity protection.

[0027] Optionally, the state space dimension reduction module for establishing clustering includes:

[0028] Feature extraction is performed on the load curve of the electric vehicle charging station:

[0029] ;

[0030] wherein, represents a set of feature vectors extracted from the load curve of the i th charging station, represents the load curve data of the th charging station, i represents the mean value of the load curve, represents the standard deviation of the load curve, represents the skewness of the load curve, represents the peak occurrence time of the load curve; Randomly select

[0031] samples as initial cluster centers, for each sample k , calculate its distance to each cluster center, and assign it to the nearest cluster;

[0032] ;

[0033] wherein represents the cluster number to which the sample X i belongs, is the k th cluster center;

[0034] After assignment, the cluster center is recalculated in the cluster k until the algorithm converges;

[0035] ;

[0036] wherein, and represent the new center and the old center of the k th cluster, respectively, represents the cluster​k the number of samples, η represents a convergence threshold.

[0037] Optionally, it further includes a charging station-electric vehicle cluster clustering mapping mechanism, specifically:

[0038] Through the K-means algorithm, the electric vehicle charging station set is clustered to generate k charging station clustering clusters ; each charging station c i The electric vehicles under the jurisdiction naturally form corresponding electric vehicle clusters , wherein M i the service vehicle number of the electric vehicle charging station c i The mapping relationship between the charging station cluster and the electric vehicle cluster is defined as:

[0039] ;

[0040] wherein ， represents the set of electric vehicles under the jurisdiction of the charging station , i.e. all electric vehicles served by the charging station, represents the k charging station clustering corresponding electric vehicle cluster, through the set operation, all electric vehicles of the charging stations in the same cluster are integrated into a unified scheduling unit, realizing the hierarchical mapping of charging station feature clustering to electric vehicle physical cluster.

[0041] Optionally, S4 specifically includes:

[0042] S41, define the solving input and the solving output, the solving input includes the charging price, the battery state of each electric vehicle cluster group and the current time period; the solving output includes the charging amount of each electric vehicle cluster group per hour;

[0043] S42, initialize iteration parameters, including setting the maximum number of iterations , initializing the learning rate α , the discount factor γ and the Q value table; based on the historical load curve data, the electric vehicle cluster clustering is generated through the K-means algorithm;

[0044] S43, input the agent state t at the current time ;

[0045] S44, select the electric vehicle cluster charging and discharging action through the greedy strategy;

[0046] S45, performing a corresponding action, obtaining an agent reward, and entering a next time, wherein a state of the agent is transferred from to , comprising:

[0047] ;

[0048] wherein is a change amount of the soc caused by the action;

[0049] S46, updating the Q value based on a time sequence difference algorithm;

[0050] S47, repeatedly performing steps S43-S46 until a maximum iteration number is reached.

[0051] Optionally, the greedy strategy is defined as:

[0052] ;

[0053] wherein is an exploration rate, is a discrete action space.

[0054] Optionally, updating the Q value based on the time sequence difference algorithm is specifically:

[0055] ;

[0056] wherein is an instant reward obtained after performing the current action, is an optimal action value of a next state.

[0057] According to the above technical solution, compared with the prior art, the application provides a Q-learning electric vehicle charging scheduling method based on clustering state space dimension reduction, which has the following beneficial effects:

[0058] 1、The application clusters the load curve of the electric vehicle charging station, uses the fixed attribution relationship between the electric vehicle and the charging station, naturally deduces the aggregation relationship of the electric vehicle cluster, compresses the high-dimensional state space to a constant complexity, and solves the "dimension disaster" problem of the traditional reinforcement learning method in the large-scale electric vehicle group cooperative scheduling scene.

[0059] 2. The non-neural network inference architecture is used innovatively, which has significant advantages in computing efficiency and data utilization: on the one hand, low computing complexity is realized by relying on lightweight algorithm design, which greatly reduces hardware resource consumption and deployment cost; on the other hand, the iteration efficiency is significantly improved by the optimized convergence mechanism, and stable convergence state can be reached without a large amount of training, and the efficient sample utilization strategy is used, so that fast convergence can be realized under the condition of small sample, and the dependence on the size of training data is greatly reduced compared with the traditional method. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0061] Figure 1 The overall flowchart provided by the present application is shown in the figure;

[0062] Figure 2 The electric vehicle charging scheduling system model provided by the present application is shown in the figure;

[0063] Figure 3 The electric vehicle cluster scheduling range constraint provided by the present application is shown in the figure;

[0064] Figure 4 The electric vehicle cluster dimensionality reduction mapping mechanism based on electric vehicle charging station clustering provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0066] The embodiment of the present application discloses a Q-learning electric vehicle charging scheduling method based on clustering state space dimensionality reduction, as shown in the figure, the specific steps are as follows: Figure 1

[0067] S1, based on the user charging intention and the charging power of the electric vehicle cluster, the electric vehicle cluster scheduling range is determined, and the specific analysis is as follows:

[0068] Based on the user charging intention and the charging power of the electric vehicle cluster, the schedulable range of the electric vehicle group participating in demand response scheduling is obtained, which is expressed as:​

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] in, This is the minimum battery state required to meet the minimum commuting needs of electric vehicle owners; in order to prevent long-term charging and discharging from damaging the battery, it is necessary to limit the depth of charge and discharge of electric vehicles. and These represent the minimum and maximum charging rates, respectively. and These represent the minimum and maximum discharge rates, respectively. and They represent t The upper and lower limits of the dispatchable range of the electric vehicle fleet at any given time. For the first i The battery status of an electric vehicle The charging rate during the charging and discharging process of an electric vehicle. The discharge rate during the charging and discharging process of an electric vehicle. Let t be the schedulable range of the electric vehicle group. Figure 3 This is a schematic diagram of the electric vehicle cluster scheduling range constraint in an example of the present invention, which intuitively shows the scheduling scenario under relevant boundary constraints.

[0074] S2. Based on the scheduling range of electric vehicle clusters, the electric vehicle charging scheduling decision is constructed as a Markov decision model, and the specific analysis is as follows:

[0075] Electric vehicle charging scheduling system model diagram as follows Figure 2 As shown, this charging scheduling system considers the random connection and disconnection of electric vehicles from the power grid throughout the day. Specifically, when an electric vehicle arrives at a charging station, the system makes a decision regarding the vehicle's charging and discharging behavior based on the current electricity price and the electric vehicle's battery status. During the decision execution process, the charging scheduling system buys and sells electricity from the grid and engages in energy exchange until the vehicle leaves the charging station. The main objective of this system is to ensure that electric vehicles meet user charging needs while reducing user costs.

[0076] Based on the above, the electric vehicle charging scheduling decision is constructed as a Markov decision process. In the system, the agent is deployed in the control module inside the charging station. After receiving the input information including the electricity price and the charging and discharging demand, the agent performs the action and makes the charging and discharging scheduling decision at the current time. Through continuous interaction with the electric vehicle charging station and the power market environment, the agent can gradually accumulate experience and optimize its decision strategy, and finally maximize the system revenue.

[0077] Further, the electric vehicle charging scheduling decision is constructed as a Markov decision model, including:

[0078] The formula for defining the state of the agent is:

[0079] ;

[0080] Wherein represents the state of the agent, t represents the current time period, represents the average battery state of the i th electric vehicle cluster, represents the charging electricity price;

[0081] The formula for the action of the agent is:

[0082] ;

[0083] Wherein represents the action of the agent, represents the low-power charging power of the charging station, represents the high-power charging power of the charging station;

[0084] The formula for the reward of the agent is:

[0085] ;

[0086] Wherein represents the maximum electricity price, represents the total charging power executed by the current cluster, represents the maximum power, represents the ideal target electricity level set, represents the lower limit of the battery state threshold, represents a weight parameter for balancing the relative importance of cost, electricity proximity, and low electricity protection.

[0087] The reward function can obtain a higher return when the electricity price is low, the average SOC is close to the target, and there is no low SOC individual. Conversely, when there is a low SOC vehicle (such as soc lowWhen this happens, additional penalties will be imposed, thereby guiding reinforcement learning strategies to prioritize vehicles that meet emergency charging needs, improving user satisfaction and system safety.

[0088] S3. Obtain the reinforcement learning algorithm. Embed the state space dimensionality reduction module of K-means clustering into the framework of the reinforcement learning algorithm to obtain the clustering reinforcement learning algorithm. The specific steps are as follows:

[0089] A clustering reinforcement learning algorithm is proposed, which embeds a state space dimensionality reduction module of K-means clustering on the basis of Q-learning reinforcement learning. A dimensionality reduction mapping mechanism for electric vehicle clusters based on electric vehicle charging station clustering is constructed, such as... Figure 4 As shown. This mechanism consists of two parts:

[0090] The process of the state space dimensionality reduction module for clustering:

[0091] Feature extraction of the load curve of electric vehicle charging stations:

[0092] ;

[0093] in, Indicates from the first i Load curve of each charging station A set of feature vectors extracted from them. Indicates the first i Load curve data for each charging station This represents the average value of the load curve. This represents the standard deviation of the load curve. Indicates the skewness of the load curve. This indicates the time when the peak of the load curve occurs;

[0094] Random selection k Each sample is used as the initial cluster center. Calculate the distance from each cluster center and assign it to the nearest cluster;

[0095] ;

[0096] in Indicates sample X i Cluster number to which it belongs For the first k Cluster centers;

[0097] After the assignment is completed, in this cluster k Recalculate the cluster centers until the algorithm converges;

[0098] ;

[0099] wherein, and represent the new and old center of the k th cluster, represents the number of samples of the cluster k , η represents the convergence threshold.

[0100] Charging station-electric vehicle cluster clustering mapping mechanism:

[0101] Through the K-means algorithm, the set of electric vehicle charging stations is clustered to generate k charging station clustering clusters ; since there is a fixed affiliation between electric vehicles and their charging stations, that is, each electric vehicle is long associated with a specific charging station, the clustering of charging stations can naturally deduce the aggregation relationship of the electric vehicle cluster. Each charging station c i subordinate electric vehicles naturally form corresponding electric vehicle clusters , wherein M i is the number of service vehicles of the electric vehicle charging station c i ; based on this, the mapping relationship between charging station clustering and electric vehicle cluster can be defined as:

[0102] ;

[0103] wherein ， represents the set of electric vehicles subordinate to the charging station , that is, all electric vehicles served by the charging station, represents the k th charging station clustering corresponding electric vehicle cluster, through the set union operation, all electric vehicles of the charging stations in the same cluster are integrated into a unified scheduling unit, realizing the hierarchical mapping of charging station feature clustering to electric vehicle physical cluster.

[0104] S4, solve the constructed Markov decision model through a clustering reinforcement learning algorithm, output the optimal scheduling strategy, so that the electric vehicle charging scheduling decision is best scheduled under the optimal scheduling strategy, the specific steps are as follows:

[0105] S41, define the solving input and the solving output, the solving input includes the charging price, the battery state of each electric vehicle cluster group and the current time period. The solving output includes the charging amount of each electric vehicle cluster group per hour.

[0106] S42, initialize the iteration parameters, including setting the maximum number of iterations , initialize learning rate α , discount factor γ and Q-value table. Based on historical load curve data, the electric vehicle cluster clustering is generated by K-means algorithm.

[0107] S43, input the current time t of the agent state .

[0108] S44, select the electric vehicle cluster charging and discharging action by - greedy strategy:

[0109] ;

[0110] wherein, is the exploration rate, is the discrete action space.

[0111] S45, execute the corresponding action, and after obtaining the agent reward, enter the next time, and the agent state is transferred from to , including:

[0112] ;

[0113] wherein is the soc change amount caused by the action;

[0114] S46, update the Q value based on the time difference (TD) algorithm:

[0115] ;

[0116] wherein, is the immediate reward obtained after executing the current action, is the optimal action value of the next state.

[0117] S47, repeat steps S43-S46 until the maximum iteration number is reached.

[0118] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0119] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1.A Q-learning method for electric vehicle group charging scheduling based on clustering state space dimension reduction, characterized in that, The method comprises the following steps: S1, determining an electric vehicle cluster scheduling range based on user charging intention and charging power of the electric vehicle cluster; S2, constructing an electric vehicle charging scheduling decision as a Markov decision model based on the electric vehicle cluster scheduling range; S3, obtaining a reinforcement learning algorithm, embedding a state space dimension reduction module of K-means clustering in a framework of the reinforcement learning algorithm to obtain a clustered reinforcement learning algorithm; S4, solving the constructed Markov decision model by using the clustered reinforcement learning algorithm, and outputting an optimal scheduling strategy, so that the electric vehicle charging scheduling decision is optimally scheduled under the optimal scheduling strategy; Based on the user charging intention and the charging power of the electric vehicle cluster, the schedulable range of the electric vehicle cluster participating in demand response scheduling is obtained, which is expressed as: ； ； ； ； wherein, is the minimum battery state satisfying the minimum commuting demand of the electric vehicle owner; and denote the minimum and maximum charging rate, respectively, and denote the minimum and maximum discharging rate, respectively; and denote t the upper and lower bounds of the dispatchable range of the electric vehicle fleet at time t, is the battery state of the i th electric vehicle, is the charging rate during the charging process of the electric vehicle, is the discharging rate during the discharging process of the electric vehicle, is the dispatchable range of the electric vehicle fleet at time t; The electric vehicle charging scheduling decision is constructed as a Markov decision model, which comprises: The formula for defining the state of the agent is: ； wherein represents the state of the agent, t represents the current time period, represents the average battery state of the i th cluster of electric vehicles, represents the charging electricity price; The formula for defining the action of the agent is: ； wherein, represents the agent action, represents the low power charging power of the charging station, represents the high power charging power of the charging station; The formula for defining the reward of the agent is: ； wherein, represents the maximum electricity price, represents the total charging power executed by the current cluster, represents the maximum power, represents the ideal target electricity level set, represents the lower battery state threshold, represents the weight parameter for balancing the relative importance of cost, electricity proximity, and low electricity protection; S4 specifically comprises: S41, defining a solving input and a solving output, wherein the solving input comprises a charging price, a battery state of each electric vehicle cluster, and a current time period; and the solving output comprises a charging amount of each electric vehicle cluster per hour; S42, initialize iteration parameters, including setting the maximum number of iterations , initialize learning rate α , discount factor γ and Q-value table; based on historical load curve data, generate electric vehicle cluster clustering through K-means algorithm; S43, inputting a current time t of the agent ; S44, by - a greedy strategy is used to select the charging and discharging actions of the cluster of electric vehicles; S45, execute the corresponding action, get the agent reward, and enter the next time, the agent state is transferred to S45, execute the corresponding action, get the agent reward, and enter the next time, the agent state is transferred to , comprising: ； wherein is the soc change amount due to the action; S46, updating the Q value based on a time series difference algorithm; S47, repeatedly executing steps S43-S46 until a maximum iteration number is reached; The - A greedy strategy is defined as: ； wherein, is the exploration rate, is the discrete action space; The specific method for updating the Q value based on the time series difference algorithm is: ； wherein, is the immediate reward obtained after performing the current action, is the optimal action value for the next state. 2.The Q-learning electric vehicle group charging scheduling method based on clustering state space dimension reduction according to claim 1, wherein, The state space dimension reduction module of clustering comprises: Feature extraction is performed on a load curve of the electric vehicle charging station: ； wherein, represents a set of feature vectors extracted from the load curve of the i represents load curve data of the i represents the mean of the load curve, represents the standard deviation of the load curve, represents the skewness of the load curve, represents the peak occurrence time of the load curve;​​​ Randomly select k one sample as initial cluster centers, for each sample , compute its distance to each cluster center, and assign to the nearest cluster; ； wherein representing samples X i the cluster number to which the sample belongs, is the k cluster center; After the assignment is done, the cluster k The cluster centers are recalculated at the bottom until the algorithm converges; ； wherein, and denote the new and old center of the k th cluster, respectively, denotes the number of samples of the cluster k , and η denotes the convergence threshold. 3.The Q-learning electric vehicle group charging scheduling method based on clustering state space dimension reduction according to claim 2, characterized in that, The clustered mapping mechanism of the charging station-electric vehicle cluster specifically comprises: Through K-means algorithm, the charging station set is clustered to generate k charging station clustering clusters ; each charging station c i under jurisdiction forms a corresponding electric vehicle cluster , wherein M i the service vehicle number of the electric vehicle charging station c i ; the mapping relationship between the charging station cluster and the electric vehicle cluster is defined as: ； wherein ， denotes a charging station a set of electric vehicles under jurisdiction, i.e. all electric vehicles served by the charging station, denotes a k th charging station cluster a corresponding cluster of electric vehicles, by set union operation, all electric vehicles of the charging stations in the same cluster are integrated into a unified scheduling unit, realizing the hierarchical mapping of charging station feature clustering to electric vehicle physical cluster.

Images