Electric vehicle charging station operation management method and system based on virtual power plant

Abstract

The electric vehicle charging station operation and management method and system based on a virtual power plant provided in the present application relate to the technical field of charging station operation methods, collect historical operation data of the charging station, and classify the data after pre-processing, model based on the charging station environment, use value function as the basis of the model, design a deep reinforcement learning model, train the deep reinforcement learning model based on the collected historical data, use a deep Q network, optimize the model parameters through interaction with the environment, so that the model learns the optimal operation strategy, evaluate the trained model based on a verification data set, evaluate the performance of the model, and make targeted adjustments and optimization to the model based on the evaluation results, deploy the optimized and adjusted model to the operation management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed back to the deep reinforcement learning model for retraining, so as to continuously optimize the operation strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of charging station operation methods, and in particular to an electric vehicle charging station operation management method and system based on a virtual power plant. Background Art

[0002] A virtual power plant is an intelligent system that integrates, optimizes and manages distributed energy resources. It integrates multiple energy technologies and flexible operation strategies to achieve effective dispatch and management of the power system. In the field of electric vehicle charging stations, the application of virtual power plants can integrate scattered charging station resources.

[0003] In today's energy transformation and sustainable development context, electric vehicles, as an important part of clean energy transportation, have an urgent need for accurate prediction and efficient management of charging demand. With the increase in the number of electric vehicles and the continuous construction of charging infrastructure, the traditional operation and management of charging stations based on manual experience can no longer meet the growing demand and complex energy system requirements. Conventional operation and management methods cannot achieve charging demand prediction, let alone accurate prediction and dynamic scheduling of charging demand, which leads to a decrease in the utilization rate of charging facilities at charging stations, which is not conducive to promoting the popularization and sustainable development of clean energy.

[0004] Therefore, there is an urgent need to propose an electric vehicle charging station operation and management method and system based on a virtual power plant. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of the prior art and provide an electric vehicle charging station operation management method and system based on a virtual power plant. The method aims to adopt a deep Q network to optimize model parameters through interaction with the environment so that the model learns the optimal operation strategy. At the same time, the ARIMA model is used to realize the prediction and processing of charging demand, so as to achieve accurate prediction and dynamic scheduling of charging demand.

[0006] To this end, the present application provides an electric vehicle charging station operation and management method based on a virtual power plant, comprising the following steps:

[0007] Step 100: Collect historical operation data of the charging station, and classify the data after pre-processing, specifically classifying it into charging demand data, charging power data, energy supply data, and customer satisfaction data;

[0008] Step 200: Modeling is performed based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined;

[0009] Step 300: Using the value function as the basis of the model, design a deep reinforcement learning model, specifically including the structure of the neural network, state representation, and action selection strategy;

[0010] Step 400: training a deep reinforcement learning model based on the collected historical data, using a deep Q network, optimizing model parameters through interaction with the environment, so that the model learns the optimal operation strategy;

[0011] Step 500: Evaluate the trained model based on the validation data set, evaluate the performance of the model, and make targeted adjustments and optimizations to the model based on the evaluation results;

[0012] Step 600: Deploy the optimized and adjusted model to the operation management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed it back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

[0013] In some specific implementations, modeling is performed based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined, specifically including:

[0014] Step 200.1, define the state space of the charging station, determine the various factors that affect the operation of the charging station, including the location of the charging station, the number of charging piles, the type and power of the charging piles, the use status of the charging piles, the charging demand, and the energy supply. After the factors are determined, combine them to form a state space, which is used to describe the environmental state of the charging station for subsequent training and optimization of the model;

[0015] Step 200.1.1, state space, assuming the state space is , a vector is used to represent the current state of the charging station environment, that is, , specifically:

[0016]

[0017] In the formula, are the dimensions in the state space, each of which represents a feature or variable in the charging station environment;

[0018] Step 200.2, define the action space of the charging station. The action space defines the decisions and operations that the charging station can take. During the definition process, three specific factors need to be considered, namely, the scheduling of charging piles, the adjustment of charging rate, and the setting of charging price;

[0019] Step 200.2.1, action space, let the action space be , a vector is used to represent the decision and operation taken by the charging station, specifically:

[0020]

[0021] In the formula, are the dimensions in the action space, each of which represents a specific action or decision taken by the charging station;

[0022] Step 200.3, define the reward mechanism of the charging station. The reward mechanism defines the behavior evaluation criteria of the deep reinforcement learning model in the charging station environment. Through the reward mechanism, the model learns the degree of change of taking different actions in different states. When determining the reward mechanism, the reward mechanism is confirmed by using maximizing the profit as the operating target benchmark;

[0023] Step 200.3.1, reward mechanism, set in a certain state Take action The reward obtained is , then the reward mechanism is represented by a function, specifically:

[0024]

[0025] In the formula, this function describes the Take action The numerical reward obtained after

[0026] Step 200.4: Establish a state transition model. The state transition model is used to describe the transition relationship between different states in the charging station environment. In this environment, the state of the charging station will change over time, and this change can be represented by the state transition model.

[0027] Assume the current state of the charging station is ,in Represents the time step, and the state transition model is specifically:

[0028]

[0029] In the formula, is in state The probability of taking an action under the current state Take action After that, the charging station will transfer to the next state The probability of

[0030] Step 200.5: Based on the data benchmarks provided in steps 200.1 to 200.4, the charging station environment is modeled and the established model is verified to ensure that it can reflect the characteristics of the charging station environment. If the verification result is not ideal, the model needs to be adjusted, including adjusting the dimension of the state space, adjusting the range of the action space, and redefining the reward mechanism.

[0031] In some specific implementations, a value function is used as the basis of the model to design a deep reinforcement learning model, specifically including the structure of the neural network, state representation, and action selection strategy, specifically including:

[0032] Step 300.1, based on the TensorFlow deep learning framework, a deep Q network learning algorithm is used to define a network structure, where the deep Q network consists of an input layer, a hidden layer, and an output layer;

[0033] The size of the input layer matches the dimension of the state representation, the hidden layer uses a recurrent layer, and the size of the output layer matches the number of actions that the charging station can take;

[0034] Step 300.1.2, input layer: ,in, is a vector containing the status of the charging station;

[0035] Hidden Layer: ,in, is the output of the hidden layer, is the activation function, and are the weights and biases of the hidden layer;

[0036] Output layer: ,in is the estimate of the action value function, and are the weights and biases of the output layer;

[0037] Step 300.2: After building the network structure, initialize the network parameters based on the TensorFlow framework. The module initializes network parameters;

[0038] Step 300.3, define the loss function. In deep Q learning, the mean square error loss function is used to measure the difference between the predicted value and the target value. The MSE loss function is specifically:

[0039]

[0040] In the formula, is the action value function output by the network, is the target value, is the sample size;

[0041] Step 300.4: During the training process, use The optimizer updates the parameters of the network, specifically:

[0042]

[0043] In the formula, As parameters, is the learning rate, is the gradient, is the square value of the exponentially weighted average of the gradient, Used to prevent the denominator from being zero.

[0044] The square of the exponentially weighted mean of the gradients The specific update rules are:

[0045]

[0046] In the formula, is the decay rate, which is close to 1 and is used to smooth the exponentially weighted average of the historical gradient square values;

[0047] Step 300.5, using the high-level features charging pile utilization and charging demand prediction value extracted by feature engineering as state representation;

[0048] Step 300.6: Use the epsilon-greedy strategy to select actions. This strategy has a certain probability In the other cases, the action with the highest value is chosen based on the current value function.

[0049] In some specific implementations, high-level features of charging pile utilization and charging demand prediction values ​​extracted through feature engineering are used as state representations, specifically including:

[0050] Step 300.5.1, define the calculation method of charging pile utilization rate. Charging pile utilization rate is defined as the proportion of charging piles used in a certain time period. It is calculated by counting the usage of each charging pile in a certain time period. Specifically, it is:

[0051]

[0052] In the formula, in the formula, Represents the total number of times it is used within a certain period of time. Represents the total available time of the charging pile in a certain period of time. Represents the utilization rate of charging piles in the station;

[0053] Step 300.5.2, using the ARIMA model to calculate the charging demand forecast value;

[0054] Step 300.5.2.1, extract charging demand data of charging stations, including hourly or daily charging volume, and analyze the characteristics of time series to obtain trend, seasonality and periodicity characteristics of the data;

[0055] Step 300.5.2.2, stationarity test: In the ARIMA model, it is necessary to ensure that the time series is stationary. The stationarity test is implemented based on the difference operation.

[0056] Assume the original time series is , the difference operation is as follows:

[0057]

[0058] The time series is made stationary by performing one or more order differences on it.

[0059] Step 300.5.2.3: Construct an ARIMA model. After the time series becomes stable, establish an ARIMA model. Indicates that is the autoregressive (AR) order, is the difference (D) order, is the moving average (MA) order; the ARIMA model is specifically:

[0060] In the formula, For the time series at time Observed value of is a constant term; is the autoregression coefficient; is the moving average coefficient; is white noise, and the model represents the current moment Observed value Compared with before The observed value at the moment and the previous The linear relationship between the residuals at each moment;

[0061] Step 300.5.2.4: Use maximum likelihood estimation to estimate the parameters in the ARIMA model. Suppose the time series is ,in , and assume that the series obeys the ARIMA model, given the parameters and the variance of the residual term , the likelihood function of the time series It can be expressed as:

[0062]

[0063] In the formula, the likelihood function represents the probability density of the observed data under given parameters;

[0064] The goal of maximum likelihood estimation is to find the parameter value that maximizes the likelihood function, take the negative logarithm of the likelihood function as the loss function, and find the parameter value that minimizes the loss function through the optimization algorithm. For the ARIMA model, the likelihood function of the normal distribution is used, so the negative logarithm of the likelihood function can be expressed as a negative log-likelihood function:

[0065]

[0066] In the formula, The goal is to minimize the negative log-likelihood function by adjusting the parameters Optimize and obtain the best parameter estimates. Based on maximum likelihood estimation, obtain the ARIMA model parameter values ​​that maximize the probability density of given data, thereby realizing the prediction of charging demand.

[0067] Step 300.5.3, construct the state representation, take the charging pile utilization rate and the charging demand prediction value as features, construct the state representation of the charging station, combine the features to form a vector, and represent the current state of the charging station, specifically:

[0068]

[0069] In the formula, Indicates the utilization rate of charging piles in the station, represents the predicted value of future charging demand, Represents the charging station state vector constructed by combining utilization rate and predicted value; Step 300.5.4, based on the fitted ARIMA model, predict the charging demand in the future period and obtain the prediction result, compare the prediction result with the actual result, evaluate the prediction performance of the model, and optimize the model in a targeted manner based on the difference between the two.

[0070] In some specific implementations, a deep reinforcement learning model is trained based on the collected historical data, a deep Q network is used, and model parameters are optimized through interaction with the environment so that the model learns the optimal operation strategy, specifically including:

[0071] Step 400.1, constructing a training loop for a deep Q-learning network;

[0072] Step 400.1.1, obtaining status based on the charging station environment;

[0073] Step 400.1.2, predicting the action value function based on the current state and the parameters of the network;

[0074] Step 400.1.3, selecting an action based on the action-value function;

[0075] Step 400.1.4, perform the action and observe the reward and next state;

[0076] Step 400.1.5, update the target value using the reward and the next state, and calculate the loss;

[0077] Step 400.1.6, storing the experience in the experience playback buffer, and using the loss to update the parameters of the network. During the training process, S400.1.1 to S400.1.6 is a complete cycle;

[0078] Step 400.2, training the deep Q network;

[0079] Step 400.2.1, randomly sampling a batch of experience data from the experience playback buffer;

[0080] Step 400.2.2, respectively calculate the target Q value and the current Q value, the specific calculation formula of the Q value is:

[0081]

[0082] in, is the current state, For the selected action, For the status Execute an action After receiving the instant reward, To perform actions The next state obtained after is a discount factor used to measure the importance of current rewards and future rewards, For the next state Select The action with the largest value The corresponding value;

[0083] Step 400.2.3, calculate the mean square error loss. The calculation formula of the mean square error loss is as follows:

[0084]

[0085] in, is the batch size, , , , Respectively represent the The state, action, immediate reward and next state of each sample. and has the same meaning as above;

[0086] Step 400.2.4, Clear The optimizer's gradient back-propagates the loss and updates the network parameters. Gradient descent is used to update the parameters. Specifically:

[0087]

[0088] In the formula, For Depth The parameters of the network, is the learning rate, is the gradient of the loss function with respect to the network parameters;

[0089] Step 400.3: Repeat steps 400.1 and 400.2 until the expected number of training times is reached and the expected loss converges.

[0090] In some specific implementations, the trained model is evaluated based on the validation data set, the performance of the model is evaluated, and the model is adjusted and optimized based on the evaluation results, specifically including:

[0091] Step 500.1: Separate an independent validation set from the data set obtained in step 100 to evaluate the model performance, and confirm that the validation set contains various representative situations and samples to ensure the accuracy of the evaluation results;

[0092] Step 500.2: Infer the trained model based on the validation data set and calculate performance indicators, specifically calculating accuracy, precision, recall and F1 score;

[0093] Step 500.2.1, calculation of accuracy, specifically:

[0094]

[0095] The calculation of precision is as follows:

[0096]

[0097] The calculation of recall rate is as follows:

[0098]

[0099] The calculation of F1 score is as follows:

[0100] Step 500.3, analyze the performance of the model on the validation set, identify its strengths and weaknesses, and determine the reasons for the performance differences;

[0101] Step 500.4: Based on the determined causes and the actual values ​​calculated, adjust and optimize the deficiencies of the model.

[0102] This application also provides an electric vehicle charging station operation and management system based on a virtual power plant, including:

[0103] A preprocessing unit is used to collect historical operation data of the charging station and classify the data after preprocessing, specifically classifying it into charging demand data, charging power data, energy supply data and customer satisfaction data;

[0104] A definition unit is used to model the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined;

[0105] Design unit, which is used to design deep reinforcement learning models using value functions as the basis of the model, including the structure of the neural network, state representation, and action selection strategy;

[0106] A construction unit is used to train a deep reinforcement learning model based on the collected historical data. The deep Q network is used to optimize the model parameters through interaction with the environment so that the model can learn the optimal operation strategy.

[0107] The evaluation unit is used to evaluate the trained model based on the validation data set, evaluate the performance of the model, and make targeted adjustments and optimizations to the model based on the evaluation results;

[0108] The optimization unit is used to deploy the optimized and adjusted model to the operation and management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

[0109] In some specific implementations, the value function is used as the basis of the model to design a deep reinforcement learning model, which specifically includes the structure of the neural network, state representation, and action selection strategy, specifically:

[0110] Define the state space of the charging station and determine the various factors that affect the operation of the charging station. The specific factors include the location of the charging station, the number of charging piles, the type and power of the charging piles, the usage status of the charging piles, the charging demand, and the energy supply. After the factors are determined, combine them to form a state space, which is used to describe the environmental status of the charging station for subsequent training and optimization of the model.

[0111] State space, let the state space be , a vector is used to represent the current state of the charging station environment, that is, , specifically:

[0112]

[0113] In the formula, are the dimensions in the state space, each of which represents a feature or variable in the charging station environment;

[0114] Define the action space of the charging station. The action space defines the decisions and operations that the charging station can take. During the definition process, three specific factors need to be considered: the scheduling of charging piles, the adjustment of charging rate, and the setting of charging price;

[0115] Action space, let the action space be , a vector is used to represent the decision and operation taken by the charging station, specifically:

[0116]

[0117] In the formula, are the dimensions in the action space, each of which represents a specific action or decision taken by the charging station;

[0118] Define the reward mechanism of the charging station. The reward mechanism defines the behavior evaluation criteria of the deep reinforcement learning model in the charging station environment. Through the reward mechanism, the model learns the degree of change of taking different actions in different states. When determining the reward mechanism, the reward mechanism is confirmed by maximizing the profit as the operating goal benchmark;

[0119] Reward mechanism, set in a certain state Take action The reward obtained is , then the reward mechanism is represented by a function, specifically:

[0120]

[0121] In the formula, this function describes the Take action The numerical reward obtained after

[0122] Establish a state transition model. The state transition model is used to describe the transition relationship between different states in the charging station environment. In this environment, the state of the charging station will change over time, and this change can be represented by the state transition model.

[0123] Assume the current state of the charging station is ,in Represents the time step, and the state transition model is specifically:

[0124]

[0125] In the formula, is in state The probability of taking an action under the current state Take action After that, the charging station will transfer to the next state The probability of

[0126] Based on the data benchmark, the charging station environment is modeled and the established model is verified to ensure that it can reflect the characteristics of the charging station environment. If the verification result is not ideal, the model needs to be adjusted, including adjusting the dimension of the state space, adjusting the range of the action space, and redefining the reward mechanism.

[0127] In some specific implementations, the value function is used as the basis of the model to design a deep reinforcement learning model, which specifically includes the structure of the neural network, state representation, and action selection strategy, specifically:

[0128] Based on the TensorFlow deep learning framework, the deep Q network learning algorithm is used to define the network structure. The deep Q network consists of an input layer, a hidden layer, and an output layer.

[0129] The size of the input layer matches the dimension of the state representation, the hidden layer uses a recurrent layer, and the size of the output layer matches the number of actions that the charging station can take;

[0130] Input Layer: ,in, is a vector containing the status of the charging station;

[0131] Hidden Layer: ,in, is the output of the hidden layer, is the activation function, and are the weights and biases of the hidden layer;

[0132] Output layer: ,in, is the estimate of the action value function, and are the weights and biases of the output layer;

[0133] After building the network structure, initialize the network parameters based on the TensorFlow framework. The module initializes network parameters;

[0134] Define the loss function. In deep Q learning, the mean square error loss function is used to measure the difference between the predicted value and the target value. The MSE loss function is specifically:

[0135]

[0136] In the formula, is the action value function output by the network, is the target value, is the sample size;

[0137] During the training process, we use The optimizer updates the parameters of the network, specifically:

[0138]

[0139] In the formula, As parameters, is the learning rate, is the gradient, is the square value of the exponentially weighted average of the gradient, Used to prevent the denominator from being zero.

[0140] The square of the exponentially weighted mean of the gradients The specific update rules are:

[0141]

[0142] In the formula, is the decay rate, which is close to 1 and is used to smooth the exponentially weighted average of the historical gradient square values;

[0143] The high-level features of charging pile utilization and charging demand prediction value extracted through feature engineering are used as state representation;

[0144] The epsilon-greedy strategy is used to select actions. This strategy has a certain probability In the other cases, the action with the highest value is chosen based on the current value function.

[0145] The electric vehicle charging station operation and management method based on a virtual power plant provided in the present application collects historical operation data of the charging station, and classifies the data after pre-processing, specifically classifying it into charging demand data, charging power data, energy supply data and customer satisfaction data, and models it based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined, and the value function is used as the basis of the model to design a deep reinforcement learning model, which specifically includes the structure of the neural network, state representation and action selection strategy. The deep reinforcement learning model is trained based on the collected historical data, and a deep Q network is used to optimize the model parameters through interaction with the environment so that the model learns the optimal operation strategy. The trained model is evaluated based on a verification data set, the performance of the model is evaluated, and the model is adjusted and optimized based on the evaluation results, and the optimized and adjusted model is deployed to the operation and management system of the charging station, and the status and environmental changes of the charging station are monitored in real time, and the feedback is fed back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0146] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0147] Figure 1 A flowchart of an electric vehicle charging station operation and management method based on a virtual power plant provided in an embodiment of the present application. DETAILED DESCRIPTION

[0148] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0149] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0150] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0151] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0152] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0153] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0154] Please refer to Figure 1 , a flowchart of an electric vehicle charging station operation and management method based on a virtual power plant provided in an embodiment of the present application.

[0155] like Figure 1 As shown, the electric vehicle charging station operation and management method based on the virtual power plant includes the following steps:

[0156] Step 100: Collect historical operation data of the charging station, and classify the data after pre-processing, specifically classifying it into charging demand data, charging power data, energy supply data and customer satisfaction data.

[0157] Here, specifically include:

[0158] Step 100.1, determine the type of data that needs to be collected, including charging demand data, charging power data, energy supply data and customer satisfaction data, determine the data collection cycle and frequency, specifically set to daily / 3 times, and configure data acquisition equipment for real-time recording and storage of data.

[0159] Step 100.2: Data cleaning and preprocessing: identifying and removing outliers, which specifically include erroneous data or abnormal charging behavior caused by backup failures; filling or deleting missing data to ensure data integrity and accuracy; and using smoothing technology to eliminate noise in the data to improve data quality.

[0160] Step 100.3: Classify the data. The charging demand data includes the charging demand, charging period and charging duration information. The charging power data includes the charging power and charging rate. The energy supply situation is recorded. The user's evaluation and feedback on the charging service is recorded, including the waiting time and charging speed.

[0161] Step 200: Modeling is performed based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined.

[0162] Here, specifically include:

[0163] Step 200.1, define the state space of the charging station, determine the various factors that affect the operation of the charging station, the specific factors include the location of the charging station, the number of charging piles, the type and power of the charging piles, the usage status of the charging piles, the charging demand and the energy supply. After the factors are determined, combine them to form a state space, which is used to describe the environmental status of the charging station for subsequent training and optimization of the model.

[0164] Step 200.1.1, state space, assuming the state space is , a vector is used to represent the current state of the charging station environment, that is, , specifically:

[0165]

[0166] In the formula, are the dimensions in the state space, each of which represents a feature or variable in the charging station environment.

[0167] Step 200.2, define the action space of the charging station. The action space defines the decisions and operations that the charging station can take. During the definition process, three specific factors need to be considered, namely, the scheduling of charging piles, the adjustment of charging rate, and the setting of charging price.

[0168] Step 200.2.1, action space, let the action space be , a vector is used to represent the decision and operation taken by the charging station, specifically:

[0169]

[0170] In the formula, These are the dimensions in the action space, each of which represents a specific action or decision taken by the charging station.

[0171] Step 200.3, define the reward mechanism of the charging station. The reward mechanism defines the behavior evaluation criteria of the deep reinforcement learning model in the charging station environment. Through the reward mechanism, the model learns the degree of change of taking different actions in different states. When determining the reward mechanism, the reward mechanism is confirmed by maximizing the profit as the operating goal benchmark.

[0172] Step 200.3.1, reward mechanism, set in a certain state Take action The reward obtained is , then the reward mechanism is represented by a function, specifically:

[0173]

[0174] In the formula, this function describes the Take action The numerical reward obtained after that.

[0175] Step 200.4: Establish a state transition model. The state transition model is used to describe the transition relationship between different states in the charging station environment. In this environment, the state of the charging station will change over time, and this change can be represented by the state transition model.

[0176] Assume the current state of the charging station is ,in Represents the time step, and the state transition model is specifically:

[0177]

[0178] In the formula, is in state The probability of taking an action under the current state Take action After that, the charging station will transfer to the next state probability.

[0179] Step 200.5: Based on the data benchmarks provided in steps 200.1 to 200.4, the charging station environment is modeled and the established model is verified to ensure that it can reflect the characteristics of the charging station environment. If the verification result is not ideal, the model needs to be adjusted, including adjusting the dimension of the state space, adjusting the range of the action space, and redefining the reward mechanism.

[0180] Step 300: Use the value function as the basis of the model to design a deep reinforcement learning model, specifically including the structure of the neural network, state representation, and action selection strategy.

[0181] Here, specifically include:

[0182] Step 300.1: Based on the TensorFlow deep learning framework, a deep Q network learning algorithm is used to define the network structure. The deep Q network consists of an input layer, a hidden layer, and an output layer.

[0183] The size of the input layer matches the dimensionality of the state representation, the hidden layers are recurrent, and the size of the output layer matches the number of actions the charging station can take.

[0184] Step 300.1.2, input layer: ,in is a vector containing the status of the charging station;

[0185] Hidden Layer: ,in, is the output of the hidden layer, is the activation function, and are the weights and biases of the hidden layer;

[0186] Output layer: ,in is the estimate of the action value function, and are the weights and biases of the output layer.

[0187] Step 300.2: After building the network structure, initialize the network parameters based on the TensorFlow framework. The module initializes network parameters.

[0188] Step 300.3, define the loss function. In deep Q learning, the mean square error loss function is used to measure the difference between the predicted value and the target value. The MSE loss function is specifically:

[0189]

[0190] In the formula, is the action value function output by the network, is the target value, is the sample size.

[0191] Step 300.4: During the training process, use The optimizer updates the parameters of the network, specifically:

[0192]

[0193] In the formula, As parameters, is the learning rate, is the gradient, is the square value of the exponentially weighted average of the gradient, Used to prevent the denominator from being zero.

[0194] The square of the exponentially weighted mean of the gradients The specific update rules are:

[0195]

[0196] In the formula, is the decay rate, which is close to 1 and is used to smooth the exponentially weighted average of the historical gradient square values.

[0197] Step 300.5: Use the high-level features charging pile utilization and charging demand prediction value extracted by feature engineering as state representation.

[0198] Here, specifically include:

[0199] Step 300.5.1, define the calculation method of charging pile utilization rate. Charging pile utilization rate is defined as the proportion of charging piles used in a certain time period. It is calculated by counting the usage of each charging pile in a certain time period. Specifically, it is:

[0200]

[0201] In the formula, in the formula, Represents the total number of times it is used within a certain period of time. Represents the total available time of the charging pile in a certain period of time. Represents the utilization rate of charging piles in the station;

[0202] Step 300.5.2: Use the ARIMA model to calculate the charging demand forecast value.

[0203] Step 300.5.2.1. Extract charging demand data of charging stations, including hourly or daily charging volume, and analyze the characteristics of time series to obtain trend, seasonality and cyclical characteristics of the data.

[0204] Step 300.5.2.2, stationarity test: In the ARIMA model, it is necessary to ensure that the time series is stationary. The stationarity test is implemented based on the difference operation.

[0205] Assume the original time series is , the difference operation is as follows:

[0206]

[0207] The time series is made stationary by performing one or more order differences on it.

[0208] Step 300.5.2.3: Construct an ARIMA model. After the time series becomes stable, establish an ARIMA model. Indicates that is the autoregressive (AR) order, is the difference (D) order, is the moving average (MA) order; the ARIMA model is specifically:

[0209] In the formula, For the time series at time Observed value of is a constant term; is the autoregression coefficient; is the moving average coefficient; is white noise, and the model represents the current moment Observed value Compared with before The observed value at the moment and the previous The linear relationship between the residuals at each moment.

[0210] Step 300.5.2.4: Use maximum likelihood estimation to estimate the parameters in the ARIMA model. Suppose the time series is ,in , and assume that the series obeys the ARIMA model, given the parameters and the variance of the residual term , the likelihood function of the time series It can be expressed as:

[0211]

[0212] Where, the likelihood function represents the probability density of the observed data under given parameters.

[0213] The goal of maximum likelihood estimation is to find the parameter value that maximizes the likelihood function, take the negative logarithm of the likelihood function as the loss function, and find the parameter value that minimizes the loss function through the optimization algorithm. For the ARIMA model, the likelihood function of the normal distribution is used, so the negative logarithm of the likelihood function can be expressed as a negative log-likelihood function:

[0214]

[0215] In the formula, The goal is to minimize the negative log-likelihood function by adjusting the parameters Optimization is performed to obtain the optimal parameter estimates. Based on maximum likelihood estimation, the ARIMA model parameter values ​​that maximize the probability density of given data are obtained, thereby realizing the prediction of charging demand.

[0216] Step 300.5.3, construct the state representation, take the charging pile utilization rate and the charging demand prediction value as features, construct the state representation of the charging station, combine the features to form a vector, and represent the current state of the charging station, specifically:

[0217]

[0218] In the formula, Indicates the utilization rate of charging piles in the station, represents the predicted value of future charging demand, Represents the charging station state vector constructed by combining utilization rate and predicted value; Step 300.5.4, based on the fitted ARIMA model, predict the charging demand in the future period and obtain the prediction result, compare the prediction result with the actual result, evaluate the prediction performance of the model, and optimize the model in a targeted manner based on the difference between the two.

[0219] Step 300.6: Use the epsilon-greedy strategy to select actions. This strategy has a certain probability In the other cases, the action with the highest value is chosen based on the current value function.

[0220] Step 400: Train a deep reinforcement learning model based on the collected historical data, using a deep Q network to optimize model parameters through interaction with the environment so that the model learns the optimal operation strategy.

[0221] Here, specifically include:

[0222] Step 400.1: Construct a training loop for the deep Q learning network.

[0223] Step 400.1.1: Obtain status based on the charging station environment.

[0224] Step 400.1.2: Predict the action-value function based on the current state and network parameters.

[0225] Step 400.1.3: Select an action based on the action-value function.

[0226] Step 400.1.4, perform the action and observe the reward and next state.

[0227] Step 400.1.5: Update the target value using the reward and the next state, and calculate the loss.

[0228] Step 400.1.6: store the experience in the experience playback buffer and use the loss to update the parameters of the network. During the training process, S400.1.1 to S400.1.6 constitute a complete cycle.

[0229] Step 400.2: Train the deep Q network.

[0230] Step 400.2.1: Randomly sample a batch of experience data from the experience playback buffer.

[0231] Step 400.2.2, respectively calculate the target Q value and the current Q value, the specific calculation formula of the Q value is:

[0232]

[0233] in, is the current state, For the selected action, For the status Execute an action After receiving the instant reward, To perform actions The next state obtained after is a discount factor used to measure the importance of current rewards and future rewards, For the next state Select The action with the largest value The corresponding value.

[0234] Step 400.2.3, calculate the mean square error loss. The calculation formula of the mean square error loss is as follows:

[0235]

[0236] in, is the batch size, , , , Respectively represent the The state, action, immediate reward and next state of each sample. and Same meaning as above.

[0237] Step 400.2.4, Clear The optimizer's gradient back-propagates the loss and updates the network parameters. Gradient descent is used to update the parameters. Specifically:

[0238]

[0239] In the formula, For Depth The parameters of the network, is the learning rate, is the gradient of the loss function with respect to the network parameters.

[0240] Step 400.3: Repeat steps 400.1 and 400.2 until the expected number of training times is reached and the expected loss converges.

[0241] Step 500: Evaluate the trained model based on the validation data set, evaluate the performance of the model, and make targeted adjustments and optimizations to the model based on the evaluation results.

[0242] Here, specifically include:

[0243] Step 500.1: Separate an independent validation set from the data set obtained in step 100 to evaluate the model performance, and confirm that the validation set contains various representative situations and samples to ensure the accuracy of the evaluation results.

[0244] Step 500.2: Infer the trained model based on the validation data set and calculate performance indicators, specifically the accuracy, precision, recall and F1 score.

[0245] Step 500.2.1, calculation of accuracy, specifically:

[0246]

[0247] The calculation of precision is as follows:

[0248]

[0249] The calculation of recall rate is as follows:

[0250]

[0251] The calculation of F1 score is as follows:

[0252] Step 500.3: Analyze the performance of the model on the validation set, identify its strengths and weaknesses, and determine the reasons for the performance differences.

[0253] Step 500.4: Based on the determined causes and the actual values ​​calculated, adjust and optimize the deficiencies of the model.

[0254] Step 600: Deploy the optimized and adjusted model to the operation management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed it back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

[0255] The electric vehicle charging station operation and management method based on a virtual power plant provided in the present application collects historical operation data of the charging station, and classifies the data after pre-processing, specifically classifying it into charging demand data, charging power data, energy supply data and customer satisfaction data, and models it based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined, and the value function is used as the basis of the model to design a deep reinforcement learning model, which specifically includes the structure of the neural network, state representation and action selection strategy. The deep reinforcement learning model is trained based on the collected historical data, and a deep Q network is used to optimize the model parameters through interaction with the environment so that the model learns the optimal operation strategy. The trained model is evaluated based on a verification data set, the performance of the model is evaluated, and the model is adjusted and optimized based on the evaluation results, and the optimized and adjusted model is deployed to the operation and management system of the charging station, and the status and environmental changes of the charging station are monitored in real time, and the feedback is fed back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

[0256] This application also provides an electric vehicle charging station operation and management system based on a virtual power plant, including:

[0257] The preprocessing unit is used to collect the historical operation data of the charging station and classify the data after preprocessing, specifically classifying it into charging demand data, charging power data, energy supply data and customer satisfaction data.

[0258] The definition unit is used to model based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined.

[0259] The design unit is used to design a deep reinforcement learning model using the value function as the basis of the model, including the structure of the neural network, state representation, and action selection strategy.

[0260] The construction unit is used to train a deep reinforcement learning model based on the collected historical data. The deep Q network is used to optimize the model parameters through interaction with the environment so that the model can learn the optimal operation strategy.

[0261] The evaluation unit is used to evaluate the trained model based on the validation data set, evaluate the performance of the model, and make targeted adjustments and optimizations to the model based on the evaluation results.

[0262] The optimization unit is used to deploy the optimized and adjusted model to the operation and management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

[0263] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

[0264] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0265] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An electric vehicle charging station operation and management method based on a virtual power plant, characterized in that: The steps include: S100, collecting historical operation data of the charging station, and classifying the data after pre-processing, specifically classifying the data into charging demand data, charging power data, energy supply data and customer satisfaction data; S200, modeling based on the charging station environment, during which the state space, action space and reward mechanism of the charging station application environment are defined; S300, using the value function as the basis of the model, designing a deep reinforcement learning model, including the structure of the neural network, state representation, and action selection strategy; S400, training a deep reinforcement learning model based on the collected historical data, using a deep Q network, optimizing model parameters through interaction with the environment, so that the model learns the optimal operation strategy; S500, evaluating the trained model based on the validation data set, evaluating the performance of the model, and making targeted adjustments and optimizations to the model based on the evaluation results; S600, deploy the optimized and adjusted model to the operation and management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed it back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

2. The method for operating and managing an electric vehicle charging station based on a virtual power plant according to claim 1, characterized in that: Modeling is performed based on the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined, including: S200.

1. Define the state space of the charging station and determine the various factors that affect the operation of the charging station. The specific factors include the location of the charging station, the number of charging piles, the type and power of the charging piles, the usage status of the charging piles, the charging demand, and the energy supply. After the factors are determined, combine them to form a state space, which is used to describe the environmental status of the charging station for subsequent training and optimization of the model. S200.1.1, state space, assuming the state space is , a vector is used to represent the current state of the charging station environment, that is, , specifically: In the formula, are the dimensions in the state space, each of which represents a feature or variable in the charging station environment; S200.2, define the action space of the charging station. The action space defines the decisions and operations that the charging station can take. During the definition process, three specific factors need to be considered, namely, the scheduling of charging piles, the adjustment of charging rate, and the setting of charging price; S200.2.1, Action Space, Suppose the action space is , a vector is used to represent the decision and operation taken by the charging station, specifically: In the formula, are the dimensions in the action space, each of which represents a specific action or decision taken by the charging station; S200.

3. Define the reward mechanism for charging stations. The reward mechanism defines the behavior evaluation criteria of the deep reinforcement learning model in the charging station environment. Through the reward mechanism, the model learns the degree of change of taking different actions in different states. When determining the reward mechanism, the reward mechanism is confirmed by using maximizing the profit as the operating target benchmark; S200.3.1, reward mechanism, set in a certain state Take action The reward obtained is , then the reward mechanism is represented by a function, specifically: In the formula, this function describes the Take action The numerical reward obtained after S200.

4. Establish a state transition model. The state transition model is used to describe the transition relationship between different states in the charging station environment. In this environment, the state of the charging station will change over time. This change can be represented by the state transition model. Assume the current state of the charging station is ,in Represents the time step, and the state transition model is specifically: In the formula, is in state The following actions are taken, Indicates the current state Take action After that, the charging station will transfer to the next state probability; S200.

5. Based on the data benchmarks provided by S200.1 to S200.4, the charging station environment is modeled and the established model is verified to ensure that it can reflect the characteristics of the charging station environment. If the verification result is not ideal, the model needs to be adjusted, including adjusting the dimension of the state space, adjusting the range of the action space, and redefining the reward mechanism.

3. The method for operating and managing an electric vehicle charging station based on a virtual power plant according to claim 2, characterized in that: Using the value function as the basis of the model, a deep reinforcement learning model is designed, including the structure of the neural network, state representation, and action selection strategy, including: S300.

1. Based on the TensorFlow deep learning framework, the deep Q network learning algorithm is used to define the network structure. The deep Q network consists of an input layer, a hidden layer, and an output layer. The size of the input layer matches the dimension of the state representation, the hidden layer uses a recurrent layer, and the size of the output layer matches the number of actions that the charging station can take; S300.1.2, Input layer: ,in is a vector containing the status of the charging station; Hidden Layer: ,in, is the output of the hidden layer, is the activation function, and are the weights and biases of the hidden layer; Output layer: ,in is the estimate of the action value function, and are the weights and biases of the output layer; S300.

2. After building the network structure, initialize the network parameters based on the TensorFlow framework. The module initializes network parameters; S300.

3. Define the loss function. In deep Q learning, the mean square error loss function is used to measure the difference between the predicted value and the target value. The MSE loss function is specifically: In the formula, is the action value function output by the network, is the target value, is the sample size; S300.4, during the training process, The optimizer updates the parameters of the network, specifically: In the formula, As parameters, is the learning rate, is the gradient, is the square value of the exponentially weighted average of the gradient, Used to prevent the denominator from being zero; The square of the exponentially weighted mean of the gradients The specific update rules are: In the formula, is the decay rate, which is close to 1 and is used to smooth the exponentially weighted average of the historical gradient square values; S300.5, using the high-level features charging pile utilization and charging demand prediction value extracted through feature engineering as state representation; S300.6, use the epsilon-greedy strategy to select actions. This strategy has a certain probability In the other cases, the action with the highest value is chosen based on the current value function.

4. The method for operating and managing an electric vehicle charging station based on a virtual power plant according to claim 3, characterized in that: The high-level features of charging pile utilization and charging demand prediction values ​​extracted through feature engineering are used as state representation, including: S300.5.

1. Define the calculation method of charging pile utilization rate. Charging pile utilization rate is defined as the proportion of charging piles used in a certain period of time. It is calculated by counting the usage of each charging pile in a certain period of time. Specifically: In the formula, Represents the total number of times it is used within a certain period of time. Represents the total available time of the charging pile in a certain period of time. Represents the utilization rate of charging piles in the station; S300.5.

2. Use the ARIMA model to calculate the charging demand forecast value; S300.5.2.

1. Extract charging demand data of charging stations, including hourly or daily charging volume, and analyze the characteristics of time series to obtain trend, seasonality and cyclical characteristics of the data; S300.5.2.2, Stationarity test: In the ARIMA model, it is necessary to ensure that the time series is stationary. The stationarity test is implemented based on the difference operation; Assume the original time series is , the differential operation is as follows: By performing first-order or multi-order difference operations on the time series, the data is made stable; S300.5.2.3, constructing the ARIMA model. After the time series becomes stable, the ARIMA model is established. The ARIMA model is used Indicates that is the autoregressive (AR) order, is the difference (D) order, is the moving average (MA) order; the ARIMA model is specifically: In the formula, For a time series at time Observed value of is a constant term; is the autoregression coefficient; is the moving average coefficient; is white noise, and the model represents the current moment Observed value Compared with before The observed value at the moment and the previous The linear relationship between the residuals at each moment; S300.5.2.4, use maximum likelihood estimation to estimate the parameters in the ARIMA model, assuming that the time series is ,in , and assume that the series obeys the ARIMA model, given the parameters and the variance of the residual term , the likelihood function of the time series It can be expressed as: In the formula, the likelihood function represents the probability density of the observed data under given parameters; The goal of maximum likelihood estimation is to find the parameter value that maximizes the likelihood function, take the negative logarithm of the likelihood function as the loss function, and find the parameter value that minimizes the loss function through the optimization algorithm. For the ARIMA model, the likelihood function of the normal distribution is used, so the negative logarithm of the likelihood function can be expressed as a negative log-likelihood function: In the formula, The goal is to minimize the negative log-likelihood function by adjusting the parameters Optimize and obtain the best parameter estimates. Based on maximum likelihood estimation, obtain the ARIMA model parameter values ​​that maximize the probability density of given data, thereby realizing the prediction of charging demand. S300.5.3, state representation construction, using the charging pile utilization rate and the charging demand prediction value as features, constructing the state representation of the charging station, combining the features to form a vector, representing the current state of the charging station, specifically: In the formula, Indicates the utilization rate of charging piles in the station, represents the predicted value of future charging demand, represents the charging station state vector constructed by combining utilization and prediction values; S300.5.

4. Predict the charging demand in the future based on the fitted ARIMA model and obtain the prediction results. Compare the prediction results with the actual results to evaluate the prediction performance of the model, and optimize the model in a targeted manner based on the difference between the two.

5. The method for operating and managing an electric vehicle charging station based on a virtual power plant according to claim 4, characterized in that: Based on the collected historical data, a deep reinforcement learning model is trained. A deep Q network is used to optimize the model parameters through interaction with the environment so that the model can learn the optimal operation strategy, including: S400.

1. Construct a training loop for a deep Q-learning network. S400.1.

1. Obtaining status based on the charging station environment; S400.1.2, predicting the action value function based on the current state and network parameters; S400.1.3, select actions based on action value function; S400.1.

4. Perform actions and observe rewards and next states; S400.1.5, update the target value using the reward and the next state, and calculate the loss; S400.1.6, storing the experience in the experience playback buffer, and using the loss to update the parameters of the network. During the training process, S400.1.1 to S400.1.6 is a complete cycle; S400.

2. Training deep Q network; S400.2.

1. Randomly sample a batch of experience data from the experience playback buffer; S400.2.

2. Calculate the target Q value and the current Q value respectively. The specific calculation formula of the Q value is: in, is the current state, For the selected action, For the status Execute an action After receiving the instant reward, To perform actions The next state obtained after is a discount factor used to measure the importance of current rewards and future rewards, For the next state Select The action with the largest value The corresponding value; S400.2.

3. Calculate the mean square error loss. The calculation formula of the mean square error loss is as follows: in, is the batch size, , , , Respectively represent the The state, action, immediate reward and next state of each sample. and has the same meaning as above; S400.2.4, Clear The optimizer's gradient back-propagates the loss and updates the network parameters. Gradient descent is used to update the parameters. Specifically: In the formula, For Depth The parameters of the network, is the learning rate, is the gradient of the loss function with respect to the network parameters; S400.

3. Repeat S400.1 and S400.2 until the expected number of training times is reached and the expected loss converges.

6. The method for operating and managing an electric vehicle charging station based on a virtual power plant according to claim 5, characterized in that: Evaluate the trained model based on the validation data set, evaluate the model's performance, and make targeted adjustments and optimizations to the model based on the evaluation results, including: S500.

1. Separate an independent validation set from the data set obtained in S100 to evaluate the model performance and confirm that the validation set contains a variety of representative situations and samples to ensure the accuracy of the evaluation results. S500.

2. Perform inference on the trained model based on the validation data set and calculate performance indicators, specifically accuracy, precision, recall, and F1 score; S500.2.

1. Calculation of accuracy, specifically: The calculation of precision is as follows: The calculation of recall rate is as follows: The calculation of F1 score is as follows: S500.

3. Analyze the performance of the model on the validation set, identify its strengths and weaknesses, and determine the reasons for performance differences; S500.

4. Based on the determined causes and actual calculated values, adjust and optimize the deficiencies of the model.

7. The electric vehicle charging station operation and management system based on virtual power plant is characterized by: include: A preprocessing unit is used to collect historical operation data of the charging station and classify the data after preprocessing, specifically classifying it into charging demand data, charging power data, energy supply data and customer satisfaction data; A definition unit is used to model the charging station environment. During the modeling process, the state space, action space and reward mechanism of the charging station application environment are defined; Design unit, which is used to design deep reinforcement learning models using value functions as the basis of the model, including the structure of the neural network, state representation, and action selection strategy; A construction unit is used to train a deep reinforcement learning model based on the collected historical data. The deep Q network is used to optimize the model parameters through interaction with the environment so that the model can learn the optimal operation strategy. The evaluation unit is used to evaluate the trained model based on the validation data set, evaluate the performance of the model, and make targeted adjustments and optimizations to the model based on the evaluation results; The optimization unit is used to deploy the optimized and adjusted model to the operation and management system of the charging station, monitor the status and environmental changes of the charging station in real time, and feed back to the deep reinforcement learning model for retraining to continuously optimize the operation strategy.

8. The electric vehicle charging station operation and management system based on a virtual power plant according to claim 7, characterized in that: Using the value function as the basis of the model, a deep reinforcement learning model is designed, including the structure of the neural network, state representation, and action selection strategy, specifically: Define the state space of the charging station and determine the various factors that affect the operation of the charging station. The specific factors include the location of the charging station, the number of charging piles, the type and power of the charging piles, the usage status of the charging piles, the charging demand, and the energy supply. After the factors are determined, combine them to form a state space, which is used to describe the environmental status of the charging station for subsequent training and optimization of the model. State space, let the state space be , a vector is used to represent the current state of the charging station environment, that is, , specifically: In the formula, are the dimensions in the state space, each of which represents a feature or variable in the charging station environment; Define the action space of the charging station. The action space defines the decisions and operations that the charging station can take. During the definition process, three specific factors need to be considered: the scheduling of charging piles, the adjustment of charging rate, and the setting of charging price; Action space, let the action space be , a vector is used to represent the decision and operation taken by the charging station, specifically: In the formula, are the dimensions in the action space, each of which represents a specific action or decision taken by the charging station; Define the reward mechanism of the charging station. The reward mechanism defines the behavior evaluation criteria of the deep reinforcement learning model in the charging station environment. Through the reward mechanism, the model learns the degree of change of taking different actions in different states. When determining the reward mechanism, the reward mechanism is confirmed by maximizing the profit as the operating goal benchmark; Reward mechanism, set in a certain state Take action The reward obtained is , then the reward mechanism is represented by a function, specifically: In the formula, this function describes the Take action The numerical reward obtained after Establish a state transition model. The state transition model is used to describe the transition relationship between different states in the charging station environment. In this environment, the state of the charging station will change over time, and this change can be represented by the state transition model. Assume the current state of the charging station is ,in Represents the time step, and the state transition model is specifically: In the formula, is in state The following actions are taken, Indicates the current state Take action After that, the charging station will transfer to the next state probability; Based on the data benchmark, the charging station environment is modeled and the established model is verified to ensure that it can reflect the characteristics of the charging station environment. If the verification result is not ideal, the model needs to be adjusted, including adjusting the dimension of the state space, adjusting the range of the action space, and redefining the reward mechanism.

9. The electric vehicle charging station operation and management system based on a virtual power plant according to claim 7, characterized in that: Using the value function as the basis of the model, a deep reinforcement learning model is designed, including the structure of the neural network, state representation, and action selection strategy, specifically: Based on the TensorFlow deep learning framework, the deep Q network learning algorithm is used to define the network structure. The deep Q network consists of an input layer, a hidden layer, and an output layer. The size of the input layer matches the dimension of the state representation, the hidden layer uses a recurrent layer, and the size of the output layer matches the number of actions that the charging station can take; Input Layer: ,in is a vector containing the status of the charging station; Hidden Layer: ,in, is the output of the hidden layer, is the activation function, and are the weights and biases of the hidden layer; Output layer: ,in is the estimate of the action value function, and are the weights and biases of the output layer; After building the network structure, initialize the network parameters based on the TensorFlow framework. The module initializes network parameters; Define the loss function. In deep Q learning, the mean square error loss function is used to measure the difference between the predicted value and the target value. The MSE loss function is specifically: In the formula, is the action value function output by the network, is the target value, is the sample size; During the training process, we use The optimizer updates the parameters of the network, specifically: In the formula, As parameters, is the learning rate, is the gradient, is the square value of the exponentially weighted average of the gradient, Used to prevent the denominator from being zero; The square of the exponentially weighted mean of the gradients The specific update rules are: In the formula, is the decay rate, which is close to 1 and is used to smooth the exponentially weighted average of the historical gradient square values; The high-level features of charging pile utilization and charging demand prediction value extracted through feature engineering are used as state representation; The epsilon-greedy strategy is used to select actions. This strategy has a certain probability In the other cases, the action with the highest value is chosen based on the current value function.

Images