Dynamic planning model for bidirectional interaction between electric vehicle and power grid

Abstract

The invention provides a dynamic programming model (BDPIM) for bidirectional interaction between an electric vehicle and a power grid, and aims to realize real-time optimization of an electric vehicle charging and discharging strategy and power grid load scheduling by combining multi-period dynamic programming and a deep reinforcement learning technology, take both user cost minimization and power grid income maximization into account, and improve the efficiency of the power grid. The intelligent power grid operation efficiency and the user experience are improved. The method comprises the steps of data collection and preprocessing, model construction, multi-period dynamic planning solving, strategy optimization and feedback mechanism and reinforcement learning iterative optimization. A multi-period charging and discharging strategy is generated through staged dynamic planning, a decision is adjusted in real time in combination with a deep Q network (DQN) and an Actor-Critic architecture, power grid load prediction and a user behavior model are fused, a bidirectional interactive optimal scheduling scheme can be output in real time, the scheduling strategy is continuously optimized through deep reinforcement learning, and closed-loop improvement is formed. Compared with the prior art, the power grid load fluctuation and the user behavior dynamic state can be accurately captured, the adaptability and the economic benefit of the system are improved, and the method has remarkable application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart grid and electric vehicle charging and discharging optimization, in particular to a dynamic programming model for two-way interaction between electric vehicles and power grids. The method realizes real-time optimization and collaborative decision-making of electric vehicle charging and discharging strategy and power grid load scheduling by combining multi-period dynamic programming and deep reinforcement learning, thereby effectively improving the operation efficiency of smart grids and the charging and discharging experience of users. BACKGROUND

[0002] With the transformation of global energy structure and the promotion of carbon neutralization goals, electric vehicles (EVs) have gained rapid development due to their zero-emission and high-energy efficiency advantages. After a large number of electric vehicles are connected to the power grid, not only does it exacerbate the load fluctuation of the power grid, but also it provides the possibility for a new energy management model of "vehicle-to-grid" two-way interaction. However, the existing charging scheduling system mainly relies on static rules or traditional optimization algorithms, which is difficult to meet the complex and changeable electricity demand and power grid operation constraints after the large-scale access of electric vehicles.

[0003] The current mainstream electric vehicle charging and discharging scheduling method mainly includes simple heuristic strategies based on fixed time-of-use (TOU) electricity prices and static optimization models based on linear or mixed integer programming. Such methods usually schedule with a single objective (such as cost minimization or revenue maximization), and rely on centralized background analysis of historical data, lacking effective response to real-time load fluctuations, user behavior preferences, and dynamic changes in battery state, which can lead to scheduling lag, low resource utilization, and poor user experience.

[0004] In addition, the charging and discharging behavior of electric vehicles is influenced by various factors, including but not limited to user travel plans, dynamic electricity prices, geographic distribution of charging stations, battery health status, and power grid load prediction errors. Existing methods are difficult to deeply integrate and collaboratively decide on the above multi-source heterogeneous data while ensuring the safe and stable operation of the power grid, and cannot balance the dual optimization needs of user cost and grid revenue.

[0005] In recent years, dynamic programming and deep reinforcement learning have shown strong decision-making capabilities in complex system scheduling, enabling them to seek global optimal solutions in continuous and phased decision-making problems, and continuously optimize strategies through online learning. However, pure dynamic programming faces the problem of state space explosion, and traditional deep reinforcement learning models are difficult to directly handle multi-stage, multi-objective coupled optimization tasks. How to organically combine the precise modeling advantages of dynamic programming with the real-time adaptive capabilities of deep reinforcement learning to build a "vehicle-to-grid" two-way interaction scheduling model that can not only finely describe the multi-period decision-making process but also realize closed-loop iterative optimization in actual operation is a key technical problem that needs to be solved. SUMMARY

[0006] To overcome the strong static nature of the existing charging and discharging scheduling scheme and the difficulty in balancing user cost and grid benefit, the present application provides a bidirectional dynamic programming interaction model (BDPIM) for the interaction between electric vehicles and power grids. The model can overcome the limitations of offline analysis and single-target optimization of traditional scheduling methods, and through the combination of multi-period dynamic programming and deep reinforcement learning, it realizes real-time optimization and closed-loop iteration of charging and discharging strategies and power grid load scheduling, significantly improving the efficiency of smart grid operation and the user charging and discharging experience.

[0007] The specific technical solutions are as follows:

[0008] A bidirectional dynamic programming interaction model for electric vehicles and power grids, comprising the following steps:

[0009] 1) Data collection and preprocessing:

[0010] Collect power grid operation data, electric vehicle characteristic data, and user charging and discharging behavior data;

[0011] Perform missing value filling and multiple interpolation processing on the collected data, and use

[0012]

[0013] wherein, is the original data, is the mean of the data, is the standard deviation of the data, ensuring standardized processing of the data.

[0014] 2) Construction of multi-period dynamic programming model:

[0015] Divide a day into charging and discharging stages, define stage state , wherein is the state of charge of the battery, is the predicted load;

[0016] Define stage action ;

[0017] State transition and reward function are:

[0018]

[0019] wherein, is the state of stage ; is the stage action; is the state transition function; is the stage reward function; is the grid revenue weight coefficient, is the grid revenue increment; is the user cost weight coefficient, is the user cost increment.

[0020] 3) Deep reinforcement learning online optimization:

[0021] Based on the dynamic programming solution, combined with deep Q-network (DQN) and Actor-Critic architecture for online fine-tuning of strategy, the following Bellman formula is used:

[0022]

[0023] where, is the action value function, representing the expected cumulative revenue of performing action in state ; is the discount factor; is the optimal action value of the next state .

[0024] 4) User behavior modeling:

[0025] Based on the user utility function and Logit model to predict the charging and discharging willingness, the utility function is expressed as:

[0026]

[0027] where, is the utility value of the user under action ; is the expected revenue; is the corresponding cost; and are the weight parameters of revenue and cost, respectively.

[0028] 5) Multi-objective optimization and Pareto frontier analysis:

[0029] Under the constraints of meeting the user cost minimization target and the grid revenue maximization target , the Pareto frontier set is defined as:

[0030]

[0031] where, is the user cost target, is the grid revenue target, is the solution set that meets the Pareto optimality.

[0032] 6) Real-time feedback and closed-loop improvement:

[0033] Real-time feedback such as user charging completion time and charging station selection is obtained through the smart grid control platform, and the deep Q learning continuous update strategy is used to realize dynamic adjustment and closed-loop optimization of charging and discharging scheduling.

[0034] Preferably, the data preprocessing step includes:

[0035] Missing values are filled using the Multiple Imputation method;

[0036] All numerical features are normalized;

[0037] Load features and battery characteristic features based on time series are extracted.

[0038] Preferably, the state transition function in the dynamic programming model is constructed in combination with a battery charging and discharging efficiency model and grid scheduling constraints;

[0039] Preferably, in the deep reinforcement learning algorithm, the Actor network outputs an action probability distribution, the Critic network evaluates the action value, and a target network and experience replay are used to alleviate training instability.

[0040] Compared with the prior art, the present application can:

[0041] Simultaneously consider user cost and grid revenue, and provide a globally optimal scheduling scheme for two-way interaction;

[0042] Combining dynamic programming and deep reinforcement learning ensures scheduling accuracy and online adaptive ability;

[0043] Based on user utility and real-time feedback, a closed-loop optimization is formed, significantly improving system adaptability and user satisfaction;

[0044] Effectively reduces charging waiting time and improves charging and discharging efficiency, bringing obvious economic and social benefits to electric vehicle users and grid operators. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0046] Figure 1 The technical solution flowchart of the present application. DETAILED DESCRIPTION

[0047] In order to better illustrate the present application, the technical solutions of the present application are described in detail in combination with specific embodiments. The following content is the detailed steps of the specific embodiments and the corresponding implementation details, which aims to help understand how the present application solves the problems existing in the prior art, and improves the intelligence, adaptability and economic benefit of the electric vehicle and power grid two-way interactive scheduling model.

[0048] S1. Data collection and preprocessing

[0049] (1) Data collection

[0050] Through the SCADA system or power exchange API, obtain the base load and peak-valley load curve, historical and real-time electricity price;

[0051] Through the vehicle-mounted OBD interface or the back-end management platform, collect the electric vehicle SOC, maximum charging and discharging power, and vehicle model;

[0052] Through the mobile App or the pile station background, record the start / end time of user charging and discharging, charging location (latitude and longitude), duration and cost;

[0053] Pull the geographic location, number of charging piles and real-time load status of each site from the charging operator database;

[0054] Interface with weather and traffic APIs to obtain weather conditions (temperature, humidity, precipitation probability) and real-time traffic flow;

[0055] Through the GPS module or map service, accurately locate the latitude and longitude of the user and each charging station (error ≤ ±5 m).

[0056] Through the above steps, multi-source raw data covering the power grid, vehicles, user behavior, charging stations and environment are obtained.

[0057] (2) Data aggregation and storage

[0058] After obtaining the raw data, align the various data by timestamp, and write into the time series database (InfluxDB) and relational database (PostgreSQL) with batch (batch) or streaming (streaming) ETL process;

[0059] Store in the database by "day / hour" partitioning, and label backup for each complete collection and preprocessing data version

[0060] At this stage, all raw data has been uniformly managed and versioned, providing guarantee for subsequent efficient query and backtracking.

[0061] (3) Data preprocessing

[0062] 1) Missing value imputation

[0063] After the data aggregation, the missing values are estimated first.

[0064] Multiple imputation by chained equations (MICE) is adopted to generate 5 sets of candidate imputation values based on predictive mean matching (PMM), and the median is taken as the final imputation result.

[0065] 2) Numerical standardization

[0066] After obtaining the data without missing values, the next step is to unify the dimension of numerical features.

[0067]

[0068] where, is the original feature value, is the mean, is the standard deviation, ensuring that each numerical feature is in the same dimension.

[0069] 3) Time series feature extraction

[0070] After standardization, the time series rules are mined.

[0071] From the unified timestamp, multi-dimensional time labels such as "hour (0-23)", "weekday / weekend (binary encoding)", "quarter (1-4)", and "intra-day cycle (0-23)" are extracted.

[0072] 4) Spatial feature construction

[0073] After enriching the time series features, the location factor is added.

[0074]

[0075] where, and are the latitude and longitude of the user and charging station, respectively, and the Euclidean distance is calculated after projection to quantify the accessibility of the user to each station.

[0076] 5) Anomaly data cleaning

[0077] After obtaining the space-time features, the data quality is further checked.

[0078] Box plot (IQR) is used to remove abnormal samples with charging time <1 min or >24 h, cost or SOC exceeding 3 times the standard deviation.

[0079] 6) Label generation and format conversion

[0080] ​After the cleaning is completed, the input / output format required for model training is constructed.

[0081] The next actual charging station ID is one-hot encoded, and the charging time is encoded as a continuous numerical value; and the final data set is exported in CSV or Parquet format for downstream models.

[0082] Through the above layer-by-layer processing procedures, the system finally obtains a structured, non-missing, non-noisy, and rich in spatio-temporal semantic multi-dimensional feature data set, providing a solid data foundation for subsequent user behavior modeling and recommendation algorithm training.

[0083] S2. User behavior modeling

[0084] After completing data preprocessing, enter the user behavior modeling phase, which is divided into four sub-steps: "input feature construction", "pre-training large model training", "sequence context embedding", and "fine-tuning behavior embedding generation". The specific steps are as follows:

[0085] (1) Input feature construction

[0086] 1) From the multi-dimensional spatio-temporal feature set obtained in S1, extract the subset directly related to user charging and discharging, including:

[0087] Time features: hour, weekday / weekend, intra-day cycle ( encoding);

[0088] Spatial features: distance between user and charging station ;

[0089] Behavioral features: charging duration, charging cost;

[0090] Station features: charging station ID (one-hot encoded) and regional clustering label.

[0091] Vectorize the above features to form a sequence input , where represents the multi-dimensional features of the th charging record.

[0092] (2) Pre-training large model training

[0093] Use the Transformer architecture to pre-train a large number of user charging sequences in a self-supervised manner;

[0094] Masked prediction task: randomly mask some of the feature dimensions in , the model reconstructs the masked part through the remaining information, and the optimization goal is to minimize the reconstruction error;

[0095] After several rounds of iterations, the model can automatically capture the potential rules and nonlinear interactions in the behavior sequence, laying the foundation for downstream fine-tuning.

[0096] (3) Sequence context embedding

[0097] On the basis of the pre-trained model, the sequence input is gradually sent into the Transformer layer for context fusion:

[0098]

[0099] where, is the context embedding of the previous moment, is the context representation of the current moment, and ;

[0100] Each layer of self-attention mechanism enables the model to capture short-term and long-term dependencies of user behavior in the global range of the sequence, outputting the sequence .

[0101] (4) Fine-tuning to generate behavior embedding

[0102] On the basis of the sequence context embedding, a prediction head is added for the "next charging station" and "next charging time" dual-task fine-tuning;

[0103] The objective function consists of cross-entropy loss and mean square error loss:

[0104]

[0105] where, is the true charging station label, is the predicted probability; , are the true and predicted charging times, respectively;

[0106] After fine-tuning, the context vector of the last moment or the low-dimensional vector obtained by full connection mapping is taken as the final user behavior embedding .

[0107] After obtaining the user behavior embedding in S2, the geographical information and electricity price time series are modeled in turn, and after dimension alignment, a unified fusion vector is formed for subsequent personalized recommendation models. The specific steps include:

[0108] (1) Time alignment and batch processing organization

[0109] 1) Align each modality to a uniform time granularity (e.g., 5-15 minutes): resample the price series, and time-window map the event-driven behavior records;

[0110] 2) Fill in the missing time steps with forward fill or linear interpolation, and associate each time step with the "most recent snapshot" of the site state;

[0111] 3) Construct mini-batches per time step, each containing samples of the same time instant, ensuring that the inputs are strictly consistent in time.

[0112] (2) Geographical feature modeling (GNN)

[0113] 1) Graph construction and edge weight definition:

[0114]

[0115] where, is the geographical distance between and , is a temperature coefficient to control the distance decay speed.

[0116] 2) Message passing and node update:

[0117]

[0118] where, is the feature of the site at the th layer, is the trainable weight matrix, is the nonlinear activation function, is the neighbor set, is the normalized adjacency weight.

[0119] 3) User-site reachability injection and aggregation:

[0120]

[0121] where, is the output of the last layer of the GNN, is the distance feature from the user to the site , is the combination of small fully connected mapping and normalization, denotes the aggregation operation (e.g., average pooling / max pooling / attention pooling), is the geographical embedding related to the current user scenario.

[0122] (3) Dynamic modeling of electricity prices (LSTM)

[0123] 1) Construction length is Sliding window:

[0124]

[0125] In the formula, For a moment Electricity price, This is the historical electricity price window for the current moment.

[0126] 2) Sequence encoding using time stamps:

[0127]

[0128] In the formula, Including "hours, weekdays / weekends, Time feature encoding such as "intraday cycle" The output is the hidden state at the last moment or its linear transformation, yielding the dynamic embedding of electricity prices. .

[0129] (4) Dimension alignment and feature splicing

[0130] 1) Project the three types of embeddings onto a unified dimension :

[0131]

[0132] In the formula, For trainable projection matrices, It is an equal-dimensional vector.

[0133] 2) Concatenation and normalization yield the fused vector:

[0134]

[0135] In the formula, This indicates splicing by dimension. This is a layer normalization operation used to stabilize the numerical range and improve the stability of downstream training.

[0136] (5) Caching and online inference acceleration

[0137] 1) Periodically cache site topology and static attributes that change slowly over time, and trigger incremental updates only when a change in topology or site status is detected;

[0138] 2) The electricity price sequence is updated using sliding updates and hidden state reuse to reduce redundant calculations;

[0139] 3) Criteria for the candidate site set Perform Top-K pre-screening, prioritizing the encoding of highly relevant sites to reduce the computational load in the fusion stage.

[0140] Through the above steps (1)-(5), a comprehensive feature vector that is aligned at the same time step, sensitive to user scenarios, and friendly to downstream models is obtained. The user's personalized behavioral preferences have been encoded at the same time. Spatial accessibility and site relationships ) and electricity price fluctuation trends ( ).

[0141] S4. Building a Personalized Recommendation Model

[0142] After completing S3, obtain the fused feature vector. Next, this embodiment constructs a personalized recommendation model based on a multilayer fully connected network (MLP), outputting "charging station recommendations" and "charging time predictions" respectively. To ensure consistency between training and deployment, a combination of multi-task joint learning, regularization, and post-inference processing is adopted. Specifically, the following steps are included:

[0143] (1) Overall structure of the model

[0144] 1) Input and Output: As input, the output includes: a) the probability distribution of candidate charging stations. b) Continuous charging time prediction .

[0145] 2) Network backbone: A multilayer perceptron (MLP) is used to extract non-linear features, which are then divided into two branches: a "site recommendation head" and a "time prediction head". The hidden layers use ReLU activation, and LayerNorm and Dropout are used for stable training.

[0146] 3) Forward propagation:

[0147]

[0148] In the formula, For the first Layers can be trained with weights and biases. For activation functions; For the first Layer output features.

[0149] (2) Site recommendation sub-model

[0150] 1) Classified output: Input site recommendation header to obtain unnormalized score ( Softmax to get probability distribution:

[0151]

[0152] where, is the predicted probability of station , is the corresponding score. 2) Cross-entropy loss:

[0153]

[0154] where, is the one-hot label of real station, is the predicted probability, which is used to measure the classification error.

[0155] 3) Class imbalance handling (optional): Introduce class weights for unpopular stations to get weighted cross-entropy

[0156]

[0157] where, is the class weight, which alleviates the impact of unbalanced distribution of training samples.

[0158] (3) Time prediction sub-model

[0159] 1) Regression output: input into the time prediction head to get scalar , which represents the recommended charging time (e.g. estimated start time or charging duration, according to business definition).

[0160] 2) Time error (MAE):

[0161]

[0162] where, is the real charging time (label) of the th sample, is the predicted value, which measures the mean absolute error.

[0163] 3) Boundary constraint (inference stage): clip to map to the operation available interval to ensure the result is executable.

[0164] (4) Joint loss and regularization

[0165] 1) Multi-task joint:

[0166]

[0167] where, are weight coefficients to balance the importance of classification and regression tasks.

[0168] 2) Weight decay and stabilization term: add regularization to the optimization objective, resulting in

[0169]

[0170] where, is a regularization coefficient to suppress overfitting and improve generalization.

[0171] (5) Training procedure and hyperparameter settings

[0172] 1) Mini-batch training: organize samples into mini-batches by time slicing to ensure temporal consistency within the same batch.

[0173] 2) Optimizer: use Adam optimizer with learning rate combined with cosine annealing or warm-up strategies; gradient clipping (e.g., ) if necessary.

[0174] 3) Regularization methods: add Dropout (e.g., 0.1–0.3) and LayerNorm after each hidden layer to alleviate overfitting and stabilize gradient distribution.

[0175] 4) Early stopping strategy: use the or Top-1 / MAE on the validation set as the monitoring indicator, and stop training early if there is no improvement for several cycles.

[0176] 5) Label smoothing (optional): apply a smoothing coefficient (e.g., 0.05) to the classification labels to further improve model robustness.

[0177] (6) Inference and post-processing

[0178] 1) Top-K recommendation and reordering: select Top-K sites as the candidate set according to ; within the candidate set, perform weighted reordering based on real-time load, queue length, available gun number, and user distance to obtain the final site .

[0179] 2) Time calibration: perform temperature scaling on the classification output to improve confidence calibration:

[0180]

[0181] where,​ Confidence for calibrated probability; meanwhile, for Combining operating period and traffic peak information for alignment at the hour or half hour, generating the final .

[0182] 3) Result packaging: output and Top-K candidate list with corresponding confidence for front-end display and decision explanation.

[0183] (7) Evaluation indicators and acceptance

[0184] 1) Offline evaluation:

[0185] Recommendation accuracy: Top-1 / Top-3 Accuracy, NDCG@K;

[0186] Time prediction: MAE, RMSE;

[0187] Comprehensive index: and Weighted score under trade-off.

[0188] 2) Online monitoring:

[0189] Adoption rate (the proportion of users who choose recommended sites), average waiting time change, unit time charging completion amount change;

[0190] Abnormal alarm (such as systematic early / late prediction time) and rollback mechanism.

[0191] S5. Real-time feedback and closed-loop optimization

[0192] After completing S4 and outputting , enter the online running phase; the system continuously receives user and platform feedback while generating recommendation results, converts the feedback into reinforcement learning signals and drives policy iteration updates, forming a "prediction - execution - feedback - update" closed loop.

[0193] (1) Online feedback collection and data channel

[0194] 1) Continuously receive site-side feedback from charging pile operation platform: queue length, available gun number, real-time load, price adjustment.

[0195] 2) Record behavior feedback from user side application: actual selected site , actual start / end time , cancellation / change, etc.

[0196] 3) Grid-side system synchronization operation information: regional load, peak and valley period marking, dispatch instruction change. 4) Align the above feedback by timestamp and perform quality check (missing data filling, abnormal data removal), generate sample entries for reinforcement learning .

[0197] (2) State, action and constraint definition

[0198] 1) State : composed of fusion features of S3 and platform-side real-time quantities, such as user location, site snapshot, regional load, electricity price window, historical Top-K recommendation and adoption results .

[0199] 2) Action : composed of "recommended site + recommended time". In implementation, candidate sites are discretized into , and time is quantized to a fixed granularity (such as 15 minutes) to form a discrete action space, or "site classification + time regression" is encoded into a joint action.

[0200] 3) Constraints: actions need to meet site capacity, operating period, user accessibility and grid safety constraints; if not met, perform action correction or resampling (e.g. replace with a suboptimal but executable action in the candidate set).

[0201] (3) Reward function and synthesis

[0202] 1) User-side reward:

[0203]

[0204] where is the actual charging time of the user, is the model recommended time, measuring the impact of time deviation on user experience (the closer the reward is, the higher the reward is).

[0205] 2) System-side reward:

[0206]

[0207] where represents the load balancing degree (the closer to the target curve, the higher the score), is the operating cost (such as peak electricity purchase, congestion penalty, etc.).

[0208] 3) Comprehensive reward:

[0209]

[0210] where ​For the synthesis of rewards; if one side goal is emphasized, weight coefficients can be introduced to weight the two.

[0211] (4) DQN training and updating process

[0212] 1) Policy execution and sampling: in the online phase, the -greedy policy is adopted to select the greedy action of the current Q network with a probability and randomly explore with a probability ; the interactions obtained are stored in the experience replay pool .

[0213] 2) Target value and loss: from , a small batch is randomly sampled to calculate the TD target

[0214]

[0215] In the formula, is the discount factor; is the output of the target network. Then the square loss is minimized

[0216]

[0217] In the formula, is the output of the current evaluation network, is the quadratic loss of the time difference error.

[0218] 3) Parameter update and stabilization: perform gradient descent update parameters on the evaluation network; every steps, soft / hard copy the evaluation network parameters to the target network to stabilize the training.

[0219] 4) Bellman update relationship (used to explain Q value backtracking):

[0220]

[0221] In the formula, is the expected return of taking action in state ; the right side "immediate reward + discounted optimal future return" reflects the recursive nature of dynamic programming.

[0222] (5) Online inference and post-processing

[0223] 1) Select Top-K candidates from the site probability distribution; combine real-time queues, distances , site health, etc. to perform weighted ranking and output the final .​

[0224] 2) Predicted time Application of operation boundary clipping and time alignment (whole point / half point) to obtain 3) If the recommendation cannot be executed (sudden outage / full load), fall back to the suboptimal candidate in order of confidence and recalculate immediately.

[0225] (6) Adaptive and concept drift processing

[0226] 1) Update exploration rate over time (such as linearly decay from 0.2 to 0.01), retain exploration in early stage and enhance utilization in later stage.

[0227] 2) Take time weighting or sliding window strategy for experience replay pool to ensure that the latest environment distribution is learned first; if necessary, perform periodic retraining or incremental fine-tuning.

[0228] 3) Set threshold alarm and automatic rollback strategy for key indicators (Top-K adoption rate, MAE, average waiting time, load deviation).

[0229] (7) Closed-loop monitoring and deployment interface

[0230] 1) Logs and indicators: record and execution results for each recommendation for offline review and online monitoring;

[0231] 2) A / B test (optional): compare the impact of different hyperparameters (e.g. ) or network structures on indicators in small traffic;

[0232] 3) Platform integration: provide standardized interfaces to push recommendation and reordering results to scheduling / billing / terminal system, and receive status feedback to maintain closed loop.

[0233] With the progress of two-way interactive scheduling process, the system continuously optimizes the scheduling strategy according to real-time feedback. Each optimization adjusts the strategy parameters through deep Q learning and Actor-Critic and other reinforcement learning mechanisms, gradually improving the accuracy and stability of charging and scheduling results. With this dynamic optimization method, the model can update decisions in real time under the environment of changing grid load, electricity price fluctuations, user behavior, and charging station status, ensuring the efficiency and accuracy of vehicle-grid two-way interactive scheduling.

[0234] The application provides a dynamic planning model for two-way interaction between an electric vehicle and a power grid, which is combined with multi-period dynamic planning, user behavior modeling and multi-modal data features, and can generate individualized and globally coordinated charging and discharging and power grid scheduling schemes in real time. Through the introduction of an online feedback and reinforcement learning closed-loop mechanism, the recommendation and scheduling strategies can be continuously and adaptively optimized, so that the system can still maintain efficient operation under different scenarios and demand changes. The application not only improves the charging and discharging efficiency and user satisfaction, but also effectively reduces the peak load and operating cost, enhances the power grid frequency modulation and renewable energy consumption capacity, and has a wide application prospect and significant economic benefits.

Claims

1. A method for dynamic programming model of bidirectional interaction between electric vehicles and power grid, characterized in that the method comprises the following steps: (1) collecting power grid operation data, vehicle battery characteristic data and user charging and discharging behavior data; (2) pre-processing the collected data, including missing value interpolation, data normalization and feature extraction; (3) constructing a multi-period dynamic programming model, dividing a day into several charging and discharging stages, and establishing a state transition equation and a stage benefit function based on power load prediction and battery state; (4) using a deep reinforcement learning algorithm to optimize the dynamic programming strategy online, the algorithm including a deep Q network (DQN) and an Actor-Critic architecture; (5) modeling user behavior based on a user utility function and a Logit model to predict the charging and discharging willingness under different electricity prices and time periods; (6) generating an optimal bidirectional interaction scheduling scheme by Pareto frontier analysis under the multi-objective constraints of minimizing user cost and maximizing power grid benefit; (7) issuing the optimal scheduling scheme to the smart grid control platform in real time, and iteratively updating the strategy using deep reinforcement learning according to platform feedback and user real-time behavior data, forming a closed-loop improvement.

2. The method according to claim 1, characterized in that, The data pre-processing step includes: Multiple interpolation processing for missing values in power grid operation data and user behavior data; Normalizing or standardizing numerical features; Extracting time series features and spatial features, and modeling spatial features by calculating the geographical distance between users and charging stations. 3.The method of claim 1, characterized in that in the multi-period dynamic programming model, the stage state is composed of the battery state of charge (SOC) and the predicted power load, the stage action includes charging, discharging and idling, and the stage benefit function is defined according to the difference between user electricity price expenditure and power grid benefit. 4.The method of claim 1, characterized in that the deep reinforcement learning algorithm includes: Using DQN to pre-train the multi-period dynamic programming strategy offline; Based on the Actor-Critic framework, the strategy is updated online, wherein the Actor network outputs the charging and discharging action probability distribution, and the Critic network evaluates the action value. 5.The method of claim 1, characterized in that the user behavior modeling uses a Logit-based utility function.

6. The method of claim 1, characterized in that the multi-objective optimization is achieved by a Pareto front analysis defining a user cost minimization objective and a grid benefit maximization objective .

Images