Power swap station energy scheduling method and system based on reinforcement learning algorithm

Abstract

The invention is suitable for the technical field of battery swap energy scheduling, and particularly relates to a battery swap station energy scheduling method and system based on a reinforcement learning algorithm, and the method comprises the steps: constructing a TCN time sequence prediction model, and carrying out the data collection and charging and discharging and battery swap operation through the interaction of an intelligent agent and an environment; using the TCN deep learning model to predict future demands so as to expand the observation space of the deep reinforcement learning model; an intelligent agent is trained by using a PPO algorithm, updating is performed by using a multi-step advantage estimation and cutting constraint control strategy, and model parameters are optimized by using interactive data, so that the energy scheduling capability is realized. According to the method, the output charging power is continuous instead of discrete gears, so that the control is more accurate, entropy regularization promotes the strategy to cover more action space, the profit can be increased by more than 50% under the condition of daily scheduling, the profit is increased by 9%, the optimal charging time period can be found more quickly, and the problems of low data processing speed and weak modeling capability in the prior art are solved.

Description

Technical Field

[0001] This invention belongs to the field of battery swapping energy dispatching technology, and particularly relates to a battery swapping station energy dispatching method and system based on reinforcement learning algorithm. Background Technology

[0002] With the rapid development of new energy vehicles and energy storage batteries, the widespread adoption of charging and battery swapping stations, and the continuous advancement of the artificial intelligence industry, all sectors are anticipating an "AI+" energy revolution. Addressing the issues of existing battery swapping systems being relatively basic and having low battery utilization efficiency, our team plans to develop an intelligent scheduling system for battery swapping stations based on real-time and predictive data. This system will not only improve the effective utilization rate of batteries and meet the ever-increasing demand for battery swapping, but also create greater profits for related enterprises and promote the sustainable and prosperous development of my country's energy dispatching industry.

[0003] Existing energy dispatching methods for battery swapping stations can be broadly categorized into three types: LSTM-based time series forecasting, model predictive control, and rule-based expert system solutions. All of these technologies suffer from long data processing times and weak modeling capabilities, severely impacting model response efficiency and hindering the practical implementation of efficient energy dispatching and utilization. Summary of the Invention

[0004] The purpose of this invention is to provide an energy scheduling method for battery swapping stations based on reinforcement learning algorithms, aiming to solve the problems of long data processing time, weak modeling ability, and low model response efficiency of the above-mentioned technologies, and to achieve intelligent and efficient energy scheduling and utilization.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The steps to construct a TCN time series prediction model include: Step 1: Prepare time-series training data, calculate the predicted value through forward propagation, calculate the loss using the MSE between the predicted value and the true value, and finally use the Adam optimizer to achieve backpropagation and gradient descent, iteratively updating the network weights to minimize the error. Step 2: Couple the demand prediction results of the TCN prediction model with the basic environmental data to form the observation space of the deep reinforcement learning model, and use the deep reinforcement learning algorithm PPO to train the model based on this, construct a multi-step advantage function estimation framework, and calculate the advantage value at each time step. Step 3: Introduce clipping constraints to limit the update range of the strategy; Step 4: Update the parameters of the PPO prediction model based on the interaction data between the agent and the environment, and use the trained PPO model for energy scheduling.

[0006] Preferably, the observation space has 36 dimensions, including at least the current time, electricity price, temperature, whether it is a holiday, temperature, and vehicle battery swapping demand forecast.

[0007] Preferably, the TCN prediction model includes an Actor-Critic network, in which the Actor network is composed of a multi-layer fully connected network, and the Actor network incorporates the ReLU activation function and LayerNorm processing of the input data.

[0008] Preferably, after the Softmax function converts the battery action scores output by the Actor network into probability values, the action with the highest probability among charging, discharging, and no operation is selected during the Actor evaluation phase; during the Actor training phase, a sampling strategy is adopted for each battery, and the corresponding log probability is recorded for subsequent policy gradient updates.

[0009] Preferably, in the Actor-Critic network, the output of the Critic network is a scalar value estimate at the current time and battery level. The Critic network is used to calculate the state value at each step and compare it with the actual reward. If the state value is greater than the actual reward, it means that the value estimate of this state is too high, and vice versa.

[0010] Preferably, a Temporal Convolutional Network (TCN) is trained to predict future electric vehicle battery swapping demand, and this predicted data is transformed into a state vector and embedded into a Deep Reinforcement Learning (DRL) framework, thereby effectively expanding the agent's observation space. The PPO algorithm is used to train the DRL, and a multi-step advantage function estimation framework based on Generalized Advantage Estimation (GAE) is constructed. This framework calculates the advantage value at each time step by introducing temporal difference (TD) residuals: a positive advantage value indicates that the benefit of taking the action is higher than average; a negative value indicates that the action performs worse than expected. Simultaneously, an exponentially weighted sum is used to process the TD residuals at each time step, effectively balancing the estimation bias and variance, ensuring the accuracy and stability of the advantage estimation.

[0011] Preferably, the system includes an interactive front-end, an interactive back-end, and an IoT platform, wherein the interactive front-end includes a view layer, a business logic layer, and a data access layer; the interactive back-end includes a presentation layer, a business logic layer, a data access layer, and a data storage layer; and the IoT platform includes a perception layer and a network layer.

[0012] Preferably, the data access layer of the interactive front end supports GET, POST, PUT and DELETE requests, and uses interceptors to achieve unified authentication and error handling.

[0013] Preferably, the presentation layer of the interactive backend adopts the Spring MVC framework based on the JavaEE platform, integrating Spring Boot and Spring Data plugins, and supports interaction with the charging pile power database and behavior database.

[0014] Preferably, the business logic layer of the interactive backend adopts a Python + Django ORM architecture and uses a Jetson Nano processor to provide computing power support for edge computing. The trained prediction model and intelligent body are embedded in the processor to achieve real-time scheduling and reduce data transmission costs.

[0015] The energy scheduling method for battery swapping stations based on reinforcement learning algorithms provided by this invention has the following advantages: 1. Intelligent adaptive scheduling capability based on PPO algorithm: By employing the Proximal Policy Optimization (PPO) deep reinforcement learning algorithm, a scheduling agent capable of autonomous decision-making is trained through learning from historical and real-time data. This agent can achieve millisecond-level response and dynamic optimization strategy generation based on dynamic environmental factors such as real-time electricity price fluctuations, load changes, and renewable energy output, significantly improving the system's adaptability and decision-making efficiency in complex and ever-changing scenarios. 2. Integrating TCN-predicted future perception capabilities: Temporal Convolutional Networks (TCNs) are introduced to predict future information in multiple dimensions, including key external variables such as electricity demand, weather conditions, and policy changes. Leveraging their ability to capture long-range dependencies and their parallel computing advantages, TCNs provide reinforcement learning agents with richer and more accurate representations of future states, enabling them to have "foresight" in scheduling decisions. This reduces the risks associated with uncertainty and enhances the system's sensitivity to the coupled effects of multiple factors. 3. Practical engineering advantages brought by cloud-edge collaborative architecture: A collaborative framework of "cloud-based training and edge-based decision-making" is adopted. Large-scale historical data training and model updates are completed in the cloud, ensuring continuous algorithm optimization; lightweight inference and real-time decision-making are performed at the edge, significantly reducing the demand for local hardware computing power, while also reducing the communication bandwidth and latency required for data upload and command issuance. This architecture not only improves the system's scalability and deployment flexibility but also significantly reduces operating costs and response latency. 4. Overall economic and social benefits: It can be applied not only to typical scenarios such as battery swapping stations, enabling low-cost power purchase and efficient power dispatch, directly improving enterprise operating profits, but also possesses excellent robustness, maintaining stable operation under extreme weather, policy adjustments, or sudden market changes. Furthermore, by participating in grid demand response, it can achieve peak shaving and valley filling, effectively smoothing grid load fluctuations, promoting the consumption of renewable energy, and possessing significant social and environmental benefits. Attached Figure Description

[0016] Figure 1 This is a structural diagram of the TCN prediction model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating an application scenario provided by an embodiment of the present invention; Figure 3 A schematic diagram of the working principle of the energy dispatching system provided in the embodiments of the present invention; Figure 4 A GAE structure diagram provided for an embodiment of the present invention; Figure 5 A schematic diagram of the view layer construction framework for the interactive front end provided in an embodiment of the present invention; Figure 6 A schematic diagram of the business layer framework of the interactive front end provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the construction of the data access layer framework provided in an embodiment of the present invention; Figure 8 An architecture diagram of the intelligent cloud-edge system framework provided in an embodiment of the present invention; Figure 9 A schematic diagram of the DS18B20 temperature sensor provided in an embodiment of the present invention; Figure 10 A schematic diagram of the cloud platform architecture provided in an embodiment of the present invention; Figure 11 Vehicle operation data and feature maps provided in embodiments of the present invention; Figure 12 This is a schematic diagram illustrating the vehicle data correlation analysis provided in an embodiment of the present invention; Figure 13 A three-dimensional diagram of the TCN network architecture provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0018] like Figure 2As shown, the operation of a typical battery swapping station is simulated: When an electric vehicle enters the station, the system selects the battery with the highest current charge (and meeting the minimum charge requirement) from the battery compartment for the user's swap. The replaced battery is returned to an idle storage location within the battery compartment and charged according to the scheduling strategy. The station's control center not only integrates environmental data but also executes specific control commands, controlling the charging and discharging behavior of each battery in the battery compartment or keeping them idle. Furthermore, while meeting the vehicle's battery swapping needs, the station can leverage its energy storage characteristics to participate in grid interaction, aiming to obtain additional power dispatch profits.

[0019] This invention provides a method for energy scheduling of battery swapping stations based on reinforcement learning algorithms, the method comprising: Constructing a TCN time series prediction model: In this step, the construction of the observation space represents the basis for the agent's decision-making and is crucial to the agent's learning and decision-making. This invention collects information such as current time, electricity price, temperature, whether it is a holiday, vehicle battery swapping demand predictions, arrival predictions, and station battery status at each time step, forming a 36-dimensional observation space that comprehensively and broadly characterizes the objective physical world.

[0020] To improve the foresight of decision-making, the state vector also embeds vehicle data for the next 30 minutes predicted by TCN (Temporal Convolutional Network algorithm).

[0021] like Figure 1 As shown, the TCN algorithm mainly consists of three basic principles: dilated convolution, dilated convolution, and residual connections. Convolution operations can slide across time-series data, extracting local features by element-wise multiplying and summing the convolution kernel with local data regions. Dilated convolution, based on standard convolution, introduces an increased margin into the convolution kernel, causing the receptive field to expand exponentially, thus enabling it to capture dependencies in more distant future time series. Residual connections establish connections between input and output, allowing information to be directly passed from one layer of the network to deeper layers, enabling gradients to propagate more smoothly in the backpropagation direction. This solves the gradient vanishing problem when training deep networks, allowing for the training of deeper pre-planning network structures.

[0022] Compared to other time series models, TCN (Temporal Convolutional Network) excels in parallel computing capabilities, long-range dependency modeling, flexibility, and versatility. In terms of parallel computing, TCN can process the entire time series data simultaneously, significantly improving computational efficiency, especially when dealing with long-series data such as those from battery swapping systems. Regarding long-range dependency modeling, through dilated convolution, TCN can more effectively capture long-range dependencies in the time series, making it suitable for predicting future vehicle arrivals and battery swapping capacity in battery swapping systems. In terms of flexibility and versatility, TCN has a relatively simple structure, is easy to adjust and extend, and can be easily combined with networks such as Actor and Critic networks to further improve model performance.

[0023] The Actor network is structurally composed of multiple fully connected layers, enabling it to learn complex relationships between different features of the input data. By incorporating the ReLU (Rectified Linear Unit) activation function and LayerNorm (layer normalization) processing of the input data into the network, the Actor network's ability to learn data features is significantly improved, shortening the network's training process, increasing stability, and avoiding problems such as gradient vanishing and exploding during training.

[0024] The Softmax function converts the battery action scores output by the Actor into probability values, significantly improving the legality of the battery's action probabilities and thus obtaining the probability distribution of charging, discharging, and inaction for a single battery. During the Actor evaluation phase, we select the action with the highest probability among charging, discharging, and inaction. During the Actor training phase, we adopt a sampling strategy for each battery action and record the corresponding log probability, allowing the Actor to be adjusted and optimized towards obtaining greater rewards while exploring more different action choices.

[0025] The Critic network is structurally similar to the Actor network, but their outputs differ. The output of the Critic network is a scalar estimate of the value at the current time and battery level, which is essentially a prediction of the cumulative reward in the future under this state.

[0026] Therefore, Critic networks can be used to calculate the state value at each step, which can then be compared with the actual reward. If the state value is greater than the actual reward, it indicates that the value of this state has been overestimated, resulting in a poor decision; conversely, if the state value is less than the actual reward, it indicates that the value of this state has been underestimated, resulting in a better decision. By comparing the state value with the actual reward, Critic networks can provide feedback on the quality of decisions made under different states, thus providing a basis for the agent to optimize its strategy.

[0027] The steps for training the TCN time series prediction model include: Prepare time-series training data, calculate predicted values ​​through forward propagation, calculate loss using the MSE between predicted and true values, and finally use the Adam optimizer to implement backpropagation and gradient descent, iteratively updating network weights to minimize error.

[0028] In this step, the training mechanism for the PPO deep reinforcement learning model is as follows: Figure 3 As shown, data collection and interaction: In each training episode, the agent interacts with the environment: based on factors such as remaining battery power and current electricity price, the agent selects actions for each battery. The environment then performs charging / discharging or battery swapping operations based on these actions, calculating the benefits and costs of each operation. These benefits and costs are converted into immediate profit rewards, which are fed back to the agent. These immediate rewards can be used to evaluate the quality of the agent's current actions, allowing the agent to continuously adjust its action selection strategy.

[0029] In a specific embodiment of the present invention, the TCN training process is as follows: By inputting predicted information as enhanced state features into the DRL agent, the TCN's ability to perceive future situations is significantly improved. Considering the strong correlation between vehicle arrival information (vehicle arrival rate and arriving battery capacity) and time factors, we analyzed the relationship between the predicted vehicle battery swapping demand at a relevant power station in Chengdu in May 2020 and multi-dimensional time factors such as hour, day of the week, and date to capture specific data characteristics; such as... Figure 11 As shown.

[0030] Figure 11 Figures (a) to (c) show the variation of vehicle arrival demand with time, week, and month. Figure 11 (d) shows the number of battery swaps that occurred during different time periods from May 1 to May 6. Figure 11 (e) shows the battery distribution of EVs arriving at the battery swapping station. The graph reveals significant temporal and state characteristics in vehicle arrivals at charging stations. On an hourly scale, a bimodal distribution is observed, with peaks occurring during the morning (7-10 AM) and evening (3-7 PM) peak hours, consistent with travel patterns, while arrivals are sparse at night. On a weekly scale, arrivals are stable on weekdays, with significant increases on Fridays and weekends (especially Fridays and Saturdays). Daily arrivals in May fluctuated between 60 and 90 vehicles, and the heatmap confirms the prevalence of peak hours throughout the week. Furthermore, the initial SOC (State of Charge) of arriving vehicles is concentrated between 30% and 60%, peaking at approximately 40% to 50%, indicating that users tend to charge before the battery is completely depleted. To further determine and quantify these relationships, we conducted an autocorrelation analysis on the data to analyze the correlation between data at different lag times. The results are as follows: Figure 12 As shown.

[0031] likeFigure 12 As shown, both the daily periodicity indicated by the red vertical line and the weekly periodicity indicated by the blue vertical line exhibit significant peaks in their autocorrelation coefficients, which all significantly exceed the 95% confidence interval represented by the gray dashed line. This phenomenon clearly demonstrates that vehicle arrival information exhibits distinct daily and weekly periodic characteristics. Based on this, conducting research on vehicle arrival time series prediction not only has a theoretical basis but also significant practical value.

[0032] Based on the data feature analysis and temporal correlation analysis above, we chose a Temporal Convolutional Network (TCN) for vehicle demand forecasting. TCN is a recognized deep learning architecture that excels in forecasting temporal data. It ensures that predictions rely only on historical information through causal convolutions, avoiding future data leakage. Simultaneously, it utilizes dilated convolutions to progressively expand the temporal receptive field, thus efficiently capturing long-term dependencies, making it highly suitable for battery swapping demand forecasting. The TCN neural network is as follows: Figure 13 As shown.

[0033] In this invention, such as Figure 13 As shown, the input data for the TCN model is a 24-time series, with each time step containing four features: arrival sign, average SOC (State of Charge), hourly information, and weekday information. The model first captures the long-term and short-term dependencies in the time series data through the TCN module. The arrival prediction model uses a TCN module with 64 filters, a kernel size of 3, and an inflation rate of [1, 2, 4, 8], outputting the arrival probability with Sigmoid activation. The SOC prediction model further expands the inflation rate to [1, 2, 4, 8, 16]. The TCN module is followed by a fully connected layer (32 nodes, ReLU activation function, and L2 regularization), outputting a linearly activated SOC value. The model training process uses the Adam optimizer and incorporates an early stopping mechanism to prevent overfitting.

[0034] Model performance: The model was trained using historical data, and its prediction accuracy reached 71.67%, with an AUC of 0.735, which is much higher than 0.5, indicating that the model has a good ability to distinguish between vehicles that have arrived and those that have not. At the same time, the model was cross-validated, and the accuracy and F1 score fluctuations between each fold were less than 5%, indicating that the model also has good stability and generalization ability.

[0035] In terms of collaboration: To address the high degree of randomness in vehicle arrival rates and battery swapping demand, we employ a Temporal Convolutional Network (TCN) to capture multi-timescale dependencies (such as daily and weekly cycles) in time-series data. Through dilated convolution and causal constraint mechanisms, we predict future vehicle arrival probabilities and battery demand characteristics, providing forward-looking information for dynamic decision-making.

[0036] Reward Design: In addition to the immediate profit rewards generated from charging, discharging, and battery swapping, the system also adds an extra reward for successful battery swaps after training. If the battery swapping success rate is high, the system will give the agent an additional reward to encourage it to maintain its excellent battery swapping performance. If the battery swapping success rate is lower than the preset success rate (e.g., 90%), the system will also impose an additional penalty on the agent. This allows the agent to pursue profits while also considering the success rate of battery swapping services, thereby maximizing overall benefits.

[0037] The demand prediction results from the TCN prediction model are coupled with basic environmental data to form the observation space of the deep reinforcement learning model. Based on this, the deep reinforcement learning algorithm PPO is used for model training, constructing a multi-step advantage function estimation framework. The advantage value at each time step is calculated. In this step, the advantage estimation and target update are performed. In reinforcement learning training, while the immediate reward an agent receives at each time step directly reflects the short-term effect of the current action, the core objective is to maximize the long-term cumulative reward. For example... Figure 4 As shown, to systematically weigh the relationship between immediate gains and future potential gains, the Generalized Advantage Estimation (GAE) constructs a multi-step advantage function estimation framework by introducing an exponentially weighted sum of time difference residuals (TD Residuals). The key to this method lies in simultaneously combining a discount factor γ (0 < γ < 1) and a trajectory adjustment factor λ (0 ≤ λ ≤ 1). The former, through the exponential decay of γ^t, embodies the rational assumption that "rewards further back in time have a weaker impact on the current decision." For example, when γ = 0.9, the weight of the reward at the k-th step in the future will decay exponentially at a rate of 0.9^k. The latter, λ, achieves a dynamic balance between the high variance of the Monte Carlo method and the high bias of the time difference method by adjusting the superposition step size of the TD error. This dual regulation mechanism not only quantifies the potential impact of the current action on the future state sequence, but also makes the contribution of recent rewards decrease geometrically through the compound decay mode of γ^t·λ^k (where k represents the k-th time step in the future). Thus, while ensuring stable policy updates, it accurately portrays the decision-making characteristics of the agent that "focuses on the present while planning for the future".

[0038] Introduce pruning constraints to limit the magnitude of policy updates.

[0039] In this step, the CLIP pruning objective is to understand the degree of policy change after the update by calculating the ratio of the probability between the old and new policies during policy updates in reinforcement learning. This helps to evaluate whether the direction and magnitude of the policy update are reasonable. Simultaneously, not only is the magnitude of the policy update limited by introducing pruning constraints (clip): if the calculated ratio exceeds a set threshold for the probability ratio between the old and new policies, the ratio will be pruned to the threshold boundary. Gradient pruning techniques are also combined to limit the gradient within a certain range, avoiding problems such as the model failing to converge or diverging due to excessively large gradients, or training being too slow due to excessively small gradients. Pruning constraints and gradient pruning techniques can effectively improve the stability and effectiveness of training, ensure the controllability of policy updates, and thus enable the model to learn and optimize more smoothly.

[0040] Furthermore, a value function loss is introduced to reflect the gap between the current policy and the optimal policy. By minimizing this gap, the policy is made to develop towards obtaining higher long-term returns. An entropy regularization term is also used to add a term related to the policy entropy to the optimization objective, preventing the policy from converging to a deterministic state too early, thereby balancing the agent's exploration and exploitation.

[0041] The parameters of the PPO prediction model are updated based on the interaction data between the agent and the environment, and the trained PPO model is used for energy scheduling.

[0042] In this step, multiple rounds of updates are performed: After the agent interacts with the environment and collects data such as immediate rewards and battery levels for each round, this data is divided into several mini-batches. Based on the mini-batch data, the parameters of the Actor and Critic are repeatedly updated, allowing the agent to fully utilize the current data to iterate and improve its strategy more stably and effectively, in order to achieve better performance and higher rewards in future interactions with the environment.

[0043] This invention also provides an energy dispatching system for battery swapping stations based on reinforcement learning algorithms. The system includes an interactive front-end, an interactive back-end, and an Internet of Things (IoT) platform. The interactive front-end includes a view layer, a business logic layer, and a data access layer. The interactive back-end includes a presentation layer, a business logic layer, a data access layer, and a data storage layer. The IoT platform includes a perception layer and a network layer.

[0044] I. Interactive Front-End: (1) View layer: like Figure 5 As shown, HTML is used to build the basic webpage structure, CSS is used to define the style layout, and Sass / Less preprocessors are used to improve code maintainability and reusability. It is responsible for rendering the user interface and handling user actions, enabling data visualization and input feedback.

[0045] (2) Business logic layer: like Figure 6 As shown, component-based development is implemented based on Vue.js, and the page is dynamically updated through a responsive system. VueRouter is integrated to support single-page application (SPA) route switching, and rules are configured according to business requirements.

[0046] (3) Data Access Layer: like Figure 7 As shown, the communication tool uses Axios (a Promise-based HTTP client), which supports GET, POST, PUT, and DELETE requests. Authentication and unified error handling are achieved through interceptors. A RESTful API is designed, defining resource paths by URLs and obtaining the status of the battery swapping station through HTTP methods, supporting centralized device management.

[0047] II. Interactive Backend: (1) Presentation layer: Based on the JavaEE platform and the Spring MVC framework, this system optimizes code structure using dependency injection and aspect-oriented programming (AOP), integrates Spring Boot and Spring Data plugins, supports efficient interaction with charging pile power and behavior databases, and simplifies code logic.

[0048] (2) Business logic layer: The system adopts a Python + Django ORM architecture to implement core business rules such as user authentication and charging pile status management, and coordinates the data access layer to process data and verify its validity.

[0049] (3) Data Access Layer: Responsible for database CRUD operations, returning structured data to the business logic layer; reducing storage layer load and improving response efficiency through data caching strategies.

[0050] (4) Data storage layer: It undertakes data persistence tasks, optimizes database table structure and index design to improve storage and retrieval performance, and formulates backup and recovery strategies to ensure data security and integrity.

[0051] III. Internet of Things (IoT) Platform: (1) Perception layer: Data is collected through built-in sensors in the charging pile, which is equipped with a 5G / wired communication module to achieve real-time interaction with the Actor-Critic network and battery swapping system.

[0052] (2) Network layer: A hybrid transmission strategy is adopted: 5G network: Supports high-volume, high-time-efficiency data (user interaction system data); NB-IoT network: used for low-power, low-data-volume scenarios; Wired Ethernet: Ensures the stability and reliability of data transmission between the battery swapping system and the Actor-Critic network.

[0053] This invention also provides an intelligent cloud-edge collaborative power station scheduling system, such as... Figure 8 As shown.

[0054] Edge computing and continuous sensing based on Jetson nano: like Figure 9 As shown, the Jetson Nano microcontroller is a small, high-performance AI computer launched by NVIDIA. Its 128-core Maxwell GPU and 472 GFLOPS of computing power give the Jetson Nano powerful computing capabilities, enabling it to efficiently process reinforcement learning models for energy scheduling systems. The Jetson Nano's rich interfaces allow for easy connection to external devices such as charging pile power sensors to collect battery status information. Furthermore, the Jetson Nano excels in energy consumption; its low-power design reduces heat dissipation requirements and is suitable for long-term operation, allowing it to work stably in relatively compact spaces. The Jetson Nano is also easy to develop and deploy, supporting the Linux operating system and providing a rich set of software development kits and sample code to facilitate application development, model training, and deployment.

[0055] This invention utilizes Jetson Nano as an edge computing device, integrating Jetson Nano's edge computing capabilities with the battery management system and energy dispatch system of the battery swapping station to achieve full-process automation from data acquisition to decision execution.

[0056] First, the Jetson Nano is installed in the control unit of the battery swapping station. Sensors (such as battery management chips, temperature sensors, vehicle detection sensors, etc.) are connected via I2C and GPIO interfaces. A wired network is used to connect the Jetson Nano to the central control system of the battery swapping station to ensure the stability and real-time performance of data transmission, thus completing the hardware deployment. Next, a suitable Linux operating system (Ubuntu NVIDIA driver provided by JetPack) is installed on the Jetson Nano. The trained TCN-PPO model is then converted to a format suitable for edge device operation and deployed on the Jetson Nano, thus completing the software development. Sensor data is collected through the Jetson Nano's GPIO or I2C interface and converted into historical time-series data, which is then input into the TCN-PPO model deployed on the Jetson Nano to generate decision commands in real time, completing the data interaction within the station. Finally, the decision commands are sent to the execution units (such as battery controllers, charging equipment, etc.) of the battery swapping station via the network, achieving local decision-making and control. This reduces reliance on the cloud and remote servers, lowers latency, improves energy efficiency, enhances system reliability, and achieves intelligent decision-making on edge devices.

[0057] Meanwhile, the computer also has good scalability. Its standardized hardware interface supports plug-and-play for multiple types of temperature sensors, which can flexibly adapt to different accuracy and environmental monitoring needs and meet the continuous monitoring of battery swapping station facilities. It also adopts a hybrid communication architecture that is compatible with 5G (high bandwidth real-time transmission), NB-IoT (low power wide area coverage) and wired Ethernet (high reliability). It can dynamically switch communication methods according to the scenario and can also seamlessly connect to future new communication protocols through software upgrades, ensuring that the system maintains efficient and stable data interaction capabilities when the scale is expanded or the technology is iterated.

[0058] Cloud platforms, such as Figure 10 : The cloud platform in this invention is a key component of the TCN-PPO-based intelligent energy dispatching system for battery swapping stations, providing strong support for the system's efficient operation and intelligent decision-making. This cloud platform employs advanced cloud computing technology and a distributed architecture, enabling real-time acquisition, storage, processing, and analysis of data from battery swapping stations. It also supports TCN (Temporal Convolutional Network) training and PPO (Proximal Policy Optimization) algorithm execution, providing accurate prediction and optimization strategies for intelligent energy dispatching at battery swapping stations.

[0059] (1) Data Acquisition Layer: It is responsible for collecting real-time data from sensors and devices, including battery status data, vehicle arrival data, environmental data, etc., and sending the data to the cloud platform through data transmission protocols.

[0060] (2) Data processing layer: The collected raw data is cleaned, transformed, and integrated, and then converted into structured data suitable for model training and analysis. Distributed database and data warehouse technologies are used to store and manage the data, ensuring efficient storage and fast retrieval.

[0061] (3) Model training layer: Based on the processed data, the TCN model is trained and optimized using the deep learning framework PyTorch to accurately predict the number of future vehicle arrivals and the amount of battery swapping capacity. Simultaneously, the PPO algorithm is run, combined with the TCN prediction data, to optimize and train the energy scheduling strategy for the battery swapping station, generating the optimal charging, discharging, and battery swapping strategy.

[0062] (4) Application service layer: It provides application service interfaces for the intelligent energy dispatching system of battery swapping stations, including functions such as real-time monitoring, intelligent decision-making, and remote control. Through integration with the front-end interaction system and the back-end management system, it enables comprehensive intelligent management of battery swapping stations.

[0063] In this invention, TCN can be replaced by Transformer, and its advantages and disadvantages are as follows: Advantages: (1) Long-range dependency modeling: Self-attention can directly calculate the correlation weight between time t and time tk (k can be arbitrarily large), which can accurately capture the cross-time linkage of "morning peak traffic flow → midday charging demand".

[0064] (2) Dynamic feature focusing: By automatically strengthening the weight of key time steps (such as abnormal peaks caused by accidents) through QKV (attention mechanism), the prediction error of sudden traffic flow is reduced by 15-20%.

[0065] (3) Multi-source data fusion: Location coding can embed external variables, improving the modeling accuracy of the impact of sudden weather changes on demand by 12%.

[0066] defect: (1) High computational resource requirements: self-attention complexity O(n²) (computational cost surges when sequence length n=288), edge device (Jetson Nano), inference latency is nearly twice that of TCN.

[0067] (2) Data hunger: 100,000+ samples are needed for stable training (TCN only requires 30,000). Due to the necessity of transferring pre-trained models, it is difficult to start up new battery swapping stations.

[0068] (3) Overfitting risk: The number of parameters is usually 2-3 times that of TCN. At small-scale sites (average daily battery swapping <100 times), the prediction error exceeds TCN by 5-8%.

[0069] (4) Limitations of position coding: Sine coding has a weak ability to express non-fixed periodic patterns (such as sudden maintenance) and is inaccurate in predicting sequence breaks caused by unexpected events.

[0070] In this invention, PPO can be replaced by SAC, and its advantages and disadvantages are as follows: Advantages: (1) Continuous action optimization: Directly output continuous charging power, rather than discrete levels (avoiding frequent full charge / deep discharge).

[0071] (2) Exploring efficiency improvement: Entropy regularization enables the strategy to cover more action space, increasing profits by 9% in the event of sudden changes in electricity prices (finding the optimal charging time faster). (3) Stable value estimation: The dual-Q network takes the minimum value as the target, which suppresses overestimation and reduces the fluctuation of the reward curve during the training process by 41%.

[0072] Disadvantages: (1) Low sample efficiency: More interactive data is needed to converge (about 1.5 times that of PPO), cloud training costs increase by 35%, and online learning latency of edge devices increases.

[0073] (2) Weak support for discrete actions: The original design is oriented towards continuous space, and discrete actions need to be modified (such as Gumbel-Softmax), which increases the cost of modifying the actuators of the old battery swapping station.

[0074] (3) Theoretical convergence is complex: it depends on the balance between policy entropy and Q value, and the convergence proof is not as rigorous as PPO. In safety-critical scenarios (such as overcharge prevention), additional verification mechanisms are required.

[0075] (4) Real-time decision delay: The policy network needs to sample a Gaussian distribution (1.8 times slower than PPO's Softmax), which will increase the delay.

[0076] Edge hardware (using Raspberry Pi + TPU instead of Jetson Nano): Hardware configuration: Raspberry Pi: Choose a high-performance version (such as Raspberry Pi 4B / 5, equipped with 4GB+ memory).

[0077] Edge TPU Accelerator: Connects via USB or PCIe interface (such as Coral USB Accelerator).

[0078] Sensor interface: Connect temperature sensor (DS18B20), battery management chip, etc. using Raspberry Pi GPIO / I²C.

[0079] Communication module: External 5G / NB-IoT HAT expansion board (such as Sixfab or WaveShare module).

[0080] Software deployment: Configure the Python environment and dependent libraries, convert the quantized TCN-PPO model to a TPU-compatible format, develop scripts to call TPU inference and collect sensor data, compile middleware to coordinate hardware collaboration, deploy MQTT to achieve cloud communication, and set up automatic startup and exception handling to ensure stability.

[0081] Operating System: Install Raspberry Pi OS (64-bit version for improved performance).

[0082] Model optimization: Convert the TCN-PPO model to TensorFlow Lite format. Quantize the model to INT8 precision using the Edge TPU compiler (TPU operator compatibility required).

[0083] Inference framework: TPU is invoked through the libedgetpu library to accelerate inference.

[0084] System Integration: Develop Python middleware to coordinate Raspberry Pi (data processing) and Edge TPU (model inference). Deploy the MQTT protocol to enable communication with cloud platforms (e.g., using Eclipse Mosquitto).

[0085] A comparison of the advantages and disadvantages of Raspberry Pi + TPU vs. Jetson Nano: Dimension Raspberry Pi 5 + Edge TPU Jetson Nano Cost About 900 yuan (Raspberry Pi 5: 500 yuan + TPU: 400 yuan) More than 1000 yuan (when global shortage, premium) Computing power TPU provides 4 TOPS (int8) GPU provides 0.5 TOPS (FP16) Power consumption Total power consumption ≤7W (TPU about 2W) 5W-10W Deployment flexibility Modular design, detachable upgrade Integrated design, hardware fixed Development environment Depend on community-driven, complex configuration (need to downgrade Python, etc.) Official support is perfect (CUDA, TensorRT, etc.) Applicable scenarios Lightweight model inference (such as YOLOv8n) Support training + inference, can run more complex models Cloud-based collaboration (federated learning replacing centralized cloud platforms) Original centralized cloud layer Federal learning alternative solution Data acquisition layer Each battery swap station locally retains original data (battery status, vehicle arrival record, etc.), and does not upload to the cloud Data processing layer Data cleaning / feature engineering is completed on the edge device Model training layer Distributed training process: - Local training: each site trains the model with local data - Parameter aggregation: sites only upload model gradient / weight to cloud coordination node - Global update: cloud aggregates parameters to generate global model and distributes to each site Application service layer Decision model is deployed on edge device, cloud only provides model version management and aggregation algorithm Advantages and disadvantages comparison: Dimension Centralized cloud platform Federal learning solution Data privacy Low (original data upload) High (data locally retained) Real-time performance Depend on network transmission (200ms~2s) Edge decision (<50 ms) Bandwidth consumption High (GB level per day) Extremely low (MB level per day) Model performance Global optimum (centralized data training) Slightly lower than global (Non-IID compromise) Expansion cost Cloud expansion cost is high Incremental investment in edge devices Fault impact Single point failure causes network downtime Local fault isolation Summary of alternative solutions Replacing TCN with Transformer improves long-range prediction accuracy but significantly increases edge computing; SAC replaces PPO to achieve continuous adjustment of charging and discharging power, but requires more training data and reduces real-time performance; Raspberry Pi + TPU replaces Jetson Nano to reduce costs by 60%, but has weak industrial stability; Federated learning replaces centralized cloud platforms to enhance privacy compliance (data does not leave the local machine) and reduce bandwidth by 90%, but faces model accuracy loss and pressure to upgrade edge devices.

[0086] Overall, this solution is suitable for scenarios with low budgets, strong privacy requirements, and non-critical environments (such as remote small and medium-sized battery swapping stations), sacrificing some real-time performance and stability in exchange for cost and privacy advantages; however, in scenarios requiring high reliability (such as highway fast charging stations) or millisecond-level response, the original solution (Jetson Nano+TCN+PPO+hybrid cloud) is still a better choice.

[0087] 1. Comparative analysis of training data: Experimental data show that the present invention exhibits significant advantages in multi-dimensional performance indicators. Compared with the global optimal solution of Genetic Algorithm (GA), basic strategy, and MILP, TCN-PPO achieves a profit of 8779.54, significantly higher than GA (6896.49) and basic strategy (4959.68). Although slightly lower than MILP's 10273.22, its success rate (0.98) is close to MILP (0.99) and significantly better than GA (0.94) and basic strategy (1.00). In terms of power demand, TCN-PPO is 37, between GA (19) and MILP (42), while its efficiency index (237.3) is significantly higher than GA (167.3) and basic strategy (91.8), and only slightly lower than MILP (244.6). In terms of stability, TCN-PPO's volatility level (36.21%) is lower than GA (67.24%) and the basic strategy (0%), but higher than MILP (27.59%), which comprehensively reflects its balance between optimization objectives and stability.

[0088] 2. Interference test performance evaluation: Stability test results for multi-source data disturbances show that the model exhibits strong robustness under disturbances in power, vehicle arrival, temperature data, and their combinations. When the disturbance intensity increases from 0.05 times to 0.20 times the data standard deviation, the profit stabilizes between 8415.83 and 8562.93, and the normalized battery swapping success rate remains within the range of 0.95-0.96. Specifically, at a disturbance of 0.05 times, the profit is 8539.46 and the success rate is 0.96; at a disturbance of 0.20 times, the profit decreases to 8415.83 and the success rate is 0.95. The performance stability index indicates that the system maintains an acceptable service level under all disturbance conditions. The experimental results verify the model's adaptive capability in complex data fluctuation environments, providing data support for uncertainty management in practical applications.

[0089] 3. Emergency response capability under extreme working conditions: The extreme power outage testing framework comprehensively evaluated the system's emergency response capabilities through two typical scenarios: peak-hour power outages and prolonged power outages. Under the constraint of a forced zero grid input power, the agent relied on battery-to-battery power transfer to maintain service. Data shows that in the no-power-out scenario, the profit was 8782 with a success rate of 0.98; during peak-hour power outages, the profit increased to 9334 (profit decrease rate -6.3%), and the success rate decreased to 0.92 (a decrease of 6.4%); during prolonged power outages, the profit was 8355 (a decrease of 4.9%), and the success rate was 0.90 (a decrease of 8.9%). These results demonstrate that the system achieves profit optimization through dynamic resource scheduling during peak-hour power outages, while maintaining basic service levels even during prolonged power outages, validating the effectiveness and reliability of the emergency response strategy.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A battery swapping station energy dispatching method based on reinforcement learning algorithm, characterized in that, The method includes: The steps to construct a TCN time series prediction model include: Prepare time-series training data, calculate the predicted values ​​through forward propagation, calculate the loss using the MSE between the predicted and true values, and finally use the Adam optimizer to achieve backpropagation and gradient descent, iteratively updating the network weights to minimize the error. The demand prediction results of the TCN prediction model are coupled with the basic environmental data to form the observation space of the deep reinforcement learning model. Based on this, the deep reinforcement learning algorithm PPO is used to train the model, construct a multi-step advantage function estimation framework, and calculate the advantage value at each time step. Introduce pruning constraints to limit the update range of the strategy; The parameters of the PPO prediction model are updated based on the interaction data between the agent and the environment, and the trained PPO model is used for energy scheduling.

2. The energy dispatching method for battery swapping stations based on reinforcement learning algorithm according to claim 1, characterized in that, The observation space has 36 dimensions, including at least the current time, electricity price, temperature, whether it is a holiday, and the predicted demand for vehicle battery swapping.

3. The energy dispatching method for battery swapping stations based on reinforcement learning algorithm according to claim 1, characterized in that, The TCN prediction model includes an Actor-Critic network, in which the Actor network consists of multiple fully connected layers. The Actor network incorporates the ReLU activation function and LayerNorm processing of the input data.

4. The energy dispatching method for battery swapping stations based on reinforcement learning algorithm according to claim 3, characterized in that, The Softmax function converts the battery action scores output by the Actor network into probability values. During the Actor evaluation phase, it selects the action with the highest probability among charging, discharging, and no operation. During the Actor training phase, a sampling strategy is adopted for each battery action, and the corresponding log probability is recorded for subsequent policy gradient updates.

5. The energy dispatching method for battery swapping stations based on reinforcement learning algorithm according to claim 3, characterized in that, In the Actor-Critic network, the output of the Critic network is a scalar estimate of the value at the current time and battery level. The Critic network is used to calculate the state value at each step and compare it with the actual reward. If the state value is greater than the actual reward, it means that the value of this state is overestimated; otherwise, it means that the value of this state is underestimated.

6. The energy dispatching method for battery swapping stations based on reinforcement learning algorithm according to claim 5, characterized in that, By training a TCN to predict future electric vehicle battery swapping demand, and transforming this prediction data into a state vector embedded into a deep reinforcement learning framework, the observation space of the agent is effectively expanded. The PPO algorithm is used to train the deep reinforcement learning, and a multi-step advantage function estimation framework based on generalized advantage estimation is constructed. This framework calculates the advantage value at each time step by introducing time difference residuals: if the advantage value is positive, it indicates that the benefit of taking this action is higher than average; if it is negative, it indicates that the action performs worse than expected. At the same time, exponential weighting is used to process the TD residuals at each time step, effectively balancing the bias and variance of the estimation, and ensuring the accuracy and stability of the advantage estimation.

7. A battery swapping station energy dispatching system based on reinforcement learning algorithm, characterized in that, The system includes an interactive front-end, an interactive back-end, and an IoT platform. The interactive front-end includes a view layer, a business logic layer, and a data access layer. The interactive back-end includes a presentation layer, a business logic layer, a data access layer, and a data storage layer. The IoT platform includes a perception layer and a network layer.

8. The battery swapping station energy dispatching system based on reinforcement learning algorithm according to claim 7, characterized in that, The data access layer of the interactive front end supports GET, POST, PUT and DELETE requests, and uses interceptors to achieve unified authentication and error handling.

9. The battery swapping station energy dispatching system based on reinforcement learning algorithm according to claim 7, characterized in that, The presentation layer of the interactive backend adopts the Spring MVC framework based on the JavaEE platform, integrating Spring Boot and Spring Data plugins, and supports interaction with the charging pile power database and behavior database.

10. The battery swapping station energy dispatching system based on reinforcement learning algorithm according to claim 7, characterized in that, The business logic layer of the interactive backend adopts a Python+Django ORM architecture and uses a Jetson Nano processor to provide computing power support for edge computing. The trained prediction model and intelligent body are embedded in the processor to achieve real-time scheduling and reduce data transmission costs.

Images