Bagged reward framework-based battery swap station scheduling method and system, and storage medium

Abstract

The invention discloses a battery swap station scheduling method and system based on a bagged reward framework and a storage medium, and the method comprises the steps: collecting and preprocessing battery state, battery swap station state and environment state data; the spatio-temporal trajectory bag construction module constructs multi-dimensional state representation and multi-dimensional action representation, and divides trajectory bags; the delayed reward aggregator module generates a multi-target reward function for a bag-level reward based on the trajectory bag and through a delayed feedback calculator. A reward bag converter RBT model maps a state-action pair of a trajectory bag to a hidden space, and reward distribution is carried out by combining a space-time attention mechanism and adopting a reward decomposition algorithm; based on an RLBR framework and a reward bag converter RBT model, and in combination with a space-time trajectory bag construction module and a delay reward aggregator module, constructing a dynamic scheduling model of the power swap station based on a reinforcement learning bagged reward framework; performing multi-task optimization and training on the model; and performing dynamic scheduling of the battery swap station according to a dynamic decision behavior output by the trained model.

Description

Technical Field

[0001] The present invention relates to the field of intersectional technology of artificial intelligence and energy management, and in particular to a method, system and storage medium for scheduling battery swap stations based on a bagged reward framework. Background Art

[0002] Existing battery swap cabinet scheduling systems often use static, rule-based inventory allocation strategies that are unable to adapt to demand fluctuations in real time, resulting in delayed dynamic scheduling responses. During peak hours in the afternoon and evening, battery swap stations experience high outage rates, leading to longer average wait times for riders. Current mainstream solutions rely on fixed charging thresholds and fail to account for the cumulative effects of historical battery charge and discharge stress, resulting in extensive battery health management. Field data shows that frequent fast charging can reduce battery cycle life from 1,500 cycles to 1,200 cycles or even lower. Platforms such as battery swap platforms often use ARIMA time series models for demand forecasting, but response to emergencies can be delayed by over two hours. Delayed reward processing is insufficient, and traditional reinforcement learning relies on single-step, immediate rewards, making it ineffective at modeling the long-term metrics of battery swap station operations. Time series correlations are broken, and existing scheduling systems have limited ability to model the long-term impact of action sequences. The impact of peak-hour charging decisions on subsequent stations over the next six hours is often overlooked. Multimodal data, such as battery status (temperature, SOC) and user requests (location, time), is difficult to uniformly represent, resulting in suboptimal strategies and deficiencies in heterogeneous data fusion. Summary of the Invention

[0003] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a battery swap station scheduling method, system and storage medium based on a bagged reward framework, a dynamic scheduling method based on the RLBR framework and the RBT model, and delayed reward modeling, action sequence correlation analysis and multi-source data fusion in the battery swap station scenario to achieve dynamic scheduling in the battery swap station.

[0004] Technical solution: To achieve the above-mentioned purpose, the battery swap station scheduling method based on the bagged reward framework of the present invention includes collecting and preprocessing battery status, battery swap station status and environmental status data; constructing multi-dimensional state representation and multi-dimensional action representation based on the spatiotemporal trajectory bag construction module, and dividing the trajectory bag; using the delayed reward aggregator module to generate a multi-objective reward function of bag-level rewards based on the trajectory bag and through the delayed feedback calculator; using the reward bag transformer RBT model to map the state-action pairs of the trajectory bag to the latent space, combining the spatiotemporal attention mechanism and using the reward decomposition algorithm to distribute rewards; based on the RLBR framework and the reward bag transformer RBT model, and combining the spatiotemporal trajectory bag construction module and the delayed reward aggregator module, a battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework is constructed; multi-task optimization is performed on the obtained model, and the model is trained; dynamic scheduling of battery swap stations is performed according to the dynamic decision-making behavior output by the trained model.

[0005] Furthermore, the battery status includes the battery state of charge SOC, battery health SOH and battery temperature; the battery swap station status includes the inventory I∈Z + and queue length Q∈Z + ; The environmental state includes a timestamp and a weather code; constructing a state vector s representing a multi-dimensional state t , s t =(SOC, SOH, battery temperature, inventory, queue length, timestamp, weather code); set the action data of the battery swap station including charging power selection α charge ∈{0, 1}, delivery instruction α deliver =(target site ID, number of batteries delivered)∈R 2 , and inventory allocation α allocate ∈{retain, call out}; construct the action vector a of multi-dimensional action representation t , a t =(α charge , α deliver , α allocate ).

[0006] Furthermore, the spatiotemporal trajectory bag construction module divides the decision cycle into trajectory bags of variable time length, sets the initial window length ΔT, and defines the state-action sequence within the time window [t0, t0+ΔT] as trajectory bag B;

[0007]

[0008] Where s t is the state vector at time t, a t is the action vector at time t, t0 is the initial time;

[0009] At the same time, a dynamic time window adjustment rule is established in the spatiotemporal trajectory bag construction module according to the set initial window length ΔT, so that the initial window length ΔT is dynamically adjusted according to the dynamic time window adjustment rule; the dynamic time window adjustment rule includes shortening the initial window length ΔT by 10% when the service efficiency variance exceeds 0.2; and extending the initial window length ΔT by 10% when the average delivery time exceeds a preset threshold.

[0010] Furthermore, the three optimization objectives of service efficiency, battery life and delivery cost are integrated through linear weighting, and a multi-objective reward function R of bag-level reward is constructed based on the trajectory bag B through a delayed feedback calculator. bag (B);

[0011]

[0012] Where η service is the service efficiency, ηdegradation is the battery life, η delivery is the delivery cost, α, β, and γ are the adjustable weights of service efficiency, battery life, and delivery cost, respectively;

[0013] The service efficiency is the ratio of the number of successful battery swaps to the total number of requests during the statistical window period.

[0014]

[0015] Where N success is the number of successful battery replacements, N total is the total number of battery swap requests;

[0016] The battery life is the reciprocal mean of all battery capacity decays within the calculation window period.

[0017]

[0018] Where N batt is the number of charge and discharge cycles within the window period, is the initial capacity of the battery, is the current capacity of the i-th battery;

[0019] The distribution cost is calculated based on the exponential decay function of the logistics path planning results;

[0020]

[0021] Where exp is the exponential function operation, k logistic is the total logistics cost, k threshold Set the cost threshold.

[0022] Furthermore, the reward bag transformer RBT model includes an input layer, a spatiotemporal attention module and a reward distributor; wherein the state-action pair (s t , a t ) is mapped to the 768-dimensional vector in the latent space to obtain the hidden state h t , based on the spatiotemporal attention module, the hidden state h is updated by weighted summation t ;

[0023] h t =GELU(W1s t +W2a t +b)

[0024] Where W1 and W2 are both trainable weight matrices, and b is the bias term.

[0025] Furthermore, the reward allocator decomposes the reward algorithm based on the multi-objective reward function of the bag-level reward. The formula for the temporal distribution of the bag-level reward is:

[0026]

[0027] Where R bag is the multi-objective reward function for bag-level rewards, is the Sigmoid function, α i ∈R is the learnable association weight, is the inner product of the hidden states at time steps i and t, h i is the hidden state at time step i, h′ t is the hidden state at time step t after the update, and n represents the total number of time steps in the trajectory bag.

[0028] Furthermore, the multi-task optimization is to train the scheduling strategy, charging strategy and delivery strategy in parallel, and achieve Pareto optimality through a weighted loss function;

[0029]

[0030] Where, is the scheduling strategy, For the charging strategy, is the distribution strategy, λ1, λ2 and λ3 are the weights of the scheduling strategy, charging strategy and distribution strategy respectively;

[0031] The scheduling strategy is to minimize the waiting time for battery replacement.

[0032]

[0033] The charging strategy is to maximize the battery life.

[0034]

[0035] The distribution strategy is to minimize the distribution cost.

[0036]

[0037] Furthermore, the maximization objective function for training the obtained battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework is:

[0038]

[0039] Where, For the old policy π old The state-action pairs sampled in (s t , a t )Find the expectation, where π old is the strategy before the current model parameters are updated; π θ(a|s) is the new strategy with parameter θ; clip is the importance sampling ratio; min is the minimization operation; ∈ is the hyperparameter; At is the advantage function, which is calculated by generalized advantage estimation GAE;

[0040]

[0041] δ t =r t +γV φ (s t+1 )-V φ (s t )

[0042] Where r t is the temporal distribution of bag-level rewards, ΔT is the initial window length, l is the step size, γ is the discount coefficient of future rewards, λ is the mixing weight for controlling multi-step TD residuals, V φ is a neural network with a state-value function parameter φ, s t is a multi-dimensional state vector, s t+1 To execute the action vector a t The next state vector after .

[0043] Furthermore, a dynamic scheduling system for battery swap stations based on a bagged reward framework based on reinforcement learning is provided. The system is used to implement a battery swap station scheduling method based on a bagged reward framework, including:

[0044] Data collection layer: used to collect battery status, battery swap station status and environmental status data;

[0045] RBT model layer: used to execute the operations of the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework;

[0046] Decision output layer: Converts the dynamic decision-making behavior output by the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework into specific decisions and outputs them to the execution agency.

[0047] Furthermore, a storage medium stores an executable program therein, and the executable program is executed by a processor to implement a battery swap station scheduling method based on a bagged reward framework.

[0048] Beneficial effects: The battery swap station scheduling method, system and storage medium based on the bagged reward framework of the present invention can significantly shorten the user waiting time and improve the battery swap success rate through dynamic scheduling and optimized decision-making, ensure that users can obtain fully charged batteries in time, and improve the overall service satisfaction and user battery swap experience; through intelligent charging strategies and battery health monitoring, it can effectively slow down the battery degradation rate, extend the battery cycle life, reduce asset maintenance costs, improve economic benefits and extend battery service life; through multi-source data fusion and intelligent path planning, it can improve the scheduling efficiency of delivery personnel, reduce logistics costs, increase the number of deliveries per unit time, optimize logistics distribution efficiency, and achieve efficient use of resources; through the introduction of reinforcement learning framework and reward distribution transformer (RBT), it can realize intelligent decision-making of battery swap station operations, reduce the need for manual intervention, and improve overall operational efficiency and system intelligence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 System diagram of the battery swap station scheduling method based on the bagged reward framework. DETAILED DESCRIPTION

[0050] The present invention will be further described below with reference to the accompanying drawings.

[0051] like Figure 1 As shown in the figure, the battery swap station scheduling method based on the bagged reward framework includes collecting and preprocessing battery status, battery swap station status and environmental status data; constructing multi-dimensional state representation and multi-dimensional action representation based on the spatiotemporal trajectory bag building module, and dividing the trajectory bag; adopting the delayed reward aggregator module to generate a multi-objective reward function of bag-level reward based on the trajectory bag and through the delayed feedback calculator; adopting the reward bag transformer RBT model to map the state-action pairs of the trajectory bag to the latent space, combining the spatiotemporal attention mechanism and adopting the reward decomposition algorithm to distribute the reward; based on the RLBR framework and the reward bag transformer RBT model, and combining the spatiotemporal trajectory bag building module and the delayed reward aggregator module, a battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework is constructed; mainly based on the RLBR framework reward bag transformer RBT model to construct a new model and introduce the spatiotemporal trajectory bag building module and the delayed reward aggregator module to construct the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework. The obtained dynamic scheduling model of battery swap stations based on the reinforcement learning bagged reward framework is optimized for multiple tasks, and the dynamic scheduling model of battery swap stations based on the reinforcement learning bagged reward framework is trained; the battery swap stations are dynamically scheduled according to the dynamic decision-making behavior output by the trained dynamic scheduling model of battery swap stations based on the reinforcement learning bagged reward framework.

[0052] The RLBR framework is a routing framework based on reinforcement learning. As the overall framework of reinforcement learning, the RLBR framework defines the task objectives and learning mechanisms for dynamic scheduling. The reward bag transformer (RBT) model is responsible for mapping the state-action pairs in the trajectory bag to the latent space and realizing fine-grained reward distribution through the spatiotemporal attention module and reward decomposition algorithm.

[0053] The battery status includes the battery state of charge SOC, battery health SOH and battery temperature; the battery exchange station status includes the inventory I∈Z + and queue length Q∈Z + The environmental state includes a timestamp and a weather code. The timestamp is an integer value from 0 to 23 representing the hour. The weather code converts different weather conditions into a one-hot vector. Construct a state vector s representing a multi-dimensional state. t , s t =(SOC, SOH, battery temperature, inventory, queue length, timestamp, weather code); set the action data of the battery swap station including charging power selection α charge ∈{0, 1} where 0 or 1 represents low / high power charging mode respectively; delivery instruction α deliver =(target site ID, number of batteries delivered)∈R 2 , expressed as a natural number tuple, the delivery instruction is to deliver the number of batteries from this battery swap station to a certain battery swap station; and the inventory allocation α allocate ∈{retain, call out}, using binary classification decision representation; similarly, the action data of the battery swap station is collected by acquisition, or sampled from the strategy of the battery swap station; the action vector a of the multi-dimensional action representation is constructed t , a t =(α charge , α deliver , α allocate ).

[0054] The spatiotemporal trajectory bag construction module divides the decision cycle into trajectory bags of variable time length. Dividing into multiple trajectory bags can reduce the sampling frequency and avoid the impact of local optimal solutions on the global optimal solution. Many collected data cannot be collected at a higher frequency, so only partial results for a period of time can be observed. Therefore, dividing the trajectory bags avoids this situation. Based on the statistical results of historical operation data, the initial window length ΔT is set, and the state-action sequence within the time window [t0, t0+ΔT] is defined as trajectory bag B.

[0055]

[0056] Where s t is the state vector at time t, a t is the action vector at time t, t0 is the initial time;

[0057] At the same time, a dynamic time window adjustment rule is established in the spatiotemporal trajectory bag construction module based on the set initial window length ΔT. The initial window length ΔT can be set to 4 hours, so that the initial window length ΔT is dynamically adjusted according to the dynamic time window adjustment rule; the dynamic time window adjustment rule includes shortening the initial window length ΔT by 10% when the service efficiency variance exceeds 0.2; and extending the initial window length ΔT by 10% when the average delivery time exceeds the preset threshold.

[0058] By linearly weighting the three optimization objectives of service efficiency, battery life and delivery cost, a multi-objective reward function R of bag-level reward is constructed based on the trajectory bag B through a delayed feedback calculator. bag (B):

[0059]

[0060] Where η service is the service efficiency, η degradation is the battery life, η delivery is the delivery cost, α, β, and γ are the adjustable weights of service efficiency, battery life, and delivery cost, respectively. The weights are adaptively adjusted according to the target variance. The weight is calculated by the inverse of the variance σ of each component. The smaller the variance, the greater the target weight, indicating that the target is more stable and more worthy of priority optimization.

[0061]

[0062] Where α, β, and γ satisfy α+β+γ=1, the variance is counted according to the trajectory bag window and the parameters are updated, and softmax is the softmax function.

[0063] The service efficiency is the ratio of the number of successful battery swaps to the total number of requests during the statistical window period.

[0064]

[0065] Where N success is the number of successful battery replacements, N total is the total number of battery swap requests;

[0066] The battery life is the reciprocal mean of all battery capacity decays within the calculation window period.

[0067]

[0068] Where N batt is the number of charge and discharge cycles within the window period, is the initial capacity of the battery, is the current capacity of the i-th battery;

[0069] The distribution cost is calculated based on the exponential decay function of the logistics path planning results, and the exponential decay function is the distribution cost;

[0070]

[0071] Where exp is the exponential function operation, k logistic is the total logistics cost, k threshold The cost threshold is set as the historical average cost or the budget upper limit if there is no historical data, to measure whether the cost is reasonable.

[0072] The reward bag transformer RBT model includes an input layer, a spatiotemporal attention module and a reward distributor; wherein the state-action pair (s t , a t ) is mapped to the 768-dimensional vector in the latent space to obtain the hidden state h t , based on the spatiotemporal attention module, the hidden state h is updated by weighted summation t , identify key decision nodes, the attention weight reflects the importance of different time steps, which in turn affects the hidden state representation of each time step and achieves the update of the hidden state;

[0073] h t =GELU(W1s t +W2a t +b)

[0074] Where W1 and W2 are both trainable weight matrices, and b is the bias term, d h =768 is the latent space dimension.

[0075] The spatiotemporal attention module is a bidirectional attention module that uses a multi-head attention function;

[0076]

[0077] Where, Q = [h t-τ ,…,h t ] is the query matrix of the historical state sequence, K=[h t ,…,h t+τ ] is the key matrix of the future state, V=[h t-τ ,…,h t+τ ] is the value matrix of the global context, d k is the single-head attention dimension, N heads is the number of heads in the multi-head attention function;

[0078] h′ t =LayerNorm(attention+h t)

[0079] Where h′ t It is the hidden state after being updated by the spatiotemporal attention module. LayerNorm is the normalization operation. The updated hidden state needs to be normalized.

[0080] The reward allocator decomposes the reward algorithm based on the multi-objective reward function of the bag-level reward, decomposes the bag-level reward into each time step, and generates pseudo-instantaneous rewards for strategy updates; the formula for the temporal distribution of the bag-level reward is:

[0081]

[0082] Where R bag is the multi-objective reward function for bag-level rewards, is the Sigmoid function, α i ∈R is the learnable association weight, is the inner product of the hidden states at time steps i and t, h i is the hidden state at time step i, h′ t is the hidden state at time step t after the update, and n represents the total number of time steps in the trajectory bag.

[0083] Temporal distribution of bag-level rewards t It is an artificially assigned guidance signal that aims to decompose delayed rewards into optimizable short-term goals, which does not need to be consistent with the real environment rewards. The generated pseudo-immediate reward directly participates in the policy gradient calculation, and drives the model to optimize action selection through the advantage function At. A specific example is the priority adjustment of delivery instructions.

[0084] The multi-task optimization is to train the scheduling strategy, charging strategy and distribution strategy in parallel, and achieve Pareto optimality through weighted loss function; the calculation of the scheduling strategy, charging strategy and distribution strategy is the calculation part of the loss function, and the strategy obtained by minimizing the loss function is applied to The old strategy π old The state-action pairs sampled in (s t , a t ) to find the expectation; here is the calculation of a part of the loss function in the calculation of the maximization objective function;

[0085]

[0086] Where, is the scheduling strategy, For the charging strategy, is the distribution strategy, λ1, λ2 and λ3 are the weights of the scheduling strategy, charging strategy and distribution strategy respectively, and can be adjusted according to actual conditions;

[0087] The scheduling strategy is to minimize the waiting time for battery replacement.

[0088]

[0089] Where E is the expected operation, which is the expected value of the scheduling delay after t, that is, the user's battery replacement time; τ t is the scheduling delay after t, is the actual waiting time;

[0090] The charging strategy is to maximize the battery life.

[0091]

[0092] Where N batt is the number of charge and discharge cycles within the window period, is the initial capacity of the battery, is the current capacity of the i-th battery;

[0093] The distribution strategy is to minimize the distribution cost and impose penalties when the distribution cost exceeds a threshold;

[0094]

[0095] In the formula, ReLU is the activation function, c k is the historical average delivery cost, C threshold is the budget cap threshold;

[0096] The maximization objective function when training the obtained battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework is:

[0097]

[0098] Where, For the old policy π old The state-action pairs sampled in t , a t )Find the expectation, where π old The strategy before the current model parameters are updated is the old strategy; Used to generate historical trajectory data state s t (such as battery SOC, inventory, timestamp) and action a t (such as charging power selection, delivery instructions) are sampled from the decision records of the old strategy. θ (a|s) is the new strategy with parameter θ, which means that in state s t Next select action a tThe probability of: The battery swap station scheduling model dynamically generates decision rules for actions (such as charging and battery allocation) based on the current status (such as queue length and battery health); π old (a|s) and π old The same is the old policy. clip is the importance sampling ratio, which is limited to the interval [1-∈, 1+∈] to prevent the policy update from being too large; min is the minimization operation; ∈ is a hyperparameter, generally set to 0.02, which controls the conservativeness of the policy update; through the importance sampling ratio, the policy is updated using historical data while limiting the update amplitude. At is the advantage function, calculated by the generalized advantage estimation GAE;

[0099]

[0100] δ t =r t +γV φ (s t+1 )-V φ (s t )

[0101] Where r t is the temporal distribution of bag-level rewards, ΔT is the initial window length, and l is the step size; γ is the discount coefficient for future rewards, ranging from [0, 1]. The smaller the value, the more attention is paid to immediate rewards (short-sighted decision-making), and the larger the value, the more attention is paid to long-term benefits (far-sighted decision-making); λ is the mixed weight controlling the multi-step TD residual, ranging from [0, 1]. V φ is a neural network with a state value function parameter of φ, predicting the state vector s t The expected cumulative reward of t is a multi-dimensional state vector, s t+1 To execute the action vector a t The next state vector after δ t Measure the current action vector a t The immediate effect of δ t+1 Measure the future l-th step action vector a t+1 The immediate effect of At is the generalized advantage value of the comprehensive multi-step residual. For example, in the scheduling of battery swap stations, if a delivery instruction (a t ) in the short term (l=0) increases the cost (δ t <0), but in the long term (l=3), it alleviates the battery shortage at multiple sites (δ t+3 >0), then At may still be a positive value.

[0102] A dynamic scheduling system for battery swap stations based on a bagged reward framework based on reinforcement learning. This system is used to implement a scheduling method for battery swap stations based on a bagged reward framework, including:

[0103] Data collection layer: used to collect battery status, battery swap station status and environmental status data;

[0104] RBT model layer: used to execute the operations of the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework;

[0105] Decision output layer: Converts the dynamic decision-making behavior output by the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework into specific decisions and outputs them to the execution agency.

[0106] The dynamic scheduling system for battery swap stations based on the reinforcement learning bagged reward framework includes a data acquisition layer, an RBT model layer, and a decision output layer. The data acquisition layer collects battery status through battery sensors, the station monitoring module collects battery swap station status, and the environmental perception module collects weather codes and timestamps. The RBT model layer is the constructed dynamic scheduling model for battery swap stations based on the reinforcement learning bagged reward framework. The decision output layer includes a multi-task decision tree and an actuator interface. The multi-task decision tree converts the output results of the RBT model layer into specific decisions, such as the selection of battery charging power, the delivery of several batteries from a certain battery swap station to another battery swap station, and the guarantee of the inventory quantity at a certain battery swap station. The actuator interface transmits information to the execution structure or computer terminal or other receiving device through a communication protocol, notifying personnel or machinery to make specific dispatches based on the specific output decisions.

[0107] A storage medium stores an executable program, which is executed by a processor to implement a battery swap station scheduling method based on a bagged reward framework.

[0108] The training strategy or method for a dynamic battery swap station scheduling model based on a reinforcement learning bagged reward framework includes an initial phase, an intermediate phase, and a final phase. The model is first trained in the initial phase, i.e., single-station training. This includes an environment configuration and training objectives. The environment configuration includes a single battery swap station, a single battery type, a fixed ΔT of 4 hours, and no unexpected events. The training objectives include maintaining the battery SOC between 90% and 100% and achieving an inventory allocation decision accuracy greater than 95%. The model is then trained in the intermediate phase, i.e., multi-station coordination. This includes an environment configuration and progressive expansion. The environment configuration includes 300-500 stations, mixed battery types, dynamic ΔT, and the introduction of path planning. Progressive expansion involves adding 50 stations every 100 training rounds until the number of trained stations reaches 500. Finally, the model is trained in the final phase, i.e., full-area scheduling. This includes an environment configuration and an anti-forgetting mechanism. The environment configuration includes 20,000+ stations, four battery types (including low-temperature ones), and random unexpected events. The anti-forgetting mechanism involves freezing the initial network layers and fine-tuning only the top-level attention module. Through the three stages of training and learning, namely the initial stage, the mid-term stage and the final stage, the obtained model is gradually trained with increasing difficulty, so that the model can output better decisions when scheduling.

[0109] When training the dynamic scheduling model of battery swap stations based on the reinforcement learning bagged reward framework, the status data and action data of each battery swap station are obtained; the status data and action data are input into the dynamic scheduling model of battery swap stations based on the reinforcement learning bagged reward framework for training; when performing dynamic scheduling through the dynamic scheduling model of battery swap stations based on the reinforcement learning bagged reward framework; first collect the current status data s of each battery swap station t The collected data is input into the dynamic scheduling model of the battery swap station based on the reinforcement learning bag reward framework. The dynamic scheduling model of the battery swap station based on the reinforcement learning bag reward framework outputs a specific decision based on the input data, which is the action a that the current state data should correspond to. t , is also based on the current status data of each battery swap station. After the action data a t After the execution of the action, the battery swap station obtains the next state data s t+1 The dynamic scheduling model of battery swap stations based on the reinforcement learning bag reward framework obtains the status data of each battery swap station at regular intervals. t , in real time, the current scheduling decisions for each battery swap station should be output, and the battery swap stations can be dynamically scheduled, forming a method for continuously outputting scheduling decisions in real time to achieve dynamic scheduling of battery swap stations.

[0110] By conducting real-time monitoring and intelligent analysis of multi-dimensional data such as battery status, user needs, and logistics distribution at battery swap stations, we can optimize scheduling strategies to meet user battery swap needs while maximizing battery utilization efficiency and reducing operating costs. Furthermore, this significantly improves user experience, shortens potential wait times for battery swaps, reduces energy waste, and promotes the intelligent and efficient development of battery swap station operations.

[0111] The above is only a description of the preferred embodiment of the present invention. Ordinary technicians in this technical field can make several modifications and optimizations based on the above disclosure without departing from the above basic principles. These improvements and optimizations should be regarded as the scope of protection understood by the present invention.

Claims

1. A battery swap station scheduling method based on a bagged reward framework, characterized by: It includes collecting and preprocessing battery status, battery swap station status and environmental status data; constructing multi-dimensional state representation and multi-dimensional action representation based on the spatiotemporal trajectory bag building module, and dividing the trajectory bag; using the delayed reward aggregator module to generate a multi-objective reward function of bag-level rewards based on the trajectory bag and through the delayed feedback calculator; using the reward bag transformer RBT model to map the state-action pairs of the trajectory bag to the latent space, combining the spatiotemporal attention mechanism and using the reward decomposition algorithm for reward distribution; based on the RLBR framework and the reward bag transformer RBT model, and combining the spatiotemporal trajectory bag building module and the delayed reward aggregator module, a dynamic scheduling model of the battery swap station based on the reinforcement learning bag reward framework is constructed; the obtained model is optimized for multiple tasks and trained; and the battery swap station is dynamically scheduled according to the dynamic decision-making behavior output by the trained model.

2. The method for scheduling battery swap stations based on a bagged reward framework according to claim 1, characterized in that: The battery status includes the battery state of charge SOC, battery health SOH and battery temperature; the battery exchange station status includes the inventory I∈Z + and queue length Q∈Z + ; The environmental state includes a timestamp and a weather code; constructing a state vector s representing a multi-dimensional state t , s t =(SOC, SOH, battery temperature, inventory, queue length, timestamp, weather code); set the action data of the battery swap station including charging power selection α charge ∈{0,1}, delivery instruction α deliver =(target site ID, number of batteries delivered)∈R 2 , and inventory allocation α allocate ∈{retain, call out}; construct the action vector a of multi-dimensional action representation t , a t =(α charge , α deliver , α allocate ).

3. The method for scheduling battery swap stations based on a bagged reward framework according to claim 2, characterized in that: In the spatiotemporal bag-of-trajectory construction module, the decision cycle is divided into bag-of-trajectory of variable time length, the initial window length ΔT is set, and the state-action sequence within the time window [t0, t0+ΔT] is defined as bag-of-trajectory B; Where s t is the state vector at time t, a t is the action vector at time t, t0 is the initial time; At the same time, a dynamic time window adjustment rule is established in the spatiotemporal trajectory bag construction module according to the set initial window length ΔT, so that the initial window length ΔT is dynamically adjusted according to the dynamic time window adjustment rule; the dynamic time window adjustment rule includes shortening the initial window length ΔT by 10% when the service efficiency variance exceeds 0.2; and extending the initial window length ΔT by 10% when the average delivery time exceeds a preset threshold.

4. The method for scheduling battery swap stations based on a bagged reward framework according to claim 1, characterized in that: By linearly weighting the three optimization objectives of service efficiency, battery life and delivery cost, a multi-objective reward function R of bag-level reward is constructed based on the trajectory bag B through a delayed feedback calculator. bag (B); Where η service is the service efficiency, η degradation is the battery life, η delivery is the delivery cost, α, β, and γ are the adjustable weights of service efficiency, battery life, and delivery cost, respectively; The service efficiency is the ratio of the number of successful battery swaps to the total number of requests during the statistical window period. Where N success is the number of successful battery replacements, N total is the total number of battery swap requests; The battery life is the reciprocal mean of all battery capacity decays within the calculation window period. Where N batt is the number of charge and discharge cycles within the window period, is the initial capacity of the battery, is the current capacity of the i-th battery; The distribution cost is calculated based on the exponential decay function of the logistics path planning results; Where exp is the exponential function operation, k logistic is the total logistics cost, k threshold Set the cost threshold.

5. The method for scheduling battery swap stations based on a bagged reward framework according to claim 1, characterized in that: The Reward Bag Transformer (RBT) model includes an input layer, a spatiotemporal attention module, and a reward allocator. The feature embedding layer embeds the state-action pairs (s t , a t ) is mapped to the 768-dimensional vector in the latent space to obtain the hidden state h t , based on the spatiotemporal attention module, the hidden state h is updated by weighted summation t ; h t =GEL(W1s t +W2a t +b) Where W1 and W2 are both trainable weight matrices, and b is the bias term.

6. The method for scheduling battery swap stations based on a bagged reward framework according to claim 1, characterized in that: The reward allocator decomposes the reward algorithm based on the multi-objective reward function of the bag-level reward. The formula for the temporal distribution of the bag-level reward is: Where R bag is the multi-objective reward function for bag-level rewards, is the Sigmoid function, α i ∈R is the learnable association weight, is the inner product of the hidden states at time steps i and t, h i is the hidden state at time step i, h′ t is the hidden state at time step t after the update, and n represents the total number of time steps in the trajectory bag.

7. The method for scheduling battery swap stations based on a bagged reward framework according to claim 1, characterized in that: The multi-task optimization is to train the scheduling strategy, charging strategy and delivery strategy in parallel, and achieve Pareto optimality through a weighted loss function; Where, is the scheduling strategy, For the charging strategy, is the distribution strategy, λ1, λ2 and λ3 are the weights of the scheduling strategy, charging strategy and distribution strategy respectively; The scheduling strategy is to minimize the waiting time for battery replacement. The charging strategy is to maximize the battery life. The distribution strategy is to minimize the distribution cost.

8. The method for scheduling battery swap stations based on a bagged reward framework according to claim 1, characterized in that: The maximization objective function when training the obtained battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework is: Where, For the old policy π old The state-action pairs sampled in t , a t )Find the expectation, where π old is the strategy before the current model parameters are updated; π θ (a|s) is the new strategy with parameter θ; clip is the importance sampling ratio; min is the minimization operation; ∈ is the hyperparameter; At is the advantage function, which is calculated by generalized advantage estimation GAE; δ t =r t +γV φ (s t+1 )-V φ (s t ) Where r t is the temporal distribution of bag-level rewards, ΔT is the initial window length, l is the step size, γ is the discount coefficient of future rewards, λ is the mixing weight for controlling multi-step TD residuals, V φ is a neural network with a state-value function parameter φ, s t is a multi-dimensional state vector, s t+1 To execute the action vector a t The next state vector after .

9. A dynamic scheduling system for battery swap stations based on a bagged reward framework based on reinforcement learning, the system being used to implement the battery swap station scheduling method based on a bagged reward framework as described in any one of claims 1 to 8, characterized in that: include: Data collection layer: used to collect battery status, battery swap station status and environmental status data; RBT model layer: used to execute the operations of the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework; Decision output layer: Converts the dynamic decision-making behavior output by the battery swap station dynamic scheduling model based on the reinforcement learning bagged reward framework into specific decisions and outputs them to the execution agency.

10. A storage medium, characterized in that: An executable program is stored therein, and the executable program is executed by a processor to implement the battery swap station scheduling method based on the bagged reward framework as described in any one of claims 1 to 8.

Images