A mobile charging service arrangement management method based on deep reinforcement learning

Abstract

The application discloses a kind of mobile charging service arrangement management methods based on deep reinforcement learning, belong to deep learning technical field.The method utilizes the social network topological structure between mobile charging car, electric vehicle and control center, when electric vehicle user sends charging request information to control center, control center will receive and process these information, then utilize transformer-based neural network model to calculate the best charging service sequence arrangement, and send the service arrangement instruction to mobile charging car.Mobile charging car will provide charging service for electric vehicle as far as possible according to instruction in the time window of user expectation, to meet the charging demand of user, to realize the best service scheduling.The application not only helps the cost control of vehicle network, but also can improve the charging experience of electric vehicle user.

Description

Technical Field

[0001] This invention relates to technology in the field of mobile charging services, and more specifically, to a method for solving the scheduling and management problem in mobile charging services. Background Technology

[0002] This invention relates to a deep reinforcement learning-based path planning method for mobile charging vehicles, addressing the charging needs of electric vehicles to adapt to the increasing prevalence of electric vehicles and growing environmental awareness in future transportation. Electric vehicles, due to their high energy efficiency and low emissions, are gradually replacing traditional internal combustion engine vehicles. The International Energy Agency predicts that by 2030, electric vehicle sales will exceed 20% of global vehicle sales. However, electric vehicle charging planning faces several challenges. Traditional fixed charging stations are inefficient and cannot meet constantly changing charging demands, while battery replacement is costly and incompatible. To address these challenges, mobile charging vehicles have emerged as an emerging solution. Mobile charging vehicles can provide charging services on demand without relying on fixed infrastructure, have wide coverage, and can adapt to the flexible charging needs of urban and rural areas. The application of smart charging technology enables mobile charging vehicles to achieve fast charging, high energy efficiency, and environmental sustainability.

[0003] Mobile charging route planning is a complex multi-objective combinatorial optimization problem, aiming to simultaneously optimize energy consumption and time cost along the charging vehicle's route. In recent years, researchers have begun to apply deep reinforcement learning algorithms to solve combinatorial optimization problems. Among these, some advanced models, such as pointer network models and transformer models, have been introduced to handle multi-objective optimization problems like the Traveling Salesman Problem (TSP), improving the efficiency and quality of problem-solving to some extent. Inspired by deep reinforcement learning methods, some scholars have proposed new deep reinforcement learning algorithms such as DRL-MOA to solve multi-objective optimization problems, achieving excellent performance and generalization ability. However, most current work focuses on solving specific problems like the TSP and has not been widely applied to multi-objective combinatorial optimization problems in real-world mobile charging scenarios.

[0004] In summary, existing technologies have certain limitations in mobile charging vehicle route planning. Therefore, this invention aims to combine deep reinforcement learning algorithms and transformer models to propose an efficient and flexible mobile charging service orchestration and management method to optimize charging route planning, adapting to the growing electric vehicle market and increasing environmental awareness. By optimizing charging vehicle routes, this invention helps promote the popularization of electric vehicles and the development of environmentally friendly transportation. Summary of the Invention

[0005] Purpose of the invention: This invention is dedicated to studying the orchestration and management of mobile charging services, aiming to provide additional charging capabilities for electric vehicles, achieve dual control of the energy consumption cost and time penalty cost of electric vehicle charging, and improve the mobile charging experience for electric vehicle users while controlling the cost of the charging network.

[0006] Technical Solution: A mobile charging service orchestration and management method based on deep reinforcement learning. The method involves electric vehicle users issuing request information, which is received by the control center. The center then orchestrates the charging service sequence based on the mobile charging vehicle service information. The steps are as follows:

[0007] (1) Construct a relational network model and establish a social relational network topology between the mobile charging vehicle, the electric vehicle and the control center. In the relational network model, the mobile charging vehicle, the electric vehicle and the control center exchange information through bidirectional communication lines.

[0008] (2) Determine the optimization objective function based on the charging service information of the mobile charging vehicle and the charging demand information of the electric vehicle. The charging service information includes the location of the charging base station and the energy storage capacity carried by the mobile charging vehicle. The charging demand information includes the location of the electric vehicle, the required energy capacity, and the expected time window.

[0009] This step comprehensively considers the benefits of minimizing the total length of the service path and the total cost of the time penalty, forming a total optimization objective function to achieve both optimization objectives simultaneously.

[0010] (3) Construct a neural network model and train it. Take charging service information and charging demand information as input, and calculate the order in which the mobile charging vehicle provides charging services to electric vehicles through the neural network.

[0011] (4) The control center schedules mobile charging vehicles to provide charging services for electric vehicles according to the overall optimization objective function, and completes the charging task.

[0012] Furthermore, the relationship network model described in step (1) is set as a vehicle-to-everything (V2X) network within a service area, including a mobile charging vehicle and multiple electric vehicles, along with a control center and a base station for replenishing the mobile charging vehicle's power. The control center determines how to schedule charging tasks based on relevant information to meet the charging needs of electric vehicles within the expected time window as much as possible. In this environment, there is no direct connection between the mobile charging vehicle and the electric vehicles; instead, the coordination and allocation of charging tasks are handled by the control center. This architecture enables efficient mobile charging vehicle service scheduling, providing convenient charging services for electric vehicle users.

[0013] Based on this, the information exchanged in step (1) includes the sending and receiving of charging service instructions, the electric vehicle sending charging demand information to the control center, and the mobile charging vehicle providing charging services to the electric vehicle under the guidance of the control center.

[0014] Through this two-way communication, charging needs and charging service instructions can be transmitted efficiently, making the entire charging scheduling process more intelligent and flexible.

[0015] Furthermore, regarding the multi-objective optimization problem of mobile charging services described in step (2), this invention considers an urban environment where electric vehicles are randomly distributed on a two-dimensional urban map, and the charging service sequence is ρ={ρ1,ρ2,…,ρ m}, where ρ m This represents the m-th access node (electric vehicle or charging base station).

[0016] Each electric vehicle's power demand is met only once during servicing, and it will not be visited twice. Mobile charging vehicles may visit charging stations multiple times to replace batteries. The starting and ending nodes of the access route are the charging base station nodes by default, denoted as cd0. Therefore, the constraint on the access order is represented as: ρ i ≠ρ j if i≠j,ρ i ≠cd0andρ j ≠ cd0. Mobile charging vehicles start their charging service journey with fully charged batteries, and the energy consumption during the journey is their own cost. Shortening the total travel distance of mobile charging vehicle services is crucial to minimizing travel costs.

[0017] Therefore, the objective function corresponding to minimizing the total length of the service path in step (2) is as follows:

[0018]

[0019] Where ρ represents the charging service sequence of the mobile charging vehicle, ρ = {ρ1, ρ2, ..., ρ...} m}, ρ m Let m represent the m-th visited node. dist calculates the travel distance between two visited nodes. When providing charging services for electric vehicles, it is crucial to meet the expected time window requirements of electric vehicle users.

[0020] If the mobile charging vehicle arrives at the electric vehicle's requested location outside the expected time window, it will receive a negative evaluation from the system, and the user will be compensated accordingly. Therefore, arriving on time within the time window is crucial to reducing penalty costs. Penalties are mainly divided into two scenarios: early arrival and late arrival. In the first scenario, if the mobile charging vehicle arrives at the designated location before the electric vehicle user, it will choose another nearby node for charging; this unnecessary journey should be avoided as much as possible. The second scenario involves late arrival; if the mobile charging vehicle exceeds the latest deadline of the time window, the user may find the car not fully charged upon arrival, leading to a degraded user experience and time-based penalties.

[0021] Therefore, the second objective is to minimize these time-based penalty costs, that is, the objective function for minimizing time penalty costs is expressed as follows:

[0022]

[0023] Where p e p l These represent the penalty coefficients for arriving early and late, respectively. This indicates that the mobile charging vehicle arrived at node ρ earlier than expected. j Location duration, mobile charging vehicle delay ρ j Duration of location, This indicates that the mobile charging vehicle has arrived at ρ. j Time, Represents ρ j The initial time point of the expected time window. If the mobile charging vehicle arrives earlier than the time window, the early arrival time is calculated; specifically, if ρ j =cd0, then the early or late arrival time is directly set to 0. This time is related to the mobile charging vehicle's travel time and the charging time of the electric vehicle before it; the travel time and charging time together constitute the early or late arrival time. The calculation formula is as follows:

[0024]

[0025] Where v represents the speed of the mobile charging vehicle, d k And ε correspond to ρ respectively k The power demand and charging rate. Similarly, the lateness time can be calculated as... Represents ρ j The expected end time of the time window.

[0026] The mobile charging service involved in this invention aims to plan a service sequence that meets users' power needs while optimizing two objectives. The first objective is to minimize the total travel distance of the mobile charging vehicle, thereby reducing the energy consumption cost of the vehicle itself. The second objective is to minimize the time penalty cost, which reflects user satisfaction and guides the scheduling in the control center. Therefore, it can be expressed as a multi-objective optimization problem: min f = [f1, f2].

[0027] In this process, step (2) for the input information of the present invention includes N electric vehicle nodes, and the node set information is represented as X = {x0, x1, ..., x}. n}, conventional electric vehicle node x i =(s i ,d i ,tw i ) contains two-dimensional coordinate information of electric vehicles. i =(a i ,b i ), charging demand d i Expected time window information tw i In this invention, a special node with index 0 is used to represent the charging base station, which is equivalent to the starting point of a mobile charging vehicle with a power capacity of D. Its location is in the center of the service area, i.e., the location of the control center. The demand and time window values ​​of this node are both set to null.

[0028] Furthermore, the neural network described in step (3) assigns weights to minimizing the travel distance and minimizing the time penalty cost based on the overall optimization objective and the optimization objective function. The structure and training process of the neural network are as follows:

[0029] (31) Decomposition of subproblems: The multi-objective optimization problem of mobile charging service is decomposed into M single-objective subproblems using weighted grouping techniques.

[0030] By defining a set of uniformly distributed weight vectors {λ 1 ,λ 2 ,…,λ M},in Let represent the weight vector corresponding to the i-th scalar optimization subproblem. The optimization objective formula for the i-th subproblem can be expressed as:

[0031] For each subproblem, a transformer-based deep neural network is used to model and solve it in order to obtain a local optimum.

[0032] The neural network model for the subproblem in this invention originates from the attention mechanism of the transformer, referred to as Att-Model in this invention, and follows an encoder-decoder architecture. The encoder of the Att-Model is essentially a graph attention network used to compute the embedding of each node. The decoder employs the self-attention computation method of the transformer model and enhances its performance through additional layers. This invention introduces a neural network input form for the subproblem: X = {x0, x1, ..., x...} n}, where x i =(s i ,d i ,tw i ) is the two-dimensional coordinate information s of the electric vehicle i =(a i ,b i ), charging demand d i Expected time window information tw i The tuples formed by these subproblems. The output of the neural network corresponding to the subproblems is the access sequence of electric vehicles and charging stations, i.e., ρ = {ρ1, ρ2, ..., ρ...} m}

[0033] Deep neural networks are trained using reinforcement learning algorithms to optimize the solution strategy for subproblems.

[0034] This invention employs a training mechanism combining the REINFORCE algorithm and the Rollout baseline. The network parameters of the Att-Model are denoted as θ, and the network used to evaluate the performance of the Att-Model is called the Rollout baseline network, denoted as RBL. The gradient calculation formula is as follows: in Let λ represent the objective function for optimizing the i-th subproblem. i It is the corresponding weight vector, p θ This indicates that node x is selected under this strategy. i The probability of being the next access node. RBL φ (X) is the baseline function computed by the Rollout baseline network, used to estimate the desired target value to reduce the variance of the gradient.

[0035] By passing parameters between neighbors, the solution strategies for each subproblem can cooperate with each other, thereby improving the quality of the global solution.

[0036] Each subproblem is modeled by a neural network and trained using reinforcement learning. When solving a subproblem, the parameters of the corresponding model are passed to the subproblem network model corresponding to the next adjacent weight vector group as its initial parameters. The Pareto front can be directly obtained through the trained model.

[0037] Beneficial Effects: The orchestration and management method for mobile charging vehicles to provide charging services for electric vehicles provided by this invention comprehensively considers the charging service information of the mobile charging vehicles and the charging demand information of the electric vehicles. It also leverages the algorithm of the control center to simultaneously minimize the charging path length cost and time penalty cost. Compared to existing fixed charging piles or battery swapping projects that may encounter problems such as waiting or queuing for full charge, this invention can significantly improve these issues, increase energy utilization, and enhance the flexibility and diversity of charging services. This brings more efficient and convenient charging services to the electric vehicle charging field, contributing to the development of environmentally friendly transportation. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the method described in this invention;

[0039] Figure 2(a) is a schematic diagram comparing the Pareto front obtained by using the method described in this invention and NSGA-II to solve an example containing 10 electric vehicles;

[0040] Figure 2(b) is a schematic diagram comparing the Pareto front obtained by using the method described in this invention and MOEA / D to solve an example containing 10 electric vehicles;

[0041] Figure 3(a) is a schematic diagram comparing the Pareto front obtained by using the method described in this invention and NSGA-II to solve an example containing 20 electric vehicles;

[0042] Figure 3(b) is a schematic diagram comparing the Pareto front obtained by using the method described in this invention and MOEA / D to solve an example containing 20 electric vehicles;

[0043] Figures 4(a) and 4(b) are schematic diagrams of charging path planning in the case of 10 electric vehicles and 20 electric vehicles, respectively, according to the embodiments of the present invention. Detailed Implementation

[0044] To illustrate the technical solutions disclosed in this invention in detail, further descriptions are provided below in conjunction with the accompanying drawings.

[0045] First, this invention provides a mobile charging service orchestration and management method based on deep reinforcement learning. Please refer to... Figure 1 The process shown in this invention focuses on using a transformer-based neural network to achieve fast online problem solving. The Pareto front performance obtained by this method is significantly better than that of traditional multi-objective genetic algorithms NSGA-II and MOEA / D.

[0046] The mobile charging vehicle service scheduling and management problem considered in this invention is mainly to solve the difficulty of cost control for mobile charging. By controlling the driving cost and time window penalty cost of mobile charging, the control center can effectively control the energy consumption cost of mobile charging vehicles while meeting the power demand of electric vehicle users and achieving the goal of meeting the user's expected time window as much as possible to ensure good user satisfaction.

[0047] The method described in this invention calculates a charging service sequence based on the service request information of electric vehicle users, including location information, power demand information, and expected time window information, combined with the status information of mobile charging vehicles, through the control center system. This achieves the dual objectives of minimizing the travel distance of the mobile charging vehicles and the time window penalty cost, thereby realizing the cost control strategy objective of the mobile charging service control center.

[0048] The specific steps of implementing this invention are further described below. A mobile charging vehicle service orchestration and management method based on deep reinforcement learning is as follows: Figure 1 As shown, it includes the following steps:

[0049] (1) Construct a relational network model.

[0050] In this step, a social network topology is first established between the mobile charging vehicle, electric vehicles, and the control center. The mobile charging vehicle is responsible for providing charging services to the electric vehicles, while the control center is responsible for calculating the charging scheduling order of the mobile charging vehicle and sending instructions to the mobile charging vehicle to guide it in providing charging services to the electric vehicles.

[0051] (2) Determine information and optimize the objective function.

[0052] Relevant information is determined for both mobile charging vehicles and electric vehicles, and corresponding optimization objective functions are established. This information includes the location of the charging vehicle, charging demand, and expected time window. The optimization objective function considers two factors: the energy consumption cost of the mobile charging vehicle and the time penalty cost caused by the time window limitation.

[0053] (3) Use a neural network to determine the next charging target.

[0054] Using information from mobile charging vehicles and electric vehicles as input, a neural network predicts the next charging target. The neural network is a deep neural network model based on transformers, combined with reinforcement learning training algorithms, which can output the optimal action based on the current state and policy.

[0055] (4) Perform the charging task.

[0056] Once the charging sequence for all electric vehicles is determined, the mobile charging vehicle completes the charging task according to the planned route. The execution of the charging task follows the route calculated online by the control center using a trained model.

[0057] To further explain the above steps, the relational network model constructed in step (1) has three main entities: electric vehicles, mobile charging vehicles, and the control center. Electric vehicles are the starting point of the relational network because they are the objects that require charging services. Mobile charging vehicles are the entities that provide charging services to electric vehicles. The control center is responsible for calculating the charging service sequence of the mobile charging vehicles, and it has a charging scheduling relationship with the mobile charging vehicles.

[0058] Step (2) determines the information for electric vehicles, including location information, charging demand, and expected time window. This invention represents the set of electric vehicles as X = {x0, x1, ..., x...}. n}, where each node is specifically represented as x i =(s i ,d i ,tw i It contains the two-dimensional position coordinate information s of the electric vehicle. i =(a i ,b i ), charging demand d i Information on the expected time window of electric vehicle users (tw) i .

[0059] In this invention, a mobile charging vehicle is considered to serve multiple electric vehicles. A special node x0 with index 0 represents the charging base station where the control center is located, which is also the starting point of the mobile charging vehicle. The charging base station is responsible for replacing the energy storage of the mobile charging vehicle. For x0, the values ​​of charging demand and time window are set to empty to distinguish it from the charging vehicle node. The mobile charging vehicle departs carrying a battery with a total capacity of D, following the charging service access route ρ = {ρ1, ρ2, ..., ρ...} calculated by the control center. m The mobile charging vehicle provides charging services for electric vehicles. For each electric vehicle user's charging request, the mobile charging vehicle will fulfill it in one go, so each electric vehicle will not be visited twice. During the service process, the mobile charging vehicle may return to the charging base station to replace the battery, so the charging base station may be visited multiple times. After completing all charging tasks, the mobile charging vehicle will return to the charging base station. Therefore, the first and last nodes in the complete access route are both charging base stations, and the access sequence satisfies the constraint: ρ i ≠ρ j if i≠j,ρ i ≠cd0andρ j≠cd0. The energy consumption cost of the mobile charging vehicle itself is an important cost factor that the control center needs to control. Therefore, the goal is to minimize the energy consumption cost by shortening the total travel distance of the mobile charging service route as much as possible. The corresponding optimization objective function is as follows:

[0060]

[0061] The value calculated by dist represents the target ρ being accessed. j To access target ρ j+1 The mobile charging vehicle's service will be penalized if it arrives outside the requested time window, as electric vehicle users have specified their desired charging time. The system will then compensate the user accordingly. Therefore, it is crucial to meet the time window requirements of electric vehicle users as much as possible, arriving at the electric vehicle's location promptly within the time window to minimize penalty costs.

[0062] The penalty cost mainly includes two scenarios: First, an early arrival penalty. If the mobile charging vehicle arrives at the designated location before the electric vehicle, it will choose another nearby node to charge, essentially taking an unnecessary detour, hence the penalty. Second, if the mobile charging vehicle arrives after the latest time window limit, the car may not be fully charged when the user picks it up, leading to a decreased user experience and also incurring a time penalty. Minimizing the time penalty is represented as:

[0063]

[0064] Where p e p l These represent the penalty coefficients for arriving early and late, respectively. These represent the mobile charging vehicle arriving earlier (ρ). j The time spent at the location and the time the mobile charging vehicle was late. The formula for calculating the early arrival time is: In the formula This indicates that the mobile charging vehicle has arrived at ρ. j At any given time point, the time it takes for the mobile charging vehicle to reach a different node location is equal to the sum of the mobile charging vehicle's travel time and the charging time of all electric vehicles prior to that point.

[0065]

[0066] v、d k Indicates the travel speed of the mobile charging vehicle, ρ k The charging demand is denoted by ε, where ε represents the charging rate. In the formula... Represents ρ jThe initial time point of the corresponding user-expected time window is used. If the mobile charging vehicle arrives earlier than the time window, the specific early arrival time will be calculated; otherwise, the early arrival time will be 0. Similarly, the duration of lateness The calculation formula is as follows: in Represents ρ j The corresponding expected time window's cutoff time. Specifically, if the arriving node is a charging base station, then both the early arrival duration and late arrival duration will be set to 0.

[0067] The charging service optimization objective of this invention is to plan a route while meeting users' electricity demand, achieving two optimization goals. The first goal is to minimize the total travel distance of the mobile charging vehicle, thereby reducing operating energy costs. The second goal is to minimize penalty costs, which is related to the analysis and improvement of service quality, and is an important indicator reflecting user satisfaction, as well as an important reference indicator for the control center to plan service schemes. Therefore, the charging service optimization objective of this invention can be expressed in the following form:

[0068] min f = [f1, f2]

[0069] In step 3, this invention employs weighted decomposition, transformer, and parameter transfer methods for network modeling and training. For the mobile charging service orchestration management problem addressed by this invention, two objective functions need to be optimized simultaneously. Therefore, this invention decomposes the problem into multiple sub-problems based on weighted decomposition. First, a set of uniformly distributed weight vectors {λ} is defined. 1 ,λ 2 ,…,λ M The multi-objective optimization problem is decomposed into M scalar optimization subproblems, each considering different objectives and corresponding to different combinations of objective weights. This represents the combination of weight vectors corresponding to the i-th scalar optimization subproblem. The optimization objective formula for the i-th subproblem can be expressed as:

[0070] Each subproblem is modeled using a neural network and trained and solved using reinforcement learning. First, each subproblem is modeled as a neural network model with a multi-head attention mechanism, derived from a transformer variant. The neural network is then trained using reinforcement learning combined with Rollout baselines. To accelerate training and optimize performance, neighbor-based parameter transfer is used for collaborative training of different subproblem models. When solving a subproblem, the parameters of the corresponding model are transferred to the subproblem network model in the next neighboring weight vector group as its initial parameters, thus accelerating training.

[0071] The neural network model for the subproblem is derived from a transformer variant and is referred to as Att-Model in this invention. It follows an encoder-decoder structure. The encoder in Att-Model is equivalent to a graph attention network used to compute the embedding of each node. The decoder employs the Self-attention computation method of the transformer model, adding more computational layers to improve model performance. The node information X = {x0, x1, ..., x...} is... n The input is fed into the encoder, where it first undergoes a linear transformation to obtain new node embeddings. Then, multiple attention layers within the encoder calculate the final node embeddings. The average of all final node embeddings is taken as the graph embedding. Finally, the node embeddings and graph embeddings are decoded by the decoder to obtain the network model's output sequence ρ = {ρ1, ρ2, ..., ρ...}. m The decoder consists of an improved multi-head attention mechanism and a probability computation mechanism. These two layers are repeated four times vertically. In each decoding step t∈{1,2,…,m}, the decoder needs to determine the current sequence π. (1:t-1) The remaining battery power D of the current mobile charging vehicle t Decisions are made based on the embeddings of each node and the embeddings of the entire graph. t The formula is as follows:

[0072] p i =p θ (π t =x i |π 1:t-1 ,X)

[0073] Each time a decision is made, the information of the last node involved in the calculation and the remaining power information are updated until the needs of all electric vehicles are met, at which point the output sequence can be obtained.

[0074] This invention employs a training mechanism combining the REINFORCE algorithm and the Rollout baseline. The network parameters of the Att-Model are denoted as θ, and the network used to evaluate the performance of the Att-Model is called the Rollout baseline network, denoted as RBL. The gradient calculation formula is as follows:

[0075]

[0076] Let λ represent the objective function of the i-th subproblem, which is a weighted sum of different objectives. i It is the corresponding weight vector, p θ This indicates that node x is selected under this strategy. i The probability of being the next access node. RBL φ (X) is the baseline function computed by the Rollout baseline network, which estimates the expected target value to reduce gradient variance. Initially, the parameters of both networks are randomly initialized. During training, a greedy strategy is used to select the action with the highest probability to ensure deterministic results and reduce variance. Then, gradient updates are used to update the parameters of the Att-Model. The updates to the baseline network parameters are determined by paired T-tests.

[0077] In step (4), the mobile charging vehicle receives the charging service sequence instructions calculated by the control center, goes to the requested location of each electric vehicle to provide charging services, and completes all charging service tasks.

[0078] Example

[0079] To verify the effectiveness and optimization of the present invention, a simulation experiment was conducted in this embodiment, setting up a 15x15km... 2 A square area is designated as the service area for mobile charging. Charging stations are located at the center of this area, and electric vehicles are randomly distributed across the map. Electric vehicle users submit charging requests, specifying their charging needs (between 5kWh and 20kWh) and selecting their desired time window, with a time penalty coefficient p. e and p l The settings are 5 and 10 respectively. The electric vehicle's speed is set to 60 km / h. The control center dispatches a mobile charging vehicle equipped with a 100 kWh battery storage capacity, traveling at 60 km / h and charging at 60 kW. If the mobile charging vehicle's battery capacity falls below 5 kWh or is insufficient to meet the needs of the next electric vehicle, the mobile charging vehicle will return to the charging station to replace the battery with a fully charged one, regardless of the time required for battery replacement.

[0080] Depend on Figure 2a and Figure 2bThe Pareto front comparison shown demonstrates that the method described in this invention significantly improves the minimization of both objective functions. Even with increased iterations, NSGA-II and MOEA / D did not show significant improvement, and their optimization performance was inferior to that of the method described in this invention.

[0081] Regarding computation time, the average data from 10 random experiments is shown in Table 1. When the number of iterations is 1000, the average solution time for NSGA-II is 9.494 seconds, and the average solution time for MOEA / D is 42.975 seconds. The method described in this invention can directly output results within an average of 1.3572 seconds. Moreover, for NSGA-II and MOEA / D, increasing the number of iterations leads to a multiple increase in computation time. Furthermore, the Pareto solution set obtained by the method described in this invention has a higher HV value than the other two methods, reflecting that the quality of the Pareto solution set obtained by the method described in this invention is superior to that of NSGA-II and MOEA / D.

[0082] Table 1. Computation time and HV values ​​for obtaining the Pareto front using different methods.

[0083] NSGAII-1000 9.494 484.2525 NSGAII-2000 18.433 490.7924 NSGAII-4000 36.054 484.2019 MOEAD-1000 42.975 488.4618 MOEAD-2000 81.596 483.7343 MOEAD-4000 160.368 488.5054 This invention 1.3572 796.9267

[0084] Using a model trained on instances containing 10 electric vehicles, further solve for instances containing 20 electric vehicles, such as... Figure 3a and Figure 3b As shown, the method described in this invention is still significantly superior to NSGA-II and MOEA / D. This embodiment illustrates that the method described in this invention has good generalization ability. After the model is trained, it can be used to solve problems of different scales but of the same type, while maintaining excellent performance, such as fast computation speed and higher HV level than NSGA-II and MOEA / D.

[0085] like Figure 4a and Figure 4b The charging service route shown allows the mobile charging vehicle to complete all charging tasks in one round trip for 10 electric vehicles. However, when the number of electric vehicles increased to 20, the mobile charging vehicle returned to the charging base station once during the service period to replace the battery, demonstrating the effectiveness and rationality of this embodiment.

Claims

1. A method for orchestrating and managing mobile charging services based on deep reinforcement learning, characterized by the following steps: include: (1) Establish a social relationship network topology between the mobile charging vehicle, the electric vehicle and the control center, and construct a relationship network model accordingly. In the relationship network model, the mobile charging vehicle, the electric vehicle and the control center exchange information through bidirectional communication lines. (2) Determine the optimization objective function based on the charging service information of the mobile charging vehicle and the charging demand information of the electric vehicle. The charging service information includes the location of the charging base station and the energy storage capacity carried by the mobile charging vehicle. The charging demand information includes the location of the electric vehicle, the required energy capacity, and the expected time window. This step comprehensively considers the benefits of minimizing the total length of the service path and the total cost of the time penalty, forming a total optimization objective function to achieve both optimization objectives simultaneously. The objective function corresponding to minimizing the total length of the service paths is as follows: in This indicates the charging service sequence of the mobile charging vehicle. , This represents the m-th access node, which could be an electric vehicle or a charging base station. Calculate the travel distance between two nodes; The mobile charging vehicle will meet the charging needs of each electric vehicle in one go, so each electric vehicle will not be accessed twice; during the service process, the mobile charging vehicle may return to the charging base station for battery replacement, so the charging base station may be accessed multiple times. The first and last nodes of the access route are charging base station nodes by default. The charging base station access sequence satisfies the following constraints: The objective function for minimizing the time penalty cost is expressed as follows: in , These represent the penalty coefficients for arriving early and late, respectively. , This indicates that the mobile charging vehicle arrived at the node early. Location duration, mobile charging vehicle lateness Duration of location; The calculation formula is: In the formula Indicates the arrival of the mobile charging vehicle The timing of this point in time is related to the travel time of the mobile charging vehicle and the charging time of the electric vehicle prior to it; the travel time and charging time together constitute the total time. The calculation formula is as follows: in This indicates the speed at which the mobile charging vehicle travels. Indicates the charging rate. express The initial time point of the corresponding expected time window will be used to calculate the early arrival time if the mobile charging vehicle arrives earlier than the time window; otherwise, the early arrival time will be 0. Similarly, the duration of lateness The calculation formula is as follows: ,in express The corresponding expected time window's cutoff time; The goal of the charging service is to optimize two objectives by planning the best service sequence while meeting the electricity needs of all users: The first objective is to minimize the total driving distance of the mobile charging vehicle, thereby reducing its energy consumption cost. This is denoted as... ; The second objective is to minimize penalty costs. This is a crucial indicator reflecting user satisfaction and also an important reference metric for control center orchestration, denoted as... ; The overall optimization objective of this method is expressed as: ; (3) Construct a neural network model and train it. Take charging service information and charging demand information as input, and calculate the order in which the mobile charging vehicle provides charging services to electric vehicles through the neural network. The neural network described above is based on an overall optimization objective and an optimization objective function. It assigns weights to minimizing the travel distance and minimizing the time penalty cost. The structure and training process of the neural network are as follows: (31) The multi-objective optimization problem of mobile charging service is decomposed into M single-objective sub-problems using the weighted grouping technique; This includes defining a set of uniformly distributed weight vectors. ,in Let represent the weight vector corresponding to the i-th scalar optimization subproblem. The optimization objective formula for the i-th subproblem can be expressed as: ; (32) For each subproblem, a transformer-based deep neural network is used to model and solve it in order to obtain a local optimum; The process of finding local optima includes the use of Att-Model, which follows an encoder-decoder architecture; The encoder of the Att-Model is used to compute the embedding of each node, and its decoder adopts the self-attention computation method of the transformer model and enhances its performance through additional layers. The method introduces a neural network input representation for the subproblem. ,in It is based on the two-dimensional coordinate information of the electric vehicle. Charging demand Expected time window information The tuples formed by these subproblems; the output of the neural network corresponding to the subproblems is the access sequence of electric vehicles and charging base stations, i.e. ; (33) The deep neural network is trained using reinforcement learning algorithms to optimize the solution strategy of subproblems, and the training is based on a training mechanism that combines the REINFORCE algorithm with the Rollout baseline. The network parameters of Att-Model are represented as follows: The network used to evaluate the performance of the Att-Model is called the Rollout baseline network, denoted as RBL; the gradient calculation formula is as follows: ， in Let i represent the objective function for optimizing the i-th subproblem. It is the corresponding weight vector. This indicates the selection of nodes under this strategy. The probability of being the next accessed node. It is the baseline function computed by the Rollout baseline network, used to estimate the desired target value to reduce the variance of the gradient; (34) By passing parameters from neighbors, the solution strategies of each subproblem can cooperate with each other, thereby improving the quality of the global solution; (4) The control center schedules mobile charging vehicles to provide charging services for electric vehicles according to the overall optimization objective function, and completes the charging task.

2. The mobile charging service orchestration and management method based on deep reinforcement learning according to claim 1, characterized in that: The information exchange in the relational network model in step (1) includes the sending and receiving of charging service instructions, the electric vehicle sending charging demand information to the control center, and the mobile charging vehicle providing charging services to the electric vehicle under the guidance of the control center.

3. The mobile charging service orchestration and management method based on deep reinforcement learning according to claim 1, characterized in that: In step (2), during the process of solving the objective function, the input information includes N electric vehicle nodes, and the node set information is represented as follows: Conventional electric vehicle nodes Includes two-dimensional coordinate information of electric vehicles Charging demand Expected time window information ; In the method, a special node with index 0 is defined to represent a charging base station, which is equivalent to the starting point of a mobile charging vehicle with a power capacity of D. Its location is in the center of the service area, i.e., the location of the control center, and the demand and time window values ​​of this node are both set to empty.

Images