A collaborative route-charging planning method and system based on graph reinforcement learning

Abstract

The application relates to the technical field of electric vehicle charging planning, and discloses a collaborative route-charging planning method and system based on graph reinforcement learning, which comprises the following steps: using a graph neural network and a reinforcement learning algorithm to perform integrated optimization on route and charging planning; representing a map as graph structure data, and extracting topological information of a road network through a Structure2Vector method; embedding the topological information into a state value function of a deep Q network (DQN) algorithm to plan a route and optimize selection of a charging station; and through self-learning, adapting to changes in traffic and charging demand to form a collaborative route-charging planning method. The application has the beneficial effects that the planning efficiency can be improved, the driving and charging waiting time can be greatly reduced, and the user experience can be enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric vehicle charging planning, and relates to a collaborative route-charging planning method and system based on graph reinforcement learning. Background Art

[0002] Against the backdrop of global climate change, reducing greenhouse gas emissions and achieving carbon peak and carbon neutrality have become urgent global priorities. As a major greenhouse gas emitter, the transportation industry faces the urgent task of reducing carbon emissions. Governments and organizations around the world have prioritized the development of new energy and clean energy vehicles. Green, low-carbon new energy vehicles have garnered global attention, with their low emissions and high efficiency holding the key to reducing carbon emissions in the transportation sector.

[0003] As a new energy vehicle, electric vehicles (EVs) are hindered in their promotion and adoption due to range anxiety. Range anxiety refers to EV users' concerns about the unpredictable distances of a single trip. This is because EVs have a limited range, requiring users to frequently consider charging points when planning their trips, adding complexity and uncertainty to travel. Furthermore, inaccessible charging infrastructure in remote and inaccessible areas further exacerbates range anxiety. Generally speaking, optimizing charging station layout and managing EV scheduling can improve the utilization of charging facilities and the operational efficiency of EVs, but these measures neglect to consider the actual travel experience of EV users. Summary of the Invention

[0004] The purpose of the present invention is to provide a collaborative route-charging planning method and system based on graph reinforcement learning, which can improve planning efficiency, greatly reduce driving and charging waiting time, and enhance user experience.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] Graph reinforcement learning addresses decision-making problems on graphs. When solving routing problems, input data is often presented in the form of graphs. To process graph data while preserving its topological information, it is necessary to convert this information into a matrix form suitable for the reinforcement learning processing paradigm. To achieve this goal, graph neural network methods have been introduced and combined with reinforcement learning techniques to fully exploit the structural properties of graphs to optimize route selection.

[0007] In a first aspect, the present invention provides a collaborative route-charging planning method based on graph reinforcement learning, which uses a graph neural network and a reinforcement learning algorithm to integrate and optimize route and charging planning, including the following steps:

[0008] The TSPHS model is introduced as a modeling scheme for the collaborative route-charging planning method to obtain an improved TSPHS model;

[0009] The obtained map is represented as graph structure data, and the topological information of the road network (i.e., the map) is extracted from the graph structure data by a graph neural network method to obtain a feature vector of each node in the map, which can reflect the topological information of the road network. The feature vector can reflect the characteristics of the road network, i.e., the topological structure, thus preserving the topological information of the road network.

[0010] Since the data form of the directly input map information is structural data, which is difficult to be directly utilized by the reinforcement learning method, the graph neural network method is used to extract the map, and the information in the form of the feature vector of each node in the map obtained after the extraction can be better utilized by the reinforcement learning. Through this feature extraction process, the reinforcement learning more intuitively understands the characteristics of the map, and the topological information of the map is thus preserved.

[0011] It should be noted that the map is obtained by a map generation method in the prior art, and the map information includes task node coordinates, charging station positions, and other information. After reading the code and performing point distance calculation, the map information can be regarded as graph structure data.

[0012] The feature vector is embedded into a state-action function of the reinforcement learning algorithm to obtain an action value of the state-action function, and the feature vector affects the action value of the state-action function, and further affects the action selection of the agent to plan a route and optimize a charging point.

[0013] The collaborative route-charging planning method is obtained by self-learning and adaptation of the reinforcement learning algorithm to changes in traffic and charging demand.

[0014] In combination with the first aspect, further, each node in the map refers to each intersection and charging station. The nodes include task nodes and nodes that can provide charging for vehicles, i.e., the charging stations are also one of the nodes.

[0015] In combination with the first aspect, further, the collaborative route-charging planning method is trained, and the weight coefficient is optimized. The specific method is to modify different coefficients and compare the effects produced.

[0016] In combination with the first aspect, further, a TSPHS model is introduced as a modeling scheme of the collaborative route-charging planning method. The hotels are compared with the charging stations, the customers are compared with the passengers, and the longest working time of the electric vehicle drivers per day is corresponded with the driving mileage of the electric vehicle to obtain an improved TSPHS model.

[0017] The improved TSPHS model includes nodes which include Charging station nodes and Task nodes , charging station node ,Task Node , the expression of the improved TSPHS model is:

[0018]

[0019] in, Indicates that if Visited a charging station or passenger in missions , and then visited a charging station or passenger ,but ,otherwise ; Indicates that the charging station or passenger to a charging station or passenger The time required, Indicates the total number of tasks, , Indicates the total number of charging station nodes, Indicates the total number of task nodes.

[0020] In combination with the first aspect, further, the specific method of extracting the topological information of the road network from the graph structure data by using the graph neural network method to obtain the feature vector of each node in the map is:

[0021] Graph neural network methods use embedded feature vectors to represent each node in the map ,node The eigenvector of The update formula is:

[0022]

[0023] in, is a general nonlinear mapping, Represents a node in map G The set of critical points, 、 represents a node, and represents a deep learning parameter used to parameterize the evaluation function Q; Representation node With node The weight of the edge between Representation node Iterated The eigenvector after the second represents the first iteration number, Representation node Iterated The eigenvector after the second Representation node Attributes, Representation node The eigenvector of .

[0024] In combination with the first aspect, further, the feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function by:

[0025] Will Planned to update dimensional embedding feature vector , to parameterize , specifically:

[0026]

[0027] in, 、 and yes The parameterized model parameters, Represents a rectified linear unit applied element-wise to its input , Represents the variables in relu; Represents an embedded node The eigenvector of Dimensions;

[0028] The reinforcement learning algorithm adopts a deep Q network algorithm, which uses embedded nodes The eigenvector of To dynamically optimize the route and charging planning method of electric vehicles, the Q function is expressed as:

[0029]

[0030] in, represents the Q function, 、 yes The parameterized model parameters, represents the connection operator, Represents the embedded feature vectors of all nodes in the map G after T iterations; Representation node The embedded feature vector after T iterations; T is the number of second iterations; , Represents the set of all nodes on the map G; denotes a combinatorial structure representing a solution; Represents the solution of the improved TSPHS model, which also corresponds to the reinforcement learning algorithm Indicates status; Represents the nodes on the map G, and corresponds to the reinforcement learning algorithm Indicates the action of selection.

[0031] In combination with the first aspect, further, the definitions of the state, action, and reward function in the deep Q network algorithm are as follows:

[0032] state Show on map A series of actions (nodes) performed on ; since we have encoded the nodes in the labeled map using embedding, we can convert the state Manifested as vectors in dimensional space, allowing representation across different maps;

[0033] action Indicates the action selected by the reinforcement learning algorithm, and the selected node cannot be the current state Similarly, we will move Expressed as dimensional embedding node embedding feature vector ,This definition is applicable to maps of different scales;

[0034] The reward function of state S Defined as performing an action and transition to a new state The changes in the cost function caused are:

[0035]

[0036] and the final state The cumulative reward function for:

[0037]

[0038] in, Represents the cost function, and the reward value is the change in the cost function between the two states. Indicates a map, Represents each state-action pair.

[0039] In combination with the first aspect, further, by minimizing the travel distance and penalties for charging station selection To optimize the driving route of electric vehicles, each state action Reward Function Expressed as:

[0040] in, is the adjustment penalty term Parameter for weight in total reward.

[0041] In order to make the reinforcement learning algorithm better perform multi-objective optimization, a weight parameter is required To ensure that the two optimization objectives are of the same magnitude.

[0042] In combination with the first aspect, further, the reinforcement learning process defined above uses Step Q learning and fitting Q iterative solution.

[0043] Step Q learning solves the problem of delayed rewards, that is, the final reward that the agent is interested in is received in the future, so One-step Q-learning is a natural extension of One-step Q-learning, which waits for t before updating the parameters of the approximator. steps, thereby collecting more accurate estimates of future rewards; 1-step Q-learning updates the parameters at each step by minimizing the squared loss:

[0044]

[0045] in, From non-terminal state definition:

[0046]

[0047] The objective function of step Q learning is:

[0048]

[0049] in, express Momentary status; express Always in action, express Moment-to-moment action; Indicates action at the next moment possible actions to be taken; express Momentary status; express Momentary status; represents the target value; represents the discount factor;

[0050] The fit Q-iterative method uses experience replay to update the function approximator with a set of samples from the dataset E, rather than the single sample currently experienced; the dataset E is populated at the previous few steps, so Step time, tuple is added to the dataset E, ; Perform stochastic gradient descent updates on a random sample of tuples drawn from dataset E using the gradient step size in the loss of the current sample; where express Moment-to-moment action; Represents the time step arrive cumulative rewards; express Always in action.

[0051] In a second aspect, the present invention provides a collaborative route-charging planning system based on graph reinforcement learning, comprising:

[0052] An improved TSPHS model module is configured to introduce the TSPHS model as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model;

[0053] a feature vector extraction module configured to represent the acquired map as graph structure data, and extract topological information of the road network from the graph structure data using a graph neural network method to obtain a feature vector of each node in the map;

[0054] The collaborative route-charging planning method module is configured to embed the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function, influence the action value of the state-action function through the feature vector, and then influence the action selection of the intelligent agent to plan the route and optimize the charging point, thereby obtaining a collaborative route-charging planning method.

[0055] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning are implemented.

[0056] In a fourth aspect, the present invention provides a computer device, comprising:

[0057] memory for storing computer programs;

[0058] A processor is configured to execute the computer program to implement the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0059] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) This invention uses graph neural networks and reinforcement learning algorithms to integrate and optimize route and charging planning. By using graph reinforcement learning, a comprehensive electric vehicle path planning and charging scheduling strategy is proposed. From the perspective of user travel experience, combined with urban road networks and electric vehicle battery curves, the problem of electric vehicle path charging collaborative planning is deeply studied. By comprehensively considering factors such as user travel destination requirements, charging facility locations, and charging time, the optimal driving and charging solutions are provided to users. This not only improves planning efficiency, but also greatly reduces driving and charging waiting time, enhances user experience, and also solves the problem that most existing solutions only complete path planning tasks and cannot take into account both tasks at the same time.

[0062] (2) The present invention introduces the TSPHS model as a modeling scheme for path charging planning, which provides a more comprehensive and accurate solution for electric vehicle path planning and charging strategy.

[0063] (3) This paper applies reinforcement learning, an advanced machine learning method, to solve the TSPHS (Traveling Salesman Problem with Hotel Selection) problem for the first time, demonstrating its strong adaptability and flexibility. The method can dynamically respond to changing traffic and charging demands, adjust the charging planning method accordingly, and enable the model to dynamically optimize path planning and charging strategies.

[0064] (4) This paper adopts the Structure2Vector graph neural network method. By converting each node in the map into a feature vector, a feature matrix that fully reflects the original map structure is constructed, providing an effective data processing tool.

[0065] (5) The present invention adapts to changes in traffic and charging demand through self-learning of the reinforcement learning algorithm, plans routes and optimizes charging points, and obtains a collaborative route-charging planning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a schematic diagram of the framework structure of the collaborative route-charging planning method proposed in Example 1 of the present invention;

[0067] Figure 2 Schematic diagram of the TSPHS model principle in Example 1 of the present invention;

[0068] Figure 3A schematic diagram of a road network abstracted as a map instance in Example 1 of the present invention;

[0069] Figure 4 The example training results of different numbers of nodes in Example 1 of the present invention are Figure 1 ;

[0070] Figure 5 The example training results of different numbers of nodes in Example 1 of the present invention are Figure 2 ;

[0071] Figure 6 The example training results of different numbers of nodes in Example 1 of the present invention are Figure 3 . DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0073] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0074] Example 1

[0075] like Figure 1 As shown in the figure, the collaborative route-charging planning method based on graph reinforcement learning proposed in the present invention uses a graph neural network and a reinforcement learning algorithm to integrate and optimize route and charging planning. Specifically, it includes the following steps:

[0076] In step S1, the TSPHS model is introduced as a modeling scheme for route charging planning (the goal of the TSPHS model is to minimize the number of trips and then the total time spent). Hotels are compared to charging stations, and the maximum daily working hours are mapped to the mileage of electric vehicles. The term "trip" is defined as an ordered set of itineraries that covers all node visits. The model also introduces a constraint on the maximum daily working hours. This means that when the expected working hours are about to exceed the limit and the electric vehicle needs to charge, the work must be terminated and charged.

[0077] The TSPHS model is composed of A collection of available hotels and Customers Definition, each customer Assigned a service time or visit time From the customer To customers Time required is known. Every trip The available time is limited to the given cost function To minimize the total cost of the entire trip. Assume that the trip ends at node The node starts and ends at the hotel, and the node is still available en route. Represents the total number of trips (i.e., the total number of tasks in the improved TSPHS model of the present invention).

[0078] The improved TSPHS model consists of a collection of charging station nodes With the task node collection Definition, where , Any task node Assigned an access time , from the node To Node The travel time is known and is expressed as Any task The available time is limited to the given cost function so that the total cost of the entire trip (i.e., the travel distance ) minimum, Represents a collection of tasks.

[0079] The improved TSPHS model can be simplified as follows. The schematic diagram of the improved TSPHS model is as follows: Figure 2 As shown:

[0080]

[0081] in, Indicates that if Visited a charging station or passenger in missions , and then visited a charging station or passenger ,but ,otherwise ; Indicates that the charging station or passenger to a charging station or passenger the time required; Indicates the total number of charging station nodes, Indicates the total number of task nodes.

[0082] Step S2: Representing the map as graph structure data and extracting the topological information of the road network through the Structure2Vector method, specifically including the following steps:

[0083] Step S21, use the instance generator in the Center for Discrete Mathematics and Theoretical Computer Science to create Euclidean instances of the cluster. The points of these instances are evenly distributed on a two-dimensional grid and organized into M / 100 different clusters. More precisely, we start from M / 100 cluster centers are randomly selected within a range, where M is the number of cluster centers (i.e., the total number of cities). For each city, a cluster center is randomly assigned, and two normally distributed variables (scaled and rounded) are added to the corresponding center coordinates. Distances between cities are determined using Euclidean distance and rounded to the nearest integer. This method generates a set of different instances, including 10 with 1,000 cities, 5 with 3,162 cities, 3 with 10,000 cities, 2 with 31,623 cities, 2 with 100,000 cities, and a challenging single instance with 316,228 cities.

[0084] Step S22, select a mapping instance as a means to abstract the intersections in the real-world urban road network into nodes on a two-dimensional plane. Figure 3 As shown, by representing intersections as nodes, the basic topological and geometric information about the road network is captured while ignoring the complexity of the actual physical space.

[0085] Step S23, using the embedded feature vector To represent each node in the map ,node The eigenvector of The update formula is:

[0086]

[0087] in, It is a general nonlinear mapping, such as a neural network kernel function. Represents a node in map G The set of critical points, 、 represents a node, and represents a deep learning parameter used to parameterize the evaluation function Q, which is used to measure the quality of the solution; Representation node With node The weight of the edge between Representation node Iterated The eigenvector after the second represents the first iteration number, Representation node Iterated The eigenvector after the second Representation node From the feature vector As can be seen from the update formula, the embedding update process depends on the topological structure of the map. A new round of inter-node embedding update will only begin after the embedding update of all nodes in the previous round is completed. In the update process, a node feature is also defined through a nonlinear propagation function The process of propagating to other nodes. In addition, the more update iterations there are, the farther the node features will propagate and gather to distant nodes in a nonlinear manner. Finally, after When the node is terminated after iterations, the embedding feature of each node will encapsulate its local neighborhood, which is determined by the graph topology, the characteristics of the nodes involved, and the propagation function Decide.

[0088] Step S3, embedding the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function to plan the route and optimize the selection of the charging point, specifically includes the following steps:

[0089] Step S31: It is proposed to update the p-dimensional embedding , to parameterize , specifically:

[0090]

[0091] in, 、 and yes The parameterized model parameters, Represents a rectified linear unit applied element-wise to its input , Represents the variables in relu; Represents an embedded node The eigenvector of dimension.

[0092] Step S32, using the Deep Q Network (DQN) algorithm to use the embedded feature vector To dynamically optimize the route and charging planning method of electric vehicles, the Q function is expressed as:

[0093]

[0094] in, represents the Q function, 、 It is parameterized The model parameters, represents the connection operator, Represents the embedded feature vectors of all nodes in the map G after T iterations; Representation node Embedded features after T iterations; T is the number of second iterations; , Represents the set of all nodes on the map G; denotes a combinatorial structure representing a solution; Represents the solution of the improved TSPHS model, which also corresponds to the reinforcement learning algorithm Indicates status; Represents the nodes on the map G, and corresponds to the reinforcement learning algorithm Indicates the action of selection.

[0095] Step S33, define the state, action, and reward function in the deep reinforcement learning algorithm:

[0096] State S means on the map A sequence of actions (nodes) to be performed on . Since we have encoded the nodes in the labeled graph using embeddings, the state Represented as a vector in p-dimensional space, allowing representation across different graphs.

[0097] action yes A node, not the current state Similarly, we will move Expressed as dimensional embedding feature vector , this definition is applicable to graphs of different scales.

[0098] Reward function for state S Defined as performing an action and transition to a new state The change in the cost function caused by

[0099]

[0100] and the final state The cumulative reward function for:

[0101]

[0102] in, Represents the cost function, and the reward value is the change in the cost function between the two states. Indicates a map, Represents each state-action pair.

[0103] Step S34, using the minimum travel distance and penalties for charging station selection To optimize the driving route of electric vehicles, each state action The reward function is expressed as:

[0104]

[0105] in, is the adjustment penalty term Parameter for weight in total reward.

[0106] Step S4: for the reinforcement learning process defined above, use The algorithm solution of step Q learning and fitting Q iteration is as follows:

[0107] Step S41, Step Q learning solves the problem of delayed rewards, that is, the final reward that the agent is interested in is received in the future, so One-step Q-learning is a natural extension of One-step Q-learning, which waits for t before updating the parameters of the approximator. steps, thereby collecting more accurate estimates of future rewards; 1-step Q-learning updates the parameters at each step by minimizing the squared loss:

[0108]

[0109] in, From non-terminal state definition:

[0110]

[0111] The objective function of step Q learning is:

[0112]

[0113] in, express Momentary status; express Always in action, express Moment-to-moment action; Indicates action at the next moment possible actions to be taken; express Momentary status; express Momentary status; represents the target value; represents the discount factor;

[0114] Step S42, the fitting Q-iterative method uses experience replay to update the function approximator with a set of samples in the dataset E, rather than the single sample currently experienced; the dataset E is filled in the previous steps, so Step time, tuple is added to the dataset E, ; Perform stochastic gradient descent updates on a random sample of tuples drawn from dataset E using the gradient step size in the loss of the current sample; where express Moment-to-moment action; Represents the time step arrive cumulative rewards; express Always in action.

[0115] Step S5: Conduct simulation tests based on the collaborative route-charging planning method to verify the performance of the collaborative route-charging planning method: determine appropriate values ​​for the weight coefficients and test the adaptability of the collaborative route-charging planning method accordingly. This specifically includes the following steps:

[0116] Step S51: Adjust the penalty item Parameters of weight in total reward The values ​​are 0, 1, 2, 3, and 4 respectively, and the observation and analysis are different The training results of the agent under the given value.

[0117] Step S52: Take the average travel reward as the observation index, output the optimized image, and compare and analyze the differences between the images with different values. When , the average travel reward has a clear convergence law and a high convergence value. This shows that under this parameter setting, the system can more effectively optimize travel rewards, while maximizing the overall reward and identifying a route that meets the charging requirements.

[0118] Step S53: Use the SOC value, i.e. the battery status of the electric vehicle during driving, as an observation index, output an optimized image, and compare and analyze the image differences with different values. When the SOC value curve rises rapidly, it means that the system responds more quickly to the charging demand and can select a charging station for the electric vehicle in time to ensure that the battery is fully charged during driving.

[0119] Step S54: Take the vehicle's travel distance as the observation index, output the optimized image, and compare and analyze the image differences with different values. In most cases, the travel distance remains relatively unchanged; The travel distance increases significantly when If the value is too large, the system may Penalizing and harming route optimization, choosing longer but more frequent charging routes.

[0120] In step S55, to evaluate the performance of the improved TSPHS model when processing larger instances, the model is applied to three test sets of different sizes: test@A, test@B, and test@C. test@A has 40-50 nodes, test@B has 50-100 nodes, and test@C has 100-200 nodes. Each test set contains 1000 instances.

[0121] Step S56, compare the moving distances of the three test sets with the moving distances of the instance itself during training, and use a box plot to visualize the results. Figures 4 to 6 The results show that despite being trained on smaller instances, the improved TSPHS model is able to generate travel distances on larger instances containing 40-50 nodes, 50-100 nodes, and 100-200 nodes that are close to the training results on the instances themselves, with deviations of 3.7%, 1.1%, and 3.2%, respectively. This validates the generalization ability of the improved TSPHS model and the adaptability of the algorithm.

[0122] Example 2

[0123] Based on the same inventive concept as Example 1, this example introduces a collaborative route-charging planning system based on graph reinforcement learning, including:

[0124] An improved TSPHS model module is configured to introduce the TSPHS model as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model;

[0125] a feature vector extraction module configured to represent the acquired map as graph structure data, and extract topological information of the road network from the graph structure data using a graph neural network method to obtain a feature vector of each node in the map;

[0126] The collaborative route-charging planning method module is configured to embed feature vectors into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function. The feature vector is used to influence the action value of the state-action function, thereby influencing the action selection of the intelligent agent to plan routes and optimize charging points, thereby obtaining a collaborative route-charging planning method.

[0127] Example 3

[0128] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning are implemented.

[0129] Example 4

[0130] Based on the same inventive concept as other embodiments, this embodiment introduces a computer device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0131] Example 5

[0132] Based on the same inventive concept as other embodiments, this embodiment introduces a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0133] This invention takes electric vehicles as the research object, represents the map with graph structure data, extracts road network topology information through the Structure2Vector method, and embeds it into the state value function of the DQN algorithm, thereby realizing route planning and optimal selection of charging points. By combining the graph network structure with reinforcement learning technology, the planning efficiency is improved, the driving and charging routes are optimized, and the smooth and efficient operation of electric vehicles is ensured. This method significantly reduces the driving and charging waiting time, thereby improving the overall utilization efficiency of electric vehicles and enhancing the user experience. The algorithm includes n-step Q learning and fitting Q iteration. Simulation experiments are carried out according to the design strategy to verify the performance of the strategy: the appropriate value of the weight coefficient in the reward function is determined, and the adaptability of the design strategy is tested accordingly. The experimental results demonstrate the robust adaptability and flexibility of the algorithm. It can dynamically respond to changing traffic and charging demands and adjust the charging planning method.

[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0138] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A collaborative route-charging planning method based on graph reinforcement learning, characterized in that: The integrated optimization of route and charging planning using graph neural networks and reinforcement learning algorithms includes the following steps: The TSPHS model is introduced as a modeling scheme for the collaborative route-charging planning method to obtain an improved TSPHS model; Representing the acquired map as graph structure data, and extracting topological information of the road network from the graph structure data using a graph neural network method to obtain a feature vector of each node in the map; The feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function. The feature vector is used to influence the action value of the state-action function, thereby influencing the action selection of the intelligent agent to plan the route and optimize the charging point, thereby obtaining a collaborative route-charging planning method; The specific method of introducing the TSPHS model as a modeling scheme for the collaborative route-charging planning method is as follows: hotels are compared to charging stations, customers are compared to passengers, and the maximum daily working hours of electric vehicle drivers are matched with the mileage of electric vehicles to obtain an improved TSPHS model; The improved TSPHS model includes nodes , which includes Charging station nodes and Task nodes , charging station node ,Task Node , the expression of the improved TSPHS model is: ； in, Indicates that if Visited a charging station or passenger in missions , and then visited a charging station or passenger ,but ,otherwise ; Indicates that the charging station or passenger to a charging station or passenger The time required, Indicates the total number of tasks, , Indicates the total number of charging station nodes, Indicates the total number of task nodes; The specific method of extracting the topological information of the road network from the graph structure data by using the graph neural network method to obtain the feature vector of each node in the map is as follows: Graph neural network methods use embedded feature vectors to represent each node in the map ,node The eigenvector of The update formula is: ； in, is a general nonlinear mapping, Represents a node in map G The set of critical points, 、 represents a node, and represents a deep learning parameter used to parameterize the evaluation function Q; Representation node With node The weight of the edge between Representation node Iterated The eigenvector after the second represents the first iteration number, Representation node Iterated The eigenvector after the second Representation node Attributes, Representation node The eigenvector of The feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function as follows: Will Planned to update dimensional embedding feature vector , to parameterize , specifically: ； in, 、 and yes The parameterized model parameters, Represents a rectified linear unit applied element-wise to its input , Represents the variables in relu; Represents an embedded node The eigenvector of Dimensions; The reinforcement learning algorithm adopts a deep Q network algorithm, which uses embedded nodes The eigenvector of To dynamically optimize the route and charging planning method of electric vehicles, the Q function is expressed as: ； in, represents the Q function, 、 yes The parameterized model parameters, represents the connection operator, Represents the embedded feature vectors of all nodes in the map G after T iterations; Representation node The embedded feature vector after T iterations; T is the number of second iterations; , Represents the set of all nodes on the map G; denotes a combinatorial structure representing a solution; Represents the solution of the improved TSPHS model, which also corresponds to the reinforcement learning algorithm Indicates status; Represents the nodes on the map G, and corresponds to the reinforcement learning algorithm Indicates the action of selection; The definitions of the state, action, and reward function in the deep Q network algorithm are as follows: state Show on map A series of actions to be performed on ; since we have encoded the nodes in the labeled map using embeddings, we can convert the state Manifested as vectors in dimensional space, allowing representation across different maps; action Indicates the action selected by the reinforcement learning algorithm, and the selected node cannot be the current state As a component of Expressed as dimensional embedding node embedding feature vector ,This definition is applicable to maps of different scales; The reward function of state S Defined as performing an action and transition to a new state The changes in the cost function caused are: ； and the final state The cumulative reward function for: ； in, Represents the cost function, and the reward value is the change in the cost function between the two states. Indicates a map, Represents each state-action pair; By minimizing travel distance and penalties for charging station selection To optimize the driving route of electric vehicles, each state action Reward Function Expressed as: ； in, is the adjustment penalty term Parameter for weight in total reward.

2. A collaborative route-charging planning system based on graph reinforcement learning, characterized in that: The method for implementing the collaborative route-charging planning method based on graph reinforcement learning according to claim 1 comprises: An improved TSPHS model module is configured to introduce the TSPHS model as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model; a feature vector extraction module configured to represent the acquired map as graph structure data, and extract topological information of the road network from the graph structure data using a graph neural network method to obtain a feature vector of each node in the map; The collaborative route-charging planning method module is configured to embed the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function, influence the action value of the state-action function through the feature vector, and then influence the action selection of the intelligent agent to plan the route and optimize the charging point, thereby obtaining a collaborative route-charging planning method.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the collaborative route-charging planning method based on graph reinforcement learning described in claim 1 are implemented.

4. A computer device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the collaborative route-charging planning method based on graph reinforcement learning according to claim 1.

5. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the collaborative route-charging planning method based on graph reinforcement learning described in claim 1 are implemented.

Images