Cooperative route-charging planning method and system based on graph reinforcement learning

Abstract

The invention relates to the technical field of electric vehicle charging planning, and discloses a cooperative route-charging planning method and system based on graph reinforcement learning, and the method comprises the steps: carrying out the integrated optimization of a route and charging planning through employing a graph neural network and a reinforcement learning algorithm; the method comprises the following steps: representing a map as graph structure data, and extracting topological information of a road network through a Straucture2Vector method; the topological information is embedded into a state value function of a deep Q network (DQN) algorithm so as to plan a route and optimize selection of charging stations; the cooperative path-charging planning method is formed by adapting to changes of traffic and charging requirements through self-learning. The method has the beneficial effects that the planning efficiency can be improved, the driving and charging waiting time can be greatly shortened, and the user experience is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric vehicle charging planning, and relates to a collaborative route-charging planning method and system based on graph reinforcement learning. Background Art

[0002] In the context of global climate change, reducing greenhouse gas emissions and achieving carbon peak and carbon neutrality has become a global priority. As the main emitter of greenhouse gases, the transportation industry faces the urgent task of reducing carbon emissions. Governments and organizations around the world have given priority to the development of new energy and clean energy vehicles. Green and low-carbon new energy vehicles have attracted global attention. Their low emissions and high efficiency are the key to reducing carbon emissions in the transportation sector.

[0003] As a new energy vehicle, electric vehicles are restricted in their promotion and application due to the problem of range anxiety. "Ranged anxiety" refers to the concerns of electric vehicle users about the unpredictable distance of a single trip. This is because electric vehicles have a limited range, requiring users to often consider charging points when planning their trips, which increases the complexity and uncertainty of travel. At the same time, in remote and inconvenient areas, the problem of insufficient coverage of charging facilities further exacerbates range anxiety. Generally speaking, optimizing the layout of charging stations and scheduling and managing electric vehicles can improve the utilization efficiency of charging facilities and the operating efficiency of electric vehicles, but these measures ignore the actual travel experience of electric vehicle users. Summary of the invention

[0004] The purpose of the present invention is to provide a collaborative route-charging planning method and system based on graph reinforcement learning, which can improve planning efficiency, greatly reduce driving and charging waiting time, and enhance user experience.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] Graph reinforcement learning is used to deal with decision-making problems on graphs. When solving problems related to paths, the input data is usually presented in the form of graphs. To process the data in the graph and retain its topological information, it is necessary to convert this information into a matrix form to adapt to the processing paradigm of reinforcement learning. To achieve this goal, graph neural network methods were introduced and combined with reinforcement learning techniques to make full use of the structural characteristics of the graph to optimize route selection.

[0007] In a first aspect, the present invention provides a collaborative route-charging planning method based on graph reinforcement learning, which uses a graph neural network and a reinforcement learning algorithm to integrate and optimize route and charging planning, including the following steps:

[0008] The TSPHS model is introduced as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model;

[0009] The acquired map is represented as graph structure data, and the topological information of the road network (the road network is the map) is extracted from the graph structure data through the graph neural network method to obtain the feature vector of each node in the map, and the feature vector can reflect the topological information of the road network. The feature vector can reflect the characteristics of the road network, that is, the topological structure, and thus retains the topological information of the road network.

[0010] Since the data form of the map information directly input is structured data, it is difficult to be directly used by the reinforcement learning method. Therefore, it is necessary to use the graph neural network method. After extracting the map, the feature vector of each node in the map is obtained. This form of information can be better used by reinforcement learning. Through this feature extraction process, reinforcement learning understands the characteristics of the map more intuitively, and the topological information of the map is therefore retained.

[0011] It should be noted that the map is obtained through the map generation method in the prior art. The map information includes the coordinates of each task node, the location of the charging station and other information. After the code is read and the point distance and other calculations are performed, it can be regarded as graph structure data.

[0012] The feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function, and the action value of the state-action function is influenced by the feature vector, thereby influencing the action selection of the intelligent agent to plan the route and optimize the charging point;

[0013] A collaborative route-charging planning method is obtained by self-learning through reinforcement learning algorithms to adapt to changes in traffic and charging demand.

[0014] In combination with the first aspect, further, each node in the map refers to each intersection and charging station. The nodes include task nodes and nodes that can provide vehicle charging, that is, the charging station is also one of the nodes.

[0015] In combination with the first aspect, the collaborative route-charging planning method is further trained to optimize weight coefficients, and the specific method is: modify different coefficients and compare the resulting effects.

[0016] In combination with the first aspect, further, a TSPHS model is introduced as a modeling scheme of the collaborative route-charging planning method, hotels are compared to charging stations, customers are compared to passengers, and the maximum daily working time of electric vehicle drivers is matched with the mileage of electric vehicles, thereby obtaining an improved TSPHS model;

[0017] The improved TSPHS model includes Nodes , which includes Charging station nodes and Task Nodes , charging station node ,Task Node , the expression of the improved TSPHS model is:

[0018]

[0019] in, Indicates that if Visited a charging station or passenger in missions , and then visited a charging station or passenger ,but ,otherwise ; Indicates that the charging station or passenger To a charging station or passenger The time required, Indicates the total number of tasks, , Represents the total number of charging station nodes, Indicates the total number of task nodes.

[0020] In combination with the first aspect, further, the topological information of the road network is extracted from the graph structure data by a graph neural network method, and a specific method for obtaining a feature vector of each node in the map is as follows:

[0021] The graph neural network method uses an embedded feature vector to represent each node in the map. ,node The eigenvector of The update formula is:

[0022]

[0023] in, is a general nonlinear mapping, Represents a node in map G The set of critical points, , represents a node, and represents a deep learning parameter used to parameterize the evaluation function Q; Representation Node With Node The weight of the edge between Representation Node Iterated The eigenvector after represents the first iteration number, Representation Node Iterated The eigenvector after Representation Node Attributes, Representation Node The feature vector of .

[0024] In combination with the first aspect, further, the feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function as follows:

[0025] Will Planned to update dimensional embedding feature vector , to parameterize , specifically:

[0026]

[0027] in, , and yes The parameterized model parameters are Represents a rectified linear unit applied element-wise to its input , Represents the variables in relu; Represents an embedded node The eigenvector of Dimensions;

[0028] The reinforcement learning algorithm adopts a deep Q network algorithm, which uses embedded nodes The eigenvector of To dynamically optimize the route and charging planning method of electric vehicles, the Q function is expressed as:

[0029]

[0030] in, represents the Q function, , yes The parameterized model parameters are represents the connection operator, Represents the embedded feature vector of all nodes in the map G after T iterations; Representation Node The embedded feature vector after T iterations; T is the number of second iterations; , Represents the set of all nodes on the map G; denotes a combinatorial structure representing a solution; Represents the solution of the improved TSPHS model, and corresponds to the reinforcement learning algorithm Indicates status; Represents the nodes on the map G, and corresponds to the nodes in the reinforcement learning algorithm Indicates the action of selection.

[0031] In combination with the first aspect, further, the definitions of the state, action and reward function in the deep Q network algorithm are as follows:

[0032] state Show on map A series of actions (nodes) performed on it; since we have encoded the nodes in the labeled map using embeddings, we can convert the state Manifested as vectors in dimensional space, allowing representation across different maps;

[0033] action Indicates the action selected by the reinforcement learning algorithm, and the selected node cannot be the current state Similarly, we will move Expressed as dimensional embedding node embedding feature vector ,This definition is applicable to maps of different scales;

[0034] The reward function of state S Defined as performing an action and transition to a new state The changes in the cost function caused are:

[0035]

[0036] With the final state The cumulative reward function for:

[0037]

[0038] in, represents the cost function, and the reward value is the change in the cost function between the two states. Indicates a map, Represents each state-action pair.

[0039] In combination with the first aspect, further, by minimizing the travel distance and penalties for charging station selection To optimize the driving route of electric vehicles, each state action Reward Function It is expressed as:

[0040] in, is the adjustment penalty term Parameter for weight in total reward.

[0041] In order for the reinforcement learning algorithm to better perform multi-objective optimization, a weight parameter is required To ensure that the two optimization objectives are of the same magnitude.

[0042] In combination with the first aspect, further, the reinforcement learning process defined above uses Step Q learning and fitting Q iterative solution.

[0043] Step Q learning solves the problem of delayed rewards, that is, the final reward that the agent is interested in is received in the future, so One-step Q-learning is a natural extension of One-step Q-learning, which waits for t before updating the parameters of the approximator. steps, thereby collecting more accurate estimates of future rewards; 1-step Q-learning updates the parameters at each step by minimizing the squared loss:

[0044]

[0045] in, From non-terminal state definition:

[0046]

[0047] The objective function of step Q learning is:

[0048]

[0049] in, express Moment status; express Always in action, express Always in action; Indicates action at the next moment possible actions to be taken; express Moment status; express Moment status; represents the target value; represents the discount factor;

[0050] The fit Q-iterative method uses experience replay to update the function approximator with a set of samples from the dataset E, rather than the single sample currently experienced; the dataset E is populated a few steps ago, so Step time, tuple is added to the dataset E, ; Perform stochastic gradient descent updates on random samples of tuples drawn from dataset E using the gradient step size in the loss of the current sample; where express Always in action; Represents the time step arrive The cumulative reward of express Always in action.

[0051] In a second aspect, the present invention provides a collaborative route-charging planning system based on graph reinforcement learning, comprising:

[0052] An improved TSPHS model module is configured to introduce a TSPHS model as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model;

[0053] A feature vector extraction module is configured to represent the acquired map as graph structure data, and extract the topological information of the road network from the graph structure data by a graph neural network method to obtain a feature vector of each node in the map;

[0054] The collaborative route-charging planning method module is configured to embed the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function, influence the action value of the state-action function through the feature vector, and then influence the action selection of the intelligent agent to plan the route and optimize the charging point, thereby obtaining a collaborative route-charging planning method.

[0055] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0056] In a fourth aspect, the present invention provides a computer device, comprising:

[0057] Memory for storing computer programs;

[0058] A processor is used to execute the computer program to implement the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0059] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) The present invention uses graph neural networks and reinforcement learning algorithms to integrate and optimize route and charging planning. By using graph reinforcement learning, a comprehensive electric vehicle path planning and charging scheduling strategy is proposed. From the perspective of user travel experience, combined with urban road networks and electric vehicle battery curves, the problem of electric vehicle path charging collaborative planning is deeply studied. By comprehensively considering factors such as user travel destination requirements, charging facility locations, and charging time, the optimal driving and charging solutions are provided to users. This not only improves planning efficiency, but also greatly reduces driving and charging waiting time, enhances user experience, and also solves the problem that most existing methods only complete path planning tasks and cannot take into account both tasks at the same time.

[0062] (2) The present invention introduces the TSPHS model as a modeling scheme for path charging planning, which provides a more comprehensive and accurate solution for electric vehicle path planning and charging strategy.

[0063] (3) This paper first applies reinforcement learning, an advanced machine learning method, to solve the TSPHS (traveling salesman problem with hotel selection) problem, demonstrating its strong adaptability and flexibility. This method can dynamically respond to changing traffic and charging demands, adjust the charging planning method accordingly, and enable the model to dynamically optimize path planning and charging strategies.

[0064] (4) The present invention adopts the graph neural network method of Structure2Vector. By converting each node in the map into a feature vector, a feature matrix that fully reflects the original map structure is constructed, providing an effective data processing tool.

[0065] (5) The present invention adapts to changes in traffic and charging demand through self-learning of the reinforcement learning algorithm, plans routes and optimizes charging points, and obtains a collaborative route-charging planning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a schematic diagram of the framework structure of the collaborative route-charging planning method proposed in Example 1 of the present invention;

[0067] Figure 2 This is a schematic diagram of the TSPHS model principle in Example 1 of the present invention;

[0068] Figure 3A schematic diagram of a road network abstracted as a map instance in Embodiment 1 of the present invention;

[0069] Figure 4 The example training results of different numbers of nodes in Example 1 of the present invention are Figure 1 ;

[0070] Figure 5 The example training results of different numbers of nodes in Example 1 of the present invention are Figure 2 ;

[0071] Figure 6 The example training results of different numbers of nodes in Example 1 of the present invention are Figure 3 . DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0073] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0074] Example 1

[0075] like Figure 1 As shown, the collaborative route-charging planning method based on graph reinforcement learning proposed in the present invention uses a graph neural network and a reinforcement learning algorithm to integrate and optimize route and charging planning, and specifically includes the following steps:

[0076] Step S1, introduce the TSPHS model as a modeling scheme for path charging planning (the goal of the TSPHS model is to minimize the number of trips and then minimize the total time spent.), compare the hotel to a charging station, and correspond the maximum daily working time to the mileage of the electric vehicle. The term "trip" is defined as a set of ordered trips covering all node visits. The model also introduces a constraint on the maximum number of working hours per day, that is, when the expected number of working hours is about to exceed the limit and the electric vehicle needs to be charged, the work must be ended and charged.

[0077] The TSPHS model consists of A collection of available hotels and Customers Definition, each customer Assigned a service time or visit time From customers To Customers Time required is known. Every trip The available time is limited to a given cost function to minimize the total cost of the entire trip. Assume that the trip ends at node The node starts and ends at the hotel on the way, and the node is still available on the way. Represents the total number of trips (i.e., the total number of tasks in the improved TSPHS model of the present invention).

[0078] The improved TSPHS model consists of a collection of charging station nodes With the task node collection Definition, where , Any task node Assigned an access time , from the node To Node The travel time is known and is expressed as . Any task The available time is limited to a given cost function so that the total cost of the entire trip (i.e., the trip distance ) minimum, Represents a collection of tasks.

[0079] The improved TSPHS model can be simplified as follows. The schematic diagram of the improved TSPHS model is as follows: Figure 2 As shown:

[0080]

[0081] in, Indicates that if Visited a charging station or passenger in missions , and then visited a charging station or passenger ,but ,otherwise ; Indicates that the charging station or passenger To a charging station or passenger the time required; Represents the total number of charging station nodes, Indicates the total number of task nodes.

[0082] Step S2, representing the map as graph structure data, and extracting the topological information of the road network through the Structure2Vector method, specifically includes the following steps:

[0083] Step S21, use the instance generator in the Center for Discrete Mathematics and Theoretical Computer Science to create Euclidean instances of clusters. The points of these instances are evenly distributed on a two-dimensional grid and organized into M / 100 different clusters. More precisely, we start with Randomly select M / 100 cluster centers in the range, where M is the number of cluster centers (i.e., the total number of cities). For each city, a cluster center is randomly assigned and two normally distributed variables (scaled and rounded) are added to the corresponding center coordinates. The distance between cities is determined using Euclidean distance and rounded to the nearest integer. This method generates a set of different instances, including 10 with 1000 cities, 5 with 3162 cities, 3 with 10000 cities, 2 with 31623 cities, 2 with 100000 cities, and a challenging single instance with 316228 cities.

[0084] Step S22, select a mapping instance as a means to abstract the intersections in the real world urban road network into nodes on a two-dimensional plane. The road network diagram abstracted into a map instance is as follows: Figure 3 As shown, by representing intersections as nodes, the basic topological and geometric information about the road network is captured while ignoring the complexity of the actual physical space.

[0085] Step S23, using the embedded feature vector To represent each node in the map ,node The eigenvector of The update formula is:

[0086]

[0087] in, is a general nonlinear mapping, such as a neural network kernel function. Represents a node in map G The set of critical points, , represents a node, and represents a deep learning parameter used to parameterize the evaluation function Q, which is used to measure the quality of the solution; Representation Node With Node The weight of the edge between Representation Node Iterated The eigenvector after represents the first iteration number, Representation Node Iterated The eigenvector after Representation Node From the feature vector As can be seen from the update formula, the embedding update process depends on the topological structure of the map. A new round of inter-node embedding updates will only begin after the embedding updates of all nodes in the previous round are completed. In the update process, a node feature is also defined through a nonlinear propagation function In addition, the more update iterations there are, the farther the node features will propagate and gather to distant nodes in a nonlinear manner. When the node is terminated after iterations, the embedding feature of each node will encapsulate its local neighborhood, which consists of the graph topology, the node characteristics involved, and the propagation function Decide.

[0088] Step S3, embedding the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function to plan the route and optimize the selection of the charging point, specifically includes the following steps:

[0089] Step S31: It is proposed to update the p-dimensional embedding , to parameterize , specifically:

[0090]

[0091] in, , and yes The parameterized model parameters are Represents a rectified linear unit applied element-wise to its input , Represents the variables in relu; Represents an embedded node The eigenvector of Dimension.

[0092] Step S32, using the deep Q network (DQN) algorithm to utilize the embedded feature vector To dynamically optimize the route and charging planning method of electric vehicles, the Q function is expressed as:

[0093]

[0094] in, represents the Q function, , It is parameterized The model parameters, represents the connection operator, Represents the embedded feature vector of all nodes in the map G after T iterations; Representation Node Embedded features after T iterations; T is the number of second iterations; , Represents the set of all nodes on the map G; denotes a combinatorial structure representing a solution; Represents the solution of the improved TSPHS model, and corresponds to the reinforcement learning algorithm Indicates status; Represents the nodes on the map G, and corresponds to the nodes in the reinforcement learning algorithm Indicates the action of selection.

[0095] Step S33, define the state, action, and reward function in the deep reinforcement learning algorithm:

[0096] State S means on the map A sequence of actions (nodes) to be performed on . Since we have encoded the nodes in the labeled graph using embeddings, the state Represented as a vector in p-dimensional space, allowing representation across different graphs.

[0097] action yes A node, not the current state Similarly, we will move Expressed as dimensional embedding feature vector , this definition is applicable to graphs of different scales.

[0098] Reward function for state S Defined as performing an action and transition to a new state The change in the cost function caused by

[0099]

[0100] With the final state The cumulative reward function for:

[0101]

[0102] in, represents the cost function, and the reward value is the change in the cost function between the two states. Indicates a map, Represents each state-action pair.

[0103] Step S34, using the minimum travel distance and penalties for charging station selection To optimize the driving route of electric vehicles, each state action The reward function is expressed as:

[0104]

[0105] in, is the adjustment penalty term Parameter for weight in total reward.

[0106] Step S4: for the reinforcement learning process defined above, use The algorithm solution of step Q learning and fitting Q iteration is as follows:

[0107] Step S41, Step Q learning solves the problem of delayed rewards, that is, the final reward that the agent is interested in is received in the future, so One-step Q-learning is a natural extension of One-step Q-learning, which waits for t before updating the parameters of the approximator. steps, thereby collecting more accurate estimates of future rewards; 1-step Q-learning updates the parameters at each step by minimizing the squared loss:

[0108]

[0109] in, From non-terminal state definition:

[0110]

[0111] The objective function of step Q learning is:

[0112]

[0113] in, express Moment status; express Always in action, express Always in action; Indicates action at the next moment possible actions to be taken; express Moment status; express Moment status; represents the target value; represents the discount factor;

[0114] Step S42, the fitting Q iteration method uses experience replay to update the function approximator with a set of samples in the data set E, rather than the single sample currently experienced; the data set E is filled in the previous steps, so in Step time, tuple is added to the dataset E, ; Perform stochastic gradient descent updates on random samples of tuples drawn from dataset E using the gradient step size in the loss of the current sample; where express Always in action; Represents the time step arrive The cumulative reward of express Always in action.

[0115] Step S5, conduct a simulation test according to the collaborative path-charging planning method to verify the performance of the collaborative path-charging planning method: determine the appropriate value of the weight coefficient, and test the adaptability of the collaborative path-charging planning method accordingly. Specifically, the following steps are included:

[0116] Step S51: adjust the penalty term Parameters of weight in total reward The values ​​are 0, 1, 2, 3, and 4 respectively, and the observation and analysis are different. The training results of the agent under the given value.

[0117] Step S52: Take the average travel reward as the observation index, output the optimized image, and compare and analyze the differences between the images with different values. When , the average travel reward has an obvious convergence law and a high convergence value. This shows that under this parameter setting, the system can optimize the travel reward more effectively, while maximizing the overall reward, and identify a route that meets the charging requirements.

[0118] Step S53, taking the SOC value, i.e. the battery status of the electric vehicle during driving, as an observation index, outputting an optimized image, and comparing and analyzing the image differences with different values. When the SOC value curve rises rapidly, it means that the system responds more quickly to the charging demand and can select a charging station for the electric vehicle in time to ensure that the battery is fully charged during driving.

[0119] Step S54, taking the vehicle travel distance as the observation index, outputting the optimized image, and comparing and analyzing the image differences with different values. In most cases, the travel distance remains relatively unchanged; The travel distance increases significantly when If the value is too large, the system may Penalizes and impairs route optimization, choosing longer but more frequent charging routes.

[0120] Step S55, to evaluate the performance of the improved TSPHS model in processing larger instances, the model is applied to three test sets of different sizes, namely test@A, test@B, and test@C. test@A has 40-50 nodes, test@B has 50-100 nodes, and test@C has 100-200 nodes. Each test set contains 1000 instances.

[0121] Step S56, compare the moving distances of the three test sets with the moving distances of the instance itself during training, and use a box plot to visualize the results. Figures 4 to 6 The results show that despite being trained on smaller instances, the improved TSPHS model is able to generate travel distances on larger instances containing 40-50 nodes, 50-100 nodes, and 100-200 nodes that are close to the training results of the instances themselves, with deviations of 3.7%, 1.1%, and 3.2%, respectively. This verifies the generalization ability of the improved TSPHS model of the present invention and the adaptability of the algorithm.

[0122] Example 2

[0123] Based on the same inventive concept as Example 1, this example introduces a collaborative route-charging planning system based on graph reinforcement learning, including:

[0124] An improved TSPHS model module is configured to introduce the TSPHS model as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model;

[0125] A feature vector extraction module is configured to represent the acquired map as graph structure data, and extract the topological information of the road network from the graph structure data by a graph neural network method to obtain a feature vector of each node in the map;

[0126] The collaborative route-charging planning method module is configured to embed the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function, influence the action value of the state-action function through the feature vector, and then influence the action selection of the intelligent agent to plan the route and optimize the charging point, thereby obtaining a collaborative route-charging planning method.

[0127] Example 3

[0128] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning are implemented.

[0129] Example 4

[0130] Based on the same inventive concept as other embodiments, this embodiment introduces a computer device, including: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning.

[0131] Example 5

[0132] Based on the same inventive concept as other embodiments, this embodiment introduces a computer program product, including a computer program, which implements the steps of the above-mentioned collaborative route-charging planning method based on graph reinforcement learning when executed by a processor.

[0133] The present invention takes electric vehicles as the research object, represents the map with graph structure data, extracts the road network topology information through the Structure2Vector method, and embeds it into the state value function of the DQN algorithm, so as to realize route planning and optimal selection of charging points. By combining the graph network structure with reinforcement learning technology, the planning efficiency is improved, the driving and charging routes are optimized, and the smooth and efficient operation of electric vehicles is ensured. The method significantly reduces the driving and charging waiting time, thereby improving the overall utilization efficiency of electric vehicles and enhancing the user experience. The algorithm includes n-step Q learning and fitting Q iteration. Simulation experiments are carried out according to the design strategy to verify the performance of the strategy: determine the appropriate value of the weight coefficient in the reward function, and test the adaptability of the design strategy accordingly. The experimental results prove the robust adaptability and flexibility of the algorithm. It can dynamically respond to changing traffic and charging needs and adjust the charging planning method.

[0134] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A collaborative route-charging planning method based on graph reinforcement learning, characterized in that: The integrated optimization of route and charging planning using graph neural network and reinforcement learning algorithm includes the following steps: The TSPHS model is introduced as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model; The obtained map is represented as graph structure data, and topological information of the road network is extracted from the graph structure data by a graph neural network method to obtain a feature vector of each node in the map; The feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function. The feature vector is used to influence the action value of the state-action function, thereby influencing the action selection of the intelligent agent to plan routes and optimize charging points, thereby obtaining a collaborative route-charging planning method.

2. The collaborative route-charging planning method based on graph reinforcement learning according to claim 1, characterized in that: The specific method of introducing the TSPHS model as a modeling scheme of the collaborative route-charging planning method is as follows: comparing a hotel to a charging station, comparing a customer to a passenger, and matching the maximum daily working time of an electric vehicle driver with the mileage of the electric vehicle to obtain an improved TSPHS model; The improved TSPHS model includes Nodes , which includes Charging station nodes and Task Nodes , charging station node ,Task Node , the expression of the improved TSPHS model is: ； in, Indicates that if Visited a charging station or passenger in missions , and then visited a charging station or passenger ,but ,otherwise ; Indicates that the charging station or passenger To a charging station or passenger The time required, Indicates the total number of tasks, , Represents the total number of charging station nodes, Indicates the total number of task nodes.

3. The collaborative route-charging planning method based on graph reinforcement learning according to claim 2 is characterized in that: The specific method of extracting the topological information of the road network from the graph structure data by using the graph neural network method to obtain the feature vector of each node in the map is as follows: The graph neural network method uses an embedded feature vector to represent each node in the map. ,node The eigenvector of The update formula is: ； in, is a general nonlinear mapping, Represents a node in map G The set of critical points, , represents a node, and represents a deep learning parameter used to parameterize the evaluation function Q; Representation Node With Node The weight of the edge between Representation Node Iterated The eigenvector after represents the first iteration number, Representation Node Iterated The eigenvector after Representation Node Attributes, Representation Node The feature vector of .

4. The collaborative route-charging planning method based on graph reinforcement learning according to claim 3 is characterized in that: The feature vector is embedded into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function as follows: Will Planned to update dimensional embedding feature vector , to parameterize , specifically: ； in, , and yes The parameterized model parameters are Represents a rectified linear unit applied element-wise to its input , Represents the variables in relu; Represents an embedded node The eigenvector of Dimensions; The reinforcement learning algorithm adopts a deep Q network algorithm, which uses embedded nodes The eigenvector of To dynamically optimize the route and charging planning method of electric vehicles, the Q function is expressed as: ； in, represents the Q function, , yes The parameterized model parameters are represents the connection operator, Represents the embedded feature vector of all nodes in the map G after T iterations; Representation Node The embedded feature vector after T iterations; T is the number of second iterations; , Represents the set of all nodes on the map G; denotes a combinatorial structure representing a solution; Represents the solution of the improved TSPHS model, and corresponds to the reinforcement learning algorithm Indicates status; Represents the nodes on the map G, and corresponds to the nodes in the reinforcement learning algorithm Indicates the action of selection.

5. The collaborative route-charging planning method based on graph reinforcement learning according to claim 4 is characterized in that: The definitions of the state, action, and reward function in the deep Q network algorithm are as follows: state Show on map A series of actions to be performed on the state; since we have encoded the nodes in the labeled map using embeddings, we can Manifested as vectors in dimensional space, allowing representation across different maps; action Indicates the action selected by the reinforcement learning algorithm, and the selected node cannot be the current state As a component of Expressed as dimensional embedding node embedding feature vector ,This definition is applicable to maps of different scales; The reward function of state S Defined as performing actions and transition to a new state The changes in the cost function caused are: ； With the final state The cumulative reward function for: ； in, represents the cost function, and the reward value is the change in the cost function between the two states. Indicates a map, Represents each state-action pair.

6. The collaborative route-charging planning method based on graph reinforcement learning according to claim 5 is characterized in that: By minimizing travel distance and penalties for charging station selection To optimize the driving route of electric vehicles, each state action Reward Function It is expressed as: ； in, is the adjustment penalty term Parameter for weight in total reward.

7. A collaborative route-charging planning system based on graph reinforcement learning, characterized in that: include: An improved TSPHS model module is configured to introduce a TSPHS model as a modeling scheme of the collaborative route-charging planning method to obtain an improved TSPHS model; A feature vector extraction module is configured to represent the acquired map as graph structure data, and extract the topological information of the road network from the graph structure data by a graph neural network method to obtain a feature vector of each node in the map; The collaborative route-charging planning method module is configured to embed the feature vector into the state-action function of the reinforcement learning algorithm to obtain the action value of the state-action function, influence the action value of the state-action function through the feature vector, and then influence the action selection of the intelligent agent to plan the route and optimize the charging point, thereby obtaining a collaborative route-charging planning method.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the collaborative route-charging planning method based on graph reinforcement learning described in any one of claims 1 to 6 are implemented.

9. A computer device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program to implement the steps of the collaborative route-charging planning method based on graph reinforcement learning according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the collaborative route-charging planning method based on graph reinforcement learning described in any one of claims 1 to 6 are implemented.

Images